Some minor langague and formatting fixes to sections: Proxmox VE
Integration, pxar Command Line Tool, Managing Remotes, Maintenance
Tasks, Host System Administration, Network Management, and Technical
Overview.
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
also remove todo item for scheduling garbage collect with cron, and add
note about schedule configuration through proxmox-backup-manager/PBS GUI
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
Update GUI section and GUI instructions to reflect current layout and
features
List OpenID connect in possible realms (user management)
Link Access Control section when referring to it (user management)
Include Tape roles in access control section
Minor formatting changes
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
Some minor changes to the sections: Introduction, Installation,
Terminology, GUI, Storage, and User Management
Mention tape backup in main features
Update epilog.rst with link for 'LXC'.
Remove FIXME from epilog.rst (I believe this was a note to repair
the not-yet-created pbs wiki link).
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
send_command serializes everything so it cannot be used to send a
raw, optimized command. Normally that means we get an error like
> 'unable to parse parameters (expected json object)'
when used that way.
Switch over to send_raw_command which does not re-serializes the
command.
Fixes: 45b8a032 ("refactor send_command")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this is a helper that removes task log files that are not referenced
by the task archive anymore
it gets the oldest task archive file, gets the first endtime (the
oldest) and removes all files in the taskdir where the mtime is older
than that
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when we trigger the first load before the panel was fully created,
there was no load mask for it (but the snapshots would "pop in" on load)
move the first reload into the 'activate' listener. this will be called
the every time a user opens the content tab of a datastore, so guard
it by a 'firstLoad' bool.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
else it's rather to subtle and not a nice interface considering that
we only want to have a thin wrapper for sd_notify_barrier..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
until now, we manually polled the systemd service state during a reload
so that the sd_notify messages get processed in the correct order
(RELOAD(old) -> MAINPID(old) -> READY(new))
with systemd >= 246 there is now 'sd_notify_barrier' which
blocks until systemd processed all prior messages
with that change, the daemon does not need to know the service name anymore
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
With the merger the shop got moved from shop.maurer-it to
shop.proxmox.com, while we transparently redirect we also want to
stop doing that in a few years, so use new domain.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It also used `CertInfo` from pbs-tools which is also server
specific.
The original helper is now in the main crate's
client_helpers instead.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
just in case we ever need any of them in async code that
requires them and loses it because of accessing such a trait
object...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
show how to generally start a daemon that serves a rest api + index page
api calls are (prefixed with either /api2/json or /api2/extjs):
/ GET listing
/ping GET returns "pong"
/items GET lists existing items
POST lets user create new items
/items/{id} GET returns the content of a single item
PUT updates an item
DELETE deletes an item
Contains a small dummy user/authinfo
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).
this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.
The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
only bit 0-2 are fatal errors, bit 3-7 are used to indicate
some drive conditions. for details see the manpage of smartctl(8)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ Thomas: resolved merge-conflict due to moved run_command ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
In order to avoid name conflicts with WorkerTaskContext
- renamed WorkerTask::log to WorkerTask::log_message
Note: Methods have different fuction signatures
Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.
Use the task_log!() and task_warn!() macros more often.
And application now needs to call init_worker_tasks() before using
worker tasks.
Notable changes:
- need to call init_worker_tasks() before using worker tasks.
- create_task_log_dirs() ís called inside init_worker_tasks()
- removed UpidExt trait
- use atomic_open_or_create_file()
- remove pbs_config and pbs_buildcfg dependency
it's only used for generating the docs for the interactive-shell
parts of the client.
Ideally we'd avoid that whole separate binary in the first place and
let the client dump it, but we'd need to have some more elaborate
"hide this command from the help/usage" mechanisms in the CLI
helper/formatter code to make that play out more nicely.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
In our simple use cases they both should generate the same code, see
[0] for notable differences. While we cannot drop proc-macro due to
that switch, all of our dependencies that use pinning already use
pin-project-lite, so this allows us to drop a whole crate in general
while not loosing anything.
[0]: https://github.com/taiki-e/pin-project-lite#pin-project-vs-pin-project-lite
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Text-width should be 80 cc in the docs.
Avoid using relative paths in examples, they only confuse users as
one has less of a specific idea what the example may do. Rather use a
"descriptive" example path.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to have the proper link between the token list and the sub routes
in the api, include the 'tokenname' property in the token listing
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this provides some generic api call mechanisms like pvesh/pmgsh.
by default it uses the https api on localhost (creating a token
if called as root, else requesting the root@pam password interactively)
this is mainly intended for debugging, but it is also useful for
situations where some api calls do not have an equivalent in a binary
and a user does not want to go through the api
not implemented are the http2 api calls (since it is a separate api an
it wouldn't be that easy to do)
there are a few quirks though, related to the 'ls' command:
i extract the 'child-link' from the property name of the
'match_all' statement of the router, but this does not
always match with the property from the relevant 'get' api call
so it fails there (e.g. /tape/drive )
this can be fixed in the respective api calls (e.g. by renaming
the parameter that comes from the path)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we want to add something to it that needs access to the
proxmox_backup::api2 stuff, so it cannot live in a sub crate
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
basically a (semantic) revert of commit
991be99c37 "buildsys: workaround
linkage issues from openid/curl build server stuff separate"
This is no longer required because we moved proxmox_restore_daemon
code into extra crate (previous commit)
Originally-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Handle auth logs the same way as access log.
- Configure with ApiConfig
- CommandoSocket command to reload auth-logs "api-auth-log-reopen"
Inside API calls, we now access the ApiConfig using the RestEnvironment.
The openid_login api now also logs failed logins and return http_err!(UNAUTHORIZED, ..)
on failed logins.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This need impl UserInformation for Arc<CachedUserInfo> which is implemented
with proxmox 0.13.2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
ApiConfig: avoid using pbs_config::backup_user()
CommandoSocket: avoid using pbs_config::backup_user()
FileLogger: avoid using pbs_config::backup_user()
- use atomic_open_or_create_file()
Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
so that we actually have the property that 'match_all' refers to for
the templated API path.
This is mostly for improving usage of the WIP pbs-shell, i.e., its
`ls` command, it has no other functional/semantic impact.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The datastore/backup debug helpers should always be available, they
can help a lot in dire times, so making them available directly via
the server package (alongside the manager CLI tool) is nicer for the
user.
Additionally, building a package can be quite time consuming in this
repo, as some tools like dwarves and other debug symbol stuff has to
scan the quite big rust binaries. So dropping a binary package shaves
of a noticeable bit of build time too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
locking during the tests as regular user failed because we try to
chown to the backup user (which is not always possible).
Instead, do not lock at all, by implementing 'open_backup_lockfile' with
'create_mocked_lock'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
by making the field an option and making it None in the mocked case
this function is only intended for testing and hidden from the docs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
seems like there was a typo in these from the beginning.
also fixes the wrong function name for proxmox-file-restore completion
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
This also moves a couple of required utilities such as
logrotate and some file descriptor methods to pbs-tools.
Note that the logrotate usage and run-dir handling should be
improved to work as a regular user as this *should* (IMHO)
be a regular unprivileged command (including running
qemu given the kvm privileges...)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Adds possibility to recover data from an index file. Options:
- chunks: path to the directory where the chunks are saved
- file: the index file that should be recovered(must be either .fidx or
didx)
- [opt] keyfile: path to a keyfile, if the data was encrypted, a keyfile is
needed
- [opt] skip-crc: boolean, if true, read chunks wont be verified with their
crc-sum, increases the restore speed by a lot
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Adds possibility to inspect .blob, .fidx and .didx files. For index
files a list of the chunks referenced will be printed in addition to
some other information. .blob files can be decoded into file or directly
into stdout. Without decode the tool just prints the size and encryption
mode of the blob file. Options:
- file: path to the file
- [opt] decode: path to a file or stdout(-), if specidied, the file will be
decoded into the specified location [only for blob files, no effect
with index files]
- [opt] keyfile: path to a keyfile, needed if decode is specified and the
data was encrypted
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Adds possibility to inspect chunks and find indexes that reference the
chunk. Options:
- chunk: path to the chunk file
- [opt] decode: path to a file or to stdout(-), if specified, the
chunk will be decoded into the specified location
- [opt] digest: needed when searching for references, if set, it will
be used for verification when decoding
- [opt] keyfile: path to a keyfile, needed if decode is specified and
the data was encrypted
- [opt] reference-filter: path in which indexes that reference the
chunk should be searched, can be a group, snapshot or the whole
datastore, if not specified no references will be searched
- [default=true] use-filename-as-digest: use chunk-filename as digest,
if no digest is specified
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
two things wrong with the old code:
* the sort function wants -1, 0 and 1 as a return value for a<b, a==b and a>b
respectively, not a bool (which a < b returns)
* we have to sort the newest backups first, since the first reason is
'keep-last'. until now, we sorted the oldest backup first, resulting
in the older backups getting the 'keep-last' reason
reported by a user in the forum:
https://forum.proxmox.com/threads/prune-ui-and-prune-schedule-simulator-dont-match.94944/
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Defined a new struct RemoteConfig (without name and password). This makes it
possible to bas64-encode the pasword in the config, but still allow plain
passwords with the API.
otherwise a user might get a task log like this:
-----
...
found 7 groups
TASK OK
-----
which could confuse the users as why there were no snapshots backed up
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it seems that for some actions or in some circumstances, two minutes is
simply too short and the command aborts. Increase the default timeout to
10 minutes.
While it should give most commands enough time to finish, in case of a real
failure the procedure now takes up to 5 times longer, but IMHO thats an
OK tradeoff.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this should make the api call much faster, since it is not reading
the whole catalog anymore
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
For some parts of the ui, we only need the snapshot list from the catalog,
and reading the whole catalog (can be multiple hundred MiB) is not
really necessary.
Instead, we write the list of snapshots into a seperate .index file. This file
is generated on demand and is much smaller and thus faster to read.
a test for a valid status_page, one with excess data
(in the descriptor as well in the page as a whole)
and a test with too little data
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if the library sends more data than advertised, simply cut it off,
but if it sends less data, bail out (depending on how much data is
missing, trying to parse it could lead to a panic, so bail out early)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in 'restore_archive', we reach that 'catalog.commit()' for
* every skipped snapshot (we already call 'commit_if_large' then before)
* every skipped chunk archive (no change in catalog since we do not read
the chunk archive in that case)
* after reading a catalog (no change in catalog)
in all other cases, we call 'commit_if_large' and return early,
meaning that the 'commit' there was executed too often and
unnecessary, so move it after the loop over the files, before
finishing the temporary database.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of having a public start/end_chunk_archive and register_chunks,
simply expose a 'register_chunk_archive' method since we always have
a list of chunks anywhere we want to add them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of having the grid be as tall as possible and the containing
panel scroll. limit the grids height to the panel size and scroll the
grid.
this has two advantages:
* if a user has many slots, it is now possible to to navigate the other
grids to the position wanted
* having the grids scroll, means it can use extjs' buffered renderer,
which makes the view much more responsive (in case of hundreds of
slots)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
We get the descriptor length from the library and use that in
'chunks_exact', which panics on length 0. Catch that case
and bail out, since that makes no sense here anyway.
This could prevent a panic, in case a library sends wrong data.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
debugging history showed that its surely nice to have more logs at
when stuff happens (and thus fails)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
now required as we always enforce lock files to be owned by the
backup user, and the restore code uses such code indirectly as the
REST server module is reused from proxmox-backup-server. Once that is
refactored out we may do away such things, but until then we need to
have a somewhat complete system env.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead of 'blindly' trusting the changer to deliver the fields written
in the specification, trust the length data it returns in the header.
we slice the descriptor data into equal sized chunks of the correct
size, then we do not have care bout the len and empty checks anymore
this also makes the code to read the rest of the page obsolete,
since the next descriptor is on the correct offset anyway
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
whether the kernel allows super-long names or weird
namespace prefixes is not our concern...
also the latter fails under fakeroot
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
We want to allow passing a secret not only directly through the
environment value, but also indirectly through a file path, an open
file descriptor or a command that can write it to standard out.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to skip test cases for faster builds or in case your local system does
not support running (all) tests..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to 255. 8 drives per changer was a rather arbitrary limitation and could
well be reached in practice with big libraries.
Altough 255 is still a arbirtrary limitation, this is much less likely
to be reached in practice.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
pbs-datastore now ended up depending on tokio after all, but
that's fine for now
for the fuse code I added pbs-fuse-loop (has the old
fuse_loop and its 'loopdev' module)
ultimately only binaries should depend on this to avoid the
library link
the only thins remaining to move out the client binary are
the api method return types, those will need to be moved to
pbs-api-types...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
dh_auto_test also checks for the build flags used, including any
`--cfg`, so it rebuilds and overwrites our carefully assembled daemon
binaries with openid support as it is run after build and before
install.
So manually ensure the order of first test then build (argh, hackes
of hackes >.<)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Factor out open_backup_lockfile() method to acquire locks owned by
user backup with permission 0660.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While this slightly duplicates code we just do not profit from the
central, lazy static variant here, as that is only really useful in
daemons to avoid doing frequent syscalls there.
proxmox just pull in far to much (e.g., tokio) and duplicating that
one line of simple code has no real maintenance cost, so just go for
that and use the nix crate directly.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we are not actually pruning the whole datastore, but only the single
group, so set that as a title
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since the api call always starts a real worker, we cannot have a
preview. It would also be very hard to show that for all groups in a
non-confusing way. We reuse the pbsPruneInputPanel and add the dry-run
field there conditionally.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to prune the whole datastore at once, with the given parameters.
We need a new api call since this can take a while and we need to start
a worker for this. The exisiting api call returns a list of removed/kept
snapshots and is synchronous.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
checks for PRIV_DATASTORE_MODIFY, or else if the auth_id is the backup
owner, and skips the group if not.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it is the same as when pruning single groups.
for prune_jobs, we never start the worker if there is no prune option set.
but if we want to call 'prune_datastore' from somewhere else, we
have to check it here again
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by using the api macro and reusing the PruneOptions from pbs-datastore
this means we can now drop the 'add_common_prune_prameters' macro
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by using the api macro on the async method and reusing the PruneOptions
from pbs-datastore with 'flatten: true'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
some libraries cannot handle a request with volume tags and DVCID set at
the same time.
So we make 2 separate requests and merge them, since we want to keep
the vendor/model/serial data.
to not overcomplicate the code, add another special type to ElementType
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The previous assumption was that the Tasks returned by the Iterator are
sorted by the starttime, but that is not actually the case, and
could never have been, since we append the tasks into the log when
they are finished (not started) and running tasks are always iterated
first.
To correctly filter (and simplify the the api call) we forgo the
combinators, and use a for loop instead. This way we only have to do
the since/until checks only once per Task, but have to do the
start/limit counting ourselves.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
LVM replaces any dashes '-' in an LV or PV name with two '--' for the
created device node in /dev/mapper/ to distinguish the seperating
character between the PV and LV name.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
This lock is held during VM startup, so that multiple calls will not
start VMs twice. But this means that the timeout needs to incorporate
the time it might take a VM to boot, so increase it quite a bit.
This could previously lead to "interrupted system call" errors when
accessing backups with many disks.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
if an error occurs, the snapshot dirs will already be created, and we
do not clean them up (some might already be finished).
Warn the user that they are not cleaned up.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
otherwise this context is missing in some tasks (e.g. tape restore)
and it is unclear where it came from
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
a single catalog can be over 100MiB, and a media-set can have multiple
catalogs to read (no technical upper limit). On slow disks, this can
take much longer than 30 seconds (the default timeout).
The real solution would be to have some kind of index only for the gui
relevant part, e.g. a table in the beginning of the catalog, or
alternatively a seperate file with that info. Until we have such a
solution increase the timeout as a stopgap.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
According to crypt(3):
"crypt places its result in a static storage area, which will be
overwritten by subsequent calls to crypt. It is not safe to call crypt
from multiple threads simultaneously."
This means that multiple login calls as a PBS-realm user can collide and
produce intermittent authentication failures. A visible case is for
file-restore, where VMs with many disks lead to just as many auth-calls
at the same time, as the GUI tries to expand each tree element on load.
Instead, use the thread-safe variant 'crypt_r', which places the result
into a pre-allocated buffer of type 'crypt_data'. The C struct is laid
out according to 'lib/crypt.h.in' and the man page mentioned above.
Use the opportunity and make both arguments to the rust 'crypt' function
take a &[u8].
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Some changers do not like the DVCID bit when querying non-drives,
this includes when querying 'all' elements.
To circumvent this, we query each type by itself (like mtx does it),
and only add the DVCID bit for drives (Data Transfer Elements).
Reported by a user in the forum:
https://forum.proxmox.com/threads/ibm-3584-ts3500-support.92291/
and limit to 1000 elements per request.
(Because some changers limit that request with the options we set)
instead of checking if the data len was equal to the allocation_len
for getting more data, we count the returned elements and compare
that with the number we requested
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
New kernel has stricter checks on tmpfs with stick-bit on directories, so some
commands (i.e. proxmox-tape changer status) fails when executed as root, because
permission checks fails when locking the drive.
This patch move the drive locks to /run/proxmox-backup/drive-lock.
Note: This is incompatible to old locking mechmanism, so users may not
run tape backups during update (or running backup can fail).
Make docs target depend directly on the some docs-only required
binaries and add a new intermediate ".do-cargo-build" target that is
explicitly not a PHONY target.
That avoids one extra set of full builds.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
as those have a hover effect and use dark-grey vs. the quite "harsh"
looking plain black. We need to override the margin though, as else
the floated layout adds another line.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
eslint is configured to not allow using quoted object keys if they
could be just passed in dot notation, e.g.,
wrong: `group["comment"]`
good: `group.comment`
It's not a big problem but eslint fails the build with the wrong one,
so this needs to be fixed anyway..
Also, rewrite to async, shorter and less indentation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Currently done a little bit hacky in a seperate API call following the
initial list_snapshots, as we previously didn't call list_groups at all
and instead calculated the groups from the snapshots.
This calls it async and updates the view with group comments when data
arrives. The editor is simply reused with the 'group-notes' API call,
since the semantics are the same.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Stored in atomically-updated 'notes' file in backup group directory.
Available via dedicated GET/PUT API calls, as well as the first line
being included in list_groups (similar to list_snapshots).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Mostly copied from PVE, slightly adapted to be consistent with other
things in the dashboard, e.g. use a store for the repository info.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
this changes the node info panel to a similar layout as in pve,
with the ksm sharing and version field removed
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that Dashboard.js will be less cluttered when we add more information
there.
No functional change, but reworked the fingerprint button disabling to
use a property of the view instead of a viewmodel
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
While it would be nice to be able to resize that window for more
snapshots/datastores in view, this would need quite some reworking on the
input panel side. So for now, disable resizing of that window, otherwise
the grids look weird as they only scale horizontally but not vertically.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since we cannot do anything with a selected row anyway, simply
disallow it
this avoids having the row in the same color as the progressbar, without
being able to deselect the row again
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this blows up build times, but we do not plan for using it longer
than required (i.e., the server is finally split into its own binary
crate providing only those binaries).
Note, using `cargo b --release` to build is naturally unaffected by
this change, so for dev builds just continue to use that.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
a match expresses the fallback slightly nicer and needs no mut,
which is always nice to avoid.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Moving icons around is not to ideal for people accustomed to the old
ones, at least if they are used for a new component on the same view.
Rather use the address-book icon, which is also used for adding a new
realm in PVE, we can rather switch over PVE to that and the text
"Realms", as that is also the label one sees when logging in, so a
better fit to keep that consistent.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The api2 one passes the whole response (for more flexibility) on
reject, so we need to adapt to that.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
so that user can add/edit/delete realms
changes the icon of tfa to 'id-badge' so that we can keep the same icon
for authentication as pve and not have duplicate icons
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
modeled like our other section config api calls
two drawbacks of doing it this way:
* we have to copy some api properties again for the update call,
since not all of them are updateable (username-claim)
* we only handle openid for now, which we would have to change
when we add ldap/ad
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
these will be used as parameters/return types for the read/create/etc.
calls for realms
for now we copy the necessary attributes (only from openid) since
our api macros/tools are not good enought to generate the necessary
api definitions for section configs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it's not used by the client and not part of the client, it
just makes use *of* the client, but is used on the
datastore/server...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
So callers get more stable results. Most noticeable, the disk list in
the web UI doesn't jump around upon reloading, and while sorting could
be done directly there, like this other callers get the benefit too.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
in preparation to also get the file system type from lsblk.
Co-developed-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
While the PVE one "bails" too, it has an eval around those and moves
the error to the message property, so lets do so too to ensure a user
can force an update on a too old subscription
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since extjs 7.0 those will get picked up by our query logic and
sent to the backend. prevent that by setting isFormField to false
(we assemble the values differently)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the systemd config/unit parsing stays in pbs for now since
that's not usually required and uses our section config
parser
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
move key_derivation to pbs-datastore
pbs-api-types should only contain "basic" types which
* are usually required by clients
* don't depend on pbs-related code directly
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These are mostly tokio specific "hacks" or "workarounds" we
only really need/want in our binaries without pulling it in
via our library crates.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Similar like we did for Proxmox VE's manager. The main title and
version should stand a bit more out compared to simple nav/button
texts.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
one really does not need a if and an extra intermediate variable for
assigning a simple bool...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the dns plugin config allow for a specified amount of time to wait for
the TXT record to be set and propagated through DNS.
This patch adds a sleep for this amount of time.
The log message was taken from the perl implementation in proxmox-acme
for consistency.
Tested with the powerdns plugin in my test setup.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
During startup most of the stuff is happening in milliseconds (or
less), so the timestamp granularity of seconds made it hard to tell
if the previous command required 990ms or 1ms, which is quite the
difference in the restore daemon context.
Using micros seems not to bring too much additional information, a
millisecond is already an ok lower time resolution for logging, so
switch only to millis for now.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
fixes file restore again.
The new Memcom tracking file lives in `/run/proxmox-backup` and is
always created on REST interaction, as CachedUserInfo uses it to
efficiently track config changes, and such a cache is used in each
REST handle_request.
Further, the Memcom infra expects the base run PBS dir to exists
already, which is an OK assumption to have, but in the file-restore
daemon we have a significantly more minimal environment, and the run
dir was simply not required there, even /run isn't a tmpfs yet.
Fixes fda19dcc6f ("fix CachedUserInfo by using a shared memory version counter")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We send it already to the user via the response body, but the
log_response does not has, nor wants to have FWIW, access to the
async body stream, so pass it through the ErrorMessageExtension
mechanism like we do else where.
Note that this is not only useful for PBS API proxy/daemon but also
the REST server of the file-restore daemon running inside the restore
VM, and it really is *very* helpful to debug things there..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
> requires a Breaks on the old restore image (else the restore daemon
> crashes because of missing lock/LVM support).
- F.G., mailing list
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Parses JSON output from 'pvs' and 'lvs' LVM utils and does two passes:
one to scan for thinpools and create a device node for their
metadata_lv, and a second to load all LVs, thin-provisioned or not.
Should support every LV-type that LVM supports, as we only parse LVM
tools and use 'vgscan --mknodes' to create device nodes for us.
Produces a two-layer BucketComponent hierarchy with VGs followed by LVs,
PVs are mapped to their respective disk node.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Reviewed-By: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Prefix zpool mount paths to avoid clashing with other mount namespaces
(like LVM).
Also ignore "already-mounted" error and return it as success instead -
as we always assume that a mount path is unique, this is a safe
assumption, as nothing else could have been mounted here.
This fixes an issue where a mountpoint=legacy subvol might be available
on different disks, and thus have different Bucket instances that don't
share the mountpoint cache, which could lead to an error if the user
tried opening it multiple times on different disks.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Reviewed-By: Fabian Grünbichler <f.gruenbichler@proxmox.com>
otherwise the path ends in an array ["foo", "bar"] instead of "foo/bar"
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Reviewed-By: Fabian Grünbichler <f.gruenbichler@proxmox.com>
To support nested BucketComponents, it is necessary to dedup them, as
otherwise two components like:
/foo/bar
/foo/baz
will result in /foo being shown twice at the first hierarchy.
Also make the size property based on index and optional, as for example
/foo in the example above might not have a size, and bar/baz might have
differing sizes.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Reviewed-By: Fabian Grünbichler <f.gruenbichler@proxmox.com>
since it pulls in lots of additional linked libraries for all binaries
compiled as part of proxmox-backup. it can easily be re-enabled with
`--cfg openid` added to the RUSTFLAGS env variable.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it's not really needed in the config module, and this makes it easier to
disable the proxmox-openid dependency linkage as a stop-gap measure.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
similar to what was done in PVE.
- factor out openid_login_param to widget-toolkit as
getOpenIDRedirectionAuthorization and use it
- use camel case to match our JS style guide and our framework (and
basically the rest of the JS world)
- minor cleanups like moving variable definition into the single if
branch their used
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we try to load the correct media in a loop until we find the correct tape.
when encountering an error or wrong tape, we want to log that (and send
an email if one is set) that requests the correct tape.
while trying to avoid printing the same errors more than once in a row,
we had at least one case (starting with an empty tape in the drive)
which would not print/send any tape request.
reworking that code to use a custom 'TapeRequest' enum, which contains
the state + error message, and a helper that prints and sends an email
when the state changes
this reduces the change check/log to a single variable, instead of 4
(tried, last_media_uuid, last_error, failure_reason)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Add test code to the first locate_file command, compute locate_offset.
Subsequent locate_file commands use that offset.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
from proxmox-widget-toolkit-dev and not as normal dependency,
else we would have to ship widget-toolkit on the wiki
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in extjs 7.0, specifying displayField overwrites the displayTpl,
which we want to use here, so remove it
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we have a static list of filesystems and their capabilities regarding
file attributes and fs features (e.g. sockets/fifos/etc) which also
includes xattrs,acls and fcaps
if we did not know a filesystem by its magic number (for example cephfs),
we did not even attempt to read xattrs, etc.
this patch adds those flags by default to unknown filesystems, and
removes them when we encounter EOPNOTSUPP (to remove the number
of syscalls)
with this, we should be able to catch xattrs/acls/fcaps on all
(unknown) fs types that support them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
These require mounting using the regular 'mount' syscall.
Auto-generates an appropriate mount path.
Note that subvols with mountpoint=none cannot be mounted this way, and
would require setting the mountpoint property, which is not possible as
the zpools have to be imported with readonly=on.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Uses the ZFS utils to detect, import and mount zpools. These are
available as a new Bucket type 'zpool'.
Requires some minor changes to the existing disk and partiton detection
code, so the ZFS-specific part can use the information gathered in the
previous pass to associate drive names with their 'drive-xxxN.img.fidx'
node.
For detecting size, the zpool has to be imported. This is only done with
pools containing 5 or less disks, as anything else might take too long
(and should be seldomly found within VMs).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Even through best efforts at keeping it small, including the ZFS tools
in the initramfs seems to have exhausted the small overhead we had left
- give it a bit more RAM to compensate.
Also disable the ZFS ARC, as it's no use in such a memory constrained
environment, and we cache on the QEMU/rust layer anyway.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
The future needs to be removed from the pending map in any case, even if
it returned an error, else all upcoming calls to access this key will
always return the same error.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
sort the chunks we want to backup to tape by inode, to gain some
speed on spinning disks. this is done per index, not globally.
costs a bit memory, but not too much, about 16 bytes per chunk which
would mean ~4MiB for a 1TiB index with 4MiB chunks.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that we can reuse that information
the removal of the adding to the corrupted list is ok, since
'get_chunks_in_order' returns them at the end of the list
and we do the same if the loading fails later in 'verify_index_chunks'
so we still mark them corrupt
(assuming that the load will fail if the stat does)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since the output:
Result: "<UPID>"
is not really interesting, show instead the task log while
the datastore is creating, since it is now run in a worker
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Setting this to 0 is not just useless, but breaks the logic horribly
enough to cause random segfaults - better forbid this, to avoid someone
else having to debug it again ;)
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
admin/datastore reads linearly only, so no need for cache (capacity of 1
basically means no cache except for the currently active chunk).
mount can do random access too, so cache last 8 chunks for possibly a
mild performance improvement.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Implemented as a seperate struct SeekableCachedChunkReader that contains
the original as an Arc, since the read_at future captures the
CachedChunkReader, which would otherwise not work with the lifetimes
required by AsyncRead. This is also the reason we cannot use a shared
read buffer and have to allocate a new one for every read. It also means
that the struct items required for AsyncRead/Seek do not need to be
included in a regular CachedChunkReader.
This is intended as a replacement for AsyncIndexReader, so we have less
code duplication and can utilize the LRU cache there too (even though
actual request concurrency is not supported in these traits).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Supports concurrent 'access' calls to the same key via a
BroadcastFuture. These are stored in a seperate HashMap, the LruCache
underneath is only modified once a valid value has been retrieved.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Explicitly test that data will stay available and can be retrieved
immediately via listen(), even if the future producing the data and
notifying the consumers was already run in the past.
Wasn't broken or anything, but helps with understanding IMO.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
in PVE, the logic how wearout gets read from the smartctl output was
changed from a vendor -> id map to a sorted list of specific
attribute field names.
copy that list to pbs (in the same order), and use that to get the
wearout
in the future we might want to split the disk logic into its own crate
and reuse it in pve
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if we want the empty value as a valid default value in a combogrid,
we have to explicitely select 'null' else the field will be marked as
dirty
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
in case an invalid drive was configured, now it marks the field
invalid instead of autoselecting the first valid one
this could have lead to users configuring the wrong drive in a
tape-backup-job when they edited one
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we skip snapshots that are older than the newest snapshot of the group in
the target datastore, log it so the user can know why it is not synced
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that a user can remove a datastore from the gui,
though no data is deleted, this has to be done elsewhere (for now)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that longer running creates (e.g. a slow storage), does not
run in a timeout and we can follow its creation
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
when we remove a datastore via api/cli, the proxy
has sometimes leftover references to that datastore in its
DATASTORE_MAP which includes an open filehandle on the
'.lock' file
this prevents unmounting/exporting the datastore even after removal,
only a reload/restart of the proxy did help
add a command to our command socket, which removes all non
configured datastores from the map, dropping the open filehandle
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
by implementing a custom error type that is either 'TimeOut' or
'Other'.
In the api, check in the worker loop for exactly 'TimeOut' errors and continue only
then. All other errors lead to a aborted task.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
removing the backup dir must acquire the snapshot lock, else it can
happen that we remove a snapshot while it is being restored
or backed up to tape
the original commit that adds the force flag
(c9756b40d1)
mentions that the prune checks itself if the snapshot is in use,
but i could not find such code, so simply set force to false
to avoid failing and aborting the prune job, warn if it could not
and continue
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that the update does not get canceled because of a bad datastore
hide the irrelevant fields in that case
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This reverts commit 75f9f40922, which is
no longer needed now that we use tokio >= 1.6 which contains the proper
fix.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it contains a bug fix that allows dropping the workaround in
75f9f40922 file-restore-daemon: work around tokio DuplexStream bug
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
from the SspDataEncryptionCapabilityPage
it seems we do not need it, since the EXTDECC flag is only used for
determining if the drive is capable to be configured via
ADI (Automation/Drive Interface) which we do not use at all.
this makes the call work with LTO-4 again
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this was previously set on the button class, but has since been removed
add it here to have the badge number centered again
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by default the treelist gets the 'nav' ui, which in newer extjs
versions has a custom styling (unlike before)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by using beforedestroy instead of destroy (like we do everywhere else)
to avoid race condition when the controller has
already removed some handlers on destruction
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
using container here is fine, we do not need panel behaviour which
is more bloated. Removes two ARIA warnings.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
looks (almost confusingly) empty else and no real disadvantage in
showing the disabled one until a media-set is selected and loaded
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
for that we need to split the prefilter additions, else
we always filter the snaphots too and giving 'undefined' filters
all snapshots...
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
for the case that the user selects only whole datastores, we do not
want to send and (exhaustive) list of snapshots that get restored,
but we only want to honor the mapping the user gives
this avoids using the backup restore codepath that iterates twice
over the tapes and would generally be slower for a lot of snapshots
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when no uuid/mediaset is given.
we change a bit how we use the uuid by moving it into the viewmodel
(instead of a simple property on the view) so that we can always
use the selected one
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to make it clear that this button is for restore and for
now we do not have any plans to add buttons here
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we want a 'media-set' selector in the gui, this makes it
very easy to do and is not as costly as reusing the media list,
since we do not need to iterate over all media (e.g. unassigned)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
normally, users will not have many tape media pools,
and are more interested in the actual media-sets, so
expand those nodes by default
if the list gets very long, the user can collapse some pools anyway
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by extracting them via the api macro into the function signature
this fixes an issue, where giving 'since' and 'until' where not
used since we tried to extract them as 'str' while they were numbers.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While the issue with vsock packets starving kernel memory is mostly
worked around by the '64k -> 4k buffer' patch in
'proxmox-backup-restore-image', let's be safe and also limit the number
of concurrent transfers. 8 downloads per VM seems like a fair value.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
The extract API call may be active for more than the watchdog timeout,
so a simple ping is not enough.
This adds an "inhibit" API, which will stop the watchdog from completing
as long as at least one WatchdogInhibitor instance is alive. Keep one in
the download task, so it will be dropped once it completes (or errors).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
See this PR for more info: https://github.com/tokio-rs/tokio/pull/3756
As a workaround use a pair of connected unix sockets - this obviously
incurs some overhead, albeit not measureable on my machine. Once tokio
includes the fix we can go back to a DuplexStream for performance and
simplicity.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Used to specify a filesystem placed directly on a disk, without a
partition table inbetween. Detected by simply attempting to mount the
disk itself.
A helper "make_dev_node" is extracted to avoid code duplication.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
A bucket might contain multiple (or 0) layers of components in its path
specification, so allow a mapping between bucket type strings and
expected component depth. For partitions, this is 1, as there is only
the partition number layer below the "part" node.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Mainly as Config -> Option is a weird name, Authentication has only
one obj. grid, the node options are only the http-proxy for now and
that is a sort of authentication, so good enough for me for now, but
should be rethought for 2.0 and/or once more node opts are added
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
for now only http-proxy lives there, but we will add more options later,
such as
* email from
* default gui language
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
It was not actually bad, so they're quite opinionated to be honest,
but at least xtypes props must go first and variable declaration
should try to be as near as possible to the actual use as long as
code stays sensible readable/short.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
a restore does not change the tape content, so a reload has no benefit here.
since we're touching those lines, change to 'autoShow' property
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by including the new snapshotselector. If a whole media-set is to be
restored, select all snapshots
to achieve this, we drop the 'restoreid' and 'datastores' properties
for the restore window, and replace them by a 'prefilter' object
(with 'store' and 'snapshot' properties)
to be able to show the snapshots, we now have to always load the
content of that media-set, so drop the short-circuit if we have
the datastores already.
change the layout of the restore window into a two-step window
so that the first tab is the selection what to restore, and on the
second tab the user chooses where to restore (drive, datastore, etc.)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
* handle not rendered call of getErrors
* return 'all' as value if all snaphots where selected
(for better distinction)
* remove the default height
* add checkChange on stores filterChange
(now change also fires on the gridfilter plugin change)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
enable scrolling by default, and handle the case that getErrors gets
called when the component is not yet rendered
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This can happen if the underlying storage failed, in which case we do
not want to fail the whole API call, as it should report the status
of all datastores. So rather add the error inline to the related
store entry and continue.
Allows to nicely visualize those stores in the gui.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this will be used for letting the user select multiple, individual
snapshots on restore (instead of having a single or the whole media-set)
if a 'prefilter' object is given, we filter the grid by those
values using the gridfilter plugins (like in pve's bulk action windows)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of having them in the toolbar. This makes the UI more consistent
with the datastore content view.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it's the only PBS-specific part in there, so let's make it
product-agnostic before moving it off to proxmox-http.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
so that a user can delete a whole group at once, until now, the fastest
way for this was to prune to one snapshot, and delete that
code is basically a copy/paste from the snapshot delete, sans
the 'backup-time' parameter
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that a user can force a new media set, e.g. if he uses the
allocation policy 'continue', but wants to manually start a new
media-set.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if the account does not exist, error with its name
if file loading fails, the error includes the full path
if the content fails to parse, show file & parse error
and in each case mention that it's about loading the acme account file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
syncs behavior with both, the displayed state in the PBS
web-interface, and the behavior of PVE/PMG.
Without this a standard setup would result in a Error like:
> TASK ERROR: no acme client configured
which was pretty confusing, as the actual error was something else
(no account configured), and the web-interface showed "default" as
selected account, so a user had no idea what actually was wrong and
how to fix it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
- refactor the combinators,
- make it take a `&T: Serialize` instead of a Value, and
allow sending the raw string via `send_raw_command`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
return a result with optional fingerprint instead of tuple, allowing
easy extraction of a meaningful error message.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
if the expected fingerprint and the one returned by the server don't
match, print a warning and allow confirmation and proceeding if running
interactive.
previous:
$ proxmox-backup-client ...
Error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915:
new:
$ proxmox-backup-client ...
WARNING: certificate fingerprint does not match expected fingerprint!
expected: ac:cb:6a:bc:d6:b7:b4:77:3e:17:05:d6:b6:29:dd:1f:05:9c:2b:3a:df:84:3b:4d:f9:06:2c:be:da:06:52:12
fingerprint: ab:cb:6a:bc:d6:b7:b4:77:3e:17:05:d6:b6:29:dd:1f:05:9c:2b:3a:df:84:3b:4d:f9:06:2c:be:da:06:52:12
Are you sure you want to continue connecting? (y/n): n
Error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915:
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this makes it possible to only restore some snapshots from a tape media-set
instead of the whole. If the user selects only a small part, this will
probably be faster (and definitely uses less space on the target
datastores).
the user has to provide a list of snapshots to restore in the form of
'store:type/group/id'
e.g. 'mystore:ct/100/2021-01-01T00:00:00Z'
we achieve this by first restoring the index to a temp dir, retrieving
a list of chunks, and using the catalog, we generate a list of
media/files that we need to (partially) restore.
finally, we copy the snapshots to the correct dir in the datastore,
and clean up the temp dir
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and create the 'email' and 'restore_owner' variable at the beginning,
so that we can reuse them and do not have to pass the sources of those
through too many functions
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by showing
'[format, ...]'
where 'format' is the simple format from the type of the items
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
if the reader thread is already gone here, we panic here, resulting in
a nondescript error message, so simply ignore/warn in that case and
return gracefully
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when changin the remote, there is a high chance that there are different
datastores, and if a user does not pay attention, now the first store
of the new remote is selected, instead of the one with the same name
disable autoSelect and let the user manually select a remote datastore
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we have 2 modi in that window:
* backup has multiple datastores
* backup has single datastore
In the first case we show a 'mapping' grid so that
the user can only restore a part. Here a user sees all source
Datastores and can select a target for each one.
In the second case we only have a single 'Datastore' selector, but
we do not show the source. Because of this, the naming is slightly ambiguous
(is it the 'Source' or the 'Target' ?), so rename it to 'Target Datastore'.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
It may make sense in the future, e.g., if the built-in standalone
type is not enough, e.g., as HTTP**s**, HTTP 2 or even QUIC (HTTP 3)
is wanted in some setups, but for now there's no scenario where one
would profit from adding a new HTTP plugin, especially as it requires
the `data` property to be set, which makes no sense..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we cannot add a plugin with an existing ID so this completion helper
is rather counterproductive...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It will be reused in a later patch in another module which should not
depend on the actual API implementation (ugly and cyclic)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We get lots of warnings due to sphinx complaining about missing
includes for generated synopsis. We do not reference to any of those
for now, so we can ignore that now and supress all standard and
warning output.
Note: Errors are still reported.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
especially for the last group, without this the progress would report:
"percentage done: 100.00% (1 of 2 groups, 1 of 1 group snapshots)"
instead of the more logical
"percentage done: 100.00% (2 of 2 groups)"
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Set PBS_QEMU_DEBUG=1 on a command that starts a VM and then connect to
the debug root shell via:
minicom -D \unix#/run/proxmox-backup/file-restore-serial-10.sock
or similar.
Note that this requires 'proxmox-backup-restore-image-debug' to work,
the postinst script is updated to also generate the corresponding image.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
A PCI bus can only support up to 32 devices, so excluding built-in
devices that left us with a maximum of about 25 drives. By adding a new
PCI bridge every 32 devices (starting at bridge ID 2 to avoid conflicts
with automatic bridges), we can theoretically support up to 8096 drives.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
The guest kernel requires more memory depending on how many disks are
attached. 256 seems to be enough for basically any reasonable and
unreasonable amount of disks though.
For debug instance, make it 1G, as these are never started automatically
anyway, and need at least 512MB since the initramfs (especially when
including a debug build of the daemon) is substantially bigger.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Helps to clean up a VM that has crashed, is not responding to vsock API
calls, but still has a running QEMU instance.
We always check the process commandline to ensure we don't kill a random
process that took over the PID.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
otherwise, the kernel driver exposes file names as iso 8859-1,
but we want to have them as utf8.
This mapping should always work, since UTF16 can be cleanly converted
to UTF8.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if we are given a 'naked' ipv6 without square brackets around it,
we need to add them ourselves, since the address is ambigious otherwise
when we add the port.
e.g. giving 'fe80::1' as address we arrive at the url (with the default port)
'https://fe80::1:8007/'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
some features we need (e.g. READ POSITION long form) are only officially
available with LTO-5, but work on many LTO-4 drives, so move LTO-4 to
'best-effort' support.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by checking the 'checked_chunks' before trying to write to disk
and by doing the existance check in the parallel handler. This way,
we do not have to check the existance of a chunk multiple times
(if multiple source datastores gets restored to the same target
datastore) and also we do not have to wait on the stat before reading
the next chunk.
We have to change the &WorkerTask to an Arc though, otherwise we
cannot log to the worker from the parallel handler
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Split out a separate function scan_chunk_archive() for catalog restores.
Note: Required, because we need to optimize restore_chunk_archive() to
write datastore in separate threads (else thape drive will stop during restore)
API like in PVE:
GET .../info => current cert information
POST .../custom => upload custom certificate
DELETE .../custom => delete custom certificate
POST .../acme/certificate => order acme certificate
PUT .../acme/certificate => renew expiring acme cert
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This is the highlevel part using proxmox-acme-rs to create
requests and our hyper code to issue them to the acme
server.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
else we sometimes forget to remove it from the 'params' variable
and use that further, running into 'invalid parameter' errors
found by giving 'output-format' paramter to proxmox-tape status
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by checking for definedness of the label (tapes without barcode
have the empty string as label-text) and falling back to the
source slot for the load action
Note: Changed the load-slot API from PUT to POST
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
This allows mounting XFS partitons with 'dirty' states, like from a
running VM. Otherwise XFS tries to write recovery information, which
fails on a read-only mount.
Tested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Drive serials have a character limit of 20, longer names like
"drive-virtio0.img.fidx" or "drive-efidisk0.img.fidx" would get cut off.
Fix this by removing the suffix, it is not necessary to uniquely
identify an image.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Some drives will always return the number of bytes given in the
allocation_length field, but correctly report the data len in the mode
sense header. Simply ignore the excess data.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
include the expected and unexpected sizes in the error message,
so that it's easier to debug in case of an error
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
With the vsock-pkt-buffer fix in proxmox-backup-restore-image, we can
use way less memory for the VM without risking any crashes. 128 MiB
seems to be the lowest it will go and still be fully reliable.
While at it, add the "panic=1" argument to the kernel command line, so
in case the kernel *does* run out of memory, it will at least restart
automatically.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Read image sizes (.pxar.fidx/.img.didx) from manifest and partition
sizes from /sys/...
Requires a change to ArchiveEntry, as DirEntryAttribute::Directory
does not have a size associated with it (and that's probably good).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
this way, the api call does not error out when the file is locked
currently (which means that job is running and we do not need
to update the time)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when a user updates a job schedule, we want to save that point in time
to calculate future runs, otherwise when a user updates a schedule to
a time that would have been between the last run and 'now' the
schedule is triggered instantly
for example:
schedule 08:00
last run today 08:00
now it is 12:00
before this patch:
update schedule to 11:00
-> triggered instantly since we calculate from 08:00
after this patch:
update schedule to 11:00
-> triggered tomorrow 11:00 since we calculate from today 12:00
the change in the enum type is ok, since by default serde does not
error on unknown fields and the new field is optional
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a backup task failed (e.g. it was aborted), show the snapshots
which were successfully backed up in the notification
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when the user start an action where we know that it locks the drive,
reload the tape store, so that the state is refreshed
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to make the following cryptic error:
proxmox-file-restore failed: Error: Invalid byte 46, offset 5.
more understandable:
proxmox-file-restore failed: Error: Failed base64-decoding path '/root.pxar.didx' - Invalid byte 46, offset 5.
when a user passes in a non-base64 path but sets `--base64`.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
basically the same as commit eeff085d9d
Will be required once we get to use a newer rustc, at least the
client build for archlinux was broken due to this.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
same functionality as crypto_parameters, except it keeps the file
descriptor passed as "keyfd" open (and seeks to the beginning after
reading), if one is given.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
For the actual partitions and blockdevices in a backup, which the
user sees like folders in the file-restore ui
Encoded as "None", to avoid cluttering DirEntryAttribute, where it
wouldn't make any sense to have.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
These can't be entered or restored anyway, and cause issues with catalog
files for example.
Also a clippy fix.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
in some configurations, samba stores NTFS-ACLs in this xattr[0], so
we should backup (if we can)
altough the 'security' namespace is special (e.g. in use by
selinux, etc.) this value is normally only used by samba and we
should be able to back it up.
to restore it, the user needs at least 'CAP_SYS_ADMIN' rights, otherwise
it cannot be set
0: https://www.samba.org/samba/docs/current/man-html/vfs_acl_xattr.8.html
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Some changer seem to append more data than we expect, but correctly
annotates that size in the subheader.
For each descriptor entry, read as much as the size given in the
subheader (or until the end of the reader), else our position in
the reader is wrong for the next entry, and we will parse
incorrect data.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
only check every 1024'th, which is cheaper to do than a modulo, as we
can just mask the 10 least-significant-bits and check if the result
is zero.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Fixes a non-negligible performance regression from commit
7f394c807b
While we skip known-verified chunks in the stat-and-inode-sort loop,
those are only the ones from previous indexes. If there's a repeated
chunk in one index they would get re-verified more often as required.
So, add the check again explicitly to the read+verify loop.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
before reading the chunks from disk in the order of the index file,
stat them first and sort them by inode number.
this can have a very positive impact on read speed on spinning disks,
even with the additional stat'ing of the chunks.
memory footprint should be tolerable, for 1_000_000 chunks
we need about ~16MiB of memory (Vec of 64bit position + 64bit inode)
(assuming 4MiB Chunks, such an index would reference 4TiB of data)
two small benchmarks (single spinner, ext4) here showed an improvement from
~430 seconds to ~330 seconds for a 32GiB fixed index
and from
~160 seconds to ~120 seconds for a 10GiB dynamic index
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a user has not configured a drive for a specified driveslot of the
changer, simply hide that slot
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The idea is that people first need to make actual backups before they
need to do maintenance tasks.
Network is already setup when installing with the ISO or on-top of
Debian, so that is not a priority either.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
when we get an error from the tape, we possibly want to ignore it,
i.e. when the file was incomplete, but we still want to error
out if the error came from e.g, the datastore, so we have to move
the error checking code to the 'next_chunk' call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
1. The exit was never called as `test ... || echo "foo" || exit 1`
can never come to the exit, as echo will not fail
2. The echo was meant to be redirected to stderr (FD #2) but it was
actually redirected to a file named '2'
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Treat filepaths like "/root.pxar.didx" without a trailing slash as
wanting to download the entire archive content instead of erroring. The
zip-creation code already works fine for this scenario.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
datastore and tape entries are very similar but differ in some points
in such a way that a nice unification is not really that helpful, but
making similar key parts the same is still nice when reading the code
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
in proxmox-backup-proxy, we log and discard any errors on 'accept',
so that we can continue to server requests
in proxmox-backup-api, we just have the StaticIncoming that accepts,
which will forward any errors from the underlying TcpListener
this patch also logs and discards the errors, like in the proxy.
Otherwise it could happen that if the api-daemon has more files open
than the proxy, it will shut itself down because of a
'too many open files' error if there are many open connections
(the service should also restart on exit i think, but this is
a separate issue)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a datastore or root is not used directly on the pool dir
(e.g. the installer creates 2 sub datasets ROOT/pbs-1), info in
/proc/self/mountinfo returns not the pool, but the path to the
dataset, which has no iostats itself in /proc/spl/kstat/zfs/
but only the pool itself
so instead of not gathering data at all, gather the info from the
underlying pool instead. if one has multiple datastores on the same
pool those rrd stats will be the same for all those datastores now
(instead of empty) similar to 'normal' directories
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
should not be a hard dependency, as one can use the file-restore tool
for pxar archives without it too
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The data on the restore daemon is either encoded into a pxar archive, to
provide the most accurate data for local restore, or encoded directly
into a zip file (or written out unprocessed for files), depending on the
'pxar' argument to the 'extract' API call.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Encodes an entire local directory into an AsyncWrite recursively.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
extract_sub_dir_seq, together with seq_files_extractor, allow extracting
files from a pxar Decoder, along with the existing option for an
Accessor. To facilitate code re-use, some helper functions are extracted
in the process.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Allows listing files and directories on a block device snapshot.
Hierarchy displayed is:
/archive.img.fidx/bucket/component/<path>
e.g.
/drive-scsi0.img.fidx/part/2/etc/passwd
(corresponding to /etc/passwd on the second partition of drive-scsi0)
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This will be triggered on updating proxmox-file-restore (via configure,
necessary since the daemon binary might change) and
proxmox-backup-restore-image (via 'activate-noawait', necessary since
the base image might change).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Includes methods to start, stop and list QEMU file-restore VMs, as well
as CLI commands do the latter two (start is implicit).
The implementation is abstracted behind the concept of a
"BlockRestoreDriver", so other methods can be implemented later (e.g.
mapping directly to loop devices on the host, using other hypervisors
then QEMU, etc...).
Starting VMs is currently unused but will be needed for further changes.
The design for the QEMU driver uses a locked 'map' file
(/run/proxmox-backup/$UID/restore-vm-map.json) containing a JSON
encoding of currently running VMs. VMs are addressed by a 'name', which
is a systemd-unit encoded combination of repository and snapshot string,
thus uniquely identifying it.
Note that currently you need to run proxmox-file-restore as root to use
this method of restoring.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Includes functionality for scanning and referring to partitions on
attached disks (i.e. snapshot images).
Fairly modular structure, so adding ZFS/LVM/etc... support in the future
should be easy.
The path is encoded as "/disk/bucket/component/path/to/file", e.g.
"/drive-scsi0/part/0/etc/passwd". See the comments for further
explanations on the design.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Add a watchdog that will automatically shut down the VM after 10
minutes, if no API call is received.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Implements the base of a small daemon to run within a file-restore VM.
The binary spawns an API server on a virtio-vsock socket, listening for
connections from the host. This happens mostly manually via the standard
Unix socket API, since tokio/hyper do not have support for vsock built
in. Once we have the accept'ed file descriptor, we can create a
UnixStream and use our tower service implementation for that.
The binary is deliberately not installed in the usual $PATH location,
since it shouldn't be executed on the host by a user anyway.
For now, only the API calls 'status' and 'stop' are implemented, to
demonstrate and test proxmox::api functionality.
Authorization is provided via a custom ApiAuth only checking a header
value against a static /ticket file.
Since the REST server implementation uses the log!() macro, we can
redirect its output to stdout by registering env_logger as the logging
target. env_logger is already in our dependency tree via zstd/bindgen.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This allows switching the base user identification/authentication method
in the rest server. Will initially be used for single file restore VMs,
where authentication is based on a ticket file, not the PBS user
backend (PAM/local).
To avoid putting generic types into the RestServer type for this, we
merge the two calls "extract_auth_data" and "check_auth" into a single
one, which can use whatever type it wants internally.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
For now it only supports 'list' and 'extract' commands for 'pxar.didx'
files. This should be the foundation for a general file-restore
interface that is shared with block-level snapshots.
This is packaged as a seperate .deb file, since for block level restore
it will need to depend on pve-qemu-kvm, which we want to seperate from
proxmox-backup-client.
[original code for proxmox-file-restore.rs]
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[code cleanups/clippy, use helpers::list_dir_content/ArchiveEntry, no
/block subdir for .fidx files, seperate binary and package]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
by using format_boolean for compression/write protect,
combining file/block posiition into one (saves a line)
and adding the missing alert-flags
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it is rather pointless to let the user select something were there
is no choice. We have to keep the window though, since the user may
want to choose a pool
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a changer only has a single drive, there is no point in showing
a window with a DriveSelector, just do want the user wanted.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to make it more like a 'dangerous' remove window
also works in the singleDrive logic to hide/show the driveselector
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
for some storages, it is valuable information, e.g. if one has datastores
on separate datasets of the same zpool
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
I made some comparision with bombardier[0], the one listed here are
30s looped requests with two concurrent clients:
[ static download of ext-all.js ]:
lvl avg / stdev / max
none 1.98 MiB 100 % 5.17ms / 1.30ms / 32.38ms
fastest 813.14 KiB 42 % 20.53ms / 2.85ms / 58.71ms
default 626.35 KiB 30 % 39.70ms / 3.98ms / 85.47ms
[ deterministic (pre-defined data), but real API call ]:
lvl avg / stdev / max
none 129.09 KiB 100 % 2.70ms / 471.58us / 26.93ms
fastest 42.12 KiB 33 % 3.47ms / 606.46us / 32.42ms
default 34.82 KiB 27 % 4.28ms / 737.99us / 33.75ms
The reduction is quite better with default, but it's also slower, but
only when testing over unconstrained network. For real world
scenarios where compression actually matters, e.g., when using a
spotty train connection, we will be faster again with better
compression.
A GPRS limited connection (Firefox developer console) requires the
following load (until the DOMContentLoaded event triggered) times:
lvl t x faster
none 9m 18.6s x 1.0
fastest 3m 20.0s x 2.8
default 2m 30.0s x 3.7
So for worst case using sligthly more CPU time on the server has a
tremendous effect on the client load time.
Using a more realistical example and limiting for "Good 2G" gives:
none 1m 1.8s x 1.0
fastest 22.6s x 2.7
default 16.6s x 3.7
16s is somewhat OK, >1m just isn't...
So, use default level to ensure we get bearable load times on
clients, and if we want to improve transmission size AND speed then
we could always use a in-memory cache, only a few MiB would be
required for the compressable static files we server.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
by using our DeflateEncoder
for this to work, we have to create wrapper reader that generates the crc32
checksum while reading.
also we need to put the target writer in an Option, so that we can take
it out of self and move it into the DeflateEncoder while writing
compressed
we can drop the internal buffer then, since that is managed by the
deflate encoder now
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
implements a deflate encoder that can compress anything that implements
AsyncRead + Unpin into a file with the helper 'compress'
if the inner type is a Stream, it implements Stream itself, this way
some streaming data can be streamed compressed
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Else it does not gets picked up on release builds...
Also the mathjax path option affects HTML not EPUB so move it to the
correct section in conf.py
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Previously we did not store GROUP_OBJ ACL entries for
directories, this means that these were lost which may
potentially elevate group permissions if they were masked
before via ACLs, so we also show a warning.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Don't override `group_obj` with `None` when handling
`ACL_TYPE_DEFAULT` entries for directories.
Reproducer: /var/log/journal ends up without a `MASK` type
entry making it invalid as it has `USER` and `GROUP`
entries.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Add a new module containing key-related functions and schemata from all
over, code moved is not changed as much as possible.
Requires adapting some 'use' statements across proxmox-backup-client and
putting the XDG helpers quite cozily into proxmox_client_tools/mod.rs
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Pass in an optional auth tag, which will be passed as an Authorization
header on every subsequent call.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
To let users find the good explanation about allocation and retention
policies from the docs easier.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This way we get a better rendering in the api-viewer.
before:
[<string>, ... ]
after:
[(<source>=)?<target>, ... ]
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding a custom field (grid) where the user can select
a target datastore for each source datastore on tape
if we have not loaded the content of the media set yet,
we have to load it on window open to get the list of datastores
on the tape
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by changing the 'store' parameter of the restore api call to a
list of mappings (or a single default datastore)
for example giving:
a=b,c=d,e
would restore
datastore 'a' from tape to local datastore 'b'
datastore 'c' from tape to local datastore 'e'
all other datastores to 'e'
this way, only a single datastore can also be restored, by only
giving a single mapping, e.g. 'a=b'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The text 'had to upload [KMG]iB' implies that this is the size we
actually had to send to the server, while in reality it is the
raw data size before compression.
Count the size of the compressed chunks and print it separately.
Split the average speed into its own line so they do not get too long.
Rename 'uploaded' into 'size_dirty' and 'vsize_h' into 'size'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in commit `asyncify pxar create_archive`, we changed from a
separate thread for creating a pxar to using async code, but the
StdChannelWriter used for both pxar and catalog can block, which
may block the tokio runtime for single (and probably dual) core
environments
this patch adds a wrapper struct for any writer that implements
'std::io::Write' and wraps the write calls with 'block_in_place'
so that if called in a tokio runtime, it knows that this code
potentially blocks
Fixes: 6afb60abf5 ("asyncify pxar create_archive")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This is just an optimization, avoiding to read the catalog into memory.
We also expose create_temporary_database_file() now (will be
used for catalog restore).
- new helper: lock_media_set()
- MediaPool: lock media set
- Expose Inventory::new() to avoid double loading
- do not lock pool on restore (only lock media-set)
- change pool lock name to ".pool-{name}"
so that a user can schedule multiple backup jobs onto a single
media pool without having to consider timing them apart
this makes sense since we can backup multiple datastores onto
the same media-set but can only specify one datastore per backup job
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the default escape handler is handlebars::html_escape, but this are
plain text emails and we manually escape them for the html part, so
set the default escape handler to 'no_escape'
this avoids double html escape for the characters: '&"<>' in emails
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead print an error and continue, the rendering functions will error
out if one of the templates could not be registered
if we `.unwrap()` here, it can lead to problems if the templates are
not correct, i.e. we could panic while holding a lock, if something holds
a mutex while this is called for the first time
add a test to catch registration issues during package build
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since we can now backup multiple datastores in the same media-set,
we show the datastores as first level below that
the final tree structucture looks like this:
tapepool A
- media set 1
- datastore I
- tape x
- ct/100
- ct/100/2020-01-01T00:00:00Z
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
I mean the whole distro uses quite some C and the like as base, so
avoid being overly strict here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It does not help users if that is spelled out, and its not a common
use of GCM, and especially in the AES 256 context its clear what is
meant. The link to Wikipedia stays, so interested people can still
read up on it and others get a better overview due to the text being
more concise.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The main feature list should provide a short overview of the, well,
main features. While enterprise support *is* a main and important
feature, it's not the place here to describe things like personal
volume/ngo/... offers and the like.
Move parts of it to getting help, which lacked mentioning the
enterprise support too and is a good place to describe the customer
portal.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this is much handier than number field, and the user can instantly
see which one is an import/export slot
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that the tape backup can be restored as any user, given
the current logged in user has the correct permission.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Else one has quite a terrible UX when installing from 1.0 ISO and
then upgrading to latest release..
commit 0ec79339f7 for the fix and some other details
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead of always using the starttime, use the last timestamp from the log
this way, one can see when the task was aborted without having to read
the log
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and reuse 'send_job_status_mail' there so that we get consistent
formatted mails from pbs (e.g. html part and author)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
found and semi-manually replaced by using:
codespell -L mut -L crate -i 3 -w
Mostly in comments, but also email notification and two occurrences
of misspelled 'reserved' struct member, which where not used and
cargo build did not complain about the change, soo ...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
from 2 to 60 seconds. To retain the response time of the gui
when adding/editing/removing, trigger a manual reload on these actions
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
iterating over a nodeinterfaces children while removing them
will lead to 'child' being undefined
instead collect the children to remove in a separate list
and iterate over them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when we do a prune during a tape backup, do not cancel the tape backup,
but continue with a warning
the task still fails and prompts the user to check the log
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Less anti-aliasing to do, so it looks better when small.
Note, I use a mask as else we could not cut out something
transparent, but only over paint it filled with white, which is not
the same and not how font awesome does it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The old value just was way to heavy, and notes/warnings/...
admonitions did not stick out anymore.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
1. Use same height as width
2. drop the top/bottom "frame" lines
3. instead of two "bearings" side by side, like VHS or music-compact
tapes have, use a single bearing and an outer circle to denote the
magnetic tape edge
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Simplifies the introduction a bit and makes it more readable.
Also some other minor language fixes throughout the section.
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
Try once first to load the correct tape before sending an email to
insert the correct one. This way, the admin does not get a mail
if the correct tape is already inserted.
Also include the error we got that prompted the email to insert the
tape. This means that if the admin gets prompted to insert e.g.
"FOO" but inserts "BAR", he'll get an email that the wrong
tape is inserted.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of silently discarding the error, else the user might be
confused because nothing happened
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding the existing credential id to the 'excludeCredentials' list
this prevents the browser from registering a token twice, which
lets authentication fail on some browser/token combinations
(e.g. onlykey/solokey+chromium)
while is seems this is currently a bug in chromium, in a future spec
update the underlying behaviour should be better defined, making this
an authenticator bug
also explicitly catch registering errors and show appropriate error messages
0: https://bugs.chromium.org/p/chromium/issues/detail?id=1087642
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
even if the options *could* be set in the frontend, the backend
actually has to do validation of those settings, thus we should not
make that a browser setting
additionally, having the value 'preferred' does not actually make sense,
since it does not add any security (the backend skips the
userverification check then)
This reverts commit aca4c2b5a9.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the current default is 'Preferred', which is not really useful, as the
(web) client can simply change this to discouraged, since the
webauthn_rs crate does not verify the 'user_verified' bit of the
response in that case
setting this to 'Required' is not really useful either at the moment,
since a user can have a mix of different authenticators that may or
may not support user verification
there is ongoing discussion in the crate how to handle that[0]
we could probably expose this setting(discouraged/required) to the user/admin
and save it to the credential and allow only registering credentials
of the same type or filter them out on login (i.e. if there is an
authenticator that can handle userVerification, require it)
in any case, the current default is not helpful for security, but
makes loggin in harder, since the key will by default want to verify
the user
0: https://github.com/kanidm/webauthn-rs/pull/49
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The previous description in the docs was false. The restore command
with the pattern parameter will search the entire backup archive,
regardless of pwd.
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
If the usage of a datastore did not change, we did not
return an estimate. The ui interpreted this as 'not enough data', but
it should actually be 'never'.
Fixing this by always setting the estimate first to 0 and overwriting
if we successfully calculated one, and checking for 'undefined' in the ui.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since the status api call blocks the drive, it only makes sense to do
when the drive is idle.
so do not reload on activate, but when the drive changes the first
time from busy to idle
also disable the reload button when the drive is busy
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we test for the config key in the API so it makes sense to have as
test here too. Actually it would be better if we'd have a expect
Value defined here and enforce that it matches, but better than
nothing.
Fix the input for test 1, where tabs got replaced by spaces, as else
it fails
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
For tapes with many snapshots/group, a simple list of them is too
big. Instead, add a level for just the groups, this makes searching
for a specific backup much easier.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since reload is an async function, the view can be destroyed during
any 'await' point. Subsequent accesses to the view will fail, and we
will land in the catch. Check there if the view is destroyed, and
do not raise an error with the user then
also cancel any outstanding timer on 'deactivate' and 'destroy'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
because we now select it directly in the left-hand tree
so we have to adapt the changer to the one set by the router
and not by the drop down field
and remove it from the TapeManagement tabpanel
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and only check TapeManagement once in the init function
we now have 2 updatestores that update individually
(one for datastores, one for drives/changers)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by generalizing the isDataStorePath logic to a 'parseRouterPath'.
We still have to keep the isDataStore logic for tabpanel handling,
If we add tabs to changer-/drivestatus panels, we have to adapt
that too.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Americans seem to sometimes use Avery 6577 which has 5/8" x 3"
labels, equaling 15.875 mm x 76.2 mm, so do not set the lower bound
to 17mm (which even breaks our used Avery 3240 sheets which have an
label height of 16.9 mm
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
By default, sphinx embeds the cloudflare CDN version of mathjax. This
is bad for privacy, webistes enforcing cross-site origin protection
and in environments with no WAN access.
Luckily there's a Debian package we can use instead.
The config is the default sphinx config used.
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
marketing noted that it looked quite heavy weight, to the point where
it was hard to read to the full black squares, bold monospace text
and bottom borders with rather distinctive darkness.
Address those by:
* change color for list points from black to mid-dark grey
* use empty circles for second heading level
* ensure monospaced text has a normal font weight in the TOC headings
* some lighter color for the bottom border
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
As 20s is really not that high, especially for loaded setups one is
connected to through a spotty network (looking at you ÖBB railnet)
and gets latency spikes of 5 - 10s for some minutes at a time..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since the PUT api call is using the 'Updater', the 'id' parameter is
already encoded in there, tripping up the api verify tests with
'Duplicate keys found in AllOf schema: id'
"fixing" it by removing the explicit id from the api call and
taking it from the Updater (and failing if it does not exists there;
even though that should never happen)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
a 'leaf' node is every file *except* directories, so we have
to reverse the logtic here
this fixes the pxar.didx browser in the web ui
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of filling them with zeroes
this fixes an issue where we could not restore a container with large
sparse files in the backup (e.g. a 10GiB sparse file in a container
with a 8GiB disk)
if the last operation of the copy was a seek, we need to truncate
the file to the correct size (seek beyond filesize does not change it)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the DiffStore only updates the fiels given in the model, so we have to
list all the fields we depend on, else we have incomplete data,
e.g. for rendering
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if the drive state is not empty, some action blocks it, so we cannot
do any task anyway. Since we now refresh the state every 5 seconds,
we can disable/enable them 'live'.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Rund `codespell` tool, but it picked up not as much as I hoped.
Rest was found with vim + (hun)spell
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
some fido2/webauthn keys can have a pin, and the client can request
a mode for the user verification.
'default' (no value set), lets the browser/device decide if the user has to
enter the pin of the device
'discouraged' requests that the user should not need to enter the pin
'preferred' requests that the user should need to enter the pin (if possible)
since we use webauthn only as a 2nd factor, having the user enter
the device pin on login may seem too much hassle for some users, so
give them the option
since this is a client option anyway, do not save it in the backend, but
in the browser local storage
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
mostly copied from pve (for now; will refactor when i add it to
pmg too (soon)) without the pve specific features like dashboard
storages
contains some eslint fixes comparing to pves window
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the statement !a === "somestring" cannot be true since
!a is either true or false and thus not a string
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
an optimize the columns for smaller layouts (1280 width)
we show either:
* Idle
* spinner + status (if no upid)
* spinner + rendered UPID (clickable, opens task viewer)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
those calls could also block, so we have to run them in a blocking
tokio task, as to not block the current thread
nice side effect is that we now also update the state for that
drive in those instances
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
simply writes into/reads from a file in /run, we will use this
for writing the upid (or potential other states) per drive
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
We do not use the media location, so there is no need to update
the online status in the list_media call. Besides, we already update
the online status when we query the changer/status.
slot is already in the title of the grid, and hide it by default for
drives (the user does not need it there)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
...including common schemata, connect(), extract_*() and completion
functions.
For later use with proxmox-file-restore binary.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
...to take advantage of the aio::Encoder from the pxar create.
Rather straightforward conversion, but does require getting rid of
references in the Archiver struct, and thus has to be given the Mutex
for the catalog directly. The callback is boxed.
archive_dir_contents can call itself recursively, and thus needs to
return a boxed future.
Users are adjusted, namely PxarBackupStream is converted to use an
Abortable future instead of a thread so it supports async in its handler
function, and the pxar bin create_archive is converted to an async API
function. One test case is made to just use 'block_on'.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
to extract some subdirectory of a pxar into a given target
this will be used in the client
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
we will reuse that code in the client, so we need to move it to
where we can access it from the client
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[clippy fixes]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
we will reuse that later in the client, so we need it somewhere
we can use from there
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[add strongly typed ArchiveEntry and put api code into helpers.rs]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Currently useful only for single file restore, but kept generic enough
to use any compatible API endpoint over a virtio-vsock[0,1] interface.
VsockClient is adapted and slimmed down from HttpClient.
A tower-compatible VsockConnector is implemented, using a wrapped
UnixStream as transfer. The UnixStream has to be wrapped in a custom
struct to implement 'Connection', Async{Read,Write} are simply forwarded
directly to the underlying stream.
[0] https://www.man7.org/linux/man-pages/man7/vsock.7.html
[1] https://wiki.qemu.org/Features/VirtioVsock
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
This allows anything that can be represented as a UnixStream to be used
as transport for an API server (e.g. virtio sockets).
A tower service expects an IP address as it's peer, which we can't
reliably provide for unix socket based transports, so just fake one.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
but auto-expand them, so no additional click is necessary
this shows the user which tapes are involved for the media sets
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a catalog is missing (or the loading otherwise throws an error), show
the error message in a msg box instead of a mask. this way a user can
still navigate the tree
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if given, erases the tape only iff the inserted tape contains that label
used to safeguard tape erasing from ui for standalone drives
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adapting and reusing the 'LabelMedia' window
shows a short notice about inserting the correct tape in the drive
and now allows to select a drive
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
listing the media can potentially hang (changer status), so do not
reload in the background, only on activate
also increase the timeout to 5 minutes and add a reload button
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since tape commands can take a while and we do not want to change
all of those to worker tasks, increase the timeout to 5 minutes
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in most uses, we want to remove the drive from the param afterwards
where we don't, we already overwrite it with the result of this function
this fixes some commands (like 'proxmox-tape read-label --drive foo')
that failed with:
parameter 'drive': duplicate parameter.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
do not only show if labeled, but the whole status (full/writeable/etc.)
and to which pool the tape belongs (if any)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of showing the snapshots directly under the pool and then the
media-sets, list the media-sets under the pool and only after the
snapshots
this has several advantages:
* we only have to read one set of tape catalog data on expand and not all of
them everytime (which does not scale)
* we can show media-sets without snapshots, this can happen when we
inventoried a set of tapes from another pbs instance, or lost the
catalog data somehow
the disadvantage is that one has to go look for the media set where the
snapshot is included, but we can solve this by implementing a search
function in the future (in the backend)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we include all tasks whose type start with 'verif' in the type 'verify'
but if we want to actually show them in the pop-up, we have to reverse
map this back to 'verif', else there are tasks missing from there
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to avoid confusing messages about using encryption keys when restoring
plaintext backups, or about loading master keys when they are not
actually used for the current operation.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
pull out the crypt-mode to logically group arms and make the whole mess
a bit more "human-parsable".
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it's needed for PVE's LXC integration, and might be interesting for
other more special usage scenarios as well.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
If the ref is named the same as the headline (once normalized), sphinx
will return a 'idX' value in node['ids'][1] which we use for the label
ID. The headline is always present at index 0.
Checking for that and using index 0 in case we do get a 'idX' helps us
to avoid using the 'idX' as keys in our OnlineHelpInfo.js and actually
use the intended key.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
With commit ec1ae7e631 some refs were
changed by getting prefixes and such. We need to adapt the places that
reference them as well
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
and fix some issues from referenced named the same as their heading
they anchor too.
This should be fixed for real in our python plugin to scan for such
references, its probably a bug there, but as most of the problematic
ones where wrong (missing chapter prefix) anyway changing them is OK
too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
"we" should be avoided, it's never quite clear who is "we" in the
context here and it leads to some technical wrong meanings, e.g., we
(here assumed to be "we developers") do not read any backup data, the
Proxmox Backup client does.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
adds explanations for:
* what datastores are
* their relation with snapshots/chunks
* basic information about chunk directory structures
* fixed-/dynamically-sized chunks
* special handling of encrypted chunks
* hash collision probability
* limitation of file-based backups
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
fixes connecting to hosts with valid certificates without a
pinned fingerprint
this was accidentally changed in the tokio-1.0 updates
apparently
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: 0f860f712f ("tokio 1.0: update to new tokio-openssl interface")
since we do not show the tapes anymore in the BackupOverview, add
another panel where we can list the available tapes in the inventory
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of grouping by tape (which is rarely interesting),
group by pool -> group -> id -> mediaset
this way a user looking for a backup of specific vm can do just that
we may want to have an additional view here were we list all snapshots
included in the selected media-set ?
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
some users might want to store the plain version of their master key for
long-term storage and rely on physical security instead of a passphrase
to protect the paper key.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
our patches got applied upstream, and a release was cut, so we no longer
need to depend on a manually patched version here.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
with the fix for #2909 (improving handling missing chunks), we
changed from bailing to warning during a garbage collection when
updating the atime of a chunk.
but, updating the atime can not only fail when the chunk is missing,
but also on other occasions, e.g. no permissions or more importantly,
no space left on the device. in that case, the atime of a valid and used
chunk cannot be updated, and the second sweep of the gc will remove that chunk.
[0] is a real world example of that happening.
instead, only warn on really missin chunks, and bail on all other
errors.
0: https://forum.proxmox.com/threads/pbs-server-full-two-days-later-almost-empty.83274/
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when executing this code as non-root, we use sg-tape-cmd (a setuid binary)
to execute various ioctls on the tape device
we give the command the open tape device fd as stdin, but did not
dup it, so the std::process:Stdio handle closed it on drop,
which let subsequent operation on that file fail (since it was closed)
fix it by dup'ing it before giving it to the command, and also refactor
the calling code, so that we do not forget to do this
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we did this for 'mtx', but missed it for the sg_pt_changer code
refactor it into the MtxStatus strut, and call it from both
code paths
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
mostly typical CRUD interface for managing drives, with an
additional actioncolumn containing some useful actions, e.g.
* reading the label
* show volume-statistics
* show the status
* label the inserted tape
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this lets the users manage changers and lets them view the status of one
by having an overview of:
* slots for tapes
* import/export slots
* drives
lets the user:
* barcode-label all the tapes in the library
* move tapes between slots, into/out of drives
* show some basic info when a tape is loaded into a drive
* show the status of a drive
* clean a drive
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
shows all tapes with the relevant info
* which pool it belongs to
* what backups are on it
* which media-set
* location
* etc.
This is very rough, and maybe not the best way to display this information.
It may make sense to reverse the tree, i.e. having pools at top-level,
then media-sets, then tapes, then snapshots..
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
includes edit windows for
* Drives
* Changers
* Media Pools
* Labeling Media
* Making new Tape Backups
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this check is not perfect since there are often multiple device
nodes per drive/changer, but from the scan api we should return always
the same, so for an api user this should be enough
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that an api user can get the drives belonging to a changer
without having to parse the config listing themselves
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
similar to the changers, create a listing at /tape/drive and put
the specific api calls below that
move the scan api call up one level
remove the status info from the config listing
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that an api user can see which drive belongs to which drivenum of a changer
for ones with multiple drives
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Fix two things:
* do not reject the login promise when we get the abort DOMException
error
* safely save the original challenge string as we work on a reference
here and avoid to convert to a UInt8 array twice to avoid an
exception.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
simple heuristic for those people who always prefer a specific TFA
method and have the others only as backup.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Improve UX by avoiding the need to click some buttons twice, or
calling TOTP and Recovery codes both "OTP" codes and showing multiple
buttons, with all having the same goal "submit a TFA token" at the
same time.
Instead use a tab panel with a single submit button.
WebAuthn can and should be still improved, but that can be OK as
followup.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
add a changer listing here (copied from api2/config/changer)
and put the status and transfer api calls below that
puts the changer scan into the top level tape api
and removes the (now redundant) info from the config api path
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
userid parameter needs to be properly encoded when shown on the browser
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
improves upid_read_status with:
* ignore multiple newlines at the end
* remove all code that could panic (array index access)
the one place where we access with '[pos+1..]' is ok since
we explicitely test the len of the vector, this is done to
let rust optimize away the range checks, so it cannot panic
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The options struct has no Drop handler and is passed by-move
so we can partially move out of it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
all the verify methods pass along the following:
- task worker
- datastore
- corrupt and verified chunks
might as well pull that out into a common type, with the added bonus of
now having a single point for construction instead of copying the
default capacaties in three different modules..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it's needed to derive Hash, and we always compare Authids or their
Userid components, never just the Tokenname part anyway..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
containing the CLI parameters that are mostly passed-through from the
client to our pxar archive creation wrapper in pxar::create
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Otherwise we run the drop handler for the scsi pt object AND
the box itself, which shouldn't even work as it should be
doing a double-free (unless the library does some kind of
reference counting in which case this should simply crash
later on?)
anyway, let's make a wrapper simply called `SgPt` containing
the pointer from `construct_scsi_pt_obj()`
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
chunk_stream one can be collapsed, since split == split_to with at set
to buffer.len() anyway.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This clarifies the fact that all communication between client and server
uses TLS for secure communication.
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
This is wrong most of the time, when not loading the web interface
with valid credentials, and thus some checks or defaults did not
evaluated correctly when the underlying value was only set later.
Needs to be set on component creation only, this can be done through
initComponent, even listeners, view controllers or cbind closures.
Use the latter, as all affected components already use cbind.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
and allow it in the one case where the entry loop is intended, but the
code is not yet implemented fully.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
present all relevant information about the TFA token to be removed,
so that a user can make a better decision.
Rework layout to match our commonly used style.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While the user chosen description is not allowed to be
empty, we do leave it empty for recovery keys, as a "dummy
description" makes little sense...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Clarify that the password of the user one wants to add TFA too is
required, which is not necessarily the one of the current logged in
user. Use an empty text for that.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
otherwise the user is confronted with a generic error like "permission
check failed" with no indication that it refers to a request made to the
remote PBS instance..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Currently there's not yet a node config and the WA config is
somewhat "tightly coupled" to the user entries in that
changing it can lock them all out, so for now I opted for
fewer reorganization and just use a digest of the
canonicalized config here, and keep it all in the tfa.json
file.
Experimentally using the flatten feature on the methods with
an`Updater` struct similar to what the api macro is supposed
to be able to derive on its own in the future.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
this is a HACK!
It seems that due to lots of binaries getting compiled from a single
crate the compiler is confused when linking in dependencies to each
binaries ELF.
It picks up the combined set (union) of all dependencies and sets
those to every ELF. This results in the client, for example, linking
to libapt-pkg or libsystemd even if none of that symbols are used..
This could be possibly fixed by restructuring the source tree into
sub crates/workspaces or what not, not really tested and *lots* of
work.
So as stop gap measure use `ldd -u` to find out unused linkage and
remove them using `patchelf`.
While this works well, and seems to not interfere with any debug
symbol usage or other usage in general it still is a hack and should
be dropped once the restructuring of the source tree has shown to
bring similar effects.
This allows for much easier re-use of the generated client .deb
package on other Debian derivaties (e.g., Ubuntu) which got blocked
until now due to wrong libt-apt verison or the like.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
modifying @pam users credentials should be only possible for root@pam,
otherwise it can have unintended consequences.
also enforce the same limit on user creation (except self_service check,
since it makes no sense during user creation)
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
update them to the new tokio-openssl API and remove socket buffer size
setting - it was removed from the TcpStream API, and is now only
available via TcpSocket (which can in turn be converted to a
TcpListener), but this is not needed for this example.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
to wrap a Receiver in a Stream. this will likely move back into tokio
proper once we have a std Stream..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Child itself is no longer a Future, but it has a new wait() async fn
that does the same thing
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
enter() now returns a guard, and the builder got revamped to make the
choice between MT and current thread explicit.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the password field should not be indented differently than the rest of
the fields, and we never have a border on the panels
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we can reuse the edit window from widget toolkit for the most part
this solves some spacing and layout issues and is less code
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
an empty schedule means 'none', so do not fill it with the default
in case we edit an existing job (like we do already for sync jobs)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
We always automatically unload tapes to free library slots,
so it should not happen that an ejected tape resides inside the drive.
This is just a safe guard to handle the situation in case it happens ...
You can manually produce the situation by ejecting a tape without unloading:
mt -f /dev/nst0 eject
Note: Our "proxmox-tape eject" does automatic unload
Try to provide generic implementation for complex operations:
- unload_to_free_slot
- load_media
- export media
- clean drive
- online_media_changer_ids
the old variant attempted to parse a tokenid as userid and returned the
cryptic parsing error to the client, which is rather confusing.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it triggered with a wrongly-formatted message on schemas that did NOT
contain any duplicates..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Use timeout futures for sections that might hang in certain error
conditions. This is mostly intended to be used as a safeguard, not a
first line of defense - i.e. best-effort avoidance of total hangs.
Not every future used for the HttpClient/H2Client is changed, only those
where a quick response is to be expected. For example, the response
reading futures are left alone, so data transfer is never capped with
timeout, only the initial server connect.
It is also used for upgrading to H2 connections, as that can take a long
time on overloaded servers.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
it seems that sometimes, the child process signal gets handled
before the parent process signal. Systemd then ignores the
childs signal (finished reloading) and only after going into
reloading state because of the parent. this will never finish.
Instead, wait for the state to change to 'reloading' after sending
that signal in the parent, an only fork afterwards. This way
we ensure that systemd knows about the reloading before actually trying
to do it.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Fabian Ebner <f.ebner@proxmox.com>
of ProcessLockSharedGuard.
We use a counter to determine if we can unlock the file again, but
we never actually decremented the writer count, so we held the
lock forever.
This fixes the issue that we could not start a garbage collect after
a reload, as long as the old process is still running, even when that
process has no active backup anymore but another long running task
(e.g. file download, terminal, etc.).
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
document all public things, add some doc links and make some
previously-public things only available for test cases or within the
crate:
previously public, now private:
- AclTreeNode::extract_user_roles (we have extract_roles())
- AclTreeNode::extract_group_roles (same)
- AclTreeNode::delete_group_role (exists on AclTree)
- AclTreeNode::delete_user_role (same)
- AclTreeNode::insert_group_role (same)
- AclTreeNode::insert_user_role (same)
- AclTree::write_config (we have save_config())
- AclTree::load (we have config()/cached_config())
previously public, now crate-internal:
- AclTree::from_raw (only used by tests)
- split_acl_path (used by some test binaries)
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
instead of just logging the error. this should never happen in practice
unless someone is messing with the keyfile, in which case, it's better
to abort.
update tests accordingly (wrong fingerprint should fail, no fingerprint
should get the expected one).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the RSA key and the encryption key itself are hard-coded to avoid
stalling the test runs because of lack of entropy, they have no special
significance otherwise.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
when restoring an encrypted key, the original one is obviously not
available to check the fingerprint with.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
This needs to happen in a separate loop, because some time intervals are not
subsets of others, i.e. weeks and months. Previously, with a daily backup
schedule, having:
* a backup on Sun, 06 Dec 2020 kept by keep-daily
* a backup on Sun, 29 Nov 2020 kept by keep-weekly
would lead to the backup on Mon, 30 Nov 2020 to be selected for keep-monthly,
because the iteration did not yet reach the backup on Sun, 29 Nov 2020 that
would mark November as being covered.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
We do not use the mt utility directly, but the package also provides
an udev helper to correctly initialize tape drives (stinit). Also,
the mt utility is helpful for debugging tap issues.
this fixes the issue that on some filesystems, you cannot recursively
remove a directory when you hold a lock on a file inside (e.g. nfs/cifs)
it is not really backwards compatible (so during an upgrade, there
could be two daemons have the lock), but since the locking was
broken before (see previous patch) it should not really matter
(also it seems very unlikely that someone will trigger this)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
'lock_manifest' returns a Result<File, Error> so we always got the result,
even when we did not get the lock, but we acted like we had.
bubble the locking error up
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if no groups were found, the task log was very confusing as it
contained no real information why nothing was synced, e.g.:
Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
Sync datastore 'local-datastore' from 'remote/datastore'
sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
TASK OK
this patch simply logs how many groups were found and are about to be synced:
Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
Sync datastore 'local-datastore' from 'remote/datastore'
found 0 groups to sync
sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
TASK OK
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
percentage of verified groups, interpolating based on snapshot count
within the group. in most cases, this will also be closer to 'real'
progress since added snapshots (those which will be verified) in active
backup groups will be roughly evenly distributed, while number of total
snapshots per group will be heavily skewed towards those groups which
have existed the longest, even though most of those old snapshots will
only be re-verified very infrequently.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
BackupInfo::list_backup_groups is identical code-wise, and makes more
sense as entry point for listing groups.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
WalkDir does not follow symlinks by default anyway, and this behaviour
is not documented anywhere. e.g., if a sysadmin mounts 'extra storage'
for some backup group or type (not knowing that only metadata is stored
in those directories), GC will ignore all the indices contained within
and happily garbage collect their chunks..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
for safety reason, GC finds and marks all index files below the
datastore base path. as a result of regular operations, only index files
within the expected scheme of <TYPE>/<ID>/<TIMESTAMP> should exist.
add a small check + warning if the index list contains index files out
side of this expected scheme, so that an admin with shell access can
investigate.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
we have messages starting the phases anyway, and limit the number of
progress updates so that context remains available at all times.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
findRecord does not match exactly, but only at the beginning and
case insensitive, by default. Change all calls to be case sensitive
and an exactmatch (we never want the default behaviour afaics).
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
before adding more fields to the tuple, let's just create the struct
inside the match arms to improve readability.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and use this information to add more information to client backup log
and guide the download manifest decision.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
the errors Vec can contain failed groups as well (e.g., if a group has
no or an invalid owner).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
else users have to manually search through a potentially very long task
log to find the entries that are different.. this is the same summary
printed at the end of a manual verify task.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
from formatting functions to main function, and pass along the key data
lines instead of the full string.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this is stricter than the check that happened on manifest load, as it
also fails if the manifest is signed but we don't have a key available.
add some additional output at the start of a backup to indicate whether
a previous manifest is available to base the backup on.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
otherwise loading will run into the signature mismatch which is
technically true, but not the complete picture in this case.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
if the manifest is signed/the contained archives/blobs are encrypted.
stored in 'unprotected' area, since there is already a strong binding
between key and manifest via the signature, and this avoids breaking
backwards compatibility for a simple usability improvement.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
and set/generate it on
- key creation
- key passphrase change
- key decryption if not already set
- key encryption with master key
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
since we systemd-encode parts of the upid string, and those can contain
characters that are invalid in urls (e.g. '\'), we have to percent encode
those
add a 'percent_encode_component' helper, so that we can maybe change
the AsciiSet for all uses at the same time
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by creating a new class that adds a clear trigger and also uses the
clear-trigger image. Code was taken from the one in PBS's prune window,
but we have default values here, so a bit of adapting was necessary. For
example, we don't want to reset to the original value (which might have
been one of the defaults) when clearing, but always to 'null'.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Simplify the phase 2 code by treating .bad files just like regular
chunks, with the exception of stat logging.
To facilitate, we need to touch .bad files in phase 1. We only do this
under the condition that 1) the original chunk is missing (as before),
and 2) the original chunk is still referenced somewhere (since the code
lives in the error handler for a failed chunk touch, it only gets called
for chunks we expect to be there, i.e. ones that are referenced).
Untouched they will then be cleaned up after 24 hours (or after the last
longer-running task finishes).
Reason 2) is also a fix for .bad files not being cleaned up at all if
the original is no longer referenced anywhere (e.g. a user deleting all
snapshots after seeing some corrupt chunks appear).
cond_touch_path is introduced to touch arbitrary paths in the chunk
store with the same logic as touching chunks.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
unprivileged users should only see the counts related to their part of
the datastore.
while we're at it, switch to a list groups, filter groups, count
snapshots approach (like list_snapshots) to speedup calls to this
endpoint when many unprivileged users share a datastore.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
used in the PBS GUI, but also for PVE usage queries which don't need all
the extra expensive information..
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
by listing groups first, then filtering, then listing group snapshots.
this cuts down the number of openat/getdirents calls for users that just
have a partial view of the datastore.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Useful to avoid the need for a long (and possibly changing) list of include-dev
options in certain situations, e.g. nested ZFS file systems. The option is
already implemented and seems to work as expected. The checks for virtual
filesystems are not affected by this option.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
avoids that it shows during store load, we do not know if there are
no datastores at that point and have already a loading mask.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
when we could not load the config (e.g. missing permissions)
show the comment from the global datastore-list
also show a messagebox for a load error instead of setting
the text of the comment box
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The patterns from the archive root's .pxarexclude file are already present in
self.patterns when encode_pxarexclude_cli is called. Pass along the number of
CLI patterns and slice accordingly.
Suggested-By: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
previously a .pxarexclude entry in the root of the archive caused the file to
be generated as well, because the patterns are read before calling
generate_directory_file_list and within the function it wasn't possible to
distinguish between a pattern coming from the CLI and a pattern coming from
archive/root/.pxarexclude
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The documentation states:
.pxarexclude files are treated as regular files and will be included in the
backup archive.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
There is no leading slash in an entry's full_path, causing an anchored
exclude at the root level to fail, e.g. having "/name" as the content of the
file archive/root/.pxarexclude didn't match the file archive/root/name
Fix this by prepending a leading slash before matching.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
the .lint-incremental target, which is implicitly used by the install
target, is still more forgiving to allow faster "change, build, test"
iteration when developing.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Add the versions command to proxmox-backup-manager with a similar output
to pveversion [-v]. It prints the packages line by line with only the
package name, followed by the version and, for proxmox-backup and
proxmox-backup-server, some additional information (running kernel,
running version).
In addition it supports the optional output-format parameter which can
be used to print the complete data in either json, json-pretty or text
format. If output-format is specified, the --verbose parameter is
ignored and the detailed list of packages is printed.
With the addition of the versions command, the report is extended as
well.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
Add an optional string field to APTUpdateInfo which can be used for
extra information.
This is used for passing running kernel and running version information
in the versions API call together with proxmox-backup and
proxmox-backup-server.
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
for now this only does the 'postfix' -> 'postfix@-' conversion,
fixes the issue that we only showed the 'postfix' service syslog
(which is rather empty in a default setup) instead of the instance one
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This patch prints the source of the encryption key when running
operations with proxmox-backup-client.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Currently if you generate a default encryption key:
`proxmox-backup-client key create --kdf none`
all backup operations which don't explicitly disable encryption will be
encrypted with this key.
I found it quite surprising, that my backups were all encrypted without
me explicitly specfying neither key nor encryption mode
This patch informs the user when the default key is used (and no
crypt-mode is provided explicitly)
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
when authenticating a token, and not just when authenticating a
user/ticket.
Reported-By: Dominik Jäger <d.jaeger@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-11 12:21:29 +01:00
637 changed files with 72107 additions and 16552 deletions
This tool implements a backup server client, i.e. it can connect to a
backup servers to issue management commands and to create or restore
..NOTE:: No further info.
backups.
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.