Backport of commit 0bd9c87010
When re-opening a datastore due to the cached entry being stale
(only on verify-new config change). On datastore open the chunk store
was also re-opened, which in turn creates a new ProcessLocker,
loosing any existing shared lock which can cause conflicts between
long running (24h+) backups and GC.
To fix this, reuse the existing ChunkStore, and thus its
ProcessLocker, when creating a up-to-date datastore instance on
lookup, since only the datastore config should be reloaded. This is
fine as the ChunkStore path is not updatable over our API.
Note that this is a precaution backport, the underlying issue this
fixes is relatively unlikely to cause any trouble in the 1.x branch
due to not often re-opening the datastore.
Originally-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this fixes the leaked memory for the cache, as we had only pointers
in the map/list which were freed, not the underlying chunks
moves the 'clear' implementation out of the trait bounds so that
Drop can reuse it
this is used e.g. for file download from a pxar
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
(cherry picked from commit 98983a9dab)
this was also leaking a refcount before, this is fixed now
See-also: proxmox/proxmox-async:
* d0a3e38006fe ("drop RawWaker usage")
* ff132e93c6fd ("rustfmt")
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
send_command serializes everything so it cannot be used to send a
raw, optimized command. Normally that means we get an error like
> 'unable to parse parameters (expected json object)'
when used that way.
Switch over to send_raw_command which does not re-serializes the
command.
Fixes: 45b8a032 ("refactor send_command")
Originally-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This lock is held during VM startup, so that multiple calls will not
start VMs twice. But this means that the timeout needs to incorporate
the time it might take a VM to boot, so increase it quite a bit.
This could previously lead to "interrupted system call" errors when
accessing backups with many disks.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
(cherry picked from commit 66501529a2)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
During startup most of the stuff is happening in milliseconds (or
less), so the timestamp granularity of seconds made it hard to tell
if the previous command required 990ms or 1ms, which is quite the
difference in the restore daemon context.
Using micros seems not to bring too much additional information, a
millisecond is already an ok lower time resolution for logging, so
switch only to millis for now.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit ecd66ecaf6)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
fixes file restore again.
The new Memcom tracking file lives in `/run/proxmox-backup` and is
always created on REST interaction, as CachedUserInfo uses it to
efficiently track config changes, and such a cache is used in each
REST handle_request.
Further, the Memcom infra expects the base run PBS dir to exists
already, which is an OK assumption to have, but in the file-restore
daemon we have a significantly more minimal environment, and the run
dir was simply not required there, even /run isn't a tmpfs yet.
Fixes fda19dcc6f ("fix CachedUserInfo by using a shared memory version counter")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 33d7292f29)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We send it already to the user via the response body, but the
log_response does not has, nor wants to have FWIW, access to the
async body stream, so pass it through the ErrorMessageExtension
mechanism like we do else where.
Note that this is not only useful for PBS API proxy/daemon but also
the REST server of the file-restore daemon running inside the restore
VM, and it really is *very* helpful to debug things there..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit f4d371d2d2)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the dns plugin config allow for a specified amount of time to wait for
the TXT record to be set and propagated through DNS.
This patch adds a sleep for this amount of time.
The log message was taken from the perl implementation in proxmox-acme
for consistency.
Tested with the powerdns plugin in my test setup.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
(cherry picked from commit 3f84541412)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
extract_challenge is used by both dns-01 and http-01 challenges.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
(cherry picked from commit f9bd5e1691)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
So callers get more stable results. Most noticeable, the disk list in
the web UI doesn't jump around upon reloading, and while sorting could
be done directly there, like this other callers get the benefit too.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
(cherry picked from commit bbff317aa7)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
a match expresses the fallback slightly nicer and needs no mut,
which is always nice to avoid.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 6b5013edb3)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
According to crypt(3):
"crypt places its result in a static storage area, which will be
overwritten by subsequent calls to crypt. It is not safe to call crypt
from multiple threads simultaneously."
This means that multiple login calls as a PBS-realm user can collide and
produce intermittent authentication failures. A visible case is for
file-restore, where VMs with many disks lead to just as many auth-calls
at the same time, as the GUI tries to expand each tree element on load.
Instead, use the thread-safe variant 'crypt_r', which places the result
into a pre-allocated buffer of type 'crypt_data'. The C struct is laid
out according to 'lib/crypt.h.in' and the man page mentioned above.
Use the opportunity and make both arguments to the rust 'crypt' function
take a &[u8].
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
(cherry picked from commit c4c4b5a3ef)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While the PVE one "bails" too, it has an eval around those and moves
the error to the message property, so lets do so too to ensure a user
can force an update on a too old subscription
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit b81818b6ad)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we try to load the correct media in a loop until we find the correct tape.
when encountering an error or wrong tape, we want to log that (and send
an email if one is set) that requests the correct tape.
while trying to avoid printing the same errors more than once in a row,
we had at least one case (starting with an empty tape in the drive)
which would not print/send any tape request.
reworking that code to use a custom 'TapeRequest' enum, which contains
the state + error message, and a helper that prints and sends an email
when the state changes
this reduces the change check/log to a single variable, instead of 4
(tried, last_media_uuid, last_error, failure_reason)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Add test code to the first locate_file command, compute locate_offset.
Subsequent locate_file commands use that offset.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
we have a static list of filesystems and their capabilities regarding
file attributes and fs features (e.g. sockets/fifos/etc) which also
includes xattrs,acls and fcaps
if we did not know a filesystem by its magic number (for example cephfs),
we did not even attempt to read xattrs, etc.
this patch adds those flags by default to unknown filesystems, and
removes them when we encounter EOPNOTSUPP (to remove the number
of syscalls)
with this, we should be able to catch xattrs/acls/fcaps on all
(unknown) fs types that support them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
since the output:
Result: "<UPID>"
is not really interesting, show instead the task log while
the datastore is creating, since it is now run in a worker
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Setting this to 0 is not just useless, but breaks the logic horribly
enough to cause random segfaults - better forbid this, to avoid someone
else having to debug it again ;)
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
admin/datastore reads linearly only, so no need for cache (capacity of 1
basically means no cache except for the currently active chunk).
mount can do random access too, so cache last 8 chunks for possibly a
mild performance improvement.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Implemented as a seperate struct SeekableCachedChunkReader that contains
the original as an Arc, since the read_at future captures the
CachedChunkReader, which would otherwise not work with the lifetimes
required by AsyncRead. This is also the reason we cannot use a shared
read buffer and have to allocate a new one for every read. It also means
that the struct items required for AsyncRead/Seek do not need to be
included in a regular CachedChunkReader.
This is intended as a replacement for AsyncIndexReader, so we have less
code duplication and can utilize the LRU cache there too (even though
actual request concurrency is not supported in these traits).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Supports concurrent 'access' calls to the same key via a
BroadcastFuture. These are stored in a seperate HashMap, the LruCache
underneath is only modified once a valid value has been retrieved.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Explicitly test that data will stay available and can be retrieved
immediately via listen(), even if the future producing the data and
notifying the consumers was already run in the past.
Wasn't broken or anything, but helps with understanding IMO.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
in PVE, the logic how wearout gets read from the smartctl output was
changed from a vendor -> id map to a sorted list of specific
attribute field names.
copy that list to pbs (in the same order), and use that to get the
wearout
in the future we might want to split the disk logic into its own crate
and reuse it in pve
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if we want the empty value as a valid default value in a combogrid,
we have to explicitely select 'null' else the field will be marked as
dirty
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
in case an invalid drive was configured, now it marks the field
invalid instead of autoselecting the first valid one
this could have lead to users configuring the wrong drive in a
tape-backup-job when they edited one
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we skip snapshots that are older than the newest snapshot of the group in
the target datastore, log it so the user can know why it is not synced
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that a user can remove a datastore from the gui,
though no data is deleted, this has to be done elsewhere (for now)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that longer running creates (e.g. a slow storage), does not
run in a timeout and we can follow its creation
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
when we remove a datastore via api/cli, the proxy
has sometimes leftover references to that datastore in its
DATASTORE_MAP which includes an open filehandle on the
'.lock' file
this prevents unmounting/exporting the datastore even after removal,
only a reload/restart of the proxy did help
add a command to our command socket, which removes all non
configured datastores from the map, dropping the open filehandle
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
by implementing a custom error type that is either 'TimeOut' or
'Other'.
In the api, check in the worker loop for exactly 'TimeOut' errors and continue only
then. All other errors lead to a aborted task.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
removing the backup dir must acquire the snapshot lock, else it can
happen that we remove a snapshot while it is being restored
or backed up to tape
the original commit that adds the force flag
(c9756b40d1)
mentions that the prune checks itself if the snapshot is in use,
but i could not find such code, so simply set force to false
to avoid failing and aborting the prune job, warn if it could not
and continue
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that the update does not get canceled because of a bad datastore
hide the irrelevant fields in that case
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This reverts commit 75f9f40922, which is
no longer needed now that we use tokio >= 1.6 which contains the proper
fix.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it contains a bug fix that allows dropping the workaround in
75f9f40922 file-restore-daemon: work around tokio DuplexStream bug
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
from the SspDataEncryptionCapabilityPage
it seems we do not need it, since the EXTDECC flag is only used for
determining if the drive is capable to be configured via
ADI (Automation/Drive Interface) which we do not use at all.
this makes the call work with LTO-4 again
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this was previously set on the button class, but has since been removed
add it here to have the badge number centered again
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by default the treelist gets the 'nav' ui, which in newer extjs
versions has a custom styling (unlike before)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by using beforedestroy instead of destroy (like we do everywhere else)
to avoid race condition when the controller has
already removed some handlers on destruction
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
using container here is fine, we do not need panel behaviour which
is more bloated. Removes two ARIA warnings.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
looks (almost confusingly) empty else and no real disadvantage in
showing the disabled one until a media-set is selected and loaded
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
for that we need to split the prefilter additions, else
we always filter the snaphots too and giving 'undefined' filters
all snapshots...
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
for the case that the user selects only whole datastores, we do not
want to send and (exhaustive) list of snapshots that get restored,
but we only want to honor the mapping the user gives
this avoids using the backup restore codepath that iterates twice
over the tapes and would generally be slower for a lot of snapshots
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when no uuid/mediaset is given.
we change a bit how we use the uuid by moving it into the viewmodel
(instead of a simple property on the view) so that we can always
use the selected one
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to make it clear that this button is for restore and for
now we do not have any plans to add buttons here
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we want a 'media-set' selector in the gui, this makes it
very easy to do and is not as costly as reusing the media list,
since we do not need to iterate over all media (e.g. unassigned)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
normally, users will not have many tape media pools,
and are more interested in the actual media-sets, so
expand those nodes by default
if the list gets very long, the user can collapse some pools anyway
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by extracting them via the api macro into the function signature
this fixes an issue, where giving 'since' and 'until' where not
used since we tried to extract them as 'str' while they were numbers.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While the issue with vsock packets starving kernel memory is mostly
worked around by the '64k -> 4k buffer' patch in
'proxmox-backup-restore-image', let's be safe and also limit the number
of concurrent transfers. 8 downloads per VM seems like a fair value.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
The extract API call may be active for more than the watchdog timeout,
so a simple ping is not enough.
This adds an "inhibit" API, which will stop the watchdog from completing
as long as at least one WatchdogInhibitor instance is alive. Keep one in
the download task, so it will be dropped once it completes (or errors).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
See this PR for more info: https://github.com/tokio-rs/tokio/pull/3756
As a workaround use a pair of connected unix sockets - this obviously
incurs some overhead, albeit not measureable on my machine. Once tokio
includes the fix we can go back to a DuplexStream for performance and
simplicity.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Used to specify a filesystem placed directly on a disk, without a
partition table inbetween. Detected by simply attempting to mount the
disk itself.
A helper "make_dev_node" is extracted to avoid code duplication.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
A bucket might contain multiple (or 0) layers of components in its path
specification, so allow a mapping between bucket type strings and
expected component depth. For partitions, this is 1, as there is only
the partition number layer below the "part" node.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Mainly as Config -> Option is a weird name, Authentication has only
one obj. grid, the node options are only the http-proxy for now and
that is a sort of authentication, so good enough for me for now, but
should be rethought for 2.0 and/or once more node opts are added
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
for now only http-proxy lives there, but we will add more options later,
such as
* email from
* default gui language
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
It was not actually bad, so they're quite opinionated to be honest,
but at least xtypes props must go first and variable declaration
should try to be as near as possible to the actual use as long as
code stays sensible readable/short.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
a restore does not change the tape content, so a reload has no benefit here.
since we're touching those lines, change to 'autoShow' property
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by including the new snapshotselector. If a whole media-set is to be
restored, select all snapshots
to achieve this, we drop the 'restoreid' and 'datastores' properties
for the restore window, and replace them by a 'prefilter' object
(with 'store' and 'snapshot' properties)
to be able to show the snapshots, we now have to always load the
content of that media-set, so drop the short-circuit if we have
the datastores already.
change the layout of the restore window into a two-step window
so that the first tab is the selection what to restore, and on the
second tab the user chooses where to restore (drive, datastore, etc.)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
* handle not rendered call of getErrors
* return 'all' as value if all snaphots where selected
(for better distinction)
* remove the default height
* add checkChange on stores filterChange
(now change also fires on the gridfilter plugin change)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
enable scrolling by default, and handle the case that getErrors gets
called when the component is not yet rendered
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This can happen if the underlying storage failed, in which case we do
not want to fail the whole API call, as it should report the status
of all datastores. So rather add the error inline to the related
store entry and continue.
Allows to nicely visualize those stores in the gui.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this will be used for letting the user select multiple, individual
snapshots on restore (instead of having a single or the whole media-set)
if a 'prefilter' object is given, we filter the grid by those
values using the gridfilter plugins (like in pve's bulk action windows)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of having them in the toolbar. This makes the UI more consistent
with the datastore content view.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it's the only PBS-specific part in there, so let's make it
product-agnostic before moving it off to proxmox-http.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
so that a user can delete a whole group at once, until now, the fastest
way for this was to prune to one snapshot, and delete that
code is basically a copy/paste from the snapshot delete, sans
the 'backup-time' parameter
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that a user can force a new media set, e.g. if he uses the
allocation policy 'continue', but wants to manually start a new
media-set.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if the account does not exist, error with its name
if file loading fails, the error includes the full path
if the content fails to parse, show file & parse error
and in each case mention that it's about loading the acme account file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
syncs behavior with both, the displayed state in the PBS
web-interface, and the behavior of PVE/PMG.
Without this a standard setup would result in a Error like:
> TASK ERROR: no acme client configured
which was pretty confusing, as the actual error was something else
(no account configured), and the web-interface showed "default" as
selected account, so a user had no idea what actually was wrong and
how to fix it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
- refactor the combinators,
- make it take a `&T: Serialize` instead of a Value, and
allow sending the raw string via `send_raw_command`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
return a result with optional fingerprint instead of tuple, allowing
easy extraction of a meaningful error message.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
if the expected fingerprint and the one returned by the server don't
match, print a warning and allow confirmation and proceeding if running
interactive.
previous:
$ proxmox-backup-client ...
Error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915:
new:
$ proxmox-backup-client ...
WARNING: certificate fingerprint does not match expected fingerprint!
expected: ac:cb:6a:bc:d6:b7:b4:77:3e:17:05:d6:b6:29:dd:1f:05:9c:2b:3a:df:84:3b:4d:f9:06:2c:be:da:06:52:12
fingerprint: ab:cb:6a:bc:d6:b7:b4:77:3e:17:05:d6:b6:29:dd:1f:05:9c:2b:3a:df:84:3b:4d:f9:06:2c:be:da:06:52:12
Are you sure you want to continue connecting? (y/n): n
Error: error trying to connect: error:1416F086:SSL routines:tls_process_server_certificate:certificate verify failed:../ssl/statem/statem_clnt.c:1915:
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
this makes it possible to only restore some snapshots from a tape media-set
instead of the whole. If the user selects only a small part, this will
probably be faster (and definitely uses less space on the target
datastores).
the user has to provide a list of snapshots to restore in the form of
'store:type/group/id'
e.g. 'mystore:ct/100/2021-01-01T00:00:00Z'
we achieve this by first restoring the index to a temp dir, retrieving
a list of chunks, and using the catalog, we generate a list of
media/files that we need to (partially) restore.
finally, we copy the snapshots to the correct dir in the datastore,
and clean up the temp dir
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and create the 'email' and 'restore_owner' variable at the beginning,
so that we can reuse them and do not have to pass the sources of those
through too many functions
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by showing
'[format, ...]'
where 'format' is the simple format from the type of the items
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
if the reader thread is already gone here, we panic here, resulting in
a nondescript error message, so simply ignore/warn in that case and
return gracefully
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when changin the remote, there is a high chance that there are different
datastores, and if a user does not pay attention, now the first store
of the new remote is selected, instead of the one with the same name
disable autoSelect and let the user manually select a remote datastore
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we have 2 modi in that window:
* backup has multiple datastores
* backup has single datastore
In the first case we show a 'mapping' grid so that
the user can only restore a part. Here a user sees all source
Datastores and can select a target for each one.
In the second case we only have a single 'Datastore' selector, but
we do not show the source. Because of this, the naming is slightly ambiguous
(is it the 'Source' or the 'Target' ?), so rename it to 'Target Datastore'.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
It may make sense in the future, e.g., if the built-in standalone
type is not enough, e.g., as HTTP**s**, HTTP 2 or even QUIC (HTTP 3)
is wanted in some setups, but for now there's no scenario where one
would profit from adding a new HTTP plugin, especially as it requires
the `data` property to be set, which makes no sense..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we cannot add a plugin with an existing ID so this completion helper
is rather counterproductive...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It will be reused in a later patch in another module which should not
depend on the actual API implementation (ugly and cyclic)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We get lots of warnings due to sphinx complaining about missing
includes for generated synopsis. We do not reference to any of those
for now, so we can ignore that now and supress all standard and
warning output.
Note: Errors are still reported.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
especially for the last group, without this the progress would report:
"percentage done: 100.00% (1 of 2 groups, 1 of 1 group snapshots)"
instead of the more logical
"percentage done: 100.00% (2 of 2 groups)"
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Set PBS_QEMU_DEBUG=1 on a command that starts a VM and then connect to
the debug root shell via:
minicom -D \unix#/run/proxmox-backup/file-restore-serial-10.sock
or similar.
Note that this requires 'proxmox-backup-restore-image-debug' to work,
the postinst script is updated to also generate the corresponding image.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
A PCI bus can only support up to 32 devices, so excluding built-in
devices that left us with a maximum of about 25 drives. By adding a new
PCI bridge every 32 devices (starting at bridge ID 2 to avoid conflicts
with automatic bridges), we can theoretically support up to 8096 drives.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
The guest kernel requires more memory depending on how many disks are
attached. 256 seems to be enough for basically any reasonable and
unreasonable amount of disks though.
For debug instance, make it 1G, as these are never started automatically
anyway, and need at least 512MB since the initramfs (especially when
including a debug build of the daemon) is substantially bigger.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Helps to clean up a VM that has crashed, is not responding to vsock API
calls, but still has a running QEMU instance.
We always check the process commandline to ensure we don't kill a random
process that took over the PID.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
otherwise, the kernel driver exposes file names as iso 8859-1,
but we want to have them as utf8.
This mapping should always work, since UTF16 can be cleanly converted
to UTF8.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if we are given a 'naked' ipv6 without square brackets around it,
we need to add them ourselves, since the address is ambigious otherwise
when we add the port.
e.g. giving 'fe80::1' as address we arrive at the url (with the default port)
'https://fe80::1:8007/'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
some features we need (e.g. READ POSITION long form) are only officially
available with LTO-5, but work on many LTO-4 drives, so move LTO-4 to
'best-effort' support.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by checking the 'checked_chunks' before trying to write to disk
and by doing the existance check in the parallel handler. This way,
we do not have to check the existance of a chunk multiple times
(if multiple source datastores gets restored to the same target
datastore) and also we do not have to wait on the stat before reading
the next chunk.
We have to change the &WorkerTask to an Arc though, otherwise we
cannot log to the worker from the parallel handler
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Split out a separate function scan_chunk_archive() for catalog restores.
Note: Required, because we need to optimize restore_chunk_archive() to
write datastore in separate threads (else thape drive will stop during restore)
API like in PVE:
GET .../info => current cert information
POST .../custom => upload custom certificate
DELETE .../custom => delete custom certificate
POST .../acme/certificate => order acme certificate
PUT .../acme/certificate => renew expiring acme cert
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
This is the highlevel part using proxmox-acme-rs to create
requests and our hyper code to issue them to the acme
server.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
else we sometimes forget to remove it from the 'params' variable
and use that further, running into 'invalid parameter' errors
found by giving 'output-format' paramter to proxmox-tape status
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by checking for definedness of the label (tapes without barcode
have the empty string as label-text) and falling back to the
source slot for the load action
Note: Changed the load-slot API from PUT to POST
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
This allows mounting XFS partitons with 'dirty' states, like from a
running VM. Otherwise XFS tries to write recovery information, which
fails on a read-only mount.
Tested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Drive serials have a character limit of 20, longer names like
"drive-virtio0.img.fidx" or "drive-efidisk0.img.fidx" would get cut off.
Fix this by removing the suffix, it is not necessary to uniquely
identify an image.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Some drives will always return the number of bytes given in the
allocation_length field, but correctly report the data len in the mode
sense header. Simply ignore the excess data.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
include the expected and unexpected sizes in the error message,
so that it's easier to debug in case of an error
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
With the vsock-pkt-buffer fix in proxmox-backup-restore-image, we can
use way less memory for the VM without risking any crashes. 128 MiB
seems to be the lowest it will go and still be fully reliable.
While at it, add the "panic=1" argument to the kernel command line, so
in case the kernel *does* run out of memory, it will at least restart
automatically.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Read image sizes (.pxar.fidx/.img.didx) from manifest and partition
sizes from /sys/...
Requires a change to ArchiveEntry, as DirEntryAttribute::Directory
does not have a size associated with it (and that's probably good).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
this way, the api call does not error out when the file is locked
currently (which means that job is running and we do not need
to update the time)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when a user updates a job schedule, we want to save that point in time
to calculate future runs, otherwise when a user updates a schedule to
a time that would have been between the last run and 'now' the
schedule is triggered instantly
for example:
schedule 08:00
last run today 08:00
now it is 12:00
before this patch:
update schedule to 11:00
-> triggered instantly since we calculate from 08:00
after this patch:
update schedule to 11:00
-> triggered tomorrow 11:00 since we calculate from today 12:00
the change in the enum type is ok, since by default serde does not
error on unknown fields and the new field is optional
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a backup task failed (e.g. it was aborted), show the snapshots
which were successfully backed up in the notification
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when the user start an action where we know that it locks the drive,
reload the tape store, so that the state is refreshed
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to make the following cryptic error:
proxmox-file-restore failed: Error: Invalid byte 46, offset 5.
more understandable:
proxmox-file-restore failed: Error: Failed base64-decoding path '/root.pxar.didx' - Invalid byte 46, offset 5.
when a user passes in a non-base64 path but sets `--base64`.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
basically the same as commit eeff085d9d
Will be required once we get to use a newer rustc, at least the
client build for archlinux was broken due to this.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
same functionality as crypto_parameters, except it keeps the file
descriptor passed as "keyfd" open (and seeks to the beginning after
reading), if one is given.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
For the actual partitions and blockdevices in a backup, which the
user sees like folders in the file-restore ui
Encoded as "None", to avoid cluttering DirEntryAttribute, where it
wouldn't make any sense to have.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
These can't be entered or restored anyway, and cause issues with catalog
files for example.
Also a clippy fix.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
in some configurations, samba stores NTFS-ACLs in this xattr[0], so
we should backup (if we can)
altough the 'security' namespace is special (e.g. in use by
selinux, etc.) this value is normally only used by samba and we
should be able to back it up.
to restore it, the user needs at least 'CAP_SYS_ADMIN' rights, otherwise
it cannot be set
0: https://www.samba.org/samba/docs/current/man-html/vfs_acl_xattr.8.html
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Some changer seem to append more data than we expect, but correctly
annotates that size in the subheader.
For each descriptor entry, read as much as the size given in the
subheader (or until the end of the reader), else our position in
the reader is wrong for the next entry, and we will parse
incorrect data.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
only check every 1024'th, which is cheaper to do than a modulo, as we
can just mask the 10 least-significant-bits and check if the result
is zero.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Fixes a non-negligible performance regression from commit
7f394c807b
While we skip known-verified chunks in the stat-and-inode-sort loop,
those are only the ones from previous indexes. If there's a repeated
chunk in one index they would get re-verified more often as required.
So, add the check again explicitly to the read+verify loop.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
before reading the chunks from disk in the order of the index file,
stat them first and sort them by inode number.
this can have a very positive impact on read speed on spinning disks,
even with the additional stat'ing of the chunks.
memory footprint should be tolerable, for 1_000_000 chunks
we need about ~16MiB of memory (Vec of 64bit position + 64bit inode)
(assuming 4MiB Chunks, such an index would reference 4TiB of data)
two small benchmarks (single spinner, ext4) here showed an improvement from
~430 seconds to ~330 seconds for a 32GiB fixed index
and from
~160 seconds to ~120 seconds for a 10GiB dynamic index
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a user has not configured a drive for a specified driveslot of the
changer, simply hide that slot
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The idea is that people first need to make actual backups before they
need to do maintenance tasks.
Network is already setup when installing with the ISO or on-top of
Debian, so that is not a priority either.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
when we get an error from the tape, we possibly want to ignore it,
i.e. when the file was incomplete, but we still want to error
out if the error came from e.g, the datastore, so we have to move
the error checking code to the 'next_chunk' call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
1. The exit was never called as `test ... || echo "foo" || exit 1`
can never come to the exit, as echo will not fail
2. The echo was meant to be redirected to stderr (FD #2) but it was
actually redirected to a file named '2'
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Treat filepaths like "/root.pxar.didx" without a trailing slash as
wanting to download the entire archive content instead of erroring. The
zip-creation code already works fine for this scenario.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
datastore and tape entries are very similar but differ in some points
in such a way that a nice unification is not really that helpful, but
making similar key parts the same is still nice when reading the code
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
in proxmox-backup-proxy, we log and discard any errors on 'accept',
so that we can continue to server requests
in proxmox-backup-api, we just have the StaticIncoming that accepts,
which will forward any errors from the underlying TcpListener
this patch also logs and discards the errors, like in the proxy.
Otherwise it could happen that if the api-daemon has more files open
than the proxy, it will shut itself down because of a
'too many open files' error if there are many open connections
(the service should also restart on exit i think, but this is
a separate issue)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a datastore or root is not used directly on the pool dir
(e.g. the installer creates 2 sub datasets ROOT/pbs-1), info in
/proc/self/mountinfo returns not the pool, but the path to the
dataset, which has no iostats itself in /proc/spl/kstat/zfs/
but only the pool itself
so instead of not gathering data at all, gather the info from the
underlying pool instead. if one has multiple datastores on the same
pool those rrd stats will be the same for all those datastores now
(instead of empty) similar to 'normal' directories
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
should not be a hard dependency, as one can use the file-restore tool
for pxar archives without it too
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The data on the restore daemon is either encoded into a pxar archive, to
provide the most accurate data for local restore, or encoded directly
into a zip file (or written out unprocessed for files), depending on the
'pxar' argument to the 'extract' API call.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Encodes an entire local directory into an AsyncWrite recursively.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
extract_sub_dir_seq, together with seq_files_extractor, allow extracting
files from a pxar Decoder, along with the existing option for an
Accessor. To facilitate code re-use, some helper functions are extracted
in the process.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Allows listing files and directories on a block device snapshot.
Hierarchy displayed is:
/archive.img.fidx/bucket/component/<path>
e.g.
/drive-scsi0.img.fidx/part/2/etc/passwd
(corresponding to /etc/passwd on the second partition of drive-scsi0)
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This will be triggered on updating proxmox-file-restore (via configure,
necessary since the daemon binary might change) and
proxmox-backup-restore-image (via 'activate-noawait', necessary since
the base image might change).
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Includes methods to start, stop and list QEMU file-restore VMs, as well
as CLI commands do the latter two (start is implicit).
The implementation is abstracted behind the concept of a
"BlockRestoreDriver", so other methods can be implemented later (e.g.
mapping directly to loop devices on the host, using other hypervisors
then QEMU, etc...).
Starting VMs is currently unused but will be needed for further changes.
The design for the QEMU driver uses a locked 'map' file
(/run/proxmox-backup/$UID/restore-vm-map.json) containing a JSON
encoding of currently running VMs. VMs are addressed by a 'name', which
is a systemd-unit encoded combination of repository and snapshot string,
thus uniquely identifying it.
Note that currently you need to run proxmox-file-restore as root to use
this method of restoring.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Includes functionality for scanning and referring to partitions on
attached disks (i.e. snapshot images).
Fairly modular structure, so adding ZFS/LVM/etc... support in the future
should be easy.
The path is encoded as "/disk/bucket/component/path/to/file", e.g.
"/drive-scsi0/part/0/etc/passwd". See the comments for further
explanations on the design.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Add a watchdog that will automatically shut down the VM after 10
minutes, if no API call is received.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Implements the base of a small daemon to run within a file-restore VM.
The binary spawns an API server on a virtio-vsock socket, listening for
connections from the host. This happens mostly manually via the standard
Unix socket API, since tokio/hyper do not have support for vsock built
in. Once we have the accept'ed file descriptor, we can create a
UnixStream and use our tower service implementation for that.
The binary is deliberately not installed in the usual $PATH location,
since it shouldn't be executed on the host by a user anyway.
For now, only the API calls 'status' and 'stop' are implemented, to
demonstrate and test proxmox::api functionality.
Authorization is provided via a custom ApiAuth only checking a header
value against a static /ticket file.
Since the REST server implementation uses the log!() macro, we can
redirect its output to stdout by registering env_logger as the logging
target. env_logger is already in our dependency tree via zstd/bindgen.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This allows switching the base user identification/authentication method
in the rest server. Will initially be used for single file restore VMs,
where authentication is based on a ticket file, not the PBS user
backend (PAM/local).
To avoid putting generic types into the RestServer type for this, we
merge the two calls "extract_auth_data" and "check_auth" into a single
one, which can use whatever type it wants internally.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
For now it only supports 'list' and 'extract' commands for 'pxar.didx'
files. This should be the foundation for a general file-restore
interface that is shared with block-level snapshots.
This is packaged as a seperate .deb file, since for block level restore
it will need to depend on pve-qemu-kvm, which we want to seperate from
proxmox-backup-client.
[original code for proxmox-file-restore.rs]
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[code cleanups/clippy, use helpers::list_dir_content/ArchiveEntry, no
/block subdir for .fidx files, seperate binary and package]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
by using format_boolean for compression/write protect,
combining file/block posiition into one (saves a line)
and adding the missing alert-flags
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it is rather pointless to let the user select something were there
is no choice. We have to keep the window though, since the user may
want to choose a pool
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a changer only has a single drive, there is no point in showing
a window with a DriveSelector, just do want the user wanted.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to make it more like a 'dangerous' remove window
also works in the singleDrive logic to hide/show the driveselector
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
for some storages, it is valuable information, e.g. if one has datastores
on separate datasets of the same zpool
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
I made some comparision with bombardier[0], the one listed here are
30s looped requests with two concurrent clients:
[ static download of ext-all.js ]:
lvl avg / stdev / max
none 1.98 MiB 100 % 5.17ms / 1.30ms / 32.38ms
fastest 813.14 KiB 42 % 20.53ms / 2.85ms / 58.71ms
default 626.35 KiB 30 % 39.70ms / 3.98ms / 85.47ms
[ deterministic (pre-defined data), but real API call ]:
lvl avg / stdev / max
none 129.09 KiB 100 % 2.70ms / 471.58us / 26.93ms
fastest 42.12 KiB 33 % 3.47ms / 606.46us / 32.42ms
default 34.82 KiB 27 % 4.28ms / 737.99us / 33.75ms
The reduction is quite better with default, but it's also slower, but
only when testing over unconstrained network. For real world
scenarios where compression actually matters, e.g., when using a
spotty train connection, we will be faster again with better
compression.
A GPRS limited connection (Firefox developer console) requires the
following load (until the DOMContentLoaded event triggered) times:
lvl t x faster
none 9m 18.6s x 1.0
fastest 3m 20.0s x 2.8
default 2m 30.0s x 3.7
So for worst case using sligthly more CPU time on the server has a
tremendous effect on the client load time.
Using a more realistical example and limiting for "Good 2G" gives:
none 1m 1.8s x 1.0
fastest 22.6s x 2.7
default 16.6s x 3.7
16s is somewhat OK, >1m just isn't...
So, use default level to ensure we get bearable load times on
clients, and if we want to improve transmission size AND speed then
we could always use a in-memory cache, only a few MiB would be
required for the compressable static files we server.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
by using our DeflateEncoder
for this to work, we have to create wrapper reader that generates the crc32
checksum while reading.
also we need to put the target writer in an Option, so that we can take
it out of self and move it into the DeflateEncoder while writing
compressed
we can drop the internal buffer then, since that is managed by the
deflate encoder now
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
implements a deflate encoder that can compress anything that implements
AsyncRead + Unpin into a file with the helper 'compress'
if the inner type is a Stream, it implements Stream itself, this way
some streaming data can be streamed compressed
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Else it does not gets picked up on release builds...
Also the mathjax path option affects HTML not EPUB so move it to the
correct section in conf.py
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Previously we did not store GROUP_OBJ ACL entries for
directories, this means that these were lost which may
potentially elevate group permissions if they were masked
before via ACLs, so we also show a warning.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Don't override `group_obj` with `None` when handling
`ACL_TYPE_DEFAULT` entries for directories.
Reproducer: /var/log/journal ends up without a `MASK` type
entry making it invalid as it has `USER` and `GROUP`
entries.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Add a new module containing key-related functions and schemata from all
over, code moved is not changed as much as possible.
Requires adapting some 'use' statements across proxmox-backup-client and
putting the XDG helpers quite cozily into proxmox_client_tools/mod.rs
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Pass in an optional auth tag, which will be passed as an Authorization
header on every subsequent call.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
To let users find the good explanation about allocation and retention
policies from the docs easier.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This way we get a better rendering in the api-viewer.
before:
[<string>, ... ]
after:
[(<source>=)?<target>, ... ]
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding a custom field (grid) where the user can select
a target datastore for each source datastore on tape
if we have not loaded the content of the media set yet,
we have to load it on window open to get the list of datastores
on the tape
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by changing the 'store' parameter of the restore api call to a
list of mappings (or a single default datastore)
for example giving:
a=b,c=d,e
would restore
datastore 'a' from tape to local datastore 'b'
datastore 'c' from tape to local datastore 'e'
all other datastores to 'e'
this way, only a single datastore can also be restored, by only
giving a single mapping, e.g. 'a=b'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The text 'had to upload [KMG]iB' implies that this is the size we
actually had to send to the server, while in reality it is the
raw data size before compression.
Count the size of the compressed chunks and print it separately.
Split the average speed into its own line so they do not get too long.
Rename 'uploaded' into 'size_dirty' and 'vsize_h' into 'size'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in commit `asyncify pxar create_archive`, we changed from a
separate thread for creating a pxar to using async code, but the
StdChannelWriter used for both pxar and catalog can block, which
may block the tokio runtime for single (and probably dual) core
environments
this patch adds a wrapper struct for any writer that implements
'std::io::Write' and wraps the write calls with 'block_in_place'
so that if called in a tokio runtime, it knows that this code
potentially blocks
Fixes: 6afb60abf5 ("asyncify pxar create_archive")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This is just an optimization, avoiding to read the catalog into memory.
We also expose create_temporary_database_file() now (will be
used for catalog restore).
- new helper: lock_media_set()
- MediaPool: lock media set
- Expose Inventory::new() to avoid double loading
- do not lock pool on restore (only lock media-set)
- change pool lock name to ".pool-{name}"
so that a user can schedule multiple backup jobs onto a single
media pool without having to consider timing them apart
this makes sense since we can backup multiple datastores onto
the same media-set but can only specify one datastore per backup job
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the default escape handler is handlebars::html_escape, but this are
plain text emails and we manually escape them for the html part, so
set the default escape handler to 'no_escape'
this avoids double html escape for the characters: '&"<>' in emails
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead print an error and continue, the rendering functions will error
out if one of the templates could not be registered
if we `.unwrap()` here, it can lead to problems if the templates are
not correct, i.e. we could panic while holding a lock, if something holds
a mutex while this is called for the first time
add a test to catch registration issues during package build
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since we can now backup multiple datastores in the same media-set,
we show the datastores as first level below that
the final tree structucture looks like this:
tapepool A
- media set 1
- datastore I
- tape x
- ct/100
- ct/100/2020-01-01T00:00:00Z
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
I mean the whole distro uses quite some C and the like as base, so
avoid being overly strict here.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
It does not help users if that is spelled out, and its not a common
use of GCM, and especially in the AES 256 context its clear what is
meant. The link to Wikipedia stays, so interested people can still
read up on it and others get a better overview due to the text being
more concise.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The main feature list should provide a short overview of the, well,
main features. While enterprise support *is* a main and important
feature, it's not the place here to describe things like personal
volume/ngo/... offers and the like.
Move parts of it to getting help, which lacked mentioning the
enterprise support too and is a good place to describe the customer
portal.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this is much handier than number field, and the user can instantly
see which one is an import/export slot
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that the tape backup can be restored as any user, given
the current logged in user has the correct permission.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Else one has quite a terrible UX when installing from 1.0 ISO and
then upgrading to latest release..
commit 0ec79339f7 for the fix and some other details
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
instead of always using the starttime, use the last timestamp from the log
this way, one can see when the task was aborted without having to read
the log
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and reuse 'send_job_status_mail' there so that we get consistent
formatted mails from pbs (e.g. html part and author)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
found and semi-manually replaced by using:
codespell -L mut -L crate -i 3 -w
Mostly in comments, but also email notification and two occurrences
of misspelled 'reserved' struct member, which where not used and
cargo build did not complain about the change, soo ...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
from 2 to 60 seconds. To retain the response time of the gui
when adding/editing/removing, trigger a manual reload on these actions
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
iterating over a nodeinterfaces children while removing them
will lead to 'child' being undefined
instead collect the children to remove in a separate list
and iterate over them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when we do a prune during a tape backup, do not cancel the tape backup,
but continue with a warning
the task still fails and prompts the user to check the log
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Less anti-aliasing to do, so it looks better when small.
Note, I use a mask as else we could not cut out something
transparent, but only over paint it filled with white, which is not
the same and not how font awesome does it.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The old value just was way to heavy, and notes/warnings/...
admonitions did not stick out anymore.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
1. Use same height as width
2. drop the top/bottom "frame" lines
3. instead of two "bearings" side by side, like VHS or music-compact
tapes have, use a single bearing and an outer circle to denote the
magnetic tape edge
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Simplifies the introduction a bit and makes it more readable.
Also some other minor language fixes throughout the section.
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
Try once first to load the correct tape before sending an email to
insert the correct one. This way, the admin does not get a mail
if the correct tape is already inserted.
Also include the error we got that prompted the email to insert the
tape. This means that if the admin gets prompted to insert e.g.
"FOO" but inserts "BAR", he'll get an email that the wrong
tape is inserted.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of silently discarding the error, else the user might be
confused because nothing happened
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding the existing credential id to the 'excludeCredentials' list
this prevents the browser from registering a token twice, which
lets authentication fail on some browser/token combinations
(e.g. onlykey/solokey+chromium)
while is seems this is currently a bug in chromium, in a future spec
update the underlying behaviour should be better defined, making this
an authenticator bug
also explicitly catch registering errors and show appropriate error messages
0: https://bugs.chromium.org/p/chromium/issues/detail?id=1087642
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
even if the options *could* be set in the frontend, the backend
actually has to do validation of those settings, thus we should not
make that a browser setting
additionally, having the value 'preferred' does not actually make sense,
since it does not add any security (the backend skips the
userverification check then)
This reverts commit aca4c2b5a9.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the current default is 'Preferred', which is not really useful, as the
(web) client can simply change this to discouraged, since the
webauthn_rs crate does not verify the 'user_verified' bit of the
response in that case
setting this to 'Required' is not really useful either at the moment,
since a user can have a mix of different authenticators that may or
may not support user verification
there is ongoing discussion in the crate how to handle that[0]
we could probably expose this setting(discouraged/required) to the user/admin
and save it to the credential and allow only registering credentials
of the same type or filter them out on login (i.e. if there is an
authenticator that can handle userVerification, require it)
in any case, the current default is not helpful for security, but
makes loggin in harder, since the key will by default want to verify
the user
0: https://github.com/kanidm/webauthn-rs/pull/49
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The previous description in the docs was false. The restore command
with the pattern parameter will search the entire backup archive,
regardless of pwd.
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
If the usage of a datastore did not change, we did not
return an estimate. The ui interpreted this as 'not enough data', but
it should actually be 'never'.
Fixing this by always setting the estimate first to 0 and overwriting
if we successfully calculated one, and checking for 'undefined' in the ui.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since the status api call blocks the drive, it only makes sense to do
when the drive is idle.
so do not reload on activate, but when the drive changes the first
time from busy to idle
also disable the reload button when the drive is busy
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we test for the config key in the API so it makes sense to have as
test here too. Actually it would be better if we'd have a expect
Value defined here and enforce that it matches, but better than
nothing.
Fix the input for test 1, where tabs got replaced by spaces, as else
it fails
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
For tapes with many snapshots/group, a simple list of them is too
big. Instead, add a level for just the groups, this makes searching
for a specific backup much easier.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since reload is an async function, the view can be destroyed during
any 'await' point. Subsequent accesses to the view will fail, and we
will land in the catch. Check there if the view is destroyed, and
do not raise an error with the user then
also cancel any outstanding timer on 'deactivate' and 'destroy'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
because we now select it directly in the left-hand tree
so we have to adapt the changer to the one set by the router
and not by the drop down field
and remove it from the TapeManagement tabpanel
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and only check TapeManagement once in the init function
we now have 2 updatestores that update individually
(one for datastores, one for drives/changers)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by generalizing the isDataStorePath logic to a 'parseRouterPath'.
We still have to keep the isDataStore logic for tabpanel handling,
If we add tabs to changer-/drivestatus panels, we have to adapt
that too.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Americans seem to sometimes use Avery 6577 which has 5/8" x 3"
labels, equaling 15.875 mm x 76.2 mm, so do not set the lower bound
to 17mm (which even breaks our used Avery 3240 sheets which have an
label height of 16.9 mm
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
By default, sphinx embeds the cloudflare CDN version of mathjax. This
is bad for privacy, webistes enforcing cross-site origin protection
and in environments with no WAN access.
Luckily there's a Debian package we can use instead.
The config is the default sphinx config used.
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
marketing noted that it looked quite heavy weight, to the point where
it was hard to read to the full black squares, bold monospace text
and bottom borders with rather distinctive darkness.
Address those by:
* change color for list points from black to mid-dark grey
* use empty circles for second heading level
* ensure monospaced text has a normal font weight in the TOC headings
* some lighter color for the bottom border
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
As 20s is really not that high, especially for loaded setups one is
connected to through a spotty network (looking at you ÖBB railnet)
and gets latency spikes of 5 - 10s for some minutes at a time..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since the PUT api call is using the 'Updater', the 'id' parameter is
already encoded in there, tripping up the api verify tests with
'Duplicate keys found in AllOf schema: id'
"fixing" it by removing the explicit id from the api call and
taking it from the Updater (and failing if it does not exists there;
even though that should never happen)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
a 'leaf' node is every file *except* directories, so we have
to reverse the logtic here
this fixes the pxar.didx browser in the web ui
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of filling them with zeroes
this fixes an issue where we could not restore a container with large
sparse files in the backup (e.g. a 10GiB sparse file in a container
with a 8GiB disk)
if the last operation of the copy was a seek, we need to truncate
the file to the correct size (seek beyond filesize does not change it)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the DiffStore only updates the fiels given in the model, so we have to
list all the fields we depend on, else we have incomplete data,
e.g. for rendering
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if the drive state is not empty, some action blocks it, so we cannot
do any task anyway. Since we now refresh the state every 5 seconds,
we can disable/enable them 'live'.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Rund `codespell` tool, but it picked up not as much as I hoped.
Rest was found with vim + (hun)spell
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
some fido2/webauthn keys can have a pin, and the client can request
a mode for the user verification.
'default' (no value set), lets the browser/device decide if the user has to
enter the pin of the device
'discouraged' requests that the user should not need to enter the pin
'preferred' requests that the user should need to enter the pin (if possible)
since we use webauthn only as a 2nd factor, having the user enter
the device pin on login may seem too much hassle for some users, so
give them the option
since this is a client option anyway, do not save it in the backend, but
in the browser local storage
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
mostly copied from pve (for now; will refactor when i add it to
pmg too (soon)) without the pve specific features like dashboard
storages
contains some eslint fixes comparing to pves window
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the statement !a === "somestring" cannot be true since
!a is either true or false and thus not a string
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
an optimize the columns for smaller layouts (1280 width)
we show either:
* Idle
* spinner + status (if no upid)
* spinner + rendered UPID (clickable, opens task viewer)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
those calls could also block, so we have to run them in a blocking
tokio task, as to not block the current thread
nice side effect is that we now also update the state for that
drive in those instances
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
simply writes into/reads from a file in /run, we will use this
for writing the upid (or potential other states) per drive
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
We do not use the media location, so there is no need to update
the online status in the list_media call. Besides, we already update
the online status when we query the changer/status.
slot is already in the title of the grid, and hide it by default for
drives (the user does not need it there)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
...including common schemata, connect(), extract_*() and completion
functions.
For later use with proxmox-file-restore binary.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
...to take advantage of the aio::Encoder from the pxar create.
Rather straightforward conversion, but does require getting rid of
references in the Archiver struct, and thus has to be given the Mutex
for the catalog directly. The callback is boxed.
archive_dir_contents can call itself recursively, and thus needs to
return a boxed future.
Users are adjusted, namely PxarBackupStream is converted to use an
Abortable future instead of a thread so it supports async in its handler
function, and the pxar bin create_archive is converted to an async API
function. One test case is made to just use 'block_on'.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
to extract some subdirectory of a pxar into a given target
this will be used in the client
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
we will reuse that code in the client, so we need to move it to
where we can access it from the client
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[clippy fixes]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
we will reuse that later in the client, so we need it somewhere
we can use from there
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[add strongly typed ArchiveEntry and put api code into helpers.rs]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Currently useful only for single file restore, but kept generic enough
to use any compatible API endpoint over a virtio-vsock[0,1] interface.
VsockClient is adapted and slimmed down from HttpClient.
A tower-compatible VsockConnector is implemented, using a wrapped
UnixStream as transfer. The UnixStream has to be wrapped in a custom
struct to implement 'Connection', Async{Read,Write} are simply forwarded
directly to the underlying stream.
[0] https://www.man7.org/linux/man-pages/man7/vsock.7.html
[1] https://wiki.qemu.org/Features/VirtioVsock
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
This allows anything that can be represented as a UnixStream to be used
as transport for an API server (e.g. virtio sockets).
A tower service expects an IP address as it's peer, which we can't
reliably provide for unix socket based transports, so just fake one.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
but auto-expand them, so no additional click is necessary
this shows the user which tapes are involved for the media sets
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a catalog is missing (or the loading otherwise throws an error), show
the error message in a msg box instead of a mask. this way a user can
still navigate the tree
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if given, erases the tape only iff the inserted tape contains that label
used to safeguard tape erasing from ui for standalone drives
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adapting and reusing the 'LabelMedia' window
shows a short notice about inserting the correct tape in the drive
and now allows to select a drive
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
listing the media can potentially hang (changer status), so do not
reload in the background, only on activate
also increase the timeout to 5 minutes and add a reload button
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since tape commands can take a while and we do not want to change
all of those to worker tasks, increase the timeout to 5 minutes
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in most uses, we want to remove the drive from the param afterwards
where we don't, we already overwrite it with the result of this function
this fixes some commands (like 'proxmox-tape read-label --drive foo')
that failed with:
parameter 'drive': duplicate parameter.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
do not only show if labeled, but the whole status (full/writeable/etc.)
and to which pool the tape belongs (if any)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of showing the snapshots directly under the pool and then the
media-sets, list the media-sets under the pool and only after the
snapshots
this has several advantages:
* we only have to read one set of tape catalog data on expand and not all of
them everytime (which does not scale)
* we can show media-sets without snapshots, this can happen when we
inventoried a set of tapes from another pbs instance, or lost the
catalog data somehow
the disadvantage is that one has to go look for the media set where the
snapshot is included, but we can solve this by implementing a search
function in the future (in the backend)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
we include all tasks whose type start with 'verif' in the type 'verify'
but if we want to actually show them in the pop-up, we have to reverse
map this back to 'verif', else there are tasks missing from there
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to avoid confusing messages about using encryption keys when restoring
plaintext backups, or about loading master keys when they are not
actually used for the current operation.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
pull out the crypt-mode to logically group arms and make the whole mess
a bit more "human-parsable".
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it's needed for PVE's LXC integration, and might be interesting for
other more special usage scenarios as well.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
If the ref is named the same as the headline (once normalized), sphinx
will return a 'idX' value in node['ids'][1] which we use for the label
ID. The headline is always present at index 0.
Checking for that and using index 0 in case we do get a 'idX' helps us
to avoid using the 'idX' as keys in our OnlineHelpInfo.js and actually
use the intended key.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
With commit ec1ae7e631 some refs were
changed by getting prefixes and such. We need to adapt the places that
reference them as well
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
and fix some issues from referenced named the same as their heading
they anchor too.
This should be fixed for real in our python plugin to scan for such
references, its probably a bug there, but as most of the problematic
ones where wrong (missing chapter prefix) anyway changing them is OK
too.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
"we" should be avoided, it's never quite clear who is "we" in the
context here and it leads to some technical wrong meanings, e.g., we
(here assumed to be "we developers") do not read any backup data, the
Proxmox Backup client does.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
adds explanations for:
* what datastores are
* their relation with snapshots/chunks
* basic information about chunk directory structures
* fixed-/dynamically-sized chunks
* special handling of encrypted chunks
* hash collision probability
* limitation of file-based backups
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
fixes connecting to hosts with valid certificates without a
pinned fingerprint
this was accidentally changed in the tokio-1.0 updates
apparently
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: 0f860f712f ("tokio 1.0: update to new tokio-openssl interface")
@ -25,4 +25,7 @@ either explain things which are different on `Proxmox Backup`_, or
tasks which are commonly used on `Proxmox Backup`_. For other topics,
please refer to the standard Debian documentation.
..include:: local-zfs.rst
..include:: services.rst
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
