before reading the chunks from disk in the order of the index file,
stat them first and sort them by inode number.
this can have a very positive impact on read speed on spinning disks,
even with the additional stat'ing of the chunks.
memory footprint should be tolerable, for 1_000_000 chunks
we need about ~16MiB of memory (Vec of 64bit position + 64bit inode)
(assuming 4MiB Chunks, such an index would reference 4TiB of data)
two small benchmarks (single spinner, ext4) here showed an improvement from
~430 seconds to ~330 seconds for a 32GiB fixed index
and from
~160 seconds to ~120 seconds for a 10GiB dynamic index
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a user has not configured a drive for a specified driveslot of the
changer, simply hide that slot
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when we get an error from the tape, we possibly want to ignore it,
i.e. when the file was incomplete, but we still want to error
out if the error came from e.g, the datastore, so we have to move
the error checking code to the 'next_chunk' call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in proxmox-backup-proxy, we log and discard any errors on 'accept',
so that we can continue to server requests
in proxmox-backup-api, we just have the StaticIncoming that accepts,
which will forward any errors from the underlying TcpListener
this patch also logs and discards the errors, like in the proxy.
Otherwise it could happen that if the api-daemon has more files open
than the proxy, it will shut itself down because of a
'too many open files' error if there are many open connections
(the service should also restart on exit i think, but this is
a separate issue)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a datastore or root is not used directly on the pool dir
(e.g. the installer creates 2 sub datasets ROOT/pbs-1), info in
/proc/self/mountinfo returns not the pool, but the path to the
dataset, which has no iostats itself in /proc/spl/kstat/zfs/
but only the pool itself
so instead of not gathering data at all, gather the info from the
underlying pool instead. if one has multiple datastores on the same
pool those rrd stats will be the same for all those datastores now
(instead of empty) similar to 'normal' directories
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
For now it only supports 'list' and 'extract' commands for 'pxar.didx'
files. This should be the foundation for a general file-restore
interface that is shared with block-level snapshots.
This is packaged as a seperate .deb file, since for block level restore
it will need to depend on pve-qemu-kvm, which we want to seperate from
proxmox-backup-client.
[original code for proxmox-file-restore.rs]
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[code cleanups/clippy, use helpers::list_dir_content/ArchiveEntry, no
/block subdir for .fidx files, seperate binary and package]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
by using format_boolean for compression/write protect,
combining file/block posiition into one (saves a line)
and adding the missing alert-flags
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
it is rather pointless to let the user select something were there
is no choice. We have to keep the window though, since the user may
want to choose a pool
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if a changer only has a single drive, there is no point in showing
a window with a DriveSelector, just do want the user wanted.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
to make it more like a 'dangerous' remove window
also works in the singleDrive logic to hide/show the driveselector
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
for some storages, it is valuable information, e.g. if one has datastores
on separate datasets of the same zpool
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by using our DeflateEncoder
for this to work, we have to create wrapper reader that generates the crc32
checksum while reading.
also we need to put the target writer in an Option, so that we can take
it out of self and move it into the DeflateEncoder while writing
compressed
we can drop the internal buffer then, since that is managed by the
deflate encoder now
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
implements a deflate encoder that can compress anything that implements
AsyncRead + Unpin into a file with the helper 'compress'
if the inner type is a Stream, it implements Stream itself, this way
some streaming data can be streamed compressed
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
This way we get a better rendering in the api-viewer.
before:
[<string>, ... ]
after:
[(<source>=)?<target>, ... ]
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding a custom field (grid) where the user can select
a target datastore for each source datastore on tape
if we have not loaded the content of the media set yet,
we have to load it on window open to get the list of datastores
on the tape
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by changing the 'store' parameter of the restore api call to a
list of mappings (or a single default datastore)
for example giving:
a=b,c=d,e
would restore
datastore 'a' from tape to local datastore 'b'
datastore 'c' from tape to local datastore 'e'
all other datastores to 'e'
this way, only a single datastore can also be restored, by only
giving a single mapping, e.g. 'a=b'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
The text 'had to upload [KMG]iB' implies that this is the size we
actually had to send to the server, while in reality it is the
raw data size before compression.
Count the size of the compressed chunks and print it separately.
Split the average speed into its own line so they do not get too long.
Rename 'uploaded' into 'size_dirty' and 'vsize_h' into 'size'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
in commit `asyncify pxar create_archive`, we changed from a
separate thread for creating a pxar to using async code, but the
StdChannelWriter used for both pxar and catalog can block, which
may block the tokio runtime for single (and probably dual) core
environments
this patch adds a wrapper struct for any writer that implements
'std::io::Write' and wraps the write calls with 'block_in_place'
so that if called in a tokio runtime, it knows that this code
potentially blocks
Fixes: 6afb60abf5 ("asyncify pxar create_archive")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that a user can schedule multiple backup jobs onto a single
media pool without having to consider timing them apart
this makes sense since we can backup multiple datastores onto
the same media-set but can only specify one datastore per backup job
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the default escape handler is handlebars::html_escape, but this are
plain text emails and we manually escape them for the html part, so
set the default escape handler to 'no_escape'
this avoids double html escape for the characters: '&"<>' in emails
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead print an error and continue, the rendering functions will error
out if one of the templates could not be registered
if we `.unwrap()` here, it can lead to problems if the templates are
not correct, i.e. we could panic while holding a lock, if something holds
a mutex while this is called for the first time
add a test to catch registration issues during package build
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since we can now backup multiple datastores in the same media-set,
we show the datastores as first level below that
the final tree structucture looks like this:
tapepool A
- media set 1
- datastore I
- tape x
- ct/100
- ct/100/2020-01-01T00:00:00Z
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
this is much handier than number field, and the user can instantly
see which one is an import/export slot
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
so that the tape backup can be restored as any user, given
the current logged in user has the correct permission.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of always using the starttime, use the last timestamp from the log
this way, one can see when the task was aborted without having to read
the log
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and reuse 'send_job_status_mail' there so that we get consistent
formatted mails from pbs (e.g. html part and author)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
from 2 to 60 seconds. To retain the response time of the gui
when adding/editing/removing, trigger a manual reload on these actions
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
iterating over a nodeinterfaces children while removing them
will lead to 'child' being undefined
instead collect the children to remove in a separate list
and iterate over them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
when we do a prune during a tape backup, do not cancel the tape backup,
but continue with a warning
the task still fails and prompts the user to check the log
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Try once first to load the correct tape before sending an email to
insert the correct one. This way, the admin does not get a mail
if the correct tape is already inserted.
Also include the error we got that prompted the email to insert the
tape. This means that if the admin gets prompted to insert e.g.
"FOO" but inserts "BAR", he'll get an email that the wrong
tape is inserted.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of silently discarding the error, else the user might be
confused because nothing happened
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by adding the existing credential id to the 'excludeCredentials' list
this prevents the browser from registering a token twice, which
lets authentication fail on some browser/token combinations
(e.g. onlykey/solokey+chromium)
while is seems this is currently a bug in chromium, in a future spec
update the underlying behaviour should be better defined, making this
an authenticator bug
also explicitly catch registering errors and show appropriate error messages
0: https://bugs.chromium.org/p/chromium/issues/detail?id=1087642
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
even if the options *could* be set in the frontend, the backend
actually has to do validation of those settings, thus we should not
make that a browser setting
additionally, having the value 'preferred' does not actually make sense,
since it does not add any security (the backend skips the
userverification check then)
This reverts commit aca4c2b5a9.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the current default is 'Preferred', which is not really useful, as the
(web) client can simply change this to discouraged, since the
webauthn_rs crate does not verify the 'user_verified' bit of the
response in that case
setting this to 'Required' is not really useful either at the moment,
since a user can have a mix of different authenticators that may or
may not support user verification
there is ongoing discussion in the crate how to handle that[0]
we could probably expose this setting(discouraged/required) to the user/admin
and save it to the credential and allow only registering credentials
of the same type or filter them out on login (i.e. if there is an
authenticator that can handle userVerification, require it)
in any case, the current default is not helpful for security, but
makes loggin in harder, since the key will by default want to verify
the user
0: https://github.com/kanidm/webauthn-rs/pull/49
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
If the usage of a datastore did not change, we did not
return an estimate. The ui interpreted this as 'not enough data', but
it should actually be 'never'.
Fixing this by always setting the estimate first to 0 and overwriting
if we successfully calculated one, and checking for 'undefined' in the ui.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since the status api call blocks the drive, it only makes sense to do
when the drive is idle.
so do not reload on activate, but when the drive changes the first
time from busy to idle
also disable the reload button when the drive is busy
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
For tapes with many snapshots/group, a simple list of them is too
big. Instead, add a level for just the groups, this makes searching
for a specific backup much easier.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
since reload is an async function, the view can be destroyed during
any 'await' point. Subsequent accesses to the view will fail, and we
will land in the catch. Check there if the view is destroyed, and
do not raise an error with the user then
also cancel any outstanding timer on 'deactivate' and 'destroy'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
because we now select it directly in the left-hand tree
so we have to adapt the changer to the one set by the router
and not by the drop down field
and remove it from the TapeManagement tabpanel
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
and only check TapeManagement once in the init function
we now have 2 updatestores that update individually
(one for datastores, one for drives/changers)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
by generalizing the isDataStorePath logic to a 'parseRouterPath'.
We still have to keep the isDataStore logic for tabpanel handling,
If we add tabs to changer-/drivestatus panels, we have to adapt
that too.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
By default, sphinx embeds the cloudflare CDN version of mathjax. This
is bad for privacy, webistes enforcing cross-site origin protection
and in environments with no WAN access.
Luckily there's a Debian package we can use instead.
The config is the default sphinx config used.
Reported-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
since the PUT api call is using the 'Updater', the 'id' parameter is
already encoded in there, tripping up the api verify tests with
'Duplicate keys found in AllOf schema: id'
"fixing" it by removing the explicit id from the api call and
taking it from the Updater (and failing if it does not exists there;
even though that should never happen)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
a 'leaf' node is every file *except* directories, so we have
to reverse the logtic here
this fixes the pxar.didx browser in the web ui
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
instead of filling them with zeroes
this fixes an issue where we could not restore a container with large
sparse files in the backup (e.g. a 10GiB sparse file in a container
with a 8GiB disk)
if the last operation of the copy was a seek, we need to truncate
the file to the correct size (seek beyond filesize does not change it)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
the DiffStore only updates the fiels given in the model, so we have to
list all the fields we depend on, else we have incomplete data,
e.g. for rendering
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
if the drive state is not empty, some action blocks it, so we cannot
do any task anyway. Since we now refresh the state every 5 seconds,
we can disable/enable them 'live'.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
