Commit Graph

4187 Commits

Author SHA1 Message Date
Dietmar Maurer 237314ad0d tape: improve catalog consistency checks
Try to check if we read the correct catalog by verifying uuid, media_set_uuid
and seq_nr.

Note: this changes the catalog format again.
2021-03-18 08:43:55 +01:00
Dominik Csapak caf76ec592 tools/subscription: ignore ENOENT for apt auth config removal
deleting a nonexistant file is hardly an error worth mentioning

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-17 20:12:58 +01:00
Dietmar Maurer 6ee3035523 tape: define magic number for catalog archives 2021-03-17 13:35:23 +01:00
Dietmar Maurer b627ebbf40 tape: improve catalog parser 2021-03-17 11:29:23 +01:00
Dietmar Maurer ef4bdf6b8b tape: proxmox-tape media content - add 'store' attribute 2021-03-17 11:17:54 +01:00
Dietmar Maurer 54722acada tape: store datastore name in tape archives and media catalog
So that we can store multiple datastores on a single media set.
Deduplication is now per datastore (not per media set).
2021-03-17 11:08:51 +01:00
Dietmar Maurer 0e2bf3aa1d SnapshotReader: add self.datastore_name() helper 2021-03-17 10:16:34 +01:00
Dietmar Maurer 365126efa9 tape: PoolWriter - remove unnecessary move_to_eom 2021-03-17 10:16:34 +01:00
Dominik Csapak 7914e62b10 tools/zip: only add zip64 field when necessary
if neither offset nor size exceeds 32bit, do not add the
zip64 extension field

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-16 09:13:39 +01:00
Dominik Csapak e36135031d ui: tape/Restore: let the user choose an owner
so that the tape backup can be restored as any user, given
the current logged in user has the correct permission.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-16 08:55:42 +01:00
Dominik Csapak 5a5ee0326e proxmox-tape: add missing notify-user to 'proxmox-tape restore'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-16 08:54:38 +01:00
Dietmar Maurer 776dabfb2e tape: use MB/s for backup speed (to match drive speed specification) 2021-03-16 08:51:49 +01:00
Dietmar Maurer 5c4755ad08 tape: speedup backup by doing read/write in parallel 2021-03-16 08:51:49 +01:00
Dominik Csapak 7c1666289d tools/zip: add missing start_disk field for zip64 extension
it is not optional, even though we give the size explicitely

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-15 12:36:40 +01:00
Thomas Lamprecht cded320e92 backup info: run rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-14 19:18:35 +01:00
Wolfgang Bumiller b31cdec225 update to pxar 0.10
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-03-12 10:48:09 +01:00
Wolfgang Bumiller 591b120d35 fix feature flag logic in pxar create
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-03-12 10:17:51 +01:00
Dietmar Maurer e8913fea12 tape: write_chunk_archive - do not consume partially written chunk at EOT
So that it is re-written to the next tape.
2021-03-12 07:14:50 +01:00
Dominik Csapak ef0ea4ba05 server/worker_task: improve endtime for unknown tasks
instead of always using the starttime, use the last timestamp from the log
this way, one can see when the task was aborted without having to read
the log

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-11 09:56:12 +01:00
Dominik Csapak 2892624783 tape/send_load_media_email: move to server/email_notifications
and reuse 'send_job_status_mail' there so that we get consistent
formatted mails from pbs (e.g. html part and author)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-11 09:56:12 +01:00
Dietmar Maurer 2c10410b0d tape: improve backup task log 2021-03-11 08:43:13 +01:00
Thomas Lamprecht d1d74c4367 typo fixes all over the place
found and semi-manually replaced by using:
 codespell -L mut -L crate -i 3 -w

Mostly in comments, but also email notification and two occurrences
of misspelled  'reserved' struct member, which where not used and
cargo build did not complain about the change, soo ...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-10 16:39:57 +01:00
Dietmar Maurer 5ef4c7bcd3 tape: fix scsi volume_statistics and cartridge_memory for quantum drives 2021-03-10 14:13:48 +01:00
Dietmar Maurer 6396bace3d tape: improve backup task log (show percentage) 2021-03-10 10:59:13 +01:00
Dietmar Maurer 713a128adf tape: improve backup task log format 2021-03-10 09:54:51 +01:00
Dietmar Maurer affc224aca tape: read_tape_mam - pass correct allocation len 2021-03-10 09:24:38 +01:00
Dietmar Maurer 6f82d32977 tape: cleanup - remove wrong inline comment 2021-03-10 08:11:51 +01:00
Dominik Csapak 2a06e08618 api2/tape/backup: continue on vanishing snapshots
when we do a prune during a tape backup, do not cancel the tape backup,
but continue with a warning

the task still fails and prompts the user to check the log

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-09 10:20:54 +01:00
Dietmar Maurer 1057b1f5a5 tape: lock artificial "__UNASSIGNED__" pool to avoid races 2021-03-09 10:00:26 +01:00
Dietmar Maurer af76234112 tape: improve MediaPool allocation by sorting tapes by ctime and label_text 2021-03-09 08:33:21 +01:00
Dominik Csapak 1d7fcbece8 tape/linux_tape: remove debug prints
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-08 14:29:19 +01:00
Dominik Csapak 8703a68a31 tape/backup: add success/error notifications
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-08 14:29:06 +01:00
Dietmar Maurer 9ce2481a69 tape: clean - fix if cleaning tape is already loaded 2021-03-08 09:08:55 +01:00
Dietmar Maurer d95c74c6e7 tape: test tape alert flags after drive clean 2021-03-08 09:08:55 +01:00
Dietmar Maurer 5ca5f8daf3 pmt: add tape-alert-flags command 2021-03-08 07:37:05 +01:00
Dominik Csapak c9793d47f9 api2/tape: add notify_user to backup(-jobs) and restore api calls
so that a user can be given that will be notified for
manual intervention (e.g. inserting a tape)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-05 17:20:37 +01:00
Dominik Csapak 9152a0077f config/tape_job: add notify-user config to tape jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-03-05 12:51:25 +01:00
Dietmar Maurer 0b90c67fb4 tape: avoid Updatable for now 2021-03-05 12:51:25 +01:00
Dietmar Maurer b4975d3102 tape: finish api permission checks 2021-03-05 11:49:07 +01:00
Dietmar Maurer ee33795b72 acl: use /tape/device instead of /tape/drive and /tape/changer 2021-03-05 10:06:19 +01:00
Dietmar Maurer 90e16be3ae tape: improve PoolWriter logging
Log reason why we allocate a new media set.
2021-03-05 09:59:38 +01:00
Dietmar Maurer cf90a369e2 cleanup: rename token_user into auth_id_filter 2021-03-05 08:36:18 +01:00
Dietmar Maurer cd5d6103ea tape: fix media_expire_time 2021-03-04 16:58:24 +01:00
Dietmar Maurer 1bed3aedc8 tape: fix wrong media set expire time 2021-03-04 12:38:28 +01:00
Dietmar Maurer ab77d660cc tape: improve media status in list_media 2021-03-04 10:17:10 +01:00
Dominik Csapak 78593b5b5c tape/drive: improve error and email handling for requesting a tape load
Try once first to load the correct tape before sending an email to
insert the correct one. This way, the admin does not get a mail
if the correct tape is already inserted.

Also include the error we got that prompted the email to insert the
tape. This means that if the admin gets prompted to insert e.g.
"FOO" but inserts "BAR", he'll get an email that the wrong
tape is inserted.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-03 16:09:39 +01:00
Dominik Csapak 831c43c91b config/tfa: webauthn: disallow registering a token twice
by adding the existing credential id to the 'excludeCredentials' list

this prevents the browser from registering a token twice, which
lets authentication fail on some browser/token combinations
(e.g. onlykey/solokey+chromium)
while is seems this is currently a bug in chromium, in a future spec
update the underlying behaviour should be better defined, making this
an authenticator bug

also explicitly catch registering errors and show appropriate error messages

0: https://bugs.chromium.org/p/chromium/issues/detail?id=1087642

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-03 14:05:05 +01:00
Dominik Csapak 7f37cacfac config/tfa: set UserVerificationPolicy to Discouraged
the current default is 'Preferred', which is not really useful, as the
(web) client can simply change this to discouraged, since the
webauthn_rs crate does not verify the 'user_verified' bit of the
response in that case

setting this to 'Required' is not really useful either at the moment,
since a user can have a mix of different authenticators that may or
may not support user verification

there is ongoing discussion in the crate how to handle that[0]

we could probably expose this setting(discouraged/required) to the user/admin
and save it to the credential and allow only registering credentials
of the same type or filter them out on login (i.e. if there is an
authenticator that can handle userVerification, require it)

in any case, the current default is not helpful for security, but
makes loggin in harder, since the key will by default want to verify
the user

0: https://github.com/kanidm/webauthn-rs/pull/49

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-03 14:05:05 +01:00
Thomas Lamprecht 3bb7e62e88 zpool parsers: small code cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-03 14:05:05 +01:00
Thomas Lamprecht 3b060167f6 zpool parsers: fixup test value
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-03 14:05:05 +01:00
Dietmar Maurer 8a76e71129 tape: impl access permission for media content list 2021-03-03 13:16:15 +01:00
Dietmar Maurer 396fd747a6 tape: impl access permissions for tape jobs 2021-03-03 12:48:06 +01:00
Dietmar Maurer 16bd08b297 tape: add acl path /tape/job/{id} 2021-03-03 12:48:06 +01:00
Dietmar Maurer ccdf327ac8 tape: impl access permissions for encryption key config 2021-03-03 12:21:55 +01:00
Dietmar Maurer 8cd63df0dc tape: impl api permissions for drive/changer/pool configuration 2021-03-03 12:10:00 +01:00
Dietmar Maurer b90cb34fd6 complete_acl_path: add more paths 2021-03-03 11:09:54 +01:00
Dietmar Maurer d6c1e12c06 add roles to implement tape backup permissions 2021-03-03 11:09:02 +01:00
Dietmar Maurer 40bf636b47 ui: enable experimental tape UI if tape.cfg exists 2021-03-03 09:02:02 +01:00
Dominik Csapak 347cde827b dashboard: fix datastore full estimate for not changing usage
If the usage of a datastore did not change, we did not
return an estimate. The ui interpreted this as 'not enough data', but
it should actually be 'never'.

Fixing this by always setting the estimate first to 0 and overwriting
if we successfully calculated one, and checking for 'undefined' in the ui.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-02 17:51:13 +01:00
Dominik Csapak 8bf5769382 api2/tape/drive: add state to DriveListEntry
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-02 12:51:19 +01:00
Dietmar Maurer db04d10d14 tape: add comment to media pool config 2021-03-02 11:42:58 +01:00
Dietmar Maurer 1d14c31658 Revert "tape: also abort backup/restore on server shutdown"
This reverts commit 9bd81bb384.

Turns out this is not really a good idea.
2021-03-02 08:00:10 +01:00
Dietmar Maurer 9bd81bb384 tape: also abort backup/restore on server shutdown 2021-03-02 07:19:31 +01:00
Thomas Lamprecht d64226efee disks/zfs: fix test input and enforce check for config key
we test for the config key in the API so it makes sense to have as
test here too. Actually it would be better if we'd have a expect
Value defined here and enforce that it matches, but better than
nothing.

Fix the input for test 1, where tabs got replaced by spaces, as else
it fails

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-01 17:36:16 +01:00
Thomas Lamprecht 2440eaa2df disks/zfs: pull common test code out in helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-01 14:48:35 +01:00
Dietmar Maurer 1552d9699c tape: add 'Restore Key' button 2021-03-01 11:11:38 +01:00
Dietmar Maurer 16f9ea6708 tape: restore - do not check if pool is configured 2021-03-01 09:43:23 +01:00
Dietmar Maurer d984a9acf0 tape: add Inventory.media_list() helper 2021-03-01 09:30:45 +01:00
Dietmar Maurer 955f4aefcd tape: show media inventory/content, even if there is no pool configuration 2021-03-01 09:21:11 +01:00
Dietmar Maurer 3832911d50 docs: remove backup protocol api dumps
We now have the api-viewer for that.
2021-02-26 19:00:08 +01:00
Dietmar Maurer 08ec39be0c tape: add api to set media status 2021-02-26 10:44:07 +01:00
Dietmar Maurer 25350f3370 tape: set media status if we detect damaged medium at start of backup 2021-02-26 09:00:50 +01:00
Dietmar Maurer 0023cfa385 tape: fix 'proxmox-tape backup' cli command 2021-02-26 08:51:24 +01:00
Dietmar Maurer 5830e5620d tape: cleanup TapeJob implementation - uses AllOf Schema with SectionConfig 2021-02-25 09:44:31 +01:00
Wolfgang Bumiller 3554fe6480 update to schema changes in proxmox
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-24 14:50:08 +01:00
Dietmar Maurer 0dadf66dc7 tape: implement backup using latest-only flag 2021-02-24 13:51:53 +01:00
Thomas Lamprecht a941bbd0c9 client: raise HTTP_TIMEOUT to 120s
As 20s is really not that high, especially for loaded setups one is
connected to through a spotty network (looking at you ÖBB railnet)
and gets latency spikes of 5 - 10s for some minutes at a time..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-02-24 11:57:21 +01:00
Dietmar Maurer 21e3ed3449 tape: add 'latest-only' property to backup job config 2021-02-24 11:19:12 +01:00
Dominik Csapak 52d8db7925 api2/config/tape_backup_job: fix duplicate id parameter
since the PUT api call is using the 'Updater', the 'id' parameter is
already encoded in there, tripping up the api verify tests with
'Duplicate keys found in AllOf schema: id'

"fixing" it by removing the explicit id from the api call and
taking it from the Updater (and failing if it does not exists there;
even though that should never happen)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-23 17:23:47 +01:00
Dominik Csapak 875d375d7a api2/types: ArchiveEntry: fix inverse 'leaf' value
a 'leaf' node is every file *except* directories, so we have
to reverse the logtic here

this fixes the pxar.didx browser in the web ui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-23 17:23:47 +01:00
Dominik Csapak cba167b874 pxar/extract: if possible create files sparesly
instead of filling them with zeroes

this fixes an issue where we could not restore a container with large
sparse files in the backup (e.g. a 10GiB sparse file in a container
with a 8GiB disk)

if the last operation of the copy was a seek, we need to truncate
the file to the correct size (seek beyond filesize does not change it)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-23 14:44:59 +01:00
Dominik Csapak 4961404c7c api2/config/tape_backup_job: enable update api call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-23 13:17:49 +01:00
Dietmar Maurer 3fbf2311e4 tape: improve backup task logging 2021-02-23 12:58:44 +01:00
Dietmar Maurer 043018cfbe doc: fix wrong api method description 2021-02-22 12:10:34 +01:00
Dietmar Maurer 2037d9af03 api-viewer: show permissions 2021-02-22 12:01:24 +01:00
Dietmar Maurer 7f07991035 docs: improve MEDIA_SET_NAMING_TEMPLATE_SCHEMA description 2021-02-22 09:50:16 +01:00
Dietmar Maurer 18ce01caff tape: use correct schema for changer path (SCSI_CHANGER_PATH_SCHEMA) 2021-02-22 09:43:13 +01:00
Dietmar Maurer 5bc8e80a99 cleanup: cleanup use clause (avoid *) 2021-02-22 09:33:29 +01:00
Dietmar Maurer 6252df4c18 docs: fix EXPORT_SLOT_LIST_SCHEMA description 2021-02-22 09:05:07 +01:00
Dietmar Maurer 451856d21d api-viewer: fix path for upgraded HTTP/2 proptocls 2021-02-22 08:38:27 +01:00
Dietmar Maurer aa30663ca5 docgen: improve regex format 2021-02-21 16:14:11 +01:00
Dietmar Maurer 8616a4afe5 docgen: dump string format (property strings) 2021-02-21 15:54:14 +01:00
Dietmar Maurer bc2358319b docgen: improve api schema dump 2021-02-21 12:00:06 +01:00
Dietmar Maurer 0bf4b81370 docs: api-viewer - add backup and restore APIs 2021-02-21 10:38:33 +01:00
Dietmar Maurer fee0fe5422 docgen: implement api schema generation (for api-viewer)
Just a start - not complete jet.
2021-02-21 09:06:58 +01:00
Dietmar Maurer 86d9f4e733 tape: extend MediaChange trait to return MtxStatus 2021-02-20 10:23:16 +01:00
Dietmar Maurer 3f16f1b006 tape: update changer status inside ScsiMediaChange implementation 2021-02-20 09:56:27 +01:00
Dietmar Maurer cbd9899389 tape: update changer status cache after load/unload 2021-02-20 09:06:17 +01:00
Dietmar Maurer cd44fb8d84 tape: create tmp dirs early at server startup 2021-02-19 18:10:02 +01:00
Dietmar Maurer 4188fd59a0 tape: cache changer state 2021-02-19 16:48:19 +01:00
Dominik Csapak 926d05ef0b api2/tape/{backup, restore}, proxmox-tape: set device state
set the drive device state everywhere we lock it, so that we
know what it currently does

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:16:30 +01:00
Dominik Csapak 8be48ddfc7 api2/tape/changer: add drive state to changer status output
if we can find the drive in the config and it has a state

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:15:15 +01:00
Dominik Csapak 41e66bfaf6 api2/tape/drive: wrap some api calls in run_drive_blocking_task
those calls could also block, so we have to run them in a blocking
tokio task, as to not block the current thread

nice side effect is that we now also update the state for that
drive in those instances

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:14:08 +01:00
Dominik Csapak 47a7241410 api2/tape/drive: use run_drive_blocking_task where possible
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:13:36 +01:00
Dominik Csapak 54c77b3d62 api2/tape/drive: add wrapper for tokio::task::spawn_blocking
similar to the worker wrapper, lock, write status, run code, unset status

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:13:21 +01:00
Dominik Csapak a1c5575308 api2/tape/drive: use 'run_drive_worker' where possible
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:11:04 +01:00
Dominik Csapak a44c934b5d api2/tape/drive: add run_drive_worker wrapper
a wrapper for locking, starting the worker and writing the state

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:08:27 +01:00
Dominik Csapak 546d2653ee tape/drive: add get/set status functions
simply writes into/reads from a file in /run, we will use this
for writing the upid (or potential other states) per drive

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:05:09 +01:00
Dominik Csapak 33c06b3388 tape/drive: add test_device_path_lock
this makes it possible to detect if the drive was locked in a
non-blocking manner

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:03:30 +01:00
Dietmar Maurer 1917ea3ce1 cleanup: use serde(flatten) for VerificationJobStatus, improve code reuse 2021-02-19 09:50:25 +01:00
Dietmar Maurer 70842b9ef2 cleanup: use serde(flatten) for SyncJobStatus, improve code reuse 2021-02-19 09:36:39 +01:00
Dietmar Maurer 7690a8e7bd api: list tape backup jobs with scheduling status 2021-02-19 09:02:13 +01:00
Dietmar Maurer 1860208560 add helper to compute job scheduling state 2021-02-19 08:58:50 +01:00
Dietmar Maurer 1689296d46 api: add new type JobScheduleStatus 2021-02-19 08:23:41 +01:00
Dietmar Maurer 9bbd83b1f2 tape: media_list API - allow to update online status for a single changer 2021-02-18 10:59:33 +01:00
Dietmar Maurer d0647e5a02 tape: use worker tasks for media load/unload 2021-02-18 09:04:51 +01:00
Stefan Reiter f1a83e9759 client: extract common functions to proxmox_client_tools module
...including common schemata, connect(), extract_*() and completion
functions.

For later use with proxmox-file-restore binary.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 10:13:10 +01:00
Stefan Reiter 6afb60abf5 asyncify pxar create_archive
...to take advantage of the aio::Encoder from the pxar create.

Rather straightforward conversion, but does require getting rid of
references in the Archiver struct, and thus has to be given the Mutex
for the catalog directly. The callback is boxed.

archive_dir_contents can call itself recursively, and thus needs to
return a boxed future.

Users are adjusted, namely PxarBackupStream is converted to use an
Abortable future instead of a thread so it supports async in its handler
function, and the pxar bin create_archive is converted to an async API
function. One test case is made to just use 'block_on'.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-17 09:24:20 +01:00
Dominik Csapak a42212fc1e pxar/extract: add extract_sub_dir
to extract some subdirectory of a pxar into a given target
this will be used in the client

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:32:41 +01:00
Dominik Csapak 2e21948156 api2/admin/datastore: refactor create_zip into pxar/extract
we will reuse that code in the client, so we need to move it to
where we can access it from the client

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>

[clippy fixes]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:32:41 +01:00
Dominik Csapak 5279ee745f api2/admin/datastore: accept "/" as path for root
makes more sense than sending "root'"

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:32:41 +01:00
Dominik Csapak 227501c063 api2/admin/datastore: refactor list_dir_content in catalog_reader
we will reuse that later in the client, so we need it somewhere
we can use from there

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>

[add strongly typed ArchiveEntry and put api code into helpers.rs]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:32:41 +01:00
Stefan Reiter 89d25b1931 client: add VsockClient to connect to virtio-vsock VMs
Currently useful only for single file restore, but kept generic enough
to use any compatible API endpoint over a virtio-vsock[0,1] interface.

VsockClient is adapted and slimmed down from HttpClient.

A tower-compatible VsockConnector is implemented, using a wrapped
UnixStream as transfer. The UnixStream has to be wrapped in a custom
struct to implement 'Connection', Async{Read,Write} are simply forwarded
directly to the underlying stream.

[0] https://www.man7.org/linux/man-pages/man7/vsock.7.html
[1] https://wiki.qemu.org/Features/VirtioVsock

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:15:43 +01:00
Stefan Reiter b57c0dbe30 rest: implement tower service for UnixStream
This allows anything that can be represented as a UnixStream to be used
as transport for an API server (e.g. virtio sockets).

A tower service expects an IP address as it's peer, which we can't
reliably provide for unix socket based transports, so just fake one.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 07:50:35 +01:00
Dietmar Maurer 8b910bb6bc tape: add regression test for compute_media_state 2021-02-17 06:57:29 +01:00
Dietmar Maurer 3cdd1a3424 tape: erase_media - automatically load media (with tape libraries). 2021-02-17 06:55:53 +01:00
Dominik Csapak be61c56c21 api2/tape/drive: add optional label-text to erase
if given, erases the tape only iff the inserted tape contains that label
used to safeguard tape erasing from ui for standalone drives

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-17 06:55:53 +01:00
Dietmar Maurer dbaef7d04d tape: add regression test for alloc_writable_media 2021-02-17 06:55:49 +01:00
Dietmar Maurer 159100b944 tape: ui - avoid changer status query in backup content panel 2021-02-15 12:17:36 +01:00
Dietmar Maurer 41a8db3576 tape: implement API to run tape backup jobs 2021-02-15 11:09:30 +01:00
Dietmar Maurer fe291ab794 cleanup: tape_backup_job.rs - do not use #[allow(non_camel_case_types)] 2021-02-15 10:41:04 +01:00
Dietmar Maurer adb65b9889 tape: fix proxmox-tape pool config command 2021-02-15 08:58:05 +01:00
Dietmar Maurer 8513626b9f proxmox-backup-proxy: schedule tape backup jobs 2021-02-15 08:30:42 +01:00
Dietmar Maurer 7ca0ba4515 tape: add tape-job.cfg manual page 2021-02-15 07:05:39 +01:00
Dietmar Maurer 42200c405a fix api sort order 2021-02-15 06:35:45 +01:00
Dietmar Maurer be327dbccd tape: add tape backup job configuration 2021-02-14 10:35:51 +01:00
Dietmar Maurer 7b1bf4c098 tape: erase_media - automatically erase catalog and inventory 2021-02-12 11:30:34 +01:00
Dominik Csapak 7c9fb570cc api2/types/tape/media_location: fix deserialization of MediaLocation::Vault
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-11 14:23:54 +01:00
Dominik Csapak 56d22c66c0 api2/tape/media: add 'move' api call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-11 14:23:27 +01:00
Dominik Csapak 85cdc4f371 proxmox-tape: change lookup_drive_name to extract_drive_name
in most uses, we want to remove the drive from the param afterwards
where we don't, we already overwrite it with the result of this function

this fixes some commands (like 'proxmox-tape read-label --drive foo')
that failed with:

parameter 'drive': duplicate parameter.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-11 14:22:50 +01:00
Dietmar Maurer 4a874665eb proxmox-backup-manger verify-job CLI
Add missing command line interface to manage verification jobs.
2021-02-11 14:06:44 +01:00
Dietmar Maurer 6f6b69946e docs: use type_text for config::acl::Role
The list of roles is simply too long, so we use type_text to generate
reasonable docs.
2021-02-11 13:47:01 +01:00
Dietmar Maurer 5b7f44555e docs: add manual page for verification.cfg 2021-02-11 13:40:11 +01:00
Dietmar Maurer 2ca396c015 docs: add manual page for acl.cfg 2021-02-11 13:13:34 +01:00
Dietmar Maurer 8f02db04f9 docs: use new type_text option for calendar events 2021-02-11 11:51:22 +01:00
Dietmar Maurer 9f35e44681 fix proxmox_backup_manager remote list: s/userid/auth-id/ 2021-02-11 11:31:20 +01:00
Dietmar Maurer 6279b8f5a5 docs: add manukal page for remote.cfg 2021-02-11 11:25:15 +01:00
Dietmar Maurer 3084232cb5 depend on proxmox 0.10.5 2021-02-11 10:49:16 +01:00
Dietmar Maurer 2322a980d0 docs: add utility binary to generate docs 2021-02-10 11:05:02 +01:00
Wolfgang Bumiller 7f9d8438ab tfa: derive WebauthnConfigUpdater via api macro
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-10 10:47:47 +01:00
Wolfgang Bumiller befd95a90a pxar: improve error messages
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-09 13:09:00 +01:00
Wolfgang Bumiller ab6cd4229b pxar: skip write_attr_fd if they are equal
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-09 13:09:00 +01:00
Wolfgang Bumiller 9213744ecb fix #3026: pxar: metadata: apply flags _after_ updating mtime
otherwise we'll fail to update the mtime of immutable files

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-09 13:08:49 +01:00
Dominik Csapak afcf8b3ed6 api2/tape/drive: add missing locks for some drive actions
else a backup/restore can easily be disrupted

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-09 08:46:03 +01:00
Fabian Grünbichler 15d2c7786e client: move test output files to target/testout
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-08 09:52:16 +01:00
Dietmar Maurer fbf8779388 doc: improve pmt/pmtx manual pages 2021-02-06 16:50:56 +01:00
Dietmar Maurer ced7838de4 tape: fix path for test files 2021-02-06 09:32:50 +01:00
Fabian Grünbichler 2f26b8668a client: track key source, print when used
to avoid confusing messages about using encryption keys when restoring
plaintext backups, or about loading master keys when they are not
actually used for the current operation.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 09:07:31 +01:00
Fabian Grünbichler 9432838914 client: refactor crypto_parameter handling
pull out the crypt-mode to logically group arms and make the whole mess
a bit more "human-parsable".

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 09:06:28 +01:00
Fabian Grünbichler 1a89a7794e client: extend tests for master key handling
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 09:05:01 +01:00
Fabian Grünbichler c0a87c12fb client: allow passing specific master key
it's needed for PVE's LXC integration, and might be interesting for
other more special usage scenarios as well.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 09:04:42 +01:00
Fabian Grünbichler c6a7ea0a2f client: refactor keyfile_parameters
no semantic changes intended

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 09:03:03 +01:00
Fabian Grünbichler 5bb057e5a2 client: add test for keyfile_parameters
this will get more complex soon, so add test to document current
behaviour.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 08:58:15 +01:00
Fabian Grünbichler 2924b37d6d key: rustfmt module
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 08:56:19 +01:00
Fabian Grünbichler 42c0f784e2 key: add show-master-pubkey command
and print public key when generating/importing..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 08:55:21 +01:00
Fabian Grünbichler 05f17d1ec4 key: make 'default' master key explicit
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-06 08:52:11 +01:00
Dominik Csapak f197c286d5 tape/inventory: fix missing parameter in error message
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-05 16:12:18 +01:00
Dietmar Maurer 4c209d6b10 install pmt binary 2021-02-05 12:42:20 +01:00
Dietmar Maurer 25aa55b5f5 tape: correctly lock the tape device 2021-02-05 10:50:21 +01:00
Dietmar Maurer cdf39e62b3 tape: MediaPool - replace use_offline_media with changer_name
This way, we can improve location_is_available, because we only
consider media from that changer as available.
2021-02-04 10:15:18 +01:00
Dietmar Maurer b81e37f6ab tape: improve code reuse 2021-02-04 09:39:16 +01:00
Dietmar Maurer ddebbb52fd tape: fix tests for BlockedReader 2021-02-04 08:54:54 +01:00
Dietmar Maurer 983e929e25 tape: add multi volume reader/writer implementations
We currently do not use it. Added anaways, to show the possibility.
2021-02-04 08:36:35 +01:00
Dietmar Maurer f47e035721 tape: cleanup - move tape file readers/writers into src/tape/file_formats folder 2021-02-04 07:59:37 +01:00
Dietmar Maurer a80d72f999 tape: allow to abort restore tasks 2021-02-04 07:05:43 +01:00
Dietmar Maurer 8de9a9917f cleanup: use task_log macro 2021-02-04 06:55:18 +01:00
Wolfgang Bumiller fa016c1697 HttpsConnector: use hostname instead of URL again
fixes connecting to hosts with valid certificates without a
pinned fingerprint
this was accidentally changed in the tokio-1.0 updates
apparently

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: 0f860f712f ("tokio 1.0: update to new tokio-openssl interface")
2021-02-03 15:18:18 +01:00
Dietmar Maurer 7d2c156eb1 tape: BlockedReader - always consume EOF 2021-02-03 13:25:59 +01:00
Dietmar Maurer b22e8c3632 tape: add media pool regression tests 2021-02-03 10:23:04 +01:00
Dietmar Maurer 7929292618 tape: add regresion test for media state 2021-02-03 09:34:31 +01:00
Dietmar Maurer 0d4e4cae7f tape: improve pmt command line completion 2021-02-03 08:54:12 +01:00
Dominik Csapak 50a4797fb1 api2/types/tape/media: add media_set_ctime to MediaContentEntry
to be able to better sort in the ui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-02 14:45:54 +01:00
Dietmar Maurer cc2a0b12f8 test: define tape tests as submodule 2021-02-02 14:38:15 +01:00
Dietmar Maurer 988e8de122 tape: set correct ownership on lock file 2021-02-02 14:18:57 +01:00
Dietmar Maurer 2f8809c6bc test: src/tape/inventory.rs - avoid chown when running tests 2021-02-02 13:43:16 +01:00
Dietmar Maurer 8937c65951 tape: add pmt stoptions/stsethoptions/stclearoptions 2021-02-02 08:58:02 +01:00
Dietmar Maurer 6c6ad82d90 tape: add pmt setblk 2021-02-02 07:19:54 +01:00
Dietmar Maurer d0f11b66f7 thape: add read_tapedev_options, display driver options with status command 2021-02-02 06:40:40 +01:00
Dietmar Maurer ca953d831f cleanup: remove MT_ST_ prefix from SetDrvBufferOptions 2021-02-01 17:54:53 +01:00
Fabian Grünbichler 01c023d50f paperkey: rustfmt
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-01 17:05:40 +01:00
Fabian Grünbichler c2113a405e paperkey: simplify block generation
the chunk-iterator already does exactly what we want here..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-01 17:05:32 +01:00
Fabian Grünbichler 5dae81d199 paperkey: allow RSA keys without passphrase
some users might want to store the plain version of their master key for
long-term storage and rely on physical security instead of a passphrase
to protect the paper key.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-01 17:05:22 +01:00
Wolfgang Bumiller 6248e51797 change half-ticket time range from -120..240 to -60..600
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-01 15:13:11 +01:00
Dietmar Maurer 19e4a36c70 tape: do not use drive.open() within pmt
Do not fail if no media is loaded. Inportant for load command.
2021-02-01 12:39:50 +01:00
Dietmar Maurer 90769e5694 tape: add pmt lock/unlock 2021-02-01 12:18:55 +01:00
Dietmar Maurer b8cbe5d65b tape: fix tape alert flag decoding 2021-02-01 12:18:55 +01:00
Dietmar Maurer d690d14568 tape: add pmt bsr/fsr 2021-02-01 10:39:04 +01:00
Dietmar Maurer 85ef624440 tape: add pmt asf 2021-02-01 10:32:21 +01:00
Dietmar Maurer e995996290 tape: pmt - fix count parameter schema 2021-02-01 10:21:25 +01:00
Dietmar Maurer 8e6ad4301d tape: add pmt fsfm/bsfm, pass count as arg_param 2021-02-01 10:18:18 +01:00
Dominik Csapak 1399c592d1 garbage_collection: only ignore 'missing chunk' errors
with the fix for #2909 (improving handling missing chunks), we
changed from bailing to warning during a garbage collection when
updating the atime of a chunk.

but, updating the atime can not only fail when the chunk is missing,
but also on other occasions, e.g. no permissions or more importantly,
no space left on the device. in that case, the atime of a valid and used
chunk cannot be updated, and the second sweep of the gc will remove that chunk.
[0] is a real world example of that happening.

instead, only warn on really missin chunks, and bail on all other
errors.

0: https://forum.proxmox.com/threads/pbs-server-full-two-days-later-almost-empty.83274/

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-01 09:18:59 +01:00
Dietmar Maurer 9883b54cba tape: remove drive from pool config 2021-02-01 09:14:28 +01:00
Dietmar Maurer 83b8949a98 tape: add pmt weof 2021-01-31 17:33:07 +01:00
Dietmar Maurer 28f60e5291 cleanup: avoid compiler warnings 2021-01-31 17:02:55 +01:00
Dietmar Maurer 1f31d06f48 tape: add pmt bsf 2021-01-31 17:00:15 +01:00
Dietmar Maurer 2f2e83c890 tape: add pmt fsf 2021-01-31 16:54:16 +01:00
Dietmar Maurer b22c618734 tape: add pmt erase 2021-01-31 16:34:10 +01:00
Dietmar Maurer 1e041082bb tape: add pmt command line tool
Experimental, not installed by now.
2021-01-31 16:19:53 +01:00
Dietmar Maurer b5b99a52cd tape: API type cleanup, use serde flatten to derive types 2021-01-30 09:36:54 +01:00
Dietmar Maurer 9586ce2f46 tape: move scan_drives API code to correct file 2021-01-30 08:03:17 +01:00
Dietmar Maurer c297835b01 tape: proxmox-tape - use API instead of direct functions calls 2021-01-29 11:49:11 +01:00
Dietmar Maurer e68269fcaf tape: proxmox-tape inventory: call API 2021-01-29 11:21:57 +01:00
Dietmar Maurer 5243df4712 tape: proxmox-tape - use API instead of direct functions calls 2021-01-29 10:50:11 +01:00
Dietmar Maurer 4470eba551 cleanup: factor out common client code to view task log/result 2021-01-29 10:10:04 +01:00
Dietmar Maurer 1f2c4713ef tape: improve backup task abort behaviour 2021-01-29 09:23:39 +01:00
Dietmar Maurer a6c16894ff worker_task: log something when we receive an abort request 2021-01-29 09:22:37 +01:00
Dietmar Maurer 271764deb9 tape: make it possible to abort tape backup tasks (check_abort)
Also use task_log makro instead of worker.log.
2021-01-29 09:07:55 +01:00
Dietmar Maurer 52f7a73009 display_task_log: make it possible to abort tasks with CTRL-C 2021-01-29 09:06:15 +01:00
Dominik Csapak bdb6e6b83f api2/reader: asyncify the reader worker task
this way, the code is much more readable

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-29 06:59:25 +01:00
Dietmar Maurer 41dacd5d3d tape: use worker task for eject-media api 2021-01-28 16:49:08 +01:00
Dietmar Maurer eb1dfb02b5 tape: proxmox-tape - use api for erase-media and rewind 2021-01-28 16:36:10 +01:00
Dietmar Maurer bdb62b20a3 tape: media_pool config api - set protected flags where required 2021-01-28 15:42:32 +01:00
Dietmar Maurer f2ca03d7d0 cleanup: avoid compiler warning 2021-01-28 15:32:21 +01:00
Dominik Csapak 00ac86c31b tape/drive/linux_tape: fix and refactor usage of sg-tape-cmd
when executing this code as non-root, we use sg-tape-cmd (a setuid binary)
to execute various ioctls on the tape device

we give the command the open tape device fd as stdin, but did not
dup it, so the std::process:Stdio handle closed it on drop,
which let subsequent operation on that file fail (since it was closed)

fix it by dup'ing it before giving it to the command, and also refactor
the calling code, so that we do not forget to do this

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:24:32 +01:00
Dominik Csapak 627d000098 tape: change changer-drive-id to changer-drivenum
because it changed in the config

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:11:22 +01:00
Dominik Csapak 4be4736603 tape/changer: refactor marking of import/export slots from config
we did this for 'mtx', but missed it for the sg_pt_changer code
refactor it into the MtxStatus strut, and call it from both
code paths

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:10:55 +01:00
Dominik Csapak 2da7aca8e8 tape/changer: add vendor/model to DriveStatus
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:10:31 +01:00
Dominik Csapak b03ec281bf api2/config/{drive, changer}: prevent adding same device multiple times
this check is not perfect since there are often multiple device
nodes per drive/changer, but from the scan api we should return always
the same, so for an api user this should be enough

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:03:56 +01:00
Dominik Csapak cef4654ff4 api2/tape/drive: change methods of some api calls from put to get
makes more sense to have retrieving api calls as get instead of put

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:02:52 +01:00
Dominik Csapak f45dceeb73 api2/tape/drive: add load_media as api call
code was already there, just add it as api call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:02:13 +01:00
Dominik Csapak 18262a88c9 api2/tape/changer: add changer filter to list_drives api call
so that an api user can get the drives belonging to a changer
without having to parse the config listing themselves

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 15:01:41 +01:00
Dietmar Maurer 87f4be7998 tape: use api to run proxmox-tape backup 2021-01-28 14:56:42 +01:00
Dietmar Maurer d737adc6be tape: rename changer_drive_id to changer_drivenum 2021-01-28 11:29:59 +01:00
Dominik Csapak 5fdaecf6f4 api2/tape/drive: reorganize drive api
similar to the changers, create a listing at /tape/drive and put
the specific api calls below that

move the scan api call up one level

remove the status info from the config listing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 11:15:17 +01:00
Dominik Csapak d8792b88ef api2/types/tape/drive: add changer_drivenum
so that an api user can see which drive belongs to which drivenum of a changer
for ones with multiple drives

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-28 11:14:28 +01:00
Dietmar Maurer 651a61f559 pmtx: implement scan command 2021-01-27 12:40:51 +01:00
Dietmar Maurer 89ccb125d1 tape: use 36 byte Inquiry (recommended size) 2021-01-27 12:35:28 +01:00
Dietmar Maurer 887f1cb90c cleanup: move scan changers API implementation 2021-01-27 09:58:16 +01:00
Dietmar Maurer 16b4d78400 tape: rename retry_command to execute_scsi_command, make retry a flag 2021-01-27 09:34:24 +01:00
Dietmar Maurer ec8d9c6b80 tape: repeat changer scsi command until successful 2021-01-27 08:59:10 +01:00
Dietmar Maurer 49c2d1dcad sgutils2: use sg_get_asc_ascq_str to produce error messages 2021-01-27 06:56:11 +01:00
Dietmar Maurer d0f51651f9 sgutils2: add ASC codes from tandeberg docs 2021-01-26 18:54:08 +01:00
Dietmar Maurer 481ccf16a5 sgutils2: further improve error messages 2021-01-26 15:19:43 +01:00
Dietmar Maurer a223458753 sgutils2: support RequestSense Descriptor format 2021-01-26 13:38:16 +01:00
Dominik Csapak e1740f3f01 tape/changer/mtx: add mtx parser test
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 12:51:26 +01:00
Dominik Csapak 740dc9d1d4 api2/tape/changer: reorganize api
add a changer listing here (copied from api2/config/changer)
and put the status and transfer api calls below that

puts the changer scan into the top level tape api
and removes the (now redundant) info from the config api path

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 12:47:34 +01:00
Dominik Csapak bbf01b644c tape: fix typos
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 12:39:54 +01:00
Dietmar Maurer c81c46c336 sgutils2: improve error messages 2021-01-26 12:24:58 +01:00
Dietmar Maurer c3747b93c8 tape: add new command line tool "pmtx"
Also improve sgutil2 error reporting
2021-01-26 11:57:15 +01:00
Dominik Csapak 6864fd0149 server/worker_task: improve newline handling in upid_read_status
improves upid_read_status with:
* ignore multiple newlines at the end
* remove all code that could panic (array index access)
  the one place where we access with '[pos+1..]' is ok since
  we explicitely test the len of the vector, this is done to
  let rust optimize away the range checks, so it cannot panic

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 10:48:15 +01:00
Wolfgang Bumiller 340c0bf9e3 pxar: don't clone patterns unnecessarily
The options struct has no Drop handler and is passed by-move
so we can partially move out of it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-26 10:24:18 +01:00
Fabian Grünbichler 4d104cd4d8 clippy: more misc fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:55 +01:00
Fabian Grünbichler 367c0ff7c6 clippy: allow api functions with many arguments
some of those can be reduced/cleaned up when we have updater support in
the api macro.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:52 +01:00
Fabian Grünbichler 9c26a3d61a verify: factor out common parameters
all the verify methods pass along the following:
- task worker
- datastore
- corrupt and verified chunks

might as well pull that out into a common type, with the added bonus of
now having a single point for construction instead of copying the
default capacaties in three different modules..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:49 +01:00
Fabian Grünbichler 93e3581ce7 derive/impl and use Default for some structs
and revamp HttpClientOptions with two constructors for the common use
cases

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:45 +01:00
Fabian Grünbichler f4e52bb27d authid: make Tokenname(Ref) derive Eq
it's needed to derive Hash, and we always compare Authids or their
Userid components, never just the Tokenname part anyway..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:40 +01:00
Fabian Grünbichler 72064fd0df pxar: extract PxarExtractOptions
same as PxarCreateOptions, but for extraction/restore rather than
create.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:36 +01:00
Fabian Grünbichler 77486a608e pxar: factor out PxarCreateOptions
containing the CLI parameters that are mostly passed-through from the
client to our pxar archive creation wrapper in pxar::create

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:32 +01:00
Fabian Grünbichler e97025ab02 pxar: typedef on_error as ErrorHandler
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:26 +01:00
Fabian Grünbichler e43b9175c0 client: factor out UploadOptions
to reduce function signature complexity.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:18 +01:00
Fabian Grünbichler 9cc1415ef5 systemd/time: extract Time/DateSpec structs
could be pulled up into CalendarEvent if desired..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:13 +01:00
Fabian Grünbichler bd215dc0e4 async index reader: typedef ReadFuture
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:58 +01:00
Fabian Grünbichler 12e874cef0 allow complex Futures in tower_service impl
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:55 +01:00
Fabian Grünbichler 6d233161b0 client: refactor catalog upload spawning
by pulling out Result type into separate struct

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:51 +01:00
Fabian Grünbichler 905a570489 broadcast_future: refactor broadcast/future binding
into its own, private struct.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:48 +01:00
Fabian Grünbichler 432fe44187 report: type-alias function call tuple
to make clippy happy.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:43 +01:00
Wolfgang Bumiller 51b938496d tools::sgutils2: name fixup
it's not a box anymore

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 15:05:52 +01:00
Wolfgang Bumiller b7f9b25e4d tools::sgutils2: use NonNull
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:56:10 +01:00
Wolfgang Bumiller fe61280b6b tools::sgutils2: extern 'C' and import ordering
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:54:25 +01:00
Wolfgang Bumiller 68c087d578 tools::sgutils2: don't transmute to a Box
Otherwise we run the drop handler for the scsi pt object AND
the box itself, which shouldn't even work as it should be
doing a double-free (unless the library does some kind of
reference counting in which case this should simply crash
later on?)

anyway, let's make a wrapper simply called `SgPt` containing
the pointer from `construct_scsi_pt_obj()`

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:48:27 +01:00
Wolfgang Bumiller d6bf87cab7 tools::sgutils2: const correctness
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:33:45 +01:00
Dietmar Maurer 2b96a43879 tape: cleanup - use ScsiMediaChange trait instead of mtx_status() 2021-01-25 13:25:22 +01:00
Dietmar Maurer 697c41c584 tape: add/use rust scsi changer implementation using libsgutil2 2021-01-25 13:14:07 +01:00
Dietmar Maurer a2379996e6 sgutils2: add scsi_inquiry command 2021-01-25 13:14:07 +01:00
Fabian Grünbichler 29077d95db http-client: further clippy cleanups
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:54 +01:00
Fabian Grünbichler dbd00a57b0 http-client: fix typoed ticket cache condition
which was even copy-pasted once without noticing.

found with clippy.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:51 +01:00
Fabian Grünbichler d08cff51a4 rework GC traversal error handling
the error message don't make sense with an empty default

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:48 +01:00
Fabian Grünbichler 3e461dec1c apt: let api handle optional bool with default
one less FIXME :)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:46 +01:00
Fabian Grünbichler 4d08e25913 clippy: rewrite ifs with identical return values
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:43 +01:00
Fabian Grünbichler 43313c2ee7 clippy: rewrite comparison chains
chunk_stream one can be collapsed, since split == split_to with at set
to buffer.len() anyway.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:39 +01:00
Fabian Grünbichler 81b2a87232 clippy: fix Mutex with unused value
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:36 +01:00
Fabian Grünbichler 3d8cd0ced7 clippy: add is_empty() when len() is implemented
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:32 +01:00
Dietmar Maurer 7c78d54231 sgutils: allow command which does not transfer any data 2021-01-24 15:19:43 +01:00
Dietmar Maurer f9d71e8b17 sgutils2: allow to set custom timeouts 2021-01-24 14:54:30 +01:00
Dietmar Maurer 0107fd323c cleanup: avoid compiler warnings 2021-01-23 17:34:26 +01:00
Dietmar Maurer 979dccc7ec tape: avoid error when clearing encryption key
Simply ignore clear request when sg_spin_data_encryption_caps fails.
Assume those are tapes without hardware encryption support.
2021-01-23 10:20:43 +01:00
Dietmar Maurer bf78f70885 improve code docs in api2
Note: API methos should be declared pub, so that they show up in the generated docu.
2021-01-22 15:57:42 +01:00
Dietmar Maurer 0d916ac531 tape: add media pool config code docs 2021-01-22 12:01:46 +01:00
Dietmar Maurer d4ab407045 tape: add drive config code docs 2021-01-22 11:51:36 +01:00
Dietmar Maurer 45212a8c78 fix mtx parser bug: s/strip_suffix/strip_prefix/ 2021-01-22 11:00:56 +01:00
Dietmar Maurer 64b83c3d70 tape: implement paperkey command for tape encryption keys 2021-01-22 09:56:14 +01:00
Dietmar Maurer 639a6782bd paperkey: move code to src/tools/paperkey.rs 2021-01-22 09:42:59 +01:00
Dietmar Maurer 5f34d69bcc tape: add volume-statistics api/command 2021-01-22 08:45:35 +01:00
Dietmar Maurer 337ff5a3cc tape: add estimated medium wearout to status 2021-01-22 08:06:25 +01:00
Dietmar Maurer 8e6459a818 tape: set encryption key on restore 2021-01-22 07:26:42 +01:00
Dietmar Maurer aff3e16194 tape: add code docs to src/config/tape_encryption_keys.rs 2021-01-21 18:23:07 +01:00
Dietmar Maurer 9372c0787d renamed src/tape/sgutils2.rs -> src/tools/sgutils2.rs 2021-01-21 17:57:17 +01:00
Dietmar Maurer 83fb2da53e tape: move MediaCatalog magic number into struct (doc cleanup) 2021-01-21 17:48:07 +01:00
Dietmar Maurer 645a044bf6 tape: further hierarchy improvements 2021-01-21 17:25:32 +01:00
Dietmar Maurer 37796ff73f tape: change code hierarchy to improve docs 2021-01-21 17:12:01 +01:00
Dietmar Maurer e1fdcb1678 tape: do not export/doc low level libsgutils2 bindings 2021-01-21 16:38:24 +01:00
Dietmar Maurer edda5039d4 tape: improve code docs 2021-01-21 13:19:07 +01:00
Dietmar Maurer 1c86893d95 cleanup: always compute fingerprint in KeyConfig constructors 2021-01-21 11:56:54 +01:00
Dietmar Maurer d543587d34 Merge branch 'master' of ssh://proxdev.maurer-it.com/rust/proxmox-backup 2021-01-21 10:56:52 +01:00
Dietmar Maurer 780bc4cad2 tape: try to set encryption key with read-label command 2021-01-21 10:31:49 +01:00
Dietmar Maurer 18bd6ba13d tape: restore_key - always update key, even if there is already an entry 2021-01-21 10:31:49 +01:00
Dietmar Maurer 4dafc513cc tape: fix file permissions for tape encryptiuon keys 2021-01-21 10:31:49 +01:00
Dietmar Maurer 7acd5c5659 cleanup: remove missleading wording from code docs 2021-01-21 10:31:49 +01:00
Dietmar Maurer 8428063d9e cleanup: KeyConfig::decrypt - show password hint on error 2021-01-21 10:31:49 +01:00
Dietmar Maurer f490dda05a tape: use type Uuid instead of String 2021-01-21 10:31:49 +01:00
Dietmar Maurer 2b191385ea tape: use specialized encryption key per media-set 2021-01-21 10:31:49 +01:00
Dietmar Maurer bc228e5eaf api: add types for UUIDs 2021-01-20 17:16:46 +01:00
Fabian Grünbichler 8be65e34de clippy: replace transmute with &*
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:41:02 +01:00
Fabian Grünbichler d967d8f1a7 clippy: remove drop(&..)
it does nothing.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:41:02 +01:00
Fabian Grünbichler 50deb0d3f8 clippy: use is_null to check for null pointers
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:41:02 +01:00
Fabian Grünbichler 1d928b25fe clippy: remove some unnecessary reference taking
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler f2f81791d1 clippy: fix for_kv_map
and allow it in the one case where the entry loop is intended, but the
code is not yet implemented fully.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 382f10a0cc clippy: fix/allow needless_range_loop
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 0d2133db98 clippy: use while let loops
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 09faa9ee95 clippy: pass &str/&[..] instead of &String/&Vec
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler ccec086e25 clippy: remove unnecessary &mut
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 05725ac9a4 clippy: remove unnecessary `let` binding
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 96b7483138 clippy: remove/replace needless explicit lifetimes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 81281d04a4 clippy: fix/allow identity_op
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler e062ebbc29 clippy: us *_or_else with function calls
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler b92cad0938 clippy: convert single match to if let
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler ea368a06cd clippy: misc. fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 3f48cdb380 clippy: don't pass along unit value
make it explicit. this whole section should probably be re-written with
select!

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 17c7b46a69 clippy: use unwrap_or_default
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler a375df6f4c clippy: use copied/cloned instead of map
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler a3775bb4e8 clippy: shorten assignments
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 1e0c6194b5 clippy: fix option_as_ref_deref
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler a6bd669854 clippy: use matches!
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 6334bdc1c5 clippy: collapse nested ifs
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 3b82f3eea5 clippy: avoid useless format!
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 38556bf60d clippy: remove explicit returns
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler d8d8af9826 clippy: use chars / byte string literals
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 3984a5fd77 clippy: is_some/none/ok/err/empty
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 397356096a clippy: remove needless bool literals
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:52 +01:00
Fabian Grünbichler 365915da9a clippy: use strip_prefix instead of manual stripping
it's less error-prone (off-by-one!)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 87152fbac6 clippy: drop redundant 'static lifetime
those declarations are already const/static..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 22a9189ee0 clippy: remove unnecessary closures
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 4428818412 clippy: remove unnecessary clones
and from::<T>(T)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 47ea98e0e3 clippy: collapse/rework nested ifs
no semantic changes (intended).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Dietmar Maurer 6dd0513546 tape: allocate new media set when pool encryption key changes 2021-01-20 15:43:39 +01:00
Dietmar Maurer 8abe51b71d improve code docs 2021-01-20 15:43:19 +01:00
Dietmar Maurer 69b8bc3bfa tape: implemenmt show key
Moved API types Kdf and KeyInfo to src/api2/types/mod.rs.
2021-01-20 15:43:19 +01:00
Dietmar Maurer 301b8aa0a5 tape: implement change-passphrase for tape encryption keys 2021-01-20 15:43:19 +01:00
Dietmar Maurer e5b6c93323 tape: add --kdf parameter to create key api 2021-01-20 15:43:19 +01:00
Dietmar Maurer 9a045790ed cleanup KeyConfig 2021-01-20 15:43:19 +01:00
Dietmar Maurer 82a103c8f9 add "password hint" to KeyConfig 2021-01-20 15:43:19 +01:00
Dietmar Maurer feb1645f37 tape: generate random encryptions keys and store key_config on media 2021-01-19 11:20:07 +01:00
Dietmar Maurer 8ca37d6a65 cleanup: factor out decrypt_key_config 2021-01-19 11:20:07 +01:00
Wolfgang Bumiller 9b6bddb24c tfa: remove/empty description for recovery keys
While the user chosen description is not allowed to be
empty, we do leave it empty for recovery keys, as a "dummy
description" makes little sense...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-18 15:20:39 +01:00
Wolfgang Bumiller ad5cee1d22 tfa: add 'created' timestamp to entries
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-18 14:06:12 +01:00
Wolfgang Bumiller ca1060862e tfa: remember recovery indices
and tell the client which keys are still available rather
than just yes/no/low

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-18 13:51:23 +01:00
Dietmar Maurer 8a0046f519 tape: implement encrypted backup - simple version
This is just a proof of concept, only storing the encryption key fingerprint
inside the media-set label.
2021-01-18 13:38:22 +01:00
Dietmar Maurer 84cbdb35c4 implement FromStr for Fingerprint 2021-01-18 13:38:22 +01:00
Dietmar Maurer 1e93fbb5c1 tape: add encrypt property to media pool configuration 2021-01-18 13:38:22 +01:00
Dietmar Maurer 619554af2b tape: clear encryption key before writing labels
We always write labels unencrypted.
2021-01-18 13:38:22 +01:00
Dietmar Maurer d5a48b5ce4 tape: add hardware encryption key managenent api 2021-01-18 13:38:22 +01:00
Fabian Grünbichler 44de5bcc00 pull: add error context for initial group list call
otherwise the user is confronted with a generic error like "permission
check failed" with no indication that it refers to a request made to the
remote PBS instance..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-18 06:51:05 +01:00
Fabian Grünbichler e2956c605d pull: rustfmt
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-18 06:50:23 +01:00
Dietmar Maurer b22b6c2299 tape: encryption scsi command cleanup 2021-01-16 18:24:04 +01:00
Dietmar Maurer 90950c9c20 tape: add scsi commands to control drive hardware encryption 2021-01-16 15:59:05 +01:00
Dietmar Maurer 0c5b9e7820 tape: sgutils2.rs - add do_out_command()
Make it possible to run commands that writes data.
2021-01-16 15:59:05 +01:00
Wolfgang Bumiller a670b99db1 tfa: add webauthn configuration API entry points
Currently there's not yet a node config and the WA config is
somewhat "tightly coupled" to the user entries in that
changing it can lock them all out, so for now I opted for
fewer reorganization and just use a digest of the
canonicalized config here, and keep it all in the tfa.json
file.

Experimentally using the flatten feature on the methods with
an`Updater` struct similar to what the api macro is supposed
to be able to derive on its own in the future.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-15 15:19:52 +01:00
Wolfgang Bumiller aefd74197a bakckup::manifest: use tools::json for canonical representation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-15 15:19:52 +01:00
Wolfgang Bumiller 9ff747ef50 add tools::json for canonical json generation
moving this from backup::manifest, no functional changes

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-15 15:19:52 +01:00
Dietmar Maurer a08a198577 tape: do not abort backup if tape drive does not support tape-alert-flags 2021-01-15 11:43:17 +01:00
Oguz Bektas 6bbe49aa14 access: restrict password changes on @pam realm to superuser
for behavior consistency with `update_user`

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2021-01-15 08:49:22 +01:00
Oguz Bektas 5aa1019010 access: limit editing pam credentials to superuser
modifying @pam users credentials should be only possible for root@pam,
otherwise it can have unintended consequences.

also enforce the same limit on user creation (except self_service check,
since it makes no sense during user creation)

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2021-01-15 08:49:22 +01:00
Fabian Grünbichler 29a59b380c proxmox 0.10: adapt to moved ParameterSchema
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 0bfcea6a11 cleanup: remove unnecessary 'mut' and '.clone()'
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 89e9134a3f hyper: use new hyper::upgrade
the old Body::on_upgrade method is no more

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler b5a202acb6 tokio 1.0: update to new Signal interface
Signal does not yet re-implement Stream (and is not yet wrapped in
tokio-stream either).

see https://github.com/tokio-rs/tokio/pull/3383

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 0f860f712f tokio 1.0: update to new tokio-openssl interface
connect/accept are now happening on pinned SslStreams

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 7c66701366 tokio 1.0: use ReceiverStream from tokio-stream
to wrap a Receiver in a Stream. this will likely move back into tokio
proper once we have a std Stream..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 585e90c0de tokio: adapt to 1.0 process:Child changes
Child itself is no longer a Future, but it has a new wait() async fn
that does the same thing

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 5c852d5b82 tokio: adapt to 1.0 runtime changes
enter() now returns a guard, and the builder got revamped to make the
choice between MT and current thread explicit.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 484172b5f8 tokio 1.0: AsyncRead/Seek with ReadBuf
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler d148958b67 proxmox 0.10: use tokio::time::timeout directly
TimeoutFutureExt is no more

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 0a8d773ad0 tokio 1.0: delay -> sleep
almost the same thing, new name(s), no longer Unpin

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 427d90e6c1 update to tokio 1.0
and various related crates

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Thomas Lamprecht 1a0b410554 manager: user/token list: fix rendering 0 (never) expire date
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-01-14 13:59:08 +01:00
Dietmar Maurer 2d50a6192f tape: sg-tape-cmd - add more ways to specify devices 2021-01-14 13:05:26 +01:00
Dietmar Maurer 781da7f6f0 tape: add --inventorize flag to read-label API/CLI 2021-01-14 11:51:23 +01:00
Dietmar Maurer 25e464c5ce tape: MediaPool - allow to allocate free tapes 2021-01-13 14:25:51 +01:00
Dietmar Maurer 8446fbca85 tape: rename changer_id to label_text 2021-01-13 13:26:59 +01:00
Dietmar Maurer 0bce2118e7 tape: improve docu 2021-01-12 16:37:23 +01:00
Dietmar Maurer 6543214dde tape: MediaListEntry - add ctime 2021-01-12 12:01:21 +01:00
Dietmar Maurer a484c9cf96 tape: automatically reload tapes inside autoloader
We always automatically unload tapes to free library slots,
so it should not happen that an ejected tape resides inside the drive.

This is just a safe guard to handle the situation in case it happens ...

You can manually produce the situation by ejecting a tape without unloading:

 mt -f /dev/nst0 eject

Note: Our "proxmox-tape eject" does automatic unload
2021-01-12 09:49:05 +01:00
Dietmar Maurer 5654d8ceba tape: make eject/export more reliable, improve logging 2021-01-12 09:16:16 +01:00
Dietmar Maurer 31cf625af5 tape: improve backup logs 2021-01-11 13:23:12 +01:00
Dietmar Maurer 93be18ffd2 tape: fix tape alert flag values 2021-01-11 13:23:12 +01:00
Wolfgang Bumiller ad0ed40a59 api: return "invalid" as CSRF token for partial tickets
So that old clients don't `unwrap` a `None` value.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:13 +01:00
Wolfgang Bumiller 7ad33e8052 tfa: use UNAUTHORIZED http status in password check
to trigger our 3s delay in the rest handler

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:13 +01:00
Wolfgang Bumiller abfe0c0e70 tfa: fixup for challenge file split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:13 +01:00
Wolfgang Bumiller f22dfb5ece tfa: remove tfa user when a user is deleted
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:10 +01:00
Wolfgang Bumiller 4bda51688b tfa: improve user existence check
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller eab25e2f33 tfa: allow deletion of entries of non-existent users
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 94bd11bae2 typo fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 759af9f00c tfa api: return types and 'pub' structs/methods
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller f58e5132aa tfa: entry access/iteration cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller d831846706 tfa: r#type parameter name
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 1fc9ac0433 tfa: _entry api method name suffix consistency
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 7f066a9b21 proxy: expose qrcodejs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 027ef213aa api: tfa management and login
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller dc1fdd6267 config: add tfa configuration
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 96918252e5 buildcfg: add rundir helper macro
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 014dc5f9d7 tools: add create_run_dir helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 59e94227af add tools::serde_filter submodule
can be used to perform filtering at parse time

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Dietmar Maurer 6638c034d2 tape: remove unused eject_on_unload method 2021-01-10 16:20:18 +01:00
Dietmar Maurer 04df41cec1 tape: more MediaChange cleanups
Try to provide generic implementation for complex operations:

- unload_to_free_slot
- load_media
- export media
- clean drive
- online_media_changer_ids
2021-01-10 15:32:52 +01:00
Dietmar Maurer 483da89d03 tape: improve export media to directly export from drive, add CLI 2021-01-10 13:44:44 +01:00
Dietmar Maurer c92e3832bf tape: cleanup: s/transfer/transfer_media/, avoid compiler warnings 2021-01-10 12:18:30 +01:00
Dietmar Maurer edb90f6afa tape: backup - implement export-media-set option 2021-01-10 11:59:55 +01:00
Dietmar Maurer 0057f0e580 tape: MediaChange - add transfer, implement export 2021-01-10 11:51:09 +01:00
Dietmar Maurer e6217b8b36 tape: renamed src/tape/changer/linux_tape.rs -> src/tape/changer/mtx.rs 2021-01-10 10:07:40 +01:00
Dietmar Maurer 6fe16039b9 tape: simplify media changer implementation - new struct MtxMediaChanger 2021-01-10 10:02:01 +01:00
Dietmar Maurer 42967bf185 tape: backup - implement --eject-media option 2021-01-09 15:17:03 +01:00
Dietmar Maurer 5843268c47 tape: abort backup when we detect critical tape alert flags 2021-01-09 12:34:00 +01:00
Dietmar Maurer 7273ba3de2 tape: change default media set naming template to "%c" 2021-01-09 10:51:51 +01:00
Dietmar Maurer 0bf1c314da tape: show catalog status in media list 2021-01-09 10:24:48 +01:00
Dietmar Maurer c7926d8e8c tape: split MediaSet into extra file 2021-01-09 08:54:58 +01:00
Dietmar Maurer df69a4fc59 tape: implement drive clean 2021-01-08 11:32:56 +01:00
Dietmar Maurer 25d3965769 tape: correctly skip cleaning tapes (not regular tapes) 2021-01-08 09:16:42 +01:00
Dietmar Maurer 879569d73f tape: changer transfer - make name parameter optional 2021-01-07 17:09:47 +01:00
Dietmar Maurer b63f833d36 tape: fix paramater name - s/slot/source-slot/ 2021-01-07 15:39:25 +01:00
Dietmar Maurer 482c6e33dd tape: changer status command: make changer name optional 2021-01-07 15:12:19 +01:00
Dietmar Maurer 46a1863f88 tape: improve MediaChange trait
We expose the whole MtxStatus, and we can load/store from/to
specified slot numbers.
2021-01-07 14:26:43 +01:00
Dietmar Maurer d5035c5600 tape: mtx_status - consider new export-slots property 2021-01-06 11:53:33 +01:00
Dietmar Maurer 38ae42b11a tape: changer - add export-slot config 2021-01-06 11:06:50 +01:00
Dietmar Maurer c4b2b9ab41 tape: only query volume stats if we can read MAM 2021-01-06 09:20:36 +01:00
Dietmar Maurer ef942e04c2 tape: add function to classify tape-alert-flags 2021-01-05 17:23:30 +01:00
Dietmar Maurer b40ab10d38 tape: add volume_mounts and medium_passes to LinuxDriveAndMediaStatus 2021-01-05 13:43:17 +01:00
Dietmar Maurer f8ccbfdedd tape: implement read_volume_statistics 2021-01-05 12:58:18 +01:00
Dietmar Maurer 470f1c798a tape: status - show thape alert flags 2021-01-04 13:15:30 +01:00
Dietmar Maurer 5c012b392a tape: use LP 12h TapeAlert Response to query tape alert flags 2021-01-04 13:14:02 +01:00
Dietmar Maurer 165b641c1d tape: changer status - show full slots (for cartridge without barcode) 2021-01-04 12:06:05 +01:00
Dietmar Maurer 66e42bec05 tape: further PoolWriter cleanups 2021-01-03 12:08:40 +01:00
Dietmar Maurer c503ea7045 tape: cleanup - rename 'info' to 'media_id'
Second try.
2021-01-03 11:38:00 +01:00
Dietmar Maurer 745ec187ce Revert "tape: cleanup - rename 'info' to 'media_id'"
This reverts commit f046313c0e.

media_id is already use as parameter, so this commit is totally buggy.
2021-01-03 11:14:58 +01:00
Dietmar Maurer f046313c0e tape: cleanup - rename 'info' to 'media_id' 2021-01-03 10:37:42 +01:00
Dietmar Maurer 74595b8821 tape: sg-tape-cmd tape-alert-flags 2021-01-03 10:09:43 +01:00
Dietmar Maurer c9fdd142a4 tape: commit missing file 2021-01-02 13:39:34 +01:00
Dietmar Maurer abaa6d0ac9 tape: decode TapeAlertFlags in cartridge-memory command 2021-01-02 10:55:30 +01:00
Dietmar Maurer cfae8f0656 tape: merge MediaStateDatabase into Inventory 2021-01-01 16:15:13 +01:00
Dietmar Maurer 54f4ecd46a tape: implement MediaPool flag to consider offline media
For standalone tape drives.
2021-01-01 10:03:59 +01:00
Dietmar Maurer b9b4b31284 tape: add basic restore api/command 2020-12-31 10:26:48 +01:00
Dietmar Maurer b4772d1c43 tape: new inventory helper - lookup_media_set_pool 2020-12-31 10:03:17 +01:00
Fabian Grünbichler 08ac90f920 api: allow tokens to list users
their owner, or all if they have the appropriate privileges.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-31 08:29:49 +01:00
Fabian Grünbichler 13f5863561 api: improve error messages for restricted endpoints
the old variant attempted to parse a tokenid as userid and returned the
cryptic parsing error to the client, which is rather confusing.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-31 08:29:09 +01:00
Dietmar Maurer 81764111fe tape: media_change - log all errors 2020-12-30 19:17:18 +01:00
Dietmar Maurer cb022525ff tape: only log to stdout in CLI environment 2020-12-30 19:01:39 +01:00
Dietmar Maurer 75656a78c6 tape: improve inline docu 2020-12-30 17:28:33 +01:00
Dietmar Maurer 284eb5daff tape: cleanup/simplify media_change code 2020-12-30 17:16:57 +01:00
Dietmar Maurer ff58c51919 tape: improve media request/load 2020-12-30 13:09:28 +01:00
Dietmar Maurer 12299b333b tape: set minimal media label length to 2 2020-12-30 10:15:02 +01:00
Dietmar Maurer b017bbc441 tape: add restore code, implement catalog api/command 2020-12-30 09:48:18 +01:00
Dietmar Maurer 9e8c0d2e33 tape: cleanup - remove debug messages 2020-12-30 08:41:30 +01:00
Dietmar Maurer 250c29edd2 tape: correctly sort media api entries 2020-12-29 12:09:51 +01:00
Dietmar Maurer c431659d05 cleanup: remove debug output 2020-12-29 11:59:57 +01:00
Dietmar Maurer a33389c391 tape: implement media content list api 2020-12-29 11:58:26 +01:00
Dietmar Maurer 3460565414 tape: create the MediaCatalog when we label a tape 2020-12-29 10:55:20 +01:00
Dietmar Maurer 26b62138ee cleanup: disable debug message when we detect a stopped worker task 2020-12-29 10:53:16 +01:00
Dietmar Maurer afb0220642 tape: cleanup LinuxDriveStatus - make density optional 2020-12-29 09:10:30 +01:00
Dietmar Maurer 0993923ed5 tape: factor out get_drive_and_media_status 2020-12-29 08:39:06 +01:00
Dietmar Maurer e0362b0d0f tape: correctly parse mtx import/export slots 2020-12-28 13:32:56 +01:00
Dietmar Maurer b27c32821c tape: install new sg-tape-cmd setuid binary 2020-12-28 11:10:25 +01:00
Dietmar Maurer 76b15a035f tape: MediaCatalog: write magic number before content 2020-12-26 11:05:25 +01:00
Dietmar Maurer eb8feb1281 tape: add LTO1 to TapeDensity 2020-12-26 10:48:32 +01:00
Dietmar Maurer fc6ce9835b tape: fix non-rewinding tape device check 2020-12-25 15:38:29 +01:00
Dietmar Maurer 8ae9f4efc2 tape: minor cleanups 2020-12-25 13:45:26 +01:00
Dietmar Maurer c9d13b0fc4 tape: expose check_tape_is_linux_tape_device 2020-12-24 15:51:49 +01:00
Dietmar Maurer bfacc1d8c3 tape: cleanup - factor out open_linux_tape_device 2020-12-24 11:24:45 +01:00
Dietmar Maurer 5ae86dfaa1 tape: return media usage info with status command 2020-12-23 11:24:34 +01:00
Dietmar Maurer dbe7e556b0 tape: implement binding for libsgutils2
So that we can read cartridge memory without calling "sg_raw". In future,
we may need further low level command to control the tape..
2020-12-23 09:44:53 +01:00
Stefan Reiter 4799280ccd http_client: add timeouts for critical connects
Use timeout futures for sections that might hang in certain error
conditions. This is mostly intended to be used as a safeguard, not a
first line of defense - i.e. best-effort avoidance of total hangs.

Not every future used for the HttpClient/H2Client is changed, only those
where a quick response is to be expected. For example, the response
reading futures are left alone, so data transfer is never capped with
timeout, only the initial server connect.

It is also used for upgrading to H2 connections, as that can take a long
time on overloaded servers.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-12-22 13:31:10 +01:00
Dietmar Maurer cb80d900b3 tape: add drive status api 2020-12-22 10:42:22 +01:00
Dietmar Maurer ee01737e87 tape: rename 'mam' api to 'cartridge-memory' 2020-12-22 09:27:34 +01:00
Dietmar Maurer 2012825913 depend on proxmox 0.9.0 2020-12-22 08:52:24 +01:00
Wolfgang Bumiller b2362a1207 adaptions for proxmox 0.9 and proxmox-api-macro 0.3
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-12-22 07:31:05 +01:00
Dietmar Maurer 1e20f819d5 tape: add command to read cartridge memory (MAM)
Thsi add an additional dependency to sg3-utils (small).
2020-12-21 12:12:33 +01:00
Dietmar Maurer 9aa58f0143 cleanup: rename mtfsf into forward_space_count_files 2020-12-18 16:57:49 +01:00
Dietmar Maurer 8835664653 tape: add tape backup api 2020-12-18 15:32:12 +01:00
Dietmar Maurer d37da6b7fc tape: add PoolWriter 2020-12-18 15:27:44 +01:00
Dietmar Maurer b9ee86efe1 tape: use SnapshotReader to create snapshot archive 2020-12-18 12:11:29 +01:00
Dietmar Maurer d108b610fd tape: fix write_media_set_label - move to correct position 2020-12-18 12:11:29 +01:00
Dominik Csapak 0ec79339f7 tools/daemon: improve reload behaviour
it seems that sometimes, the child process signal gets handled
before the parent process signal. Systemd then ignores the
childs signal (finished reloading) and only after going into
reloading state because of the parent. this will never finish.

Instead, wait for the state to change to 'reloading' after sending
that signal in the parent, an only fork afterwards. This way
we ensure that systemd knows about the reloading before actually trying
to do it.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Fabian Ebner <f.ebner@proxmox.com>
2020-12-18 10:30:37 +01:00
Dietmar Maurer 2afdc7f27d tape: MediaPool::with_config() - remove name parameter
Not required, because config already contains the pool name.
2020-12-18 08:14:24 +01:00
Dietmar Maurer 26aa9aca40 tape: return current_file_number as u64 2020-12-18 07:44:50 +01:00
Dominik Csapak 3e2984bcb9 tools/process_locker: Decrement writer count in drop handler
of ProcessLockSharedGuard.

We use a counter to determine if we can unlock the file again, but
we never actually decremented the writer count, so we held the
lock forever.

This fixes the issue that we could not start a garbage collect after
a reload, as long as the old process is still running, even when that
process has no active backup anymore but another long running task
(e.g. file download, terminal, etc.).

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-18 07:15:08 +01:00
Fabian Grünbichler a7a5406c32 acl: rustfmt module
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-18 07:07:01 +01:00
Fabian Grünbichler 4f727a783e acl: reformat privileges
for better readability, and tell rustfmt to leave those definitions
alone.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-18 07:05:45 +01:00
Fabian Grünbichler 23dc68fdea acl: add docs and adapt visibility
document all public things, add some doc links and make some
previously-public things only available for test cases or within the
crate:

previously public, now private:
- AclTreeNode::extract_user_roles (we have extract_roles())
- AclTreeNode::extract_group_roles (same)
- AclTreeNode::delete_group_role (exists on AclTree)
- AclTreeNode::delete_user_role (same)
- AclTreeNode::insert_group_role (same)
- AclTreeNode::insert_user_role (same)
- AclTree::write_config (we have save_config())
- AclTree::load (we have config()/cached_config())

previously public, now crate-internal:
- AclTree::from_raw (only used by tests)
- split_acl_path (used by some test binaries)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-18 07:05:11 +01:00
Dietmar Maurer b532dd00c4 tape: add helper to read snapshot contents
- lock the snapshot for reading
- use openat to open files
- provides an iterator over all chunks
2020-12-17 13:07:52 +01:00
Fabian Grünbichler c01742855a KeyConfig: bail on wrong fingerprint
instead of just logging the error. this should never happen in practice
unless someone is messing with the keyfile, in which case, it's better
to abort.

update tests accordingly (wrong fingerprint should fail, no fingerprint
should get the expected one).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 11:27:06 +01:00
Dietmar Maurer 9c953dd260 tape: add code to write backup snapshot files (without chunks) to tape 2020-12-17 08:28:47 +01:00
Dietmar Maurer 3fbf2d2fcd tape: cleanup MediaCatalog 2020-12-17 08:05:53 +01:00
Fabian Grünbichler e0af222ec3 KeyConfig: always calculate fingerprint
and warn if stored and calculated fingerprint don't match.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:52:55 +01:00
Fabian Grünbichler 73b5011786 KeyConfig: add encrypt/decrypt test
the RSA key and the encryption key itself are hard-coded to avoid
stalling the test runs because of lack of entropy, they have no special
significance otherwise.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:47:45 +01:00
Fabian Grünbichler 7137630d43 client: add 'import-with-master-key' command
to import an encrypted encryption key using a master key.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:46:24 +01:00
Fabian Grünbichler 8acfd15d6e key: move RSA-encryption to KeyConfig
since that is what gets encrypted, and not a CryptConfig.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:43:34 +01:00
Fabian Grünbichler 48fbbfeb7e fix #3197: skip fingerprint check when restoring key
when restoring an encrypted key, the original one is obviously not
available to check the fingerprint with.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:37:54 +01:00
Fabian Grünbichler 9990af3042 master key: store blob name in constant
since we will use it in more than one place.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:36:06 +01:00
Dietmar Maurer fe6c19383b tape: remove MediaLabelInfo, use MediaId instead
The additional content_uuid was quite useless...
2020-12-16 13:31:32 +01:00
Dietmar Maurer 9839d3f778 tape: improve docu 2020-12-16 12:43:51 +01:00
Dietmar Maurer dd59e3c2a1 tape: improve docu 2020-12-16 12:23:52 +01:00
Dietmar Maurer 0b7432ae09 tape: add chunk archive reader/writer 2020-12-16 12:08:34 +01:00
Dietmar Maurer c1c2c8f635 tape: cleanup MediaLocation type for direct use with API 2020-12-16 10:49:01 +01:00
Dietmar Maurer 42298d5896 tape: add magic number to identify media catalog files 2020-12-16 09:00:14 +01:00
Dietmar Maurer 205e187613 tape: add MediaCatalog implementation 2020-12-15 13:40:49 +01:00
Dietmar Maurer a78348acbb tape: rename DriveLabel to MediaLabel 2020-12-14 17:37:16 +01:00
Dietmar Maurer 410611b4f2 tape: improve file format docu 2020-12-14 17:29:57 +01:00
Dietmar Maurer af07ec8f29 tape: minor code cleanup 2020-12-14 16:56:26 +01:00
Dietmar Maurer 3f803af00b tape: scan - print more debug info 2020-12-14 13:16:18 +01:00
Dietmar Maurer ac461bd651 tape: implement scan command (useful for debug) 2020-12-14 12:55:49 +01:00
Dietmar Maurer ce955e1635 tape: implement eod cli command (debug tool) 2020-12-14 09:56:59 +01:00
Dietmar Maurer e20d008c6a tape: rename cli 'media media-destroy' toö 'media destroy' 2020-12-14 09:30:32 +01:00
Dietmar Maurer fb657d8ee5 tape: implement destroy_media 2020-12-14 08:58:40 +01:00
Dietmar Maurer fba0b77469 tape: add media api 2020-12-14 07:55:57 +01:00
Dietmar Maurer b5c1296eaa tape: make changer get_status async 2020-12-14 07:14:24 +01:00
Dietmar Maurer 065df12872 tape: split api type definitions for changers into extra file 2020-12-13 09:31:02 +01:00
Dietmar Maurer 7e1d4712b8 tape: rename CHANGER_ID_SCHEMA to CHANGER_NAME_SCHEMA 2020-12-13 09:22:08 +01:00
Dietmar Maurer 49c965a497 tape: rename DRIVE_ID_SCHEMA to DRIVE_NAME_SCHEMA 2020-12-13 09:18:16 +01:00
Dietmar Maurer 6fe9aedd0b tape: correctly call Async handler in proxmox-tape 2020-12-12 09:58:47 +01:00
Dietmar Maurer 42cb9bd6a5 tape: avoid executor blocking in changer api 2020-12-12 09:45:08 +01:00
Dietmar Maurer 66dbe5639e tape: avoid executor blocking in drive API
By using tokio::task::spawn_blocking().
2020-12-12 09:20:04 +01:00
Dietmar Maurer 663ef85992 tape: use WorkerTask for erase and rewind 2020-12-11 11:19:33 +01:00
Dietmar Maurer e92c75815b tape: split inventory api
inventory: sync, list labels with uuids,
update_inventory: WorkerTask, updates database
2020-12-11 10:42:29 +01:00
Dietmar Maurer 6dbad5b4b5 tape: run label commands as WorkerTask (threads) 2020-12-11 09:10:22 +01:00
Dietmar Maurer bff7e3f3e4 tape: implement barcode-label-mdedia 2020-12-11 07:50:19 +01:00
Dietmar Maurer 83abc7497d tape: implement inventory command 2020-12-11 07:39:28 +01:00
Dietmar Maurer be1a8c94ae fix build: add missing file 2020-12-10 13:40:20 +01:00
Dietmar Maurer 4606f34353 tape: implement read-label command 2020-12-10 13:20:39 +01:00
Dietmar Maurer 7bb720cb4d tape: implement label command 2020-12-10 12:30:27 +01:00
Dietmar Maurer c4d8542ec1 tape: add media pool handling 2020-12-10 11:41:35 +01:00
Dietmar Maurer 9700d5374a tape: add media pool cli 2020-12-10 11:13:12 +01:00
Dietmar Maurer 05e90d6463 tape: add media pool config api 2020-12-10 10:52:27 +01:00
Dietmar Maurer 55118ca18e tape: correctly sort drive api subdir 2020-12-10 10:09:12 +01:00
Dietmar Maurer f70d8091d3 tape: implement option changer-drive-id 2020-12-10 09:09:06 +01:00
Dietmar Maurer a3c709ef21 tape: cli cleanup - avoid api redefinition 2020-12-10 08:35:11 +01:00
Dietmar Maurer 4917f1e2d4 tape: implement delete property for drive update command 2020-12-10 08:25:46 +01:00
Dietmar Maurer 93829fc680 tape: cleanup load-slot api 2020-12-10 08:04:55 +01:00
Dietmar Maurer 5605ca5619 tape: cli cleanup - rename scana-for-* into scan 2020-12-10 07:58:45 +01:00
Dietmar Maurer e49f0c03d9 tape: implement load-media command 2020-12-10 07:52:56 +01:00
Dietmar Maurer 0098b712a5 tape: implement eject 2020-12-09 17:50:48 +01:00
Dietmar Maurer 5fb694e8c0 tape: implement rewind 2020-12-09 17:43:38 +01:00
Dietmar Maurer 583a68a446 tape: implement erase media 2020-12-09 17:35:31 +01:00
Dietmar Maurer e6604cf391 tape: add command line interface proxmox-tape 2020-12-09 13:00:20 +01:00
Dietmar Maurer 43cfb3c35a tape: do not remove changer while still used 2020-12-09 12:55:54 +01:00
Dietmar Maurer 8a16c571d2 tape: add changer property to drive create api 2020-12-09 12:55:10 +01:00
Dietmar Maurer 314652a499 tape: set protected flag for configuration change api methods 2020-12-09 12:02:55 +01:00
Dietmar Maurer 6b68e5d597 client: move connect_to_localhost into client module 2020-12-09 11:59:50 +01:00
Dietmar Maurer cafd51bf42 tape: add media state database 2020-12-09 11:21:56 +01:00
Wolfgang Bumiller 9b93c62044 remove unused descriptions from api macros
these are now a hard error in the api macro

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-12-09 10:55:18 +01:00
Dietmar Maurer 5d90860688 tape: expose basic tape/changer functionality at api2/tape/ 2020-12-08 15:42:50 +01:00
Dietmar Maurer 5ba83ed099 tape: check digest on config update 2020-12-08 11:24:38 +01:00
Dietmar Maurer 50bf10ad56 tape: add changer configuration API 2020-12-08 09:04:56 +01:00
Dietmar Maurer 16d444c979 tape: add tape drive configuration API 2020-12-07 13:04:32 +01:00
Dietmar Maurer fa9c9be737 tape: add tape device driver 2020-12-07 08:29:22 +01:00
Dietmar Maurer 2e7014e31d tape: add BlockeReader/BlockedWriter streams
This is the basic format used to write data to tapes.
2020-12-06 12:09:55 +01:00
Dietmar Maurer a84050c1f0 tape: add BlockHeader impl 2020-12-06 10:26:24 +01:00
Dietmar Maurer 7c9835465e tape: add helpers to emulate tape read/write behavior 2020-12-06 09:41:16 +01:00
Dietmar Maurer ec00200411 fix bug #3189: fix change_password permission checks, run protected 2020-12-05 16:20:29 +01:00
Dietmar Maurer b107fdb99a tape: add tape changer support using 'mtx' command 2020-12-05 14:54:12 +01:00
Dietmar Maurer 7320e9ff4b tape: add media invenotry 2020-12-05 12:54:15 +01:00
Dietmar Maurer c4d2d54a6d tape: define useful constants 2020-12-05 12:20:46 +01:00
Dietmar Maurer 1142350e8d tape: add media pool config 2020-12-05 11:59:38 +01:00
Dietmar Maurer d735b31345 tape: add tape read trait 2020-12-05 10:54:38 +01:00
Dietmar Maurer e211fee562 tape: add tape write trait 2020-12-05 10:51:34 +01:00
Dietmar Maurer 8c15560b68 tape: add file format definitions 2020-12-05 10:45:08 +01:00
Dietmar Maurer 327e93711f commit missing file: tape api type definitions 2020-12-04 16:00:52 +01:00
Dietmar Maurer a076571470 tape support: add drive configuration 2020-12-04 15:42:32 +01:00
Dominik Csapak 179145dc24 backup/datastore: move manifest locking to /run
this fixes the issue that on some filesystems, you cannot recursively
remove a directory when you hold a lock on a file inside (e.g. nfs/cifs)

it is not really backwards compatible (so during an upgrade, there
could be two daemons have the lock), but since the locking was
broken before (see previous patch) it should not really matter
(also it seems very unlikely that someone will trigger this)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-03 09:56:42 +01:00
Dominik Csapak 6bd0a00c46 backup/datastore: really lock manifest on delete
'lock_manifest' returns a Result<File, Error> so we always got the result,
even when we did not get the lock, but we acted like we had.

bubble the locking error up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-02 14:37:05 +01:00
Dominik Csapak f6e28f4e62 client/pull: log how many groups to pull were found
if no groups were found, the task log was very confusing as it
contained no real information why nothing was synced, e.g.:

 Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
 Sync datastore 'local-datastore' from 'remote/datastore'
 sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
 TASK OK

this patch simply logs how many groups were found and are about to be synced:

 Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
 Sync datastore 'local-datastore' from 'remote/datastore'
 found 0 groups to sync
 sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
 TASK OK

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-02 07:22:50 +01:00
Dietmar Maurer 2260f065d4 cleanup: use extra file for StoreProgress 2020-12-01 06:34:33 +01:00
Dietmar Maurer 6eff8dec4f cleanup: remove unnecessary StoreProgress clone() 2020-12-01 06:29:11 +01:00
Fabian Grünbichler 7e25b9aaaa verify: use same progress as pull
percentage of verified groups, interpolating based on snapshot count
within the group. in most cases, this will also be closer to 'real'
progress since added snapshots (those which will be verified) in active
backup groups will be roughly evenly distributed, while number of total
snapshots per group will be heavily skewed towards those groups which
have existed the longest, even though most of those old snapshots will
only be re-verified very infrequently.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:22:55 +01:00
Fabian Grünbichler f867ef9c4a progress: add format variants
for iterating over a single group, or iterating just on the group level

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:22:12 +01:00
Fabian Grünbichler fc8920e35d pull: factor out interpolated progress
and add group/snapshot count info.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:13:11 +01:00
Fabian Grünbichler 7f3b0f67e7 remove BackupGroup::list_groups
BackupInfo::list_backup_groups is identical code-wise, and makes more
sense as entry point for listing groups.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:09:44 +01:00
Fabian Grünbichler 844660036b gc: don't limit index listing to same filesystem
WalkDir does not follow symlinks by default anyway, and this behaviour
is not documented anywhere. e.g., if a sysadmin mounts 'extra storage'
for some backup group or type (not knowing that only metadata is stored
in those directories), GC will ignore all the indices contained within
and happily garbage collect their chunks..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:07:09 +01:00
Fabian Grünbichler efcac39d34 gc: remove duplicate variable
list_images already returns absolute paths, we don't need to prepend
anything.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:06:51 +01:00
Fabian Grünbichler cb4b721cb0 gc: log index files found outside of expected scheme
for safety reason, GC finds and marks all index files below the
datastore base path. as a result of regular operations, only index files
within the expected scheme of <TYPE>/<ID>/<TIMESTAMP> should exist.

add a small check + warning if the index list contains index files out
side of this expected scheme, so that an admin with shell access can
investigate.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:06:17 +01:00
Fabian Grünbichler 7956877f14 gc: shorten progress messages
we have messages starting the phases anyway, and limit the number of
progress updates so that context remains available at all times.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:04:13 +01:00
Wolfgang Bumiller 43e60ceb41 file logger: remove test.log after test as well
and a doc formatting fixup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 14:13:21 +01:00
Wolfgang Bumiller b760d8a23f derive PartialEq for Userid
the manual implementation is equivalent

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 14:10:17 +01:00
Wolfgang Bumiller 2c1592263d tiny clippy hint
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 14:03:43 +01:00
Wolfgang Bumiller 616533823c don't enforce Vec and String in tools::join
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:56:59 +01:00
Wolfgang Bumiller 913dddea85 minor cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:56:21 +01:00
Wolfgang Bumiller 3530430365 tools avoid unnecessary copying of parameters/properties
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:53:49 +01:00
Wolfgang Bumiller a4ba60be8f minor cleanups
whitespace, formatting and superfluous lifetime annotations

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:47:31 +01:00
Wolfgang Bumiller 99e98f605c network helpers: fix fd leak in get_network_interfaces
This one always leaked.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller 935ee97b17 use fd_change_cloexec helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller 6b9bfd7fe9 minor cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller dd519bbad1 pxar: stricter file descriptor guards
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller 35fe981c7d client: use tools::pipe instead of nix
nix::unistd::pipe returns unguarded RawFds which should be
avoided

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller b6570abe79 changes for proxmox 0.8
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Fabian Grünbichler 490560e0c6 restore: print to STDERR
else restoring to STDOUT is broken..

Reported-by: Dominic Jäger <d.jaeger@proxmox.com>

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-25 14:38:02 +01:00
Dietmar Maurer 0c9209b04c cli: rename command "upload-log" to "snapshot upload-log" 2020-11-25 07:57:39 +01:00
Dietmar Maurer edebd52374 cli: rename command "forget" to "snapshot forget" 2020-11-25 07:57:39 +01:00
Dietmar Maurer 61205f00fb cli: rename command "files" to "snapshot files" 2020-11-25 07:57:39 +01:00
Dietmar Maurer a303e00289 fingerprint: add new() method 2020-11-25 07:57:39 +01:00
Fabian Grünbichler af9f72e9d8 fingerprint: add bytes() accessor
needed for libproxmox-backup-qemu0

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-25 06:34:34 +01:00
Dietmar Maurer 731eeef25b cli: use new alias feature for "snapshots"
Now maps to "snapshot list".
2020-11-24 13:26:43 +01:00
Dominik Csapak a65e3e4bc0 client: add 'snapshot notes show/update' command
to show and update snapshot notes from the cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-24 11:44:19 +01:00
Fabian Grünbichler 035c40e638 list_snapshots: return manifest fingerprint
for display in clients.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:44:55 +01:00
Fabian Grünbichler 79c535955d refactor BackupInfo -> SnapshotListItem helper
before adding more fields to the tuple, let's just create the struct
inside the match arms to improve readability.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:44:55 +01:00
Fabian Grünbichler 8b7f8d3f3d expose previous backup time in backup env
and use this information to add more information to client backup log
and guide the download manifest decision.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:44:55 +01:00
Fabian Grünbichler 23e4e90540 verification: fix message in notification mail
the errors Vec can contain failed groups as well (e.g., if a group has
no or an invalid owner).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:33:20 +01:00
Fabian Grünbichler a4fa3fc241 verification job: log failed dirs
else users have to manually search through a potentially very long task
log to find the entries that are different.. this is the same summary
printed at the end of a manual verify task.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:33:20 +01:00
Dietmar Maurer 81d10c3b37 cleanup: remove dead code 2020-11-24 08:03:00 +01:00
Fabian Grünbichler f1e2904150 paperkey: refactor common code
from formatting functions to main function, and pass along the key data
lines instead of the full string.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:57:21 +01:00
Fabian Grünbichler 23f9503a31 client: check fingerprint after downloading manifest
this is stricter than the check that happened on manifest load, as it
also fails if the manifest is signed but we don't have a key available.

add some additional output at the start of a backup to indicate whether
a previous manifest is available to base the backup on.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:55:12 +01:00
Fabian Grünbichler a0ef68b93c manifest: check fingerprint when loading with key
otherwise loading will run into the signature mismatch which is
technically true, but not the complete picture in this case.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:49:51 +01:00
Fabian Grünbichler 6b127e6ea0 fix #3139: add key fingerprint to manifest
if the manifest is signed/the contained archives/blobs are encrypted.
stored in 'unprotected' area, since there is already a strong binding
between key and manifest via the signature, and this avoids breaking
backwards compatibility for a simple usability improvement.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:45:11 +01:00
Dietmar Maurer 5e17dbf2bb cli: cleanup 'key show' - use format_and_print_result_full
We now expose all key derivation functions on the cli, so users can
choose between scrypt or pbkdf2.
2020-11-24 07:32:34 +01:00
Fabian Grünbichler dfb04575ad client: add 'key show' command
for (pretty-)printing a keyfile.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:15:29 +01:00
Fabian Grünbichler 6f2626ae19 client: print key fingerprint and master key
for operations where it makes sense.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:11:26 +01:00
Fabian Grünbichler 37e60ddcde key: add fingerprint to key config
and set/generate it on
- key creation
- key passphrase change
- key decryption if not already set
- key encryption with master key

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:03:46 +01:00
Fabian Grünbichler 05cdc05347 crypt config: add fingerprint mechanism
by computing the ID digest of a hash of a static string.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:03:16 +01:00
Dominik Csapak 968a0ab261 fix systemd-encoded upid strings in http client
since we systemd-encode parts of the upid string, and those can contain
characters that are invalid in urls (e.g. '\'), we have to percent encode
those

add a 'percent_encode_component' helper, so that we can maybe change
the AsciiSet for all uses at the same time

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-19 11:01:19 +01:00
Stefan Reiter fd19256470 gc: treat .bad files like regular chunks
Simplify the phase 2 code by treating .bad files just like regular
chunks, with the exception of stat logging.

To facilitate, we need to touch .bad files in phase 1. We only do this
under the condition that 1) the original chunk is missing (as before),
and 2) the original chunk is still referenced somewhere (since the code
lives in the error handler for a failed chunk touch, it only gets called
for chunks we expect to be there, i.e. ones that are referenced).

Untouched they will then be cleaned up after 24 hours (or after the last
longer-running task finishes).

Reason 2) is also a fix for .bad files not being cleaned up at all if
the original is no longer referenced anywhere (e.g. a user deleting all
snapshots after seeing some corrupt chunks appear).

cond_touch_path is introduced to touch arbitrary paths in the chunk
store with the same logic as touching chunks.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-18 14:04:49 +01:00
Fabian Grünbichler 1ed022576c api: include store in invalid owner errors
since a group might exist in plenty stores

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:11:24 +01:00
Fabian Grünbichler f6aa7b38bf drop now unused BackupInfo::list_backups
all global backup listing now happens via BackupGroup

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:11:21 +01:00
Fabian Grünbichler fdfcb74d67 api: filter snapshot counts
unprivileged users should only see the counts related to their part of
the datastore.

while we're at it, switch to a list groups, filter groups, count
snapshots approach (like list_snapshots) to speedup calls to this
endpoint when many unprivileged users share a datastore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:05:50 +01:00
Fabian Grünbichler 98afc7b152 api: make expensive parts of datastore status opt-in
used in the PBS GUI, but also for PVE usage queries which don't need all
the extra expensive information..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:05:47 +01:00
Fabian Grünbichler 0d08fceeb9 improve group/snapshot listing
by listing groups first, then filtering, then listing group snapshots.

this cuts down the number of openat/getdirents calls for users that just
have a partial view of the datastore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 10:37:04 +01:00
Dominik Csapak 3c945d73c2 client/http_client: add put method
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-16 16:59:14 +01:00
Fabian Ebner 58fcbf5ab7 client: expose all-file-systems option
Useful to avoid the need for a long (and possibly changing) list of include-dev
options in certain situations, e.g. nested ZFS file systems. The option is
already implemented and seems to work as expected. The checks for virtual
filesystems are not affected by this option.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-16 16:59:14 +01:00
Fabian Ebner c9097ff801 pxar: avoid including archive root's exclude patterns in .pxarexclude-cli
The patterns from the archive root's .pxarexclude file are already present in
self.patterns when encode_pxarexclude_cli is called. Pass along the number of
CLI patterns and slice accordingly.

Suggested-By: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 13:05:09 +01:00
Wolfgang Bumiller fb01fd3af6 visibility cleanups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:53:50 +01:00
Fabian Ebner fa4bcbcad0 pxar: only generate .pxarexclude-cli if there were CLI parameters
previously a .pxarexclude entry in the root of the archive caused the file to
be generated as well, because the patterns are read before calling
generate_directory_file_list and within the function it wasn't possible to
distinguish between a pattern coming from the CLI and a pattern coming from
archive/root/.pxarexclude

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:18:08 +01:00
Fabian Ebner 189cdb7427 pxar: include .pxarexclude files in the archive
The documentation states:
.pxarexclude files are treated as regular files and will be included in the
backup archive.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:18:06 +01:00
Fabian Ebner 874bd5454d pxar: fix anchored exclusion at archive root
There is no leading slash in an entry's full_path, causing an anchored
exclude at the root level to fail, e.g. having "/name" as the content of the
file archive/root/.pxarexclude didn't match the file archive/root/name

Fix this by prepending a leading slash before matching.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:18:04 +01:00
Wolfgang Bumiller b649887e9a remove unused function
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:15:15 +01:00
Thomas Lamprecht 8c62c15f56 follouwp: whitespace cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-12 11:02:45 +01:00
Mira Limbeck 51ac17b56e api: apt/versions: fix running_kernel string for unknown package case
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-11-12 11:02:20 +01:00
Thomas Lamprecht fc5a012068 manager: versions: non-verbose should actually print server pkg info
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-12 10:28:03 +01:00
Stefan Reiter 5e293f1315 apt: use typed response for get_versions
...and cleanup get_versions for manager CLI.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-12 10:15:32 +01:00
Wolfgang Bumiller 97030c9407 cleanup clippy leftovers
this used to contain a pointer cast, now it doesn't

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 09:43:38 +01:00
Wolfgang Bumiller 5d1d0f5d6c use pin-project to remove more unsafe blocks
we already have it in our dependency tree, so use it

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 09:43:38 +01:00
Thomas Lamprecht 294466ee61 manager: versions: unify printing
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00
Mira Limbeck c100fe9108 add versions command to proxmox-backup-manager
Add the versions command to proxmox-backup-manager with a similar output
to pveversion [-v]. It prints the packages line by line with only the
package name, followed by the version and, for proxmox-backup and
proxmox-backup-server, some additional information (running kernel,
running version).

In addition it supports the optional output-format parameter which can
be used to print the complete data in either json, json-pretty or text
format. If output-format is specified, the --verbose parameter is
ignored and the detailed list of packages is printed.

With the addition of the versions command, the report is extended as
well.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-11-11 18:30:33 +01:00
Thomas Lamprecht e754da3ac2 api: versions: add version also in server package unknown case
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00
Thomas Lamprecht bc1e52bc38 api: versions: rust fmt cleanups
line length limit is 100

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00
Thomas Lamprecht 6f0073bbb5 api: apt update info: do not serialize extra info if none
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00