Commit Graph

1073 Commits

Author SHA1 Message Date
Stefan Reiter
0dfce17a43 api/datastore: allow pxar file download of entire archive
Treat filepaths like "/root.pxar.didx" without a trailing slash as
wanting to download the entire archive content instead of erroring. The
zip-creation code already works fine for this scenario.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-13 08:26:41 +02:00
Dietmar Maurer
318b310638 tape: improve EOT error handling 2021-04-12 13:27:34 +02:00
Dietmar Maurer
109ccd300f cleanup: move tape SCSI code to src/tape/drive/lto/sg_tape/ 2021-04-09 11:34:45 +02:00
Stefan Reiter
a26ebad5f9 file-restore-daemon: add watchdog module
Add a watchdog that will automatically shut down the VM after 10
minutes, if no API call is received.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:58:29 +02:00
Stefan Reiter
dd9cef56fc file-restore-daemon: add binary with virtio-vsock API server
Implements the base of a small daemon to run within a file-restore VM.

The binary spawns an API server on a virtio-vsock socket, listening for
connections from the host. This happens mostly manually via the standard
Unix socket API, since tokio/hyper do not have support for vsock built
in. Once we have the accept'ed file descriptor, we can create a
UnixStream and use our tower service implementation for that.

The binary is deliberately not installed in the usual $PATH location,
since it shouldn't be executed on the host by a user anyway.

For now, only the API calls 'status' and 'stop' are implemented, to
demonstrate and test proxmox::api functionality.

Authorization is provided via a custom ApiAuth only checking a header
value against a static /ticket file.

Since the REST server implementation uses the log!() macro, we can
redirect its output to stdout by registering env_logger as the logging
target. env_logger is already in our dependency tree via zstd/bindgen.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-08 13:57:57 +02:00
Dietmar Maurer
1a23132262 tape: add TapeDensity::Unknown 2021-04-08 12:23:54 +02:00
Dietmar Maurer
15d1435789 tape: add vendor, product and revision to LtoDriveAndMediaStatus 2021-04-08 08:34:46 +02:00
Dietmar Maurer
0892a512bc tape: correctly set/display drive option 2021-04-08 07:28:30 +02:00
Dietmar Maurer
e29f456efc tape: implement format/erase 2021-04-08 07:28:30 +02:00
Dietmar Maurer
a79082a0dd tape: implement LTO userspace driver 2021-04-08 07:28:30 +02:00
Dietmar Maurer
0db5712493 tape: introduce trait BlockRead 2021-04-08 07:28:30 +02:00
Dominik Csapak
93fb2e0d21 api2/types: add type_text to DATASTORE_MAP_FORMAT
This way we get a better rendering in the api-viewer.
before:
 [<string>, ... ]

after:
 [(<source>=)?<target>, ... ]

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-25 13:18:10 +01:00
Dietmar Maurer
c553407e98 tape: add --scan option for catalog restore 2021-03-25 13:08:34 +01:00
Dietmar Maurer
4830de408b tape: avoid writing catalogs for empty backup tasks 2021-03-25 12:50:40 +01:00
Dominik Csapak
e244b9d03d api2/types: expand DATASTORE_MAP_LIST_SCHEMA description
and give an example

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-25 12:18:14 +01:00
Thomas Lamprecht
a32bb86df9 api subscription: drop old hack for api-macro issue
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-25 12:03:33 +01:00
Dominik Csapak
4c4e5c2b1e api2/tape/restore: enable restore mapping of datastores
by changing the 'store' parameter of the restore api call to a
list of mappings (or a single default datastore)

for example giving:
a=b,c=d,e

would restore
datastore 'a' from tape to local datastore 'b'
datastore 'c' from tape to local datastore 'e'
all other datastores to 'e'

this way, only a single datastore can also be restored, by only
giving a single mapping, e.g. 'a=b'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-25 07:46:12 +01:00
Dietmar Maurer
870681013a tape: fix catalog restore
We need to rewind the tape if fast_catalog_restore() fail ...
2021-03-24 10:09:23 +01:00
Dietmar Maurer
8b1289f3e4 tape: skip catalog archives in restore 2021-03-24 09:33:39 +01:00
Dietmar Maurer
074503f288 tape: implement fast catalog restore 2021-03-24 08:40:34 +01:00
Dietmar Maurer
30316192b3 tape: improve locking (lock media-sets)
- new helper: lock_media_set()

- MediaPool: lock media set

- Expose Inventory::new() to avoid double loading

- do not lock pool on restore (only lock media-set)

- change pool lock name to ".pool-{name}"
2021-03-24 06:32:59 +01:00
Dominik Csapak
54fcb7f5d8 api2/tape/backup: wait indefinitely for lock in scheduled backup jobs
so that a user can schedule multiple backup jobs onto a single
media pool without having to consider timing them apart

this makes sense since we can backup multiple datastores onto
the same media-set but can only specify one datastore per backup job

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-19 09:04:32 +01:00
Dominik Csapak
4abd4dbe38 api2/tape/backup: include a summary on notification e-mails
for now only contains the list of included snapshots (if any),
as well as the backup duration

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-19 09:03:52 +01:00
Dietmar Maurer
32b75d36a8 tape: backup media catalogs 2021-03-19 06:58:46 +01:00
Dietmar Maurer
237314ad0d tape: improve catalog consistency checks
Try to check if we read the correct catalog by verifying uuid, media_set_uuid
and seq_nr.

Note: this changes the catalog format again.
2021-03-18 08:43:55 +01:00
Dietmar Maurer
54722acada tape: store datastore name in tape archives and media catalog
So that we can store multiple datastores on a single media set.
Deduplication is now per datastore (not per media set).
2021-03-17 11:08:51 +01:00
Dominik Csapak
e36135031d ui: tape/Restore: let the user choose an owner
so that the tape backup can be restored as any user, given
the current logged in user has the correct permission.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-16 08:55:42 +01:00
Dietmar Maurer
5c4755ad08 tape: speedup backup by doing read/write in parallel 2021-03-16 08:51:49 +01:00
Dietmar Maurer
2c10410b0d tape: improve backup task log 2021-03-11 08:43:13 +01:00
Thomas Lamprecht
d1d74c4367 typo fixes all over the place
found and semi-manually replaced by using:
 codespell -L mut -L crate -i 3 -w

Mostly in comments, but also email notification and two occurrences
of misspelled  'reserved' struct member, which where not used and
cargo build did not complain about the change, soo ...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-10 16:39:57 +01:00
Dietmar Maurer
6396bace3d tape: improve backup task log (show percentage) 2021-03-10 10:59:13 +01:00
Dominik Csapak
2a06e08618 api2/tape/backup: continue on vanishing snapshots
when we do a prune during a tape backup, do not cancel the tape backup,
but continue with a warning

the task still fails and prompts the user to check the log

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-09 10:20:54 +01:00
Dietmar Maurer
af76234112 tape: improve MediaPool allocation by sorting tapes by ctime and label_text 2021-03-09 08:33:21 +01:00
Dominik Csapak
8703a68a31 tape/backup: add success/error notifications
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-08 14:29:06 +01:00
Dietmar Maurer
d95c74c6e7 tape: test tape alert flags after drive clean 2021-03-08 09:08:55 +01:00
Dominik Csapak
c9793d47f9 api2/tape: add notify_user to backup(-jobs) and restore api calls
so that a user can be given that will be notified for
manual intervention (e.g. inserting a tape)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-05 17:20:37 +01:00
Dominik Csapak
9152a0077f config/tape_job: add notify-user config to tape jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-03-05 12:51:25 +01:00
Dietmar Maurer
0b90c67fb4 tape: avoid Updatable for now 2021-03-05 12:51:25 +01:00
Dietmar Maurer
b4975d3102 tape: finish api permission checks 2021-03-05 11:49:07 +01:00
Dietmar Maurer
ee33795b72 acl: use /tape/device instead of /tape/drive and /tape/changer 2021-03-05 10:06:19 +01:00
Dietmar Maurer
90e16be3ae tape: improve PoolWriter logging
Log reason why we allocate a new media set.
2021-03-05 09:59:38 +01:00
Dietmar Maurer
cf90a369e2 cleanup: rename token_user into auth_id_filter 2021-03-05 08:36:18 +01:00
Dietmar Maurer
ab77d660cc tape: improve media status in list_media 2021-03-04 10:17:10 +01:00
Dietmar Maurer
8a76e71129 tape: impl access permission for media content list 2021-03-03 13:16:15 +01:00
Dietmar Maurer
396fd747a6 tape: impl access permissions for tape jobs 2021-03-03 12:48:06 +01:00
Dietmar Maurer
ccdf327ac8 tape: impl access permissions for encryption key config 2021-03-03 12:21:55 +01:00
Dietmar Maurer
8cd63df0dc tape: impl api permissions for drive/changer/pool configuration 2021-03-03 12:10:00 +01:00
Dominik Csapak
347cde827b dashboard: fix datastore full estimate for not changing usage
If the usage of a datastore did not change, we did not
return an estimate. The ui interpreted this as 'not enough data', but
it should actually be 'never'.

Fixing this by always setting the estimate first to 0 and overwriting
if we successfully calculated one, and checking for 'undefined' in the ui.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-02 17:51:13 +01:00
Dominik Csapak
8bf5769382 api2/tape/drive: add state to DriveListEntry
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-02 12:51:19 +01:00
Dietmar Maurer
db04d10d14 tape: add comment to media pool config 2021-03-02 11:42:58 +01:00
Dietmar Maurer
1d14c31658 Revert "tape: also abort backup/restore on server shutdown"
This reverts commit 9bd81bb384.

Turns out this is not really a good idea.
2021-03-02 08:00:10 +01:00
Dietmar Maurer
9bd81bb384 tape: also abort backup/restore on server shutdown 2021-03-02 07:19:31 +01:00
Dietmar Maurer
1552d9699c tape: add 'Restore Key' button 2021-03-01 11:11:38 +01:00
Dietmar Maurer
16f9ea6708 tape: restore - do not check if pool is configured 2021-03-01 09:43:23 +01:00
Dietmar Maurer
955f4aefcd tape: show media inventory/content, even if there is no pool configuration 2021-03-01 09:21:11 +01:00
Dietmar Maurer
08ec39be0c tape: add api to set media status 2021-02-26 10:44:07 +01:00
Dietmar Maurer
5830e5620d tape: cleanup TapeJob implementation - uses AllOf Schema with SectionConfig 2021-02-25 09:44:31 +01:00
Dietmar Maurer
0dadf66dc7 tape: implement backup using latest-only flag 2021-02-24 13:51:53 +01:00
Dietmar Maurer
21e3ed3449 tape: add 'latest-only' property to backup job config 2021-02-24 11:19:12 +01:00
Dominik Csapak
52d8db7925 api2/config/tape_backup_job: fix duplicate id parameter
since the PUT api call is using the 'Updater', the 'id' parameter is
already encoded in there, tripping up the api verify tests with
'Duplicate keys found in AllOf schema: id'

"fixing" it by removing the explicit id from the api call and
taking it from the Updater (and failing if it does not exists there;
even though that should never happen)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-23 17:23:47 +01:00
Dominik Csapak
875d375d7a api2/types: ArchiveEntry: fix inverse 'leaf' value
a 'leaf' node is every file *except* directories, so we have
to reverse the logtic here

this fixes the pxar.didx browser in the web ui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-23 17:23:47 +01:00
Dominik Csapak
4961404c7c api2/config/tape_backup_job: enable update api call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-23 13:17:49 +01:00
Dietmar Maurer
043018cfbe doc: fix wrong api method description 2021-02-22 12:10:34 +01:00
Dietmar Maurer
7f07991035 docs: improve MEDIA_SET_NAMING_TEMPLATE_SCHEMA description 2021-02-22 09:50:16 +01:00
Dietmar Maurer
18ce01caff tape: use correct schema for changer path (SCSI_CHANGER_PATH_SCHEMA) 2021-02-22 09:43:13 +01:00
Dietmar Maurer
5bc8e80a99 cleanup: cleanup use clause (avoid *) 2021-02-22 09:33:29 +01:00
Dietmar Maurer
6252df4c18 docs: fix EXPORT_SLOT_LIST_SCHEMA description 2021-02-22 09:05:07 +01:00
Dietmar Maurer
86d9f4e733 tape: extend MediaChange trait to return MtxStatus 2021-02-20 10:23:16 +01:00
Dietmar Maurer
3f16f1b006 tape: update changer status inside ScsiMediaChange implementation 2021-02-20 09:56:27 +01:00
Dietmar Maurer
cbd9899389 tape: update changer status cache after load/unload 2021-02-20 09:06:17 +01:00
Dietmar Maurer
4188fd59a0 tape: cache changer state 2021-02-19 16:48:19 +01:00
Dominik Csapak
926d05ef0b api2/tape/{backup, restore}, proxmox-tape: set device state
set the drive device state everywhere we lock it, so that we
know what it currently does

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:16:30 +01:00
Dominik Csapak
8be48ddfc7 api2/tape/changer: add drive state to changer status output
if we can find the drive in the config and it has a state

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:15:15 +01:00
Dominik Csapak
41e66bfaf6 api2/tape/drive: wrap some api calls in run_drive_blocking_task
those calls could also block, so we have to run them in a blocking
tokio task, as to not block the current thread

nice side effect is that we now also update the state for that
drive in those instances

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:14:08 +01:00
Dominik Csapak
47a7241410 api2/tape/drive: use run_drive_blocking_task where possible
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:13:36 +01:00
Dominik Csapak
54c77b3d62 api2/tape/drive: add wrapper for tokio::task::spawn_blocking
similar to the worker wrapper, lock, write status, run code, unset status

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:13:21 +01:00
Dominik Csapak
a1c5575308 api2/tape/drive: use 'run_drive_worker' where possible
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:11:04 +01:00
Dominik Csapak
a44c934b5d api2/tape/drive: add run_drive_worker wrapper
a wrapper for locking, starting the worker and writing the state

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-19 10:08:27 +01:00
Dietmar Maurer
1917ea3ce1 cleanup: use serde(flatten) for VerificationJobStatus, improve code reuse 2021-02-19 09:50:25 +01:00
Dietmar Maurer
70842b9ef2 cleanup: use serde(flatten) for SyncJobStatus, improve code reuse 2021-02-19 09:36:39 +01:00
Dietmar Maurer
7690a8e7bd api: list tape backup jobs with scheduling status 2021-02-19 09:02:13 +01:00
Dietmar Maurer
1689296d46 api: add new type JobScheduleStatus 2021-02-19 08:23:41 +01:00
Dietmar Maurer
9bbd83b1f2 tape: media_list API - allow to update online status for a single changer 2021-02-18 10:59:33 +01:00
Dietmar Maurer
d0647e5a02 tape: use worker tasks for media load/unload 2021-02-18 09:04:51 +01:00
Dominik Csapak
2e21948156 api2/admin/datastore: refactor create_zip into pxar/extract
we will reuse that code in the client, so we need to move it to
where we can access it from the client

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>

[clippy fixes]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:32:41 +01:00
Dominik Csapak
5279ee745f api2/admin/datastore: accept "/" as path for root
makes more sense than sending "root'"

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:32:41 +01:00
Dominik Csapak
227501c063 api2/admin/datastore: refactor list_dir_content in catalog_reader
we will reuse that later in the client, so we need it somewhere
we can use from there

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>

[add strongly typed ArchiveEntry and put api code into helpers.rs]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:32:41 +01:00
Dietmar Maurer
3cdd1a3424 tape: erase_media - automatically load media (with tape libraries). 2021-02-17 06:55:53 +01:00
Dominik Csapak
be61c56c21 api2/tape/drive: add optional label-text to erase
if given, erases the tape only iff the inserted tape contains that label
used to safeguard tape erasing from ui for standalone drives

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-17 06:55:53 +01:00
Dietmar Maurer
159100b944 tape: ui - avoid changer status query in backup content panel 2021-02-15 12:17:36 +01:00
Dietmar Maurer
41a8db3576 tape: implement API to run tape backup jobs 2021-02-15 11:09:30 +01:00
Dietmar Maurer
fe291ab794 cleanup: tape_backup_job.rs - do not use #[allow(non_camel_case_types)] 2021-02-15 10:41:04 +01:00
Dietmar Maurer
8513626b9f proxmox-backup-proxy: schedule tape backup jobs 2021-02-15 08:30:42 +01:00
Dietmar Maurer
42200c405a fix api sort order 2021-02-15 06:35:45 +01:00
Dietmar Maurer
be327dbccd tape: add tape backup job configuration 2021-02-14 10:35:51 +01:00
Dietmar Maurer
7b1bf4c098 tape: erase_media - automatically erase catalog and inventory 2021-02-12 11:30:34 +01:00
Dominik Csapak
7c9fb570cc api2/types/tape/media_location: fix deserialization of MediaLocation::Vault
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-11 14:23:54 +01:00
Dominik Csapak
56d22c66c0 api2/tape/media: add 'move' api call
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-02-11 14:23:27 +01:00
Dietmar Maurer
8f02db04f9 docs: use new type_text option for calendar events 2021-02-11 11:51:22 +01:00
Wolfgang Bumiller
7f9d8438ab tfa: derive WebauthnConfigUpdater via api macro
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-10 10:47:47 +01:00