Commit Graph

1054 Commits

Author SHA1 Message Date
Dominik Csapak 49f9aca627 tape/restore: optimize chunk restore behaviour
by checking the 'checked_chunks' before trying to write to disk
and by doing the existance check in the parallel handler. This way,
we do not have to check the existance of a chunk multiple times
(if multiple source datastores gets restored to the same target
datastore) and also we do not have to wait on the stat before reading
the next chunk.

We have to change the &WorkerTask to an Arc though, otherwise we
cannot log to the worker from the parallel handler

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-04 13:06:31 +02:00
Dietmar Maurer 39c5db7f0f move basic ACME types into src/api2/types/acme.rs
And rename AccountName into AcmeAccountName.
2021-05-04 11:32:18 +02:00
Dietmar Maurer 603aa09d54 tape restore: do not verify restored files
Because this is too slow and causes the tape motor to stop. Instead,
remove the verify_state from the manifest.
2021-05-04 11:05:32 +02:00
Dietmar Maurer 88aa3076f0 tape restore: add restore speed to logs 2021-05-04 11:05:32 +02:00
Dietmar Maurer 5400fe171c tape restore: write datastore in separate thread 2021-05-04 11:05:32 +02:00
Dietmar Maurer 87bf9f569f tape restore: split restore_chunk_archive
Split out a separate function scan_chunk_archive() for catalog restores.

Note: Required, because we need to optimize restore_chunk_archive() to
write datastore in separate threads (else thape drive will stop during restore)
2021-05-04 11:05:32 +02:00
Wolfgang Bumiller 09989d9963 add node/{node}/config api path
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-05-04 09:32:42 +02:00
Wolfgang Bumiller 4088d5bc62 add node/{node}/certificates api call
API like in PVE:

GET    .../info             => current cert information
POST   .../custom           => upload custom certificate
DELETE .../custom           => delete custom certificate
POST   .../acme/certificate => order acme certificate
PUT    .../acme/certificate => renew expiring acme cert

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-05-04 09:31:30 +02:00
Wolfgang Bumiller d4b84c1dec add config/acme api path
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-05-04 09:30:49 +02:00
Dominik Csapak d1bee4344d ui: tape: handle tapes in changers without barcode
by checking for definedness of the label (tapes without barcode
have the empty string as label-text) and falling back to the
source slot for the load action

Note: Changed the load-slot API from PUT to POST

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-04-30 10:23:53 +02:00
Wolfgang Bumiller d724116c0c add dns alias schema
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-30 08:10:57 +02:00
Dietmar Maurer 467bd01cdf api: add schema for http proxy configuration - HTTP_PROXY_SCHEMA 2021-04-28 11:23:06 +02:00
Stefan Reiter 6a59fa0e18 file-restore: add size to image files and components
Read image sizes (.pxar.fidx/.img.didx) from manifest and partition
sizes from /sys/...

Requires a change to ArchiveEntry, as DirEntryAttribute::Directory
does not have a size associated with it (and that's probably good).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-26 15:46:37 +02:00
Dominik Csapak a588b67906 api2/config/datastore: use update_job_last_run_time for schedules
this way, the api call does not error out when the file is locked
currently (which means that job is running and we do not need
to update the time)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-26 10:51:06 +02:00
Dominik Csapak 37a634f550 server/jobstate: improve name of 'try_update_state_file'
and improve comment

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-26 10:50:36 +02:00
Dominik Csapak 951fe0cb7d server/jobstate: add 'updatd' to Finish variant
when a user updates a job schedule, we want to save that point in time
to calculate future runs, otherwise when a user updates a schedule to
a time that would have been between the last run and 'now' the
schedule is triggered instantly

for example:
schedule 08:00
last run today 08:00
now it is 12:00

before this patch:
update schedule to 11:00
 -> triggered instantly since we calculate from 08:00

after this patch:
update schedule to 11:00
 -> triggered tomorrow 11:00 since we calculate from today 12:00

the change in the enum type is ok, since by default serde does not
error on unknown fields and the new field is optional

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-26 09:48:34 +02:00
Dominik Csapak 4ca3f0c6ae api2/tape/backup: list backed up snapshots on failed backup notification
if a backup task failed (e.g. it was aborted), show the snapshots
which were successfully backed up in the notification

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 16:25:17 +02:00
Dominik Csapak 398636b61c api2/node/status: extend node status
to be more on par with pve

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 10:30:30 +02:00
Dominik Csapak eb70464839 api2/nodes/status: use NodeStatus struct
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 10:30:30 +02:00
Dominik Csapak 75054859ff api2/types: add necessary types for node status
we want to use concrete types instead of value

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 10:30:30 +02:00
Stefan Reiter 4d0dc29951 file-restore: Add 'v' (Virtual) ArchiveEntry type
For the actual partitions and blockdevices in a backup, which the
user sees like folders in the file-restore ui

Encoded as "None", to avoid cluttering DirEntryAttribute, where it
wouldn't make any sense to have.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-21 17:19:40 +02:00
Dietmar Maurer 9104152a83 HttpsConnector: add proxy support 2021-04-21 15:29:17 +02:00
Dietmar Maurer 26153589ba new http client implementation SimpleHttp (avoid static HTTP_CLIENT)
This one will have proxy support.
2021-04-21 15:29:17 +02:00
Dietmar Maurer 28570d19a6 tape restore: avoid multiple stat calls for same chunk 2021-04-16 13:17:17 +02:00
Dietmar Maurer 1369bcdbba tape restore: verify if all chunks exist 2021-04-16 12:20:44 +02:00
Dietmar Maurer 5e4d81e957 tape restore: simplify log (list datastores on single line) 2021-04-16 11:35:05 +02:00
Dietmar Maurer 0f4721f305 tape restore: fix datastore locking 2021-04-16 09:09:05 +02:00
Dietmar Maurer 80acdd71fa tape: do not try to backup unfinished backups 2021-04-15 10:24:14 +02:00
Dominik Csapak a4e871f52c api2/access/user: remove password for @pbs users on removal
so that their password entry is not left in the shadow.json

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-15 08:33:20 +02:00
Dominik Csapak 1dd1c9eb5c api2/tape/restore: restore_chunk_archive: only ignore tape related errors
when we get an error from the tape, we possibly want to ignore it,
i.e. when the file was incomplete, but we still want to error
out if the error came from e.g, the datastore, so we have to move
the error checking code to the 'next_chunk' call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-14 10:38:26 +02:00
Dietmar Maurer 90461b76fb TapeRead: add skip_data() 2021-04-13 13:32:45 +02:00
Stefan Reiter 0dfce17a43 api/datastore: allow pxar file download of entire archive
Treat filepaths like "/root.pxar.didx" without a trailing slash as
wanting to download the entire archive content instead of erroring. The
zip-creation code already works fine for this scenario.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-13 08:26:41 +02:00
Dietmar Maurer 318b310638 tape: improve EOT error handling 2021-04-12 13:27:34 +02:00
Dietmar Maurer 109ccd300f cleanup: move tape SCSI code to src/tape/drive/lto/sg_tape/ 2021-04-09 11:34:45 +02:00
Stefan Reiter a26ebad5f9 file-restore-daemon: add watchdog module
Add a watchdog that will automatically shut down the VM after 10
minutes, if no API call is received.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:58:29 +02:00
Stefan Reiter dd9cef56fc file-restore-daemon: add binary with virtio-vsock API server
Implements the base of a small daemon to run within a file-restore VM.

The binary spawns an API server on a virtio-vsock socket, listening for
connections from the host. This happens mostly manually via the standard
Unix socket API, since tokio/hyper do not have support for vsock built
in. Once we have the accept'ed file descriptor, we can create a
UnixStream and use our tower service implementation for that.

The binary is deliberately not installed in the usual $PATH location,
since it shouldn't be executed on the host by a user anyway.

For now, only the API calls 'status' and 'stop' are implemented, to
demonstrate and test proxmox::api functionality.

Authorization is provided via a custom ApiAuth only checking a header
value against a static /ticket file.

Since the REST server implementation uses the log!() macro, we can
redirect its output to stdout by registering env_logger as the logging
target. env_logger is already in our dependency tree via zstd/bindgen.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-08 13:57:57 +02:00
Dietmar Maurer 1a23132262 tape: add TapeDensity::Unknown 2021-04-08 12:23:54 +02:00
Dietmar Maurer 15d1435789 tape: add vendor, product and revision to LtoDriveAndMediaStatus 2021-04-08 08:34:46 +02:00
Dietmar Maurer 0892a512bc tape: correctly set/display drive option 2021-04-08 07:28:30 +02:00
Dietmar Maurer e29f456efc tape: implement format/erase 2021-04-08 07:28:30 +02:00
Dietmar Maurer a79082a0dd tape: implement LTO userspace driver 2021-04-08 07:28:30 +02:00
Dietmar Maurer 0db5712493 tape: introduce trait BlockRead 2021-04-08 07:28:30 +02:00
Dominik Csapak 93fb2e0d21 api2/types: add type_text to DATASTORE_MAP_FORMAT
This way we get a better rendering in the api-viewer.
before:
 [<string>, ... ]

after:
 [(<source>=)?<target>, ... ]

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-25 13:18:10 +01:00
Dietmar Maurer c553407e98 tape: add --scan option for catalog restore 2021-03-25 13:08:34 +01:00
Dietmar Maurer 4830de408b tape: avoid writing catalogs for empty backup tasks 2021-03-25 12:50:40 +01:00
Dominik Csapak e244b9d03d api2/types: expand DATASTORE_MAP_LIST_SCHEMA description
and give an example

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-25 12:18:14 +01:00
Thomas Lamprecht a32bb86df9 api subscription: drop old hack for api-macro issue
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-25 12:03:33 +01:00
Dominik Csapak 4c4e5c2b1e api2/tape/restore: enable restore mapping of datastores
by changing the 'store' parameter of the restore api call to a
list of mappings (or a single default datastore)

for example giving:
a=b,c=d,e

would restore
datastore 'a' from tape to local datastore 'b'
datastore 'c' from tape to local datastore 'e'
all other datastores to 'e'

this way, only a single datastore can also be restored, by only
giving a single mapping, e.g. 'a=b'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-25 07:46:12 +01:00
Dietmar Maurer 870681013a tape: fix catalog restore
We need to rewind the tape if fast_catalog_restore() fail ...
2021-03-24 10:09:23 +01:00
Dietmar Maurer 8b1289f3e4 tape: skip catalog archives in restore 2021-03-24 09:33:39 +01:00