Commit Graph

5315 Commits

Author SHA1 Message Date
Dominik Csapak
ff99780303 api2/tape/restore: add optional snapshots to 'restore'
this makes it possible to only restore some snapshots from a tape media-set
instead of the whole. If the user selects only a small part, this will
probably be faster (and definitely uses less space on the target
datastores).

the user has to provide a list of snapshots to restore in the form of
'store:type/group/id'
e.g. 'mystore:ct/100/2021-01-01T00:00:00Z'

we achieve this by first restoring the index to a temp dir, retrieving
a list of chunks, and using the catalog, we generate a list of
media/files that we need to (partially) restore.

finally, we copy the snapshots to the correct dir in the datastore,
and clean up the temp dir

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-11 12:53:38 +02:00
Dominik Csapak
fa9507020a api2/tape/restore: refactor restore code into its own function
and create the 'email' and 'restore_owner' variable at the beginning,
so that we can reuse them and do not have to pass the sources of those
through too many functions

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-11 12:53:25 +02:00
Dietmar Maurer
1bff50afea tape locate_file: fix off by one error 2021-05-11 12:37:04 +02:00
Dominik Csapak
37ff72720b docs/api-viewer: improve rendering of array format
by showing
'[format, ...]'
where 'format' is the simple format from the type of the items

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-11 09:12:55 +02:00
Dominik Csapak
2d5d264f99 tape/pool_writer: do not unwrap on channel send
if the reader thread is already gone here, we panic here, resulting in
a nondescript error message, so simply ignore/warn in that case and
return gracefully

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-11 09:07:45 +02:00
Dominik Csapak
c9c07445b7 ui: window/SyncJobEdit: disable autoSelect for remote datastore
when changin the remote, there is a high chance that there are different
datastores, and if a user does not pay attention, now the first store
of the new remote is selected, instead of the one with the same name

disable autoSelect and let the user manually select a remote datastore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-10 16:56:42 +02:00
Dominik Csapak
a4388ffc36 ui: tape: rename 'Datastore' to 'Target Datastore'
we have 2 modi in that window:
* backup has multiple datastores
* backup has single datastore

In the first case we show a 'mapping' grid so that
the user can only restore a part. Here a user sees all source
Datastores and can select a target for each one.

In the second case we only have a single 'Datastore' selector, but
we do not show the source. Because of this, the naming is slightly ambiguous
(is it the 'Source' or the 'Target' ?), so rename it to 'Target Datastore'.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-10 16:56:37 +02:00
Thomas Lamprecht
ea1458923e manager: acme plugin: auto-complete available DNS challenge types
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-10 15:55:49 +02:00
Thomas Lamprecht
e857f1fae8 completion: ACME plugin type: comment out http type for now, not useful
It may make sense in the future, e.g., if the built-in standalone
type is not enough, e.g., as HTTP**s**, HTTP 2 or even QUIC (HTTP 3)
is wanted in some setups, but for now there's no scenario where one
would profit from adding a new HTTP plugin, especially as it requires
the `data` property to be set, which makes no sense..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-10 15:50:08 +02:00
Thomas Lamprecht
3ec42e81b1 manager: acme plugin: remove ID completion helper from add command
we cannot add a plugin with an existing ID so this completion helper
is rather counterproductive...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-10 15:47:37 +02:00
Thomas Lamprecht
be1163acfe config: acme: drop now unused foreach_dns_plugin
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-10 14:41:44 +02:00
Thomas Lamprecht
d308dc8af7 acme: use proxmox-acme-plugins and load schema from there
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-10 14:41:12 +02:00
Thomas Lamprecht
60643023ad api: move AcmeChallengeSchema to acme types module
It will be reused in a later patch in another module which should not
depend on the actual API implementation (ugly and cyclic)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-10 14:39:07 +02:00
Thomas Lamprecht
875d53ef6c api: acme: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-10 11:56:38 +02:00
Wolfgang Bumiller
b41f9e9fec acme: fix bad nonce retry counter
Actually return the error on the 3rd try.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-05-10 11:52:04 +02:00
Dylan Whyte
a1b71c3c7d fix #3296: use proxy client to retrieve changelog
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2021-05-10 08:48:52 +02:00
Dylan Whyte
013fa2d886 fix #3296: use proxy for subscriptions
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2021-05-10 08:48:05 +02:00
Dylan Whyte
72e311c6b2 fix 3296: add http_proxy to node config, and provide a cli
Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-05-10 08:37:46 +02:00
Dietmar Maurer
2732c47466 cleanup src/api2/node/config.rs
- add return type
- fix permissions
- fix descriptions
2021-05-10 08:25:43 +02:00
Dietmar Maurer
0466089316 move api related type/regx definition from backup_info.rs to src/api2/types/mod.rs 2021-05-07 12:45:44 +02:00
Dominik Csapak
5e42d38598 api2/types: add TAPE_RESTORE_SNAPSHOT_SCHEMA
which is 'store:type/id/time'

needed to refactor SNAPSHOT_PATH_REGEX_STR from backup_info

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-07 12:27:50 +02:00
Dominik Csapak
82a4bb5e80 api2/tape/restore: return backup manifest in try_restore_snapshot_archive
we'll use that for partial snapshot restore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-07 12:25:30 +02:00
Fabian Grünbichler
94bc7957c1 progress: shorter format
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-05-07 12:14:37 +02:00
Fabian Grünbichler
c9e6b07145 progress: add current group to output
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-05-07 12:14:23 +02:00
Thomas Lamprecht
3c06eba17a docs: online help info: surpress warnings during scan
We get lots of warnings due to sphinx complaining about missing
includes for generated synopsis. We do not reference to any of those
for now, so we can ignore that now and supress all standard and
warning output.

Note: Errors are still reported.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-07 11:51:30 +02:00
Fabian Grünbichler
8081e4aa7b fix #3331: improve progress for last snapshot in group
especially for the last group, without this the progress would report:

"percentage done: 100.00% (1 of 2 groups, 1 of 1 group snapshots)"

instead of the more logical

"percentage done: 100.00% (2 of 2 groups)"

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-05-07 11:20:17 +02:00
Dietmar Maurer
d8769d659e use build.rs to pass REPOID to rustc-env 2021-05-07 10:11:39 +02:00
Stefan Reiter
572cd0381b file-restore: add debug mode with serial access
Set PBS_QEMU_DEBUG=1 on a command that starts a VM and then connect to
the debug root shell via:
  minicom -D \unix#/run/proxmox-backup/file-restore-serial-10.sock
or similar.

Note that this requires 'proxmox-backup-restore-image-debug' to work,
the postinst script is updated to also generate the corresponding image.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 10:00:12 +02:00
Thomas Lamprecht
5e91b40087 d/control: update for cargo manifest update
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-05-07 09:26:41 +02:00
Stefan Reiter
936eceda61 file-restore: support more drives
A PCI bus can only support up to 32 devices, so excluding built-in
devices that left us with a maximum of about 25 drives. By adding a new
PCI bridge every 32 devices (starting at bridge ID 2 to avoid conflicts
with automatic bridges), we can theoretically support up to 8096 drives.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 09:03:17 +02:00
Stefan Reiter
61c4087041 file-restore: add more RAM for VMs with many drives or debug
The guest kernel requires more memory depending on how many disks are
attached. 256 seems to be enough for basically any reasonable and
unreasonable amount of disks though.

For debug instance, make it 1G, as these are never started automatically
anyway, and need at least 512MB since the initramfs (especially when
including a debug build of the daemon) is substantially bigger.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 09:03:17 +02:00
Stefan Reiter
7d39e47182 file-restore: try to kill VM when stale
Helps to clean up a VM that has crashed, is not responding to vsock API
calls, but still has a running QEMU instance.

We always check the process commandline to ensure we don't kill a random
process that took over the PID.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 09:03:17 +02:00
Wolfgang Bumiller
c4e1af3069 make sure URI paths start with a slash
Otherwise we get an empty error message.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-05-07 08:46:47 +02:00
Dietmar Maurer
3e234af16e tape: improve inline docs for READ POSITION LONG 2021-05-06 11:45:40 +02:00
Dietmar Maurer
bbbf662d20 tape: use LOCATE(16) SCSI command
Turns out this works on LTO4 and newer.
2021-05-06 10:51:59 +02:00
Wolfgang Bumiller
25d78b1068 client: use build_authority in build_uri
so we don't need to also duplicate the IPv6 bracket logic

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-05-06 10:27:40 +02:00
Dietmar Maurer
78bf292343 call create_run_dir() at daemon startup 2021-05-06 10:23:54 +02:00
Dietmar Maurer
e5ef69ecf7 cleanup: split SimpleHttp client into extra file 2021-05-06 10:22:24 +02:00
Dominik Csapak
b7b9a57425 api2/tape/restore: remove unnecessary params from (try_)restore_snapshot_archive
we do not need them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-06 08:02:14 +02:00
Dominik Csapak
c4a04b7c62 api2/tape/restore: factor out check_datastore_privs
so that we can reuse it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-06 08:01:31 +02:00
Dominik Csapak
2e41dbe828 tape/media_catalog: add helpers to look for snapshot/chunk files
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-06 07:58:03 +02:00
Dominik Csapak
56d36ca439 tape/drive: add 'move_to_file' to TapeDriver trait
so that we can directly move to a specified file on the tape

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-06 07:55:08 +02:00
Dietmar Maurer
e0ba5553be http proxy: add necessary brackets for IPv6 proxy 2021-05-05 11:57:04 +02:00
Dominik Csapak
8d6fb677c1 proxmox_restore_daemon: mount ntfs with 'utf8' option
otherwise, the kernel driver exposes file names as iso 8859-1,
but we want to have them as utf8.

This mapping should always work, since UTF16 can be cleanly converted
to UTF8.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-05 11:07:31 +02:00
Dietmar Maurer
a2daecc25d client/http_client: add necessary brackets
if we are given a 'naked' ipv6 without square brackets around it,
we need to add them ourselves, since the address is ambigious otherwise
when we add the port.

e.g. giving 'fe80::1' as address we arrive at the url (with the default port)
'https://fe80::1:8007/'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-05-05 10:30:39 +02:00
Wolfgang Bumiller
ee0c5c8e01 use api_string_type macro
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-05-05 08:24:37 +02:00
Dominik Csapak
ae5b1e188f docs: tape: clarify LTO-4/5 support
some features we need (e.g. READ POSITION long form) are only officially
available with LTO-5, but work on many LTO-4 drives, so move LTO-4 to
'best-effort' support.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-04 13:21:44 +02:00
Dominik Csapak
49f9aca627 tape/restore: optimize chunk restore behaviour
by checking the 'checked_chunks' before trying to write to disk
and by doing the existance check in the parallel handler. This way,
we do not have to check the existance of a chunk multiple times
(if multiple source datastores gets restored to the same target
datastore) and also we do not have to wait on the stat before reading
the next chunk.

We have to change the &WorkerTask to an Arc though, otherwise we
cannot log to the worker from the parallel handler

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-04 13:06:31 +02:00
Dietmar Maurer
4cba875379 bump version to 1.1.6-2 2021-05-04 12:25:16 +02:00
Dietmar Maurer
7ab4382476 update debian/control 2021-05-04 12:24:16 +02:00