4429 Commits

Author SHA1 Message Date
Dominik Csapak
a2ef36d445 tape: media_catalog: improve chunk_archive interface
instead of having a public start/end_chunk_archive and register_chunks,
simply expose a 'register_chunk_archive' method since we always have
a list of chunks anywhere we want to add them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:18:13 +02:00
Dominik Csapak
42b010174e tape: changer: handle invalid descriptor data from library in status page
We get the descriptor length from the library and use that in
'chunks_exact', which panics on length 0. Catch that case
and bail out, since that makes no sense here anyway.

This could prevent a panic, in case a library sends wrong data.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:05:37 +02:00
Dietmar Maurer
68e77657e6 datastore config: cleanup code (use flatten attribute) 2021-07-23 12:43:33 +02:00
Thomas Lamprecht
cc99866ea3 restore daemon: rust format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:26:10 +02:00
Thomas Lamprecht
1ea3f23f7e file restore: improve some comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:25:34 +02:00
Thomas Lamprecht
3f780ddf73 restore daemon: log about doing basic system env setup
debugging history showed that its surely nice to have more logs at
when stuff happens (and thus fails)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:24:30 +02:00
Thomas Lamprecht
9edf96e6b6 restore daemon: setup backup system user and group
now required as we always enforce lock files to be owned by the
backup user, and the restore code uses such code indirectly as the
REST server module is reused from proxmox-backup-server. Once that is
refactored out we may do away such things, but until then we need to
have a somewhat complete system env.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:19:38 +02:00
Thomas Lamprecht
73e1ba65ca restore daemon: add setup_system_env helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:10:55 +02:00
Dominik Csapak
02631056b8 tape: changer: handle missing dvcid information
the dvcid information is not always available, so skip it if is missing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 12:00:30 +02:00
Dominik Csapak
131d0f10c2 tape: changer: improve error message on wrong counts
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 11:37:14 +02:00
Dominik Csapak
f9aa980c7d tape: changer: correctly consume data in decode_element_status_page
instead of 'blindly' trusting the changer to deliver the fields written
in the specification, trust the length data it returns in the header.

we slice the descriptor data into equal sized chunks of the correct
size, then we do not have care bout the len and empty checks anymore

this also makes the code to read the rest of the page obsolete,
since the next descriptor is on the correct offset anyway

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 11:37:14 +02:00
Dominik Csapak
c1feb447e8 tape: changer: sg_pt: fix typo
ok, since its a private struct

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 17:02:16 +02:00
Dominik Csapak
62a0e190cb tape: changer: sg_pt: add SCSI_VOLUME_TAG_LEN const
so that we do have less 'magic' constants without description

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 17:01:37 +02:00
Dominik Csapak
5890143920 api: types: CHANGER_DRIVENUM_SCHEMA: increase maximum drives per changer
to 255. 8 drives per changer was a rather arbitrary limitation and could
well be reached in practice with big libraries.

Altough 255 is still a arbirtrary limitation, this is much less likely
to be reached in practice.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 16:59:02 +02:00
Wolfgang Bumiller
ef4df211ab move CachedChunkReader to pbs-datastore
this was actually still missing from the previous commit

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:20:03 +02:00
Wolfgang Bumiller
eb5e0ae65a move remaining client tools to pbs-tools/datastore
pbs-datastore now ended up depending on tokio after all, but
that's fine for now

for the fuse code I added pbs-fuse-loop (has the old
fuse_loop and its 'loopdev' module)
ultimately only binaries should depend on this to avoid the
library link

the only thins remaining to move out the client binary are
the api method return types, those will need to be moved to
pbs-api-types...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:12:24 +02:00
Wolfgang Bumiller
bbc71e3b02 client: fix panic message
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 13:28:55 +02:00
Dietmar Maurer
ac81ed17b9 fix regression test file permission problems
By simply using the current user/group instead of backup:backup
2021-07-21 09:30:22 +02:00
Dietmar Maurer
a301c362e3 add helpers to write configuration files 2021-07-20 18:54:23 +02:00
Dietmar Maurer
7526d86419 use new atomic_open_or_create_file
Factor out open_backup_lockfile() method to acquire locks owned by
user backup with permission 0660.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-20 18:54:23 +02:00
Wolfgang Bumiller
fc5870be53 move channel/stream helpers to pbs-tools
pbs_tools
  ::blocking: std/async wrapping with block_in_place
  ::stream: stream <-> AsyncRead/AsyncWrite wrapping

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 11:27:40 +02:00
Wolfgang Bumiller
3c8c2827cb move required_X_param to pbs_tools::json
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 11:09:52 +02:00
Wolfgang Bumiller
6c221244df move lru cachers to pbs-tools
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 10:57:22 +02:00
Wolfgang Bumiller
38629c3961 move ChunkStream to pbs-client
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 10:52:21 +02:00
Wolfgang Bumiller
3fa1b4b48c cleanup unused imports
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:55:19 +02:00
Wolfgang Bumiller
58a3fae773 move pxar binary to separate crate
and move its few remaining proxmox_backup deps out to
pbs-tools

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:53:43 +02:00
Wolfgang Bumiller
0889806a3c resolve some more client imports
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:03:24 +02:00
Wolfgang Bumiller
51ec8a3c62 move some api types to pbs-api-types
and resolve some imports in the client binary

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:01:03 +02:00
Wolfgang Bumiller
a12b1be728 move build.rs and friends to pbs-buildcfg
with this the main crate won't be re-compiled every time a
*binary* is modified

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 14:59:18 +02:00
Wolfgang Bumiller
a3399f4337 doc and tests fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 14:16:28 +02:00
Wolfgang Bumiller
2b7f8dd5ea move client to pbs-client subcrate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 12:58:43 +02:00
Wolfgang Bumiller
72fbe9ffa5 move 'wait_for_local_worker' from client to server
this just made no sense in the client

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:44:44 +02:00
Wolfgang Bumiller
4805edc4ec move more tools for the client into subcrates
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:07:12 +02:00
Wolfgang Bumiller
9eb784076c move more helpers to pbs-tools
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:07:12 +02:00
Wolfgang Bumiller
b9c5cd8291 add proxmox-backup-banner binary crate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:07:12 +02:00
Dominik Csapak
9805207aa5 api: admin/datastore: add new 'prune-datastore' api call
to prune the whole datastore at once, with the given parameters.
We need a new api call since this can take a while and we need to start
a worker for this. The exisiting api call returns a list of removed/kept
snapshots and is synchronous.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:40:05 +02:00
Dominik Csapak
8e0b852f24 server/prune_job: add proper permission checks to 'prune_datastore'
checks for PRIV_DATASTORE_MODIFY, or else if the auth_id is the backup
owner, and skips the group if not.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:39:01 +02:00
Dominik Csapak
0052dc6d28 server/prune_job: add 'keep_all' logic to 'prune_datastore'
it is the same as when pruning single groups.
for prune_jobs, we never start the worker if there is no prune option set.
but if we want to call 'prune_datastore' from somewhere else, we
have to check it here again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:38:28 +02:00
Dominik Csapak
61f05679d2 server/prune_job: factor out 'prune_datastore'
we want to use that outside of a prune job

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:36:45 +02:00
Dominik Csapak
9751ef4b36 backup/datastore: refactor check_backup_owner there
and add a 'owns_backup' convenience function

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:36:02 +02:00
Dominik Csapak
0a240aaa9a api: admin/datastore: simplify prune api call
by using the api macro and reusing the PruneOptions from pbs-datastore

this means we can now drop the 'add_common_prune_prameters' macro

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:34:36 +02:00
Dominik Csapak
e0665a64bd client: simplify prune api method
by using the api macro on the async method and reusing the PruneOptions
from pbs-datastore with 'flatten: true'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:34:28 +02:00
Dominik Csapak
ced694589d api-types: move PRUNE_SCHEMA_KEEP_* to pbs-api-types
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:26:09 +02:00
Dominik Csapak
6c053ffc89 tape: changer: sg_pt: make extra scsi request for dvcid
some libraries cannot handle a request with volume tags and DVCID set at
the same time.

So we make 2 separate requests and merge them, since we want to keep
the vendor/model/serial data.

to not overcomplicate the code, add another special type to ElementType

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 08:46:06 +02:00
Dietmar Maurer
bfd357c5a1 depend on proxmox 0.11.6 (changed make_tmp_file() return type) 2021-07-14 13:37:26 +02:00
Dominik Csapak
9517a5759a fix #3526: correctly filter tasks with 'since' and 'until'
The previous assumption was that the Tasks returned by the Iterator are
sorted by the starttime, but that is not actually the case, and
could never have been, since we append the tasks into the log when
they are finished (not started) and running tasks are always iterated
first.

To correctly filter (and simplify the the api call) we forgo the
combinators, and use a for loop instead. This way we only have to do
the since/until checks only once per Task, but have to do the
start/limit counting ourselves.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-14 09:39:14 +02:00
Stefan Reiter
d9822cd3cb fix #3515: file-restore-daemon: allow LVs/PVs with dash in name
LVM replaces any dashes '-' in an LV or PV name with two '--' for the
created device node in /dev/mapper/ to distinguish the seperating
character between the PV and LV name.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-07-13 12:07:51 +02:00
Stefan Reiter
66501529a2 file-restore: increase lock timeout on QEMU map
This lock is held during VM startup, so that multiple calls will not
start VMs twice. But this means that the timeout needs to incorporate
the time it might take a VM to boot, so increase it quite a bit.

This could previously lead to "interrupted system call" errors when
accessing backups with many disks.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-07-13 12:07:23 +02:00
Dominik Csapak
2072dede4a api2: tape: restore: add warning for list restore
if an error occurs, the snapshot dirs will already be created, and we
do not clean them up (some might already be finished).

Warn the user that they are not cleaned up.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-13 12:02:01 +02:00
Dietmar Maurer
9ee4c23833 tape: changer: sg_pt: always retry until timeout 2021-07-13 10:39:28 +02:00