Commit Graph

4076 Commits

Author SHA1 Message Date
Dietmar Maurer
e3619d4101 moved tape_job.rs to pbs_config workspace 2021-09-07 12:40:15 +02:00
Dietmar Maurer
5839c469c1 move tape_encryption_keys.rs to pbs_config workspace 2021-09-07 10:37:08 +02:00
Dietmar Maurer
bbdda58b35 moved key_derivation.rs from pbs_datastore to pbs-config/src/key_config.rs
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
2021-09-07 10:12:17 +02:00
Dietmar Maurer
45d5d873ce move Kdf and KeyInfo to pbs_api_types workspace 2021-09-07 09:59:59 +02:00
Dominik Csapak
f46806414a tape/inventory: fix the tape tests as user, by mocking the lock
locking during the tests as regular user failed because we try to
chown to the backup user (which is not always possible).

Instead, do not lock at all, by implementing 'open_backup_lockfile' with
'create_mocked_lock'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-07 08:42:04 +02:00
Dietmar Maurer
aad2d162ab move media_pool config to pbs_config workspace 2021-09-06 08:56:04 +02:00
Dietmar Maurer
1ce8e905ea move drive config to pbs_config workspace
Also moved the tape type definitions to pbs_api_types.
2021-09-03 09:10:18 +02:00
Dietmar Maurer
6afdda8832 move remote config into pbs-config workspace 2021-09-02 14:25:15 +02:00
Dietmar Maurer
2121174827 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Wolfgang Bumiller
4c1b776168 another import cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 14:46:01 +02:00
Wolfgang Bumiller
42dad3abd3 fixup imports in tests and examples
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:32:21 +02:00
Wolfgang Bumiller
6c76aa434d split proxmox-file-restore into its own crate
This also moves a couple of required utilities such as
logrotate and some file descriptor methods to pbs-tools.

Note that the logrotate usage and run-dir handling should be
improved to work as a regular user as this *should* (IMHO)
be a regular unprivileged command (including running
qemu given the kvm privileges...)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:23:29 +02:00
Wolfgang Bumiller
e5f9b7f79e split out proxmox-backup-debug binary
and introduce pbs_tools::cli module

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 14:45:48 +02:00
Wolfgang Bumiller
dd2162f6bd more import cleanups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 14:01:03 +02:00
Wolfgang Bumiller
cabdabba3d fixup imports in debug binary
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:29:06 +02:00
Hannes Laimer
3e593a2459 add index recovery to pb-debug
Adds possibility to recover data from an index file. Options:
 - chunks: path to the directory where the chunks are saved
 - file: the index file that should be recovered(must be either .fidx or
   didx)
 - [opt] keyfile: path to a keyfile, if the data was encrypted, a keyfile is
   needed
 - [opt] skip-crc: boolean, if true, read chunks wont be verified with their
   crc-sum, increases the restore speed by a lot

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:19:56 +02:00
Hannes Laimer
7c5287bb95 add file inspection to pb-debug
Adds possibility to inspect .blob, .fidx and .didx files. For index
files a list of the chunks referenced will be printed in addition to
some other information. .blob files can be decoded into file or directly
into stdout. Without decode the tool just prints the size and encryption
mode of the blob file. Options:
 - file: path to the file
 - [opt] decode: path to a file or stdout(-), if specidied, the file will be
   decoded into the specified location [only for blob files, no effect
   with index files]
 - [opt] keyfile: path to a keyfile, needed if decode is specified and the
   data was encrypted

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:19:54 +02:00
Hannes Laimer
7c72ae04f1 add chunk inspection to pb-debug
Adds possibility to inspect chunks and find indexes that reference the
chunk. Options:
 - chunk: path to the chunk file
 - [opt] decode: path to a file or to stdout(-), if specified, the
   chunk will be decoded into the specified location
 - [opt] digest: needed when searching for references, if set, it will
   be used for verification when decoding
 - [opt] keyfile: path to a keyfile, needed if decode is specified and
   the data was encrypted
 - [opt] reference-filter: path in which indexes that reference the
   chunk should be searched, can be a group, snapshot or the whole
   datastore, if not specified no references will be searched
 - [default=true] use-filename-as-digest: use chunk-filename as digest,
   if no digest is specified

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:19:51 +02:00
Wolfgang Bumiller
86582454e8 make api2::helpers::list_dir_content a CatalogReader method
this is its natural place and everything required is already
part of the catalog module

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller
013b1e8bca move some more API types
ArchiveEntry -> pbs-datastore
RestoreDaemonStatus -> pbs-api-types

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller
b2065dc7d2 cleanup proxmox_backup::backup module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 14:14:04 +02:00
Dietmar Maurer
97dfc62f0d remote config: derive and use Updater
Defined a new struct RemoteConfig (without name and password). This makes it
possible to bas64-encode the pasword in the config, but still allow plain
passwords with the API.
2021-08-30 12:48:45 +02:00
Wolfgang Bumiller
e351ac786d split out proxmox-backup-client binary
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 11:39:01 +02:00
Wolfgang Bumiller
7b570c177d move some API return types to pbs-api-types
they'll be required by the api client

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 11:39:01 +02:00
Dietmar Maurer
dbda1513c5 tape: media_pool: derive and use Updater 2021-08-30 11:17:14 +02:00
Dietmar Maurer
c62a6acb2e drive config cleanup: derive and use Updater 2021-08-30 10:50:20 +02:00
Dietmar Maurer
e4a5c072b4 openid cleanup: derive and use Updater 2021-08-30 09:48:53 +02:00
Wolfgang Bumiller
a37c8d2431 use ApiType trait
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-25 10:43:58 +02:00
Dietmar Maurer
a8a20e9210 use new api updater features
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-25 10:43:58 +02:00
Dominik Csapak
9f58e312d7 tape/pool_writer: fix typo
s/wrinting/writing/

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-09 09:36:38 +02:00
Dominik Csapak
cffe0b81e3 tape backup: mention groups that were empty
otherwise a user might get a task log like this:

-----
...
found 7 groups
TASK OK
-----

which could confuse the users as why there were no snapshots backed up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-09 09:28:01 +02:00
Dietmar Maurer
bb14ed8cab cleanup: simplify next_expired_media() 2021-08-04 11:01:18 +02:00
Dietmar Maurer
e5545c9804 cli: proxmox-tape backup-job list: use status api and display next-run an d next-media-label 2021-08-04 09:59:12 +02:00
Dietmar Maurer
efe96ec039 tape: compute next-media-label for each tape backup job 2021-08-04 09:59:12 +02:00
Dietmar Maurer
1d3ae83359 tape: media_pool: implement guess_next_writable_media() 2021-08-04 09:59:12 +02:00
Dominik Csapak
4bb3876352 tape: lto: increase default timeout to 10 minutes
it seems that for some actions or in some circumstances, two minutes is
simply too short and the command aborts. Increase the default timeout to
10 minutes.

While it should give most commands enough time to finish, in case of a real
failure the procedure now takes up to 5 times longer, but IMHO thats an
OK tradeoff.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-03 09:19:13 +02:00
Dominik Csapak
8be69a8453 api/ui: allow zstd compression for new zpools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-30 17:51:13 +02:00
Fabian Ebner
0b12a5a698 api: apt: adapt to further proxmox-apt back-end changes
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-30 10:37:27 +02:00
Fabian Ebner
2eac359430 api: apt: adapt to proxmox-apt back-end changes
It's up to the caller to provide the current release for standard
repository detection/addition.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-30 10:37:27 +02:00
Dominik Csapak
855b55dc14 api2: tape: media: use MediaCatalog::snapshot_list for content listing
this should make the api call much faster, since it is not reading
the whole catalog anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-07-29 13:34:36 +02:00
Dietmar Maurer
5ad40a3dd1 tape: media_catalog: add snapshot list cache for catalog
For some parts of the ui, we only need the snapshot list from the catalog,
and reading the whole catalog (can be multiple hundred MiB) is not
really necessary.

Instead, we write the list of snapshots into a seperate .index file. This file
is generated on demand and is much smaller and thus faster to read.
2021-07-29 13:34:31 +02:00
Dietmar Maurer
7116a2d9da tape: lock media_catalog file to to get a consistent view with load_catalog 2021-07-29 13:34:25 +02:00
Dietmar Maurer
0d5e990a62 cleanup: factor out tape catalog path helpers 2021-07-29 13:34:18 +02:00
Dominik Csapak
4f57f4ad84 tape: changer: add tests for decode_element_status_page
a test for a valid status_page, one with excess data
(in the descriptor as well in the page as a whole)
and a test with too little data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:23:21 +02:00
Dominik Csapak
13e13d836f tape: changer: handle libraries that sends wrong amount of data
if the library sends more data than advertised, simply cut it off,
but if it sends less data, bail out (depending on how much data is
missing, trying to parse it could lead to a panic, so bail out early)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:22:48 +02:00
Dominik Csapak
3ab2432ab6 tape: changer: remove unnecesary inquiry parameter
this is never used, so remove it.
Ok, since they are only non public functions.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:17:07 +02:00
Dominik Csapak
76e8565076 api2: tape/restore: commit temporary catalog at the end
in 'restore_archive', we reach that 'catalog.commit()' for
* every skipped snapshot (we already call 'commit_if_large' then before)
* every skipped chunk archive (no change in catalog since we do not read
  the chunk archive in that case)
* after reading a catalog (no change in catalog)

in all other cases, we call 'commit_if_large' and return early,
meaning that the 'commit' there was executed too often and
unnecessary, so move it after the loop over the files, before
finishing the temporary database.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 11:28:03 +02:00
Dominik Csapak
a2ef36d445 tape: media_catalog: improve chunk_archive interface
instead of having a public start/end_chunk_archive and register_chunks,
simply expose a 'register_chunk_archive' method since we always have
a list of chunks anywhere we want to add them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:18:13 +02:00
Dominik Csapak
42b010174e tape: changer: handle invalid descriptor data from library in status page
We get the descriptor length from the library and use that in
'chunks_exact', which panics on length 0. Catch that case
and bail out, since that makes no sense here anyway.

This could prevent a panic, in case a library sends wrong data.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:05:37 +02:00
Dietmar Maurer
68e77657e6 datastore config: cleanup code (use flatten attribute) 2021-07-23 12:43:33 +02:00