Commit Graph

55 Commits

Author SHA1 Message Date
Wolfgang Bumiller
38774184a9 tree-wide: replace serde_json::from_value(a_value.clone())
`&Value` itself implements `Deserializer` and can therefore
be passed directly to `T::deserialize` without requiring an
intermediate `clone()`. (This also enables optionally
borrowing strings if the result has a short enough lifetime)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-12 16:12:15 +02:00
Hannes Laimer
5fd823c3f2 api: add get_active_operations endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer
4bc84a6549 pbs-datastore: add active operations tracking
Saves the currently active read/write operation counts in a file. The
file is updated whenever a reference returned by lookup_datastore is
dropped and whenever a reference is returned by lookup_datastore. The
files are locked before every access, there is one file per datastore.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer
e9d2fc9362 datastore: add check for maintenance in lookup
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Dominik Csapak
118deb4db8 pbs-datastore: use ConfigVersionCache for datastore
instead of relying on the content of some configs

previously, we always read and parsed the config file, and only
generated a new config object when the path or the 'verify-new' option
changed.

now, we increase the datastore generation on config save, and if that
changed (or the last load is 1 minute in the past), we always
generate a new object

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-03-01 08:16:27 +01:00
Dominik Csapak
fef61684b4 datastore: add tuning option for chunk order
currently, we sort chunks by inode when verifying or backing up to tape.
we get the inode# by stat'ing each chunk, which may be more expensive
than the gains of reading the chunks in order

Since that is highly dependent on the underlying storage of the datastore,
introduce a tuning option  so that the admin can tune that behaviour
for each datastore.

The default stays the same (sorting by inode)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-02-23 09:06:03 +01:00
Dominik Csapak
f26d7ca5c5 use io_format_err, io_bail, io_err_other from proxmox-lang
and move the comment from the local io_bail in pbs-client/src/pxar/fuse.rs
to the only use

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:24:13 +01:00
Dominik Csapak
dcd9c17fff tape/pool_writer: skip already backed up chunks in iterator
currently, the iterator goes over *all* chunks of the index, even
those already backed up by a previous snapshots in the same tape
backup. this is bad since for each iterator, we stat each chunk to
sort by inode number. so to avoid stat'ing the same chunks over
and over for consecutive snapshots, add a 'skip_fn' to the iterator
and in the pool writer and check the catalog_set if we can skip it

this means we can drop the later check for the catalog_set
(since we don't modify that here)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-02-18 10:40:41 +01:00
Fabian Grünbichler
dcf5a0f62d misc clippy fixes
the trivial ones ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-08 14:57:16 +01:00
Fabian Grünbichler
540fca5c9e tree-wide: cleanup manual map/flatten
found with clippy, best viewed with `-w` ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 15:02:07 +01:00
Fabian Grünbichler
9a37bd6c84 tree-wide: fix needless borrows
found and fixed via clippy

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 13:55:33 +01:00
Dietmar Maurer
8a8a1850d0 remove trait BufferedRead from pbs-tools/src/io.rs
We do not need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-25 09:45:47 +01:00
Dietmar Maurer
9eb58647c1 pbs-datastore: use hex::serde feature
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-24 13:06:14 +01:00
Dietmar Maurer
25877d05ac update to proxmox-sys 0.2 crate
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
  proxmox-serde instead)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-24 10:32:27 +01:00
Thomas Lamprecht
a58a5cf795 move HumanByte to pbs-abi-types crate
Originally-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-20 19:35:24 +01:00
Dietmar Maurer
d5790a9f27 use new proxmox-sys crate
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-19 11:06:35 +01:00
Fabian Grünbichler
0ceb97538a BackupGroup: add filter helper
to have a single implementation of "group is matched by group filter".

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Dominik Csapak
342ed4aea0 PruneMark: implement display without the write! macro
by using write_str instead

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 12:54:54 +02:00
Dominik Csapak
d4e9d5470e PruneMark: use copied values instead of references
the type is small enough

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 12:54:54 +02:00
Dominik Csapak
8292d3d20e api2/admin/datastore: add get/set_protection
for gettin/setting the protected flag for snapshots (akin to notes)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:31:11 +02:00
Dominik Csapak
de91418b79 backup/datastore: prevent protected snapshots to be removed
by throwing an error for remove_backup_dir, and skipping for
remove_backup_group

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:31:00 +02:00
Dominik Csapak
02db72678f add protected info of snapshots to api and task logs
adds the info that a snapshot is protected to:
* snapshot list
* manual pruning (also dry-run)
* prune jobs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:51 +02:00
Dominik Csapak
db4b469285 pbs-datastore: skip protected backups in pruning
as a separate keep reason so it will not be calculated for the other reasons

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:47 +02:00
Dominik Csapak
92c5cf42d1 pbs-datastore: add protection info to BackupInfo
and add necessary helper functions (protected_file/is_protected)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:44 +02:00
Dietmar Maurer
e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
Wolfgang Bumiller
5a8726e6d2 pbs-tools: drop borrow module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-13 14:14:03 +02:00
Wolfgang Bumiller
6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Dietmar Maurer
8a23ea4656 move src/backup/read_chunk.rs to pbs-datastore/src/local_chunk_reader.rs 2021-09-27 11:10:14 +02:00
Dietmar Maurer
c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
Dietmar Maurer
6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dietmar Maurer
0fd55b08d9 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer
c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer
b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer
89725197c0 move PruneOptions to pbs_api_types workspace 2021-09-10 09:21:27 +02:00
Dietmar Maurer
58bfa3b19c remove dead code
backup_user() and backup_group() are now in pbs_config workspace
2021-09-08 06:34:44 +02:00
Dietmar Maurer
bbdda58b35 moved key_derivation.rs from pbs_datastore to pbs-config/src/key_config.rs
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
2021-09-07 10:12:17 +02:00
Dietmar Maurer
ed2080762c move data_blob encode/decode from crypt_config.rs to data_blob.rs 2021-09-07 10:00:05 +02:00
Dietmar Maurer
45d5d873ce move Kdf and KeyInfo to pbs_api_types workspace 2021-09-07 09:59:59 +02:00
Wolfgang Bumiller
42dad3abd3 fixup imports in tests and examples
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:32:21 +02:00
Wolfgang Bumiller
86582454e8 make api2::helpers::list_dir_content a CatalogReader method
this is its natural place and everything required is already
part of the catalog module

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller
013b1e8bca move some more API types
ArchiveEntry -> pbs-datastore
RestoreDaemonStatus -> pbs-api-types

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller
ef4df211ab move CachedChunkReader to pbs-datastore
this was actually still missing from the previous commit

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:20:03 +02:00
Wolfgang Bumiller
eb5e0ae65a move remaining client tools to pbs-tools/datastore
pbs-datastore now ended up depending on tokio after all, but
that's fine for now

for the fuse code I added pbs-fuse-loop (has the old
fuse_loop and its 'loopdev' module)
ultimately only binaries should depend on this to avoid the
library link

the only thins remaining to move out the client binary are
the api method return types, those will need to be moved to
pbs-api-types...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:12:24 +02:00
Dominik Csapak
dc46aa9a00 pbs-datastore/prune: make PruneOptions an api type
so that we can reuse it from here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:34:18 +02:00
Dominik Csapak
31c94d1645 chunk_store/insert_chunk: add more information to file errors
otherwise this context is missing in some tasks (e.g. tape restore)
and it is unclear where it came from

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-13 11:55:33 +02:00
Wolfgang Bumiller
ea584a7510 move more api types for the client
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 15:17:53 +02:00
Wolfgang Bumiller
2f02e431b0 moving more code to pbs-datastore
prune and fixed/dynamic index

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 10:40:14 +02:00
Wolfgang Bumiller
e64f38cb6b move chunk_stat, read_chunk to pbs-datastore
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 10:40:14 +02:00
Wolfgang Bumiller
c23192d34e move chunk_store to pbs-datastore
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 14:37:47 +02:00
Wolfgang Bumiller
f8a74456cc test fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 12:17:10 +02:00