Commit Graph

117 Commits

Author SHA1 Message Date
Fabian Grünbichler
11ca834317 update to nix 0.24 / rustyline 9 / proxmox-sys 0.3
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-06-02 14:33:33 +02:00
Wolfgang Bumiller
dba37e212b add prune jobs api
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-30 13:58:43 +02:00
Wolfgang Bumiller
5557af0efb api-types: add PruneJobConfig
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-30 13:58:43 +02:00
Thomas Lamprecht
b8858d5186 datastore: avoid unsafe transmute, use to_ne_bytes
which is stable since rustc 1.32 but wasn't available in out
toolchain when this was originally written in commit 7bc1d7277

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-26 11:42:13 +02:00
Thomas Lamprecht
bc001e12e2 datastore: clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-26 11:42:13 +02:00
Fabian Grünbichler
abd8248520 tree-wide: remove DatastoreWithNamespace
instead move the acl_path helper to BackupNamespace, and introduce a new
helper for printing a store+ns when logging/generating error messages.

Suggested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-26 11:42:10 +02:00
Fabian Grünbichler
d22363ad08 BackupDir/BackupGroup: add ns to Debug impl
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
7784698948 BackupGroup: stop implementing Display
this shouldn't be printed/logged - use DatastoreWithNamespace /
pbs_api_types::BackupGroup instead.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
eefa297aa0 BackupDir: stop implementing Display
the api type implements it already, all call sites should rather use
DatastoreWithName and pbs_api_types::BackupDir for logging/..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
f2fe00f1e2 BackupDir: fix manifest_lock_path
this definitely shouldn't rely on BackupDir's Display implementation..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
1afce610c7 tree-wide: prefer api-type BackupDir for logging
in combination with DatastoreWithNamespace to not lose the namespace
information.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Fabian Grünbichler
f15601f1c9 BackupDir: add group/dir accessors
for getting the respective api type references for convenient
printing/logging.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-16 18:49:19 +02:00
Thomas Lamprecht
a7646fe42a Revert "fix #4001: datastore/catalog: add number of files to directory entry"
causes trouble with UI and is inconsistent as its still missing in
file restore (daemon)

We probably want to use a separate property to safe this to avoid
confusion with size.

This reverts commit 66ad63bac2.
2022-05-16 17:51:35 +02:00
Thomas Lamprecht
456456483e datastore: ns iter: clamp depth to MAX_NAMESPACE_DEPTH from datastore root
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-16 09:37:38 +02:00
Thomas Lamprecht
3aafa61362 namespace deletion: propagate delete-groups=false but ENOTEMPTY as error
after all we couldn't delete all that got requested, ideally this
should become a task where we can log what got deleted and what
not...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 16:32:49 +02:00
Thomas Lamprecht
ca3f8757ba datastore: clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 16:01:09 +02:00
Thomas Lamprecht
118e984996 datastore: move backup dir/group/namespace iter to own module
no changes in interface for users of the crate as we re-export
anyway, so more for avoiding to crowd the datastore module to much

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 15:59:43 +02:00
Thomas Lamprecht
d1f9cceada namespace deletion: make destroying groups separate choice
And make that opt-in in the API endpoint, to avoid bad surprises by
default.

If not set we'll only prune empty namespaces.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 14:09:25 +02:00
Stefan Sterz
66ad63bac2 fix #4001: datastore/catalog: add number of files to directory entry
When listing the content of a catalog, add the number of files
contained in the directory as its size. Also removes redundant code,
the `mtime` and the `size` of a file is already set when creating the
archive entry, but we naturally need to override the size now for
directories.

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-15 14:08:51 +02:00
Dominik Csapak
fc99c2791b api: tape/restore: check and create target namespace
checks the privilegs for the target namespace. If that does not exist,
try to recursively create them while checking the privileges.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 14:08:32 +02:00
Thomas Lamprecht
2a088b9975 datastore: drop bogus chunk size check, can cause trouble
other sizes can happen in legitimate and illegitimate ways:
 - illegitimate: encryped chunks and bad actor client
 - legitimate: same chunk but newer zstd version (or compression
   level) can compress it better (or worse) so the

Ideally we could take the actual smaller chunk so that improved zstd
tech gets leveraged, but we could only allow to do that for
un-encrypted ones.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 15:41:20 +02:00
Thomas Lamprecht
0408f60b58 datastore: add new Lookup for operations tracking
We sometimes need to do some in-memory only stuff, e.g., to check if
GC is already running for a datastore, which is a try_lock on a mutex
that is in-memory.

Actually the whole thing would be nicer if we could guarantee to hold
the correct contract statically, e.g., like
https://docs.rust-embedded.org/book/static-guarantees/design-contracts.html

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:36:56 +02:00
Thomas Lamprecht
9ccf933be5 datastore: move update_manifest into BackupDir impl
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht
5566099849 datastore: move cleanup_unreferenced_files to BackupDir impl and fix NS awareness
sync failed on cleanup due to always trying to do so in the root NS

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Wolfgang Bumiller
53d073ec1a datastore: minor cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller
30ccc3003e datastore: relative path fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Thomas Lamprecht
59229bd7f1 api: verify: support namespaces
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller
133d718fe4 split the namespace out of BackupGroup/Dir api types
We decided to go this route because it'll most likely be
safer in the API as we need to explicitly add namespaces
support to the various API endpoints this way.

For example, 'pull' should have 2 namespaces: local and
remote, and the GroupFilter (which would otherwise contain
exactly *one* namespace parameter) needs to be applied for
both sides (to decide what to pull from the remote, and what
to *remove* locally as cleanup).

The *datastore* types still contain the namespace and have a
`.backup_ns()` getter.

Note that the datastore's `Display` implementations are no
longer safe to use as a deserializable string.

Additionally, some datastore based methods now have been
exposed via the BackupGroup/BackupDir types to avoid a
"round trip" in code.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Thomas Lamprecht
15a9272495 datastore: add max-depth to recursive namespace iter
on depth == 0 we only yield the anchor ns, this simplifies usage in
the API as for, e.g. list-snapthos the depth == 0 means only the
snapshots from the passed namespace, nothing below.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Thomas Lamprecht
dc3d716bdb datastore: add create_namespace
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller
13d6de3787 datastore: include namespace in full_path
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Thomas Lamprecht
4c7cc5b39e datastore: add helpers to destroy whole namespaces
The behavior on "any snapshot was protected" isn't yet ideal, as we
then do not cleanup any (sub) namespace, even if some of them where
cleaned from groups & snapshots completely. But that isn't easy to do
with our current depth-first pre-order iterator behavior, and it's
also not completely wrong either, the user can re-do the removal on
the sub-namespaces, so leave that for later.

Should get moved to a datastore::BackupNamespace type once/if we get
one

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Thomas Lamprecht
90e3869690 datastore: add single-level and recursive namespace iterators
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller
11ffd737e3 datastore: add backup_ns accessor
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller
8c74349b08 api-types: add namespace to BackupGroup
Make it easier by adding an helper accepting either group or
directory

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Thomas Lamprecht
d49025064c datastore: chunk store: leverage new format str variable reference
makes it often compact enough for rustfmt to move it into a single
line

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-10 09:46:51 +02:00
Dominik Csapak
dd612daab0 chunk_store: insert_chunk: write chunk again if it is empty on disk
and issue a warning. We can do this, because we know an empty chunk
cannot be valid, and we (assumedly) have a valid chunk in memory.

Having empty chunks on disk is currently possible when PBS crashes,
but the rename of the chunk was flushed to disk, when the actual data
was not.

If it's not empty but there is a size mismatch, return an error.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-10 08:47:40 +02:00
Thomas Lamprecht
0f198b82f5 cargo fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-28 10:26:00 +02:00
Thomas Lamprecht
1eef52c206 datastore: move blob loading into BackupDir impl and adapt call sites
data blobs can only appear in a BackupDir (snapshot) in the backup
hierachy, so makes more sense that it lives in there.

As it wasn't widely used anyway it's easy to move the single
non-package call site over to the new one directly and drop the
implementation from Datastore completely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 20:16:58 +02:00
Thomas Lamprecht
f03649b8f3 datastore: move destroying group or dir into respective impl
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 20:14:39 +02:00
Thomas Lamprecht
5c9c23b6b2 datastore: move manifest locking into BackupDir impl
the manifest is owned by the backup dir (snapshot) so it should also
handle locking, makes no sense to have the implementation somewhere
higher up.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 20:10:43 +02:00
Thomas Lamprecht
b298e9f16e datastore: s/fail_if_not_exist/assert_exists/
avoid putting whole sentences in parameter names

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 20:10:34 +02:00
Thomas Lamprecht
cc295e2c7a datastore: improve backup group/snapshot iters
move the check for directory before doing the OSString -> String
conversion, which should be a bit more efficient.

Also let the match return the entry in the non-skip/return case to
reduce indentation level for the inner "yield element" part, making
it slightly easier to follow.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 20:02:58 +02:00
Thomas Lamprecht
4b77d300a2 datastore: replace manual path assembly by group/dir full_path
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 19:57:20 +02:00
Thomas Lamprecht
df5c6a11cd datastore: list snapshots iter: report group dir in error
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 19:57:10 +02:00
Wolfgang Bumiller
6da20161f0 reference the datastore in BackupGroup/Dir
And drop the base_path parameter on a first bunch of
functions (more reordering will follow).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 15:31:04 +02:00
Wolfgang Bumiller
6b0c6492f7 datastore: cleanup and document backup group/dir openers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 13:24:57 +02:00
Wolfgang Bumiller
10a0059602 datastore: drop Hash from BackupGroup
same as for Eq/Ord/...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 13:08:44 +02:00
Wolfgang Bumiller
5203cfcff9 datastore: drop PartialEq and PartialOrd from BackupGroup
Same as previous commits: this will be linked to a
particular DataStore and Eq/Ord is now only part of the
api types, for now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 12:23:14 +02:00
Wolfgang Bumiller
cf320b6ba1 datastore: drop Eq and PartialEq from BackupDir
Same as previous commit: this is supposed to be connected to
a datastore and Eq/PartialEq only make sense for the
api-type part.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 12:22:57 +02:00