The idea is to have namespaces in a datastore to allow grouping and
namespacing backups from different (but similar trusted) sources,
e.g., different PVE clusters, geo sites, use-cases or company
service-branches, without separating the underlying
deduplication domain and thus blowing up data and (GC/verify)
resource usage.
To avoid namespace ID clashes with anything existing or future
usecases use a intermediate `ns` level on *each* depth.
The current implementation treats that as internal and thus hides
that fact from the API, iow., the namespace path the users passes
along or gets returned won't include the `ns` level, they do not
matter there at all.
The max-depth of 8 is chosen with the following in mind:
- assume that end-users already are in a deeper level of a hierarchy,
most often they'll start at level one or two, as the higher ones
are used by the seller/admin to namespace different users/groups,
so lower than four would be very limiting for a lot of target use
cases
- all the more, a PBS could be used as huge second level archive in a
big company, so one could imagine a namespace structure like:
/<state>/<intra-state-location>/<datacenter>/<company-branch>/<workload-type>/<service-type>/
e.g.: /us/east-coast/dc12345/financial/report-storage/cassandra/
that's six levels that one can imagine for a reasonable use-case,
leave some room for the ones harder to imagine ;-)
- on the other hand, we do not want to allow unlimited levels as we
have request parameter limits and deep nesting can create other
issues as well (e.g., stack exhaustion), so doubling the minimum
level of 4 (1st point) we got room to breath even for the
more odd (or huge) use cases (2nd point)
- a per-level length of 32 (-1 due to separator) is enough to use
telling names, making lives of users and admin simpler, but not
blowing up parameter total length with the max depth of 8
- 8 * 32 = 256 which is nice buffer size
Much thanks for Wolfgang for all the great work on the type
implementation and assisting greatly with the design.
Co-authored-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Co-authored-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Returning the GC status was dropped by mistake in commit 762f7d15
("datastore status: factor out api type DataStoreStatusListItem")
As this is considered a breaking change which we also felt, due to
the gc-status being used in the web interface for the datastore
overview list (not the dashboard), re add it.
Fixes: 762f7d15
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
[ T: add reference to breaking commit, reword message ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
so our examples can more easily access a datastore without
going over a configuration & cache
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
let BackupGroup implement Hash
let BackupGroup and BackupDir be AsRef<BackupGroup>
let BackupDir be AsRef<BackupDir>
the pbs-datastore types will implement these AsRefs as well
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
The type is a real enum.
All are API types and implement Display and FromStr. The
ordering is the same as it is in pbs-datastore.
Also, they are now flattened into a few structs instead of
being copied manually.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
add support for multi-line comments to node.cfg and the api, similar to
how pve handles multi-line comments
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
the latest changes to this api call changed/removed some things that
were actually necessary for the gui. Readd those and document them this
time.
The change from u64 to i64 limits us to 8EiB of Datastore sizes (instead if
16EiB) but if we reach that, we must adapt most other parts to use 128bit
sizes anyway
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
currently, we sort chunks by inode when verifying or backing up to tape.
we get the inode# by stat'ing each chunk, which may be more expensive
than the gains of reading the chunks in order
Since that is highly dependent on the underlying storage of the datastore,
introduce a tuning option so that the admin can tune that behaviour
for each datastore.
The default stays the same (sorting by inode)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
these need to be checked (and are) via libssl anyway before persisting,
and newer versions might contain new ciphers/variants/... (and things
like @STRENGTH or @SECLEVEL=n were missing).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
there isn't really a concept of 'nodes' in PBS (yet) anyway - and if
there ever is, it needs to be handled by the rest-server / specific API
endpoints (like in PVE), and not by the schema.
this allows dropping proxmox-sys from pbs-api-types (and thus nix and
some other transitive deps as well).
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
it's the only thing requiring openssl in pbs-api-types, and it's only
used by the client to pretty-print the 'master' key, which is
client-specific.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Some users want to import historical backups but they run into the
original lower backuo-time limit one can pass. That original limit
was derived from the initial PBS development start in 2019, it was
assumed that no older backup can exist with PBS before it existing,
but imports of older backups is a legitimate thing.
I pondered using 683071200 (1991-08-25), aka the first time Linux was
publicly announced by Linus Torvalds as new limit but at the end I
did not wanted to risk that and backup software is IMO to serious for
such easter eggs, so I went for 1, to differ between the bogus 0 some
tools fallback too if there's something off with time.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
commit c42a54795d introcuded a bug by
using fp.to_string(). Replace this with fp.signature() which correctly
returns the full fingerprint instead of the short version.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
otherwise we cannot properly parse the api return value from older
versions, since that field does not exist there.
fixes sync from older versions without the protected feature
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Adapted from Dietmar's v3 on pbs-devel but some changes:
- reworked with a strip_suffix fn that does matching, way shorter and
even easier to read IMO
- make b/B byte symbol fully optional, not just for base-10
- also trim trailing whitespace for SizeUnit::Byte
- simplify the FromStr impl
- adapt parser unit tests such that we actually see the failed test's
definition line, simplifies debugging a bit
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
The new SizeUnit type takes over the auto scaling logic and could be
used on its own too.
Switch the internal type of HumanByte from u64 to f64, this results
in a slight reduce of usable sizes we can represent (there's no
unsigned float type after all) but we support pebibyte now with quite
the precision and ebibytes should be also work out ok, and that
really should us have covered for a while..
Partially adapted by Dietmar's version, but split up and change so:
* there's no None type, for a SizeUnit that does not makes much sense
* print the unit for byte too, better consistency and one can still
use as_u64() or as_f64() if they do not want/need the unit rendered
* left the "From usize/u64" impls intact, just convenient to have and
avoids all over the tree changes to adapt to loosing that
* move auto-scaling into SizeUnit, good fit there and I could see
some re-use potential for non-human-byte users in the future
* impl Display for SizeUnit instead of the separate unit_str method,
better usability as it can be used directly in format (with zero
alloc/copy) and saw no real reason of not having that this way
* switch the place where we auto-scale in HumanByte's to the new_X
helpers which allows for slightly reduced code usage and simplify
implementation where possible
* use rounding for the precision limit algorithm. This is a stupid
problem as in practices there are cases for requiring every variant:
- flooring would be good for limits, better less than to much
- ceiling would be good for file sizes, to less can mean ENOSPACE
and user getting angry if their working value is messed with
- rounding can be good for rendering benchmark, closer to reality
and no real impact
So going always for rounding is really not the best solution..
Some of those changes where naturally opinionated, if there's a good
practical reason we can switch back (or to something completely
different).
The single thing I kept and am not _that_ happy with is being able to
have fractional bytes (1.1 B or even 0.01 B), which just does not
makes much sense as most of those values cannot exist at all in
reality - I say most as multiple of 1/8 Byte can exists, those are
bits.o
Note, the precission also changed from fixed 2 to max 3 (trailing
zeros stripped), while that can be nice we should see if we get
a better precision limiting algorithm, e.g., directly in the printer.
Rust sadly does not supports "limit to precision of 3 but avoid
trailing zeros" so we'd need to adapt their Grisu based algorithm our
own - way to much complexity for this though..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Thomas Lamprecht
Dominik Csapak
Hannes Laimer
Wolfgang Bumiller
Dietmar Maurer
Stefan Sterz
Fabian Grünbichler