Commit Graph

68 Commits

Author SHA1 Message Date
Wolfgang Bumiller 133d718fe4 split the namespace out of BackupGroup/Dir api types
We decided to go this route because it'll most likely be
safer in the API as we need to explicitly add namespaces
support to the various API endpoints this way.

For example, 'pull' should have 2 namespaces: local and
remote, and the GroupFilter (which would otherwise contain
exactly *one* namespace parameter) needs to be applied for
both sides (to decide what to pull from the remote, and what
to *remove* locally as cleanup).

The *datastore* types still contain the namespace and have a
`.backup_ns()` getter.

Note that the datastore's `Display` implementations are no
longer safe to use as a deserializable string.

Additionally, some datastore based methods now have been
exposed via the BackupGroup/BackupDir types to avoid a
"round trip" in code.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller c18d481fd7 pbs-client: don't include empty backup-ns in requests
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Wolfgang Bumiller 8c74349b08 api-types: add namespace to BackupGroup
Make it easier by adding an helper accepting either group or
directory

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:33:50 +02:00
Dominik Csapak 07a683d266 pbs-client: extract: add top-level dir in tar.zst
when download a folder, include that folder as first entry (except '/')

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-22 11:35:55 +02:00
Dominik Csapak 7098f5d885 pbs-client: extract: rewrite create_zip with sequential decoder
instead of an async recursive function. Not only is it less code,
recursive futures are not really nice and it should be faster too.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-22 11:35:53 +02:00
Wolfgang Bumiller db87d93efc make datastore BackupGroup/Dir ctors private
And use the api-types for their contents.

These are supposed to be instances for a datastore, the pure
specifications are the ones in pbs_api_types which should be
preferred in crates like clients which do not need to deal
with the datastore directly.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 11:56:23 +02:00
Wolfgang Bumiller 988d575dbb api-types: introduce BackupType enum and Group/Dir api types
The type is a real enum.

All are API types and implement Display and FromStr. The
ordering is the same as it is in pbs-datastore.

Also, they are now flattened into a few structs instead of
being copied manually.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 13:12:46 +02:00
Thomas Lamprecht bdfa637058 client: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 14:25:05 +02:00
Wolfgang Bumiller e3746a329e pbs-client: pxar: avoid some more clones
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:30:40 +02:00
Wolfgang Bumiller 7546e9c997 pbs-client: pxar: avoid some vec extensions
The `Components` `Iterator` has an `as_path()` method to get
the remainder as a borrowed path. This is more efficient
iterating and joining the components into a new `PathBuf`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:28:01 +02:00
Wolfgang Bumiller 0bb4036f25 pbs-client: pxar: drop link_to_pathbuf
pxar's Hardlink and Symlink structs implement `AsRef<OsStr>`
and have an `.as_os_str()` method.

Simply use `Path::new(link)`.

Also, the function was not very well written, and we don't
always need an owned copy.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:18:31 +02:00
Wolfgang Bumiller 84d3af3a0e pbs-client: pxar: fmt
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:17:20 +02:00
Dominik Csapak 23af572d3f pbs-client: add 'create_tar' helper function
similar to create_zip, uses an accessor to write a tar into an output
that implements AsyncWrite, but we use a Decoder to iterate instead
of having a recursive function. This is done so that we get the
entries in the correct order, and it should be faster as well.

Includes files, directories, symlinks, hardlink, block/char devs, fifos
into the tar. If the hardlink points to outside the current dir to
archive, promote the first instance to a 'real' file, and use a
hardlink for the rest.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:05:08 +02:00
Wolfgang Bumiller 99f09fd3c1 bump proxmox-compression dependency to 0.1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:37:20 +02:00
Wolfgang Bumiller ebb85c1ca3 bump proxmox-schema dependency to 1.3.1 for streaming attribute
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:20:27 +02:00
Wolfgang Bumiller fb6e48f402 bump proxmox-router dependency to 1.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:17:08 +02:00
Wolfgang Bumiller 38774184a9 tree-wide: replace serde_json::from_value(a_value.clone())
`&Value` itself implements `Deserializer` and can therefore
be passed directly to `T::deserialize` without requiring an
intermediate `clone()`. (This also enables optionally
borrowing strings if the result has a short enough lifetime)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-12 16:12:15 +02:00
Fabian Ebner 2b323a359d pxar: create: add entry: fix anchored path pattern matching
Similar to 874bd545 ("pxar: fix anchored exclusion at archive root"),
but this time for inclusion. Because of the inconsistency, it could
happen that a file included in generate_directory_file_list() got
excluded in add_entry(), e.g. with a .pxarexclude file like
> *
> !/supposed-to-be-included

Reported-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-06 16:58:08 +02:00
Dominik Csapak 7c22932c64 pbs-client: print error when we couldn't download previous fidx/didx
When we have a previous manifest, we try to download the fidx/didx files
to get the known chunks list. We continue if that fails (which is ok),
but we did not print any error, leading to a confusing backup output,
since the users would expect that chunks will be reused.

Printing the error should at least make it apparent that something did
not work correctly.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 10:12:29 +02:00
Wolfgang Bumiller bd4562e4b1 bump proxmox-schema dep to 1.3
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-04 09:50:21 +01:00
Wolfgang Bumiller 9c890d72b9 bump proxmox-async dep to 0.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:25:37 +01:00
Wolfgang Bumiller 229c1788c1 bump proxmox-lang dep to 1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:24:24 +01:00
Dominik Csapak f26d7ca5c5 use io_format_err, io_bail, io_err_other from proxmox-lang
and move the comment from the local io_bail in pbs-client/src/pxar/fuse.rs
to the only use

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:24:13 +01:00
Dominik Csapak b066586a47 depend on new 'proxmox-compression' crate
the compression utilities live there now

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:23:43 +01:00
Fabian Grünbichler dcf5a0f62d misc clippy fixes
the trivial ones ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-08 14:57:16 +01:00
Fabian Grünbichler aa174e8e8a tree-wide: drop redundant clones
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 15:02:07 +01:00
Fabian Grünbichler 9a37bd6c84 tree-wide: fix needless borrows
found and fixed via clippy

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 13:55:33 +01:00
Wolfgang Bumiller c3b8e74fdf bump regex dep to 1.5
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-12-16 11:25:02 +01:00
Wolfgang Bumiller 821aa8eae6 bump proxmox-schema to 1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-12-16 11:25:02 +01:00
Dietmar Maurer d4877712f8 pbs-client: avoid mut self in http_client methods.
It is not necessary, so avoid it. The client can now be used
with multiple threads (without using a Mutex).

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-12-04 14:44:05 +01:00
Wolfgang Bumiller 73fba2edea fix a warning in io_return macro
newer compilers warn about the semicolon there, so put
braces around it

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-11-29 11:26:25 +01:00
Dietmar Maurer 726b9d4469 use proxmox-sys 0.2.1 and proxmox-io 1.0.1
And remove unused code from pbs-tools.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-25 12:30:03 +01:00
Dietmar Maurer 577095e2f7 move pbs-tools/src/percent_encoding.rs to pbs-api-types/src/percent_encoding.rs
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-25 11:48:52 +01:00
Dietmar Maurer 92ef0b56d8 move pbs-tools/src/str.rs to pbs-client/src/pxar/create.rs
Code is only used there.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-25 10:43:22 +01:00
Dietmar Maurer 25877d05ac update to proxmox-sys 0.2 crate
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
  proxmox-serde instead)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-24 10:32:27 +01:00
Dietmar Maurer 2d5287fbbc use RateLimitConfig for HttpClient and pull
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-22 07:49:41 +01:00
Thomas Lamprecht a58a5cf795 move HumanByte to pbs-abi-types crate
Originally-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-20 19:35:24 +01:00
Dietmar Maurer 92a8f0bc82 depend on proxmox-async 0.2 2021-11-20 17:14:02 +01:00
Dietmar Maurer 9a1b24b6b1 use new proxmox-async crate
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-19 18:03:22 +01:00
Dietmar Maurer 860eaec58f use proxmox::tools::fd::fd_change_cloexec from proxmox 0.15.3
Depend on proxmox 0.15.3

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-18 13:43:41 +01:00
Dietmar Maurer 09f999337a update to proxmox-http 0.5.4
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-14 08:27:45 +01:00
Dietmar Maurer d5f58006d3 cached_traffic_control: use ShareableRateLimit trait object 2021-11-13 17:49:38 +01:00
Dietmar Maurer 2419dc0de9 pbs-client: add option to use the new RateLimiter
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-10 10:15:32 +01:00
Dietmar Maurer e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
Dietmar Maurer b3f279e2d9 use complete_file_name from proxmox-router 1.1 2021-10-13 14:10:02 +02:00
Wolfgang Bumiller 890b88cbef remove pbs-tools::ops::ControlFlow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-12 14:36:40 +02:00
Wolfgang Bumiller 6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Wolfgang Bumiller e3f3359c86 bump proxmox dependency to 0.14.0 and proxmox-http to 0.5.0
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-08 11:18:22 +02:00
Wolfgang Bumiller f66d66aafe drop dynamic_index.rs duplicate in pbs-client
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-06 15:29:27 +02:00
Wolfgang Bumiller 347e0d4c57 fix deprecated use of std::u64/... modules
integer primitive type modules are deprecated, use
associated constants instead

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-04 15:02:30 +02:00