Commit Graph

255 Commits

Author SHA1 Message Date
Fabian Grünbichler
c1c4a18f48 fix #2865: detect and skip vanished snapshots
also when they have been removed/forgotten since we retrieved the
snapshot list for the currently syncing backup group.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-22 09:19:29 +02:00
Fabian Grünbichler
91f5594c08 api: translate ENOTFOUND to 404 for downloads
and percolate the HttpError back up on the client side

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-22 09:19:29 +02:00
Fabian Grünbichler
86f6f74114 fix #2860: skip in-progress snapshots when syncing
they don't have a final manifest yet and are not done, so they can't be
synced either.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-22 09:19:29 +02:00
Dietmar Maurer
dde18bbb85 proxmox-backup-client benchmark: improve output format 2020-07-10 09:13:52 +02:00
Dietmar Maurer
323b2f3dd6 proxmox-backup-client benchmark: add --verbose flag 2020-07-09 16:16:39 +02:00
Dietmar Maurer
3dacedce71 src/backup/manifest.rs: use serde_json::from_value() to deserialize data
Also modified from_data compute signature ditectly from json.
2020-07-09 09:50:28 +02:00
Dietmar Maurer
b53f637914 src/backup/manifest.rs: cleanup signature generation 2020-07-09 09:20:49 +02:00
Dietmar Maurer
2107a5aebc src/backup/manifest.rs: include signature inside the manifest
This is more flexible, because we can choose what fileds we want to sign.
2020-07-08 16:23:26 +02:00
Dietmar Maurer
3638341aa4 src/backup/file_formats.rs: remove signed chunks
We can include signature in the manifest instead (patch will follow).
2020-07-08 16:23:26 +02:00
Dietmar Maurer
3bad3e6e52 src/client/backup_writer.rs - upload_stream: add crypt_mode 2020-07-08 10:43:28 +02:00
Wolfgang Bumiller
f28d9088ed introduce a CryptMode enum
This also replaces the recently introduced --encryption
parameter on the client with a --crypt-mode parameter.

This can be "none", "encrypt" or "sign-only".

Note that this introduces various changes in the API types
which previously did not take the above distinction into
account properly:

Both `BackupContent` and the manifest's `FileInfo`:
    lose `encryption: Option<bool>`
    gain `crypt_mode: Option<CryptMode>`

Within the backup manifest itself, the "crypt-mode" property
will always be set.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-07 15:24:19 +02:00
Dietmar Maurer
7443a6e092 src/client/remote_chunk_reader.rs: implement clone for RemoteChunkReader 2020-07-07 07:34:58 +02:00
Dietmar Maurer
e9764238df make ReadChunk not require mutable self.
That way we can reduce lock contentions because we lock for much shorter
times.
2020-07-03 07:37:29 +02:00
Dietmar Maurer
4059285649 fix typo 2020-06-29 12:59:25 +02:00
Dietmar Maurer
b957aa81bd update backup api for incremental backup
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-26 07:17:08 +02:00
Dominik Csapak
16021f6ab7 use the existing async method for read_raw_chunk
does the same, except the manual drop, but thats handled there by
letting the value go out of scope

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:23:17 +02:00
Wolfgang Bumiller
3d571d5509 some internal combinator-influenced api cleanup
The download methods used to take the destination by value
and return them again, since this was required when using
combinators before we had `async fn`.
But this is just an ugly left-over now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:46:42 +02:00
Wolfgang Bumiller
4d16badf6f add an AsyncReadChunk trait
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:38:21 +02:00
Wolfgang Bumiller
a609cf210e more cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:01:04 +02:00
Wolfgang Bumiller
5444fa940b turn pxar::flags into bitflags, pxar::Flags
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 11:05:53 +02:00
Wolfgang Bumiller
239e49f927 pxar: create .pxarexclude-cli file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-09 13:17:59 +02:00
Wolfgang Bumiller
c443f58b09 switch to external pxar and fuse crates
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 13:56:58 +02:00
Wolfgang Bumiller
a595f0fee0 client: improve connection/new fingerprint query
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-02 10:40:31 +02:00
Thomas Lamprecht
add5861e8d typo fixes all over the place
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-30 16:39:08 +02:00
Dietmar Maurer
1610c45a86 src/client/pull.rs: also download client.log.blob 2020-05-30 14:51:33 +02:00
Dietmar Maurer
7cc3473a4e src/client/backup_specification.rs: split code into extra file 2020-05-30 10:54:38 +02:00
Dietmar Maurer
4856a21836 src/client/pull.rs: more verbose logging 2020-05-30 08:12:43 +02:00
Thomas Lamprecht
090decbe76 BACKUP_REPO_URL_REGEX: move to api2::types and allow all valid data store names
The repo URL consists of
* optional userid
* optional host
* datastore name

All three have defined regex or format, but none of that is used, so
for example not all valid datastore names are accepted.

Move definition of the regex over to api2::types where we can access
all required regexes easily.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-29 06:29:23 +02:00
Dietmar Maurer
07ad6470ca src/client/pull.rs: split out pull related code 2020-05-22 08:04:20 +02:00
Wolfgang Bumiller
f7d4e4b506 switch from failure to anyhow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Christian Ebner
189996cf4a proxmox-backup-client: expose exclude match patterns to cli.
Allows to pass exclude match patterns to the 'backup' command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 07:56:09 +01:00
Dietmar Maurer
ca2dbb8af1 src/client/backup_writer.rs: also print archive size 2020-02-25 14:12:29 +01:00
Dietmar Maurer
6da73c823f src/client/backup_writer.rs: prefix upload stats with archive name 2020-02-24 13:24:46 +01:00
Dietmar Maurer
e02c3d461f src/client/backup_writer.rs: implement verbose options 2020-02-24 12:48:40 +01:00
Dietmar Maurer
ee5fe978e2 src/client/backup_writer.rs: reduce verbosity 2020-02-24 11:33:00 +01:00
Dietmar Maurer
a6e3da987c src/client/http_client.rs: do not store fingerprints passed via options in cache 2020-02-11 11:54:43 +01:00
Dietmar Maurer
1bd6f32b43 src/client/http_client.rs: always convert fingerprint to lowercase 2020-02-11 11:42:03 +01:00
Dietmar Maurer
d1c657276a src/client/http_client.rs: remove useless password_env 2020-02-11 11:42:03 +01:00
Dietmar Maurer
501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Dietmar Maurer
99d863d770 src/client/http_client.rs: new password_env option 2020-01-27 10:42:37 +01:00
Dietmar Maurer
5030b7cea4 src/client/http_client.rs: new prefix options 2020-01-27 10:20:55 +01:00
Dietmar Maurer
5a74756c15 src/client/http_client.rs: implement fingerprint cache 2020-01-25 15:37:34 +01:00
Dietmar Maurer
d59dbeca1b src/client/http_client.rs: new struct HttpClientOptions 2020-01-25 12:18:00 +01:00
Dietmar Maurer
02141b4d9b src/client/pxar_backup_stream.rs: use a channel instead of a pipe 2020-01-22 11:37:16 +01:00
Wolfgang Bumiller
9ea4bce444 bump proxmox crate to 0.1.7
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).

When we need to depend on "parts" of the crate later on
we'll just have to use features.

The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 13:48:37 +01:00
Wolfgang Bumiller
d973aa827c introduce new runtime tokio helpers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 13:12:40 +01:00
Dietmar Maurer
138910bcd4 src/client/pxar_backup_stream.rs: use std:🧵:Builder to name the thread
To make debugging easier.
2020-01-18 12:21:22 +01:00
Dietmar Maurer
625d19621f src/client/remote_chunk_reader.rs: use tokio::task::block_in_place to avoid blocking
Else we block the current executor.
2020-01-16 13:55:19 +01:00
Christian Ebner
6fc053ed85 pxar: encoder: limit number of max entries held at once in memory during archive creation.
Limit the total number of entries and therefore the approximate memory
consumption instead of doing this on a per directory basis as it was previously.
This makes more sense as it limits not only the width but also the depth of the
directory tree.

Further, instead of hardcoding this value, allow to pass this information as
additional optional parameter 'entires-max'.
By this, creation of the archive with directories containing a large number of
entries is possible.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:08 +01:00
Dietmar Maurer
5e1d1bead1 src/client/backup_reader.rs: make compute_file_csum() public 2020-01-05 14:50:54 +01:00