Commit Graph

6417 Commits

Author SHA1 Message Date
Thomas Lamprecht df5c6a11cd datastore: list snapshots iter: report group dir in error
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-24 19:57:10 +02:00
Dominik Csapak 07a683d266 pbs-client: extract: add top-level dir in tar.zst
when download a folder, include that folder as first entry (except '/')

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-22 11:35:55 +02:00
Dominik Csapak 7098f5d885 pbs-client: extract: rewrite create_zip with sequential decoder
instead of an async recursive function. Not only is it less code,
recursive futures are not really nice and it should be faster too.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-22 11:35:53 +02:00
Thomas Lamprecht f37d8540e1 server pull: fix comment w.r.t. initial downloaded chunk capacity
> The hash set will be able to hold at least capacity elements
> without reallocating. If capacity is 0, the hash set will not
> allocate.
-- rustdoc, HashSet::with_capacity

So, the number we pass is the amount of chunk "IDs" we safe, which is
then 64Ki, not 16Ki and thus the size we can reference too is also
256 GiB, not 64 GiB.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-21 15:55:03 +02:00
Dietmar Maurer eb1cd24e21 pbs-tape: sgutils2: check sense data when status is 'CHECK_CONDITION'
Some raid controllers return a 'transport error' when we expected a
'sense error'. it seems the correct way to check the sense data is when
either the result category is 'SENSE' or when the status is 'CHECK_CONDITION',
so do that. (similar to how 'sg_raw' returns the errors)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-04-21 09:35:52 +02:00
Wolfgang Bumiller 6da20161f0 reference the datastore in BackupGroup/Dir
And drop the base_path parameter on a first bunch of
functions (more reordering will follow).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 15:31:04 +02:00
Wolfgang Bumiller bb628c295a api-types: DataStoreConfig::new for testing
so our examples can more easily access a datastore without
going over a configuration & cache

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 15:31:04 +02:00
Fabian Grünbichler 2c88dc97fd api2: read_remote: also return RemoteWithoutPassword
like for the index, instead of manually stripping it.

this (and the previous change) is backwards-compatible since `Remote`
already skipped serializing empty strings, so the returned JSON is
identical.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-04-20 13:58:41 +02:00
Wolfgang Bumiller 6b0c6492f7 datastore: cleanup and document backup group/dir openers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 13:24:57 +02:00
Wolfgang Bumiller 10a0059602 datastore: drop Hash from BackupGroup
same as for Eq/Ord/...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 13:08:44 +02:00
Wolfgang Bumiller 5203cfcff9 datastore: drop PartialEq and PartialOrd from BackupGroup
Same as previous commits: this will be linked to a
particular DataStore and Eq/Ord is now only part of the
api types, for now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 12:23:14 +02:00
Wolfgang Bumiller cf320b6ba1 datastore: drop Eq and PartialEq from BackupDir
Same as previous commit: this is supposed to be connected to
a datastore and Eq/PartialEq only make sense for the
api-type part.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 12:22:57 +02:00
Wolfgang Bumiller 5116453b6d datastore: drop Ord from BackupGroup
This one is supposed to be linked to a datastore instance,
so it won't be Ord for now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 12:20:30 +02:00
Wolfgang Bumiller db87d93efc make datastore BackupGroup/Dir ctors private
And use the api-types for their contents.

These are supposed to be instances for a datastore, the pure
specifications are the ones in pbs_api_types which should be
preferred in crates like clients which do not need to deal
with the datastore directly.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 11:56:23 +02:00
Wolfgang Bumiller 38aa71fcc8 api-types: use BackupType for GroupFilter::BackupType
instead of a string

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 11:49:01 +02:00
Wolfgang Bumiller 1f6a45c938 rename BackupDir's group_path to relative_group_path
datastore's group_path will be moved to BackupDir soon and
this is required to be able to properly distinguish them

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 10:04:02 +02:00
Wolfgang Bumiller b444eb68af api-types: datastore type improvements
let BackupGroup implement Hash

let BackupGroup and BackupDir be AsRef<BackupGroup>
let BackupDir be AsRef<BackupDir>

the pbs-datastore types will implement these AsRefs as well

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 10:03:39 +02:00
Wolfgang Bumiller 2d5c20c8f5 datastore: remove unused list_files function
it also doesn't belong into this type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 10:00:33 +02:00
Wolfgang Bumiller c4b2d26cdb datastore: move last_backup from BackupInfo to BackupGroup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-20 10:00:25 +02:00
Dietmar Maurer fe94c9962e AuthId: derive Ord and PartialOrd
So the we can sort...

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-04-20 09:58:52 +02:00
Dietmar Maurer 24cb5c7a81 RemoteWithoutPassword: new API type
To make it explicit that we do not return the password.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-04-20 09:42:46 +02:00
Wolfgang Bumiller 988d575dbb api-types: introduce BackupType enum and Group/Dir api types
The type is a real enum.

All are API types and implement Display and FromStr. The
ordering is the same as it is in pbs-datastore.

Also, they are now flattened into a few structs instead of
being copied manually.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 13:12:46 +02:00
Thomas Lamprecht 33eb23d57e datastore: add snapshot iterator and provide example
will be more used in the future, when the upend-datastore master plan
comes in effect.

also a preparatory work for namespaces

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht 249dde8b63 backup: switch over to streaming Iterator improving memory usage
Avoid collecting the whole group list in memory only to iterate and
filter over it again.

Note that the change could result in a indentation change, so best
viewed with `-w` flag.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht 7b125de3e1 datastore: add helper to get a iterator for backup groups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht de015ce7e1 datastore: implement Iterator for backup group listing
While currently it's still only used in a collected() way, most call
sites can be switched over to use the iterator directly, as often
they already convert the not-so-cheap, in-memory vector back in
.into_iter() anyway.

somewhat also preparatory (yak shaving) work for namespaces

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht 72f8154571 api datastore: some code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht 693f3285eb datastore: backup info: drop deprecated list_backup_groups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht 7d9cb8c458 replace deprecated list_backup_group from BackupInfo with Datastore one
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht c90dbb5c7b datastore: move list_backup_groups into Datastore impl
Having that as static method in BackupInfo makes zero sense and just
complicates call sites, which need to extract the base_path from the
store manually upfront.

Mark old fn as deprecated so that we can do the move in a separate
step.

It's also planned to add an Iterator impl for this to allow more
efficient usage in the future.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-15 12:38:16 +02:00
Thomas Lamprecht bdfa637058 client: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 14:25:05 +02:00
Thomas Lamprecht f9a5beaa15 backup client: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 14:06:15 +02:00
Thomas Lamprecht 00ae34dfda tools: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 14:05:17 +02:00
Thomas Lamprecht 9531d2c570 rust fmt for pbs src
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 14:03:46 +02:00
Thomas Lamprecht ee0ea73500 server: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 14:01:25 +02:00
Thomas Lamprecht dc7a5b3491 api: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 13:33:01 +02:00
Thomas Lamprecht 35f151e010 config: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 13:32:04 +02:00
Thomas Lamprecht 42c2b5bec9 datastore: rustfmt whole package
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 13:27:53 +02:00
Thomas Lamprecht fb3c007f8a d/changelog: fixup released
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 09:48:00 +02:00
Thomas Lamprecht ff7568f1d9 bump version to 2.1.6-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-13 17:31:21 +02:00
Thomas Lamprecht 1fd46218ea cli: tape key-restore: print more info for better ux
as getting the marker error if we passed valid json (but not valid
key) is confusing

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-13 16:59:47 +02:00
Thomas Lamprecht ede9dc0d1a api: tape key restore: fix optional param handling and code refactoring
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-13 16:59:47 +02:00
Markus Frank ae60eed310 proxmox-tape: api: restore_key-code moved to tape-encryption-keys
The restore_key api-endpoint is tape/drive/{drive}/restore-key.
Since I cannot set the url parameter for the drivename to null or
undefined, when restoring by exported-key, I moved the
added restore_key-api-code to
"create_key aka POST api2/json/config/tape-encryption-keys" and
added an ApiHandler call in the cli's "restore_key" to call
"create_key" in the api.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2022-04-13 16:31:17 +02:00
Wolfgang Bumiller e3746a329e pbs-client: pxar: avoid some more clones
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:30:40 +02:00
Wolfgang Bumiller 7546e9c997 pbs-client: pxar: avoid some vec extensions
The `Components` `Iterator` has an `as_path()` method to get
the remainder as a borrowed path. This is more efficient
iterating and joining the components into a new `PathBuf`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:28:01 +02:00
Wolfgang Bumiller 0bb4036f25 pbs-client: pxar: drop link_to_pathbuf
pxar's Hardlink and Symlink structs implement `AsRef<OsStr>`
and have an `.as_os_str()` method.

Simply use `Path::new(link)`.

Also, the function was not very well written, and we don't
always need an owned copy.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:18:31 +02:00
Wolfgang Bumiller 84d3af3a0e pbs-client: pxar: fmt
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:17:20 +02:00
Dominik Csapak 055eab54ff ui: datastore/Content: enable tar download in ui
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:08:34 +02:00
Dominik Csapak 984ddb2ff2 api: admin/datastore: add tar support for pxar_file_download
by using the newly added 'create_tar' and the 'ZstdEncoder'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:08:26 +02:00
Dominik Csapak 23af572d3f pbs-client: add 'create_tar' helper function
similar to create_zip, uses an accessor to write a tar into an output
that implements AsyncWrite, but we use a Decoder to iterate instead
of having a recursive function. This is done so that we get the
entries in the correct order, and it should be faster as well.

Includes files, directories, symlinks, hardlink, block/char devs, fifos
into the tar. If the hardlink points to outside the current dir to
archive, promote the first instance to a 'real' file, and use a
hardlink for the rest.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:05:08 +02:00