Commit Graph

732 Commits

Author SHA1 Message Date
Dietmar Maurer b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer fd6d243843 move ApiConfig, FileLogger and CommandoSocket to proxmox-rest-server workspace
ApiConfig: avoid using  pbs_config::backup_user()
CommandoSocket: avoid using  pbs_config::backup_user()
FileLogger: avoid using  pbs_config::backup_user()
- use atomic_open_or_create_file()

Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer ccc3896ff3 avoid type re-exports 2021-09-14 08:35:43 +02:00
Dietmar Maurer cef5c72682 move src/tape/helpers/snapshot_reader.rs to src/backup/snapshot_reader.rs 2021-09-14 07:42:06 +02:00
Dietmar Maurer 6227654ad8 more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dietmar Maurer e7d4be9d85 move datastore config to pbs_config workspace 2021-09-10 08:40:58 +02:00
Dietmar Maurer bbdda58b35 moved key_derivation.rs from pbs_datastore to pbs-config/src/key_config.rs
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
2021-09-07 10:12:17 +02:00
Dietmar Maurer 2121174827 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Wolfgang Bumiller 6c76aa434d split proxmox-file-restore into its own crate
This also moves a couple of required utilities such as
logrotate and some file descriptor methods to pbs-tools.

Note that the logrotate usage and run-dir handling should be
improved to work as a regular user as this *should* (IMHO)
be a regular unprivileged command (including running
qemu given the kvm privileges...)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:23:29 +02:00
Wolfgang Bumiller b2065dc7d2 cleanup proxmox_backup::backup module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 14:14:04 +02:00
Wolfgang Bumiller ef4df211ab move CachedChunkReader to pbs-datastore
this was actually still missing from the previous commit

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:20:03 +02:00
Wolfgang Bumiller eb5e0ae65a move remaining client tools to pbs-tools/datastore
pbs-datastore now ended up depending on tokio after all, but
that's fine for now

for the fuse code I added pbs-fuse-loop (has the old
fuse_loop and its 'loopdev' module)
ultimately only binaries should depend on this to avoid the
library link

the only thins remaining to move out the client binary are
the api method return types, those will need to be moved to
pbs-api-types...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:12:24 +02:00
Dietmar Maurer ac81ed17b9 fix regression test file permission problems
By simply using the current user/group instead of backup:backup
2021-07-21 09:30:22 +02:00
Dietmar Maurer a301c362e3 add helpers to write configuration files 2021-07-20 18:54:23 +02:00
Dietmar Maurer 7526d86419 use new atomic_open_or_create_file
Factor out open_backup_lockfile() method to acquire locks owned by
user backup with permission 0660.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-20 18:54:23 +02:00
Wolfgang Bumiller 6c221244df move lru cachers to pbs-tools
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 10:57:22 +02:00
Wolfgang Bumiller 38629c3961 move ChunkStream to pbs-client
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 10:52:21 +02:00
Wolfgang Bumiller 2b7f8dd5ea move client to pbs-client subcrate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 12:58:43 +02:00
Dominik Csapak 8e0b852f24 server/prune_job: add proper permission checks to 'prune_datastore'
checks for PRIV_DATASTORE_MODIFY, or else if the auth_id is the backup
owner, and skips the group if not.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:39:01 +02:00
Dominik Csapak 9751ef4b36 backup/datastore: refactor check_backup_owner there
and add a 'owns_backup' convenience function

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:36:02 +02:00
Wolfgang Bumiller ea584a7510 move more api types for the client
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 15:17:53 +02:00
Wolfgang Bumiller 2f02e431b0 moving more code to pbs-datastore
prune and fixed/dynamic index

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 10:40:14 +02:00
Wolfgang Bumiller e64f38cb6b move chunk_stat, read_chunk to pbs-datastore
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 10:40:14 +02:00
Wolfgang Bumiller c23192d34e move chunk_store to pbs-datastore
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 14:37:47 +02:00
Wolfgang Bumiller a5951b4f38 move manifest and backup_info to pbs-datastore
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 11:34:56 +02:00
Wolfgang Bumiller bfff4eaa7f move backup id related types to pbs-api-types
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 11:34:56 +02:00
Wolfgang Bumiller 86fb38776b add pbs-api-types subcrate, move key_derivation
move key_derivation to pbs-datastore

pbs-api-types should only contain "basic" types which
* are usually required by clients
* don't depend on pbs-related code directly

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 09:04:09 +02:00
Wolfgang Bumiller f323e90602 add pbs-datastore module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-06 15:11:52 +02:00
Wolfgang Bumiller 770a36e53a add pbs-tools subcrate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-06 15:10:37 +02:00
Wolfgang Bumiller d420962fbc split out pbs-runtime module
These are mostly tokio specific "hacks" or "workarounds" we
only really need/want in our binaries without pulling it in
via our library crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-06 14:52:25 +02:00
Wolfgang Bumiller 85beb7d875 tree-wide: switch to using mod.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-06 12:04:52 +02:00
Dominik Csapak 4921a411ad backup/datastore: refactor chunk inode sorting to the datastore
so that we can reuse that information

the removal of the adding to the corrupted list is ok, since
'get_chunks_in_order' returns them at the end of the list
and we do the same if the loading fails later in 'verify_index_chunks'
so we still mark them corrupt
(assuming that the load will fail if the stat does)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 12:14:52 +02:00
Hannes Laimer 037e6c0ca8 verify-job: move snapshot filter into function
preparatory steps for fixing #3459

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 11:03:44 +02:00
Fabian Grünbichler 90ff75f85c update to zstd 0.6
compatible with libzstd from bullseye.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-06-14 13:01:43 +02:00
Stefan Reiter da78b90f9c backup: remove AsyncIndexReader
superseded by CachedChunkReader, with less code and more speed

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:46 +02:00
Stefan Reiter 10351f7075 backup: add AsyncRead/Seek to CachedChunkReader
Implemented as a seperate struct SeekableCachedChunkReader that contains
the original as an Arc, since the read_at future captures the
CachedChunkReader, which would otherwise not work with the lifetimes
required by AsyncRead. This is also the reason we cannot use a shared
read buffer and have to allocate a new one for every read. It also means
that the struct items required for AsyncRead/Seek do not need to be
included in a regular CachedChunkReader.

This is intended as a replacement for AsyncIndexReader, so we have less
code duplication and can utilize the LRU cache there too (even though
actual request concurrency is not supported in these traits).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:40 +02:00
Stefan Reiter 70a152deb7 backup: add CachedChunkReader utilizing AsyncLruCache
Provides a fast arbitrary read implementation with full async and
concurrency support.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:37 +02:00
Dominik Csapak 2de4dc3a81 backup/chunk_store: optionally log progress on creation
and enable it for the worker variants

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-04 09:32:09 +02:00
Dominik Csapak 062cf75cdf proxmox-backup-proxy: fix leftover references on datastore removal
when we remove a datastore via api/cli, the proxy
has sometimes leftover references to that datastore in its
DATASTORE_MAP which includes an open filehandle on the
'.lock' file

this prevents unmounting/exporting the datastore even after removal,
only a reload/restart of the proxy did help

add a command to our command socket, which removes all non
configured datastores from the map, dropping the open filehandle

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-06-04 08:22:53 +02:00
Dietmar Maurer 0466089316 move api related type/regx definition from backup_info.rs to src/api2/types/mod.rs 2021-05-07 12:45:44 +02:00
Dominik Csapak 5e42d38598 api2/types: add TAPE_RESTORE_SNAPSHOT_SCHEMA
which is 'store:type/id/time'

needed to refactor SNAPSHOT_PATH_REGEX_STR from backup_info

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-05-07 12:27:50 +02:00
Fabian Grünbichler 94bc7957c1 progress: shorter format
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-05-07 12:14:37 +02:00
Fabian Grünbichler c9e6b07145 progress: add current group to output
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-05-07 12:14:23 +02:00
Fabian Grünbichler 8081e4aa7b fix #3331: improve progress for last snapshot in group
especially for the last group, without this the progress would report:

"percentage done: 100.00% (1 of 2 groups, 1 of 1 group snapshots)"

instead of the more logical

"percentage done: 100.00% (2 of 2 groups)"

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-05-07 11:20:17 +02:00
Hannes Laimer f0d23e5370 add ctime and size function to IndexFile trait
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2021-04-30 11:40:45 +02:00
Wolfgang Bumiller 5e58381ea9 catalog shell: replace LoopState with ControlFlow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-29 11:17:21 +02:00
Dietmar Maurer 28570d19a6 tape restore: avoid multiple stat calls for same chunk 2021-04-16 13:17:17 +02:00
Dietmar Maurer 1369bcdbba tape restore: verify if all chunks exist 2021-04-16 12:20:44 +02:00
Thomas Lamprecht 2e1b63fb25 backup verify: do not check every loop iteration for abort/shutdown
only check every 1024'th, which is cheaper to do than a modulo, as we
can just mask the 10 least-significant-bits and check if the result
is zero.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 13:21:36 +02:00
Thomas Lamprecht 7b2d3a5fe9 backup verify: unify check if chunk can be skipped
This also re-checks the corrupt chunk list before actually loading a
chunk.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 13:21:07 +02:00