Commit Graph

487 Commits

Author SHA1 Message Date
Dominik Csapak
e693818afc refactor time functions to tools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:10 +02:00
Wolfgang Bumiller
26e78a2efb downgrade some FIXMEs to TODOs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 11:09:23 +02:00
Wolfgang Bumiller
5444fa940b turn pxar::flags into bitflags, pxar::Flags
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 11:05:53 +02:00
Wolfgang Bumiller
fab2413741 catalog: remove unused SenderWriter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 10:42:42 +02:00
Wolfgang Bumiller
c443f58b09 switch to external pxar and fuse crates
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 13:56:58 +02:00
Dominik Csapak
a95a3fb893 fix csum calculation of not 'chunk_size' aligned images
the last chunk does not have to be as big as the chunk_size,
just use the already available 'chunk_end' function which does the
correct thing

this fixes restoration of images whose sizes are not a multiple of
'chunk_size' as well

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-04 10:18:30 +02:00
Thomas Lamprecht
55919bf141 verify_file: add missing closing parenthesis in error message
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-03 19:10:01 +02:00
Dietmar Maurer
4e14781aec fix typo 2020-06-03 06:59:43 +02:00
Thomas Lamprecht
add5861e8d typo fixes all over the place
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-30 16:39:08 +02:00
Dietmar Maurer
1610c45a86 src/client/pull.rs: also download client.log.blob 2020-05-30 14:51:33 +02:00
Dietmar Maurer
96d65fbcd0 cleanup: define/use const for predefined blob file names. 2020-05-30 14:04:15 +02:00
Dietmar Maurer
8545480a31 src/bin/proxmox-backup-proxy.rs: add simple task scheduler for garbage collection 2020-05-20 08:59:45 +02:00
Dietmar Maurer
07ce44a633 avoid compiler warnings 2020-05-19 07:03:41 +02:00
Dietmar Maurer
7b22acd0c2 src/config/network.rs: make it compatible with pve
and depend on proxmox 0.1.26
2020-05-07 09:28:25 +02:00
Dietmar Maurer
99641a6bbb garbage_collect: call fail_on_abort to abort GV when requested. 2020-05-05 09:06:34 +02:00
Dietmar Maurer
54552dda59 implemnt backup ownership, improve datastore access permissions 2020-04-28 10:22:25 +02:00
Wolfgang Bumiller
f7d4e4b506 switch from failure to anyhow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Dietmar Maurer
cf459b1982 gc: log pending removals 2020-04-06 09:50:40 +02:00
Christian Ebner
32d192a952 catalog: shell: Use the new logic including resolving symlinks for catalog
and remove the old unused code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:51:01 +01:00
Christian Ebner
fee5528e59 catalog: shell: introduce new CatalogPathStack to navigate in catalog shell.
This is basically a rewrite of the current logic for navigating the catalog,
but in addition allows to follow symlinks.
Following symlinks introduces the issue that generation of canonical paths
(needed in the actual pxar archive) is more complex, as symlinks have to be
resolved and loops avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:50:43 +01:00
Christian Ebner
4145c36749 catalog: add missing function documentation.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:50:08 +01:00
Christian Ebner
c2f9149461 catalog: introduce is_symlink() to check if DirEntry is a symlink.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:49:55 +01:00
Christian Ebner
11ee5c0563 catalog: derive PartEq for equality checks of two DirEntry's.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:49:39 +01:00
Christian Ebner
536683e73b src/backup/dynamic_index.rs: Add LruCache for chunks.
In order to improve non-sequential reads of chunks as e.g. in FUSE.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:47:16 +01:00
Christian Ebner
35ddf0b419 catalog: shell: Introduce clear-selected command.
'clear-selected' allows to clear all the match patterns from the list of
patterns for a subsequent restore.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:40:55 +01:00
Christian Ebner
8e464141cf catalog: shell: Improve list-selected command.
'list-selected' now shows the filenames matching the patterns for a restore
instead of the patterns themselfs.
The patterns can be displayed by passing the '--pattern' flag.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:40:42 +01:00
Christian Ebner
03f779c6f5 catalog: shell: Improve output of stat command.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:53:17 +01:00
Dietmar Maurer
8ce49a76da src/backup/backup_info.rs: fix SNAPSHOT_PATH_REGEX and GROUP_PATH_REGEX 2020-02-18 13:16:35 +01:00
Christian Ebner
ba050e3788 catalog: shell: Allow two or more successive slashes in path.
Two or more successive slashes should be allowed and treated as a single slash.
We also do not treat two successive slashes at the beginning of a path any
different.

Details are found here:
https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap04.html#tag_04_11

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-05 09:32:29 +01:00
Dietmar Maurer
501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Dietmar Maurer
d08bc483db use proxmox 0.1.9 with new cli command helpers 2020-01-27 18:08:43 +01:00
Dietmar Maurer
a92830dc39 src/api2/types.rs: define and use api type GarbageCollectionStatus 2020-01-23 13:40:12 +01:00
Dietmar Maurer
51534c8de9 src/backup/manifest.rs: check if manifest contains files 2020-01-23 11:16:12 +01:00
Dietmar Maurer
6abce6c2bb src/backup/datastore.rs: remove_backup_(group/dir) - return Error instead of io::Error 2020-01-23 10:14:46 +01:00
Dietmar Maurer
8a1d68c8b9 src/backup/datastore.rs: improve error messages 2020-01-23 09:58:14 +01:00
Dietmar Maurer
41b373eced src/backup/datastore.rs: new helpers
last_successful_backup: Returns the time of the last successful backup
group_path: Returns the absolute path for a backup_group
snapshot_path: Returns the absolute path for a backup_dir
2020-01-22 15:05:47 +01:00
Wolfgang Bumiller
9ea4bce444 bump proxmox crate to 0.1.7
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).

When we need to depend on "parts" of the crate later on
we'll just have to use features.

The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 13:48:37 +01:00
Wolfgang Bumiller
d973aa827c introduce new runtime tokio helpers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 13:12:40 +01:00
Dietmar Maurer
4b4eba0b9e src/api2/pull.rs: implement delete flag for vanished groups 2020-01-17 11:24:55 +01:00
Dietmar Maurer
11d89239c3 src/backup/backup_info.rs: new list_groups helper 2020-01-17 10:42:03 +01:00
Dietmar Maurer
d2dd827877 src/backup/catalog.rs - SenderWriter: use tokio::task::block_in_place
Make sure we do not block the executor.
2020-01-16 14:24:15 +01:00
Dietmar Maurer
d0187a51a9 src/section_config.rs - convert_to_array: optionally add digest
datastore::config() -> also return digest
remotes::config() -> also return digest
2020-01-14 12:57:03 +01:00
Oguz Bektas
6d20a29d73 fix typo for function name load_and_decrypt_key
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-13 11:14:47 +01:00
Christian Ebner
90fc97af6a pxar::decoder::Decoder: include xattrs and payload size in DirectoryEntry.
By reading and including xattrs and payload size in struct `DirectoryEntry`,
the tuple of return types is avoided and the code is simpler.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:40 +01:00
Dietmar Maurer
8f14e8fe4c src/backup/backup_info.rs: implement last_successful_backup()
A backup is considered successful if there is a manifest file.
2020-01-06 11:35:22 +01:00
Dietmar Maurer
247a8ca5bb src/backup/manifest.rs: impl TryFrom DataBlob 2020-01-05 16:28:18 +01:00
Dietmar Maurer
7759eef552 src/backup/datastore.rs: implement cleanup_backup_dir() 2020-01-05 15:15:12 +01:00
Dietmar Maurer
3758b398cd src/backup/read_chunk.rs: add read_raw_chunk()
Allow to read chunk data without decoding.
2020-01-02 13:29:10 +01:00
Dietmar Maurer
2585a8a4e2 src/backup/chunk_store.rs: implement cond_touch_chunk()
This will be used by backup sync to test if a chunk already exists.
2020-01-02 13:26:28 +01:00
Dietmar Maurer
1cf5178ac5 src/backup/datastore.rs: new helper try_shared_chunk_store_lock() 2020-01-02 11:00:33 +01:00
Dietmar Maurer
1e8da0a789 src/backup/manifest.rs: new helper archive_type() 2019-12-31 15:23:41 +01:00
Dietmar Maurer
645995634a src/api2/config/datastore.rs - create: pass uid and gid instead of User 2019-12-20 09:23:58 +01:00
Dietmar Maurer
e67770d496 src/backup/chunk_store.rs - create: pass User instead of CreateOptions 2019-12-20 09:11:40 +01:00
Wolfgang Bumiller
afdcfb5bc9 let ChunkStore::create take CreateOptions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-19 13:14:49 +01:00
Dietmar Maurer
f74a03da1f remove tools::getpwnam_ugid, impl. crate::backup::backup_user()
And use new nix::unistd::User struct.
2019-12-19 10:20:13 +01:00
Dietmar Maurer
868c585219 src/backup/datastore.rs: avoid divide by zero 2019-12-19 07:14:23 +01:00
Dietmar Maurer
7e210bd0b4 src/backup/chunk_store.rs: create lock file with correct owner 2019-12-19 06:55:53 +01:00
Dietmar Maurer
0b97bc6158 src/backup/chunk_store.rs: use proxmox::tools::fs::create_path 2019-12-18 12:26:43 +01:00
Wolfgang Bumiller
feaa1ad35f replace file_set_contents with replace_file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-18 11:16:04 +01:00
Oguz Bektas
14f1e63067 chunk_store: create parent directories
'datastore create storename /path/to/dir/that/may/not/exist' should
work.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-12-17 15:39:42 +01:00
Christian Ebner
38d9a69875 catalog: decoder and shell: fix incorrect logic in find matching
The find matching was incorrectly performed starting from the parent directroy
and not as intended from the entries of the parent directory.

Further, the match pattern passed from the catalog shell contains the absolute
path of the search entry point as prefix, so find() must always start from the
archive root. This is because the match pattern has to be stored in the selected
list for a subsequent restore-selected command in the shell.
All matching paths are shown as absolute paths with all contents in the subdir,
equal to what would be restored by the given pattern.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 14:57:34 +01:00
Christian Ebner
25cdd0e0a1 catalog_shell: impl find for catalog shell.
Implements the find command which allows to find and select files for subsequent
restore.
Files selected for restore are now stored in a Vec instead of a HashSet.
This is needed, since instead of the full paths for each file, selected files are
now identified by a list of match pattern, where ordering matters.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
Christian Ebner
90dfd0a71d catalog: impl find() for file search by match pattern.
find() iterates over the file tree and matches each node against a list of match
patterns provided at function call.
For each matching node, a callback function with the current directroy stack is
called.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
Dietmar Maurer
236a396aa1 src/api2/admin/datastore.rs - prune: log retention options 2019-12-14 16:32:16 +01:00
Wolfgang Bumiller
db0cb9ce0b update a chunk of stuff to the hyper release
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:24:41 +01:00
Christian Ebner
f084505ec5 src/backup/catalog_shell.rs: sort output of list-selected
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-11 10:54:20 +01:00
Dietmar Maurer
73e57f244e src/backup/datastore.rs - garbage-collection: fix average chunk size 2019-12-10 11:25:29 +01:00
Dietmar Maurer
48ef3c3327 cli: avoid useless .into()
This needs latest proxmox-api updates.
2019-12-09 17:40:34 +01:00
Dietmar Maurer
102d8d4136 src/backup/prune.rs: implement --keep-hourly 2019-12-07 11:23:33 +01:00
Dietmar Maurer
052c4aa160 src/backup/prune.rs: prune - correctly use iso_week year 2019-12-06 19:41:38 +01:00
Dietmar Maurer
9e3f008804 src/backup/prune.rs: add new helper keeps_something() 2019-12-06 12:28:31 +01:00
Dietmar Maurer
55c3cb69cd rename catalog_shell_api() into catalog_shell_cli() 2019-12-06 09:42:05 +01:00
Dietmar Maurer
ecbaa38fa4 src/backup/catalog_shell.rs: factor out catalog_shell_api()
In order to extract documentation ...
2019-12-06 09:35:47 +01:00
Christian Ebner
cca67dc0c3 src/backup/readline.rs: removed in favor of new readline impl
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 09:03:04 +01:00
Christian Ebner
6934c6fe77 src/backup/catalog_shell.rs: adapt to use API Schema definition and rustyline
This major refactoring of the catalog based shell utilizes the new API macro and
the API Schema as well as rustyline instead of the old GNU readline C API.

The code now has these 3 main components:
 * The `Shell` which handles the readline loop via rustyline.
 * The shell functions defined via the API macro.
 * The `Context` which holds catalog and decoder instances.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 09:02:44 +01:00
Dietmar Maurer
9b78352188 src/backup/prune.rs: define new struct PruneOptions 2019-12-06 08:56:27 +01:00
Dietmar Maurer
92acbd69f7 src/backup/backup_info.rs: remove old prune code 2019-12-06 08:29:27 +01:00
Dietmar Maurer
408434360b src/backup/prune.rs: add PruneMark::KeepPartial
Use separate mark for partial backup - dont mess them up with regular ones.
2019-12-06 08:15:32 +01:00
Dietmar Maurer
a8c8366cfa src/backup/prune.rs: factor out code into remove_incomplete_snapshots() 2019-12-06 08:12:08 +01:00
Dietmar Maurer
dc18849156 src/backup/prune.rs: moved prune related code into extra file 2019-12-06 08:06:21 +01:00
Dietmar Maurer
6f47dd8a0f src/backup/backup_info.rs - prune: improve algorythm 2019-12-05 18:41:28 +01:00
Dietmar Maurer
8f0b4c1f90 src/api2/admin/datastore.rs - prune: log info about all snapshots 2019-12-05 15:35:11 +01:00
Dietmar Maurer
9ce42759ec src/backup/backup_info.rs - compute_prune_list: remove unfinished backups 2019-12-05 11:18:10 +01:00
Dietmar Maurer
2c034f8d0a src/backup/backup_info.rs: improve prune algorithm 2019-12-05 08:55:19 +01:00
Dietmar Maurer
aeeac29bb1 src/backup/backup_info.rs: new compute_prune_list helper
So that we can write regression tests for this.
2019-12-04 15:49:11 +01:00
Christian Ebner
314bb35868 catalog_shell: fix several clippy warnings
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-27 10:13:50 +01:00
Christian Ebner
951cf17ee3 catalog_shell: major refactoring of cli command definition and parsing
By changing the way shell commands are defined and parsed, this makes it more
straight forward to extend the current functionality.
The readline input is parsed based on the provided command definition and the
given parameters and options are passed to a command specific callback function.
In addition, the provided command definition including its description is used
to generate a help string to display.
The help command shows a list of all supported commands or the help string for
the provided command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 13:01:00 +01:00
Christian Ebner
59bc6ad676 catalog_shell: check and return on empty list in order to avoid division by zero
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 12:55:33 +01:00
Christian Ebner
4e56b2f792 catalog_shell: do not restore full archive if no entries are selected for restore-selected
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 12:55:18 +01:00
Dietmar Maurer
99b5b6cba9 src/pxar/decoder.rs: use trait object as reader
So that the Decoder is no longer genertic.
2019-11-26 10:45:11 +01:00
Dietmar Maurer
f701d0335e src/pxar/sequential_decoder.rs: remove callback from new()
And use an extra functzion  set_callback() to configure that.

Also rewrite pxar/fuse.rs and implement a generic Session (will get
further cleanups with next patches).
2019-11-26 09:56:48 +01:00
Christian Ebner
f14c96ea38 src/backup/catalog_shell.rs: impl shell to inspect and restore a snapshot via the catalog
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:54:04 +01:00
Christian Ebner
38446a9551 src/backup/readline.rs: impl wrapper for GNU readline
In order to provide the context needed for tab completion via the readline
callback, the needed mut ref is passed via a thread local storage key.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:46:24 +01:00
Christian Ebner
6dba015043 src/backup/catalog.rs: derive Clone for DirEntry and DirEntryAttribute
This is needed in order to explicitly clone the values when needed in the
catalog shell implementation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:15:39 +01:00
Christian Ebner
c9f002213e src/backup/catalog.rs: impl DirEntry::is_directory()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:14:00 +01:00
Wolfgang Bumiller
f569acc5e2 drop uuid crate dependency
proxmox::tools now has a Uuid module using the native
libuuid.

Adds build dependency: libuuid1 (which is a Pre-Depends of
util-linux, so always installed anyway).

Drops uuid + 16 more crate dependencies.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-14 10:22:29 +01:00
Christian Ebner
b423958d8e catalog: fix issue with DirEntry start
start has to be the parent start - offset

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-12 17:18:13 +01:00
Dietmar Maurer
780dd2b0a1 src/backup/catalog.rs: add some docs 2019-11-12 13:23:40 +01:00
Dietmar Maurer
6aa906b547 src/backup/catalog.rs - CatalogReader: verify magic number 2019-11-12 12:54:06 +01:00
Dietmar Maurer
2ec208aef5 src/backup/catalog.rs - CatalogReader::dump(): use root() 2019-11-12 12:47:21 +01:00