tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
6,304 Commits 2 Branches 54 Tags 14 MiB
473063e9ec
Commit Graph

755 Commits

Author SHA1 Message Date
Dietmar Maurer
0bf7ba6c92 src/backup/manifest.rs: cleanup - avoid recursive call to write_canonical_json 2020-07-27 08:48:11 +02:00
Thomas Lamprecht
3a3af6e2b6 backup manifest: make lookup_file_info public 
useful to get info like, was the previous snapshot encrypted in
libproxmox-backup-qemu

Requested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-23 10:39:21 +02:00
Thomas Lamprecht
7e42ccdaf2 fixed index: chunk_from_offset: avoid slow modulo operation 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 17:46:07 +02:00
Stefan Reiter
e713ee5c56 remove BufferedFixedReader interface 
replaced by AsyncIndexReader

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-07-22 17:28:49 +02:00
Stefan Reiter
ec5f9d3525 implement AsyncSeek for AsyncIndexReader 
Requires updating the AsyncRead implementation to cope with byte-wise
seeks to intra-chunk positions.

Uses chunk_from_offset to get locations within chunks, but tries to
avoid it for sequential read to not reduce performance from before.

AsyncSeek needs to use the temporary seek_to_pos to avoid changing the
position in case an invalid seek is given and it needs to error in
poll_complete.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-07-22 17:28:49 +02:00
Stefan Reiter
d0463b67ca add and implement chunk_from_offset for IndexFile 
Necessary for byte-wise seeking through chunks in an index.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-07-22 17:28:49 +02:00
Thomas Lamprecht
2ff4c2cd5f datastore/chunker: fix comment typos 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 16:12:49 +02:00
Thomas Lamprecht
c3b090ac8a backup: list images: handle walkdir error, catch "lost+found" 
We support using an ext4 mountpoint directly as datastore and even do
so ourself when creating one through the disk manage code.

Such ext4 ountpoints have a lost+found directory which only root can
traverse into. As the GC list images is done as backup:backup user
walkdir gets an error.

We cannot ignore just all permission errors, as they could lead to
missing some backup indexes and thus possibly sweeping more chunks
than desired. While *normally* that should not happen through our
stack, we had already user report that they do rsyncs to move a
datastore from old to new server and got the permission wrong.

So for now be still very strict, only allow a "lost+found" directory
as immediate child of the datastore base directory, nothing else.

If deemed safe, this can always be made less strict. Possibly by
filtering the known backup-types on the highest level first.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 16:01:55 +02:00
Thomas Lamprecht
c47e294ea7 datastore: fix typo 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 15:04:14 +02:00
Fabian Grünbichler
25455bd06d fix #2871: close FDs when scanning backup group 
otherwise we leak those descriptors and run into EMFILE when a backup
group contains many snapshots.

fcntl::openat and Dir::openat are not the same ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 2020-07-22 09:19:29 +02:00
Aaron Lauterer
b96b11cdb7 chunk_store: Fix typo in bail message 
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
 2020-07-21 12:51:41 +02:00
Fabian Grünbichler
1b1110581a manifest: revert canonicalization to old behaviour 
JSON keys MUST be quoted. this is a one-time break in signature
validation for backups created with the broken canonicalization code.
QEMU backups are not affected, as libproxmox-backup-qemu never linked
the broken versions.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 2020-07-20 11:37:53 +02:00
Dominik Csapak
ac5e9e770b catalog_shell: add exit command 
it is nice to have a command to exit from the shell instead of
only allowing ctrl+d or ctrl+c

the api method is just for documentation/help purposes and does nothing
by itself, the real logic is directly in the read loop

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-15 12:19:57 +02:00
Dietmar Maurer
f58233a73a src/backup/data_blob_reader.rs: avoid unwrap() - return error instead 2020-07-10 11:28:19 +02:00
Dietmar Maurer
62593aba1e src/backup/manifest.rs: fix signature (exclude 'signature' property) 2020-07-10 10:36:45 +02:00
Stoiko Ivanov
c687da9e8e datastore: chown base dir on creation 
When creating a new datastore the basedir is only owned by the backup
user if it did not exist beforehand (create_path chowns only if it
creates the directory), and returns false if it did not create the
directory).

This improves the experience when adding a new datastore on a fresh
disk or existing directory (not owned by backup) - backups/pulls can
be run instead of terminating with EPERM.

Tested on my local testinstall with a new disk, and a existing directory:

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
 2020-07-09 18:20:16 +02:00
Wolfgang Bumiller
20a4e4e252 minor optimization to 'to_canonical_json' 
* don't clone hash keys, just use references
* we don't need a String, stick to Vec<u8> and use
  serde_json::to_writer to avoid a temporary strings
  altogether

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-09 13:32:11 +02:00
Dietmar Maurer
4459ffe30e src/backup/manifest.rs: add default toömake it compatible with older backus 2020-07-09 13:25:38 +02:00
Dietmar Maurer
dfa517ad6c src/backup/manifest.rs: rename into_string -> to_string 
And do not consume self.
 2020-07-09 11:28:05 +02:00
Dietmar Maurer
3dacedce71 src/backup/manifest.rs: use serde_json::from_value() to deserialize data 
Also modified from_data compute signature ditectly from json.
 2020-07-09 09:50:28 +02:00
Dietmar Maurer
b53f637914 src/backup/manifest.rs: cleanup signature generation 2020-07-09 09:20:49 +02:00
Dietmar Maurer
2107a5aebc src/backup/manifest.rs: include signature inside the manifest 
This is more flexible, because we can choose what fileds we want to sign.
 2020-07-08 16:23:26 +02:00
Dietmar Maurer
3638341aa4 src/backup/file_formats.rs: remove signed chunks 
We can include signature in the manifest instead (patch will follow).
 2020-07-08 16:23:26 +02:00
Wolfgang Bumiller
0351f23ba4 client: introduce --keyfd parameter 
This is a more convenient way to pass along the key when
creating encrypted backups of unprivileged containers in PVE
where the unprivileged user namespace cannot access
`/etc/pve/priv`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 13:56:38 +02:00
Dietmar Maurer
c1ff544eff src/backup/crypt_config.rs - compute_digest: make it more secure 2020-07-08 12:53:04 +02:00
Wolfgang Bumiller
b65390ebc9 client: xdg usage: place() vs find() 
place() is used when creating a file, as it will create
intermediate directories, only use it when actually placing
a new file.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 10:57:28 +02:00
Dietmar Maurer
3bad3e6e52 src/client/backup_writer.rs - upload_stream: add crypt_mode 2020-07-08 10:43:28 +02:00
Wolfgang Bumiller
521a0acb2e DataStore::load_manifest: also return CryptMode 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 09:19:53 +02:00
Wolfgang Bumiller
3b66040de6 add DataBlob::crypt_mode 
and move use statements up

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 09:19:53 +02:00
Wolfgang Bumiller
af3a0ae7b1 remove CryptMode::sign_only special method 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 09:19:53 +02:00
Dietmar Maurer
4e36f78438 src/backup/manifest.rs: support old encrypted property 
Just to avoid confusion.
 2020-07-08 08:52:27 +02:00
Wolfgang Bumiller
f28d9088ed introduce a CryptMode enum 
This also replaces the recently introduced --encryption
parameter on the client with a --crypt-mode parameter.

This can be "none", "encrypt" or "sign-only".

Note that this introduces various changes in the API types
which previously did not take the above distinction into
account properly:

Both `BackupContent` and the manifest's `FileInfo`:
    lose `encryption: Option<bool>`
    gain `crypt_mode: Option<CryptMode>`

Within the backup manifest itself, the "crypt-mode" property
will always be set.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-07 15:24:19 +02:00
Wolfgang Bumiller
e13c4f66bb minor style & whitespace fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-06 10:55:25 +02:00
Dietmar Maurer
e9764238df make ReadChunk not require mutable self. 
That way we can reduce lock contentions because we lock for much shorter
times.
 2020-07-03 07:37:29 +02:00
Dietmar Maurer
2e079b8bf2 partially revert commit 1f82f9b7b5 
do it backwards compatible. Also, code was wrong because FixedIndexWriter
still computed old style csums...
 2020-06-29 12:44:45 +02:00
Dietmar Maurer
817bcda848 src/backup/verify.rs: do not stop on server shutdown 
This is a read-only task, so there is no need to stop.
 2020-06-26 09:45:59 +02:00
Dietmar Maurer
1f82f9b7b5 src/backup/index.rs: add compute_csum 
And use it for fixed and dynamic index. Please note that this
changes checksums for fixed indexes, so restore older backups
will fails now (not backward compatible).
 2020-06-26 09:00:34 +02:00
Dietmar Maurer
fdaab0df4e src/backup/index.rs: add chunk_info method 2020-06-26 08:14:45 +02:00
Dietmar Maurer
b957aa81bd update backup api for incremental backup 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-06-26 07:17:08 +02:00
Dietmar Maurer
8ea00f6e49 allow to abort verify jobs 
And improve job description rendering on gui.
 2020-06-25 12:56:36 +02:00
Dietmar Maurer
c2009e5309 src/api2/admin/datastore.rs: add verify api 2020-06-24 13:35:21 +02:00
Dietmar Maurer
23f74c190e src/backup/backup_info.rs: impl Display for BackupGroup 2020-06-24 13:35:21 +02:00
Wolfgang Bumiller
a6f8728339 update to pxar 0.1.9, update ReadAt implementations 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-24 11:57:12 +02:00
Stefan Reiter
facd9801cf add incremental backup support 
To support incremental backups (where not all chunks are sent to the
server), a new parameter "reuse-csum" is introduced on the
"create_fixed_index" API call. When set and equal to last backups'
checksum, the backup writer clones the data from the last index of this
archive file, and only updates chunks it actually receives.

In incremental mode some checks usually done on closing an index cannot
be made, since they would be inaccurate.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-06-24 10:01:25 +02:00
Dominik Csapak
05d18b907a add From<&DirEntryAttribute to CatalogEntryType and make it pub(crate) 
we want to get a string representation of the DirEntryAttribute
like 'f' for file, etc. and since we have such a mapping already
in the CatalogEntryType, use that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:08:50 +02:00
Dominik Csapak
e44fe0c9f5 derive Clone for the LocalChunkReader 
this will be necessary for accessing local pxar behind didx files

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:07:28 +02:00
Dominik Csapak
4cf0ced950 add LocalDynamicReadAt 
mostly copied from BufferedDynamicReadAt from proxmox-backup-client
but the reader is wrapped in an Arc in addition to the Mutex

we will use this for local access to a pxar behind a didx file

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:05:31 +02:00
Dietmar Maurer
60f9a6ea8f src/backup/datastore.rs: add new helpers to load blobs and verify chunks 2020-06-24 06:58:14 +02:00
Dietmar Maurer
1090fd4424 src/backup/data_blob.rs: cleanup - improve code reuse 2020-06-24 06:56:48 +02:00
Dietmar Maurer
92c3fd2e22 src/backup/chunk_store.rs: allow to read name() 
This is helpful for logging ...
 2020-06-24 06:54:21 +02:00
Dietmar Maurer
d6d3b353be cleanup: implement FromStr for BackupGroup 2020-06-23 08:16:56 +02:00
Dietmar Maurer
a67f7d0a07 cleanup: implement FromStr for BackupDir 2020-06-23 08:09:52 +02:00
Dominik Csapak
0b99e5aebc remove debug prints 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-23 06:33:58 +02:00
Dominik Csapak
f386f512d0 add AsyncReaderStream 
and replace AsyncIndexReader's stream implementation with that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-23 06:33:31 +02:00
Dominik Csapak
eeaa2c212b impl Sync for DataBlobReader 
this is safe for the reason explained in the comment

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 08:37:47 +02:00
Dominik Csapak
4a3adc3de8 add AsyncIndexReader 
implements AsyncRead as well as Stream for an IndexFile and a store
that implements AsyncReadChunk

we can use this to asyncread or stream the content of a FixedIndex or
DynamicIndex

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 08:32:33 +02:00
Dominik Csapak
abdb976340 add Display trait to BackupDir 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 08:28:35 +02:00
Dominik Csapak
3b62116ce6 implement AsyncReadChunk for LocalChunkReader 
same as the sync ReadChunk but uses tokio::fs::read instead
of file_get_contents

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 07:54:23 +02:00
Dominik Csapak
e181d2f6da add encrypted info to Manifest 
we want to save if a file of a backup is encrypted, so that we can
* show that info on the gui
* can later decide if we need to decrypt the backup

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 07:35:39 +02:00
Dominik Csapak
bde8e243cf remove unsafe copy code 
copy_nonoverlapping is basically a memcpy which can also be done
via copy_from_slice which is not unsafe
(copy_from_slice uses copy_nonoverlapping internally)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 06:56:15 +02:00
Wolfgang Bumiller
bb59df9134 catalog: don't panic on invalid file mtimes 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 11:25:54 +02:00
Wolfgang Bumiller
4264e52220 reuse some extractor code in catalog shell 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 10:54:54 +02:00
Wolfgang Bumiller
6988b29bdc use O_EXCL when creating files during extraction 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 10:33:27 +02:00
Wolfgang Bumiller
d30c192589 AsyncReadChunk: require Send 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 09:50:29 +02:00
Wolfgang Bumiller
7a6b549270 dynamic index: make it hard to mess up endianess 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-15 09:19:35 +02:00
Wolfgang Bumiller
57e50fb906 use new Mmap helper for dynamic index 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 13:57:56 +02:00
Wolfgang Bumiller
4d16badf6f add an AsyncReadChunk trait 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 11:38:21 +02:00
Wolfgang Bumiller
1498659b4e cleanup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 10:59:34 +02:00
Dominik Csapak
e693818afc refactor time functions to tools 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 13:31:10 +02:00
Wolfgang Bumiller
26e78a2efb downgrade some FIXMEs to TODOs 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 11:09:23 +02:00
Wolfgang Bumiller
5444fa940b turn pxar::flags into bitflags, pxar::Flags 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 11:05:53 +02:00
Wolfgang Bumiller
fab2413741 catalog: remove unused SenderWriter 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 10:42:42 +02:00
Wolfgang Bumiller
c443f58b09 switch to external pxar and fuse crates 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-08 13:56:58 +02:00
Dominik Csapak
a95a3fb893 fix csum calculation of not 'chunk_size' aligned images 
the last chunk does not have to be as big as the chunk_size,
just use the already available 'chunk_end' function which does the
correct thing

this fixes restoration of images whose sizes are not a multiple of
'chunk_size' as well

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-04 10:18:30 +02:00
Thomas Lamprecht
55919bf141 verify_file: add missing closing parenthesis in error message 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-06-03 19:10:01 +02:00
Dietmar Maurer
4e14781aec fix typo 2020-06-03 06:59:43 +02:00
Thomas Lamprecht
add5861e8d typo fixes all over the place 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-05-30 16:39:08 +02:00
Dietmar Maurer
1610c45a86 src/client/pull.rs: also download client.log.blob 2020-05-30 14:51:33 +02:00
Dietmar Maurer
96d65fbcd0 cleanup: define/use const for predefined blob file names. 2020-05-30 14:04:15 +02:00
Dietmar Maurer
8545480a31 src/bin/proxmox-backup-proxy.rs: add simple task scheduler for garbage collection 2020-05-20 08:59:45 +02:00
Dietmar Maurer
07ce44a633 avoid compiler warnings 2020-05-19 07:03:41 +02:00
Dietmar Maurer
7b22acd0c2 src/config/network.rs: make it compatible with pve 
and depend on proxmox 0.1.26
 2020-05-07 09:28:25 +02:00
Dietmar Maurer
99641a6bbb garbage_collect: call fail_on_abort to abort GV when requested. 2020-05-05 09:06:34 +02:00
Dietmar Maurer
54552dda59 implemnt backup ownership, improve datastore access permissions 2020-04-28 10:22:25 +02:00
Wolfgang Bumiller
f7d4e4b506 switch from failure to anyhow 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-04-17 18:43:30 +02:00
Dietmar Maurer
cf459b1982 gc: log pending removals 2020-04-06 09:50:40 +02:00
Christian Ebner
32d192a952 catalog: shell: Use the new logic including resolving symlinks for catalog 
and remove the old unused code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:51:01 +01:00
Christian Ebner
fee5528e59 catalog: shell: introduce new CatalogPathStack to navigate in catalog shell. 
This is basically a rewrite of the current logic for navigating the catalog,
but in addition allows to follow symlinks.
Following symlinks introduces the issue that generation of canonical paths
(needed in the actual pxar archive) is more complex, as symlinks have to be
resolved and loops avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:50:43 +01:00
Christian Ebner
4145c36749 catalog: add missing function documentation. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:50:08 +01:00
Christian Ebner
c2f9149461 catalog: introduce is_symlink() to check if DirEntry is a symlink. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:49:55 +01:00
Christian Ebner
11ee5c0563 catalog: derive PartEq for equality checks of two DirEntry's. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:49:39 +01:00
Christian Ebner
536683e73b src/backup/dynamic_index.rs: Add LruCache for chunks. 
In order to improve non-sequential reads of chunks as e.g. in FUSE.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:47:16 +01:00
Christian Ebner
35ddf0b419 catalog: shell: Introduce clear-selected command. 
'clear-selected' allows to clear all the match patterns from the list of
patterns for a subsequent restore.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:40:55 +01:00
Christian Ebner
8e464141cf catalog: shell: Improve list-selected command. 
'list-selected' now shows the filenames matching the patterns for a restore
instead of the patterns themselfs.
The patterns can be displayed by passing the '--pattern' flag.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:40:42 +01:00
Christian Ebner
03f779c6f5 catalog: shell: Improve output of stat command. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-27 06:53:17 +01:00
Dietmar Maurer
8ce49a76da src/backup/backup_info.rs: fix SNAPSHOT_PATH_REGEX and GROUP_PATH_REGEX 2020-02-18 13:16:35 +01:00
Christian Ebner
ba050e3788 catalog: shell: Allow two or more successive slashes in path. 
Two or more successive slashes should be allowed and treated as a single slash.
We also do not treat two successive slashes at the beginning of a path any
different.

Details are found here:
https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap04.html#tag_04_11

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-05 09:32:29 +01:00
Dietmar Maurer
501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Dietmar Maurer
d08bc483db use proxmox 0.1.9 with new cli command helpers 2020-01-27 18:08:43 +01:00
Dietmar Maurer
a92830dc39 src/api2/types.rs: define and use api type GarbageCollectionStatus 2020-01-23 13:40:12 +01:00
Dietmar Maurer
51534c8de9 src/backup/manifest.rs: check if manifest contains files 2020-01-23 11:16:12 +01:00
Dietmar Maurer
6abce6c2bb src/backup/datastore.rs: remove_backup_(group/dir) - return Error instead of io::Error 2020-01-23 10:14:46 +01:00
Dietmar Maurer
8a1d68c8b9 src/backup/datastore.rs: improve error messages 2020-01-23 09:58:14 +01:00
Dietmar Maurer
41b373eced src/backup/datastore.rs: new helpers 
last_successful_backup: Returns the time of the last successful backup
group_path: Returns the absolute path for a backup_group
snapshot_path: Returns the absolute path for a backup_dir
 2020-01-22 15:05:47 +01:00
Wolfgang Bumiller
9ea4bce444 bump proxmox crate to 0.1.7 
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).

When we need to depend on "parts" of the crate later on
we'll just have to use features.

The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-21 13:48:37 +01:00
Wolfgang Bumiller
d973aa827c introduce new runtime tokio helpers 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-20 13:12:40 +01:00
Dietmar Maurer
4b4eba0b9e src/api2/pull.rs: implement delete flag for vanished groups 2020-01-17 11:24:55 +01:00
Dietmar Maurer
11d89239c3 src/backup/backup_info.rs: new list_groups helper 2020-01-17 10:42:03 +01:00
Dietmar Maurer
d2dd827877 src/backup/catalog.rs - SenderWriter: use tokio::task::block_in_place 
Make sure we do not block the executor.
 2020-01-16 14:24:15 +01:00
Dietmar Maurer
d0187a51a9 src/section_config.rs - convert_to_array: optionally add digest 
datastore::config() -> also return digest
remotes::config() -> also return digest
 2020-01-14 12:57:03 +01:00
Oguz Bektas
6d20a29d73 fix typo for function name load_and_decrypt_key 
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
 2020-01-13 11:14:47 +01:00
Christian Ebner
90fc97af6a pxar::decoder::Decoder: include xattrs and payload size in DirectoryEntry. 
By reading and including xattrs and payload size in struct `DirectoryEntry`,
the tuple of return types is avoided and the code is simpler.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-01-10 13:45:40 +01:00
Dietmar Maurer
8f14e8fe4c src/backup/backup_info.rs: implement last_successful_backup() 
A backup is considered successful if there is a manifest file.
 2020-01-06 11:35:22 +01:00
Dietmar Maurer
247a8ca5bb src/backup/manifest.rs: impl TryFrom DataBlob 2020-01-05 16:28:18 +01:00
Dietmar Maurer
7759eef552 src/backup/datastore.rs: implement cleanup_backup_dir() 2020-01-05 15:15:12 +01:00
Dietmar Maurer
3758b398cd src/backup/read_chunk.rs: add read_raw_chunk() 
Allow to read chunk data without decoding.
 2020-01-02 13:29:10 +01:00
Dietmar Maurer
2585a8a4e2 src/backup/chunk_store.rs: implement cond_touch_chunk() 
This will be used by backup sync to test if a chunk already exists.
 2020-01-02 13:26:28 +01:00
Dietmar Maurer
1cf5178ac5 src/backup/datastore.rs: new helper try_shared_chunk_store_lock() 2020-01-02 11:00:33 +01:00
Dietmar Maurer
1e8da0a789 src/backup/manifest.rs: new helper archive_type() 2019-12-31 15:23:41 +01:00
Dietmar Maurer
645995634a src/api2/config/datastore.rs - create: pass uid and gid instead of User 2019-12-20 09:23:58 +01:00
Dietmar Maurer
e67770d496 src/backup/chunk_store.rs - create: pass User instead of CreateOptions 2019-12-20 09:11:40 +01:00
Wolfgang Bumiller
afdcfb5bc9 let ChunkStore::create take CreateOptions 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-12-19 13:14:49 +01:00
Dietmar Maurer
f74a03da1f remove tools::getpwnam_ugid, impl. crate::backup::backup_user() 
And use new nix::unistd::User struct.
 2019-12-19 10:20:13 +01:00
Dietmar Maurer
868c585219 src/backup/datastore.rs: avoid divide by zero 2019-12-19 07:14:23 +01:00
Dietmar Maurer
7e210bd0b4 src/backup/chunk_store.rs: create lock file with correct owner 2019-12-19 06:55:53 +01:00
Dietmar Maurer
0b97bc6158 src/backup/chunk_store.rs: use proxmox::tools::fs::create_path 2019-12-18 12:26:43 +01:00
Wolfgang Bumiller
feaa1ad35f replace file_set_contents with replace_file 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-12-18 11:16:04 +01:00
Oguz Bektas
14f1e63067 chunk_store: create parent directories 
'datastore create storename /path/to/dir/that/may/not/exist' should
work.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
 2019-12-17 15:39:42 +01:00
Christian Ebner
38d9a69875 catalog: decoder and shell: fix incorrect logic in find matching 
The find matching was incorrectly performed starting from the parent directroy
and not as intended from the entries of the parent directory.

Further, the match pattern passed from the catalog shell contains the absolute
path of the search entry point as prefix, so find() must always start from the
archive root. This is because the match pattern has to be stored in the selected
list for a subsequent restore-selected command in the shell.
All matching paths are shown as absolute paths with all contents in the subdir,
equal to what would be restored by the given pattern.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-12-17 14:57:34 +01:00
Christian Ebner
25cdd0e0a1 catalog_shell: impl find for catalog shell. 
Implements the find command which allows to find and select files for subsequent
restore.
Files selected for restore are now stored in a Vec instead of a HashSet.
This is needed, since instead of the full paths for each file, selected files are
now identified by a list of match pattern, where ordering matters.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-12-17 11:12:34 +01:00
Christian Ebner
90dfd0a71d catalog: impl find() for file search by match pattern. 
find() iterates over the file tree and matches each node against a list of match
patterns provided at function call.
For each matching node, a callback function with the current directroy stack is
called.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-12-17 11:12:34 +01:00
Dietmar Maurer
236a396aa1 src/api2/admin/datastore.rs - prune: log retention options 2019-12-14 16:32:16 +01:00
Wolfgang Bumiller
db0cb9ce0b update a chunk of stuff to the hyper release 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-12-13 11:24:41 +01:00
Christian Ebner
f084505ec5 src/backup/catalog_shell.rs: sort output of list-selected 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-12-11 10:54:20 +01:00
Dietmar Maurer
73e57f244e src/backup/datastore.rs - garbage-collection: fix average chunk size 2019-12-10 11:25:29 +01:00
Dietmar Maurer
48ef3c3327 cli: avoid useless .into() 
This needs latest proxmox-api updates.
 2019-12-09 17:40:34 +01:00
Dietmar Maurer
102d8d4136 src/backup/prune.rs: implement --keep-hourly 2019-12-07 11:23:33 +01:00
Dietmar Maurer
052c4aa160 src/backup/prune.rs: prune - correctly use iso_week year 2019-12-06 19:41:38 +01:00
Dietmar Maurer
9e3f008804 src/backup/prune.rs: add new helper keeps_something() 2019-12-06 12:28:31 +01:00
Dietmar Maurer
55c3cb69cd rename catalog_shell_api() into catalog_shell_cli() 2019-12-06 09:42:05 +01:00
Dietmar Maurer
ecbaa38fa4 src/backup/catalog_shell.rs: factor out catalog_shell_api() 
In order to extract documentation ...
 2019-12-06 09:35:47 +01:00
Christian Ebner
cca67dc0c3 src/backup/readline.rs: removed in favor of new readline impl 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-12-06 09:03:04 +01:00
Christian Ebner
6934c6fe77 src/backup/catalog_shell.rs: adapt to use API Schema definition and rustyline 
This major refactoring of the catalog based shell utilizes the new API macro and
the API Schema as well as rustyline instead of the old GNU readline C API.

The code now has these 3 main components:
 * The `Shell` which handles the readline loop via rustyline.
 * The shell functions defined via the API macro.
 * The `Context` which holds catalog and decoder instances.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-12-06 09:02:44 +01:00
Dietmar Maurer
9b78352188 src/backup/prune.rs: define new struct PruneOptions 2019-12-06 08:56:27 +01:00
Dietmar Maurer
92acbd69f7 src/backup/backup_info.rs: remove old prune code 2019-12-06 08:29:27 +01:00
Dietmar Maurer
408434360b src/backup/prune.rs: add PruneMark::KeepPartial 
Use separate mark for partial backup - dont mess them up with regular ones.
 2019-12-06 08:15:32 +01:00
Dietmar Maurer
a8c8366cfa src/backup/prune.rs: factor out code into remove_incomplete_snapshots() 2019-12-06 08:12:08 +01:00
Dietmar Maurer
dc18849156 src/backup/prune.rs: moved prune related code into extra file 2019-12-06 08:06:21 +01:00
Dietmar Maurer
6f47dd8a0f src/backup/backup_info.rs - prune: improve algorythm 2019-12-05 18:41:28 +01:00
Dietmar Maurer
8f0b4c1f90 src/api2/admin/datastore.rs - prune: log info about all snapshots 2019-12-05 15:35:11 +01:00
Dietmar Maurer
9ce42759ec src/backup/backup_info.rs - compute_prune_list: remove unfinished backups 2019-12-05 11:18:10 +01:00
Dietmar Maurer
2c034f8d0a src/backup/backup_info.rs: improve prune algorithm 2019-12-05 08:55:19 +01:00
Dietmar Maurer
aeeac29bb1 src/backup/backup_info.rs: new compute_prune_list helper 
So that we can write regression tests for this.
 2019-12-04 15:49:11 +01:00
Christian Ebner
314bb35868 catalog_shell: fix several clippy warnings 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-27 10:13:50 +01:00
Christian Ebner
951cf17ee3 catalog_shell: major refactoring of cli command definition and parsing 
By changing the way shell commands are defined and parsed, this makes it more
straight forward to extend the current functionality.
The readline input is parsed based on the provided command definition and the
given parameters and options are passed to a command specific callback function.
In addition, the provided command definition including its description is used
to generate a help string to display.
The help command shows a list of all supported commands or the help string for
the provided command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-26 13:01:00 +01:00
Christian Ebner
59bc6ad676 catalog_shell: check and return on empty list in order to avoid division by zero 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-26 12:55:33 +01:00
Christian Ebner
4e56b2f792 catalog_shell: do not restore full archive if no entries are selected for restore-selected 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-26 12:55:18 +01:00
Dietmar Maurer
99b5b6cba9 src/pxar/decoder.rs: use trait object as reader 
So that the Decoder is no longer genertic.
 2019-11-26 10:45:11 +01:00
Dietmar Maurer
f701d0335e src/pxar/sequential_decoder.rs: remove callback from new() 
And use an extra functzion  set_callback() to configure that.

Also rewrite pxar/fuse.rs and implement a generic Session (will get
further cleanups with next patches).
 2019-11-26 09:56:48 +01:00
Christian Ebner
f14c96ea38 src/backup/catalog_shell.rs: impl shell to inspect and restore a snapshot via the catalog 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-22 07:54:04 +01:00
Christian Ebner
38446a9551 src/backup/readline.rs: impl wrapper for GNU readline 
In order to provide the context needed for tab completion via the readline
callback, the needed mut ref is passed via a thread local storage key.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-22 07:46:24 +01:00
Christian Ebner
6dba015043 src/backup/catalog.rs: derive Clone for DirEntry and DirEntryAttribute 
This is needed in order to explicitly clone the values when needed in the
catalog shell implementation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-22 07:15:39 +01:00
Christian Ebner
c9f002213e src/backup/catalog.rs: impl DirEntry::is_directory() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-22 07:14:00 +01:00
Wolfgang Bumiller
f569acc5e2 drop uuid crate dependency 
proxmox::tools now has a Uuid module using the native
libuuid.

Adds build dependency: libuuid1 (which is a Pre-Depends of
util-linux, so always installed anyway).

Drops uuid + 16 more crate dependencies.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-11-14 10:22:29 +01:00
Christian Ebner
b423958d8e catalog: fix issue with DirEntry start 
start has to be the parent start - offset

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-12 17:18:13 +01:00
Dietmar Maurer
780dd2b0a1 src/backup/catalog.rs: add some docs 2019-11-12 13:23:40 +01:00
Dietmar Maurer
6aa906b547 src/backup/catalog.rs - CatalogReader: verify magic number 2019-11-12 12:54:06 +01:00
Dietmar Maurer
2ec208aef5 src/backup/catalog.rs - CatalogReader::dump(): use root() 2019-11-12 12:47:21 +01:00
Dietmar Maurer
8f24a9ea18 src/backup/catalog.rs - Direntry::parse: abort if callback return false 2019-11-12 11:41:44 +01:00
Dietmar Maurer
7d017123fd src/backup/catalog.rs: impl read_dir() and lookup(), refactor common code 2019-11-12 11:41:44 +01:00
Dietmar Maurer
dc9596de45 src/backup/catalog.rs: move CatalogEntryType from src/pxar/catalog.rs 2019-11-12 11:41:44 +01:00
Dietmar Maurer
55c0b3cc7c src/backup/catalog.rs - DirEntry: factor out name attribute 
And avoid allocations inside parse().
 2019-11-12 11:41:44 +01:00
Christian Ebner
3f1c5b5e65 catalog: impl std::fmt::Display trait for CatalogEntryType 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-12 07:00:08 +01:00
Dietmar Maurer
f89359c2f3 src/backup/catalog.rs: clenup - use single impl block 2019-11-10 11:38:31 +01:00
Dietmar Maurer
e8d8fdc61d src/backup/catalog.rs: remove unused code 2019-11-10 11:37:06 +01:00
Dietmar Maurer
c74c074bfd src/backup/catalog.rs: write MAGIC at start of file 2019-11-10 11:36:45 +01:00
Dietmar Maurer
265664517a src/backup/catalog.rs: new parse() helper 2019-11-09 14:37:33 +01:00
Dietmar Maurer
5d92935e03 src/backup/catalog.rs - dump_dir: make sure we parsed the whole data block 2019-11-09 13:21:20 +01:00
Dietmar Maurer
12a1975af9 src/backup/catalog.rs: store u64 compressed 
This way catalogs get about 1/3 smaller.
 2019-11-09 13:07:47 +01:00
Dietmar Maurer
89245fb530 renamed: src/backup/catalog_blob.rs -> src/backup/catalog.rs 2019-11-08 10:41:00 +01:00
Dietmar Maurer
bf6e321744 change catalog format, use dynamic index to store catalog. 
In order to remove size restriction of a single blob.
 2019-11-08 10:35:48 +01:00
Christian Ebner
b82a101c86 catalog: store current writer position in CatalogBlobWriter 
Counts the bytes written by the CatalogBlobWriter in order to obtain the
stream position, needed to get offset to reference catalog items.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-31 09:11:51 +01:00
Dietmar Maurer
62ee2eb405 avoid some clippy warnings 2019-10-26 11:42:05 +02:00
Dietmar Maurer
834a2f95a0 avoid some clippy warnings 2019-10-25 18:44:51 +02:00
Dietmar Maurer
9fe2f639b8 allow(clippy::cast_ptr_alignment) 2019-10-25 18:03:55 +02:00
Dietmar Maurer
cd69d36b45 src/backup/dynamic_index.rs: use write_all() 2019-10-25 17:22:41 +02:00
Dietmar Maurer
649c5ee7b1 load_and_decrtypt_key: allow function closures to readf passphrase 2019-10-16 13:23:52 +02:00
Dietmar Maurer
60ffa9649c src/backup/manifest.rs: fix property names (use '-' instead of '_') 2019-10-14 12:28:27 +02:00
Dietmar Maurer
4c9f753c56 src/backup/data_blob.rs: define create_zero_chunk as member function 
and rename it to build_zero_chunk()
 2019-10-14 10:58:26 +02:00
Dietmar Maurer
2745151d93 src/backup/data_blob.rs: add create_zero_chunk() helper 2019-10-14 10:44:46 +02:00
Dietmar Maurer
f06b820ac0 src/backup/manifest.rs: add verify_file 2019-10-13 10:09:12 +02:00
Dietmar Maurer
511a47bd73 cleanup 2019-10-13 09:39:21 +02:00
Dietmar Maurer
e17d5d864d src/backup/manifest.rs: improve parser (try_from) 2019-10-13 08:39:49 +02:00
Dietmar Maurer
59e9ba01c6 src/backup/manifest.rs: new class to generate/parse index.json 2019-10-12 17:58:08 +02:00
Dietmar Maurer
7123ff7d43 src/backup/data_blob.rs: avoid Arc<CryptConfig> 
use simple reference instead.
 2019-10-07 11:41:35 +02:00
Dietmar Maurer
4ee8f53d07 remove DataChunk file format - use DataBlob instead 2019-10-06 10:31:06 +02:00
Dietmar Maurer
9e603e254c src/backup/fixed_index.rs: do not use offset for checksum 2019-09-23 11:30:20 +02:00
Dietmar Maurer
2aba16bde1 src/backup/data_chunk.rs: add into_raw() 2019-09-19 14:24:17 +02:00
Wolfgang Bumiller
ba5e67475a clippy: fix access to unaligned buffer in dynamic reader 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:50:45 +02:00
Wolfgang Bumiller
b526bd1408 backup: cleanup unused imports 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:46:46 +02:00
Wolfgang Bumiller
4dc79bb1d9 clippy: fix access to unaligned buffer 
via the newly introduced ReadExt::read_host_value_boxed()
from proxmox::tools::io.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:45:44 +02:00
Wolfgang Bumiller
ccdf3ad1ec backup: use static assertion in fixed index reader 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:43:25 +02:00
Wolfgang Bumiller
b21851cc5c clippy: collapse identical if branches 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 13:57:58 +02:00
Wolfgang Bumiller
653b1ca10e some clippy fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 12:14:05 +02:00
Dietmar Maurer
9d456cf8ec src/backup/catalog_blob.rs: use write_all() instead of write() 
To correctly handle interrupted system calls.
 2019-09-05 11:40:48 +02:00
Dietmar Maurer
0a51fe0011 src/backup/fixed_index.rs: new helper to compute checksum and file size 2019-09-03 13:11:45 +02:00
Dietmar Maurer
82c85a21a1 src/backup/dynamic_index.rs: new helper to compute checksum and file size 2019-09-03 13:11:09 +02:00
Wolfgang Bumiller
745e652a7f src/backup/index.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
32bef1e2d1 src/backup/chunk_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
5c1130df9f implement Sync for DynamicIndexReader and FixedIndexReader 
hyper's wrap_stream now needs this

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
d21f8a5b5b backup/dynamic_index: explicitly zero-initialize the header 
The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 11:42:35 +02:00
Wolfgang Bumiller
990b930f22 backup/dynamic_index: use static assertion, fix size 
The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 11:42:35 +02:00
Wolfgang Bumiller
7d83440c60 remove proxmox-protocol subcrate 
AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)

Moved the chunker back into src/backup/chunker.rs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 14:08:25 +02:00
Wolfgang Bumiller
5e58e1bb7d {dynamic,fixed}_index: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:35:36 +02:00
Wolfgang Bumiller
f93b55b057 src/backup/index: style fixup & unsafe copy removal 
We can use the safe .copy_from_slice alternative in this
case.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:31:45 +02:00
Wolfgang Bumiller
fa17b1ce2a src/backup/index: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:27:35 +02:00
Wolfgang Bumiller
367f002ee2 use statement cleanup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-16 12:28:40 +02:00
Dietmar Maurer
9d135fe617 src/backup/catalog_blob.rs: moved catalog impl. from pxar 
And avoid loading catalog into memory.
 2019-08-16 12:27:17 +02:00
Wolfgang Bumiller
9025312aa6 avoid lifetimes in blob reader/writer 2019-08-16 10:07:24 +02:00
Dietmar Maurer
71d08e00b7 src/backup/data_blob_reader.rs: fix compiler warning 2019-08-16 08:10:39 +02:00
Dietmar Maurer
b791804f4b src/backup/data_blob_reader.rs: impl for Read instead of BufRead 2019-08-14 14:56:01 +02:00
Dietmar Maurer
39a4df61d6 improve docs 2019-08-14 14:08:27 +02:00
Dietmar Maurer
018d11bb84 src/backup/data_blob.rs: move parts into single files 2019-08-14 13:24:41 +02:00
Dietmar Maurer
e27c5a559f src/backup/data_blob.rs: avoid excessive stack usage 2019-08-14 12:44:08 +02:00
Dietmar Maurer
c638542b4b src/backup/file_formats.rs: moved header_size 2019-08-14 12:35:53 +02:00
Dietmar Maurer
6082216343 src/backup/data_blob.rs - CryptReader/CryptWriter: fix buffer size handling 2019-08-13 15:23:15 +02:00
Dietmar Maurer
548c9489d7 src/backup/data_blob.rs - DataBlobReader: impl compressed, encrypted blobs 2019-08-13 13:17:08 +02:00
Dietmar Maurer
2aa0bfff59 src/backup/data_blob.rs - DataBlobReader: implement reader for encrtypted blobs 2019-08-13 13:17:08 +02:00
Dietmar Maurer
e9a385a78e src/backup/data_blob.rs - DataBlobReader: impl compressed, signed blobs 2019-08-12 17:47:25 +02:00
Dietmar Maurer
4bfa147eaf src/backup/data_blob.rs - DataBlobReader: impl. reader for signed blobs 2019-08-12 17:41:25 +02:00
Dietmar Maurer
09785b2795 src/backup/data_blob.rs - DataBlobReader: start serious impl. 2019-08-12 17:20:41 +02:00
Dietmar Maurer
a32bd8a516 src/backup/data_blob.rs - DataBlobWriter: fix decryption 2019-08-12 11:57:29 +02:00
Dietmar Maurer
5d15cb49b4 src/backup/data_blob.rs - compute_crc: start after blob header 2019-08-12 11:32:26 +02:00
Dietmar Maurer
1cbdfd19a2 src/backup/file_formats.rs; use const instzead of static 2019-08-12 11:20:21 +02:00
Dietmar Maurer
18be4ec24a src/backup/data_blob.rs - DataBlobWriter: fix magic number for uncompressed blobs 2019-08-12 10:46:05 +02:00
Dietmar Maurer
5622a3fcdb src/backup/data_blob.rs - DataBlobWriter: impl. compressed encrypted blobs 2019-08-12 10:16:42 +02:00
Dietmar Maurer
f4942e9ffd src/backup/data_blob.rs - DataBlobWriter: impl. encrypted blobs 2019-08-12 10:08:00 +02:00
Dietmar Maurer
c57ec43a53 src/backup/crypt_config.rs: new helper data_crypter to get openssl Crypter 2019-08-12 10:06:51 +02:00
Dietmar Maurer
7776becf5d src/backup/data_blob.rs - DataBlobWriter: impl. compressed signed blobs 2019-08-12 09:01:08 +02:00
Dietmar Maurer
706638f803 src/backup/data_blob.rs: impl. ChecksumWriter 
To correctly compute crc,hmac of final data.
 2019-08-12 08:53:53 +02:00
Dietmar Maurer
f796351c54 src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs 2019-08-12 07:34:41 +02:00
Dietmar Maurer
cb0eea29d9 src/backup/crypt_config.rs: new method to get hmac signer 2019-08-12 07:33:15 +02:00
Dietmar Maurer
a762ce54fd src/backup/data_blob.rs: make DataBlobWriter more generic 
Allow to write compressed and uncompressed blobs.
 2019-08-11 12:14:04 +02:00
Dietmar Maurer
1f26fdef19 src/backup/data_blob.rs: impl. simple reader/writer 
To avoid loading blob into memory.
 2019-08-11 11:32:36 +02:00
Dietmar Maurer
781ac11c6a src/backup/data_blob.rs: define const MAX_BLOB_SIZE 2019-08-09 11:49:06 +02:00
Dietmar Maurer
863be2e6e2 src/backup/data_blob.rs: allow largert blob sizes (128MB) 
Need this for catalog files.
 2019-08-09 10:22:56 +02:00
Dietmar Maurer
eecb23560b src/backup/data_blob.rs - encode: always compute crc 2019-08-07 08:30:27 +02:00
Dietmar Maurer
38a6cddad7 src/backup/backup_info.rs: add BackupInfo::new constructor 2019-08-07 08:27:52 +02:00
Dietmar Maurer
f889b15867 src/backup/data_blob.rs: always compute crc 2019-08-06 11:42:14 +02:00
Dietmar Maurer
e18a6c9ee5 update to nix 0.14, use code from proxmox:tools 2019-08-03 13:05:38 +02:00
Dietmar Maurer
69ecd8d5b0 src/backup/data_blob.rs: implement signed blobs 2019-08-02 09:56:01 +02:00
Dietmar Maurer
93205f942a src/backup/crypt_config.rs: new compute_auth_tag helper 2019-08-02 08:55:37 +02:00
Dietmar Maurer
c68d2170d5 src/backup/crypt_config.rs: fix typo 2019-08-02 08:29:40 +02:00
Dietmar Maurer
fa5d6977dd Revert previous, commit, use UTC RFC3339 without timezone (Z) 
We now have human readable data/time, and names are still sortable.
 2019-07-22 11:49:30 +02:00
Dietmar Maurer
10c2a21cfe src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339 
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
 2019-07-22 07:57:03 +02:00
Dietmar Maurer
f4bf7dfcc7 src/client/remote_chunk_reader.rs: implement simple caching 2019-07-05 10:42:46 +02:00
Dietmar Maurer
afb4cd28be src/backup/fixed_index.rs: implement BufferedFixedReader 2019-07-05 09:19:56 +02:00
Dietmar Maurer
b46c3fad03 src/backup/fixed_index.rs: use correct size 
We need to consider that the last chunk may have smaller size.
 2019-07-04 15:13:22 +02:00
Dietmar Maurer
5e04ec707d src/api2/backup/environment.rs: improve chunk alignment/size checks 2019-07-04 13:40:43 +02:00
Dietmar Maurer
a24e3993e0 src/backup/chunk_store.rs: coding style fixes 2019-07-04 11:39:10 +02:00
Dietmar Maurer
e4c2fbf170 src/backup/chunk_store.rs: additionally log chunk count 2019-07-04 11:27:11 +02:00
Dietmar Maurer
9850bcdf19 src/backup/chunk_store.rs: improve error reporting 2019-07-04 11:21:54 +02:00
Wolfgang Bumiller
a3f3e91da2 backup/chunk_store: rework chunk iterator 
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-04 10:55:17 +02:00
Dietmar Maurer
a57360983b src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item 2019-07-04 09:26:44 +02:00
Dietmar Maurer
0f0a35b390 src/backup/dynamic_index.rs: lock file inside new, code cleanup 2019-07-04 08:17:30 +02:00
Dietmar Maurer
a7c72ad9eb src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store 2019-07-04 08:13:16 +02:00
Dietmar Maurer
a660978c9a src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats 2019-07-04 07:57:43 +02:00
Dietmar Maurer
1e0784841c src/backup/read_chunk.rs: use &mut self 2019-07-03 14:25:18 +02:00
Dietmar Maurer
877dd11133 src/backup/{fixed, dynamic}_index.rs: improve error messages 2019-07-02 08:56:56 +02:00
Dietmar Maurer
5c593a4d56 src/backup/read_chunk.rs: use Arc for DataStore 2019-07-02 08:49:16 +02:00
Dietmar Maurer
b850673634 src/backup/read_chunk.rs: move read chunk trait into extra file 
And implement LocalChunkReader.
 2019-07-02 08:22:29 +02:00
Wolfgang Bumiller
f35197f449 replace tools::vec with proxmox::tools::vec 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 11:03:25 +02:00
Wolfgang Bumiller
9b2b627fe0 update remaining users of tools::io::ops 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:44:03 +02:00
Wolfgang Bumiller
5485b579a1 backup: use new proxmox::tools::io traits 
This replaces all `write_value` calls with `write_le_value`
calls!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:37:02 +02:00
Dietmar Maurer
d48a9955a5 src/backup/dynamic_index.rs: introduce ReadChunk trait 2019-06-28 16:35:00 +02:00
Dietmar Maurer
81a6ce6fde src/backup/chunk_store.rs: new method chunk_path() 
Returns the absolute path.
 2019-06-28 15:48:09 +02:00
Dietmar Maurer
b208da8393 src/backup/data_{chunk,blob}.rs: add verify_crc method 2019-06-26 09:54:25 +02:00
Dietmar Maurer
bb8231409e src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json 
Use the KeyConfig serialization with kdf = None.
 2019-06-26 07:32:34 +02:00
Dietmar Maurer
c6d203bbd3 src/backup/backup_info.rs: also list .blob files 2019-06-25 07:07:45 +02:00
Dietmar Maurer
9f46c7de4b src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup 2019-06-24 13:56:37 +02:00
Dietmar Maurer
cb08ac3efe src/api2/backup.rs: replace upload_config with upload_blob 2019-06-23 11:44:30 +02:00
Dietmar Maurer
a38c5d4d12 src/backup/data_blob.rs: implement from_raw() 2019-06-23 09:35:44 +02:00
Dietmar Maurer
6f083b7a92 src/api2/backup/upload_chunk.rs: compute crc on upload 2019-06-23 09:04:42 +02:00
Dietmar Maurer
ba01828d38 try to use use proxmox::tools::io::ops::ReadExtOps 2019-06-22 16:29:10 +02:00
Dietmar Maurer
27042ce637 src/backup/file_formats.rs: improve docs 2019-06-22 15:45:36 +02:00
Dietmar Maurer
9f83e0f7ee src/backup/crypt_config.rs: simplify decode interface 2019-06-22 13:24:29 +02:00
Dietmar Maurer
0066c6d972 src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead 2019-06-22 13:02:53 +02:00
Dietmar Maurer
ee8a7e8080 src/backup/crypt_config.rs: implement encrypt_to (output to writer) 2019-06-22 12:25:04 +02:00
Dietmar Maurer
991abfa8b4 src/backup/file_formats.rs: split out file format data 2019-06-22 09:12:25 +02:00
Dietmar Maurer
b7f4f27d6c add crc field for binary blobs formats 2019-06-21 18:49:04 +02:00
Dietmar Maurer
3025b3a53c src/backup/data_blob.rs: new file format for binary blobs 2019-06-21 11:32:07 +02:00
Dietmar Maurer
077a8cae63 src/backup/crypt_config.rs: make code more flexible 
Allow to pass magic numbers as parameters.
 2019-06-21 10:50:48 +02:00
Dietmar Maurer
c38266c18d renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs 2019-06-21 09:51:18 +02:00
Dietmar Maurer
ab44acff57 src/backup/key_derivation.rs: add modified field to key file 2019-06-20 07:58:33 +02:00
Dietmar Maurer
6d0983dbe1 src/bin/proxmox-backup-client.rs: add keyfile parameter for backup 2019-06-19 17:16:41 +02:00
Dietmar Maurer
181f097af9 src/bin/proxmox-backup-client.rs - key API: pass kdf parameter 
Allow to store keys without password.
 2019-06-18 18:21:01 +02:00
Dietmar Maurer
826f309bf5 src/backup/key_derivation.rs: move kdf code into separate file 2019-06-18 11:17:22 +02:00
Dietmar Maurer
11515438cc Cargo.toml: use serde feature derive 2019-06-18 06:23:25 +02:00
Dietmar Maurer
f2401311b0 src/bin/proxmox-backup-client.rs: start key management 2019-06-17 10:33:24 +02:00
Dietmar Maurer
a7dd483097 src/backup/*_index.rs: used generated magic numbers 2019-06-14 14:58:37 +02:00