Christian Ebner
b82a101c86
catalog: store current writer position in CatalogBlobWriter
...
Counts the bytes written by the CatalogBlobWriter in order to obtain the
stream position, needed to get offset to reference catalog items.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-31 09:11:51 +01:00
Dietmar Maurer
62ee2eb405
avoid some clippy warnings
2019-10-26 11:42:05 +02:00
Dietmar Maurer
834a2f95a0
avoid some clippy warnings
2019-10-25 18:44:51 +02:00
Dietmar Maurer
9fe2f639b8
allow(clippy::cast_ptr_alignment)
2019-10-25 18:03:55 +02:00
Dietmar Maurer
cd69d36b45
src/backup/dynamic_index.rs: use write_all()
2019-10-25 17:22:41 +02:00
Dietmar Maurer
649c5ee7b1
load_and_decrtypt_key: allow function closures to readf passphrase
2019-10-16 13:23:52 +02:00
Dietmar Maurer
60ffa9649c
src/backup/manifest.rs: fix property names (use '-' instead of '_')
2019-10-14 12:28:27 +02:00
Dietmar Maurer
4c9f753c56
src/backup/data_blob.rs: define create_zero_chunk as member function
...
and rename it to build_zero_chunk()
2019-10-14 10:58:26 +02:00
Dietmar Maurer
2745151d93
src/backup/data_blob.rs: add create_zero_chunk() helper
2019-10-14 10:44:46 +02:00
Dietmar Maurer
f06b820ac0
src/backup/manifest.rs: add verify_file
2019-10-13 10:09:12 +02:00
Dietmar Maurer
511a47bd73
cleanup
2019-10-13 09:39:21 +02:00
Dietmar Maurer
e17d5d864d
src/backup/manifest.rs: improve parser (try_from)
2019-10-13 08:39:49 +02:00
Dietmar Maurer
59e9ba01c6
src/backup/manifest.rs: new class to generate/parse index.json
2019-10-12 17:58:08 +02:00
Dietmar Maurer
7123ff7d43
src/backup/data_blob.rs: avoid Arc<CryptConfig>
...
use simple reference instead.
2019-10-07 11:41:35 +02:00
Dietmar Maurer
4ee8f53d07
remove DataChunk file format - use DataBlob instead
2019-10-06 10:31:06 +02:00
Dietmar Maurer
9e603e254c
src/backup/fixed_index.rs: do not use offset for checksum
2019-09-23 11:30:20 +02:00
Dietmar Maurer
2aba16bde1
src/backup/data_chunk.rs: add into_raw()
2019-09-19 14:24:17 +02:00
Wolfgang Bumiller
ba5e67475a
clippy: fix access to unaligned buffer in dynamic reader
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:50:45 +02:00
Wolfgang Bumiller
b526bd1408
backup: cleanup unused imports
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:46:46 +02:00
Wolfgang Bumiller
4dc79bb1d9
clippy: fix access to unaligned buffer
...
via the newly introduced ReadExt::read_host_value_boxed()
from proxmox::tools::io.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:45:44 +02:00
Wolfgang Bumiller
ccdf3ad1ec
backup: use static assertion in fixed index reader
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:43:25 +02:00
Wolfgang Bumiller
b21851cc5c
clippy: collapse identical if branches
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:57:58 +02:00
Wolfgang Bumiller
653b1ca10e
some clippy fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 12:14:05 +02:00
Dietmar Maurer
9d456cf8ec
src/backup/catalog_blob.rs: use write_all() instead of write()
...
To correctly handle interrupted system calls.
2019-09-05 11:40:48 +02:00
Dietmar Maurer
0a51fe0011
src/backup/fixed_index.rs: new helper to compute checksum and file size
2019-09-03 13:11:45 +02:00
Dietmar Maurer
82c85a21a1
src/backup/dynamic_index.rs: new helper to compute checksum and file size
2019-09-03 13:11:09 +02:00
Wolfgang Bumiller
745e652a7f
src/backup/index.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
32bef1e2d1
src/backup/chunk_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
5c1130df9f
implement Sync for DynamicIndexReader and FixedIndexReader
...
hyper's wrap_stream now needs this
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
d21f8a5b5b
backup/dynamic_index: explicitly zero-initialize the header
...
The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
Wolfgang Bumiller
990b930f22
backup/dynamic_index: use static assertion, fix size
...
The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
Wolfgang Bumiller
7d83440c60
remove proxmox-protocol subcrate
...
AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)
Moved the chunker back into src/backup/chunker.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 14:08:25 +02:00
Wolfgang Bumiller
5e58e1bb7d
{dynamic,fixed}_index: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:35:36 +02:00
Wolfgang Bumiller
f93b55b057
src/backup/index: style fixup & unsafe copy removal
...
We can use the safe .copy_from_slice alternative in this
case.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:31:45 +02:00
Wolfgang Bumiller
fa17b1ce2a
src/backup/index: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:27:35 +02:00
Wolfgang Bumiller
367f002ee2
use statement cleanup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-16 12:28:40 +02:00
Dietmar Maurer
9d135fe617
src/backup/catalog_blob.rs: moved catalog impl. from pxar
...
And avoid loading catalog into memory.
2019-08-16 12:27:17 +02:00
Wolfgang Bumiller
9025312aa6
avoid lifetimes in blob reader/writer
2019-08-16 10:07:24 +02:00
Dietmar Maurer
71d08e00b7
src/backup/data_blob_reader.rs: fix compiler warning
2019-08-16 08:10:39 +02:00
Dietmar Maurer
b791804f4b
src/backup/data_blob_reader.rs: impl for Read instead of BufRead
2019-08-14 14:56:01 +02:00
Dietmar Maurer
39a4df61d6
improve docs
2019-08-14 14:08:27 +02:00
Dietmar Maurer
018d11bb84
src/backup/data_blob.rs: move parts into single files
2019-08-14 13:24:41 +02:00
Dietmar Maurer
e27c5a559f
src/backup/data_blob.rs: avoid excessive stack usage
2019-08-14 12:44:08 +02:00
Dietmar Maurer
c638542b4b
src/backup/file_formats.rs: moved header_size
2019-08-14 12:35:53 +02:00
Dietmar Maurer
6082216343
src/backup/data_blob.rs - CryptReader/CryptWriter: fix buffer size handling
2019-08-13 15:23:15 +02:00
Dietmar Maurer
548c9489d7
src/backup/data_blob.rs - DataBlobReader: impl compressed, encrypted blobs
2019-08-13 13:17:08 +02:00
Dietmar Maurer
2aa0bfff59
src/backup/data_blob.rs - DataBlobReader: implement reader for encrtypted blobs
2019-08-13 13:17:08 +02:00
Dietmar Maurer
e9a385a78e
src/backup/data_blob.rs - DataBlobReader: impl compressed, signed blobs
2019-08-12 17:47:25 +02:00
Dietmar Maurer
4bfa147eaf
src/backup/data_blob.rs - DataBlobReader: impl. reader for signed blobs
2019-08-12 17:41:25 +02:00
Dietmar Maurer
09785b2795
src/backup/data_blob.rs - DataBlobReader: start serious impl.
2019-08-12 17:20:41 +02:00
Dietmar Maurer
a32bd8a516
src/backup/data_blob.rs - DataBlobWriter: fix decryption
2019-08-12 11:57:29 +02:00
Dietmar Maurer
5d15cb49b4
src/backup/data_blob.rs - compute_crc: start after blob header
2019-08-12 11:32:26 +02:00
Dietmar Maurer
1cbdfd19a2
src/backup/file_formats.rs; use const instzead of static
2019-08-12 11:20:21 +02:00
Dietmar Maurer
18be4ec24a
src/backup/data_blob.rs - DataBlobWriter: fix magic number for uncompressed blobs
2019-08-12 10:46:05 +02:00
Dietmar Maurer
5622a3fcdb
src/backup/data_blob.rs - DataBlobWriter: impl. compressed encrypted blobs
2019-08-12 10:16:42 +02:00
Dietmar Maurer
f4942e9ffd
src/backup/data_blob.rs - DataBlobWriter: impl. encrypted blobs
2019-08-12 10:08:00 +02:00
Dietmar Maurer
c57ec43a53
src/backup/crypt_config.rs: new helper data_crypter to get openssl Crypter
2019-08-12 10:06:51 +02:00
Dietmar Maurer
7776becf5d
src/backup/data_blob.rs - DataBlobWriter: impl. compressed signed blobs
2019-08-12 09:01:08 +02:00
Dietmar Maurer
706638f803
src/backup/data_blob.rs: impl. ChecksumWriter
...
To correctly compute crc,hmac of final data.
2019-08-12 08:53:53 +02:00
Dietmar Maurer
f796351c54
src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs
2019-08-12 07:34:41 +02:00
Dietmar Maurer
cb0eea29d9
src/backup/crypt_config.rs: new method to get hmac signer
2019-08-12 07:33:15 +02:00
Dietmar Maurer
a762ce54fd
src/backup/data_blob.rs: make DataBlobWriter more generic
...
Allow to write compressed and uncompressed blobs.
2019-08-11 12:14:04 +02:00
Dietmar Maurer
1f26fdef19
src/backup/data_blob.rs: impl. simple reader/writer
...
To avoid loading blob into memory.
2019-08-11 11:32:36 +02:00
Dietmar Maurer
781ac11c6a
src/backup/data_blob.rs: define const MAX_BLOB_SIZE
2019-08-09 11:49:06 +02:00
Dietmar Maurer
863be2e6e2
src/backup/data_blob.rs: allow largert blob sizes (128MB)
...
Need this for catalog files.
2019-08-09 10:22:56 +02:00
Dietmar Maurer
eecb23560b
src/backup/data_blob.rs - encode: always compute crc
2019-08-07 08:30:27 +02:00
Dietmar Maurer
38a6cddad7
src/backup/backup_info.rs: add BackupInfo::new constructor
2019-08-07 08:27:52 +02:00
Dietmar Maurer
f889b15867
src/backup/data_blob.rs: always compute crc
2019-08-06 11:42:14 +02:00
Dietmar Maurer
e18a6c9ee5
update to nix 0.14, use code from proxmox:tools
2019-08-03 13:05:38 +02:00
Dietmar Maurer
69ecd8d5b0
src/backup/data_blob.rs: implement signed blobs
2019-08-02 09:56:01 +02:00
Dietmar Maurer
93205f942a
src/backup/crypt_config.rs: new compute_auth_tag helper
2019-08-02 08:55:37 +02:00
Dietmar Maurer
c68d2170d5
src/backup/crypt_config.rs: fix typo
2019-08-02 08:29:40 +02:00
Dietmar Maurer
fa5d6977dd
Revert previous, commit, use UTC RFC3339 without timezone (Z)
...
We now have human readable data/time, and names are still sortable.
2019-07-22 11:49:30 +02:00
Dietmar Maurer
10c2a21cfe
src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339
...
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
2019-07-22 07:57:03 +02:00
Dietmar Maurer
f4bf7dfcc7
src/client/remote_chunk_reader.rs: implement simple caching
2019-07-05 10:42:46 +02:00
Dietmar Maurer
afb4cd28be
src/backup/fixed_index.rs: implement BufferedFixedReader
2019-07-05 09:19:56 +02:00
Dietmar Maurer
b46c3fad03
src/backup/fixed_index.rs: use correct size
...
We need to consider that the last chunk may have smaller size.
2019-07-04 15:13:22 +02:00
Dietmar Maurer
5e04ec707d
src/api2/backup/environment.rs: improve chunk alignment/size checks
2019-07-04 13:40:43 +02:00
Dietmar Maurer
a24e3993e0
src/backup/chunk_store.rs: coding style fixes
2019-07-04 11:39:10 +02:00
Dietmar Maurer
e4c2fbf170
src/backup/chunk_store.rs: additionally log chunk count
2019-07-04 11:27:11 +02:00
Dietmar Maurer
9850bcdf19
src/backup/chunk_store.rs: improve error reporting
2019-07-04 11:21:54 +02:00
Wolfgang Bumiller
a3f3e91da2
backup/chunk_store: rework chunk iterator
...
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-04 10:55:17 +02:00
Dietmar Maurer
a57360983b
src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item
2019-07-04 09:26:44 +02:00
Dietmar Maurer
0f0a35b390
src/backup/dynamic_index.rs: lock file inside new, code cleanup
2019-07-04 08:17:30 +02:00
Dietmar Maurer
a7c72ad9eb
src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store
2019-07-04 08:13:16 +02:00
Dietmar Maurer
a660978c9a
src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats
2019-07-04 07:57:43 +02:00
Dietmar Maurer
1e0784841c
src/backup/read_chunk.rs: use &mut self
2019-07-03 14:25:18 +02:00
Dietmar Maurer
877dd11133
src/backup/{fixed, dynamic}_index.rs: improve error messages
2019-07-02 08:56:56 +02:00
Dietmar Maurer
5c593a4d56
src/backup/read_chunk.rs: use Arc for DataStore
2019-07-02 08:49:16 +02:00
Dietmar Maurer
b850673634
src/backup/read_chunk.rs: move read chunk trait into extra file
...
And implement LocalChunkReader.
2019-07-02 08:22:29 +02:00
Wolfgang Bumiller
f35197f449
replace tools::vec with proxmox::tools::vec
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 11:03:25 +02:00
Wolfgang Bumiller
9b2b627fe0
update remaining users of tools::io::ops
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:03 +02:00
Wolfgang Bumiller
5485b579a1
backup: use new proxmox::tools::io traits
...
This replaces all `write_value` calls with `write_le_value`
calls!
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:37:02 +02:00
Dietmar Maurer
d48a9955a5
src/backup/dynamic_index.rs: introduce ReadChunk trait
2019-06-28 16:35:00 +02:00
Dietmar Maurer
81a6ce6fde
src/backup/chunk_store.rs: new method chunk_path()
...
Returns the absolute path.
2019-06-28 15:48:09 +02:00
Dietmar Maurer
b208da8393
src/backup/data_{chunk,blob}.rs: add verify_crc method
2019-06-26 09:54:25 +02:00
Dietmar Maurer
bb8231409e
src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json
...
Use the KeyConfig serialization with kdf = None.
2019-06-26 07:32:34 +02:00
Dietmar Maurer
c6d203bbd3
src/backup/backup_info.rs: also list .blob files
2019-06-25 07:07:45 +02:00
Dietmar Maurer
9f46c7de4b
src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup
2019-06-24 13:56:37 +02:00
Dietmar Maurer
cb08ac3efe
src/api2/backup.rs: replace upload_config with upload_blob
2019-06-23 11:44:30 +02:00