tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
3,842 Commits 2 Branches 54 Tags 14 MiB
37e60ddcde
Commit Graph

615 Commits

Author SHA1 Message Date
Dietmar Maurer
f796351c54 src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs 2019-08-12 07:34:41 +02:00
Dietmar Maurer
cb0eea29d9 src/backup/crypt_config.rs: new method to get hmac signer 2019-08-12 07:33:15 +02:00
Dietmar Maurer
a762ce54fd src/backup/data_blob.rs: make DataBlobWriter more generic 
Allow to write compressed and uncompressed blobs.
 2019-08-11 12:14:04 +02:00
Dietmar Maurer
1f26fdef19 src/backup/data_blob.rs: impl. simple reader/writer 
To avoid loading blob into memory.
 2019-08-11 11:32:36 +02:00
Dietmar Maurer
781ac11c6a src/backup/data_blob.rs: define const MAX_BLOB_SIZE 2019-08-09 11:49:06 +02:00
Dietmar Maurer
863be2e6e2 src/backup/data_blob.rs: allow largert blob sizes (128MB) 
Need this for catalog files.
 2019-08-09 10:22:56 +02:00
Dietmar Maurer
eecb23560b src/backup/data_blob.rs - encode: always compute crc 2019-08-07 08:30:27 +02:00
Dietmar Maurer
38a6cddad7 src/backup/backup_info.rs: add BackupInfo::new constructor 2019-08-07 08:27:52 +02:00
Dietmar Maurer
f889b15867 src/backup/data_blob.rs: always compute crc 2019-08-06 11:42:14 +02:00
Dietmar Maurer
e18a6c9ee5 update to nix 0.14, use code from proxmox:tools 2019-08-03 13:05:38 +02:00
Dietmar Maurer
69ecd8d5b0 src/backup/data_blob.rs: implement signed blobs 2019-08-02 09:56:01 +02:00
Dietmar Maurer
93205f942a src/backup/crypt_config.rs: new compute_auth_tag helper 2019-08-02 08:55:37 +02:00
Dietmar Maurer
c68d2170d5 src/backup/crypt_config.rs: fix typo 2019-08-02 08:29:40 +02:00
Dietmar Maurer
fa5d6977dd Revert previous, commit, use UTC RFC3339 without timezone (Z) 
We now have human readable data/time, and names are still sortable.
 2019-07-22 11:49:30 +02:00
Dietmar Maurer
10c2a21cfe src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339 
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
 2019-07-22 07:57:03 +02:00
Dietmar Maurer
f4bf7dfcc7 src/client/remote_chunk_reader.rs: implement simple caching 2019-07-05 10:42:46 +02:00
Dietmar Maurer
afb4cd28be src/backup/fixed_index.rs: implement BufferedFixedReader 2019-07-05 09:19:56 +02:00
Dietmar Maurer
b46c3fad03 src/backup/fixed_index.rs: use correct size 
We need to consider that the last chunk may have smaller size.
 2019-07-04 15:13:22 +02:00
Dietmar Maurer
5e04ec707d src/api2/backup/environment.rs: improve chunk alignment/size checks 2019-07-04 13:40:43 +02:00
Dietmar Maurer
a24e3993e0 src/backup/chunk_store.rs: coding style fixes 2019-07-04 11:39:10 +02:00
Dietmar Maurer
e4c2fbf170 src/backup/chunk_store.rs: additionally log chunk count 2019-07-04 11:27:11 +02:00
Dietmar Maurer
9850bcdf19 src/backup/chunk_store.rs: improve error reporting 2019-07-04 11:21:54 +02:00
Wolfgang Bumiller
a3f3e91da2 backup/chunk_store: rework chunk iterator 
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-04 10:55:17 +02:00
Dietmar Maurer
a57360983b src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item 2019-07-04 09:26:44 +02:00
Dietmar Maurer
0f0a35b390 src/backup/dynamic_index.rs: lock file inside new, code cleanup 2019-07-04 08:17:30 +02:00
Dietmar Maurer
a7c72ad9eb src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store 2019-07-04 08:13:16 +02:00
Dietmar Maurer
a660978c9a src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats 2019-07-04 07:57:43 +02:00
Dietmar Maurer
1e0784841c src/backup/read_chunk.rs: use &mut self 2019-07-03 14:25:18 +02:00
Dietmar Maurer
877dd11133 src/backup/{fixed, dynamic}_index.rs: improve error messages 2019-07-02 08:56:56 +02:00
Dietmar Maurer
5c593a4d56 src/backup/read_chunk.rs: use Arc for DataStore 2019-07-02 08:49:16 +02:00
Dietmar Maurer
b850673634 src/backup/read_chunk.rs: move read chunk trait into extra file 
And implement LocalChunkReader.
 2019-07-02 08:22:29 +02:00
Wolfgang Bumiller
f35197f449 replace tools::vec with proxmox::tools::vec 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 11:03:25 +02:00
Wolfgang Bumiller
9b2b627fe0 update remaining users of tools::io::ops 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:44:03 +02:00
Wolfgang Bumiller
5485b579a1 backup: use new proxmox::tools::io traits 
This replaces all `write_value` calls with `write_le_value`
calls!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:37:02 +02:00
Dietmar Maurer
d48a9955a5 src/backup/dynamic_index.rs: introduce ReadChunk trait 2019-06-28 16:35:00 +02:00
Dietmar Maurer
81a6ce6fde src/backup/chunk_store.rs: new method chunk_path() 
Returns the absolute path.
 2019-06-28 15:48:09 +02:00
Dietmar Maurer
b208da8393 src/backup/data_{chunk,blob}.rs: add verify_crc method 2019-06-26 09:54:25 +02:00
Dietmar Maurer
bb8231409e src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json 
Use the KeyConfig serialization with kdf = None.
 2019-06-26 07:32:34 +02:00
Dietmar Maurer
c6d203bbd3 src/backup/backup_info.rs: also list .blob files 2019-06-25 07:07:45 +02:00
Dietmar Maurer
9f46c7de4b src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup 2019-06-24 13:56:37 +02:00
Dietmar Maurer
cb08ac3efe src/api2/backup.rs: replace upload_config with upload_blob 2019-06-23 11:44:30 +02:00
Dietmar Maurer
a38c5d4d12 src/backup/data_blob.rs: implement from_raw() 2019-06-23 09:35:44 +02:00
Dietmar Maurer
6f083b7a92 src/api2/backup/upload_chunk.rs: compute crc on upload 2019-06-23 09:04:42 +02:00
Dietmar Maurer
ba01828d38 try to use use proxmox::tools::io::ops::ReadExtOps 2019-06-22 16:29:10 +02:00
Dietmar Maurer
27042ce637 src/backup/file_formats.rs: improve docs 2019-06-22 15:45:36 +02:00
Dietmar Maurer
9f83e0f7ee src/backup/crypt_config.rs: simplify decode interface 2019-06-22 13:24:29 +02:00
Dietmar Maurer
0066c6d972 src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead 2019-06-22 13:02:53 +02:00
Dietmar Maurer
ee8a7e8080 src/backup/crypt_config.rs: implement encrypt_to (output to writer) 2019-06-22 12:25:04 +02:00
Dietmar Maurer
991abfa8b4 src/backup/file_formats.rs: split out file format data 2019-06-22 09:12:25 +02:00
Dietmar Maurer
b7f4f27d6c add crc field for binary blobs formats 2019-06-21 18:49:04 +02:00
Dietmar Maurer
3025b3a53c src/backup/data_blob.rs: new file format for binary blobs 2019-06-21 11:32:07 +02:00
Dietmar Maurer
077a8cae63 src/backup/crypt_config.rs: make code more flexible 
Allow to pass magic numbers as parameters.
 2019-06-21 10:50:48 +02:00
Dietmar Maurer
c38266c18d renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs 2019-06-21 09:51:18 +02:00
Dietmar Maurer
ab44acff57 src/backup/key_derivation.rs: add modified field to key file 2019-06-20 07:58:33 +02:00
Dietmar Maurer
6d0983dbe1 src/bin/proxmox-backup-client.rs: add keyfile parameter for backup 2019-06-19 17:16:41 +02:00
Dietmar Maurer
181f097af9 src/bin/proxmox-backup-client.rs - key API: pass kdf parameter 
Allow to store keys without password.
 2019-06-18 18:21:01 +02:00
Dietmar Maurer
826f309bf5 src/backup/key_derivation.rs: move kdf code into separate file 2019-06-18 11:17:22 +02:00
Dietmar Maurer
11515438cc Cargo.toml: use serde feature derive 2019-06-18 06:23:25 +02:00
Dietmar Maurer
f2401311b0 src/bin/proxmox-backup-client.rs: start key management 2019-06-17 10:33:24 +02:00
Dietmar Maurer
a7dd483097 src/backup/*_index.rs: used generated magic numbers 2019-06-14 14:58:37 +02:00
Dietmar Maurer
bffd40d6b7 src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools 2019-06-14 11:40:04 +02:00
Dietmar Maurer
9335d74eba src/backup/fixed_index.rs: compute checksum over the index 2019-06-14 10:34:29 +02:00
Dietmar Maurer
16ff6b7cd8 src/backup/dynamic_index.rs: compute checksum over the index 2019-06-14 10:33:59 +02:00
Dietmar Maurer
fa148dbd11 src/api2/backup/upload_chunk.rs: verify chunk data if possible 2019-06-14 06:23:46 +02:00
Dietmar Maurer
f640f23ac5 src/backup/data_chunk.rs: fix typo 2019-06-13 16:42:55 +02:00
Dietmar Maurer
dba72d5016 src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:56:50 +02:00
Dietmar Maurer
68ab376ae9 src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:47:54 +02:00
Dietmar Maurer
f98ac774ee backup: Add support for client side encryption 
first try ...
 2019-06-13 11:47:23 +02:00
Dietmar Maurer
51929e4532 src/backup/data_chunk.rs - decode: make crypt_config optional 2019-06-13 11:44:38 +02:00
Dietmar Maurer
046bcb9773 src/backup/crypt_setup.rs: add comment for possible improvements 2019-06-13 07:35:59 +02:00
Dietmar Maurer
bd0e3c7cfa src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs 2019-06-13 07:35:02 +02:00
Dietmar Maurer
b595cb9d2c src/backup/data_chunk.rs: Data Chunk builder 
Make handling encoded chunks easier.
 2019-06-12 13:37:01 +02:00
Dietmar Maurer
ef27200cec src/backup/crypt_setup.rs: fix encoder for uncompressed chunks 2019-06-10 09:56:06 +02:00
Dietmar Maurer
74792b95b2 src/backup/crypt_setup.rs: allow compressed and uncompressed chunks 2019-06-10 08:27:35 +02:00
Dietmar Maurer
9e0187a203 src/backup/crypt_setup.rs: remove unused CryptData struct 2019-06-10 07:55:36 +02:00
Dietmar Maurer
bec8498ae3 src/backup/crypt_setup.rs: avoid useless allocations 2019-06-09 11:44:17 +02:00
Dietmar Maurer
48b4b40b57 src/backup/crypt_setup.rs: crypto helpers 2019-06-08 09:56:51 +02:00
Wolfgang Bumiller
dd5495d6dc tree-wide: use 'dyn' for all trait objects 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-06-07 13:13:48 +02:00
Dietmar Maurer
36898ffce6 src/backup/chunk_stream.rs: add optional chunk_size parameter 2019-05-30 13:28:24 +02:00
Dietmar Maurer
976595e1a9 src/backup/dynamic_index.rs: split class DynamicIndexWriter 2019-05-29 08:49:57 +02:00
Dietmar Maurer
cb0708dd46 src/backup/fixed_index.rs: remove ChunkStat from struct 2019-05-29 07:08:34 +02:00
Dietmar Maurer
01af11f340 src/backup/fixed_index.rs: improve consistency checks 2019-05-29 07:02:45 +02:00
Dietmar Maurer
006f3ff407 src/api2/admin/datastore/backup/environment.rs: add more consistency checks 2019-05-28 09:21:31 +02:00
Dietmar Maurer
fc14b84937 src/backup/fixed_index.rs: improve bound checks inside add_digest 2019-05-28 08:47:03 +02:00
Dietmar Maurer
e1225de467 src/backup/fixed_index.rs: store index_length to avoid code duplication 2019-05-28 08:27:57 +02:00
Dietmar Maurer
29ae5c86a2 src/backup/fixed_index.rs: make chunk_size public 2019-05-28 06:18:24 +02:00
Wolfgang Bumiller
8ea3b1d188 tree-wide: use the new vec/io tools modules 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-23 09:49:10 +02:00
Dietmar Maurer
0cc0fffd1e src/backup/chunk_stream.rs: use BytesMut in ChunkStream 2019-05-22 09:39:02 +02:00
Dietmar Maurer
c052be5c86 src/backup/chunk_stream.rs: use more generics 2019-05-22 09:18:05 +02:00
Dietmar Maurer
169c0e060f src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut) 2019-05-22 09:05:35 +02:00
Dietmar Maurer
3be3f3dcc4 src/backup/chunk_stream.rs: fix behaviour for large input buffers 2019-05-22 08:05:27 +02:00
Dietmar Maurer
02fa54ff3c src/backup/chunk_stream.rs: assemble data with correct order 2019-05-22 07:44:51 +02:00
Dietmar Maurer
7f3d2ffab9 src/backup/index.rs: rename ChunkListReader into DigestListEncoder 2019-05-21 12:28:44 +02:00
Dietmar Maurer
ee53955f52 src/backup/index.rs: implement DigestListDecoder 2019-05-21 12:21:22 +02:00
Dietmar Maurer
40f4e198a8 src/backup/dynamic_index.rs: add chunk_info method 2019-05-21 09:52:19 +02:00
Dietmar Maurer
ff77dbbea3 Revert "src/backup/chunk_stream.rs: simplify code" 
This reverts commit e058744d8f.
The optimization was wrong, and produces wrong chunk boundaries.
 2019-05-19 11:05:56 +02:00
Dietmar Maurer
e058744d8f src/backup/chunk_stream.rs: simplify code 2019-05-18 10:55:46 +02:00
Dietmar Maurer
8a7cc7565f src/backup/chunk_stream.rs: implement fixed size chunk stream 2019-05-18 10:55:46 +02:00
Dietmar Maurer
dafc27aeff src/backup/chunk_stream.rs: async chunk stream 2019-05-14 10:05:29 +02:00
Dietmar Maurer
457531e7fe src/backup/index.rs: new ChunkListReader 2019-05-11 16:01:42 +02:00
Dietmar Maurer
51a4f63ff7 src/backup/datastore.rs: move last_backup to BackupInfo 2019-05-11 12:07:09 +02:00
Dietmar Maurer
b02a52e3a5 src/backup/datastore.rs: impl last_backup helper 2019-05-11 11:36:46 +02:00
Dietmar Maurer
c09775011a src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups 2019-05-11 10:19:34 +02:00
Dietmar Maurer
35a2d8a6a6 src/backup/backup_info.rs: Derive Clone 2019-05-10 06:59:23 +02:00
Dietmar Maurer
f2b99c34f7 src/api2/admin/datastore.rs: implement API to return last GC status 2019-04-11 12:04:25 +02:00
Dietmar Maurer
d4b59ae0b8 src/backup/datastore.rs: pass WorkerTask to GC 
So that we can log to the task log file.
 2019-04-06 17:57:38 +02:00
Dietmar Maurer
92da93b245 abort GC on server shutdown 2019-04-01 12:13:02 +02:00
Dietmar Maurer
11861a482d src/backup/chunk_store.rs: fix GC 
Added option to get oldest_writer timestamp from ProcessLocker.
 2019-03-31 17:21:36 +02:00
Dietmar Maurer
d85987aeeb fix last commit: the filename var was not ment to be removed, sorry 2019-03-31 16:16:14 +02:00
Dietmar Maurer
15a77c4c2e src/backup/chunk_store.rs: avoid create/unlink race 2019-03-31 10:03:01 +02:00
Dietmar Maurer
10eea49d81 src/backup/fixed_index.rs: store reference to file 
Keep it open and locked while index is in use.
 2019-03-27 10:17:03 +01:00
Dietmar Maurer
c597a92c81 index file readers: aquire shared lock 2019-03-27 10:11:14 +01:00
Dietmar Maurer
43b1303398 datastore: use new ProcessLocker 
To make sure only one process runs garbage collection while having active writers.
 2019-03-22 09:42:15 +01:00
Dietmar Maurer
515688d1f1 avoid compiler warnings 2019-03-18 10:00:58 +01:00
Dietmar Maurer
8968258b66 rename catar into pxar 
To avoid confusion with the casync implementation.
 2019-03-14 10:54:09 +01:00
Dietmar Maurer
8e39232acc src/bin/proxmox-backup-client.rs: strip .didx file extensions 2019-03-11 10:54:02 +01:00
Wolfgang Bumiller
06178f13fa move chunker into proxmox_protocol 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-08 09:29:47 +01:00
Dietmar Maurer
141f062e08 src/backup/chunk_store.rs: use zstd compression insteadf of lz4 
Provides better compressionm rate, and is still fast.
 2019-03-07 11:42:59 +01:00
Dietmar Maurer
d2690f74bb src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end 2019-03-07 10:40:01 +01:00
Dietmar Maurer
58e99e138e src/backup/backup_info.rs: new helper to list files 2019-03-06 11:02:08 +01:00
Wolfgang Bumiller
e3062f87b1 fixed_index: split add_chunk/add_digest 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
c3bb97e59c fixed_index: impl Send for FixedIndexWriter 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
3b7ade9e78 backup/datastore: expose insert_chunk method 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
391d310741 backup_info: BackupDir shortcut 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-05 10:15:06 +01:00
Wolfgang Bumiller
93b49ce38a backup_info: use two independent Into<String> types 
there's no reason to force them to be the same input type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-05 09:13:08 +01:00
Dietmar Maurer
d57474e055 src/backup/backup_info.rs: improve docs 2019-03-05 07:28:13 +01:00
Dietmar Maurer
b3483782d1 src/backup/backup_info.rs: move code into separate file 
Also changed create_backup_dir() parameters - uses &BackupDir now.
 2019-03-05 07:18:12 +01:00
Dietmar Maurer
2b01a22507 src/backup/datastore.rs: add helper to sort backup lists 2019-03-04 18:20:57 +01:00
Dietmar Maurer
875fb1c01a src/backup/datastore.rs: use unix epoch to create DateTime 
To make sure that we have a timestamp without nanosecond.
 2019-03-04 17:58:22 +01:00
Dietmar Maurer
38f8815925 src/backup/datastore.rs: use new() function to create structs 2019-03-04 17:44:27 +01:00
Dietmar Maurer
9b492eb256 src/backup/datastore.rs: protect BackupDir fields, impl new() 2019-03-04 13:51:36 +01:00
Dietmar Maurer
1e9a94e579 src/backup/datastore.rs: protect BackupGroup fields, impl new() 2019-03-04 13:38:23 +01:00
Dietmar Maurer
f72dfd2d10 src/backup/datastore.rs: make sure timestamp does not contains nanoseconds 
Because we want simple, readable file names.
 2019-03-04 09:37:10 +01:00
Dietmar Maurer
6f62c9240a src/api2/admin/datastore.rs: imp delete_snapshot 2019-03-03 11:29:00 +01:00
Dietmar Maurer
184f17afad src/api2/admin/datastore.rs: impl list_snapshots 2019-03-02 16:20:50 +01:00
Dietmar Maurer
e909522f06 src/backup/datastore.rs: use timestamp with zone offset 2019-03-01 17:36:40 +01:00
Dietmar Maurer
dc4c09faae src/backup/datastore.rs: split BackupDir 2019-03-01 16:26:35 +01:00
Dietmar Maurer
38b0dfa511 src/backup/datastore.rs: split BackupInfo 2019-03-01 16:26:35 +01:00
Wolfgang Bumiller
dcd15d10bb backup/index: remove IndexIterator 
removed the need for it, to avoid copies

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-01 13:29:36 +01:00
Dietmar Maurer
541b908ea4 src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch 2019-02-28 17:03:48 +01:00
Wolfgang Bumiller
5de2bced2d datastore: add generic open_index 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
7bc1d72778 add IndexFile trait 
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.

Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5be4065baf make FixedIndexReader Send and update comments 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
fded74d0c7 style: BufferedReader -> BufferedRead 
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Dietmar Maurer
8f57971714 src/api2/admin/datastore.rs: implement prune 2019-02-28 12:51:27 +01:00
Wolfgang Bumiller
8731e40a7f backup/datastore: create_backup_dir: show if path existed 
To enable asserting the creation of a new backup rather than
adding to an existing one.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-27 10:15:36 +01:00
Wolfgang Bumiller
f661f37427 backup/dynamic_index: split out add_chunk 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:50:45 +01:00
Wolfgang Bumiller
9ac6ec868a backup/chunk_store: split insert_chunk 
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
3a50ddd0c9 backup/chunk_store: mark ChunkIterator as fused 
Ranges are FusedIterators and it can be useful to propagate
that property to users.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
eff25ecaf2 backup/chunk_store: make percentage output optional 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
d59397e60e backup/datastore: expose chunk_store.get_chunk_iterator 
will be used by the protocol handler to iterate over
existing chunks

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Dietmar Maurer
8eb1203bda src/backup/chunk_stat.rs: log write speed 2019-02-25 13:21:00 +01:00
Dietmar Maurer
7e3365554e src/backup/chunk_stat.rs: new struct to track chunk statistics 2019-02-25 12:52:10 +01:00
Dietmar Maurer
798f7fa065 src/backup/chunk_store.rs: return info about compressed chunk size 
So that we can generate better statistics ..
 2019-02-25 11:36:05 +01:00
Dietmar Maurer
78216a5ab1 src/backup/chunk_store.rs: use lz4 for compression 
And depend on lz4 lib now.
 2019-02-25 10:00:28 +01:00
Dietmar Maurer
f252ca654c src/backup/chunker.rs: simplify break test 
To make it a little bit faster ...
 2019-02-22 13:02:11 +01:00
Dietmar Maurer
61834c1cbf src/backup/chunker.rs: another try to speedup the chunker 2019-02-22 11:15:10 +01:00
Dietmar Maurer
8c75372b79 src/backup/datastore.rs: list_backups() - include list of index files 2019-02-20 13:37:44 +01:00
Dietmar Maurer
fa4ebf0fce src/backup/chunker.rs: simplify code 2019-02-20 11:17:19 +01:00
Dietmar Maurer
1b9c959c32 src/backup/chunker.rs: change window size back to 48 
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
 2019-02-20 11:01:09 +01:00
Dietmar Maurer
247cdbce72 src/api2/admin/datastore/catar.rs: allow to configure chunk-size 2019-02-19 15:19:12 +01:00
Dietmar Maurer
c584aa21fd src/backup/chunker.rs: start() - correctly store hash in self.h 2019-02-19 12:40:12 +01:00
Dietmar Maurer
cd1598302a src/backup/chunker.rs: use 64 byte chunker window size 
This results in faster code.
 2019-02-19 12:27:37 +01:00
Dietmar Maurer
9a3e869cf5 src/backup/chunker.rs: simplify buzzhash start code 2019-02-19 12:17:17 +01:00
Dietmar Maurer
4d2881dd4c src/backup/chunker.rs: correctly compute window index 'idx' 
And add a simple test.
 2019-02-18 16:59:23 +01:00
Dietmar Maurer
f0a6112451 backup/datastore.rs: improve error reporting 2019-02-15 14:33:03 +01:00
Dietmar Maurer
af6f80d3db Revert "backup/datastore: fn open is a private implementation detail" 
This reverts commit 3dacd9a660.

I currently use this in some test ...
 2019-02-14 13:13:49 +01:00
Wolfgang Bumiller
62f2422f6a backup/chunk_store: verify chunk file names 
get_chunk_iterator() should skip over files which aren't an
actual chunk

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
3dacd9a660 backup/datastore: fn open is a private implementation detail 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f backup/chunk_store: open base dir in get_chunk_iterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
c7f481b642 backup: replace ChunkIterator 
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Dietmar Maurer
e5064ba607 simplify backup lib structure (pub use xxx:*), improve doc 2019-02-12 14:13:31 +01:00
Dietmar Maurer
93d5d77952 rename ArchiveIndex to DynamicIndex 
also changed the file extension from .aidx to .didx
 2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd rename ImageIndex to FixedIndex 
also changed the file extension fron .iidx to .fidx
 2019-02-12 11:50:45 +01:00
Wolfgang Bumiller
82bc0ad40c backup: fixup warnings and readd lost file type check 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:43:43 +01:00
Wolfgang Bumiller
9349d2a1d2 backup: cleanup long lines 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
fdd71f5286 backup: use ChunkIterator for sweep_unused_chunks 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
c9e7f4dc8b backup: add optional progress callback to ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
4070096c6a backup: add ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
e0a5d1ca2f tests: chunk_store: cleanup .testdir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Dietmar Maurer
9f49fe1d5d avoid compiler warnings 2019-01-30 18:25:37 +01:00
Dietmar Maurer
2296860027 tools.rs: move digest_to_hex() into tools 2019-01-25 10:58:28 +01:00
Dietmar Maurer
332dcc2277 backup/chunk_store.rs: fix test case - use absolute path 2019-01-25 10:14:25 +01:00
Dietmar Maurer
784252dbc5 backup/datastore.rs: use single lazy_static block 2019-01-24 14:58:47 +01:00
Dietmar Maurer
7ca80246ea backup/datastore.rs: use DateTime<Utc> to store backup time 2019-01-21 10:51:52 +01:00
Dietmar Maurer
be0084b001 white space fix 2019-01-21 10:32:07 +01:00
Dietmar Maurer
e25736b4c3 backup/datastore.rs: implement list_backups 2019-01-20 18:02:27 +01:00
Dietmar Maurer
68469eebb4 backup/chunk_store.rs: verify base path is absolute 2019-01-20 16:49:22 +01:00
Dietmar Maurer
0b05fd5830 api3/admin/datastore/catar.rs: implement download 2019-01-20 09:39:32 +01:00
Dietmar Maurer
6a4c091616 api3/admin/datastore/catar.rs: start impl. download 2019-01-19 17:27:02 +01:00
Dietmar Maurer
728797d0c1 reduce compiler warnings 2019-01-18 16:50:15 +01:00
Dietmar Maurer
95cea65b04 backup/datastore.rs: list all index files using walkdir crate 2019-01-18 12:24:58 +01:00
Dietmar Maurer
ff3d3100d4 backup/datastore.rs: try to create useful directory layout 
store/type/ID/TIME/xxx.yyy
 2019-01-18 12:01:37 +01:00
Dietmar Maurer
150f1bd8f6 backup/datastore.rs: use Arc for chunk_store references 2019-01-15 12:36:16 +01:00
Dietmar Maurer
1629d2ad7b api3/admin/datastore/upload_catar.rs: implement upload future 2019-01-15 11:38:26 +01:00
Dietmar Maurer
ddbdf80d51 avoid compiler warnings 2019-01-11 08:41:33 +01:00
Dietmar Maurer
373ef4a504 backup/archive_index.rs: correctly check for errors 2019-01-10 11:19:54 +01:00
Dietmar Maurer
4624fe29cc impl Read for BufferedArchiveReader 2019-01-06 10:04:45 +01:00
Dietmar Maurer
343370504b impl Seek for BufferedArchiveReader 2019-01-06 09:35:39 +01:00
Dietmar Maurer
318564ac03 buffered_read: return empty buffer on eof 2019-01-06 09:17:28 +01:00
Dietmar Maurer
0a72e26704 define and use BufferedReader trait 2019-01-05 17:28:20 +01:00
Dietmar Maurer
39c6bd86cc backup/archive_index.rs: implement BufferedArchiveReader 
Implement relativly fast random read using binary search.
 2019-01-05 14:47:56 +01:00
Dietmar Maurer
060c4811a0 backup/chunk_store.rs: resize buffer by powers of two 2019-01-05 11:34:49 +01:00
Dietmar Maurer
df9973e8f5 backup/chunk_store.rs: avoid allocation in read_chunk 2019-01-04 17:16:56 +01:00
Dietmar Maurer
7394ca3e95 white space cleanups 2019-01-04 12:51:43 +01:00
Dietmar Maurer
96df2fb44b backup/chunk_store.rs: implement read_chunk() 2019-01-04 12:50:54 +01:00
Dietmar Maurer
bc616633d5 backup/chunk_store.rs: only use one directory level 2019-01-04 10:35:22 +01:00
Dietmar Maurer
b663789bdb backup/archive_index.rs: pass correct size to unmap 2019-01-04 09:28:41 +01:00
Dietmar Maurer
48d0d35688 backup/image_index.rs: use little endian 2019-01-04 08:45:45 +01:00
Dietmar Maurer
050f8a6355 chunker.rs: make sure chunk_size_avg is a power of two 2019-01-03 15:10:54 +01:00
Dietmar Maurer
ea4ea34baf improve chunker speed (avoid modulo) 2019-01-03 14:35:19 +01:00
Dietmar Maurer
0b8e75ed16 image_index.rs: verify file size 2019-01-02 18:14:02 +01:00
Dietmar Maurer
77703d95aa implement garbage collection for .aidx files 2019-01-02 14:27:04 +01:00
Dietmar Maurer
a360f6fa2d image_index.rs: verify header magic/version 2019-01-02 13:13:13 +01:00
Dietmar Maurer
44b3f62b42 fix typo 2019-01-02 12:56:04 +01:00
Dietmar Maurer
5032b57b44 archive_index.rs: correctly write .aidx file 2019-01-02 12:54:40 +01:00
Dietmar Maurer
580dc84bac image_index.rs: print stats 2019-01-02 12:53:49 +01:00
Dietmar Maurer
5e7a09be0d src/backup/archive_index.rs: use close() instead of flush() 
Also pass a reference to the encoder.
 2019-01-02 11:02:56 +01:00
Dietmar Maurer
94a882e900 src/backup/archive_index.rs: implement flush() 2018-12-31 18:01:07 +01:00
Dietmar Maurer
0433db1949 src/backup/archive_index.rs: first try 2018-12-31 17:30:08 +01:00
Dietmar Maurer
cbdd8c54ae create backup mod in backup.rs, improve docu 2018-12-31 16:08:04 +01:00
Dietmar Maurer
cb4412b18e move chunker.rs to backup/ 2018-12-31 15:46:16 +01:00
Dietmar Maurer
bcd879cfb3 backup-client: allow to specify directories (prepare for catar backup) 2018-12-27 10:11:11 +01:00
Dietmar Maurer
a198d74fc0 cleanup: reduce compiler warnings 2018-12-25 13:29:27 +01:00
Dietmar Maurer
991de6007b remove outdated comment 2018-12-25 13:24:50 +01:00
Dietmar Maurer
7ee2aa1b94 touch_chunk: use libc::lutimensat 2018-12-25 12:27:25 +01:00
Dietmar Maurer
2c32fdde86 move lookup_datastore() to backup/datastore.rs 2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835 gather usage statistics during garbage collection 2018-12-22 16:58:16 +01:00
Dietmar Maurer
176e4af964 sweep_used_chunks: print percentage 2018-12-22 15:39:05 +01:00
Dietmar Maurer
7b2b40a893 use openssl for faster hashing 2018-12-22 14:31:59 +01:00
Dietmar Maurer
e95950e40a chunk_store: reduce number of directories 
Else, scans everything takes too long ...
 2018-12-22 14:04:05 +01:00
Dietmar Maurer
03e4753d8e fix mutability for chunk store 2018-12-21 12:15:26 +01:00
Dietmar Maurer
d62e6e2264 backup-client: use 1M chunks, make chunk_size configurable 2018-12-21 08:36:57 +01:00
Dietmar Maurer
f7dd683b46 fix tests 2018-12-19 13:54:22 +01:00
Dietmar Maurer
277fc5a3ce try to improve error messages 2018-12-19 13:40:26 +01:00
Dietmar Maurer
eae8aa3aa9 cleanups: avoid compiler warnings 2018-12-19 12:49:23 +01:00
Dietmar Maurer
6c20a13d3c cleanup - avoid too much indentation 2018-12-19 12:40:26 +01:00
Dietmar Maurer
1c43c56b06 cleanup 2018-12-19 12:30:37 +01:00
Dietmar Maurer
2bf5f6b220 ChunkStore::sweep_used_chunks - use openat to speedup directory walks 2018-12-19 12:20:01 +01:00
Wolfgang Bumiller
1628a4c731 use timers with a signal for file locking 
* rename lock_file -> open_file_locked,
* add lock_file as a function working on already-opened
  files
* change timeout types to std::time::Duration

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-19 11:23:00 +01:00
Dietmar Maurer
365bb90f17 move lock_file to tools.rs 2018-12-19 10:02:24 +01:00
Dietmar Maurer
08481a0b60 impl sweep_used_chunks, first try 2018-12-19 09:51:33 +01:00
Dietmar Maurer
6ea3a0b7fc move sweep_used_chunks to chunk_store 2018-12-18 11:18:55 +01:00
Dietmar Maurer
3d5c11e5d2 start impl garbage collection 2018-12-18 11:06:03 +01:00
Dietmar Maurer
529de6c7a3 start implementing DataStore 2018-12-17 13:00:39 +01:00
Dietmar Maurer
4818c8b6f7 start implementing ImageIndexReader 2018-12-16 14:44:44 +01:00
Dietmar Maurer
af3e7d7551 fix chunk store file paths 2018-12-16 13:52:16 +01:00
Dietmar Maurer
0cd9d4206f improve error messages 2018-12-16 13:43:19 +01:00
Dietmar Maurer
4fbb72a8b4 image_index.rs: implement atomic write/rename 2018-12-16 13:39:21 +01:00
Dietmar Maurer
798881a68a rebane ImageIndex to ImageIndexWriter 2018-12-16 11:51:16 +01:00
Dietmar Maurer
5e5b7f1c2d use u64 to store time 2018-12-16 11:48:03 +01:00
Dietmar Maurer
d13e3745b3 image_index.rs: write header 2018-12-15 17:05:49 +01:00
Dietmar Maurer
606ce64bb0 image_index.rs:write idx of chunks (use mmap) 2018-12-15 14:51:05 +01:00
Dietmar Maurer
5ba6968922 add comment to make intensions more clear 2018-12-12 11:21:00 +01:00
Dietmar Maurer
ce55dbbcc6 simplify code 2018-12-09 09:42:17 +01:00
Dietmar Maurer
461e62fcdf remove some compiler warnings 2018-12-08 14:55:54 +01:00
Dietmar Maurer
391a2e43ff chunk_store.insert_chunk: return information about existing chunks 2018-12-08 11:25:11 +01:00
Dietmar Maurer
12bb93b3ad code cleanup 2018-12-08 11:20:02 +01:00
Dietmar Maurer
022fb421b2 lock with timeout 
clumsy, but I have node idea how to make it better
 2018-12-08 10:57:09 +01:00
Dietmar Maurer
2989f6bf20 improve error messages 2018-12-08 08:38:42 +01:00
Dietmar Maurer
b8d4766a4d code cleanup 2018-12-08 08:21:20 +01:00
Dietmar Maurer
45773720ab depend on nix, use flock 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2018-12-07 18:14:07 +01:00
Dietmar Maurer
c5d82e5f34 chunk_store: use mutex 2018-12-07 16:12:45 +01:00
Dietmar Maurer
128b37feea impl insert_chunk 2018-12-07 14:44:56 +01:00
Dietmar Maurer
35cf5daa53 start experiments with simple chunk store 2018-12-07 10:51:53 +01:00