Commit Graph

374 Commits

Author SHA1 Message Date
Dietmar Maurer a32bd8a516 src/backup/data_blob.rs - DataBlobWriter: fix decryption 2019-08-12 11:57:29 +02:00
Dietmar Maurer 5d15cb49b4 src/backup/data_blob.rs - compute_crc: start after blob header 2019-08-12 11:32:26 +02:00
Dietmar Maurer 1cbdfd19a2 src/backup/file_formats.rs; use const instzead of static 2019-08-12 11:20:21 +02:00
Dietmar Maurer 18be4ec24a src/backup/data_blob.rs - DataBlobWriter: fix magic number for uncompressed blobs 2019-08-12 10:46:05 +02:00
Dietmar Maurer 5622a3fcdb src/backup/data_blob.rs - DataBlobWriter: impl. compressed encrypted blobs 2019-08-12 10:16:42 +02:00
Dietmar Maurer f4942e9ffd src/backup/data_blob.rs - DataBlobWriter: impl. encrypted blobs 2019-08-12 10:08:00 +02:00
Dietmar Maurer c57ec43a53 src/backup/crypt_config.rs: new helper data_crypter to get openssl Crypter 2019-08-12 10:06:51 +02:00
Dietmar Maurer 7776becf5d src/backup/data_blob.rs - DataBlobWriter: impl. compressed signed blobs 2019-08-12 09:01:08 +02:00
Dietmar Maurer 706638f803 src/backup/data_blob.rs: impl. ChecksumWriter
To correctly compute crc,hmac of final data.
2019-08-12 08:53:53 +02:00
Dietmar Maurer f796351c54 src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs 2019-08-12 07:34:41 +02:00
Dietmar Maurer cb0eea29d9 src/backup/crypt_config.rs: new method to get hmac signer 2019-08-12 07:33:15 +02:00
Dietmar Maurer a762ce54fd src/backup/data_blob.rs: make DataBlobWriter more generic
Allow to write compressed and uncompressed blobs.
2019-08-11 12:14:04 +02:00
Dietmar Maurer 1f26fdef19 src/backup/data_blob.rs: impl. simple reader/writer
To avoid loading blob into memory.
2019-08-11 11:32:36 +02:00
Dietmar Maurer 781ac11c6a src/backup/data_blob.rs: define const MAX_BLOB_SIZE 2019-08-09 11:49:06 +02:00
Dietmar Maurer 863be2e6e2 src/backup/data_blob.rs: allow largert blob sizes (128MB)
Need this for catalog files.
2019-08-09 10:22:56 +02:00
Dietmar Maurer eecb23560b src/backup/data_blob.rs - encode: always compute crc 2019-08-07 08:30:27 +02:00
Dietmar Maurer 38a6cddad7 src/backup/backup_info.rs: add BackupInfo::new constructor 2019-08-07 08:27:52 +02:00
Dietmar Maurer f889b15867 src/backup/data_blob.rs: always compute crc 2019-08-06 11:42:14 +02:00
Dietmar Maurer e18a6c9ee5 update to nix 0.14, use code from proxmox:tools 2019-08-03 13:05:38 +02:00
Dietmar Maurer 69ecd8d5b0 src/backup/data_blob.rs: implement signed blobs 2019-08-02 09:56:01 +02:00
Dietmar Maurer 93205f942a src/backup/crypt_config.rs: new compute_auth_tag helper 2019-08-02 08:55:37 +02:00
Dietmar Maurer c68d2170d5 src/backup/crypt_config.rs: fix typo 2019-08-02 08:29:40 +02:00
Dietmar Maurer fa5d6977dd Revert previous, commit, use UTC RFC3339 without timezone (Z)
We now have human readable data/time, and names are still sortable.
2019-07-22 11:49:30 +02:00
Dietmar Maurer 10c2a21cfe src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
2019-07-22 07:57:03 +02:00
Dietmar Maurer f4bf7dfcc7 src/client/remote_chunk_reader.rs: implement simple caching 2019-07-05 10:42:46 +02:00
Dietmar Maurer afb4cd28be src/backup/fixed_index.rs: implement BufferedFixedReader 2019-07-05 09:19:56 +02:00
Dietmar Maurer b46c3fad03 src/backup/fixed_index.rs: use correct size
We need to consider that the last chunk may have smaller size.
2019-07-04 15:13:22 +02:00
Dietmar Maurer 5e04ec707d src/api2/backup/environment.rs: improve chunk alignment/size checks 2019-07-04 13:40:43 +02:00
Dietmar Maurer a24e3993e0 src/backup/chunk_store.rs: coding style fixes 2019-07-04 11:39:10 +02:00
Dietmar Maurer e4c2fbf170 src/backup/chunk_store.rs: additionally log chunk count 2019-07-04 11:27:11 +02:00
Dietmar Maurer 9850bcdf19 src/backup/chunk_store.rs: improve error reporting 2019-07-04 11:21:54 +02:00
Wolfgang Bumiller a3f3e91da2 backup/chunk_store: rework chunk iterator
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-04 10:55:17 +02:00
Dietmar Maurer a57360983b src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item 2019-07-04 09:26:44 +02:00
Dietmar Maurer 0f0a35b390 src/backup/dynamic_index.rs: lock file inside new, code cleanup 2019-07-04 08:17:30 +02:00
Dietmar Maurer a7c72ad9eb src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store 2019-07-04 08:13:16 +02:00
Dietmar Maurer a660978c9a src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats 2019-07-04 07:57:43 +02:00
Dietmar Maurer 1e0784841c src/backup/read_chunk.rs: use &mut self 2019-07-03 14:25:18 +02:00
Dietmar Maurer 877dd11133 src/backup/{fixed, dynamic}_index.rs: improve error messages 2019-07-02 08:56:56 +02:00
Dietmar Maurer 5c593a4d56 src/backup/read_chunk.rs: use Arc for DataStore 2019-07-02 08:49:16 +02:00
Dietmar Maurer b850673634 src/backup/read_chunk.rs: move read chunk trait into extra file
And implement LocalChunkReader.
2019-07-02 08:22:29 +02:00
Wolfgang Bumiller f35197f449 replace tools::vec with proxmox::tools::vec
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 11:03:25 +02:00
Wolfgang Bumiller 9b2b627fe0 update remaining users of tools::io::ops
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:03 +02:00
Wolfgang Bumiller 5485b579a1 backup: use new proxmox::tools::io traits
This replaces all `write_value` calls with `write_le_value`
calls!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:37:02 +02:00
Dietmar Maurer d48a9955a5 src/backup/dynamic_index.rs: introduce ReadChunk trait 2019-06-28 16:35:00 +02:00
Dietmar Maurer 81a6ce6fde src/backup/chunk_store.rs: new method chunk_path()
Returns the absolute path.
2019-06-28 15:48:09 +02:00
Dietmar Maurer b208da8393 src/backup/data_{chunk,blob}.rs: add verify_crc method 2019-06-26 09:54:25 +02:00
Dietmar Maurer bb8231409e src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json
Use the KeyConfig serialization with kdf = None.
2019-06-26 07:32:34 +02:00
Dietmar Maurer c6d203bbd3 src/backup/backup_info.rs: also list .blob files 2019-06-25 07:07:45 +02:00
Dietmar Maurer 9f46c7de4b src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup 2019-06-24 13:56:37 +02:00
Dietmar Maurer cb08ac3efe src/api2/backup.rs: replace upload_config with upload_blob 2019-06-23 11:44:30 +02:00
Dietmar Maurer a38c5d4d12 src/backup/data_blob.rs: implement from_raw() 2019-06-23 09:35:44 +02:00
Dietmar Maurer 6f083b7a92 src/api2/backup/upload_chunk.rs: compute crc on upload 2019-06-23 09:04:42 +02:00
Dietmar Maurer ba01828d38 try to use use proxmox::tools::io::ops::ReadExtOps 2019-06-22 16:29:10 +02:00
Dietmar Maurer 27042ce637 src/backup/file_formats.rs: improve docs 2019-06-22 15:45:36 +02:00
Dietmar Maurer 9f83e0f7ee src/backup/crypt_config.rs: simplify decode interface 2019-06-22 13:24:29 +02:00
Dietmar Maurer 0066c6d972 src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead 2019-06-22 13:02:53 +02:00
Dietmar Maurer ee8a7e8080 src/backup/crypt_config.rs: implement encrypt_to (output to writer) 2019-06-22 12:25:04 +02:00
Dietmar Maurer 991abfa8b4 src/backup/file_formats.rs: split out file format data 2019-06-22 09:12:25 +02:00
Dietmar Maurer b7f4f27d6c add crc field for binary blobs formats 2019-06-21 18:49:04 +02:00
Dietmar Maurer 3025b3a53c src/backup/data_blob.rs: new file format for binary blobs 2019-06-21 11:32:07 +02:00
Dietmar Maurer 077a8cae63 src/backup/crypt_config.rs: make code more flexible
Allow to pass magic numbers as parameters.
2019-06-21 10:50:48 +02:00
Dietmar Maurer c38266c18d renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs 2019-06-21 09:51:18 +02:00
Dietmar Maurer ab44acff57 src/backup/key_derivation.rs: add modified field to key file 2019-06-20 07:58:33 +02:00
Dietmar Maurer 6d0983dbe1 src/bin/proxmox-backup-client.rs: add keyfile parameter for backup 2019-06-19 17:16:41 +02:00
Dietmar Maurer 181f097af9 src/bin/proxmox-backup-client.rs - key API: pass kdf parameter
Allow to store keys without password.
2019-06-18 18:21:01 +02:00
Dietmar Maurer 826f309bf5 src/backup/key_derivation.rs: move kdf code into separate file 2019-06-18 11:17:22 +02:00
Dietmar Maurer 11515438cc Cargo.toml: use serde feature derive 2019-06-18 06:23:25 +02:00
Dietmar Maurer f2401311b0 src/bin/proxmox-backup-client.rs: start key management 2019-06-17 10:33:24 +02:00
Dietmar Maurer a7dd483097 src/backup/*_index.rs: used generated magic numbers 2019-06-14 14:58:37 +02:00
Dietmar Maurer bffd40d6b7 src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools 2019-06-14 11:40:04 +02:00
Dietmar Maurer 9335d74eba src/backup/fixed_index.rs: compute checksum over the index 2019-06-14 10:34:29 +02:00
Dietmar Maurer 16ff6b7cd8 src/backup/dynamic_index.rs: compute checksum over the index 2019-06-14 10:33:59 +02:00
Dietmar Maurer fa148dbd11 src/api2/backup/upload_chunk.rs: verify chunk data if possible 2019-06-14 06:23:46 +02:00
Dietmar Maurer f640f23ac5 src/backup/data_chunk.rs: fix typo 2019-06-13 16:42:55 +02:00
Dietmar Maurer dba72d5016 src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:56:50 +02:00
Dietmar Maurer 68ab376ae9 src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:47:54 +02:00
Dietmar Maurer f98ac774ee backup: Add support for client side encryption
first try ...
2019-06-13 11:47:23 +02:00
Dietmar Maurer 51929e4532 src/backup/data_chunk.rs - decode: make crypt_config optional 2019-06-13 11:44:38 +02:00
Dietmar Maurer 046bcb9773 src/backup/crypt_setup.rs: add comment for possible improvements 2019-06-13 07:35:59 +02:00
Dietmar Maurer bd0e3c7cfa src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs 2019-06-13 07:35:02 +02:00
Dietmar Maurer b595cb9d2c src/backup/data_chunk.rs: Data Chunk builder
Make handling encoded chunks easier.
2019-06-12 13:37:01 +02:00
Dietmar Maurer ef27200cec src/backup/crypt_setup.rs: fix encoder for uncompressed chunks 2019-06-10 09:56:06 +02:00
Dietmar Maurer 74792b95b2 src/backup/crypt_setup.rs: allow compressed and uncompressed chunks 2019-06-10 08:27:35 +02:00
Dietmar Maurer 9e0187a203 src/backup/crypt_setup.rs: remove unused CryptData struct 2019-06-10 07:55:36 +02:00
Dietmar Maurer bec8498ae3 src/backup/crypt_setup.rs: avoid useless allocations 2019-06-09 11:44:17 +02:00
Dietmar Maurer 48b4b40b57 src/backup/crypt_setup.rs: crypto helpers 2019-06-08 09:56:51 +02:00
Wolfgang Bumiller dd5495d6dc tree-wide: use 'dyn' for all trait objects
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-07 13:13:48 +02:00
Dietmar Maurer 36898ffce6 src/backup/chunk_stream.rs: add optional chunk_size parameter 2019-05-30 13:28:24 +02:00
Dietmar Maurer 976595e1a9 src/backup/dynamic_index.rs: split class DynamicIndexWriter 2019-05-29 08:49:57 +02:00
Dietmar Maurer cb0708dd46 src/backup/fixed_index.rs: remove ChunkStat from struct 2019-05-29 07:08:34 +02:00
Dietmar Maurer 01af11f340 src/backup/fixed_index.rs: improve consistency checks 2019-05-29 07:02:45 +02:00
Dietmar Maurer 006f3ff407 src/api2/admin/datastore/backup/environment.rs: add more consistency checks 2019-05-28 09:21:31 +02:00
Dietmar Maurer fc14b84937 src/backup/fixed_index.rs: improve bound checks inside add_digest 2019-05-28 08:47:03 +02:00
Dietmar Maurer e1225de467 src/backup/fixed_index.rs: store index_length to avoid code duplication 2019-05-28 08:27:57 +02:00
Dietmar Maurer 29ae5c86a2 src/backup/fixed_index.rs: make chunk_size public 2019-05-28 06:18:24 +02:00
Wolfgang Bumiller 8ea3b1d188 tree-wide: use the new vec/io tools modules
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Dietmar Maurer 0cc0fffd1e src/backup/chunk_stream.rs: use BytesMut in ChunkStream 2019-05-22 09:39:02 +02:00
Dietmar Maurer c052be5c86 src/backup/chunk_stream.rs: use more generics 2019-05-22 09:18:05 +02:00
Dietmar Maurer 169c0e060f src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut) 2019-05-22 09:05:35 +02:00
Dietmar Maurer 3be3f3dcc4 src/backup/chunk_stream.rs: fix behaviour for large input buffers 2019-05-22 08:05:27 +02:00
Dietmar Maurer 02fa54ff3c src/backup/chunk_stream.rs: assemble data with correct order 2019-05-22 07:44:51 +02:00
Dietmar Maurer 7f3d2ffab9 src/backup/index.rs: rename ChunkListReader into DigestListEncoder 2019-05-21 12:28:44 +02:00
Dietmar Maurer ee53955f52 src/backup/index.rs: implement DigestListDecoder 2019-05-21 12:21:22 +02:00
Dietmar Maurer 40f4e198a8 src/backup/dynamic_index.rs: add chunk_info method 2019-05-21 09:52:19 +02:00
Dietmar Maurer ff77dbbea3 Revert "src/backup/chunk_stream.rs: simplify code"
This reverts commit e058744d8f.
The optimization was wrong, and produces wrong chunk boundaries.
2019-05-19 11:05:56 +02:00
Dietmar Maurer e058744d8f src/backup/chunk_stream.rs: simplify code 2019-05-18 10:55:46 +02:00
Dietmar Maurer 8a7cc7565f src/backup/chunk_stream.rs: implement fixed size chunk stream 2019-05-18 10:55:46 +02:00
Dietmar Maurer dafc27aeff src/backup/chunk_stream.rs: async chunk stream 2019-05-14 10:05:29 +02:00
Dietmar Maurer 457531e7fe src/backup/index.rs: new ChunkListReader 2019-05-11 16:01:42 +02:00
Dietmar Maurer 51a4f63ff7 src/backup/datastore.rs: move last_backup to BackupInfo 2019-05-11 12:07:09 +02:00
Dietmar Maurer b02a52e3a5 src/backup/datastore.rs: impl last_backup helper 2019-05-11 11:36:46 +02:00
Dietmar Maurer c09775011a src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups 2019-05-11 10:19:34 +02:00
Dietmar Maurer 35a2d8a6a6 src/backup/backup_info.rs: Derive Clone 2019-05-10 06:59:23 +02:00
Dietmar Maurer f2b99c34f7 src/api2/admin/datastore.rs: implement API to return last GC status 2019-04-11 12:04:25 +02:00
Dietmar Maurer d4b59ae0b8 src/backup/datastore.rs: pass WorkerTask to GC
So that we can log to the task log file.
2019-04-06 17:57:38 +02:00
Dietmar Maurer 92da93b245 abort GC on server shutdown 2019-04-01 12:13:02 +02:00
Dietmar Maurer 11861a482d src/backup/chunk_store.rs: fix GC
Added option to get oldest_writer timestamp from ProcessLocker.
2019-03-31 17:21:36 +02:00
Dietmar Maurer d85987aeeb fix last commit: the filename var was not ment to be removed, sorry 2019-03-31 16:16:14 +02:00
Dietmar Maurer 15a77c4c2e src/backup/chunk_store.rs: avoid create/unlink race 2019-03-31 10:03:01 +02:00
Dietmar Maurer 10eea49d81 src/backup/fixed_index.rs: store reference to file
Keep it open and locked while index is in use.
2019-03-27 10:17:03 +01:00
Dietmar Maurer c597a92c81 index file readers: aquire shared lock 2019-03-27 10:11:14 +01:00
Dietmar Maurer 43b1303398 datastore: use new ProcessLocker
To make sure only one process runs garbage collection while having active writers.
2019-03-22 09:42:15 +01:00
Dietmar Maurer 515688d1f1 avoid compiler warnings 2019-03-18 10:00:58 +01:00
Dietmar Maurer 8968258b66 rename catar into pxar
To avoid confusion with the casync implementation.
2019-03-14 10:54:09 +01:00
Dietmar Maurer 8e39232acc src/bin/proxmox-backup-client.rs: strip .didx file extensions 2019-03-11 10:54:02 +01:00
Wolfgang Bumiller 06178f13fa move chunker into proxmox_protocol
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-08 09:29:47 +01:00
Dietmar Maurer 141f062e08 src/backup/chunk_store.rs: use zstd compression insteadf of lz4
Provides better compressionm rate, and is still fast.
2019-03-07 11:42:59 +01:00
Dietmar Maurer d2690f74bb src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end 2019-03-07 10:40:01 +01:00
Dietmar Maurer 58e99e138e src/backup/backup_info.rs: new helper to list files 2019-03-06 11:02:08 +01:00
Wolfgang Bumiller e3062f87b1 fixed_index: split add_chunk/add_digest
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Wolfgang Bumiller c3bb97e59c fixed_index: impl Send for FixedIndexWriter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Wolfgang Bumiller 3b7ade9e78 backup/datastore: expose insert_chunk method
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Wolfgang Bumiller 391d310741 backup_info: BackupDir shortcut
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 10:15:06 +01:00
Wolfgang Bumiller 93b49ce38a backup_info: use two independent Into<String> types
there's no reason to force them to be the same input type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 09:13:08 +01:00
Dietmar Maurer d57474e055 src/backup/backup_info.rs: improve docs 2019-03-05 07:28:13 +01:00
Dietmar Maurer b3483782d1 src/backup/backup_info.rs: move code into separate file
Also changed create_backup_dir() parameters - uses &BackupDir now.
2019-03-05 07:18:12 +01:00
Dietmar Maurer 2b01a22507 src/backup/datastore.rs: add helper to sort backup lists 2019-03-04 18:20:57 +01:00
Dietmar Maurer 875fb1c01a src/backup/datastore.rs: use unix epoch to create DateTime
To make sure that we have a timestamp without nanosecond.
2019-03-04 17:58:22 +01:00
Dietmar Maurer 38f8815925 src/backup/datastore.rs: use new() function to create structs 2019-03-04 17:44:27 +01:00
Dietmar Maurer 9b492eb256 src/backup/datastore.rs: protect BackupDir fields, impl new() 2019-03-04 13:51:36 +01:00
Dietmar Maurer 1e9a94e579 src/backup/datastore.rs: protect BackupGroup fields, impl new() 2019-03-04 13:38:23 +01:00
Dietmar Maurer f72dfd2d10 src/backup/datastore.rs: make sure timestamp does not contains nanoseconds
Because we want simple, readable file names.
2019-03-04 09:37:10 +01:00
Dietmar Maurer 6f62c9240a src/api2/admin/datastore.rs: imp delete_snapshot 2019-03-03 11:29:00 +01:00
Dietmar Maurer 184f17afad src/api2/admin/datastore.rs: impl list_snapshots 2019-03-02 16:20:50 +01:00
Dietmar Maurer e909522f06 src/backup/datastore.rs: use timestamp with zone offset 2019-03-01 17:36:40 +01:00
Dietmar Maurer dc4c09faae src/backup/datastore.rs: split BackupDir 2019-03-01 16:26:35 +01:00
Dietmar Maurer 38b0dfa511 src/backup/datastore.rs: split BackupInfo 2019-03-01 16:26:35 +01:00
Wolfgang Bumiller dcd15d10bb backup/index: remove IndexIterator
removed the need for it, to avoid copies

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-01 13:29:36 +01:00
Dietmar Maurer 541b908ea4 src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch 2019-02-28 17:03:48 +01:00
Wolfgang Bumiller 5de2bced2d datastore: add generic open_index
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00