tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,218 Commits 2 Branches 54 Tags 14 MiB
0f0a35b390
Commit Graph

241 Commits

Author SHA1 Message Date
Dietmar Maurer
9e0187a203 src/backup/crypt_setup.rs: remove unused CryptData struct 2019-06-10 07:55:36 +02:00
Dietmar Maurer
bec8498ae3 src/backup/crypt_setup.rs: avoid useless allocations 2019-06-09 11:44:17 +02:00
Dietmar Maurer
48b4b40b57 src/backup/crypt_setup.rs: crypto helpers 2019-06-08 09:56:51 +02:00
Wolfgang Bumiller
dd5495d6dc tree-wide: use 'dyn' for all trait objects 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-06-07 13:13:48 +02:00
Dietmar Maurer
36898ffce6 src/backup/chunk_stream.rs: add optional chunk_size parameter 2019-05-30 13:28:24 +02:00
Dietmar Maurer
976595e1a9 src/backup/dynamic_index.rs: split class DynamicIndexWriter 2019-05-29 08:49:57 +02:00
Dietmar Maurer
cb0708dd46 src/backup/fixed_index.rs: remove ChunkStat from struct 2019-05-29 07:08:34 +02:00
Dietmar Maurer
01af11f340 src/backup/fixed_index.rs: improve consistency checks 2019-05-29 07:02:45 +02:00
Dietmar Maurer
006f3ff407 src/api2/admin/datastore/backup/environment.rs: add more consistency checks 2019-05-28 09:21:31 +02:00
Dietmar Maurer
fc14b84937 src/backup/fixed_index.rs: improve bound checks inside add_digest 2019-05-28 08:47:03 +02:00
Dietmar Maurer
e1225de467 src/backup/fixed_index.rs: store index_length to avoid code duplication 2019-05-28 08:27:57 +02:00
Dietmar Maurer
29ae5c86a2 src/backup/fixed_index.rs: make chunk_size public 2019-05-28 06:18:24 +02:00
Wolfgang Bumiller
8ea3b1d188 tree-wide: use the new vec/io tools modules 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-23 09:49:10 +02:00
Dietmar Maurer
0cc0fffd1e src/backup/chunk_stream.rs: use BytesMut in ChunkStream 2019-05-22 09:39:02 +02:00
Dietmar Maurer
c052be5c86 src/backup/chunk_stream.rs: use more generics 2019-05-22 09:18:05 +02:00
Dietmar Maurer
169c0e060f src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut) 2019-05-22 09:05:35 +02:00
Dietmar Maurer
3be3f3dcc4 src/backup/chunk_stream.rs: fix behaviour for large input buffers 2019-05-22 08:05:27 +02:00
Dietmar Maurer
02fa54ff3c src/backup/chunk_stream.rs: assemble data with correct order 2019-05-22 07:44:51 +02:00
Dietmar Maurer
7f3d2ffab9 src/backup/index.rs: rename ChunkListReader into DigestListEncoder 2019-05-21 12:28:44 +02:00
Dietmar Maurer
ee53955f52 src/backup/index.rs: implement DigestListDecoder 2019-05-21 12:21:22 +02:00
Dietmar Maurer
40f4e198a8 src/backup/dynamic_index.rs: add chunk_info method 2019-05-21 09:52:19 +02:00
Dietmar Maurer
ff77dbbea3 Revert "src/backup/chunk_stream.rs: simplify code" 
This reverts commit e058744d8f.
The optimization was wrong, and produces wrong chunk boundaries.
 2019-05-19 11:05:56 +02:00
Dietmar Maurer
e058744d8f src/backup/chunk_stream.rs: simplify code 2019-05-18 10:55:46 +02:00
Dietmar Maurer
8a7cc7565f src/backup/chunk_stream.rs: implement fixed size chunk stream 2019-05-18 10:55:46 +02:00
Dietmar Maurer
dafc27aeff src/backup/chunk_stream.rs: async chunk stream 2019-05-14 10:05:29 +02:00
Dietmar Maurer
457531e7fe src/backup/index.rs: new ChunkListReader 2019-05-11 16:01:42 +02:00
Dietmar Maurer
51a4f63ff7 src/backup/datastore.rs: move last_backup to BackupInfo 2019-05-11 12:07:09 +02:00
Dietmar Maurer
b02a52e3a5 src/backup/datastore.rs: impl last_backup helper 2019-05-11 11:36:46 +02:00
Dietmar Maurer
c09775011a src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups 2019-05-11 10:19:34 +02:00
Dietmar Maurer
35a2d8a6a6 src/backup/backup_info.rs: Derive Clone 2019-05-10 06:59:23 +02:00
Dietmar Maurer
f2b99c34f7 src/api2/admin/datastore.rs: implement API to return last GC status 2019-04-11 12:04:25 +02:00
Dietmar Maurer
d4b59ae0b8 src/backup/datastore.rs: pass WorkerTask to GC 
So that we can log to the task log file.
 2019-04-06 17:57:38 +02:00
Dietmar Maurer
92da93b245 abort GC on server shutdown 2019-04-01 12:13:02 +02:00
Dietmar Maurer
11861a482d src/backup/chunk_store.rs: fix GC 
Added option to get oldest_writer timestamp from ProcessLocker.
 2019-03-31 17:21:36 +02:00
Dietmar Maurer
d85987aeeb fix last commit: the filename var was not ment to be removed, sorry 2019-03-31 16:16:14 +02:00
Dietmar Maurer
15a77c4c2e src/backup/chunk_store.rs: avoid create/unlink race 2019-03-31 10:03:01 +02:00
Dietmar Maurer
10eea49d81 src/backup/fixed_index.rs: store reference to file 
Keep it open and locked while index is in use.
 2019-03-27 10:17:03 +01:00
Dietmar Maurer
c597a92c81 index file readers: aquire shared lock 2019-03-27 10:11:14 +01:00
Dietmar Maurer
43b1303398 datastore: use new ProcessLocker 
To make sure only one process runs garbage collection while having active writers.
 2019-03-22 09:42:15 +01:00
Dietmar Maurer
515688d1f1 avoid compiler warnings 2019-03-18 10:00:58 +01:00
Dietmar Maurer
8968258b66 rename catar into pxar 
To avoid confusion with the casync implementation.
 2019-03-14 10:54:09 +01:00
Dietmar Maurer
8e39232acc src/bin/proxmox-backup-client.rs: strip .didx file extensions 2019-03-11 10:54:02 +01:00
Wolfgang Bumiller
06178f13fa move chunker into proxmox_protocol 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-08 09:29:47 +01:00
Dietmar Maurer
141f062e08 src/backup/chunk_store.rs: use zstd compression insteadf of lz4 
Provides better compressionm rate, and is still fast.
 2019-03-07 11:42:59 +01:00
Dietmar Maurer
d2690f74bb src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end 2019-03-07 10:40:01 +01:00
Dietmar Maurer
58e99e138e src/backup/backup_info.rs: new helper to list files 2019-03-06 11:02:08 +01:00
Wolfgang Bumiller
e3062f87b1 fixed_index: split add_chunk/add_digest 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
c3bb97e59c fixed_index: impl Send for FixedIndexWriter 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
3b7ade9e78 backup/datastore: expose insert_chunk method 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
391d310741 backup_info: BackupDir shortcut 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-05 10:15:06 +01:00
Wolfgang Bumiller
93b49ce38a backup_info: use two independent Into<String> types 
there's no reason to force them to be the same input type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-05 09:13:08 +01:00
Dietmar Maurer
d57474e055 src/backup/backup_info.rs: improve docs 2019-03-05 07:28:13 +01:00
Dietmar Maurer
b3483782d1 src/backup/backup_info.rs: move code into separate file 
Also changed create_backup_dir() parameters - uses &BackupDir now.
 2019-03-05 07:18:12 +01:00
Dietmar Maurer
2b01a22507 src/backup/datastore.rs: add helper to sort backup lists 2019-03-04 18:20:57 +01:00
Dietmar Maurer
875fb1c01a src/backup/datastore.rs: use unix epoch to create DateTime 
To make sure that we have a timestamp without nanosecond.
 2019-03-04 17:58:22 +01:00
Dietmar Maurer
38f8815925 src/backup/datastore.rs: use new() function to create structs 2019-03-04 17:44:27 +01:00
Dietmar Maurer
9b492eb256 src/backup/datastore.rs: protect BackupDir fields, impl new() 2019-03-04 13:51:36 +01:00
Dietmar Maurer
1e9a94e579 src/backup/datastore.rs: protect BackupGroup fields, impl new() 2019-03-04 13:38:23 +01:00
Dietmar Maurer
f72dfd2d10 src/backup/datastore.rs: make sure timestamp does not contains nanoseconds 
Because we want simple, readable file names.
 2019-03-04 09:37:10 +01:00
Dietmar Maurer
6f62c9240a src/api2/admin/datastore.rs: imp delete_snapshot 2019-03-03 11:29:00 +01:00
Dietmar Maurer
184f17afad src/api2/admin/datastore.rs: impl list_snapshots 2019-03-02 16:20:50 +01:00
Dietmar Maurer
e909522f06 src/backup/datastore.rs: use timestamp with zone offset 2019-03-01 17:36:40 +01:00
Dietmar Maurer
dc4c09faae src/backup/datastore.rs: split BackupDir 2019-03-01 16:26:35 +01:00
Dietmar Maurer
38b0dfa511 src/backup/datastore.rs: split BackupInfo 2019-03-01 16:26:35 +01:00
Wolfgang Bumiller
dcd15d10bb backup/index: remove IndexIterator 
removed the need for it, to avoid copies

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-01 13:29:36 +01:00
Dietmar Maurer
541b908ea4 src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch 2019-02-28 17:03:48 +01:00
Wolfgang Bumiller
5de2bced2d datastore: add generic open_index 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
7bc1d72778 add IndexFile trait 
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.

Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5be4065baf make FixedIndexReader Send and update comments 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
fded74d0c7 style: BufferedReader -> BufferedRead 
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Dietmar Maurer
8f57971714 src/api2/admin/datastore.rs: implement prune 2019-02-28 12:51:27 +01:00
Wolfgang Bumiller
8731e40a7f backup/datastore: create_backup_dir: show if path existed 
To enable asserting the creation of a new backup rather than
adding to an existing one.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-27 10:15:36 +01:00
Wolfgang Bumiller
f661f37427 backup/dynamic_index: split out add_chunk 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:50:45 +01:00
Wolfgang Bumiller
9ac6ec868a backup/chunk_store: split insert_chunk 
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
3a50ddd0c9 backup/chunk_store: mark ChunkIterator as fused 
Ranges are FusedIterators and it can be useful to propagate
that property to users.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
eff25ecaf2 backup/chunk_store: make percentage output optional 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
d59397e60e backup/datastore: expose chunk_store.get_chunk_iterator 
will be used by the protocol handler to iterate over
existing chunks

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Dietmar Maurer
8eb1203bda src/backup/chunk_stat.rs: log write speed 2019-02-25 13:21:00 +01:00
Dietmar Maurer
7e3365554e src/backup/chunk_stat.rs: new struct to track chunk statistics 2019-02-25 12:52:10 +01:00
Dietmar Maurer
798f7fa065 src/backup/chunk_store.rs: return info about compressed chunk size 
So that we can generate better statistics ..
 2019-02-25 11:36:05 +01:00
Dietmar Maurer
78216a5ab1 src/backup/chunk_store.rs: use lz4 for compression 
And depend on lz4 lib now.
 2019-02-25 10:00:28 +01:00
Dietmar Maurer
f252ca654c src/backup/chunker.rs: simplify break test 
To make it a little bit faster ...
 2019-02-22 13:02:11 +01:00
Dietmar Maurer
61834c1cbf src/backup/chunker.rs: another try to speedup the chunker 2019-02-22 11:15:10 +01:00
Dietmar Maurer
8c75372b79 src/backup/datastore.rs: list_backups() - include list of index files 2019-02-20 13:37:44 +01:00
Dietmar Maurer
fa4ebf0fce src/backup/chunker.rs: simplify code 2019-02-20 11:17:19 +01:00
Dietmar Maurer
1b9c959c32 src/backup/chunker.rs: change window size back to 48 
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
 2019-02-20 11:01:09 +01:00
Dietmar Maurer
247cdbce72 src/api2/admin/datastore/catar.rs: allow to configure chunk-size 2019-02-19 15:19:12 +01:00
Dietmar Maurer
c584aa21fd src/backup/chunker.rs: start() - correctly store hash in self.h 2019-02-19 12:40:12 +01:00
Dietmar Maurer
cd1598302a src/backup/chunker.rs: use 64 byte chunker window size 
This results in faster code.
 2019-02-19 12:27:37 +01:00
Dietmar Maurer
9a3e869cf5 src/backup/chunker.rs: simplify buzzhash start code 2019-02-19 12:17:17 +01:00
Dietmar Maurer
4d2881dd4c src/backup/chunker.rs: correctly compute window index 'idx' 
And add a simple test.
 2019-02-18 16:59:23 +01:00
Dietmar Maurer
f0a6112451 backup/datastore.rs: improve error reporting 2019-02-15 14:33:03 +01:00
Dietmar Maurer
af6f80d3db Revert "backup/datastore: fn open is a private implementation detail" 
This reverts commit 3dacd9a660.

I currently use this in some test ...
 2019-02-14 13:13:49 +01:00
Wolfgang Bumiller
62f2422f6a backup/chunk_store: verify chunk file names 
get_chunk_iterator() should skip over files which aren't an
actual chunk

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
3dacd9a660 backup/datastore: fn open is a private implementation detail 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f backup/chunk_store: open base dir in get_chunk_iterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
c7f481b642 backup: replace ChunkIterator 
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Dietmar Maurer
e5064ba607 simplify backup lib structure (pub use xxx:*), improve doc 2019-02-12 14:13:31 +01:00
Dietmar Maurer
93d5d77952 rename ArchiveIndex to DynamicIndex 
also changed the file extension from .aidx to .didx
 2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd rename ImageIndex to FixedIndex 
also changed the file extension fron .iidx to .fidx
 2019-02-12 11:50:45 +01:00