tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
801 Commits 2 Branches 54 Tags 14 MiB
15a77c4c2e
Commit Graph

63 Commits

Author SHA1 Message Date
Dietmar Maurer
15a77c4c2e src/backup/chunk_store.rs: avoid create/unlink race 2019-03-31 10:03:01 +02:00
Dietmar Maurer
43b1303398 datastore: use new ProcessLocker 
To make sure only one process runs garbage collection while having active writers.
 2019-03-22 09:42:15 +01:00
Dietmar Maurer
141f062e08 src/backup/chunk_store.rs: use zstd compression insteadf of lz4 
Provides better compressionm rate, and is still fast.
 2019-03-07 11:42:59 +01:00
Dietmar Maurer
d2690f74bb src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end 2019-03-07 10:40:01 +01:00
Wolfgang Bumiller
9ac6ec868a backup/chunk_store: split insert_chunk 
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
3a50ddd0c9 backup/chunk_store: mark ChunkIterator as fused 
Ranges are FusedIterators and it can be useful to propagate
that property to users.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
eff25ecaf2 backup/chunk_store: make percentage output optional 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Dietmar Maurer
798f7fa065 src/backup/chunk_store.rs: return info about compressed chunk size 
So that we can generate better statistics ..
 2019-02-25 11:36:05 +01:00
Dietmar Maurer
78216a5ab1 src/backup/chunk_store.rs: use lz4 for compression 
And depend on lz4 lib now.
 2019-02-25 10:00:28 +01:00
Dietmar Maurer
247cdbce72 src/api2/admin/datastore/catar.rs: allow to configure chunk-size 2019-02-19 15:19:12 +01:00
Wolfgang Bumiller
62f2422f6a backup/chunk_store: verify chunk file names 
get_chunk_iterator() should skip over files which aren't an
actual chunk

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f backup/chunk_store: open base dir in get_chunk_iterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
c7f481b642 backup: replace ChunkIterator 
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Dietmar Maurer
e5064ba607 simplify backup lib structure (pub use xxx:*), improve doc 2019-02-12 14:13:31 +01:00
Wolfgang Bumiller
82bc0ad40c backup: fixup warnings and readd lost file type check 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:43:43 +01:00
Wolfgang Bumiller
9349d2a1d2 backup: cleanup long lines 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
fdd71f5286 backup: use ChunkIterator for sweep_unused_chunks 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
c9e7f4dc8b backup: add optional progress callback to ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
4070096c6a backup: add ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
e0a5d1ca2f tests: chunk_store: cleanup .testdir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Dietmar Maurer
2296860027 tools.rs: move digest_to_hex() into tools 2019-01-25 10:58:28 +01:00
Dietmar Maurer
332dcc2277 backup/chunk_store.rs: fix test case - use absolute path 2019-01-25 10:14:25 +01:00
Dietmar Maurer
68469eebb4 backup/chunk_store.rs: verify base path is absolute 2019-01-20 16:49:22 +01:00
Dietmar Maurer
060c4811a0 backup/chunk_store.rs: resize buffer by powers of two 2019-01-05 11:34:49 +01:00
Dietmar Maurer
df9973e8f5 backup/chunk_store.rs: avoid allocation in read_chunk 2019-01-04 17:16:56 +01:00
Dietmar Maurer
7394ca3e95 white space cleanups 2019-01-04 12:51:43 +01:00
Dietmar Maurer
96df2fb44b backup/chunk_store.rs: implement read_chunk() 2019-01-04 12:50:54 +01:00
Dietmar Maurer
bc616633d5 backup/chunk_store.rs: only use one directory level 2019-01-04 10:35:22 +01:00
Dietmar Maurer
77703d95aa implement garbage collection for .aidx files 2019-01-02 14:27:04 +01:00
Dietmar Maurer
a198d74fc0 cleanup: reduce compiler warnings 2018-12-25 13:29:27 +01:00
Dietmar Maurer
991de6007b remove outdated comment 2018-12-25 13:24:50 +01:00
Dietmar Maurer
7ee2aa1b94 touch_chunk: use libc::lutimensat 2018-12-25 12:27:25 +01:00
Dietmar Maurer
2c32fdde86 move lookup_datastore() to backup/datastore.rs 2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835 gather usage statistics during garbage collection 2018-12-22 16:58:16 +01:00
Dietmar Maurer
176e4af964 sweep_used_chunks: print percentage 2018-12-22 15:39:05 +01:00
Dietmar Maurer
7b2b40a893 use openssl for faster hashing 2018-12-22 14:31:59 +01:00
Dietmar Maurer
e95950e40a chunk_store: reduce number of directories 
Else, scans everything takes too long ...
 2018-12-22 14:04:05 +01:00
Dietmar Maurer
03e4753d8e fix mutability for chunk store 2018-12-21 12:15:26 +01:00
Dietmar Maurer
f7dd683b46 fix tests 2018-12-19 13:54:22 +01:00
Dietmar Maurer
277fc5a3ce try to improve error messages 2018-12-19 13:40:26 +01:00
Dietmar Maurer
eae8aa3aa9 cleanups: avoid compiler warnings 2018-12-19 12:49:23 +01:00
Dietmar Maurer
6c20a13d3c cleanup - avoid too much indentation 2018-12-19 12:40:26 +01:00
Dietmar Maurer
1c43c56b06 cleanup 2018-12-19 12:30:37 +01:00
Dietmar Maurer
2bf5f6b220 ChunkStore::sweep_used_chunks - use openat to speedup directory walks 2018-12-19 12:20:01 +01:00
Wolfgang Bumiller
1628a4c731 use timers with a signal for file locking 
* rename lock_file -> open_file_locked,
* add lock_file as a function working on already-opened
  files
* change timeout types to std::time::Duration

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-19 11:23:00 +01:00
Dietmar Maurer
365bb90f17 move lock_file to tools.rs 2018-12-19 10:02:24 +01:00
Dietmar Maurer
08481a0b60 impl sweep_used_chunks, first try 2018-12-19 09:51:33 +01:00
Dietmar Maurer
6ea3a0b7fc move sweep_used_chunks to chunk_store 2018-12-18 11:18:55 +01:00
Dietmar Maurer
3d5c11e5d2 start impl garbage collection 2018-12-18 11:06:03 +01:00
Dietmar Maurer
af3e7d7551 fix chunk store file paths 2018-12-16 13:52:16 +01:00