d2690f74bb
src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end
2019-03-07 10:40:01 +01:00
9ac6ec868a
backup/chunk_store: split insert_chunk
...
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
3a50ddd0c9
backup/chunk_store: mark ChunkIterator as fused
...
Ranges are FusedIterators and it can be useful to propagate
that property to users.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
eff25ecaf2
backup/chunk_store: make percentage output optional
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
798f7fa065
src/backup/chunk_store.rs: return info about compressed chunk size
...
So that we can generate better statistics ..
2019-02-25 11:36:05 +01:00
78216a5ab1
src/backup/chunk_store.rs: use lz4 for compression
...
And depend on lz4 lib now.
2019-02-25 10:00:28 +01:00
247cdbce72
src/api2/admin/datastore/catar.rs: allow to configure chunk-size
2019-02-19 15:19:12 +01:00
62f2422f6a
backup/chunk_store: verify chunk file names
...
get_chunk_iterator() should skip over files which aren't an
actual chunk
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
9739aca41f
backup/chunk_store: open base dir in get_chunk_iterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
c7f481b642
backup: replace ChunkIterator
...
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
e5064ba607
simplify backup lib structure (pub use xxx:*), improve doc
2019-02-12 14:13:31 +01:00
82bc0ad40c
backup: fixup warnings and readd lost file type check
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:43:43 +01:00
9349d2a1d2
backup: cleanup long lines
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
fdd71f5286
backup: use ChunkIterator for sweep_unused_chunks
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
c9e7f4dc8b
backup: add optional progress callback to ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
4070096c6a
backup: add ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
e0a5d1ca2f
tests: chunk_store: cleanup .testdir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
2296860027
tools.rs: move digest_to_hex() into tools
2019-01-25 10:58:28 +01:00
332dcc2277
backup/chunk_store.rs: fix test case - use absolute path
2019-01-25 10:14:25 +01:00
68469eebb4
backup/chunk_store.rs: verify base path is absolute
2019-01-20 16:49:22 +01:00
060c4811a0
backup/chunk_store.rs: resize buffer by powers of two
2019-01-05 11:34:49 +01:00
df9973e8f5
backup/chunk_store.rs: avoid allocation in read_chunk
2019-01-04 17:16:56 +01:00
7394ca3e95
white space cleanups
2019-01-04 12:51:43 +01:00
96df2fb44b
backup/chunk_store.rs: implement read_chunk()
2019-01-04 12:50:54 +01:00
bc616633d5
backup/chunk_store.rs: only use one directory level
2019-01-04 10:35:22 +01:00
77703d95aa
implement garbage collection for .aidx files
2019-01-02 14:27:04 +01:00
a198d74fc0
cleanup: reduce compiler warnings
2018-12-25 13:29:27 +01:00
991de6007b
remove outdated comment
2018-12-25 13:24:50 +01:00
7ee2aa1b94
touch_chunk: use libc::lutimensat
2018-12-25 12:27:25 +01:00
2c32fdde86
move lookup_datastore() to backup/datastore.rs
2018-12-22 17:37:25 +01:00
64e53b2835
gather usage statistics during garbage collection
2018-12-22 16:58:16 +01:00
176e4af964
sweep_used_chunks: print percentage
2018-12-22 15:39:05 +01:00
7b2b40a893
use openssl for faster hashing
2018-12-22 14:31:59 +01:00
e95950e40a
chunk_store: reduce number of directories
...
Else, scans everything takes too long ...
2018-12-22 14:04:05 +01:00
03e4753d8e
fix mutability for chunk store
2018-12-21 12:15:26 +01:00
f7dd683b46
fix tests
2018-12-19 13:54:22 +01:00
277fc5a3ce
try to improve error messages
2018-12-19 13:40:26 +01:00
eae8aa3aa9
cleanups: avoid compiler warnings
2018-12-19 12:49:23 +01:00
6c20a13d3c
cleanup - avoid too much indentation
2018-12-19 12:40:26 +01:00
1c43c56b06
cleanup
2018-12-19 12:30:37 +01:00
2bf5f6b220
ChunkStore::sweep_used_chunks - use openat to speedup directory walks
2018-12-19 12:20:01 +01:00
1628a4c731
use timers with a signal for file locking
...
* rename lock_file -> open_file_locked,
* add lock_file as a function working on already-opened
files
* change timeout types to std::time::Duration
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-19 11:23:00 +01:00
365bb90f17
move lock_file to tools.rs
2018-12-19 10:02:24 +01:00
08481a0b60
impl sweep_used_chunks, first try
2018-12-19 09:51:33 +01:00
6ea3a0b7fc
move sweep_used_chunks to chunk_store
2018-12-18 11:18:55 +01:00
3d5c11e5d2
start impl garbage collection
2018-12-18 11:06:03 +01:00
af3e7d7551
fix chunk store file paths
2018-12-16 13:52:16 +01:00
606ce64bb0
image_index.rs:write idx of chunks (use mmap)
2018-12-15 14:51:05 +01:00
5ba6968922
add comment to make intensions more clear
2018-12-12 11:21:00 +01:00
ce55dbbcc6
simplify code
2018-12-09 09:42:17 +01:00
461e62fcdf
remove some compiler warnings
2018-12-08 14:55:54 +01:00
391a2e43ff
chunk_store.insert_chunk: return information about existing chunks
2018-12-08 11:25:11 +01:00
12bb93b3ad
code cleanup
2018-12-08 11:20:02 +01:00
022fb421b2
lock with timeout
...
clumsy, but I have node idea how to make it better
2018-12-08 10:57:09 +01:00
2989f6bf20
improve error messages
2018-12-08 08:38:42 +01:00
b8d4766a4d
code cleanup
2018-12-08 08:21:20 +01:00
45773720ab
depend on nix, use flock
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2018-12-07 18:14:07 +01:00
c5d82e5f34
chunk_store: use mutex
2018-12-07 16:12:45 +01:00
128b37feea
impl insert_chunk
2018-12-07 14:44:56 +01:00
35cf5daa53
start experiments with simple chunk store
2018-12-07 10:51:53 +01:00
Dietmar Maurer
Wolfgang Bumiller