Dietmar Maurer
1e9a94e579
src/backup/datastore.rs: protect BackupGroup fields, impl new()
2019-03-04 13:38:23 +01:00
Dietmar Maurer
f72dfd2d10
src/backup/datastore.rs: make sure timestamp does not contains nanoseconds
...
Because we want simple, readable file names.
2019-03-04 09:37:10 +01:00
Dietmar Maurer
6f62c9240a
src/api2/admin/datastore.rs: imp delete_snapshot
2019-03-03 11:29:00 +01:00
Dietmar Maurer
184f17afad
src/api2/admin/datastore.rs: impl list_snapshots
2019-03-02 16:20:50 +01:00
Dietmar Maurer
e909522f06
src/backup/datastore.rs: use timestamp with zone offset
2019-03-01 17:36:40 +01:00
Dietmar Maurer
dc4c09faae
src/backup/datastore.rs: split BackupDir
2019-03-01 16:26:35 +01:00
Dietmar Maurer
38b0dfa511
src/backup/datastore.rs: split BackupInfo
2019-03-01 16:26:35 +01:00
Wolfgang Bumiller
dcd15d10bb
backup/index: remove IndexIterator
...
removed the need for it, to avoid copies
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-01 13:29:36 +01:00
Dietmar Maurer
541b908ea4
src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch
2019-02-28 17:03:48 +01:00
Wolfgang Bumiller
5de2bced2d
datastore: add generic open_index
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
7bc1d72778
add IndexFile trait
...
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.
Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5be4065baf
make FixedIndexReader Send and update comments
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
fded74d0c7
style: BufferedReader -> BufferedRead
...
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Dietmar Maurer
8f57971714
src/api2/admin/datastore.rs: implement prune
2019-02-28 12:51:27 +01:00
Wolfgang Bumiller
8731e40a7f
backup/datastore: create_backup_dir: show if path existed
...
To enable asserting the creation of a new backup rather than
adding to an existing one.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-27 10:15:36 +01:00
Wolfgang Bumiller
f661f37427
backup/dynamic_index: split out add_chunk
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:50:45 +01:00
Wolfgang Bumiller
9ac6ec868a
backup/chunk_store: split insert_chunk
...
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
3a50ddd0c9
backup/chunk_store: mark ChunkIterator as fused
...
Ranges are FusedIterators and it can be useful to propagate
that property to users.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
eff25ecaf2
backup/chunk_store: make percentage output optional
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
d59397e60e
backup/datastore: expose chunk_store.get_chunk_iterator
...
will be used by the protocol handler to iterate over
existing chunks
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Dietmar Maurer
8eb1203bda
src/backup/chunk_stat.rs: log write speed
2019-02-25 13:21:00 +01:00
Dietmar Maurer
7e3365554e
src/backup/chunk_stat.rs: new struct to track chunk statistics
2019-02-25 12:52:10 +01:00
Dietmar Maurer
798f7fa065
src/backup/chunk_store.rs: return info about compressed chunk size
...
So that we can generate better statistics ..
2019-02-25 11:36:05 +01:00
Dietmar Maurer
78216a5ab1
src/backup/chunk_store.rs: use lz4 for compression
...
And depend on lz4 lib now.
2019-02-25 10:00:28 +01:00
Dietmar Maurer
f252ca654c
src/backup/chunker.rs: simplify break test
...
To make it a little bit faster ...
2019-02-22 13:02:11 +01:00
Dietmar Maurer
61834c1cbf
src/backup/chunker.rs: another try to speedup the chunker
2019-02-22 11:15:10 +01:00
Dietmar Maurer
8c75372b79
src/backup/datastore.rs: list_backups() - include list of index files
2019-02-20 13:37:44 +01:00
Dietmar Maurer
fa4ebf0fce
src/backup/chunker.rs: simplify code
2019-02-20 11:17:19 +01:00
Dietmar Maurer
1b9c959c32
src/backup/chunker.rs: change window size back to 48
...
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
2019-02-20 11:01:09 +01:00
Dietmar Maurer
247cdbce72
src/api2/admin/datastore/catar.rs: allow to configure chunk-size
2019-02-19 15:19:12 +01:00
Dietmar Maurer
c584aa21fd
src/backup/chunker.rs: start() - correctly store hash in self.h
2019-02-19 12:40:12 +01:00
Dietmar Maurer
cd1598302a
src/backup/chunker.rs: use 64 byte chunker window size
...
This results in faster code.
2019-02-19 12:27:37 +01:00
Dietmar Maurer
9a3e869cf5
src/backup/chunker.rs: simplify buzzhash start code
2019-02-19 12:17:17 +01:00
Dietmar Maurer
4d2881dd4c
src/backup/chunker.rs: correctly compute window index 'idx'
...
And add a simple test.
2019-02-18 16:59:23 +01:00
Dietmar Maurer
f0a6112451
backup/datastore.rs: improve error reporting
2019-02-15 14:33:03 +01:00
Dietmar Maurer
af6f80d3db
Revert "backup/datastore: fn open is a private implementation detail"
...
This reverts commit 3dacd9a660
.
I currently use this in some test ...
2019-02-14 13:13:49 +01:00
Wolfgang Bumiller
62f2422f6a
backup/chunk_store: verify chunk file names
...
get_chunk_iterator() should skip over files which aren't an
actual chunk
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
3dacd9a660
backup/datastore: fn open is a private implementation detail
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f
backup/chunk_store: open base dir in get_chunk_iterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
c7f481b642
backup: replace ChunkIterator
...
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Dietmar Maurer
e5064ba607
simplify backup lib structure (pub use xxx:*), improve doc
2019-02-12 14:13:31 +01:00
Dietmar Maurer
93d5d77952
rename ArchiveIndex to DynamicIndex
...
also changed the file extension from .aidx to .didx
2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd
rename ImageIndex to FixedIndex
...
also changed the file extension fron .iidx to .fidx
2019-02-12 11:50:45 +01:00
Wolfgang Bumiller
82bc0ad40c
backup: fixup warnings and readd lost file type check
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:43:43 +01:00
Wolfgang Bumiller
9349d2a1d2
backup: cleanup long lines
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
fdd71f5286
backup: use ChunkIterator for sweep_unused_chunks
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
c9e7f4dc8b
backup: add optional progress callback to ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
4070096c6a
backup: add ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
e0a5d1ca2f
tests: chunk_store: cleanup .testdir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
Dietmar Maurer
9f49fe1d5d
avoid compiler warnings
2019-01-30 18:25:37 +01:00