tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
3,595 Commits 2 Branches 54 Tags 14 MiB
e1c1533790
Commit Graph

605 Commits

Author SHA1 Message Date
Dietmar Maurer
247cdbce72 src/api2/admin/datastore/catar.rs: allow to configure chunk-size 2019-02-19 15:19:12 +01:00
Dietmar Maurer
c584aa21fd src/backup/chunker.rs: start() - correctly store hash in self.h 2019-02-19 12:40:12 +01:00
Dietmar Maurer
cd1598302a src/backup/chunker.rs: use 64 byte chunker window size 
This results in faster code.
 2019-02-19 12:27:37 +01:00
Dietmar Maurer
9a3e869cf5 src/backup/chunker.rs: simplify buzzhash start code 2019-02-19 12:17:17 +01:00
Dietmar Maurer
4d2881dd4c src/backup/chunker.rs: correctly compute window index 'idx' 
And add a simple test.
 2019-02-18 16:59:23 +01:00
Dietmar Maurer
f0a6112451 backup/datastore.rs: improve error reporting 2019-02-15 14:33:03 +01:00
Dietmar Maurer
af6f80d3db Revert "backup/datastore: fn open is a private implementation detail" 
This reverts commit 3dacd9a660.

I currently use this in some test ...
 2019-02-14 13:13:49 +01:00
Wolfgang Bumiller
62f2422f6a backup/chunk_store: verify chunk file names 
get_chunk_iterator() should skip over files which aren't an
actual chunk

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
3dacd9a660 backup/datastore: fn open is a private implementation detail 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f backup/chunk_store: open base dir in get_chunk_iterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
c7f481b642 backup: replace ChunkIterator 
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Dietmar Maurer
e5064ba607 simplify backup lib structure (pub use xxx:*), improve doc 2019-02-12 14:13:31 +01:00
Dietmar Maurer
93d5d77952 rename ArchiveIndex to DynamicIndex 
also changed the file extension from .aidx to .didx
 2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd rename ImageIndex to FixedIndex 
also changed the file extension fron .iidx to .fidx
 2019-02-12 11:50:45 +01:00
Wolfgang Bumiller
82bc0ad40c backup: fixup warnings and readd lost file type check 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:43:43 +01:00
Wolfgang Bumiller
9349d2a1d2 backup: cleanup long lines 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
fdd71f5286 backup: use ChunkIterator for sweep_unused_chunks 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
c9e7f4dc8b backup: add optional progress callback to ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
4070096c6a backup: add ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
e0a5d1ca2f tests: chunk_store: cleanup .testdir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Dietmar Maurer
9f49fe1d5d avoid compiler warnings 2019-01-30 18:25:37 +01:00
Dietmar Maurer
2296860027 tools.rs: move digest_to_hex() into tools 2019-01-25 10:58:28 +01:00
Dietmar Maurer
332dcc2277 backup/chunk_store.rs: fix test case - use absolute path 2019-01-25 10:14:25 +01:00
Dietmar Maurer
784252dbc5 backup/datastore.rs: use single lazy_static block 2019-01-24 14:58:47 +01:00
Dietmar Maurer
7ca80246ea backup/datastore.rs: use DateTime<Utc> to store backup time 2019-01-21 10:51:52 +01:00
Dietmar Maurer
be0084b001 white space fix 2019-01-21 10:32:07 +01:00
Dietmar Maurer
e25736b4c3 backup/datastore.rs: implement list_backups 2019-01-20 18:02:27 +01:00
Dietmar Maurer
68469eebb4 backup/chunk_store.rs: verify base path is absolute 2019-01-20 16:49:22 +01:00
Dietmar Maurer
0b05fd5830 api3/admin/datastore/catar.rs: implement download 2019-01-20 09:39:32 +01:00
Dietmar Maurer
6a4c091616 api3/admin/datastore/catar.rs: start impl. download 2019-01-19 17:27:02 +01:00
Dietmar Maurer
728797d0c1 reduce compiler warnings 2019-01-18 16:50:15 +01:00
Dietmar Maurer
95cea65b04 backup/datastore.rs: list all index files using walkdir crate 2019-01-18 12:24:58 +01:00
Dietmar Maurer
ff3d3100d4 backup/datastore.rs: try to create useful directory layout 
store/type/ID/TIME/xxx.yyy
 2019-01-18 12:01:37 +01:00
Dietmar Maurer
150f1bd8f6 backup/datastore.rs: use Arc for chunk_store references 2019-01-15 12:36:16 +01:00
Dietmar Maurer
1629d2ad7b api3/admin/datastore/upload_catar.rs: implement upload future 2019-01-15 11:38:26 +01:00
Dietmar Maurer
ddbdf80d51 avoid compiler warnings 2019-01-11 08:41:33 +01:00
Dietmar Maurer
373ef4a504 backup/archive_index.rs: correctly check for errors 2019-01-10 11:19:54 +01:00
Dietmar Maurer
4624fe29cc impl Read for BufferedArchiveReader 2019-01-06 10:04:45 +01:00
Dietmar Maurer
343370504b impl Seek for BufferedArchiveReader 2019-01-06 09:35:39 +01:00
Dietmar Maurer
318564ac03 buffered_read: return empty buffer on eof 2019-01-06 09:17:28 +01:00
Dietmar Maurer
0a72e26704 define and use BufferedReader trait 2019-01-05 17:28:20 +01:00
Dietmar Maurer
39c6bd86cc backup/archive_index.rs: implement BufferedArchiveReader 
Implement relativly fast random read using binary search.
 2019-01-05 14:47:56 +01:00
Dietmar Maurer
060c4811a0 backup/chunk_store.rs: resize buffer by powers of two 2019-01-05 11:34:49 +01:00
Dietmar Maurer
df9973e8f5 backup/chunk_store.rs: avoid allocation in read_chunk 2019-01-04 17:16:56 +01:00
Dietmar Maurer
7394ca3e95 white space cleanups 2019-01-04 12:51:43 +01:00
Dietmar Maurer
96df2fb44b backup/chunk_store.rs: implement read_chunk() 2019-01-04 12:50:54 +01:00
Dietmar Maurer
bc616633d5 backup/chunk_store.rs: only use one directory level 2019-01-04 10:35:22 +01:00
Dietmar Maurer
b663789bdb backup/archive_index.rs: pass correct size to unmap 2019-01-04 09:28:41 +01:00
Dietmar Maurer
48d0d35688 backup/image_index.rs: use little endian 2019-01-04 08:45:45 +01:00
Dietmar Maurer
050f8a6355 chunker.rs: make sure chunk_size_avg is a power of two 2019-01-03 15:10:54 +01:00
Dietmar Maurer
ea4ea34baf improve chunker speed (avoid modulo) 2019-01-03 14:35:19 +01:00
Dietmar Maurer
0b8e75ed16 image_index.rs: verify file size 2019-01-02 18:14:02 +01:00
Dietmar Maurer
77703d95aa implement garbage collection for .aidx files 2019-01-02 14:27:04 +01:00
Dietmar Maurer
a360f6fa2d image_index.rs: verify header magic/version 2019-01-02 13:13:13 +01:00
Dietmar Maurer
44b3f62b42 fix typo 2019-01-02 12:56:04 +01:00
Dietmar Maurer
5032b57b44 archive_index.rs: correctly write .aidx file 2019-01-02 12:54:40 +01:00
Dietmar Maurer
580dc84bac image_index.rs: print stats 2019-01-02 12:53:49 +01:00
Dietmar Maurer
5e7a09be0d src/backup/archive_index.rs: use close() instead of flush() 
Also pass a reference to the encoder.
 2019-01-02 11:02:56 +01:00
Dietmar Maurer
94a882e900 src/backup/archive_index.rs: implement flush() 2018-12-31 18:01:07 +01:00
Dietmar Maurer
0433db1949 src/backup/archive_index.rs: first try 2018-12-31 17:30:08 +01:00
Dietmar Maurer
cbdd8c54ae create backup mod in backup.rs, improve docu 2018-12-31 16:08:04 +01:00
Dietmar Maurer
cb4412b18e move chunker.rs to backup/ 2018-12-31 15:46:16 +01:00
Dietmar Maurer
bcd879cfb3 backup-client: allow to specify directories (prepare for catar backup) 2018-12-27 10:11:11 +01:00
Dietmar Maurer
a198d74fc0 cleanup: reduce compiler warnings 2018-12-25 13:29:27 +01:00
Dietmar Maurer
991de6007b remove outdated comment 2018-12-25 13:24:50 +01:00
Dietmar Maurer
7ee2aa1b94 touch_chunk: use libc::lutimensat 2018-12-25 12:27:25 +01:00
Dietmar Maurer
2c32fdde86 move lookup_datastore() to backup/datastore.rs 2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835 gather usage statistics during garbage collection 2018-12-22 16:58:16 +01:00
Dietmar Maurer
176e4af964 sweep_used_chunks: print percentage 2018-12-22 15:39:05 +01:00
Dietmar Maurer
7b2b40a893 use openssl for faster hashing 2018-12-22 14:31:59 +01:00
Dietmar Maurer
e95950e40a chunk_store: reduce number of directories 
Else, scans everything takes too long ...
 2018-12-22 14:04:05 +01:00
Dietmar Maurer
03e4753d8e fix mutability for chunk store 2018-12-21 12:15:26 +01:00
Dietmar Maurer
d62e6e2264 backup-client: use 1M chunks, make chunk_size configurable 2018-12-21 08:36:57 +01:00
Dietmar Maurer
f7dd683b46 fix tests 2018-12-19 13:54:22 +01:00
Dietmar Maurer
277fc5a3ce try to improve error messages 2018-12-19 13:40:26 +01:00
Dietmar Maurer
eae8aa3aa9 cleanups: avoid compiler warnings 2018-12-19 12:49:23 +01:00
Dietmar Maurer
6c20a13d3c cleanup - avoid too much indentation 2018-12-19 12:40:26 +01:00
Dietmar Maurer
1c43c56b06 cleanup 2018-12-19 12:30:37 +01:00
Dietmar Maurer
2bf5f6b220 ChunkStore::sweep_used_chunks - use openat to speedup directory walks 2018-12-19 12:20:01 +01:00
Wolfgang Bumiller
1628a4c731 use timers with a signal for file locking 
* rename lock_file -> open_file_locked,
* add lock_file as a function working on already-opened
  files
* change timeout types to std::time::Duration

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-19 11:23:00 +01:00
Dietmar Maurer
365bb90f17 move lock_file to tools.rs 2018-12-19 10:02:24 +01:00
Dietmar Maurer
08481a0b60 impl sweep_used_chunks, first try 2018-12-19 09:51:33 +01:00
Dietmar Maurer
6ea3a0b7fc move sweep_used_chunks to chunk_store 2018-12-18 11:18:55 +01:00
Dietmar Maurer
3d5c11e5d2 start impl garbage collection 2018-12-18 11:06:03 +01:00
Dietmar Maurer
529de6c7a3 start implementing DataStore 2018-12-17 13:00:39 +01:00
Dietmar Maurer
4818c8b6f7 start implementing ImageIndexReader 2018-12-16 14:44:44 +01:00
Dietmar Maurer
af3e7d7551 fix chunk store file paths 2018-12-16 13:52:16 +01:00
Dietmar Maurer
0cd9d4206f improve error messages 2018-12-16 13:43:19 +01:00
Dietmar Maurer
4fbb72a8b4 image_index.rs: implement atomic write/rename 2018-12-16 13:39:21 +01:00
Dietmar Maurer
798881a68a rebane ImageIndex to ImageIndexWriter 2018-12-16 11:51:16 +01:00
Dietmar Maurer
5e5b7f1c2d use u64 to store time 2018-12-16 11:48:03 +01:00
Dietmar Maurer
d13e3745b3 image_index.rs: write header 2018-12-15 17:05:49 +01:00
Dietmar Maurer
606ce64bb0 image_index.rs:write idx of chunks (use mmap) 2018-12-15 14:51:05 +01:00
Dietmar Maurer
5ba6968922 add comment to make intensions more clear 2018-12-12 11:21:00 +01:00
Dietmar Maurer
ce55dbbcc6 simplify code 2018-12-09 09:42:17 +01:00
Dietmar Maurer
461e62fcdf remove some compiler warnings 2018-12-08 14:55:54 +01:00
Dietmar Maurer
391a2e43ff chunk_store.insert_chunk: return information about existing chunks 2018-12-08 11:25:11 +01:00
Dietmar Maurer
12bb93b3ad code cleanup 2018-12-08 11:20:02 +01:00
Dietmar Maurer
022fb421b2 lock with timeout 
clumsy, but I have node idea how to make it better
 2018-12-08 10:57:09 +01:00
Dietmar Maurer
2989f6bf20 improve error messages 2018-12-08 08:38:42 +01:00
Dietmar Maurer
b8d4766a4d code cleanup 2018-12-08 08:21:20 +01:00
Dietmar Maurer
45773720ab depend on nix, use flock 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2018-12-07 18:14:07 +01:00
Dietmar Maurer
c5d82e5f34 chunk_store: use mutex 2018-12-07 16:12:45 +01:00
Dietmar Maurer
128b37feea impl insert_chunk 2018-12-07 14:44:56 +01:00
Dietmar Maurer
35cf5daa53 start experiments with simple chunk store 2018-12-07 10:51:53 +01:00