tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

tape/pool_writer: skip already backed up chunks in iterator

Browse Source 
currently, the iterator goes over *all* chunks of the index, even
those already backed up by a previous snapshots in the same tape
backup. this is bad since for each iterator, we stat each chunk to
sort by inode number. so to avoid stat'ing the same chunks over
and over for consecutive snapshots, add a 'skip_fn' to the iterator
and in the pool writer and check the catalog_set if we can skip it

this means we can drop the later check for the catalog_set
(since we don't modify that here)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
This commit is contained in:
Dominik Csapak
2022-02-17 14:50:40 +01:00
committed by Dietmar Maurer
parent f30757df50
commit dcd9c17fff
2 changed files with 17 additions and 15 deletions
Download Patch File Download Diff File Expand all files Collapse all files

13
src/tape/pool_writer/new_chunks_iterator.rs
View File

@ -34,11 +34,16 @@ impl NewChunksIterator {
let mut chunk_index: HashSet<[u8;32]> = HashSet::new();
let datastore_name = snapshot_reader.datastore_name();
let datastore_name = snapshot_reader.datastore_name().to_string();
let result: Result<(), Error> = proxmox_lang::try_block!({
let mut chunk_iter = snapshot_reader.chunk_iterator()?;
let mut chunk_iter = snapshot_reader.chunk_iterator(move |digest| {
catalog_set
.lock()
.unwrap()
.contains_chunk(&datastore_name, digest)
})?;
loop {
let digest = match chunk_iter.next() {
@ -53,10 +58,6 @@ impl NewChunksIterator {
continue;
}
if catalog_set.lock().unwrap().contains_chunk(datastore_name, &digest) {
continue;
};
let blob = datastore.load_chunk(&digest)?;
//println!("LOAD CHUNK {}", hex::encode(&digest));
match tx.send(Ok(Some((digest, blob)))) {