39f18b30b6
src/backup/data_blob.rs: new load_from_reader(), which verifies the CRC
...
And make verify_crc private for now. We always call load_from_reader() to
verify the CRC.
Also add load_chunk() to datastore.rs (from chunk_store::read_chunk())
2020-07-28 10:23:16 +02:00
b96b11cdb7
chunk_store: Fix typo in bail message
...
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-07-21 12:51:41 +02:00
c687da9e8e
datastore: chown base dir on creation
...
When creating a new datastore the basedir is only owned by the backup
user if it did not exist beforehand (create_path chowns only if it
creates the directory), and returns false if it did not create the
directory).
This improves the experience when adding a new datastore on a fresh
disk or existing directory (not owned by backup) - backups/pulls can
be run instead of terminating with EPERM.
Tested on my local testinstall with a new disk, and a existing directory:
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-07-09 18:20:16 +02:00
92c3fd2e22
src/backup/chunk_store.rs: allow to read name()
...
This is helpful for logging ...
2020-06-24 06:54:21 +02:00
07ce44a633
avoid compiler warnings
2020-05-19 07:03:41 +02:00
99641a6bbb
garbage_collect: call fail_on_abort to abort GV when requested.
2020-05-05 09:06:34 +02:00
f7d4e4b506
switch from failure to anyhow
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
cf459b1982
gc: log pending removals
2020-04-06 09:50:40 +02:00
a92830dc39
src/api2/types.rs: define and use api type GarbageCollectionStatus
2020-01-23 13:40:12 +01:00
2585a8a4e2
src/backup/chunk_store.rs: implement cond_touch_chunk()
...
This will be used by backup sync to test if a chunk already exists.
2020-01-02 13:26:28 +01:00
645995634a
src/api2/config/datastore.rs - create: pass uid and gid instead of User
2019-12-20 09:23:58 +01:00
e67770d496
src/backup/chunk_store.rs - create: pass User instead of CreateOptions
2019-12-20 09:11:40 +01:00
afdcfb5bc9
let ChunkStore::create take CreateOptions
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-19 13:14:49 +01:00
f74a03da1f
remove tools::getpwnam_ugid, impl. crate::backup::backup_user()
...
And use new nix::unistd::User struct.
2019-12-19 10:20:13 +01:00
7e210bd0b4
src/backup/chunk_store.rs: create lock file with correct owner
2019-12-19 06:55:53 +01:00
0b97bc6158
src/backup/chunk_store.rs: use proxmox::tools::fs::create_path
2019-12-18 12:26:43 +01:00
14f1e63067
chunk_store: create parent directories
...
'datastore create storename /path/to/dir/that/may/not/exist' should
work.
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-12-17 15:39:42 +01:00
62ee2eb405
avoid some clippy warnings
2019-10-26 11:42:05 +02:00
4ee8f53d07
remove DataChunk file format - use DataBlob instead
2019-10-06 10:31:06 +02:00
a24e3993e0
src/backup/chunk_store.rs: coding style fixes
2019-07-04 11:39:10 +02:00
e4c2fbf170
src/backup/chunk_store.rs: additionally log chunk count
2019-07-04 11:27:11 +02:00
9850bcdf19
src/backup/chunk_store.rs: improve error reporting
2019-07-04 11:21:54 +02:00
a3f3e91da2
backup/chunk_store: rework chunk iterator
...
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-04 10:55:17 +02:00
a57360983b
src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item
2019-07-04 09:26:44 +02:00
a660978c9a
src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats
2019-07-04 07:57:43 +02:00
81a6ce6fde
src/backup/chunk_store.rs: new method chunk_path()
...
Returns the absolute path.
2019-06-28 15:48:09 +02:00
11515438cc
Cargo.toml: use serde feature derive
2019-06-18 06:23:25 +02:00
bffd40d6b7
src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools
2019-06-14 11:40:04 +02:00
f98ac774ee
backup: Add support for client side encryption
...
first try ...
2019-06-13 11:47:23 +02:00
36898ffce6
src/backup/chunk_stream.rs: add optional chunk_size parameter
2019-05-30 13:28:24 +02:00
f2b99c34f7
src/api2/admin/datastore.rs: implement API to return last GC status
2019-04-11 12:04:25 +02:00
92da93b245
abort GC on server shutdown
2019-04-01 12:13:02 +02:00
11861a482d
src/backup/chunk_store.rs: fix GC
...
Added option to get oldest_writer timestamp from ProcessLocker.
2019-03-31 17:21:36 +02:00
d85987aeeb
fix last commit: the filename var was not ment to be removed, sorry
2019-03-31 16:16:14 +02:00
15a77c4c2e
src/backup/chunk_store.rs: avoid create/unlink race
2019-03-31 10:03:01 +02:00
43b1303398
datastore: use new ProcessLocker
...
To make sure only one process runs garbage collection while having active writers.
2019-03-22 09:42:15 +01:00
141f062e08
src/backup/chunk_store.rs: use zstd compression insteadf of lz4
...
Provides better compressionm rate, and is still fast.
2019-03-07 11:42:59 +01:00
d2690f74bb
src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end
2019-03-07 10:40:01 +01:00
9ac6ec868a
backup/chunk_store: split insert_chunk
...
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
3a50ddd0c9
backup/chunk_store: mark ChunkIterator as fused
...
Ranges are FusedIterators and it can be useful to propagate
that property to users.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
eff25ecaf2
backup/chunk_store: make percentage output optional
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
798f7fa065
src/backup/chunk_store.rs: return info about compressed chunk size
...
So that we can generate better statistics ..
2019-02-25 11:36:05 +01:00
78216a5ab1
src/backup/chunk_store.rs: use lz4 for compression
...
And depend on lz4 lib now.
2019-02-25 10:00:28 +01:00
247cdbce72
src/api2/admin/datastore/catar.rs: allow to configure chunk-size
2019-02-19 15:19:12 +01:00
62f2422f6a
backup/chunk_store: verify chunk file names
...
get_chunk_iterator() should skip over files which aren't an
actual chunk
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
9739aca41f
backup/chunk_store: open base dir in get_chunk_iterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
c7f481b642
backup: replace ChunkIterator
...
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
e5064ba607
simplify backup lib structure (pub use xxx:*), improve doc
2019-02-12 14:13:31 +01:00
82bc0ad40c
backup: fixup warnings and readd lost file type check
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:43:43 +01:00
9349d2a1d2
backup: cleanup long lines
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Dietmar Maurer
Aaron Lauterer
Stoiko Ivanov
Wolfgang Bumiller
Oguz Bektas