Wolfgang Bumiller
e7cb4dc50d
introduce Username, Realm and Userid api types
...
and begin splitting up types.rs as it has grown quite large
already
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 12:05:01 +02:00
Oguz Bektas
2f57a433b1
fix #2909 : handle missing chunks gracefully in garbage collection
...
instead of bailing and stopping the entire GC process, warn about the
missing chunks and continue.
this results in "TASK WARNINGS: X" as the status.
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-08-06 06:36:48 +02:00
Aaron Lauterer
d3d566f7bd
GC: use time pre phase1 to calculate min_atime in phase2
...
Used chunks are marked in phase1 of the garbage collection process by
using the atime property. Each used chunk gets touched so that the atime
gets updated (if older than 24h, see relatime).
Should there ever be a situation in which the phase1 in the GC run needs
a very long time to finish, it could happen that the grace period
calculated in phase2 is not long enough and thus the marking of the
chunks (atime) becomes invalid. This would result in the removal of
needed chunks.
Even though the likelyhood of this happening is very low, using the
timestamp from right before phase1 is started, to calculate the grace
period in phase2 should avoid this situation.
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-08-04 10:19:05 +02:00
Fabian Grünbichler
8819d1f2f5
blobs: attempt to verify on decode when possible
...
regular chunks are only decoded when their contents are accessed, in
which case we need to have the key anyway and want to verify the digest.
for blobs we need to verify beforehand, since their checksums are always
calculated based on their raw content, and stored in the manifest.
manifests are also stored as blobs, but don't have a digest in the
traditional sense (they might have a signature covering parts of their
contents, but that is verified already when loading the manifest).
this commit does not cover pull/sync code which copies blobs and chunks
as-is without decoding them.
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-04 07:27:56 +02:00
Dietmar Maurer
ff86ef00a7
cleanup: manifest is always CryptMode::None
2020-07-31 10:25:30 +02:00
Dietmar Maurer
e443902583
src/backup/datastore.rs: add helpers to load/store manifest
...
We want this to modify the manifest "unprotected" data, for example
to add upload statistics, notes, ...
2020-07-31 07:45:47 +02:00
Dietmar Maurer
1fc82c41f2
src/api2/backup.rs: aquire backup lock earlier in create_locked_backup_group()
2020-07-30 11:03:05 +02:00
Stefan Reiter
c9756b40d1
datastore: prevent deletion of snaps in use as "previous backup"
...
To prevent a race with a background GC operation, do not allow deletion
of backups who's index might currently be referenced as the "known chunk
list" for successive backups. Otherwise the GC could delete chunks it
thinks are no longer referenced, while at the same time telling the
client that it doesn't need to upload said chunks because they already
exist.
Additionally, prevent deletion of whole backup groups, if there are
snapshots contained that appear to be currently in-progress. This is
currently unlikely to trigger, as that function is only used for sync
jobs, but it's a useful safeguard either way.
Deleting a single snapshot has a 'force' parameter, which is necessary
to allow deleting incomplete snapshots on an aborted backup. Pruning
also sets force=true to avoid the check, since it calculates which
snapshots to keep on its own.
To avoid code duplication, the is_finished method is factored out.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:26:01 +02:00
Dietmar Maurer
39f18b30b6
src/backup/data_blob.rs: new load_from_reader(), which verifies the CRC
...
And make verify_crc private for now. We always call load_from_reader() to
verify the CRC.
Also add load_chunk() to datastore.rs (from chunk_store::read_chunk())
2020-07-28 10:23:16 +02:00
Thomas Lamprecht
c3b090ac8a
backup: list images: handle walkdir error, catch "lost+found"
...
We support using an ext4 mountpoint directly as datastore and even do
so ourself when creating one through the disk manage code.
Such ext4 ountpoints have a lost+found directory which only root can
traverse into. As the GC list images is done as backup:backup user
walkdir gets an error.
We cannot ignore just all permission errors, as they could lead to
missing some backup indexes and thus possibly sweeping more chunks
than desired. While *normally* that should not happen through our
stack, we had already user report that they do rsyncs to move a
datastore from old to new server and got the permission wrong.
So for now be still very strict, only allow a "lost+found" directory
as immediate child of the datastore base directory, nothing else.
If deemed safe, this can always be made less strict. Possibly by
filtering the known backup-types on the highest level first.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-22 16:01:55 +02:00
Thomas Lamprecht
c47e294ea7
datastore: fix typo
...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-22 15:04:14 +02:00
Wolfgang Bumiller
521a0acb2e
DataStore::load_manifest: also return CryptMode
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 09:19:53 +02:00
Dietmar Maurer
60f9a6ea8f
src/backup/datastore.rs: add new helpers to load blobs and verify chunks
2020-06-24 06:58:14 +02:00
Dietmar Maurer
1610c45a86
src/client/pull.rs: also download client.log.blob
2020-05-30 14:51:33 +02:00
Dietmar Maurer
8545480a31
src/bin/proxmox-backup-proxy.rs: add simple task scheduler for garbage collection
2020-05-20 08:59:45 +02:00
Dietmar Maurer
99641a6bbb
garbage_collect: call fail_on_abort to abort GV when requested.
2020-05-05 09:06:34 +02:00
Dietmar Maurer
54552dda59
implemnt backup ownership, improve datastore access permissions
2020-04-28 10:22:25 +02:00
Wolfgang Bumiller
f7d4e4b506
switch from failure to anyhow
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Dietmar Maurer
cf459b1982
gc: log pending removals
2020-04-06 09:50:40 +02:00
Dietmar Maurer
a92830dc39
src/api2/types.rs: define and use api type GarbageCollectionStatus
2020-01-23 13:40:12 +01:00
Dietmar Maurer
6abce6c2bb
src/backup/datastore.rs: remove_backup_(group/dir) - return Error instead of io::Error
2020-01-23 10:14:46 +01:00
Dietmar Maurer
8a1d68c8b9
src/backup/datastore.rs: improve error messages
2020-01-23 09:58:14 +01:00
Dietmar Maurer
41b373eced
src/backup/datastore.rs: new helpers
...
last_successful_backup: Returns the time of the last successful backup
group_path: Returns the absolute path for a backup_group
snapshot_path: Returns the absolute path for a backup_dir
2020-01-22 15:05:47 +01:00
Dietmar Maurer
4b4eba0b9e
src/api2/pull.rs: implement delete flag for vanished groups
2020-01-17 11:24:55 +01:00
Dietmar Maurer
d0187a51a9
src/section_config.rs - convert_to_array: optionally add digest
...
datastore::config() -> also return digest
remotes::config() -> also return digest
2020-01-14 12:57:03 +01:00
Dietmar Maurer
7759eef552
src/backup/datastore.rs: implement cleanup_backup_dir()
2020-01-05 15:15:12 +01:00
Dietmar Maurer
2585a8a4e2
src/backup/chunk_store.rs: implement cond_touch_chunk()
...
This will be used by backup sync to test if a chunk already exists.
2020-01-02 13:26:28 +01:00
Dietmar Maurer
1cf5178ac5
src/backup/datastore.rs: new helper try_shared_chunk_store_lock()
2020-01-02 11:00:33 +01:00
Dietmar Maurer
1e8da0a789
src/backup/manifest.rs: new helper archive_type()
2019-12-31 15:23:41 +01:00
Dietmar Maurer
868c585219
src/backup/datastore.rs: avoid divide by zero
2019-12-19 07:14:23 +01:00
Dietmar Maurer
73e57f244e
src/backup/datastore.rs - garbage-collection: fix average chunk size
2019-12-10 11:25:29 +01:00
Dietmar Maurer
4ee8f53d07
remove DataChunk file format - use DataBlob instead
2019-10-06 10:31:06 +02:00
Wolfgang Bumiller
b21851cc5c
clippy: collapse identical if branches
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:57:58 +02:00
Wolfgang Bumiller
367f002ee2
use statement cleanup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-16 12:28:40 +02:00
Dietmar Maurer
a57360983b
src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item
2019-07-04 09:26:44 +02:00
Dietmar Maurer
a7c72ad9eb
src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store
2019-07-04 08:13:16 +02:00
Dietmar Maurer
a660978c9a
src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats
2019-07-04 07:57:43 +02:00
Dietmar Maurer
d48a9955a5
src/backup/dynamic_index.rs: introduce ReadChunk trait
2019-06-28 16:35:00 +02:00
Dietmar Maurer
f98ac774ee
backup: Add support for client side encryption
...
first try ...
2019-06-13 11:47:23 +02:00
Dietmar Maurer
976595e1a9
src/backup/dynamic_index.rs: split class DynamicIndexWriter
2019-05-29 08:49:57 +02:00
Dietmar Maurer
51a4f63ff7
src/backup/datastore.rs: move last_backup to BackupInfo
2019-05-11 12:07:09 +02:00
Dietmar Maurer
b02a52e3a5
src/backup/datastore.rs: impl last_backup helper
2019-05-11 11:36:46 +02:00
Dietmar Maurer
c09775011a
src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups
2019-05-11 10:19:34 +02:00
Dietmar Maurer
f2b99c34f7
src/api2/admin/datastore.rs: implement API to return last GC status
2019-04-11 12:04:25 +02:00
Dietmar Maurer
d4b59ae0b8
src/backup/datastore.rs: pass WorkerTask to GC
...
So that we can log to the task log file.
2019-04-06 17:57:38 +02:00
Dietmar Maurer
92da93b245
abort GC on server shutdown
2019-04-01 12:13:02 +02:00
Dietmar Maurer
11861a482d
src/backup/chunk_store.rs: fix GC
...
Added option to get oldest_writer timestamp from ProcessLocker.
2019-03-31 17:21:36 +02:00
Dietmar Maurer
43b1303398
datastore: use new ProcessLocker
...
To make sure only one process runs garbage collection while having active writers.
2019-03-22 09:42:15 +01:00
Dietmar Maurer
515688d1f1
avoid compiler warnings
2019-03-18 10:00:58 +01:00
Dietmar Maurer
58e99e138e
src/backup/backup_info.rs: new helper to list files
2019-03-06 11:02:08 +01:00
Wolfgang Bumiller
3b7ade9e78
backup/datastore: expose insert_chunk method
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Dietmar Maurer
b3483782d1
src/backup/backup_info.rs: move code into separate file
...
Also changed create_backup_dir() parameters - uses &BackupDir now.
2019-03-05 07:18:12 +01:00
Dietmar Maurer
2b01a22507
src/backup/datastore.rs: add helper to sort backup lists
2019-03-04 18:20:57 +01:00
Dietmar Maurer
875fb1c01a
src/backup/datastore.rs: use unix epoch to create DateTime
...
To make sure that we have a timestamp without nanosecond.
2019-03-04 17:58:22 +01:00
Dietmar Maurer
38f8815925
src/backup/datastore.rs: use new() function to create structs
2019-03-04 17:44:27 +01:00
Dietmar Maurer
9b492eb256
src/backup/datastore.rs: protect BackupDir fields, impl new()
2019-03-04 13:51:36 +01:00
Dietmar Maurer
1e9a94e579
src/backup/datastore.rs: protect BackupGroup fields, impl new()
2019-03-04 13:38:23 +01:00
Dietmar Maurer
f72dfd2d10
src/backup/datastore.rs: make sure timestamp does not contains nanoseconds
...
Because we want simple, readable file names.
2019-03-04 09:37:10 +01:00
Dietmar Maurer
6f62c9240a
src/api2/admin/datastore.rs: imp delete_snapshot
2019-03-03 11:29:00 +01:00
Dietmar Maurer
184f17afad
src/api2/admin/datastore.rs: impl list_snapshots
2019-03-02 16:20:50 +01:00
Dietmar Maurer
e909522f06
src/backup/datastore.rs: use timestamp with zone offset
2019-03-01 17:36:40 +01:00
Dietmar Maurer
dc4c09faae
src/backup/datastore.rs: split BackupDir
2019-03-01 16:26:35 +01:00
Dietmar Maurer
38b0dfa511
src/backup/datastore.rs: split BackupInfo
2019-03-01 16:26:35 +01:00
Dietmar Maurer
541b908ea4
src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch
2019-02-28 17:03:48 +01:00
Wolfgang Bumiller
5de2bced2d
datastore: add generic open_index
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Dietmar Maurer
8f57971714
src/api2/admin/datastore.rs: implement prune
2019-02-28 12:51:27 +01:00
Wolfgang Bumiller
8731e40a7f
backup/datastore: create_backup_dir: show if path existed
...
To enable asserting the creation of a new backup rather than
adding to an existing one.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-27 10:15:36 +01:00
Wolfgang Bumiller
eff25ecaf2
backup/chunk_store: make percentage output optional
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
d59397e60e
backup/datastore: expose chunk_store.get_chunk_iterator
...
will be used by the protocol handler to iterate over
existing chunks
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Dietmar Maurer
8c75372b79
src/backup/datastore.rs: list_backups() - include list of index files
2019-02-20 13:37:44 +01:00
Dietmar Maurer
f0a6112451
backup/datastore.rs: improve error reporting
2019-02-15 14:33:03 +01:00
Dietmar Maurer
af6f80d3db
Revert "backup/datastore: fn open is a private implementation detail"
...
This reverts commit 3dacd9a660
.
I currently use this in some test ...
2019-02-14 13:13:49 +01:00
Wolfgang Bumiller
3dacd9a660
backup/datastore: fn open is a private implementation detail
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Dietmar Maurer
e5064ba607
simplify backup lib structure (pub use xxx:*), improve doc
2019-02-12 14:13:31 +01:00
Dietmar Maurer
93d5d77952
rename ArchiveIndex to DynamicIndex
...
also changed the file extension from .aidx to .didx
2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd
rename ImageIndex to FixedIndex
...
also changed the file extension fron .iidx to .fidx
2019-02-12 11:50:45 +01:00
Dietmar Maurer
9f49fe1d5d
avoid compiler warnings
2019-01-30 18:25:37 +01:00
Dietmar Maurer
784252dbc5
backup/datastore.rs: use single lazy_static block
2019-01-24 14:58:47 +01:00
Dietmar Maurer
7ca80246ea
backup/datastore.rs: use DateTime<Utc> to store backup time
2019-01-21 10:51:52 +01:00
Dietmar Maurer
be0084b001
white space fix
2019-01-21 10:32:07 +01:00
Dietmar Maurer
e25736b4c3
backup/datastore.rs: implement list_backups
2019-01-20 18:02:27 +01:00
Dietmar Maurer
6a4c091616
api3/admin/datastore/catar.rs: start impl. download
2019-01-19 17:27:02 +01:00
Dietmar Maurer
95cea65b04
backup/datastore.rs: list all index files using walkdir crate
2019-01-18 12:24:58 +01:00
Dietmar Maurer
ff3d3100d4
backup/datastore.rs: try to create useful directory layout
...
store/type/ID/TIME/xxx.yyy
2019-01-18 12:01:37 +01:00
Dietmar Maurer
150f1bd8f6
backup/datastore.rs: use Arc for chunk_store references
2019-01-15 12:36:16 +01:00
Dietmar Maurer
1629d2ad7b
api3/admin/datastore/upload_catar.rs: implement upload future
2019-01-15 11:38:26 +01:00
Dietmar Maurer
bc616633d5
backup/chunk_store.rs: only use one directory level
2019-01-04 10:35:22 +01:00
Dietmar Maurer
77703d95aa
implement garbage collection for .aidx files
2019-01-02 14:27:04 +01:00
Dietmar Maurer
0433db1949
src/backup/archive_index.rs: first try
2018-12-31 17:30:08 +01:00
Dietmar Maurer
bcd879cfb3
backup-client: allow to specify directories (prepare for catar backup)
2018-12-27 10:11:11 +01:00
Dietmar Maurer
a198d74fc0
cleanup: reduce compiler warnings
2018-12-25 13:29:27 +01:00
Dietmar Maurer
2c32fdde86
move lookup_datastore() to backup/datastore.rs
2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835
gather usage statistics during garbage collection
2018-12-22 16:58:16 +01:00
Dietmar Maurer
e95950e40a
chunk_store: reduce number of directories
...
Else, scans everything takes too long ...
2018-12-22 14:04:05 +01:00
Dietmar Maurer
03e4753d8e
fix mutability for chunk store
2018-12-21 12:15:26 +01:00
Dietmar Maurer
d62e6e2264
backup-client: use 1M chunks, make chunk_size configurable
2018-12-21 08:36:57 +01:00
Dietmar Maurer
277fc5a3ce
try to improve error messages
2018-12-19 13:40:26 +01:00
Dietmar Maurer
365bb90f17
move lock_file to tools.rs
2018-12-19 10:02:24 +01:00
Dietmar Maurer
6ea3a0b7fc
move sweep_used_chunks to chunk_store
2018-12-18 11:18:55 +01:00
Dietmar Maurer
3d5c11e5d2
start impl garbage collection
2018-12-18 11:06:03 +01:00