887018bb79
client: use default encryption key if it is available
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
9696f5193b
client: move key management into separate module
...
and use api macro for methods and Kdf type
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
e13c4f66bb
minor style & whitespace fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 10:55:25 +02:00
08cb2038bd
api: disks: indentation fixup
...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-03 17:59:30 +02:00
9de69cdb1a
src/bin/proxmox_backup_client/catalog.rs: split out catalog code
2020-07-03 16:45:47 +02:00
e9764238df
make ReadChunk not require mutable self.
...
That way we can reduce lock contentions because we lock for much shorter
times.
2020-07-03 07:37:29 +02:00
cc7995ac40
src/bin/proxmox_backup_client/task.rs: split out task command
2020-07-02 18:04:29 +02:00
43abba4b4f
src/bin/proxmox_backup_client/mount.rs: split out mount code
2020-07-02 17:49:59 +02:00
caea8d611f
proxmox-backup-client: add benchmark command
...
This is just a start, We need to add more useful things here...
2020-07-02 14:01:57 +02:00
7d0754a6d2
pxar: fixup 'vanished-file' logic a bit
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-30 14:41:42 +02:00
5afa0755ea
pxar: fix missing newlines in warnings
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-30 14:37:20 +02:00
2162e2c15d
src/api2/admin/datastore.rs: avoid slash in UPID strings
2020-06-30 13:11:22 +02:00
4059285649
fix typo
2020-06-29 12:59:25 +02:00
2e079b8bf2
partially revert commit 1f82f9b7b5
...
do it backwards compatible. Also, code was wrong because FixedIndexWriter
still computed old style csums...
2020-06-29 12:44:45 +02:00
a8e2940ff3
pxar: deal with files changing size during archiving
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-26 11:49:51 +02:00
4eb4e94918
fix test output
...
field separator for pools is always a tab when using -H
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 10:31:11 +02:00
817bcda848
src/backup/verify.rs: do not stop on server shutdown
...
This is a read-only task, so there is no need to stop.
2020-06-26 09:45:59 +02:00
f6de2c7359
WorkerTask: add warnings and count them
...
so that we have one level more between errors and OK
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 09:42:11 +02:00
402c8861d8
fix typo
...
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 09:12:29 +02:00
3f683799a8
improve 'debug' parameter
...
instead of checking on '1' or 'true', check that it is there and not
'0' and 'false'. this allows using simply
https://foo:8007/?debug
instead of
https://foo:8007/?debug=1
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 09:12:14 +02:00
1f82f9b7b5
src/backup/index.rs: add compute_csum
...
And use it for fixed and dynamic index. Please note that this
changes checksums for fixed indexes, so restore older backups
will fails now (not backward compatible).
2020-06-26 09:00:34 +02:00
fe3e65c3ea
src/api2/backup.rs: call register_chunk in previous download api
2020-06-26 08:22:46 +02:00
fdaab0df4e
src/backup/index.rs: add chunk_info method
2020-06-26 08:14:45 +02:00
b957aa81bd
update backup api for incremental backup
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-26 07:17:08 +02:00
8ea00f6e49
allow to abort verify jobs
...
And improve job description rendering on gui.
2020-06-25 12:56:36 +02:00
e22f4882e7
extract create_download_response API helper
...
and put it into a new "api2::helpers" module.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-25 11:57:37 +02:00
355c055e81
src/bin/proxmox-backup-manager.rs: implement verify
2020-06-24 13:35:21 +02:00
c2009e5309
src/api2/admin/datastore.rs: add verify api
2020-06-24 13:35:21 +02:00
23f74c190e
src/backup/backup_info.rs: impl Display for BackupGroup
2020-06-24 13:35:21 +02:00
a6f8728339
update to pxar 0.1.9, update ReadAt implementations
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-24 11:57:12 +02:00
facd9801cf
add incremental backup support
...
To support incremental backups (where not all chunks are sent to the
server), a new parameter "reuse-csum" is introduced on the
"create_fixed_index" API call. When set and equal to last backups'
checksum, the backup writer clones the data from the last index of this
archive file, and only updates chunks it actually receives.
In incremental mode some checks usually done on closing an index cannot
be made, since they would be inaccurate.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-24 10:01:25 +02:00
21302088de
remove debug println
2020-06-24 09:15:13 +02:00
8268c9d161
fix overflow panic during upload
...
if *only* data chunks are registered (high chance during incremental
backup), then chunk_count might be one lower then upload_stat.count
because of the zero chunk being unconditionally uploaded but not used.
Thus when subtracting the two, an overflow would occur.
In general, don't let the client make the server panic, instead just set
duplicates to 0.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-24 09:07:22 +02:00
b91b7d9ffd
api2/node/disks/zfs: check if default zfs mount path exists
...
and if it does bail, because otherwise we would get an
error on mounting and have a zpool that is not imported
and disks that are used
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:22:39 +02:00
d33d8f4e6a
api2/admin/datastore: add pxar-file-download api call
...
streams a file from a pxar file of an unencrypted backup
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:21:15 +02:00
5b1cfa01f1
api2/admin/datastore: add 'catalog' api call
...
returns the dir listing of the given filepath of the backup snapshot
the filepath has to be base64 encoded or 'root'
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:16:12 +02:00
05d18b907a
add From<&DirEntryAttribute to CatalogEntryType and make it pub(crate)
...
we want to get a string representation of the DirEntryAttribute
like 'f' for file, etc. and since we have such a mapping already
in the CatalogEntryType, use that
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:08:50 +02:00
e44fe0c9f5
derive Clone for the LocalChunkReader
...
this will be necessary for accessing local pxar behind didx files
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:07:28 +02:00
4cf0ced950
add LocalDynamicReadAt
...
mostly copied from BufferedDynamicReadAt from proxmox-backup-client
but the reader is wrapped in an Arc in addition to the Mutex
we will use this for local access to a pxar behind a didx file
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:05:31 +02:00
60f9a6ea8f
src/backup/datastore.rs: add new helpers to load blobs and verify chunks
2020-06-24 06:58:14 +02:00
1090fd4424
src/backup/data_blob.rs: cleanup - improve code reuse
2020-06-24 06:56:48 +02:00
92c3fd2e22
src/backup/chunk_store.rs: allow to read name()
...
This is helpful for logging ...
2020-06-24 06:54:21 +02:00
d6d3b353be
cleanup: implement FromStr for BackupGroup
2020-06-23 08:16:56 +02:00
a67f7d0a07
cleanup: implement FromStr for BackupDir
2020-06-23 08:09:52 +02:00
c8137518fe
src/bin/proxmox_backup_manager/disk.rs: add renderer for wearout
...
So that we display the same value as the GUI.
2020-06-23 07:44:09 +02:00
cbef49bf4f
remove absolute paths when executing binaries
...
we set the paths manually, so this is ok
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-23 07:09:06 +02:00
0b99e5aebc
remove debug prints
...
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-23 06:33:58 +02:00
f386f512d0
add AsyncReaderStream
...
and replace AsyncIndexReader's stream implementation with that
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-23 06:33:31 +02:00
3ddb14889a
src/tools/daemon.rs: reopen STDOUT/STDERR journald streams to get correct PID in logs
2020-06-22 13:06:53 +02:00
c08fac4d69
tools::daemon: sync with child after MainPid message
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-22 10:58:04 +02:00
c40440092d
tools: add socketpair helper
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-22 10:49:56 +02:00
dc2ef2b54f
tools::daemon: fetch exe name in the beginning
...
We get the path to our executable via a readlink() on
"/proc/self/exe", which appends a " (deleted)" during
package reloads.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-22 10:31:54 +02:00
f28cfb322a
avoid compiler warnings
2020-06-20 07:24:02 +02:00
3bbe291c51
zpool_status.rs - indented_list_to_tree: do not set name property
...
This is no necessary. We only touch/set 'children' and 'leaf' properties.
2020-06-20 07:19:25 +02:00
42d19fdf69
src/api2/node/disks/zfs.rs: always set pool name
2020-06-20 07:15:32 +02:00
215968e033
src/tools/disks/zpool_status.rs: add 'leaf' attribute to root node, rename 'prev' into 'parent'
2020-06-20 06:49:06 +02:00
eddd1a1b9c
src/tools/disks/zpool_status.rs: move use clause top of file
2020-06-20 06:17:22 +02:00
d2ce211899
fixup for previous commit
2020-06-20 06:15:26 +02:00
1cb46c6f65
src/tools/disks/zpool_status.rs - cleanup: use struct StackItem instead of tuple
2020-06-19 18:58:57 +02:00
5d88c3a1c8
src/tools/disks/zpool_status.rs: remove unnecessary checks
...
Thos things can never happen, so simply use unwrap().
2020-06-19 18:27:39 +02:00
07fb504943
src/tools/disks/zpool_status.rs: simplify code by using serde_json::to_value
2020-06-19 17:51:13 +02:00
f675c5e978
src/tools/disks/zpool_status.rs - add all attributes to the tree
2020-06-19 16:55:28 +02:00
4e37d9ce67
add general indented_list_to_tree implementation
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-19 14:37:40 +02:00
e303077132
lru_cache: restrict and annotate Send impl
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-19 09:37:34 +02:00
6ef9bb59eb
api2/admin/datastore: add download-decoded endpoint
...
similar to 'download', but streams the decoded file
when it is not encrypted
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:39:15 +02:00
eeaa2c212b
impl Sync for DataBlobReader
...
this is safe for the reason explained in the comment
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:37:47 +02:00
4a3adc3de8
add AsyncIndexReader
...
implements AsyncRead as well as Stream for an IndexFile and a store
that implements AsyncReadChunk
we can use this to asyncread or stream the content of a FixedIndex or
DynamicIndex
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:32:33 +02:00
abdb976340
add Display trait to BackupDir
...
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:28:35 +02:00
3b62116ce6
implement AsyncReadChunk for LocalChunkReader
...
same as the sync ReadChunk but uses tokio::fs::read instead
of file_get_contents
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:54:23 +02:00
1c090810f5
api2/admin/datastore/snapshos: show encrypted and size info per file
...
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:39:56 +02:00
e181d2f6da
add encrypted info to Manifest
...
we want to save if a file of a backup is encrypted, so that we can
* show that info on the gui
* can later decide if we need to decrypt the backup
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:35:39 +02:00
16021f6ab7
use the existing async method for read_raw_chunk
...
does the same, except the manual drop, but thats handled there by
letting the value go out of scope
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:23:17 +02:00
ba694720fc
api2/admin/datastore: log stream error during file download
...
the client cannot get an error during an chunked http transfer, so at
least log it server side
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 06:58:55 +02:00
bde8e243cf
remove unsafe copy code
...
copy_nonoverlapping is basically a memcpy which can also be done
via copy_from_slice which is not unsafe
(copy_from_slice uses copy_nonoverlapping internally)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 06:56:15 +02:00
3352ee5656
parse_zpool_status_field: handle tabs without copying input
2020-06-18 19:40:01 +02:00
b29cbc414d
parse_zpool_status_vdev: consider tabs as 8 spaces
2020-06-18 18:38:56 +02:00
026dc1d11f
src/api2/node/disks/zfs.rs: add zpool_details api
2020-06-18 15:04:46 +02:00
9438aca6c9
src/tools/disks/zpool_status.rs: improve parser
2020-06-18 14:55:22 +02:00
547f0c97e4
src/tools/nom.rs: new helper parse_complete_line() for single line parsers
...
Like parse_complete(), but generates simpler error messages.
2020-06-18 12:57:55 +02:00
177a2de992
src/tools/nom.rs: move nom helpers into separate file
2020-06-18 12:41:13 +02:00
0686b1f4db
src/tools/disks/zpool_list.rs: split code into separate file
2020-06-18 10:31:07 +02:00
0727e56a06
src/tools/disks/zpool_status.rs: parse zpool status output
2020-06-18 10:23:15 +02:00
2fd3d57490
src/tools/disks/zfs.rs: rename ZFSPoolStatus into ZFSPoolInfo, fix error message
2020-06-17 09:08:26 +02:00
3f851d1321
src/api2/node/disks/directory.rs: add early check if disk is unused
2020-06-17 08:31:11 +02:00
1aef491e24
src/bin/proxmox_backup_manager/disk.rs: add cli to create mounted disks
2020-06-17 08:07:54 +02:00
d0eccae37d
avoid compiler warning
2020-06-17 08:07:42 +02:00
a34154d900
src/tools/disks/zfs.rs: cleanup parse_pool_header
2020-06-17 07:47:11 +02:00
c2cc32b4dd
src/tools/disks/zfs.rs: add more parser tests
2020-06-17 07:38:19 +02:00
46405fa35d
src/tools/disks/zfs.rs: add comment
2020-06-17 07:14:26 +02:00
66af7f51bc
src/tools/disks/zfs.rs: make zfs list parser private
2020-06-17 07:00:54 +02:00
c72ccd4e33
src/tools/disks/zfs.rs: add regression tests for parse_zfs_list
2020-06-16 18:14:35 +02:00
902b2cc278
src/tools/disks/zfs.rs: simplify code
2020-06-16 17:51:17 +02:00
8ecd7c9c21
move api dump binaries back to src/bin for package building
...
they're required for docs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 14:48:01 +02:00
fb5a066500
src/api2/node/disks.rs: expose directory api
2020-06-16 13:36:32 +02:00
d19c96d507
move test binaries to examples/
...
These aren't installed and are only used for manual testing,
so there's no reason to force them to be built all the time.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 13:32:24 +02:00
929a13b357
src/api2/node/disks/zfs.rs: add zpool api
2020-06-16 13:25:53 +02:00
36c65ee0b0
src/tools/disks/zfs.rs: cleanup (rename usage properties)
...
And allow to parse zpool list output without -v flag.
2020-06-16 13:25:53 +02:00
3378fd9fe5
src/tools/disks/zfs.rs: parse more infos (dedup, fragmentation, health)
2020-06-16 13:25:53 +02:00
58c51cf3d9
avoid compiler warnings
2020-06-16 13:25:53 +02:00
5509b199fb
use new run_command helper
2020-06-16 13:25:53 +02:00
Wolfgang Bumiller
Thomas Lamprecht
Dietmar Maurer
Dominik Csapak
Stefan Reiter