Dietmar Maurer
045e8a568d
src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key
2019-06-25 07:00:07 +02:00
Dietmar Maurer
79bc7345e8
src/cli/command.rs: fix ReST output format
2019-06-25 06:19:51 +02:00
Dietmar Maurer
9f46c7de4b
src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup
2019-06-24 13:56:37 +02:00
Christian Ebner
cd7dc87903
pxar: impl .pxarexclude parsing and exclude matching
...
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.
Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-24 10:26:20 +02:00
Dietmar Maurer
46bd880041
src/api2/backup/environment.rs: new helper add_blob()
2019-06-24 09:35:37 +02:00
Dietmar Maurer
cb08ac3efe
src/api2/backup.rs: replace upload_config with upload_blob
2019-06-23 11:44:30 +02:00
Dietmar Maurer
a38c5d4d12
src/backup/data_blob.rs: implement from_raw()
2019-06-23 09:35:44 +02:00
Dietmar Maurer
6f083b7a92
src/api2/backup/upload_chunk.rs: compute crc on upload
2019-06-23 09:04:42 +02:00
Dietmar Maurer
ba01828d38
try to use use proxmox::tools::io::ops::ReadExtOps
2019-06-22 16:29:10 +02:00
Dietmar Maurer
27042ce637
src/backup/file_formats.rs: improve docs
2019-06-22 15:45:36 +02:00
Dietmar Maurer
9f83e0f7ee
src/backup/crypt_config.rs: simplify decode interface
2019-06-22 13:24:29 +02:00
Dietmar Maurer
0066c6d972
src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead
2019-06-22 13:02:53 +02:00
Dietmar Maurer
ee8a7e8080
src/backup/crypt_config.rs: implement encrypt_to (output to writer)
2019-06-22 12:25:04 +02:00
Dietmar Maurer
991abfa8b4
src/backup/file_formats.rs: split out file format data
2019-06-22 09:12:25 +02:00
Dietmar Maurer
b7f4f27d6c
add crc field for binary blobs formats
2019-06-21 18:49:04 +02:00
Dietmar Maurer
a7f67a9a9c
depend on crc32fast
...
We can use this to compute fast checksums to test file integrity (not crypto safe).
2019-06-21 15:58:00 +02:00
Dietmar Maurer
3025b3a53c
src/backup/data_blob.rs: new file format for binary blobs
2019-06-21 11:32:07 +02:00
Dietmar Maurer
077a8cae63
src/backup/crypt_config.rs: make code more flexible
...
Allow to pass magic numbers as parameters.
2019-06-21 10:50:48 +02:00
Dietmar Maurer
c38266c18d
renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs
2019-06-21 09:51:18 +02:00
Dietmar Maurer
3ea8bfc9c0
src/bin/proxmox-backup-client.rs: add import-master-pubkey command
2019-06-21 07:34:17 +02:00
Dietmar Maurer
37c5a17505
src/bin/proxmox-backup-client.rs: helper to generate RSA master key
2019-06-21 07:12:19 +02:00
Dietmar Maurer
ab44acff57
src/backup/key_derivation.rs: add modified field to key file
2019-06-20 07:58:33 +02:00
Dietmar Maurer
6d0983dbe1
src/bin/proxmox-backup-client.rs: add keyfile parameter for backup
2019-06-19 17:16:41 +02:00
Dietmar Maurer
181f097af9
src/bin/proxmox-backup-client.rs - key API: pass kdf parameter
...
Allow to store keys without password.
2019-06-18 18:21:01 +02:00
Dietmar Maurer
9b06db4563
src/bin/proxmox-backup-client.rs: use file system path to reference keys
2019-06-18 17:38:50 +02:00
Dietmar Maurer
091540298e
src/bin/proxmox-backup-client.rs - create_key: always read password from tty
2019-06-18 17:27:52 +02:00
Dietmar Maurer
826f309bf5
src/backup/key_derivation.rs: move kdf code into separate file
2019-06-18 11:17:22 +02:00
Dietmar Maurer
11515438cc
Cargo.toml: use serde feature derive
2019-06-18 06:23:25 +02:00
Dietmar Maurer
ac71623483
src/bin/proxmox-backup-client.rs: implement "key change-passphrase"
2019-06-17 13:10:00 +02:00
Dietmar Maurer
f2401311b0
src/bin/proxmox-backup-client.rs: start key management
2019-06-17 10:33:24 +02:00
Dietmar Maurer
a7dd483097
src/backup/*_index.rs: used generated magic numbers
2019-06-14 14:58:37 +02:00
Dietmar Maurer
bffd40d6b7
src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools
2019-06-14 11:40:04 +02:00
Dietmar Maurer
00388226e6
src/api2/backup/environment.rs: log index checksums
2019-06-14 10:36:20 +02:00
Dietmar Maurer
9335d74eba
src/backup/fixed_index.rs: compute checksum over the index
2019-06-14 10:34:29 +02:00
Dietmar Maurer
16ff6b7cd8
src/backup/dynamic_index.rs: compute checksum over the index
2019-06-14 10:33:59 +02:00
Dietmar Maurer
36075475a6
src/api2/backup/environment.rs: log duplicate chunks
2019-06-14 07:12:30 +02:00
Dietmar Maurer
fa148dbd11
src/api2/backup/upload_chunk.rs: verify chunk data if possible
2019-06-14 06:23:46 +02:00
Dietmar Maurer
0f37577092
src/bin/cipherbench.rs: add test for zstd compression
2019-06-13 17:24:57 +02:00
Dietmar Maurer
9399c98f82
src/bin/cipherbench.rs: cipher speed test with large blocks
2019-06-13 17:16:43 +02:00
Dietmar Maurer
f640f23ac5
src/backup/data_chunk.rs: fix typo
2019-06-13 16:42:55 +02:00
Dietmar Maurer
dba72d5016
src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:56:50 +02:00
Dietmar Maurer
68ab376ae9
src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:47:54 +02:00
Dietmar Maurer
f98ac774ee
backup: Add support for client side encryption
...
first try ...
2019-06-13 11:47:23 +02:00
Dietmar Maurer
51929e4532
src/backup/data_chunk.rs - decode: make crypt_config optional
2019-06-13 11:44:38 +02:00
Dietmar Maurer
046bcb9773
src/backup/crypt_setup.rs: add comment for possible improvements
2019-06-13 07:35:59 +02:00
Dietmar Maurer
bd0e3c7cfa
src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs
2019-06-13 07:35:02 +02:00
Dietmar Maurer
6762db70d6
src/api2/types.rs: define CHUNK_DIGEST_SCHEMA
2019-06-13 07:30:42 +02:00
Dietmar Maurer
b595cb9d2c
src/backup/data_chunk.rs: Data Chunk builder
...
Make handling encoded chunks easier.
2019-06-12 13:37:01 +02:00
Dietmar Maurer
ef27200cec
src/backup/crypt_setup.rs: fix encoder for uncompressed chunks
2019-06-10 09:56:06 +02:00
Dietmar Maurer
74792b95b2
src/backup/crypt_setup.rs: allow compressed and uncompressed chunks
2019-06-10 08:27:35 +02:00
Dietmar Maurer
9e0187a203
src/backup/crypt_setup.rs: remove unused CryptData struct
2019-06-10 07:55:36 +02:00
Dietmar Maurer
bec8498ae3
src/backup/crypt_setup.rs: avoid useless allocations
2019-06-09 11:44:17 +02:00
Dietmar Maurer
c20b17b460
src/backup.rs: define magic numbers for chunk store file format
2019-06-08 10:34:20 +02:00
Dietmar Maurer
48b4b40b57
src/backup/crypt_setup.rs: crypto helpers
2019-06-08 09:56:51 +02:00
Wolfgang Bumiller
dd5495d6dc
tree-wide: use 'dyn' for all trait objects
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-07 13:13:48 +02:00
Christian Ebner
e993db91cd
pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder
...
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:20 +02:00
Christian Ebner
1f319e766e
src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:06 +02:00
Christian Ebner
9b77388f2a
src/pxar/encoder.rs: minor refactoring
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:55 +02:00
Christian Ebner
a4cc8eb7c5
src/pxar/encoder.rs: check if ioctl supported by filesystem
...
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:31 +02:00
Dietmar Maurer
4800df7ba6
src/api2/admin/datastore.rs: remove test-upload api
2019-06-07 07:52:11 +02:00
Dietmar Maurer
113898d7e9
src/api2/admin/datastore/upload.rs: remove unused upload api
2019-06-07 07:39:41 +02:00
Dietmar Maurer
4247fccb0f
src/client/http_client.rs: automatically close connection in finish
2019-06-05 11:41:50 +02:00
Wolfgang Bumiller
cb4426b348
make h2 client connection cancellable
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-05 09:45:59 +02:00
Dietmar Maurer
986bef16be
src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1
...
So that we can include it in static doc strings.
2019-06-05 08:41:20 +02:00
Dietmar Maurer
c9ec0956cf
src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1
2019-06-05 08:12:13 +02:00
Dietmar Maurer
44c54845b3
src/tools/xattr.rs: fix test when run as root
2019-06-05 07:57:42 +02:00
Dietmar Maurer
7773ccc11f
src/api2.rs: move backup api to /backup
2019-06-05 07:23:21 +02:00
Christian Ebner
97eeea3b4a
src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:29:02 +02:00
Dietmar Maurer
1ce2f7cfb9
src/pxar/encoder.rs: style fix
2019-06-05 06:27:39 +02:00
Christian Ebner
e7b60a16c8
pxar: impl storing/dumping/restoring of quota project ids
...
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:26:02 +02:00
Christian Ebner
7312ab9eef
src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID
...
Defines the types needed to store/restore quota project ids in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:23:06 +02:00
Christian Ebner
042babe4e7
src/tools/fs.rs: impl ioctl calls to get/set fsxattr
...
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:22:49 +02:00
Dietmar Maurer
22e5aa1d3e
src/bin/dump-backup-api.rs: helper to generate backup API docs
2019-06-04 13:12:42 +02:00
Dietmar Maurer
5e0f305142
src/api_schema/format.rs: depend on textwrap, new wrap_text helper
2019-06-04 12:42:02 +02:00
Dietmar Maurer
339ddfcbfa
src/cli/command.rs: move doc generator code to src/api_schema/format.rs
2019-06-04 12:32:22 +02:00
Dietmar Maurer
ef39bf95b3
src/client/http_client.rs - upload_config: improve error message
2019-06-03 10:53:09 +02:00
Dietmar Maurer
ec8a9bb953
src/bin/proxmox-backup-client.rs: implement upload_config
2019-06-03 10:39:44 +02:00
Dietmar Maurer
39d6846e1e
src/api2/admin/datastore/backup.rs: implement config file upload
2019-06-03 09:45:14 +02:00
Dietmar Maurer
1c0472e88f
avoid compiler warnings
2019-06-03 06:43:47 +02:00
Dietmar Maurer
36898ffce6
src/backup/chunk_stream.rs: add optional chunk_size parameter
2019-05-30 13:28:24 +02:00
Dietmar Maurer
49ef316bcd
src/bin/proxmox-backup-client.rs: backup_image() - pass correct chunk size
2019-05-30 12:52:12 +02:00
Dietmar Maurer
202cadc29e
src/bin/proxmox-backup-client.rs: remove unused code
2019-05-30 12:47:35 +02:00
Dietmar Maurer
6af905c18f
src/bin/proxmox-backup-client.rs: implement image/block-device upload
2019-05-30 12:46:01 +02:00
Dietmar Maurer
eb1804c567
src/bin/proxmox-backup-client.rs: use std lib instead of nix
2019-05-30 12:19:23 +02:00
Dietmar Maurer
6e907505aa
src/bin/proxmox-backup-client.rs: fix file type test
2019-05-30 12:08:34 +02:00
Dietmar Maurer
96482891ae
src/api2/admin/datastore/backup/environment.rs: log upload statistics
2019-05-30 09:21:19 +02:00
Dietmar Maurer
642322b433
src/api2/admin/datastore/backup.rs: use separate api entry points for chunk upload
...
So that we can provide better statistics (patches follows).
2019-05-30 09:21:19 +02:00
Wolfgang Bumiller
390e83c9b9
typo fix and doc improvement
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:57:28 +02:00
Wolfgang Bumiller
7fb49397d5
doc fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:56:14 +02:00
Wolfgang Bumiller
382609b04e
tools: add Cancellable, start a futures submodule
...
To make a future cancellable, use:
let (future, canceller) =
crate::tools::futures::cancellable(future);
Proceed with using `future` as usual, `canceller` is
clonable and can cancel the future via the `.cancel()`
method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Wolfgang Bumiller
0d32d71fb7
tools: add AsyncMutex::new_locked
...
Allows creating a pre-locked mutex, returning the mutex and
a guard.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Dietmar Maurer
e6389f4e75
src/api2/admin/datastore/backup/environment.rs: do not allow empty backups (no files)
2019-05-29 10:38:57 +02:00
Dietmar Maurer
39e60bd6fa
src/api2/admin/datastore/backup.rs: improve debug logging
2019-05-29 10:17:38 +02:00
Dietmar Maurer
4ae6551c78
src/api_schema/router.rs: disable debug println
2019-05-29 09:42:16 +02:00
Dietmar Maurer
a42d1f55c3
src/api2/admin/datastore/backup.rs: add debug flag
2019-05-29 09:35:21 +02:00
Dietmar Maurer
976595e1a9
src/backup/dynamic_index.rs: split class DynamicIndexWriter
2019-05-29 08:49:57 +02:00
Dietmar Maurer
cb0708dd46
src/backup/fixed_index.rs: remove ChunkStat from struct
2019-05-29 07:08:34 +02:00
Dietmar Maurer
01af11f340
src/backup/fixed_index.rs: improve consistency checks
2019-05-29 07:02:45 +02:00
Christian Ebner
9b38443314
pxar: change logic and impl all command line flags for xattrs/fcaps/acls
...
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-29 06:51:20 +02:00
Dietmar Maurer
60e589a111
src/api2/admin/datastore/backup.rs: log sucessful finish call
2019-05-28 12:55:22 +02:00
Dietmar Maurer
c4ff3dcefd
src/bin/proxmox-backup-client.rs: use new BackupClient to upload data
2019-05-28 10:12:44 +02:00
Dietmar Maurer
10241c20ea
src/tools.rs: implement image_size helper
2019-05-28 09:35:08 +02:00
Dietmar Maurer
006f3ff407
src/api2/admin/datastore/backup/environment.rs: add more consistency checks
2019-05-28 09:21:31 +02:00
Dietmar Maurer
3dc5b2a203
src/client/http_client.rs: use start of chunk as offset
2019-05-28 09:21:31 +02:00
Dietmar Maurer
8de20e5c1f
src/client/http_client.rs: code cleanup
2019-05-28 08:53:27 +02:00
Dietmar Maurer
fc14b84937
src/backup/fixed_index.rs: improve bound checks inside add_digest
2019-05-28 08:47:03 +02:00
Dietmar Maurer
e1225de467
src/backup/fixed_index.rs: store index_length to avoid code duplication
2019-05-28 08:27:57 +02:00
Dietmar Maurer
e3d525feb7
src/api2/admin/datastore/backup.rs: fix typo
2019-05-28 07:13:19 +02:00
Christian Ebner
af309d4d5c
src/bin/pxar.rs: set archive permissions to 0o640
...
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:24:47 +02:00
Christian Ebner
ab87f167f1
src/pxar/encoder.rs: Refactor file stat
...
Introduce helper functions to check file stats
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:23:49 +02:00
Christian Ebner
578b601159
src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
...
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:22:16 +02:00
Dietmar Maurer
a42fa400ee
src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api
2019-05-28 06:18:55 +02:00
Dietmar Maurer
29ae5c86a2
src/backup/fixed_index.rs: make chunk_size public
2019-05-28 06:18:24 +02:00
Dietmar Maurer
7dd1bcac58
src/client/http_client.rs: correctly parse error response in download_chunk_list
2019-05-27 09:35:30 +02:00
Dietmar Maurer
a1e7cff3ea
src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk
2019-05-27 07:57:43 +02:00
Dietmar Maurer
771953f9eb
src/api2/admin/datastore/backup.rs: use smaller window sizes
2019-05-27 07:53:40 +02:00
Dietmar Maurer
174ad378d8
src/client/http_client.rs: fix request pipelining
2019-05-27 07:24:32 +02:00
Dietmar Maurer
624362226e
src/client/merge_known_chunks.rs: use MergedChunkInfo as input
2019-05-26 10:52:56 +02:00
Dietmar Maurer
05cba08c9c
src/client/http_client.rs: append chunks in correct order
...
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
2019-05-26 09:57:34 +02:00
Dietmar Maurer
417cb0731e
src/api2/admin/datastore/backup.rs: verify chunk offset
...
This clearly shows that the current approach does not work - seems we get
chunks out of order.
2019-05-26 08:49:29 +02:00
Dietmar Maurer
34114e2606
src/server/rest.rs: correctly verify json parameters
2019-05-24 09:10:59 +02:00
Dietmar Maurer
eb2bdd1bcc
src/client/http_client.rs: encode parameters inside body
...
To avoid problems with long uris.
2019-05-24 08:32:55 +02:00
Dietmar Maurer
9bb675ec23
src/client/merge_known_chunks.rs: reduce RAM usage
...
There is no need to store chunk data for known chunks.
2019-05-24 07:42:58 +02:00
Dietmar Maurer
ecb21b4794
src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough
2019-05-24 07:36:09 +02:00
Christian Ebner
2dbba78b98
src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:27:24 +02:00
Christian Ebner
6a19448004
src/pxar/encoder.rs: impl support to store ACLs in pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:21:33 +02:00
Christian Ebner
7cfaade7c8
src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:16:55 +02:00
Christian Ebner
b344461b33
pxar: pass feature_flags to encoder/decoder instead of individual flags
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner
687407741b
src/tools/xattr.rs: add test for is_valid_xattr_name()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner
357e4614e2
Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Wolfgang Bumiller
c05a8c8d18
doc-test fixup
...
cargo test by default compiles and runs all code snippets
found in the documentation...
oops...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 15:22:33 +02:00
Wolfgang Bumiller
cc84a830c5
RFC: schema: make enums static
...
I don't see a reason to allow these to be dynamically
modifiable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 12:38:51 +02:00
Dietmar Maurer
aa1b2e04fe
src/client/merge_known_chunks.rs: merge known chunks
...
To decrease the number of api calls required...
2019-05-23 12:31:09 +02:00
Dietmar Maurer
91320f0879
src/client/http_client.rs: use ChunkInfo streams
...
This will make out of order uploads possible...
2019-05-23 12:31:09 +02:00
Wolfgang Bumiller
8ea3b1d188
tree-wide: use the new vec/io tools modules
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Wolfgang Bumiller
9cdda3f7c7
tools: add helpful vector and read operations
...
After importing the I/O ops trait via:
use crate::tools::io::ops::*;
Instead of:
let mut buffer = vec![0u8; 65536];
file.read_exact(&mut buffer)?;
use:
let buffer = file.read_exact_allocated(65536)?;
After importing the vector helpers via:
use crate::tools::vec::{self, ops::*};
For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
let mut buffer = vec![0u8; len];
use:
let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Dietmar Maurer
8bea85b42e
src/api2/admin/datastore/backup.rs: verify file size and chunk count on close
2019-05-23 08:50:36 +02:00
Dietmar Maurer
1179e15842
src/server/rest.rs: improve error handling
2019-05-23 08:15:32 +02:00
Dietmar Maurer
907f5bb0d8
src/api2/admin/datastore/backup/service.rs: improve error logging
2019-05-23 08:05:39 +02:00
Dietmar Maurer
32e1bf4260
src/bin/upload-speed.rs: code cleanup
2019-05-23 07:20:05 +02:00
Christian Ebner
0d9bab05ab
src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:50 +02:00
Christian Ebner
4add706092
src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:32 +02:00
Christian Ebner
54b14ad741
src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:09 +02:00
Christian Ebner
1201360def
src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:39:16 +02:00
Christian Ebner
8bc8791338
src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:37:37 +02:00
Christian Ebner
a45a123950
src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
...
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:34:36 +02:00
Dietmar Maurer
d6f204eda6
src/client/http_client.rs: re-order code - public functions first
2019-05-23 06:28:10 +02:00
Dietmar Maurer
9af37c8f0e
src/client/http_client.rs: move low level H2 code into separate class
2019-05-22 17:28:25 +02:00
Dietmar Maurer
d2c48afc6e
src/client/http_client.rs: code cleanup
2019-05-22 13:24:33 +02:00
Dietmar Maurer
6ab34afa88
src/client/http_client.rs: cleanup h2 backup client
2019-05-22 13:05:51 +02:00
Dietmar Maurer
a55fcd740f
src/api2/admin/datastore/backup.rs: suse POST for finish() api
2019-05-22 13:04:58 +02:00
Dietmar Maurer
de6742c04f
src/bin/h2test.rs: removed
...
This test does no work anymore.
2019-05-22 11:14:56 +02:00
Dietmar Maurer
5a0b521809
src/api2/admin/datastore/backup.rs: remove test callbacks
2019-05-22 10:55:27 +02:00
Christian Ebner
de61bc9264
src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr
...
As git does not preserve xattrs they have to be set before they can be read.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-22 10:01:10 +02:00
Dietmar Maurer
553610b43e
src/client/http_client.rs: implement download_chunk_list
2019-05-22 09:46:02 +02:00
Dietmar Maurer
0cc0fffd1e
src/backup/chunk_stream.rs: use BytesMut in ChunkStream
2019-05-22 09:39:02 +02:00
Dietmar Maurer
c052be5c86
src/backup/chunk_stream.rs: use more generics
2019-05-22 09:18:05 +02:00
Dietmar Maurer
169c0e060f
src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut)
2019-05-22 09:05:35 +02:00
Dietmar Maurer
3be3f3dcc4
src/backup/chunk_stream.rs: fix behaviour for large input buffers
2019-05-22 08:05:27 +02:00
Dietmar Maurer
02fa54ff3c
src/backup/chunk_stream.rs: assemble data with correct order
2019-05-22 07:44:51 +02:00
Dietmar Maurer
d0afeaa831
src/api2/admin/datastore/backup.rs: improve error message
2019-05-22 07:43:54 +02:00
Dietmar Maurer
11867a2b54
src/tools.rs: fix hex_to_digest
2019-05-21 13:42:13 +02:00
Dietmar Maurer
7f3d2ffab9
src/backup/index.rs: rename ChunkListReader into DigestListEncoder
2019-05-21 12:28:44 +02:00
Dietmar Maurer
ee53955f52
src/backup/index.rs: implement DigestListDecoder
2019-05-21 12:21:22 +02:00
Dietmar Maurer
a95849321c
src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks
2019-05-21 09:54:11 +02:00
Dietmar Maurer
40f4e198a8
src/backup/dynamic_index.rs: add chunk_info method
2019-05-21 09:52:19 +02:00
Christian Ebner
596182056b
src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:15:30 +02:00
Christian Ebner
b303057a19
src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:06:26 +02:00
Christian Ebner
bee8d8ea78
src/tools/xattr.rs: add functions name_store and security_capability
...
These functions allow to check if the name for xattrs/fcaps are valid.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:05:31 +02:00
Dietmar Maurer
a09c0e38d8
src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks
2019-05-20 18:05:10 +02:00
Dietmar Maurer
97f03eff13
src/client/pxar_backup_stream.rs: fix docu
2019-05-20 17:29:00 +02:00
Dietmar Maurer
82ab72304e
src/client/http_client.rs: implement upload_stream
2019-05-20 14:19:24 +02:00
Dietmar Maurer
2698e8a514
src/client/pxar_backup_stream.rs: close pipe inside drop
2019-05-20 11:20:33 +02:00
Dietmar Maurer
f2269d8f99
src/tools.rs: implement hex_to_digest
2019-05-19 12:15:46 +02:00
Dietmar Maurer
751030aa7c
src/bin/test_chunk_speed2.rs: test max chunk size
2019-05-19 11:09:02 +02:00
Dietmar Maurer
ff77dbbea3
Revert "src/backup/chunk_stream.rs: simplify code"
...
This reverts commit e058744d8f
.
The optimization was wrong, and produces wrong chunk boundaries.
2019-05-19 11:05:56 +02:00
Dietmar Maurer
178ac042c4
src/api_schema/schema.rs: improve error messages
2019-05-19 10:18:51 +02:00
Dietmar Maurer
0c516b123d
src/client/pxar_backup_stream.rs: use WrappedStreamReader
...
to make it fully async ...
2019-05-18 15:13:31 +02:00
Dietmar Maurer
c6e28b66c6
src/client/pxar_backup_stream.rs: use 1M pipe buffer size
2019-05-18 12:57:43 +02:00
Dietmar Maurer
6276448519
src/bin/test_chunk_speed2.rs: test chunker speed with real data
2019-05-18 11:59:17 +02:00
Dietmar Maurer
e058744d8f
src/backup/chunk_stream.rs: simplify code
2019-05-18 10:55:46 +02:00
Dietmar Maurer
8a7cc7565f
src/backup/chunk_stream.rs: implement fixed size chunk stream
2019-05-18 10:55:46 +02:00
Dietmar Maurer
9c9ad941c0
src/client/http_client.rs: improve speed test
...
pipeline responses to a separate async channel ...
2019-05-18 10:55:46 +02:00
Christian Ebner
2dcdd3b427
src/tools/xattrs.rs: impl libc wrapper functions to get/set xattrs and fcaps
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:21 +02:00
Christian Ebner
de2016d58a
src/pxar/format_definitions.rs: add format definitions for xattrs and fcaps
...
In addition to the format definition, the traits needed for sorting of xattr
entries by name are derived.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:13 +02:00
Christian Ebner
dec1ff18b8
src/tools/acl.rs: impl add_entry_full for ACL and set_permissions for ACLEntry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:10:06 +02:00
Dietmar Maurer
adec8ea263
src/api2/admin/datastore/backup.rs: add speedtest api, improve upload speed
...
We need to disable tcp Nagle algorythm (set_nodelay), and use larger window size for http2
2019-05-16 10:24:23 +02:00
Dietmar Maurer
372724afea
src/api2/admin/datastore/backup.rs: implement finish_backup
2019-05-15 12:58:55 +02:00
Dietmar Maurer
bb105f9dae
src/api2/admin/datastore/backup.rs: improve logging
2019-05-15 10:09:46 +02:00
Dietmar Maurer
25ddc15fe0
src/client/pipe_to_stream.rs: add missing file
2019-05-15 09:07:19 +02:00
Dietmar Maurer
a20772525f
src/api2/admin/datastore/backup.rs: implement close for dynamic writers
2019-05-15 07:58:05 +02:00
Dietmar Maurer
4e93f8c164
src/api2/types.rs: add schema/format for file names
2019-05-15 07:14:08 +02:00
Dietmar Maurer
e3dbd41b20
src/client/http_client.rs: add flow control for h2 upload
2019-05-14 14:54:21 +02:00
Dietmar Maurer
97f22ce57d
src/client/http_client.rs: implement h2 upload
...
without flow control so far ...
2019-05-14 12:58:59 +02:00
Dietmar Maurer
dafc27aeff
src/backup/chunk_stream.rs: async chunk stream
2019-05-14 10:05:29 +02:00
Dietmar Maurer
c0e22141ed
src/client/http_client.rs: derive Clone for H2Client
2019-05-14 09:00:24 +02:00
Christian Ebner
25e205a1d3
src/tools.rs: add mod acl to tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:34:24 +02:00
Christian Ebner
d22096ef4b
src/tools/acl.rs: impl POSIX access control list (ACL) manipulation tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:28:42 +02:00
Dietmar Maurer
1133fe9a63
src/server/rest.rs: do not log 1xx status codes as errors
2019-05-14 06:23:22 +02:00
Dietmar Maurer
850ac6d0f0
src/client/http_client.rs: directly return H2Client on upgrade
2019-05-13 16:44:59 +02:00
Dietmar Maurer
10130cf48c
src/client/http_client.rs: cleanup, remove debug println
2019-05-13 12:11:18 +02:00
Dietmar Maurer
b57cb26406
src/client/http_client.rs: implement http2 client wrapper
2019-05-13 10:27:22 +02:00
Dietmar Maurer
9e391bb7f0
src/client/http_client.rs: allow to pass parameters (encode them as query string)
2019-05-13 09:12:03 +02:00
Dietmar Maurer
fbb798f6d2
src/api2/admin/datastore/backup.rs: ignore errors from last_backup
2019-05-13 07:06:56 +02:00
Dietmar Maurer
d36113665b
src/api2/admin/datastore/backup.rs: implement chunk index download from previous backup
2019-05-11 16:05:50 +02:00
Dietmar Maurer
457531e7fe
src/backup/index.rs: new ChunkListReader
2019-05-11 16:01:42 +02:00
Dietmar Maurer
af1d68185c
src/api2/admin/datastore/backup.rs: add missing 'archive-name' parameter to schema
2019-05-11 12:25:15 +02:00
Dietmar Maurer
6b95c7df07
src/api2/admin/datastore/backup/environment.rs: do not store path
2019-05-11 12:13:44 +02:00
Dietmar Maurer
51a4f63ff7
src/backup/datastore.rs: move last_backup to BackupInfo
2019-05-11 12:07:09 +02:00
Dietmar Maurer
b02a52e3a5
src/backup/datastore.rs: impl last_backup helper
2019-05-11 11:36:46 +02:00
Dietmar Maurer
c09775011a
src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups
2019-05-11 10:19:34 +02:00
Dietmar Maurer
f9578f3c79
backup-api: allow to create DynamicIndexWriter and add chunks to to
2019-05-10 10:25:40 +02:00
Dietmar Maurer
35a2d8a6a6
src/backup/backup_info.rs: Derive Clone
2019-05-10 06:59:23 +02:00
Dietmar Maurer
0506d1ed04
upload_chunk.rs: add comment about possible unnecessary code
2019-05-10 06:15:25 +02:00
Dietmar Maurer
5f42ef009a
cleanup: remove unused code
2019-05-09 18:11:39 +02:00
Dietmar Maurer
b4b63e520d
handle_async_api_request: put rpcenv into a Box
...
So that we can pass rpcenv into futures.
2019-05-09 18:01:24 +02:00
Dietmar Maurer
2c41096a4b
src/api2/admin/datastore/backup/upload_chunk.rs: fix maximal chunk size
2019-05-09 13:29:50 +02:00
Dietmar Maurer
78b5191550
src/server/formatter.rs: further cleanups and renaming ...
2019-05-09 13:28:26 +02:00
Dietmar Maurer
d55037e471
src/server/formatter.rs: rename format_result to format_data
...
To avoid confusions with Rust Result type.
2019-05-09 13:15:15 +02:00
Dietmar Maurer
21ee7912fa
src/api2/admin/datastore/backup.rs: implement upload chunk
2019-05-09 13:06:09 +02:00
Dietmar Maurer
4ebf0eabb0
src/api2/types.rs: use separate file for schema definitions
2019-05-09 07:44:09 +02:00
Dietmar Maurer
bd1507c4fa
src/api2/admin/datastore/backup/service.rs: move service code into extra file
2019-05-08 17:36:19 +02:00
Dietmar Maurer
0aadd40b4f
src/api2/admin/datastore/backup.rs: add required parameters, cleanups
2019-05-08 13:17:23 +02:00
Dietmar Maurer
d95ced64ad
src/api2/admin/datastore/backup/environment.rs: use separate file
2019-05-08 12:41:58 +02:00
Dietmar Maurer
ca60c371e2
api: rename h2upload to backup
2019-05-08 12:35:15 +02:00
Dietmar Maurer
d906b1124f
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 11:36:43 +02:00
Dietmar Maurer
58c8d7d91f
rc/api2/admin/datastore/h2upload.rs: implement BackupEnvironment
...
To pass arbitrary information/state to api methods.
2019-05-08 11:26:54 +02:00
Dietmar Maurer
f757b30efc
src/server/rest.rs: use generics to pass RpcEnvironment
2019-05-08 11:09:01 +02:00
Dietmar Maurer
2edc341b29
src/tools.rs: Add AsAyn Trait
2019-05-08 11:07:36 +02:00
Dietmar Maurer
090ac9f79a
src/api2/admin/datastore/h2upload.rs: avoid unwrap
2019-05-08 08:21:18 +02:00
Dietmar Maurer
4e3da4b366
src/api2/admin/datastore/h2upload.rs: do not log client address
2019-05-08 08:15:55 +02:00
Dietmar Maurer
e3a44552ee
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 08:12:49 +02:00
Dietmar Maurer
a66ab8ae5e
src/api2/admin/datastore/h2upload.rs: make worker abortable
2019-05-08 07:14:40 +02:00
Dietmar Maurer
72375ce6c6
src/api2/admin/datastore/h2upload.rs: log to worker task
2019-05-07 14:07:00 +02:00
Dietmar Maurer
d9bd06eae8
src/api2/admin/datastore/h2upload.rs: use WorkerTask
2019-05-07 13:42:00 +02:00
Dietmar Maurer
52cf506e48
src/api2/admin/datastore/h2upload.rs: implement async test api call
2019-05-07 12:26:55 +02:00
Dietmar Maurer
92ac375aff
src/api2/admin/datastore/h2upload.rs: improve http2 service
2019-05-07 11:24:44 +02:00
Dietmar Maurer
78f7e225e4
src/server/rest.rs: make handle_(a)sync_api_request public
2019-05-07 11:23:52 +02:00
Dietmar Maurer
75097495ad
RestEnvironment: derive Clone
2019-05-07 11:09:18 +02:00
Dietmar Maurer
01bf3b7b5f
move find_method to Router class
2019-05-07 11:08:30 +02:00
Dietmar Maurer
3578d99f3e
move normalize_path to tools::normalize_uri_path
2019-05-07 09:44:34 +02:00
Dietmar Maurer
96e95fc179
src/api2/admin/datastore/h2upload.rs: test if version < 2
...
We can only upgrade http 1.x connections.
2019-05-06 10:29:34 +02:00
Dietmar Maurer
45cdce069e
src/client/http_client.rs: request password at creation time
2019-04-30 11:44:35 +02:00
Dietmar Maurer
e45afdff9c
src/server/state.rs: use new BroadcastData helper
2019-04-30 10:21:48 +02:00
Dietmar Maurer
824b5ee4ee
src/tools/broadcast_future.rs: cleanup, decompose into two classes
...
In order to make it more usable.
2019-04-30 08:57:20 +02:00
Wolfgang Bumiller
d82ed9b0f5
tools: add AsyncMutex
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-29 14:19:03 +02:00
Dietmar Maurer
3a9bea3918
src/bin/h2test.rs: test h2 upgrade
2019-04-29 11:59:18 +02:00
Dietmar Maurer
cf639a47de
rc/client/http_client.rs: add experimental h2 upgrade code
2019-04-29 11:57:58 +02:00
Dietmar Maurer
5a2df00004
src/client/http_client.rs: cleanup, make login fully async
2019-04-28 10:55:03 +02:00
Dietmar Maurer
3dceb9b304
src/tools/broadcast_future.rs: add new constructor new_oneshot()
...
To simplify usage. Also added a test case.
2019-04-27 10:56:49 +02:00