Dietmar Maurer
da7d67213e
src/api2/admin/datastore/pxar.rs: remove pxar upload api
...
Upload needs to be done using the backup protocol.
2019-06-25 09:45:35 +02:00
Dietmar Maurer
dd8e744f9a
src/api2/admin/datastore.rs: use a worker log log prune actions
2019-06-25 08:12:25 +02:00
Dietmar Maurer
6899dbfb47
src/bin/proxmox-backup-client.rs: strip .blob extension
2019-06-25 07:11:02 +02:00
Dietmar Maurer
c6d203bbd3
src/backup/backup_info.rs: also list .blob files
2019-06-25 07:07:45 +02:00
Dietmar Maurer
045e8a568d
src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key
2019-06-25 07:00:07 +02:00
Dietmar Maurer
79bc7345e8
src/cli/command.rs: fix ReST output format
2019-06-25 06:19:51 +02:00
Dietmar Maurer
9f46c7de4b
src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup
2019-06-24 13:56:37 +02:00
Christian Ebner
cd7dc87903
pxar: impl .pxarexclude parsing and exclude matching
...
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.
Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-24 10:26:20 +02:00
Dietmar Maurer
46bd880041
src/api2/backup/environment.rs: new helper add_blob()
2019-06-24 09:35:37 +02:00
Dietmar Maurer
cb08ac3efe
src/api2/backup.rs: replace upload_config with upload_blob
2019-06-23 11:44:30 +02:00
Dietmar Maurer
a38c5d4d12
src/backup/data_blob.rs: implement from_raw()
2019-06-23 09:35:44 +02:00
Dietmar Maurer
6f083b7a92
src/api2/backup/upload_chunk.rs: compute crc on upload
2019-06-23 09:04:42 +02:00
Dietmar Maurer
ba01828d38
try to use use proxmox::tools::io::ops::ReadExtOps
2019-06-22 16:29:10 +02:00
Dietmar Maurer
27042ce637
src/backup/file_formats.rs: improve docs
2019-06-22 15:45:36 +02:00
Dietmar Maurer
9f83e0f7ee
src/backup/crypt_config.rs: simplify decode interface
2019-06-22 13:24:29 +02:00
Dietmar Maurer
0066c6d972
src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead
2019-06-22 13:02:53 +02:00
Dietmar Maurer
ee8a7e8080
src/backup/crypt_config.rs: implement encrypt_to (output to writer)
2019-06-22 12:25:04 +02:00
Dietmar Maurer
991abfa8b4
src/backup/file_formats.rs: split out file format data
2019-06-22 09:12:25 +02:00
Dietmar Maurer
b7f4f27d6c
add crc field for binary blobs formats
2019-06-21 18:49:04 +02:00
Dietmar Maurer
a7f67a9a9c
depend on crc32fast
...
We can use this to compute fast checksums to test file integrity (not crypto safe).
2019-06-21 15:58:00 +02:00
Dietmar Maurer
3025b3a53c
src/backup/data_blob.rs: new file format for binary blobs
2019-06-21 11:32:07 +02:00
Dietmar Maurer
077a8cae63
src/backup/crypt_config.rs: make code more flexible
...
Allow to pass magic numbers as parameters.
2019-06-21 10:50:48 +02:00
Dietmar Maurer
c38266c18d
renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs
2019-06-21 09:51:18 +02:00
Dietmar Maurer
3ea8bfc9c0
src/bin/proxmox-backup-client.rs: add import-master-pubkey command
2019-06-21 07:34:17 +02:00
Dietmar Maurer
37c5a17505
src/bin/proxmox-backup-client.rs: helper to generate RSA master key
2019-06-21 07:12:19 +02:00
Dietmar Maurer
ab44acff57
src/backup/key_derivation.rs: add modified field to key file
2019-06-20 07:58:33 +02:00
Dietmar Maurer
6d0983dbe1
src/bin/proxmox-backup-client.rs: add keyfile parameter for backup
2019-06-19 17:16:41 +02:00
Dietmar Maurer
181f097af9
src/bin/proxmox-backup-client.rs - key API: pass kdf parameter
...
Allow to store keys without password.
2019-06-18 18:21:01 +02:00
Dietmar Maurer
9b06db4563
src/bin/proxmox-backup-client.rs: use file system path to reference keys
2019-06-18 17:38:50 +02:00
Dietmar Maurer
091540298e
src/bin/proxmox-backup-client.rs - create_key: always read password from tty
2019-06-18 17:27:52 +02:00
Dietmar Maurer
826f309bf5
src/backup/key_derivation.rs: move kdf code into separate file
2019-06-18 11:17:22 +02:00
Dietmar Maurer
11515438cc
Cargo.toml: use serde feature derive
2019-06-18 06:23:25 +02:00
Dietmar Maurer
ac71623483
src/bin/proxmox-backup-client.rs: implement "key change-passphrase"
2019-06-17 13:10:00 +02:00
Dietmar Maurer
f2401311b0
src/bin/proxmox-backup-client.rs: start key management
2019-06-17 10:33:24 +02:00
Dietmar Maurer
a7dd483097
src/backup/*_index.rs: used generated magic numbers
2019-06-14 14:58:37 +02:00
Dietmar Maurer
bffd40d6b7
src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools
2019-06-14 11:40:04 +02:00
Dietmar Maurer
00388226e6
src/api2/backup/environment.rs: log index checksums
2019-06-14 10:36:20 +02:00
Dietmar Maurer
9335d74eba
src/backup/fixed_index.rs: compute checksum over the index
2019-06-14 10:34:29 +02:00
Dietmar Maurer
16ff6b7cd8
src/backup/dynamic_index.rs: compute checksum over the index
2019-06-14 10:33:59 +02:00
Dietmar Maurer
36075475a6
src/api2/backup/environment.rs: log duplicate chunks
2019-06-14 07:12:30 +02:00
Dietmar Maurer
fa148dbd11
src/api2/backup/upload_chunk.rs: verify chunk data if possible
2019-06-14 06:23:46 +02:00
Dietmar Maurer
0f37577092
src/bin/cipherbench.rs: add test for zstd compression
2019-06-13 17:24:57 +02:00
Dietmar Maurer
9399c98f82
src/bin/cipherbench.rs: cipher speed test with large blocks
2019-06-13 17:16:43 +02:00
Dietmar Maurer
f640f23ac5
src/backup/data_chunk.rs: fix typo
2019-06-13 16:42:55 +02:00
Dietmar Maurer
dba72d5016
src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:56:50 +02:00
Dietmar Maurer
68ab376ae9
src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:47:54 +02:00
Dietmar Maurer
f98ac774ee
backup: Add support for client side encryption
...
first try ...
2019-06-13 11:47:23 +02:00
Dietmar Maurer
51929e4532
src/backup/data_chunk.rs - decode: make crypt_config optional
2019-06-13 11:44:38 +02:00
Dietmar Maurer
046bcb9773
src/backup/crypt_setup.rs: add comment for possible improvements
2019-06-13 07:35:59 +02:00
Dietmar Maurer
bd0e3c7cfa
src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs
2019-06-13 07:35:02 +02:00
Dietmar Maurer
6762db70d6
src/api2/types.rs: define CHUNK_DIGEST_SCHEMA
2019-06-13 07:30:42 +02:00
Dietmar Maurer
b595cb9d2c
src/backup/data_chunk.rs: Data Chunk builder
...
Make handling encoded chunks easier.
2019-06-12 13:37:01 +02:00
Dietmar Maurer
ef27200cec
src/backup/crypt_setup.rs: fix encoder for uncompressed chunks
2019-06-10 09:56:06 +02:00
Dietmar Maurer
74792b95b2
src/backup/crypt_setup.rs: allow compressed and uncompressed chunks
2019-06-10 08:27:35 +02:00
Dietmar Maurer
9e0187a203
src/backup/crypt_setup.rs: remove unused CryptData struct
2019-06-10 07:55:36 +02:00
Dietmar Maurer
bec8498ae3
src/backup/crypt_setup.rs: avoid useless allocations
2019-06-09 11:44:17 +02:00
Dietmar Maurer
c20b17b460
src/backup.rs: define magic numbers for chunk store file format
2019-06-08 10:34:20 +02:00
Dietmar Maurer
48b4b40b57
src/backup/crypt_setup.rs: crypto helpers
2019-06-08 09:56:51 +02:00
Wolfgang Bumiller
dd5495d6dc
tree-wide: use 'dyn' for all trait objects
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-07 13:13:48 +02:00
Christian Ebner
e993db91cd
pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder
...
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:20 +02:00
Christian Ebner
1f319e766e
src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:06 +02:00
Christian Ebner
9b77388f2a
src/pxar/encoder.rs: minor refactoring
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:55 +02:00
Christian Ebner
a4cc8eb7c5
src/pxar/encoder.rs: check if ioctl supported by filesystem
...
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:31 +02:00
Dietmar Maurer
4800df7ba6
src/api2/admin/datastore.rs: remove test-upload api
2019-06-07 07:52:11 +02:00
Dietmar Maurer
113898d7e9
src/api2/admin/datastore/upload.rs: remove unused upload api
2019-06-07 07:39:41 +02:00
Dietmar Maurer
4247fccb0f
src/client/http_client.rs: automatically close connection in finish
2019-06-05 11:41:50 +02:00
Wolfgang Bumiller
cb4426b348
make h2 client connection cancellable
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-05 09:45:59 +02:00
Dietmar Maurer
986bef16be
src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1
...
So that we can include it in static doc strings.
2019-06-05 08:41:20 +02:00
Dietmar Maurer
c9ec0956cf
src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1
2019-06-05 08:12:13 +02:00
Dietmar Maurer
44c54845b3
src/tools/xattr.rs: fix test when run as root
2019-06-05 07:57:42 +02:00
Dietmar Maurer
7773ccc11f
src/api2.rs: move backup api to /backup
2019-06-05 07:23:21 +02:00
Christian Ebner
97eeea3b4a
src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:29:02 +02:00
Dietmar Maurer
1ce2f7cfb9
src/pxar/encoder.rs: style fix
2019-06-05 06:27:39 +02:00
Christian Ebner
e7b60a16c8
pxar: impl storing/dumping/restoring of quota project ids
...
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:26:02 +02:00
Christian Ebner
7312ab9eef
src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID
...
Defines the types needed to store/restore quota project ids in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:23:06 +02:00
Christian Ebner
042babe4e7
src/tools/fs.rs: impl ioctl calls to get/set fsxattr
...
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:22:49 +02:00
Dietmar Maurer
22e5aa1d3e
src/bin/dump-backup-api.rs: helper to generate backup API docs
2019-06-04 13:12:42 +02:00
Dietmar Maurer
5e0f305142
src/api_schema/format.rs: depend on textwrap, new wrap_text helper
2019-06-04 12:42:02 +02:00
Dietmar Maurer
339ddfcbfa
src/cli/command.rs: move doc generator code to src/api_schema/format.rs
2019-06-04 12:32:22 +02:00
Dietmar Maurer
ef39bf95b3
src/client/http_client.rs - upload_config: improve error message
2019-06-03 10:53:09 +02:00
Dietmar Maurer
ec8a9bb953
src/bin/proxmox-backup-client.rs: implement upload_config
2019-06-03 10:39:44 +02:00
Dietmar Maurer
39d6846e1e
src/api2/admin/datastore/backup.rs: implement config file upload
2019-06-03 09:45:14 +02:00
Dietmar Maurer
1c0472e88f
avoid compiler warnings
2019-06-03 06:43:47 +02:00
Dietmar Maurer
36898ffce6
src/backup/chunk_stream.rs: add optional chunk_size parameter
2019-05-30 13:28:24 +02:00
Dietmar Maurer
49ef316bcd
src/bin/proxmox-backup-client.rs: backup_image() - pass correct chunk size
2019-05-30 12:52:12 +02:00
Dietmar Maurer
202cadc29e
src/bin/proxmox-backup-client.rs: remove unused code
2019-05-30 12:47:35 +02:00
Dietmar Maurer
6af905c18f
src/bin/proxmox-backup-client.rs: implement image/block-device upload
2019-05-30 12:46:01 +02:00
Dietmar Maurer
eb1804c567
src/bin/proxmox-backup-client.rs: use std lib instead of nix
2019-05-30 12:19:23 +02:00
Dietmar Maurer
6e907505aa
src/bin/proxmox-backup-client.rs: fix file type test
2019-05-30 12:08:34 +02:00
Dietmar Maurer
96482891ae
src/api2/admin/datastore/backup/environment.rs: log upload statistics
2019-05-30 09:21:19 +02:00
Dietmar Maurer
642322b433
src/api2/admin/datastore/backup.rs: use separate api entry points for chunk upload
...
So that we can provide better statistics (patches follows).
2019-05-30 09:21:19 +02:00
Wolfgang Bumiller
390e83c9b9
typo fix and doc improvement
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:57:28 +02:00
Wolfgang Bumiller
7fb49397d5
doc fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:56:14 +02:00
Wolfgang Bumiller
382609b04e
tools: add Cancellable, start a futures submodule
...
To make a future cancellable, use:
let (future, canceller) =
crate::tools::futures::cancellable(future);
Proceed with using `future` as usual, `canceller` is
clonable and can cancel the future via the `.cancel()`
method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Wolfgang Bumiller
0d32d71fb7
tools: add AsyncMutex::new_locked
...
Allows creating a pre-locked mutex, returning the mutex and
a guard.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Dietmar Maurer
e6389f4e75
src/api2/admin/datastore/backup/environment.rs: do not allow empty backups (no files)
2019-05-29 10:38:57 +02:00
Dietmar Maurer
39e60bd6fa
src/api2/admin/datastore/backup.rs: improve debug logging
2019-05-29 10:17:38 +02:00
Dietmar Maurer
4ae6551c78
src/api_schema/router.rs: disable debug println
2019-05-29 09:42:16 +02:00
Dietmar Maurer
a42d1f55c3
src/api2/admin/datastore/backup.rs: add debug flag
2019-05-29 09:35:21 +02:00
Dietmar Maurer
976595e1a9
src/backup/dynamic_index.rs: split class DynamicIndexWriter
2019-05-29 08:49:57 +02:00
Dietmar Maurer
cb0708dd46
src/backup/fixed_index.rs: remove ChunkStat from struct
2019-05-29 07:08:34 +02:00
Dietmar Maurer
01af11f340
src/backup/fixed_index.rs: improve consistency checks
2019-05-29 07:02:45 +02:00
Christian Ebner
9b38443314
pxar: change logic and impl all command line flags for xattrs/fcaps/acls
...
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-29 06:51:20 +02:00
Dietmar Maurer
60e589a111
src/api2/admin/datastore/backup.rs: log sucessful finish call
2019-05-28 12:55:22 +02:00
Dietmar Maurer
c4ff3dcefd
src/bin/proxmox-backup-client.rs: use new BackupClient to upload data
2019-05-28 10:12:44 +02:00
Dietmar Maurer
10241c20ea
src/tools.rs: implement image_size helper
2019-05-28 09:35:08 +02:00
Dietmar Maurer
006f3ff407
src/api2/admin/datastore/backup/environment.rs: add more consistency checks
2019-05-28 09:21:31 +02:00
Dietmar Maurer
3dc5b2a203
src/client/http_client.rs: use start of chunk as offset
2019-05-28 09:21:31 +02:00
Dietmar Maurer
8de20e5c1f
src/client/http_client.rs: code cleanup
2019-05-28 08:53:27 +02:00
Dietmar Maurer
fc14b84937
src/backup/fixed_index.rs: improve bound checks inside add_digest
2019-05-28 08:47:03 +02:00
Dietmar Maurer
e1225de467
src/backup/fixed_index.rs: store index_length to avoid code duplication
2019-05-28 08:27:57 +02:00
Dietmar Maurer
e3d525feb7
src/api2/admin/datastore/backup.rs: fix typo
2019-05-28 07:13:19 +02:00
Christian Ebner
af309d4d5c
src/bin/pxar.rs: set archive permissions to 0o640
...
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:24:47 +02:00
Christian Ebner
ab87f167f1
src/pxar/encoder.rs: Refactor file stat
...
Introduce helper functions to check file stats
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:23:49 +02:00
Christian Ebner
578b601159
src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
...
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:22:16 +02:00
Dietmar Maurer
a42fa400ee
src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api
2019-05-28 06:18:55 +02:00
Dietmar Maurer
29ae5c86a2
src/backup/fixed_index.rs: make chunk_size public
2019-05-28 06:18:24 +02:00
Dietmar Maurer
7dd1bcac58
src/client/http_client.rs: correctly parse error response in download_chunk_list
2019-05-27 09:35:30 +02:00
Dietmar Maurer
a1e7cff3ea
src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk
2019-05-27 07:57:43 +02:00
Dietmar Maurer
771953f9eb
src/api2/admin/datastore/backup.rs: use smaller window sizes
2019-05-27 07:53:40 +02:00
Dietmar Maurer
174ad378d8
src/client/http_client.rs: fix request pipelining
2019-05-27 07:24:32 +02:00
Dietmar Maurer
624362226e
src/client/merge_known_chunks.rs: use MergedChunkInfo as input
2019-05-26 10:52:56 +02:00
Dietmar Maurer
05cba08c9c
src/client/http_client.rs: append chunks in correct order
...
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
2019-05-26 09:57:34 +02:00
Dietmar Maurer
417cb0731e
src/api2/admin/datastore/backup.rs: verify chunk offset
...
This clearly shows that the current approach does not work - seems we get
chunks out of order.
2019-05-26 08:49:29 +02:00
Dietmar Maurer
34114e2606
src/server/rest.rs: correctly verify json parameters
2019-05-24 09:10:59 +02:00
Dietmar Maurer
eb2bdd1bcc
src/client/http_client.rs: encode parameters inside body
...
To avoid problems with long uris.
2019-05-24 08:32:55 +02:00
Dietmar Maurer
9bb675ec23
src/client/merge_known_chunks.rs: reduce RAM usage
...
There is no need to store chunk data for known chunks.
2019-05-24 07:42:58 +02:00
Dietmar Maurer
ecb21b4794
src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough
2019-05-24 07:36:09 +02:00
Christian Ebner
2dbba78b98
src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:27:24 +02:00
Christian Ebner
6a19448004
src/pxar/encoder.rs: impl support to store ACLs in pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:21:33 +02:00
Christian Ebner
7cfaade7c8
src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:16:55 +02:00
Christian Ebner
b344461b33
pxar: pass feature_flags to encoder/decoder instead of individual flags
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner
687407741b
src/tools/xattr.rs: add test for is_valid_xattr_name()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner
357e4614e2
Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Wolfgang Bumiller
c05a8c8d18
doc-test fixup
...
cargo test by default compiles and runs all code snippets
found in the documentation...
oops...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 15:22:33 +02:00
Wolfgang Bumiller
cc84a830c5
RFC: schema: make enums static
...
I don't see a reason to allow these to be dynamically
modifiable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 12:38:51 +02:00
Dietmar Maurer
aa1b2e04fe
src/client/merge_known_chunks.rs: merge known chunks
...
To decrease the number of api calls required...
2019-05-23 12:31:09 +02:00
Dietmar Maurer
91320f0879
src/client/http_client.rs: use ChunkInfo streams
...
This will make out of order uploads possible...
2019-05-23 12:31:09 +02:00
Wolfgang Bumiller
8ea3b1d188
tree-wide: use the new vec/io tools modules
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Wolfgang Bumiller
9cdda3f7c7
tools: add helpful vector and read operations
...
After importing the I/O ops trait via:
use crate::tools::io::ops::*;
Instead of:
let mut buffer = vec![0u8; 65536];
file.read_exact(&mut buffer)?;
use:
let buffer = file.read_exact_allocated(65536)?;
After importing the vector helpers via:
use crate::tools::vec::{self, ops::*};
For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
let mut buffer = vec![0u8; len];
use:
let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Dietmar Maurer
8bea85b42e
src/api2/admin/datastore/backup.rs: verify file size and chunk count on close
2019-05-23 08:50:36 +02:00
Dietmar Maurer
1179e15842
src/server/rest.rs: improve error handling
2019-05-23 08:15:32 +02:00
Dietmar Maurer
907f5bb0d8
src/api2/admin/datastore/backup/service.rs: improve error logging
2019-05-23 08:05:39 +02:00
Dietmar Maurer
32e1bf4260
src/bin/upload-speed.rs: code cleanup
2019-05-23 07:20:05 +02:00
Christian Ebner
0d9bab05ab
src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:50 +02:00
Christian Ebner
4add706092
src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:32 +02:00
Christian Ebner
54b14ad741
src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:09 +02:00
Christian Ebner
1201360def
src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:39:16 +02:00
Christian Ebner
8bc8791338
src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:37:37 +02:00
Christian Ebner
a45a123950
src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
...
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:34:36 +02:00
Dietmar Maurer
d6f204eda6
src/client/http_client.rs: re-order code - public functions first
2019-05-23 06:28:10 +02:00
Dietmar Maurer
9af37c8f0e
src/client/http_client.rs: move low level H2 code into separate class
2019-05-22 17:28:25 +02:00
Dietmar Maurer
d2c48afc6e
src/client/http_client.rs: code cleanup
2019-05-22 13:24:33 +02:00
Dietmar Maurer
6ab34afa88
src/client/http_client.rs: cleanup h2 backup client
2019-05-22 13:05:51 +02:00
Dietmar Maurer
a55fcd740f
src/api2/admin/datastore/backup.rs: suse POST for finish() api
2019-05-22 13:04:58 +02:00
Dietmar Maurer
de6742c04f
src/bin/h2test.rs: removed
...
This test does no work anymore.
2019-05-22 11:14:56 +02:00
Dietmar Maurer
5a0b521809
src/api2/admin/datastore/backup.rs: remove test callbacks
2019-05-22 10:55:27 +02:00
Christian Ebner
de61bc9264
src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr
...
As git does not preserve xattrs they have to be set before they can be read.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-22 10:01:10 +02:00
Dietmar Maurer
553610b43e
src/client/http_client.rs: implement download_chunk_list
2019-05-22 09:46:02 +02:00
Dietmar Maurer
0cc0fffd1e
src/backup/chunk_stream.rs: use BytesMut in ChunkStream
2019-05-22 09:39:02 +02:00
Dietmar Maurer
c052be5c86
src/backup/chunk_stream.rs: use more generics
2019-05-22 09:18:05 +02:00
Dietmar Maurer
169c0e060f
src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut)
2019-05-22 09:05:35 +02:00
Dietmar Maurer
3be3f3dcc4
src/backup/chunk_stream.rs: fix behaviour for large input buffers
2019-05-22 08:05:27 +02:00
Dietmar Maurer
02fa54ff3c
src/backup/chunk_stream.rs: assemble data with correct order
2019-05-22 07:44:51 +02:00
Dietmar Maurer
d0afeaa831
src/api2/admin/datastore/backup.rs: improve error message
2019-05-22 07:43:54 +02:00
Dietmar Maurer
11867a2b54
src/tools.rs: fix hex_to_digest
2019-05-21 13:42:13 +02:00
Dietmar Maurer
7f3d2ffab9
src/backup/index.rs: rename ChunkListReader into DigestListEncoder
2019-05-21 12:28:44 +02:00
Dietmar Maurer
ee53955f52
src/backup/index.rs: implement DigestListDecoder
2019-05-21 12:21:22 +02:00
Dietmar Maurer
a95849321c
src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks
2019-05-21 09:54:11 +02:00
Dietmar Maurer
40f4e198a8
src/backup/dynamic_index.rs: add chunk_info method
2019-05-21 09:52:19 +02:00
Christian Ebner
596182056b
src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:15:30 +02:00
Christian Ebner
b303057a19
src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:06:26 +02:00
Christian Ebner
bee8d8ea78
src/tools/xattr.rs: add functions name_store and security_capability
...
These functions allow to check if the name for xattrs/fcaps are valid.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:05:31 +02:00
Dietmar Maurer
a09c0e38d8
src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks
2019-05-20 18:05:10 +02:00
Dietmar Maurer
97f03eff13
src/client/pxar_backup_stream.rs: fix docu
2019-05-20 17:29:00 +02:00
Dietmar Maurer
82ab72304e
src/client/http_client.rs: implement upload_stream
2019-05-20 14:19:24 +02:00
Dietmar Maurer
2698e8a514
src/client/pxar_backup_stream.rs: close pipe inside drop
2019-05-20 11:20:33 +02:00
Dietmar Maurer
f2269d8f99
src/tools.rs: implement hex_to_digest
2019-05-19 12:15:46 +02:00
Dietmar Maurer
751030aa7c
src/bin/test_chunk_speed2.rs: test max chunk size
2019-05-19 11:09:02 +02:00
Dietmar Maurer
ff77dbbea3
Revert "src/backup/chunk_stream.rs: simplify code"
...
This reverts commit e058744d8f
.
The optimization was wrong, and produces wrong chunk boundaries.
2019-05-19 11:05:56 +02:00
Dietmar Maurer
178ac042c4
src/api_schema/schema.rs: improve error messages
2019-05-19 10:18:51 +02:00
Dietmar Maurer
0c516b123d
src/client/pxar_backup_stream.rs: use WrappedStreamReader
...
to make it fully async ...
2019-05-18 15:13:31 +02:00
Dietmar Maurer
c6e28b66c6
src/client/pxar_backup_stream.rs: use 1M pipe buffer size
2019-05-18 12:57:43 +02:00
Dietmar Maurer
6276448519
src/bin/test_chunk_speed2.rs: test chunker speed with real data
2019-05-18 11:59:17 +02:00
Dietmar Maurer
e058744d8f
src/backup/chunk_stream.rs: simplify code
2019-05-18 10:55:46 +02:00
Dietmar Maurer
8a7cc7565f
src/backup/chunk_stream.rs: implement fixed size chunk stream
2019-05-18 10:55:46 +02:00
Dietmar Maurer
9c9ad941c0
src/client/http_client.rs: improve speed test
...
pipeline responses to a separate async channel ...
2019-05-18 10:55:46 +02:00
Christian Ebner
2dcdd3b427
src/tools/xattrs.rs: impl libc wrapper functions to get/set xattrs and fcaps
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:21 +02:00
Christian Ebner
de2016d58a
src/pxar/format_definitions.rs: add format definitions for xattrs and fcaps
...
In addition to the format definition, the traits needed for sorting of xattr
entries by name are derived.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:13 +02:00
Christian Ebner
dec1ff18b8
src/tools/acl.rs: impl add_entry_full for ACL and set_permissions for ACLEntry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:10:06 +02:00
Dietmar Maurer
adec8ea263
src/api2/admin/datastore/backup.rs: add speedtest api, improve upload speed
...
We need to disable tcp Nagle algorythm (set_nodelay), and use larger window size for http2
2019-05-16 10:24:23 +02:00
Dietmar Maurer
372724afea
src/api2/admin/datastore/backup.rs: implement finish_backup
2019-05-15 12:58:55 +02:00
Dietmar Maurer
bb105f9dae
src/api2/admin/datastore/backup.rs: improve logging
2019-05-15 10:09:46 +02:00
Dietmar Maurer
25ddc15fe0
src/client/pipe_to_stream.rs: add missing file
2019-05-15 09:07:19 +02:00
Dietmar Maurer
a20772525f
src/api2/admin/datastore/backup.rs: implement close for dynamic writers
2019-05-15 07:58:05 +02:00
Dietmar Maurer
4e93f8c164
src/api2/types.rs: add schema/format for file names
2019-05-15 07:14:08 +02:00
Dietmar Maurer
e3dbd41b20
src/client/http_client.rs: add flow control for h2 upload
2019-05-14 14:54:21 +02:00
Dietmar Maurer
97f22ce57d
src/client/http_client.rs: implement h2 upload
...
without flow control so far ...
2019-05-14 12:58:59 +02:00
Dietmar Maurer
dafc27aeff
src/backup/chunk_stream.rs: async chunk stream
2019-05-14 10:05:29 +02:00
Dietmar Maurer
c0e22141ed
src/client/http_client.rs: derive Clone for H2Client
2019-05-14 09:00:24 +02:00
Christian Ebner
25e205a1d3
src/tools.rs: add mod acl to tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:34:24 +02:00
Christian Ebner
d22096ef4b
src/tools/acl.rs: impl POSIX access control list (ACL) manipulation tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:28:42 +02:00
Dietmar Maurer
1133fe9a63
src/server/rest.rs: do not log 1xx status codes as errors
2019-05-14 06:23:22 +02:00
Dietmar Maurer
850ac6d0f0
src/client/http_client.rs: directly return H2Client on upgrade
2019-05-13 16:44:59 +02:00
Dietmar Maurer
10130cf48c
src/client/http_client.rs: cleanup, remove debug println
2019-05-13 12:11:18 +02:00
Dietmar Maurer
b57cb26406
src/client/http_client.rs: implement http2 client wrapper
2019-05-13 10:27:22 +02:00
Dietmar Maurer
9e391bb7f0
src/client/http_client.rs: allow to pass parameters (encode them as query string)
2019-05-13 09:12:03 +02:00
Dietmar Maurer
fbb798f6d2
src/api2/admin/datastore/backup.rs: ignore errors from last_backup
2019-05-13 07:06:56 +02:00
Dietmar Maurer
d36113665b
src/api2/admin/datastore/backup.rs: implement chunk index download from previous backup
2019-05-11 16:05:50 +02:00
Dietmar Maurer
457531e7fe
src/backup/index.rs: new ChunkListReader
2019-05-11 16:01:42 +02:00
Dietmar Maurer
af1d68185c
src/api2/admin/datastore/backup.rs: add missing 'archive-name' parameter to schema
2019-05-11 12:25:15 +02:00
Dietmar Maurer
6b95c7df07
src/api2/admin/datastore/backup/environment.rs: do not store path
2019-05-11 12:13:44 +02:00
Dietmar Maurer
51a4f63ff7
src/backup/datastore.rs: move last_backup to BackupInfo
2019-05-11 12:07:09 +02:00
Dietmar Maurer
b02a52e3a5
src/backup/datastore.rs: impl last_backup helper
2019-05-11 11:36:46 +02:00
Dietmar Maurer
c09775011a
src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups
2019-05-11 10:19:34 +02:00
Dietmar Maurer
f9578f3c79
backup-api: allow to create DynamicIndexWriter and add chunks to to
2019-05-10 10:25:40 +02:00
Dietmar Maurer
35a2d8a6a6
src/backup/backup_info.rs: Derive Clone
2019-05-10 06:59:23 +02:00
Dietmar Maurer
0506d1ed04
upload_chunk.rs: add comment about possible unnecessary code
2019-05-10 06:15:25 +02:00
Dietmar Maurer
5f42ef009a
cleanup: remove unused code
2019-05-09 18:11:39 +02:00
Dietmar Maurer
b4b63e520d
handle_async_api_request: put rpcenv into a Box
...
So that we can pass rpcenv into futures.
2019-05-09 18:01:24 +02:00
Dietmar Maurer
2c41096a4b
src/api2/admin/datastore/backup/upload_chunk.rs: fix maximal chunk size
2019-05-09 13:29:50 +02:00
Dietmar Maurer
78b5191550
src/server/formatter.rs: further cleanups and renaming ...
2019-05-09 13:28:26 +02:00
Dietmar Maurer
d55037e471
src/server/formatter.rs: rename format_result to format_data
...
To avoid confusions with Rust Result type.
2019-05-09 13:15:15 +02:00
Dietmar Maurer
21ee7912fa
src/api2/admin/datastore/backup.rs: implement upload chunk
2019-05-09 13:06:09 +02:00
Dietmar Maurer
4ebf0eabb0
src/api2/types.rs: use separate file for schema definitions
2019-05-09 07:44:09 +02:00
Dietmar Maurer
bd1507c4fa
src/api2/admin/datastore/backup/service.rs: move service code into extra file
2019-05-08 17:36:19 +02:00
Dietmar Maurer
0aadd40b4f
src/api2/admin/datastore/backup.rs: add required parameters, cleanups
2019-05-08 13:17:23 +02:00
Dietmar Maurer
d95ced64ad
src/api2/admin/datastore/backup/environment.rs: use separate file
2019-05-08 12:41:58 +02:00
Dietmar Maurer
ca60c371e2
api: rename h2upload to backup
2019-05-08 12:35:15 +02:00
Dietmar Maurer
d906b1124f
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 11:36:43 +02:00
Dietmar Maurer
58c8d7d91f
rc/api2/admin/datastore/h2upload.rs: implement BackupEnvironment
...
To pass arbitrary information/state to api methods.
2019-05-08 11:26:54 +02:00
Dietmar Maurer
f757b30efc
src/server/rest.rs: use generics to pass RpcEnvironment
2019-05-08 11:09:01 +02:00
Dietmar Maurer
2edc341b29
src/tools.rs: Add AsAyn Trait
2019-05-08 11:07:36 +02:00
Dietmar Maurer
090ac9f79a
src/api2/admin/datastore/h2upload.rs: avoid unwrap
2019-05-08 08:21:18 +02:00
Dietmar Maurer
4e3da4b366
src/api2/admin/datastore/h2upload.rs: do not log client address
2019-05-08 08:15:55 +02:00
Dietmar Maurer
e3a44552ee
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 08:12:49 +02:00
Dietmar Maurer
a66ab8ae5e
src/api2/admin/datastore/h2upload.rs: make worker abortable
2019-05-08 07:14:40 +02:00
Dietmar Maurer
72375ce6c6
src/api2/admin/datastore/h2upload.rs: log to worker task
2019-05-07 14:07:00 +02:00
Dietmar Maurer
d9bd06eae8
src/api2/admin/datastore/h2upload.rs: use WorkerTask
2019-05-07 13:42:00 +02:00
Dietmar Maurer
52cf506e48
src/api2/admin/datastore/h2upload.rs: implement async test api call
2019-05-07 12:26:55 +02:00
Dietmar Maurer
92ac375aff
src/api2/admin/datastore/h2upload.rs: improve http2 service
2019-05-07 11:24:44 +02:00
Dietmar Maurer
78f7e225e4
src/server/rest.rs: make handle_(a)sync_api_request public
2019-05-07 11:23:52 +02:00
Dietmar Maurer
75097495ad
RestEnvironment: derive Clone
2019-05-07 11:09:18 +02:00
Dietmar Maurer
01bf3b7b5f
move find_method to Router class
2019-05-07 11:08:30 +02:00
Dietmar Maurer
3578d99f3e
move normalize_path to tools::normalize_uri_path
2019-05-07 09:44:34 +02:00
Dietmar Maurer
96e95fc179
src/api2/admin/datastore/h2upload.rs: test if version < 2
...
We can only upgrade http 1.x connections.
2019-05-06 10:29:34 +02:00
Dietmar Maurer
45cdce069e
src/client/http_client.rs: request password at creation time
2019-04-30 11:44:35 +02:00
Dietmar Maurer
e45afdff9c
src/server/state.rs: use new BroadcastData helper
2019-04-30 10:21:48 +02:00
Dietmar Maurer
824b5ee4ee
src/tools/broadcast_future.rs: cleanup, decompose into two classes
...
In order to make it more usable.
2019-04-30 08:57:20 +02:00
Wolfgang Bumiller
d82ed9b0f5
tools: add AsyncMutex
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-29 14:19:03 +02:00
Dietmar Maurer
3a9bea3918
src/bin/h2test.rs: test h2 upgrade
2019-04-29 11:59:18 +02:00
Dietmar Maurer
cf639a47de
rc/client/http_client.rs: add experimental h2 upgrade code
2019-04-29 11:57:58 +02:00
Dietmar Maurer
5a2df00004
src/client/http_client.rs: cleanup, make login fully async
2019-04-28 10:55:03 +02:00
Dietmar Maurer
3dceb9b304
src/tools/broadcast_future.rs: add new constructor new_oneshot()
...
To simplify usage. Also added a test case.
2019-04-27 10:56:49 +02:00
Dietmar Maurer
152764ec15
src/api2/admin/datastore/h2upload.rs: add test code for http2
2019-04-27 08:57:35 +02:00
Dietmar Maurer
490be29ed6
src/tools/broadcast_future.rs: new helper class
2019-04-26 17:57:57 +02:00
Wolfgang Bumiller
e888155712
proxy: allow .key/.pem files instead of a pkcs12/.pfx file
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-26 10:47:33 +02:00
Christian Ebner
8b7c0def84
src/tools/procfs.rs: clean up tab indentation and use whitespaces instead
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-25 13:14:52 +02:00
Wolfgang Bumiller
5e5eed5c3b
use double-fork for reload
...
To ensure the new process' parent is pid 1, so systemd won't
complain about supervising a process it does not own.
Fixes the following log spam on reloads:
Apr 25 10:50:54 deb-dev systemd[1]: proxmox-backup.service: Supervising process 1625 which is not our child. We'll most likely not notice when it exits.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
Wolfgang Bumiller
efd1536eb7
add a wrapper around nix::unistd::pipe2
...
Using O_CLOEXEC by default, and returning Fd handles to
ensure they get dropped on bail!() or panic!() if the RawFds
aren't used yet.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
Wolfgang Bumiller
897982e237
tools: add read/write convenience helpers
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
Wolfgang Bumiller
d98c9a7a38
use service Type=notify
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:33 +00:00
Wolfgang Bumiller
9c351a36c4
tools/daemon: add sd_notify wrapper
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:28 +00:00
Dietmar Maurer
d8871f2f18
src/tools/procfs.rs: avoid useless allocations
2019-04-24 09:35:15 +02:00
Christian Ebner
ae694f44ef
src/tools/procfs.rs: add tests for read_proc_net_route() and read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:52 +02:00
Christian Ebner
ec8db837fb
src/tools/procfs.rs: refactor read_proc_net_route() and read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:32 +02:00
Christian Ebner
eb46c3d1ec
src/tools/procfs.rs: implement read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-23 14:31:23 +02:00
Dietmar Maurer
6358b68236
cleanup: avoid compiler warnings
2019-04-23 12:17:24 +02:00
Dietmar Maurer
e06ffb10d1
src/api_schema/router.rs: fix typo (subdira -> subdir)
2019-04-16 12:20:17 +02:00
Dietmar Maurer
13f1cc17ea
src/api_schema/router.rs: implement list_subdirs() helper
2019-04-16 12:07:02 +02:00
Wolfgang Bumiller
062d4916ff
api_schema: allow generic api handler functions
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-16 11:22:23 +02:00
Christian Ebner
286f0d4099
src/tools/procfs.rs: implement read_proc_net_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-15 11:13:16 +02:00
Dietmar Maurer
1a6c9415ae
src/api2/node/services.rs: add proxmox-backup-proxy service
2019-04-15 09:48:02 +02:00
Dietmar Maurer
75bc49bed7
src/server/worker_task.rs: implement abort channel
2019-04-15 09:38:05 +02:00
Dietmar Maurer
912524752e
src/api2/admin/datastore/pxar.rs: create worker task for file upload
2019-04-15 08:38:14 +02:00
Dietmar Maurer
1be71fb02b
src/server/worker_task.rs_ do not write ERROR: prefix into task index
2019-04-15 08:37:13 +02:00
Dietmar Maurer
418def7a78
src/server/worker_task.rs: move common code into log_result
...
Make new() pub
2019-04-15 07:49:03 +02:00
Dietmar Maurer
37b87869b5
src/server/upid.rs: check for illegal characters
2019-04-15 07:47:51 +02:00
Christian Ebner
d2981e2738
src/tools/procfs.rs: implement read_proc_net_dev()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:59 +02:00
Christian Ebner
13abfdc57a
src/tools/procfs.rs: implement read_memory_usage()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:01 +02:00
Christian Ebner
3005ef48b6
src/tools/procfs.rs: Rename variable and struct to be more consistent
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:34 +02:00
Christian Ebner
cc93bacb72
src/tools/procfs.rs: simplify read_cpuinfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:14 +02:00
Dietmar Maurer
f2b99c34f7
src/api2/admin/datastore.rs: implement API to return last GC status
2019-04-11 12:04:25 +02:00
Dietmar Maurer
59961b8954
src/server/command_socket.rs: check control socket permissions
2019-04-11 10:51:59 +02:00
Dietmar Maurer
ba70040dbf
src/server/worker_task.rs: change task list sort order - latest on top
2019-04-11 09:43:33 +02:00
Dietmar Maurer
d3f4c08f33
src/server/worker_task.rs: catch panics in worker threads
2019-04-11 07:56:06 +02:00
Wolfgang Bumiller
620dccf1a1
tools/daemon: dup the TcpListener file descriptor
...
Now that we let hyper shutdown gracefully we need an owned
version of the listening socket to prevent it from closing
before running the reload preparations.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
Wolfgang Bumiller
d96bb7f163
tools: add Fd helper
...
stores a raw file descriptor with a drop handler for
safekeeping in closures
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
Christian Ebner
1cd33633aa
/src/tools: Cleanup tempfiles in folder
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:56:22 +02:00
Dietmar Maurer
a665dea128
src/api2/node/tasks.rs: implement stop_task api
2019-04-10 13:55:05 +02:00
Dietmar Maurer
98a181f055
src/server/worker_task.rs: log message when we call request_abort()
2019-04-10 13:54:28 +02:00
Christian Ebner
2b6a3e1396
src/tools/procfs.rs: implement read_cpuinfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:11:11 +02:00
Dietmar Maurer
321070b4fa
src/server/worker_task.rs: implement abort_worker (via command_socket)
2019-04-10 12:42:24 +02:00
Dietmar Maurer
cfb2d3c17c
src/server/command_socket.rs: correctly handle/spawn handle parallel connections
2019-04-10 11:05:00 +02:00
Dietmar Maurer
9b002cbc5f
src/server/worker_task.rs: use abstract socket
2019-04-10 09:03:17 +02:00
Dietmar Maurer
5f550fd99f
start hyper server using with_graceful_shutdown()
...
Without, hyper keeps some futures running, and the server does not
correctly shutdown.
2019-04-10 08:24:32 +02:00
Dietmar Maurer
7f2078de59
src/tools/procfs.rs: avoid compiler warning
2019-04-10 06:40:29 +02:00
Christian Ebner
1f6fbf1c57
src/tools/procfs.rs: implement read_meminfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-09 15:21:49 +02:00
Dietmar Maurer
c360bd73d3
src/api2/node/tasks.rs: fix task status api
2019-04-09 15:12:20 +02:00
Dietmar Maurer
0bfd87bc5e
src/server/worker_task.rs: speedup upid_read_status()
2019-04-09 15:11:48 +02:00
Dietmar Maurer
d8d40dd0c2
src/api2/node.rs: add node parameter
2019-04-09 14:43:30 +02:00
Dietmar Maurer
7ae0bf52dd
src/server/command_socket.rs: code cleanup - fix error message
2019-04-09 12:47:42 +02:00
Dietmar Maurer
d607b8861b
src/server/worker_task.rs: implement task control socket
2019-04-09 12:15:06 +02:00
Dietmar Maurer
e201753629
src/server/command_socket.rs: implement auto_remove flag
...
Remove the socket file on close.
2019-04-09 11:47:23 +02:00
Dietmar Maurer
634132fe10
src/server/upid.rs: moved code into separate file
2019-04-08 18:43:26 +02:00
Dietmar Maurer
78a39e0517
src/server/command_socket.rs: simple command socket
2019-04-08 17:59:39 +02:00
Dietmar Maurer
e3f41f2165
src/tools/daemon.rs: use new ServerState handler
2019-04-08 14:00:23 +02:00
Dietmar Maurer
7a630df709
implement server state/signal handling, depend on tokio-signal
2019-04-08 13:59:07 +02:00
Dietmar Maurer
5a12c0e2fb
src/api2/node/tasks.rs: start upid API
2019-04-07 14:36:57 +02:00
Dietmar Maurer
b75b968147
src/server/worker_task.rs: implement is_active()
2019-04-07 14:36:25 +02:00
Dietmar Maurer
4494d07819
src/server/worker_task.rs: make log_path a method on UPID
2019-04-07 13:51:46 +02:00
Dietmar Maurer
d2a2e02bb7
src/api2/node/tasks.rs: implement userfilter
2019-04-07 13:17:19 +02:00
Dietmar Maurer
3c3bee2e32
src/api2/node/tasks.rs: correctly handle start parameter
2019-04-07 12:41:24 +02:00
Dietmar Maurer
063ca5be77
src/api2/node/tasks.rs: implement list_tasks
2019-04-07 12:18:58 +02:00
Dietmar Maurer
d4b59ae0b8
src/backup/datastore.rs: pass WorkerTask to GC
...
So that we can log to the task log file.
2019-04-06 17:57:38 +02:00
Dietmar Maurer
35950380a9
src/server/worker_task.rs: carefully handle file permissions
2019-04-06 17:53:12 +02:00
Dietmar Maurer
1619a72063
src/tools.rs: implement create_dir_chown
...
Combinded mkdir with chown.
2019-04-06 16:50:23 +02:00
Dietmar Maurer
eea8131952
src/tools.rs: implement file_set_contents_full()
2019-04-06 13:53:43 +02:00
Dietmar Maurer
0f778e0652
src/api2/admin/datastore.rs: use new WorkerTask::new_thread()
2019-04-06 11:27:23 +02:00
Dietmar Maurer
244abab769
src/tools/file_logger.rs: fix test
2019-04-06 11:24:37 +02:00
Dietmar Maurer
660c684640
src/server/worker_task.rs: return upid_str
2019-04-06 11:23:53 +02:00
Dietmar Maurer
93aebb38bc
src/server/worker_task.rs: implement read_task_list()
2019-04-06 10:17:11 +02:00
Dietmar Maurer
4a36a2c999
src/server/worker_task.rs: improve UPID format description
2019-04-06 09:36:50 +02:00
Dietmar Maurer
882594c5e9
src/server.rs: improve crate layout
2019-04-06 09:17:25 +02:00
Dietmar Maurer
4b01c983f0
src/server/worker_task.rs: implement persistent task list
2019-04-05 12:37:35 +02:00
Christian Ebner
a2474d328f
src/tools/procfs.rs: implement read_proc_uptime and read_proc_uptime_ticks
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-05 06:22:21 +02:00
Christian Ebner
08daa81af6
procfs: implement check_process_running and check_process_running_pstart
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-04 13:54:17 +02:00
Dietmar Maurer
9a0d0ff774
server/worker_task.rs: implement UPID parser
2019-04-04 13:49:35 +02:00
Dietmar Maurer
49cf9f3d0b
src/tools.rs: file_get_json() - add new default parameter
2019-04-04 12:24:18 +02:00
Dietmar Maurer
79ad7b7ec2
src/tools/procfs.rs: split regex into several lines
2019-04-04 09:25:19 +02:00
Dietmar Maurer
479f6e40ec
src/server/worker_task.rs: implement task registry
2019-04-04 08:58:39 +02:00
Dietmar Maurer
52ddc2522d
src/tools/procfs.rs: use libc::pid_t
2019-04-04 08:05:43 +02:00
Dietmar Maurer
d3d7ab1273
src/tools/file_logger.rs: new - accept AsRef<Path>
2019-04-03 14:13:33 +02:00
Dietmar Maurer
3c2012f97a
src/tools/procfs.rs: start helper tools to read procfs
2019-04-03 13:39:20 +02:00
Dietmar Maurer
3489936ebb
src/tools/file_logger.rs: change timestamp format to rfc3339
2019-04-03 08:58:43 +02:00
Dietmar Maurer
92da93b245
abort GC on server shutdown
2019-04-01 12:13:02 +02:00
Dietmar Maurer
9136f857cc
add global var to indicate server shutdown requests
2019-04-01 12:05:11 +02:00
Dietmar Maurer
3e6a7deeaa
src/api2/admin/datastore.rs: start GC in a separate thread
2019-04-01 12:02:46 +02:00
Dietmar Maurer
ad281d1bd9
www/DataStoreConfig.js: add button to start GC
2019-04-01 08:08:34 +02:00
Dietmar Maurer
49d123ee3a
src/server/rest.rs: use formatter to encode errors
2019-04-01 08:04:12 +02:00
Dietmar Maurer
7d4ef127cb
src/server/rest.rs: generate csrf token if we have a valid ticket
...
This is important if the user reloads the browser page.
2019-04-01 07:52:30 +02:00
Dietmar Maurer
11861a482d
src/backup/chunk_store.rs: fix GC
...
Added option to get oldest_writer timestamp from ProcessLocker.
2019-03-31 17:21:36 +02:00
Dietmar Maurer
d85987aeeb
fix last commit: the filename var was not ment to be removed, sorry
2019-03-31 16:16:14 +02:00
Dietmar Maurer
15a77c4c2e
src/backup/chunk_store.rs: avoid create/unlink race
2019-03-31 10:03:01 +02:00
Dietmar Maurer
c374f05499
src/backup.rs: improve GC problem description
2019-03-31 09:44:35 +02:00
Dietmar Maurer
c8ec450e37
src/backup.rs: start explaining different GC algorithm
2019-03-30 17:21:40 +01:00
Dietmar Maurer
8a4757342c
src/backup.rs: describe the garbage collection problem
2019-03-30 16:26:52 +01:00
Dietmar Maurer
10eea49d81
src/backup/fixed_index.rs: store reference to file
...
Keep it open and locked while index is in use.
2019-03-27 10:17:03 +01:00
Dietmar Maurer
c597a92c81
index file readers: aquire shared lock
2019-03-27 10:11:14 +01:00
Dietmar Maurer
0465218953
src/backup.rs: add documentation about ChunkStore locking
2019-03-22 10:14:50 +01:00
Dietmar Maurer
43b1303398
datastore: use new ProcessLocker
...
To make sure only one process runs garbage collection while having active writers.
2019-03-22 09:42:15 +01:00
Dietmar Maurer
abfc001f25
src/tools/process_locker.rs: new() - use generic path parameter
2019-03-22 09:41:32 +01:00
Dietmar Maurer
a650f50301
src/tools/process_locker.rs: implement inter-process reader-writer locks
2019-03-22 08:05:51 +01:00
Wolfgang Bumiller
eb90c9e372
tools: style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-20 09:57:13 +01:00
Christian Ebner
e04d1bfadc
Cleanup of if statement with unwraps by a more compact match statement
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-03-20 09:49:18 +01:00
Dietmar Maurer
8346f0d59b
src/server/rest.rs: correctly extract content type
2019-03-19 12:50:15 +01:00
Wolfgang Bumiller
a690ecacf0
daemon: simplify daemon creation
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 12:12:54 +01:00
Wolfgang Bumiller
e4311382d9
tools: daemon: rename some structs
...
Reloadable resources are now 'Reloadable' instead of
'ReexecContinue'.
The struct handling the reload is a 'Reloader', not a
'ReexecStore'.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 11:09:46 +01:00
Wolfgang Bumiller
5e7bc50af1
add reload support to api daemon
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
a2ca713701
add reload support to proxy
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
af70c1811b
tools: implement ReexecContinue for tokio's TcpListener
...
This is the only thing we currently need to keep alive for
reloads.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
4422ba2c7f
tools: daemon: add a default signalfd helper
...
Proxy and daemon for now just want to handle reload via
`SIGHUP`, so provide a helper creating the signalfd stream
doing that - this is simply a filtered stream which passes
the remaining signals through, so it can be used exactly
like the signalfd stream could before to add more signals.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
dce94d0e09
tools: add daemon helpers
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
ff7049d481
tools: add fd_change_cloexec helper
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
1c042cdc6c
proxy: listener error handling fixup
...
If the listening socket goes into some error state we'll get
std::io::Errors rather than higher level errors from the
native_tls::TlsAcceptor, those are usually fatal. (Ran into
this after performing a shutdown() on the file descriptor,
after which the future just endlessly loops in accept().)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Dietmar Maurer
60864b1b56
src/pxar.rs: improve docu
2019-03-18 12:27:30 +01:00
Dietmar Maurer
515688d1f1
avoid compiler warnings
2019-03-18 10:00:58 +01:00
Dietmar Maurer
ba10f2b0d4
src/bin/dynamic-files.rs: test case for file shrink during backup
2019-03-16 14:22:14 +01:00
Dietmar Maurer
af572aaa4c
pxar: implement hardlinks
...
So we are no longer compatible with catar ...
2019-03-16 11:02:12 +01:00
Dietmar Maurer
812dd935ce
src/pxar/encoder.rs: detect hardlinks
2019-03-15 12:19:51 +01:00
Dietmar Maurer
896e803feb
src/pxar/sequential_decoder.rs: dump_entry - fix FIFO/SOCKET handling
2019-03-15 12:15:38 +01:00
Dietmar Maurer
7dfa17c7cc
src/pxar/binary_search_tree.rs: fix test
2019-03-15 11:34:31 +01:00
Dietmar Maurer
9307279faf
cleanup: remove dead code
2019-03-15 10:18:28 +01:00
Dietmar Maurer
46cfe7ab7e
src/pxar/decoder.rs: implement seekable decoder
2019-03-15 09:36:05 +01:00
Dietmar Maurer
0e21b87be3
renamed: src/pxar/decoder.rs -> src/pxar/sequential_decoder.rs
2019-03-15 08:36:02 +01:00
Dietmar Maurer
3dbfe5b142
simplify pxar module structure
2019-03-15 08:24:32 +01:00
Dietmar Maurer
1ef46b81d3
src/bin/pxar.rs: implement extract
2019-03-15 08:03:44 +01:00
Dietmar Maurer
02c93163cf
src/pxar/decoder.rs: correctly call callback()
2019-03-15 08:02:04 +01:00
Dietmar Maurer
5defa71bf9
rename PxarBackupWriter into PxarDecodeWriter
2019-03-15 07:20:22 +01:00
Dietmar Maurer
b005ed12d5
src/client/http_client.rs: avoid compiler warning
2019-03-15 07:16:26 +01:00
Dietmar Maurer
9e26abf16b
src/cli/command.rs: fix iterator usage
2019-03-15 07:12:40 +01:00
Dietmar Maurer
f98746ba6c
src/pxar/decoder.rs: improve docu
2019-03-15 07:12:11 +01:00
Dietmar Maurer
0b2d7fdc81
src/pxar/decoder.rs: avoid compiler warnings
2019-03-14 17:45:47 +01:00
Dietmar Maurer
37940aa1f9
src/bin/pxar.rs: implement dump/list
2019-03-14 17:43:11 +01:00
Dietmar Maurer
40360fde6e
src/pxar/decoder.rs: implement dump_archive
...
use this to debug archive problems ...
2019-03-14 13:10:27 +01:00
Dietmar Maurer
8968258b66
rename catar into pxar
...
To avoid confusion with the casync implementation.
2019-03-14 10:54:09 +01:00
Dietmar Maurer
7c4dd94670
rename catar into pxar
...
Seems impossible to stay compatible with the original catar specification,
so we change the name to avoid confusions.
2019-03-14 10:14:32 +01:00
Dietmar Maurer
806500cd09
src/tools.rs: improve file name completion
...
Use current working directory by default.
2019-03-14 08:17:35 +01:00
Dietmar Maurer
08dc340a12
src/bin/proxmox-backup-client.rs: complete archive names
2019-03-14 08:09:35 +01:00
Wolfgang Bumiller
6f90a6a764
protocol: cleanup finish_backup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-13 14:16:17 +01:00
Dietmar Maurer
b238851832
src/bin/proxmox-backup-client.rs: implement completion for snapshots
2019-03-13 13:31:29 +01:00
Dietmar Maurer
4981134708
src/bin/proxmox-backup-client.rs: implement chunk-size completion
2019-03-13 12:26:01 +01:00
Dietmar Maurer
98f0b97293
src/bin/proxmox-backup-client.rs: cleanup
2019-03-13 12:17:39 +01:00
Dietmar Maurer
024f11bb55
src/bin/proxmox-backup-client.rs: implement completion for backup groups
2019-03-13 11:56:37 +01:00
Dietmar Maurer
edd3c8c605
src/client/backup_repo.rs: implement FromStr trait
2019-03-13 10:09:39 +01:00
Dietmar Maurer
874acb7039
src/client/backup_repo.rs: implement Display trait
2019-03-13 09:57:36 +01:00
Dietmar Maurer
d0a03d40ce
src/bin/proxmox-backup-client.rs: implement repository completions
2019-03-13 09:47:12 +01:00
Dietmar Maurer
bff110308b
src/bin/proxmox-backup-client.rs: fix BACKUPSPEC_REGEX - correctly specify non-capture group
...
And improve bash completion.
2019-03-13 07:23:59 +01:00
Dietmar Maurer
1a71509aad
src/cli/command.rs: fix completion - never display argument parameters as option
2019-03-12 14:53:41 +01:00
Dietmar Maurer
496a67846f
src/cli/command.rs: pass parsed parameters to completion function
2019-03-12 14:39:51 +01:00
Dietmar Maurer
b5fa28251c
src/cli/getopts.rs: factor out reusable code
2019-03-12 14:38:47 +01:00
Dietmar Maurer
e39efdbd33
src/cli/command.rs: use a HashMap to store fixed param as key/value pairs
2019-03-12 12:54:16 +01:00
Dietmar Maurer
f4e4d58327
src/cli/command.rs: improve completions
2019-03-12 12:07:45 +01:00
Wolfgang Bumiller
4ba7b6099c
signalfd cleanup
...
- add comment about the boxed file descriptor (and others)
- reexport SigSet and Signal for convenience
- remove Result wrapper (only used once)
- enforce drop order of the PollEvented2 struct with respect
to the boxed fd
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-12 11:07:09 +01:00
Wolfgang Bumiller
f54c19989c
tools: add async signalfd handler
...
This provides a Stream<Item = siginfo> via nix' signalfd,
by wrapping it in tokio's PollEvented2 struct to allow
polling via tokio's event loop.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-12 08:04:28 +01:00
Dietmar Maurer
d5c34d98c0
src/catar/decoder.rs: simplify public restore API
2019-03-11 14:31:01 +01:00
Dietmar Maurer
8e39232acc
src/bin/proxmox-backup-client.rs: strip .didx file extensions
2019-03-11 10:54:02 +01:00
Dietmar Maurer
74cdb52107
src/bin/proxmox-backup-client.rs: include file extension in backupspec
2019-03-11 10:27:28 +01:00
Dietmar Maurer
fc2bf37ead
src/catar/decoder.rs: cleanup style
2019-03-11 08:14:28 +01:00
Dietmar Maurer
656b23e107
src/catar/decoder.rs: dir_mkdirat - add create_new flag
2019-03-11 07:27:45 +01:00
Dietmar Maurer
e9c9409a99
src/catar/decoder.rs: implement real sequential decoder (no Seek)
...
I copied the old code to src/catar/inspector.rs. Will modify
that later to implement a random access decoder ...
2019-03-08 16:55:54 +01:00
Dietmar Maurer
eed6db3923
src/bin/proxmox-backup-client.rs: impl option all-file-systems
2019-03-08 09:36:40 +01:00
Wolfgang Bumiller
06178f13fa
move chunker into proxmox_protocol
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-08 09:29:47 +01:00
Dietmar Maurer
e3c30c50a6
src/bin/catar.rs: implement all-file-systems option
2019-03-08 09:28:12 +01:00
Dietmar Maurer
65092b1e27
src/catar/encoder.rs: log skipped mount points
2019-03-08 09:25:00 +01:00
Dietmar Maurer
6b64d25d64
src/catar/encoder.rs: remove debug output
2019-03-08 08:23:07 +01:00
Dietmar Maurer
2689810c92
src/bin/catar.rs: implement verbose flags
2019-03-08 08:20:56 +01:00
Dietmar Maurer
79c281fdac
src/catar/encoder.rs: use simply flag "all_file_systems"
2019-03-08 08:14:26 +01:00
Dietmar Maurer
25e77d3877
src/cli/command.rs: print_bash_completion - avoid crash
2019-03-07 12:14:26 +01:00
Dietmar Maurer
141f062e08
src/backup/chunk_store.rs: use zstd compression insteadf of lz4
...
Provides better compressionm rate, and is still fast.
2019-03-07 11:42:59 +01:00
Dietmar Maurer
d2690f74bb
src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end
2019-03-07 10:40:01 +01:00
Dietmar Maurer
2ae7d196b9
src/bin/proxmox-backup-client.rs: correctly strip file extension
2019-03-06 11:18:46 +01:00
Dietmar Maurer
9f91249351
src/bin/proxmox-backup-client.rs: impl restore
2019-03-06 11:02:08 +01:00
Dietmar Maurer
01a1342364
src/api2/admin/datastore.rs: impl list_snapshot_files
2019-03-06 11:02:08 +01:00
Dietmar Maurer
58e99e138e
src/backup/backup_info.rs: new helper to list files
2019-03-06 11:02:08 +01:00
Dietmar Maurer
dd79265a8a
src/api2/admin/datastore/catar.rs: use archive_name directly
2019-03-06 11:02:08 +01:00
Dietmar Maurer
a6b7551311
src/client/http_client.rs: impl download
2019-03-06 11:02:08 +01:00
Wolfgang Bumiller
6716f30bbc
add protocol test client
...
This uses futures for everything which is mostly useful as a
test to see if the protocol crate's non-blocking I/O support
can handle it...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:26:12 +01:00
Wolfgang Bumiller
e2d007f76e
api2/admin/datastore: add a backup protocol test api path
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:26:12 +01:00
Wolfgang Bumiller
e3062f87b1
fixed_index: split add_chunk/add_digest
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
c3bb97e59c
fixed_index: impl Send for FixedIndexWriter
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
3b7ade9e78
backup/datastore: expose insert_chunk method
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
91ebea8262
router: add upgrade method
...
This is just an Async GET method, but semantically
different, so it gets its own router method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
Dietmar Maurer
80822b95b1
src/bin/proxmox-backup-client.rs: sort list results
2019-03-06 07:20:51 +01:00
Dietmar Maurer
fba30411fb
src/bin/proxmox-backup-client.rs: allow to set backup-id
2019-03-06 07:02:52 +01:00
Dietmar Maurer
21ea0158b4
src/client/http_client.rs: remove debug statement
2019-03-06 06:45:51 +01:00
Dietmar Maurer
ba3a60b2a4
src/client/http_client.rs: store/load ticket in xdg runtime dir, depend on crate xdg
2019-03-05 12:56:21 +01:00
Dietmar Maurer
f8f94534d4
src/api2/access.rs: authenticate_user() - add ticket login
2019-03-05 12:56:21 +01:00
Dietmar Maurer
e5662b04e9
src/tools/ticket.rs: define const TICKET_LIFETIME
2019-03-05 12:56:21 +01:00
Dietmar Maurer
53cafb5925
src/tools.rs: impl file_get_json()
2019-03-05 12:56:21 +01:00
Wolfgang Bumiller
391d310741
backup_info: BackupDir shortcut
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 10:15:06 +01:00
Wolfgang Bumiller
93b49ce38a
backup_info: use two independent Into<String> types
...
there's no reason to force them to be the same input type
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 09:13:08 +01:00
Dietmar Maurer
3ec3ec3f9a
src/bin/proxmox-backup-client.rs: print backup duration
2019-03-05 08:11:40 +01:00
Dietmar Maurer
d57474e055
src/backup/backup_info.rs: improve docs
2019-03-05 07:28:13 +01:00
Dietmar Maurer
b3483782d1
src/backup/backup_info.rs: move code into separate file
...
Also changed create_backup_dir() parameters - uses &BackupDir now.
2019-03-05 07:18:12 +01:00
Dietmar Maurer
2b01a22507
src/backup/datastore.rs: add helper to sort backup lists
2019-03-04 18:20:57 +01:00
Dietmar Maurer
875fb1c01a
src/backup/datastore.rs: use unix epoch to create DateTime
...
To make sure that we have a timestamp without nanosecond.
2019-03-04 17:58:22 +01:00
Dietmar Maurer
38f8815925
src/backup/datastore.rs: use new() function to create structs
2019-03-04 17:44:27 +01:00
Dietmar Maurer
9b492eb256
src/backup/datastore.rs: protect BackupDir fields, impl new()
2019-03-04 13:51:36 +01:00
Dietmar Maurer
1e9a94e579
src/backup/datastore.rs: protect BackupGroup fields, impl new()
2019-03-04 13:38:23 +01:00
Dietmar Maurer
cdebd467e6
src/bin/proxmox-backup-client.rs: simply log to stdout
2019-03-04 09:38:34 +01:00
Dietmar Maurer
f72dfd2d10
src/backup/datastore.rs: make sure timestamp does not contains nanoseconds
...
Because we want simple, readable file names.
2019-03-04 09:37:10 +01:00
Dietmar Maurer
219ef0e6ca
src/catar/encoder.rs: add verbose flags
2019-03-04 08:01:09 +01:00
Dietmar Maurer
e0b8ce6424
src/catar/encoder.rs: skip RAMFS
...
unless explicitly selected.
2019-03-04 06:48:47 +01:00
Dietmar Maurer
597a92039e
src/bin/proxmox-backup-client.rs: rename create into backup
2019-03-03 11:46:53 +01:00
Dietmar Maurer
6f62c9240a
src/api2/admin/datastore.rs: imp delete_snapshot
2019-03-03 11:29:00 +01:00
Dietmar Maurer
af926291e0
avoid compiler warning
2019-03-03 09:52:06 +01:00
Dietmar Maurer
e77a02ed87
catar api: cleanup parameter names
2019-03-03 09:48:32 +01:00
Dietmar Maurer
421c9ba1d9
src/bin/proxmox-backup-client.rs: use new json_object_to_query
2019-03-03 09:44:25 +01:00
Dietmar Maurer
f5f13ebc5a
src/tools.rs: new helper json_object_to_query
2019-03-03 08:51:37 +01:00
Dietmar Maurer
49dc074007
parameter name cleanups (use '-' instead of '_')
2019-03-02 16:34:10 +01:00
Dietmar Maurer
ad20d19877
src/api2/admin/datastore.rs: rename get_group_list to list_groups, cleanups
2019-03-02 16:28:36 +01:00
Dietmar Maurer
184f17afad
src/api2/admin/datastore.rs: impl list_snapshots
2019-03-02 16:20:50 +01:00
Dietmar Maurer
af53186e6a
src/tools.rs: add simply string join
2019-03-02 16:15:45 +01:00
Dietmar Maurer
812c6f8716
src/api2/admin/datastore.rs: impl list_backup_groups
2019-03-02 11:29:05 +01:00
Dietmar Maurer
e909522f06
src/backup/datastore.rs: use timestamp with zone offset
2019-03-01 17:36:40 +01:00
Dietmar Maurer
dc4c09faae
src/backup/datastore.rs: split BackupDir
2019-03-01 16:26:35 +01:00
Dietmar Maurer
38b0dfa511
src/backup/datastore.rs: split BackupInfo
2019-03-01 16:26:35 +01:00
Wolfgang Bumiller
dcd15d10bb
backup/index: remove IndexIterator
...
removed the need for it, to avoid copies
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-01 13:29:36 +01:00
Dietmar Maurer
17d6979afd
src/bin/proxmox-backup-client.rs: generate a log file
2019-03-01 09:35:41 +01:00
Dietmar Maurer
3b151414f8
src/tools/file_logger.rs: class to log into files
2019-03-01 09:34:29 +01:00
Dietmar Maurer
5a2892f0ae
src/bin/proxmox-backup-client.rs: use chrono::DateTime
2019-03-01 06:59:32 +01:00
Dietmar Maurer
51144821f9
src/bin/proxmox-backup-client.rs: login before starting backup
2019-03-01 06:48:41 +01:00
Dietmar Maurer
541b908ea4
src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch
2019-02-28 17:03:48 +01:00
Dietmar Maurer
b55cee9227
src/cli/command.rs: generate_usage_str() - fix option separator (newline)
2019-02-28 16:20:00 +01:00
Dietmar Maurer
c4c5174000
src/cli/command.rs: get_schema_type_text() - display min/max
2019-02-28 16:11:49 +01:00
Wolfgang Bumiller
df67070b1c
client/backup_repo: allow ip addresses for BackupRepository
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5de2bced2d
datastore: add generic open_index
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
7bc1d72778
add IndexFile trait
...
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.
Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5be4065baf
make FixedIndexReader Send and update comments
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
fded74d0c7
style: BufferedReader -> BufferedRead
...
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
Dietmar Maurer
8f57971714
src/api2/admin/datastore.rs: implement prune
2019-02-28 12:51:27 +01:00
Dietmar Maurer
83b7db0209
src/api2/admin/datastore.rs: start prune api
...
just a dummy for now.
2019-02-27 16:53:17 +01:00
Dietmar Maurer
0ffbccce5e
rc/server/rest.rs: allow to pass parameters as application/json
2019-02-27 12:37:53 +01:00
Dietmar Maurer
f98ea63de2
src/bin/proxmox-backup-client.rs: fix bash completion
2019-02-27 12:37:53 +01:00
Wolfgang Bumiller
8731e40a7f
backup/datastore: create_backup_dir: show if path existed
...
To enable asserting the creation of a new backup rather than
adding to an existing one.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-27 10:15:36 +01:00
Dietmar Maurer
25f1650b71
src/bin/proxmox-backup-client.rs: minor code cleanup
2019-02-27 07:21:57 +01:00
Dietmar Maurer
9bc3ddb822
src/cli/command.rs: improve generate_usage_str
...
correctly format positional array arguments ({<name>}).
2019-02-26 12:40:51 +01:00
Dietmar Maurer
ae0be2dda9
src/bin/proxmox-backup-client.rs: allow to upload more that one directory
2019-02-26 12:27:28 +01:00
Dietmar Maurer
a477d6889a
src/client/http_client.rs: login once, store and reuse ticket/token
2019-02-26 11:59:10 +01:00
Dietmar Maurer
d02c2ed701
src/cli/getopts.rs: cleanup, avoid compiler warning
2019-02-26 11:55:06 +01:00
Dietmar Maurer
f8dfbb45c0
src/tools.rs: new helper required_array_param()
2019-02-26 08:48:17 +01:00
Dietmar Maurer
aaba6faace
src/cli/getopts.rs: support positional array arguments
2019-02-26 08:47:34 +01:00
Dietmar Maurer
f33fa27307
src/cli/command.rs: improve error message
2019-02-26 08:46:36 +01:00
Dietmar Maurer
92a173feb0
src/api_schema/schema.rs: improve ParameterError display function
2019-02-26 07:11:24 +01:00
Wolfgang Bumiller
f661f37427
backup/dynamic_index: split out add_chunk
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:50:45 +01:00
Wolfgang Bumiller
9ac6ec868a
backup/chunk_store: split insert_chunk
...
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
3a50ddd0c9
backup/chunk_store: mark ChunkIterator as fused
...
Ranges are FusedIterators and it can be useful to propagate
that property to users.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
eff25ecaf2
backup/chunk_store: make percentage output optional
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
d59397e60e
backup/datastore: expose chunk_store.get_chunk_iterator
...
will be used by the protocol handler to iterate over
existing chunks
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
Dietmar Maurer
8eb1203bda
src/backup/chunk_stat.rs: log write speed
2019-02-25 13:21:00 +01:00
Dietmar Maurer
7e3365554e
src/backup/chunk_stat.rs: new struct to track chunk statistics
2019-02-25 12:52:10 +01:00
Dietmar Maurer
798f7fa065
src/backup/chunk_store.rs: return info about compressed chunk size
...
So that we can generate better statistics ..
2019-02-25 11:36:05 +01:00
Dietmar Maurer
78216a5ab1
src/backup/chunk_store.rs: use lz4 for compression
...
And depend on lz4 lib now.
2019-02-25 10:00:28 +01:00
Dietmar Maurer
793b0f4d77
src/cli/command.rs: improve help command completion
2019-02-23 18:07:32 +01:00
Dietmar Maurer
6949d91564
src/cli/command.rs: cleanup, avoid array manipulation
2019-02-23 17:50:33 +01:00
Dietmar Maurer
9d78d57960
src/cli/command.rs: implement bash completion for help
2019-02-23 17:38:10 +01:00
Dietmar Maurer
698d9d4402
src/cli/command.rs: add help command
2019-02-23 15:10:48 +01:00
Dietmar Maurer
8b6dd2240d
src/cli/command.rs::allow to generate ReST docu
2019-02-23 11:29:18 +01:00
Dietmar Maurer
2f3f2bb77f
src/cli/command.rs: improve usage imformation
2019-02-22 17:40:37 +01:00
Dietmar Maurer
f252ca654c
src/backup/chunker.rs: simplify break test
...
To make it a little bit faster ...
2019-02-22 13:02:11 +01:00
Dietmar Maurer
6adb40eefc
src/cli/command.rs: start implementing generate_usage_str()
2019-02-22 12:02:59 +01:00
Dietmar Maurer
33256db6dd
src/cli/command.rs: simplify cli handler
2019-02-22 11:15:10 +01:00
Dietmar Maurer
61834c1cbf
src/backup/chunker.rs: another try to speedup the chunker
2019-02-22 11:15:10 +01:00
Dietmar Maurer
532a69eb04
src/cli/getopts.rs: improve docs
2019-02-21 09:36:52 +01:00
Dietmar Maurer
4de0e142a0
cleanup cli library structure
2019-02-21 09:08:03 +01:00
Wolfgang Bumiller
56458d9764
http_client: on a tty, read password if no env var is set
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-20 15:03:34 +01:00
Wolfgang Bumiller
c9b296f117
tools: add tty helper module
...
with read_password() and stdin_isatty() functions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-20 14:43:07 +01:00
Dietmar Maurer
8cc0d6aff3
src/bin/proxmox-backup-client.rs: implement garbage-collection sub-command
2019-02-20 14:10:45 +01:00
Dietmar Maurer
81da38c143
src/client/http_client.rs: implement post
...
without parameters for now...
2019-02-20 14:09:55 +01:00
Dietmar Maurer
8c75372b79
src/backup/datastore.rs: list_backups() - include list of index files
2019-02-20 13:37:44 +01:00
Dietmar Maurer
14691fc1f3
src/bin/test_chunk_size.rs: avoid compilöer warnings, exit after 1GB
2019-02-20 12:05:28 +01:00
Dietmar Maurer
1604ec2654
src/bin/test_chunk_size.rs: another chunker test - gather some stats
2019-02-20 11:50:10 +01:00
Dietmar Maurer
fa4ebf0fce
src/backup/chunker.rs: simplify code
2019-02-20 11:17:19 +01:00
Dietmar Maurer
1b9c959c32
src/backup/chunker.rs: change window size back to 48
...
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
2019-02-20 11:01:09 +01:00
Dietmar Maurer
247cdbce72
src/api2/admin/datastore/catar.rs: allow to configure chunk-size
2019-02-19 15:19:12 +01:00
Dietmar Maurer
c584aa21fd
src/backup/chunker.rs: start() - correctly store hash in self.h
2019-02-19 12:40:12 +01:00
Dietmar Maurer
cd1598302a
src/backup/chunker.rs: use 64 byte chunker window size
...
This results in faster code.
2019-02-19 12:27:37 +01:00
Dietmar Maurer
9a3e869cf5
src/backup/chunker.rs: simplify buzzhash start code
2019-02-19 12:17:17 +01:00
Dietmar Maurer
1c7a88aeff
UploadCaTar: currectly write all data using write_all()
2019-02-18 18:31:02 +01:00
Dietmar Maurer
4d2881dd4c
src/backup/chunker.rs: correctly compute window index 'idx'
...
And add a simple test.
2019-02-18 16:59:23 +01:00
Dietmar Maurer
a4a5c78c01
src/client/http_client.rs: send csrf token
2019-02-18 16:59:23 +01:00
Wolfgang Bumiller
fc7f03523c
cleanup
...
Error::from is already a function taking 1 parameter,
there's no need to wrap it with `|e| Error::from(e)`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-18 13:21:27 +01:00
Wolfgang Bumiller
8fdef1a81c
fix typo: confi{g}dir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-18 13:11:42 +01:00
Dietmar Maurer
1cb99c234e
src/server/rest.rs: correctly insert NoLogExtension()
2019-02-18 06:54:12 +01:00
Dietmar Maurer
591f570b22
src/client/http_client.rs: trim path
2019-02-18 06:24:28 +01:00
Dietmar Maurer
7f1685230a
src/server/rest.rs: get_index() include username and CSRF token
...
When we have an valid ticket. Also delay get_index() if called with
an invalid ticket.
2019-02-17 19:28:32 +01:00
Dietmar Maurer
141de8374a
src/server/rest.rs: factor our normalize_path()
2019-02-17 17:31:53 +01:00
Dietmar Maurer
8225aa2ff6
src/server/rest.rs: improve logs for unauthorized request
2019-02-17 17:18:44 +01:00
Dietmar Maurer
54d15ab5c5
src/api_schema/schema.rs: improve docu
2019-02-17 12:13:06 +01:00
Dietmar Maurer
51ebd07974
src/api_schema/router.rs: add docu for RpcEnvironmentType
2019-02-17 11:59:18 +01:00
Dietmar Maurer
d21c2710da
api_schema/router.rs: improve doc
2019-02-17 10:30:41 +01:00
Dietmar Maurer
ef2f2efbcc
improve api_schema module structure
2019-02-17 10:16:33 +01:00
Dietmar Maurer
dc9a007b11
rename src/api to src/api_schema
2019-02-17 09:59:20 +01:00
Dietmar Maurer
5ddf8cb10a
cleanup auth code, verify CSRF prevention token
2019-02-16 15:52:55 +01:00
Dietmar Maurer
95aee7e032
tools/common_regex.rs: add more docu
2019-02-16 12:36:01 +01:00
Dietmar Maurer
99be8844c7
buildcfg.rs: improve doc
2019-02-16 12:31:39 +01:00
Dietmar Maurer
a8f268afbb
fix configuration dir permission
...
Carefully set and check permissions ...
2019-02-16 12:19:13 +01:00
Dietmar Maurer
aada2a9719
config/datastore.rs: remove unused code
2019-02-16 10:10:45 +01:00
Dietmar Maurer
d11594db4c
config/datastore.rs: return empty config if file does not exist
2019-02-16 10:06:08 +01:00
Dietmar Maurer
12400210a2
tools::file_get_contents - improve error message
2019-02-16 09:42:00 +01:00
Dietmar Maurer
6235a41862
tools::file_read_firstline - improve error message
2019-02-16 09:36:29 +01:00
Dietmar Maurer
17ed456c2e
change proxy user from www-data to backup
2019-02-16 09:29:04 +01:00
Dietmar Maurer
5d14eb6a76
tools.rs: new helper to get uid/gid for the sepcified system user.
...
Or is there an easier way to get that info?
2019-02-15 15:33:12 +01:00
Dietmar Maurer
f0a6112451
backup/datastore.rs: improve error reporting
2019-02-15 14:33:03 +01:00
Dietmar Maurer
9b50c16103
section_config.rs: simplify parser by using new try_block macro
2019-02-15 12:13:15 +01:00
Dietmar Maurer
fe651dd6e5
tools.rs: fix doc tests
2019-02-15 11:33:51 +01:00
Dietmar Maurer
e80503d2a6
tools.rs: document try_block macro
2019-02-15 11:20:45 +01:00
Dietmar Maurer
f0dbba8cbe
tools.rs: new try_block macro
2019-02-15 10:59:12 +01:00
Wolfgang Bumiller
1d77b6cf6b
update to pam 0.7 (renamed from pam-auth)
...
It now supports custom conversation methods, so instead of
new() we ask for a default authenticator taking a password
via with_password(). Since the password is now handled by
the now separate conversation handler, `set_credentials()`
is now called on the handler we get via `.get_handler()`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-15 10:34:40 +01:00
Dietmar Maurer
d4736445fe
server/rest.rs: add method to log message
2019-02-15 10:16:12 +01:00
Dietmar Maurer
f02e6fc478
api2/admin/datastore/catar.rs: simplify/fix debug message
2019-02-15 10:04:37 +01:00
Dietmar Maurer
44c00c0dfd
server/rest.rs: log full error messages
2019-02-15 09:55:12 +01:00
Dietmar Maurer
7171b3e079
server/rest.rs: use a protocol extension to avoid double log
...
Instead of modifying the response header itself.
2019-02-14 16:04:24 +01:00
Dietmar Maurer
7e03988caf
avoid double logging of proxied requests
2019-02-14 13:28:41 +01:00
Dietmar Maurer
af6f80d3db
Revert "backup/datastore: fn open is a private implementation detail"
...
This reverts commit 3dacd9a660
.
I currently use this in some test ...
2019-02-14 13:13:49 +01:00
Dietmar Maurer
78a1fa6722
src/server/rest.rs: log failed requests
2019-02-14 13:07:34 +01:00
Wolfgang Bumiller
2cc90617dd
tools/fs: mark ReadDir as Send
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
62f2422f6a
backup/chunk_store: verify chunk file names
...
get_chunk_iterator() should skip over files which aren't an
actual chunk
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
3dacd9a660
backup/datastore: fn open is a private implementation detail
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f
backup/chunk_store: open base dir in get_chunk_iterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
Dietmar Maurer
db4868ef4f
src/client/backup_repo.rs - improve docu
2019-02-14 11:17:08 +01:00
Dietmar Maurer
151c6ce27d
src/client.rs: improve library structure and docu
2019-02-14 11:11:39 +01:00
Wolfgang Bumiller
dd1bfe6c12
tools/timer: add module documentation
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:13:42 +01:00
Wolfgang Bumiller
c7f481b642
backup: replace ChunkIterator
...
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
121f18efb7
tools: use scan_subdir iterator for scandir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
32286b039c
tools/fs: add file_name_utf8_unchecked shortcut
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
806d7a6a35
tools/fs: change scan_subdir to use filter_file_name_regex
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
53e1e7ca72
tools/fs: add filter_file_name_regex to FileIterOps
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
23fba9d777
tools/fs: add FileIterOps trait and FileTypeFilter
...
FileIterOps will provide operations on file entry iterators.
For now this provides a file type filter.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
1db416150d
tools/fs: let ReadDir iterator yield ReadDirEntry
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
b12505da69
tools/fs: add ReadDirEntry helper
...
This wraps nix::dir::Entry with the parent and additionally
exposes the parent directory file descriptor since we need
it to perform openat().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
a25f863afd
tools/fs: add scan_subdir helper
...
This filters the results of read_subdir with a regex.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
b4d5787de9
tools: start fs submodule, add read_subdir
...
This creates an iterator relative to a RawFd via
nix::dir::Dir over nix::dir::Entrys.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
Dietmar Maurer
0dffe3f99a
src/client/http_client.rs: try to login
...
use an environment var to store passphrase (PBS_PASSWORD)
2019-02-13 14:31:43 +01:00
Dietmar Maurer
33d64b81e0
src/bin/proxmox-backup-client.rs: define and use BackupRepository
2019-02-13 12:30:52 +01:00
Wolfgang Bumiller
4a3f65172c
client: use hyper-tls for now
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-13 11:03:09 +01:00
Wolfgang Bumiller
42d6e4fb05
tools: borrow: implement Deref for Tied
...
Eg. if you have an x: Tied<Dir, Iterator> now you can simply
call x.next() as it'll dereference to the Iterator.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-13 11:03:09 +01:00
Dietmar Maurer
541a302224
api2/admin/datastore/catar.rs: use nested file extensions
...
i.e. archive.catar.didx
2019-02-13 10:37:00 +01:00
Dietmar Maurer
e5064ba607
simplify backup lib structure (pub use xxx:*), improve doc
2019-02-12 14:13:31 +01:00
Dietmar Maurer
d78345bcfc
src/backup.rs - improve doc
2019-02-12 13:27:11 +01:00
Dietmar Maurer
93d5d77952
rename ArchiveIndex to DynamicIndex
...
also changed the file extension from .aidx to .didx
2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd
rename ImageIndex to FixedIndex
...
also changed the file extension fron .iidx to .fidx
2019-02-12 11:50:45 +01:00
Wolfgang Bumiller
82bc0ad40c
backup: fixup warnings and readd lost file type check
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:43:43 +01:00
Wolfgang Bumiller
9349d2a1d2
backup: cleanup long lines
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
fdd71f5286
backup: use ChunkIterator for sweep_unused_chunks
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
c9e7f4dc8b
backup: add optional progress callback to ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
4070096c6a
backup: add ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
6ed25cbe3a
tools: add Tied value
...
Tie two dependent values together, such as a nix::Dir with
its nix::dir::Iter<'a> where 'a is tied to the Dir's
lifetime, making it otherwise impossible to return them or
store them in a struct.
Alternatively we could try the 'rental' crate.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
9ac1045c7b
bin: manager: fixup datastore parameter name
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
Dietmar Maurer
8f8d5a420c
avoid compiler warning
2019-02-11 15:12:01 +01:00
Dietmar Maurer
aa5a4060ab
bin/proxmox-backup-api.rs: improve error handling
2019-02-11 15:11:42 +01:00
Dietmar Maurer
4223d9f800
bin/proxmox-backup-proxy.rs: improve error handling
2019-02-11 14:43:26 +01:00
Wolfgang Bumiller
ace9e3531a
access: use proxmox-backup-auth for pam
...
allows customization via /etc/pam.d/proxmox-backup-auth
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-07 13:40:22 +01:00
Wolfgang Bumiller
13f8310cac
api2: time: use /etc/localtime as fallback
...
in case there is no /etc/timezone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-05 14:14:56 +01:00
Wolfgang Bumiller
a7cac14ed9
tools: use std::fs::read for file_get_contents
...
Until we add the extra options like we have in pve, just use
the already existing (since 1.26) shortcut.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:35:32 +01:00
Wolfgang Bumiller
e937034c86
remove some rather inconvenient debug output
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:34:38 +01:00
Wolfgang Bumiller
9d02e5c38a
proxy: use <configdir>/server.pfx for tls
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:19:32 +01:00
Wolfgang Bumiller
9f4962d396
introduce buildcfg module and PROXMOX_CONFIGDIR
...
buildcfg.rs should contain convenience variables or macros
for using build-time configured variables
For now we replace hardcoded "/etc/proxmox-backup/<foo>"
with configdir!("<foo>").
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:13:08 +01:00
Wolfgang Bumiller
0d176f3681
proxy: use TLS via tokio-tls
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 14:56:07 +01:00
Wolfgang Bumiller
e0a5d1ca2f
tests: chunk_store: cleanup .testdir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
a4f9188300
tests: import lazy_static as needed
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
6285b251e7
don't hardcode /usr/share paths
...
Add a defines.mk file defining the standard default layout
for installation (PREFIX, DATAROOTDIR, LIBDIR etc.).
Add a $(JSDIR) variable and use env!("JSDIR") in place of
the hardcoded /usr/share/javascript/proxmox-backup path.
Have defines.mk include an *optional* local.mak to override
JSDIR to www/ during local development.
Add `make install` targets and remove debian/install.
Note that service files are now in $libdir/proxmox-backup/
as there's usually no point in starting them from the
command line, so /usr/sbin makes no sense.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
8e4df103dd
remove useless command option from daemons
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
Dietmar Maurer
6278365fd9
improve ApiMethod docu
2019-02-01 10:38:07 +01:00
Dietmar Maurer
4b2cdeb9a6
implement relead_timezone flag
2019-02-01 10:04:46 +01:00
Dietmar Maurer
c82bc1a1f9
depend on pam-auth, use pam for root login
...
Please use username "root@pam" for login via GUI.
2019-02-01 09:30:50 +01:00
Dietmar Maurer
a154a8e8a4
delay unauthorized request (rate limit)
2019-01-31 14:34:21 +01:00
Dietmar Maurer
5996577ab6
move http error class to router.rs
2019-01-31 13:22:30 +01:00
Dietmar Maurer
b9903d6331
server/rest.rs: verify auth cookie
2019-01-31 12:22:00 +01:00
Dietmar Maurer
f484eed3c2
Utils.js: fix cookie handling
...
Use unsecure cookie foör testing.
2019-01-31 10:08:08 +01:00
Dietmar Maurer
9f49fe1d5d
avoid compiler warnings
2019-01-30 18:25:37 +01:00
Dietmar Maurer
34f956bc25
api2/access.rs: add ticket api
2019-01-30 15:16:10 +01:00
Wolfgang Bumiller
1bf446a33e
fix timer_delete call
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-30 12:04:20 +01:00
Dietmar Maurer
cf671670a0
auth_helpers.rs: add timestamp to csrf token
2019-01-29 17:50:03 +01:00
Dietmar Maurer
1e76cbc6a2
auth_helpers.rs: implement assemble_csrf_prevention_token
2019-01-29 17:41:45 +01:00
Dietmar Maurer
d01e2420f7
load auth keys on startup
2019-01-29 17:21:58 +01:00
Dietmar Maurer
6c30068ebf
auth_helpers.rs: split code into separate file
2019-01-29 16:55:49 +01:00
Dietmar Maurer
39a90ca6c5
bin/proxmox-backup-api.rs: gererate secret for csrf token
2019-01-29 13:15:33 +01:00
Dietmar Maurer
26027a151e
tools/ticket.rs: remove debug statements
2019-01-29 13:04:01 +01:00
Dietmar Maurer
8d04280b29
tools/ticket.rs: add basic ticket support
2019-01-29 12:59:07 +01:00
Dietmar Maurer
c8f3f9b1af
server/rest.rs: simplify proxy code
...
Only pass neccessary parameters.
2019-01-28 18:22:16 +01:00
Dietmar Maurer
a859fa95a9
add protected flag for some api methods
2019-01-28 18:16:47 +01:00
Dietmar Maurer
a3da38ddfd
server/rest.rs: implement proxy_sync_api_request
2019-01-28 18:06:42 +01:00
Dietmar Maurer
f120483353
server/rest.rs: add proxy_sync_api_request() dummy
2019-01-28 17:30:39 +01:00
Dietmar Maurer
4e4df8e2f0
api/router.rs: add new 'protected' flag to ApiMethod
2019-01-28 17:18:42 +01:00
Dietmar Maurer
5bb3398171
server/formatter.rs: fix extjs error format
2019-01-28 13:44:48 +01:00
Dietmar Maurer
02c7a75520
src/bin/proxmox-backup-proxy.rs: implement unpriviledged server
...
We want to run the public server as user www-data. Requests needing
root priviledges needs to be proxied to the proxmox-backup.service, which
now listens to 127.0.0.1:82.
2019-01-28 13:29:58 +01:00
Dietmar Maurer
48849593ee
api2/node/services.rs: implement service commands
2019-01-27 12:40:31 +01:00
Dietmar Maurer
d7d23785f0
RpcEnvironment: implement set_user() and get_user()
2019-01-27 10:52:26 +01:00
Dietmar Maurer
162b979394
RpcEnvironment: add environment type enum RpcEnvironmentType
2019-01-27 10:33:42 +01:00
Dietmar Maurer
e82dad9700
also pass rpcenv to async handlers
2019-01-27 10:18:52 +01:00
Dietmar Maurer
d2ab5f19e2
api2/node/services.rs: start services api
2019-01-26 17:54:18 +01:00
Dietmar Maurer
0f253593c6
move rpc environment implementation to separate files
2019-01-26 15:08:02 +01:00
Dietmar Maurer
6049b71f41
api: pass RpcEnvirnment to api handlers
2019-01-26 14:50:37 +01:00
Dietmar Maurer
d96d82736d
use crate log and syslog
2019-01-26 10:56:11 +01:00
Dietmar Maurer
e182ab4a4a
api2/node/syslog.rs: improve error handling
2019-01-26 10:23:52 +01:00
Dietmar Maurer
0441886842
api2/node/syslog.rs: implement dump journal
2019-01-25 18:20:51 +01:00
Dietmar Maurer
4f9a726833
api2/node/syslog.rs: add syslog api schema
2019-01-25 17:17:30 +01:00
Dietmar Maurer
1631c54fca
tools::get_hardware_address - fix FILENAME, use file_get_contents()
2019-01-25 12:23:47 +01:00
Dietmar Maurer
af2fddea4f
api2/node/dns.rs: implement concurrent update protection
2019-01-25 11:38:59 +01:00
Dietmar Maurer
de6b0721fa
pi2/node/dns.rs: return file digest
2019-01-25 11:03:41 +01:00
Dietmar Maurer
2296860027
tools.rs: move digest_to_hex() into tools
2019-01-25 10:58:28 +01:00
Dietmar Maurer
53157ca6cf
tools.rs: implement file_get_contents()
2019-01-25 10:44:40 +01:00
Dietmar Maurer
46b79b9ee5
api2/node/dns.rs: improve dns api
2019-01-25 10:15:32 +01:00
Dietmar Maurer
332dcc2277
backup/chunk_store.rs: fix test case - use absolute path
2019-01-25 10:14:25 +01:00
Dietmar Maurer
8f973f8181
api2/node/dns.rs: implement resolv.conf parser
2019-01-24 14:59:40 +01:00
Dietmar Maurer
784252dbc5
backup/datastore.rs: use single lazy_static block
2019-01-24 14:58:47 +01:00
Dietmar Maurer
e6ffeb918c
api2/node/time.rs: implement set timezone
2019-01-24 12:05:06 +01:00
Dietmar Maurer
cd8b91f78c
api2/node/time.rs: add return schema
2019-01-24 11:07:11 +01:00
Dietmar Maurer
447787ab7c
tools.rs: implement file_read_firstline
2019-01-24 10:43:30 +01:00
Dietmar Maurer
0463602a79
api2/node/time.rs: implement read api
2019-01-23 15:14:14 +01:00
Dietmar Maurer
b2b3485d5f
src/api2/node: start node configuration api
2019-01-23 13:09:17 +01:00
Dietmar Maurer
d15009c0ce
server/rest.rs: fake login cookie
2019-01-23 12:49:10 +01:00
Dietmar Maurer
248cb51862
add missing api2.rs
2019-01-22 13:56:27 +01:00
Dietmar Maurer
7e13b2d67f
api2/subscription.rs: add subscription api class
2019-01-22 12:50:19 +01:00
Dietmar Maurer
576e3bf252
rename api3 back to api2
...
There is no real need to change the path, so using api2 we can reuse
all helpers (like tools from proxmox widget toolkit).
2019-01-22 12:10:38 +01:00
Dietmar Maurer
1adb353d6f
lient/http_client.rs: simplify code
2019-01-22 10:36:04 +01:00
Dietmar Maurer
41c039e143
bin/proxmox-backup-client.rs: implement list command
2019-01-21 18:58:14 +01:00
Dietmar Maurer
1fdb4c6ffa
client/http_client.rs: implement get, share common code
2019-01-21 18:56:48 +01:00
Dietmar Maurer
83dbd80bd8
api3/admin/datastore.rs: implement list backups
2019-01-21 11:22:55 +01:00
Dietmar Maurer
7ca80246ea
backup/datastore.rs: use DateTime<Utc> to store backup time
2019-01-21 10:51:52 +01:00
Dietmar Maurer
be0084b001
white space fix
2019-01-21 10:32:07 +01:00
Dietmar Maurer
e25736b4c3
backup/datastore.rs: implement list_backups
2019-01-20 18:02:27 +01:00
Dietmar Maurer
cce1676a9e
tools.rs: use FnMut for scandir callback
2019-01-20 17:49:11 +01:00
Dietmar Maurer
443f3743bb
tools.rs: new scandir() helper
2019-01-20 17:31:43 +01:00
Dietmar Maurer
68469eebb4
backup/chunk_store.rs: verify base path is absolute
2019-01-20 16:49:22 +01:00
Dietmar Maurer
059ca7c3a1
remove crate tokio-codec (seems to be part of tokio now)
2019-01-20 14:28:06 +01:00
Dietmar Maurer
8eeef8492a
tools/wrapped_reader_stream.rs: allocate Vec<u8> for buffer
2019-01-20 11:01:18 +01:00
Dietmar Maurer
ff8bdf3b64
tools/wrapped_reader_stream.rs: only implement Stream (without AsyncRead)
...
Looks more efficent ...
2019-01-20 10:41:21 +01:00
Dietmar Maurer
8e89d9cafe
tools/wrapped_reader_stream.rs: use tokio_threadpool::blocking
2019-01-20 10:28:51 +01:00
Dietmar Maurer
ce7ba139f1
catar: improve bash completion
2019-01-20 09:55:20 +01:00
Dietmar Maurer
29f34b8e1a
/api3/admin/datastore.rs: add missing subdir to directory index
2019-01-20 09:47:32 +01:00
Dietmar Maurer
0b05fd5830
api3/admin/datastore/catar.rs: implement download
2019-01-20 09:39:32 +01:00
Dietmar Maurer
7f0d67cf79
tools/wrapped_reader_stream.rs: new helper class
...
Will use that to download catar files.
2019-01-20 09:38:28 +01:00
Dietmar Maurer
6a4c091616
api3/admin/datastore/catar.rs: start impl. download
2019-01-19 17:27:02 +01:00
Dietmar Maurer
50cfb695ae
api/router.rs: rename ApiUploadMethod to ApiAsyncMethod
...
We can use this for uploads and downloads ...
2019-01-19 16:42:43 +01:00
Dietmar Maurer
379ea0edb6
Revert "/api/schema.rs: implement Schema::Option"
...
This reverts commit 0a35462c1e
.
I am not sure this add much value, and the old approach needs
less memory. If we really need single optional values, we can still
implement such Option while keeping the hash based approach...
2019-01-19 12:53:32 +01:00
Dietmar Maurer
80f069656d
Revert "api/schema.rs: avoid Option(Option( nesting"
...
This reverts commit ffdac1af01
.
2019-01-19 12:52:51 +01:00
Dietmar Maurer
ca3d861513
remove unused type PropertyMap
2019-01-18 17:54:06 +01:00
Dietmar Maurer
ffdac1af01
api/schema.rs: avoid Option(Option( nesting
2019-01-18 17:49:19 +01:00
Dietmar Maurer
0a35462c1e
/api/schema.rs: implement Schema::Option
2019-01-18 17:40:37 +01:00
Dietmar Maurer
728797d0c1
reduce compiler warnings
2019-01-18 16:50:15 +01:00
Dietmar Maurer
383e857749
move complete_file_name() helper into tools.rs
2019-01-18 13:42:52 +01:00
Dietmar Maurer
73e844dbfe
bin/proxmox-backup-client.rs: improve file name completion
2019-01-18 13:40:05 +01:00
Dietmar Maurer
95cea65b04
backup/datastore.rs: list all index files using walkdir crate
2019-01-18 12:24:58 +01:00
Dietmar Maurer
ff3d3100d4
backup/datastore.rs: try to create useful directory layout
...
store/type/ID/TIME/xxx.yyy
2019-01-18 12:01:37 +01:00
Dietmar Maurer
0d38dcb43c
cleanup nodename()
2019-01-18 10:13:45 +01:00
Dietmar Maurer
74a69302a7
tools.rs: new helper nodename()
2019-01-18 09:58:15 +01:00
Dietmar Maurer
5a778d92b3
api3/admin/datastore.rs: rename parameter "name" to "store"
2019-01-18 08:33:11 +01:00
Dietmar Maurer
244d9b17a8
bin/proxmox-backup-client.rs: implement file name completions - first try ...
2019-01-17 14:24:20 +01:00
Dietmar Maurer
83bdac1e3b
api3/admin/datastore/upload_catar.rs: verify content type ("application/x-proxmox-backup-catar")
2019-01-17 12:43:29 +01:00
Dietmar Maurer
06aeb76a5c
client/catar_backup_stream.rs: remove debug code
2019-01-17 12:16:36 +01:00
Dietmar Maurer
0fe5d605e7
tools.rs: new helper required_string_param()
...
To extract parameters from Value.
2019-01-17 12:14:02 +01:00
Dietmar Maurer
0ee0ad5bf3
api/router.rs: return Result in upload handler
2019-01-17 12:03:38 +01:00
Dietmar Maurer
23bb8780d4
proxmox-backup-client: implement upload to server
2019-01-17 11:38:22 +01:00
Dietmar Maurer
597641fdd5
client/http_client.rs: new helper class
2019-01-17 11:38:03 +01:00
Dietmar Maurer
264f52cf70
api3/admin/datastore.rs: change api patch for upload_catar
2019-01-17 09:16:50 +01:00
Wolfgang Bumiller
37b98249f4
upload: don't return NotReady
...
We need to kick (poll) the stream's next element, so simply
reiterate, the try_read!() will then return NotReady from
the next stream item for us.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-16 15:17:26 +01:00
Wolfgang Bumiller
1dcd52e5df
cleanup previous commit...
...
This kind of notation is weird...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-16 14:19:13 +01:00
Wolfgang Bumiller
6a50b77e72
listen on :: instead of 0.0.0.0
...
Since this supports both ipv6 & ipv4 unless v6only is
explicitly enabled.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-16 14:18:16 +01:00
Dietmar Maurer
cf16af2ab3
server/rest.rs: correctly pass query/url parameters
2019-01-16 13:58:36 +01:00
Dietmar Maurer
2085142ed4
reduce compiler warnings
2019-01-16 10:19:49 +01:00
Dietmar Maurer
e8edbbd49c
client/catar_backup_stream.rs: new helper for catar uploads to server
2019-01-16 10:15:39 +01:00
Dietmar Maurer
150f1bd8f6
backup/datastore.rs: use Arc for chunk_store references
2019-01-15 12:36:16 +01:00
Dietmar Maurer
1629d2ad7b
api3/admin/datastore/upload_catar.rs: implement upload future
2019-01-15 11:38:26 +01:00
Dietmar Maurer
7e21da6e23
api/router.rs: allow different types of api methods
...
Added a prototype for file/backup uploads.
2019-01-14 12:26:04 +01:00
Dietmar Maurer
b41d1aab8c
catar: define all feature flags
2019-01-12 16:49:02 +01:00
Dietmar Maurer
4c0fd4871e
catar/encoder.rs: allow to pass list of devices
...
We can use this to implement the --one-file-system like options. This
is a bit more convenient, because this way we can select to include
multiple devices (which is required by container backups).
2019-01-12 15:43:20 +01:00
Dietmar Maurer
a8a0495612
catar/encoder.rs: disable backup of virtual file systems
2019-01-12 11:56:53 +01:00
Dietmar Maurer
d7bfac8605
catar/encoder.rs: only test fat attribute on MSDOS and FUSE file systems
2019-01-12 10:28:26 +01:00
Dietmar Maurer
13d98013b2
/catar/encoder.rs: do not store data from virtual file systems
2019-01-12 10:20:08 +01:00
Dietmar Maurer
80881f60ef
catar/encoder.rs: add helpers to test file system type
2019-01-12 09:18:21 +01:00
Dietmar Maurer
1c4804cf67
catar/encoder.rs: pass magic fs number to encode_dir
2019-01-12 08:51:44 +01:00
Dietmar Maurer
dce0ebd505
catar/encoder.rs: use O_DIRECTORY to open dirs
2019-01-11 16:53:17 +01:00
Dietmar Maurer
490683ec93
catar: encode/decode sockets and fifos
2019-01-11 13:26:05 +01:00
Dietmar Maurer
20e2043a08
catar/decoder.rs: corrently pass mode to mknodat
2019-01-11 13:12:55 +01:00
Dietmar Maurer
a7e3713122
catar: encode/decode devices
2019-01-11 12:22:00 +01:00
Dietmar Maurer
1bdb3130f7
catar/format_definition.rs: add more feature flags
2019-01-11 10:44:57 +01:00
Dietmar Maurer
5c76c2f331
catar/encoder.rs: store fat file system attributes
2019-01-11 10:18:22 +01:00
Dietmar Maurer
8c1dfa6c72
catar/encoder.rs: cleanups
2019-01-11 10:01:51 +01:00
Dietmar Maurer
4f6892eb74
catar/encoder.rs: encode linux fs attributes, cleanup encoder
...
Try to avoid duplicate stat calls (assume file tree is stable during backup).
2019-01-11 09:20:10 +01:00
Dietmar Maurer
ddbdf80d51
avoid compiler warnings
2019-01-11 08:41:33 +01:00
Dietmar Maurer
373ef4a504
backup/archive_index.rs: correctly check for errors
2019-01-10 11:19:54 +01:00
Dietmar Maurer
23f68e53b4
catar/decoder.rs: restore mode, uid, gid and mtime
2019-01-10 09:02:50 +01:00
Dietmar Maurer
25f60394d2
catar/decoder.rs: avoid slashes in file names
2019-01-10 09:02:50 +01:00
Dietmar Maurer
9b1bb5a277
catar/decoder.rs: impl restore dirs, files, and symlinks
2019-01-10 09:01:36 +01:00
Wolfgang Bumiller
1ed86a0bae
rest: rename utf-8-checked 'bytes' to 'utf8'
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-08 14:22:43 +01:00
Wolfgang Bumiller
9fec7ae593
rest: don't copy the body
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-08 14:21:54 +01:00
Wolfgang Bumiller
d9fbcc4631
tools/timer: fixup doc formatting
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-08 14:11:10 +01:00
Dietmar Maurer
676bf71ad4
catar/decoder.rs: restore_sequential - read payload
2019-01-08 09:26:07 +01:00
Dietmar Maurer
c6a0d3fc6b
catar/decoder.rs: improve restore_sequential
2019-01-07 19:18:59 +01:00
Dietmar Maurer
b6ebfb8dab
catar/decoder.rs: start implementing sequential decoder, cleanups
...
And use [derive(Endian)] for endian conversion.
2019-01-07 19:07:03 +01:00
Dietmar Maurer
02c7d8e522
catar: implement create
2019-01-07 13:25:41 +01:00
Dietmar Maurer
e86c49247e
bin/catar.rs: add list command
2019-01-06 17:42:23 +01:00
Dietmar Maurer
3d8c24ecf8
catar/decoder.rs: start implementation
2019-01-06 17:27:57 +01:00
Dietmar Maurer
4b864ad436
format_definition.rs: remove wrong docs
2019-01-06 17:27:22 +01:00
Dietmar Maurer
4624fe29cc
impl Read for BufferedArchiveReader
2019-01-06 10:04:45 +01:00
Dietmar Maurer
343370504b
impl Seek for BufferedArchiveReader
2019-01-06 09:35:39 +01:00
Dietmar Maurer
318564ac03
buffered_read: return empty buffer on eof
2019-01-06 09:17:28 +01:00
Dietmar Maurer
0a72e26704
define and use BufferedReader trait
2019-01-05 17:28:20 +01:00
Dietmar Maurer
1006acd626
remove dead code
2019-01-05 16:55:21 +01:00
Dietmar Maurer
51b499db74
tools.rs: improve docs
2019-01-05 16:53:28 +01:00
Dietmar Maurer
39c6bd86cc
backup/archive_index.rs: implement BufferedArchiveReader
...
Implement relativly fast random read using binary search.
2019-01-05 14:47:56 +01:00
Dietmar Maurer
060c4811a0
backup/chunk_store.rs: resize buffer by powers of two
2019-01-05 11:34:49 +01:00
Dietmar Maurer
9409255aae
catar/encoder.rs: code cleanup
...
Note: Rust Vec resize uses len(), not capacity()!
2019-01-04 17:23:01 +01:00
Dietmar Maurer
df9973e8f5
backup/chunk_store.rs: avoid allocation in read_chunk
2019-01-04 17:16:56 +01:00
Dietmar Maurer
7394ca3e95
white space cleanups
2019-01-04 12:51:43 +01:00
Dietmar Maurer
96df2fb44b
backup/chunk_store.rs: implement read_chunk()
2019-01-04 12:50:54 +01:00
Dietmar Maurer
691c89a0fb
proxmox-backup-manager: add garbage collection cli
2019-01-04 11:34:32 +01:00
Dietmar Maurer
07b4694a33
cleanup crate names
2019-01-04 10:49:52 +01:00
Dietmar Maurer
4bdba5f4f1
rename backup-client to proxmox-backup-client
2019-01-04 10:41:37 +01:00
Dietmar Maurer
a689f6df0e
rename pbs to proxmox-backup-manager
2019-01-04 10:39:32 +01:00
Dietmar Maurer
bc616633d5
backup/chunk_store.rs: only use one directory level
2019-01-04 10:35:22 +01:00
Dietmar Maurer
030cc2a903
move admin API to /admin/ subfolder
2019-01-04 10:32:27 +01:00
Dietmar Maurer
b663789bdb
backup/archive_index.rs: pass correct size to unmap
2019-01-04 09:28:41 +01:00
Dietmar Maurer
48d0d35688
backup/image_index.rs: use little endian
2019-01-04 08:45:45 +01:00
Wolfgang Bumiller
4968bc3ac0
cli::command: wrap usage errors in a UsageError
...
So we can distinguish them and show usage output
conditionally.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-03 16:08:51 +01:00
Dietmar Maurer
c7fa487250
catar/encoder.rs: make sure we write all data with write_all()
2019-01-03 15:47:32 +01:00
Dietmar Maurer
050f8a6355
chunker.rs: make sure chunk_size_avg is a power of two
2019-01-03 15:10:54 +01:00
Dietmar Maurer
ea4ea34baf
improve chunker speed (avoid modulo)
2019-01-03 14:35:19 +01:00
Dietmar Maurer
3329ae8c2e
add benchmark for chunker
...
We get about 300MB/s - so this is still not fast enough
Note: This is quite slow with debug target, so please compile with --release
2019-01-02 19:18:53 +01:00
Dietmar Maurer
0b8e75ed16
image_index.rs: verify file size
2019-01-02 18:14:02 +01:00
Dietmar Maurer
77703d95aa
implement garbage collection for .aidx files
2019-01-02 14:27:04 +01:00
Dietmar Maurer
a360f6fa2d
image_index.rs: verify header magic/version
2019-01-02 13:13:13 +01:00
Dietmar Maurer
44b3f62b42
fix typo
2019-01-02 12:56:04 +01:00
Dietmar Maurer
594fa52016
disable debug code
2019-01-02 12:55:18 +01:00
Dietmar Maurer
5032b57b44
archive_index.rs: correctly write .aidx file
2019-01-02 12:54:40 +01:00
Dietmar Maurer
580dc84bac
image_index.rs: print stats
2019-01-02 12:53:49 +01:00
Dietmar Maurer
5e7a09be0d
src/backup/archive_index.rs: use close() instead of flush()
...
Also pass a reference to the encoder.
2019-01-02 11:02:56 +01:00
Dietmar Maurer
94a882e900
src/backup/archive_index.rs: implement flush()
2018-12-31 18:01:07 +01:00
Dietmar Maurer
0433db1949
src/backup/archive_index.rs: first try
2018-12-31 17:30:08 +01:00
Dietmar Maurer
cbdd8c54ae
create backup mod in backup.rs, improve docu
2018-12-31 16:08:04 +01:00
Dietmar Maurer
cb4412b18e
move chunker.rs to backup/
2018-12-31 15:46:16 +01:00
Dietmar Maurer
1c287cb19a
rc/catar/chunker.rs: first try
2018-12-31 13:01:06 +01:00
Dietmar Maurer
bf205f94c5
avoid compiler warnings
2018-12-31 10:54:25 +01:00
Dietmar Maurer
1b0dc9f680
src/catar/encoder.rs: correctly sort goodbye items by hash key
2018-12-31 10:11:28 +01:00
Dietmar Maurer
c60d34bdbf
new catar binary
...
currently used for debugging
2018-12-30 18:02:46 +01:00
Dietmar Maurer
e75eac73ca
fix doc syntax
2018-12-30 17:43:53 +01:00
Dietmar Maurer
0866748de6
catar cleanups ...
2018-12-30 17:32:52 +01:00
Dietmar Maurer
6cd28d200e
binary_search_tree.rs: fix docs
2018-12-30 15:34:43 +01:00
Dietmar Maurer
389e562524
src/catar/encoder.rs: auto-resize file_copy_buffer, limit number of dirs
2018-12-30 14:09:59 +01:00
Dietmar Maurer
4fa71e0573
improve catar docs
2018-12-30 13:47:27 +01:00
Dietmar Maurer
48147efd12
add a comment about posible improvements
2018-12-29 19:43:25 +01:00
Dietmar Maurer
0b78833d8e
src/catar/binary_search_tree.rs: add regression tests and fix one bug
2018-12-29 18:32:03 +01:00
Dietmar Maurer
b17d7149d2
src/catar/binary_search_tree.rs: improve docu
2018-12-29 17:38:50 +01:00
Dietmar Maurer
985567fb45
src/catar/encoder.rs: cleanup, factor out write_goodbye_table
2018-12-29 17:26:32 +01:00
Dietmar Maurer
95bd5dfec7
src/catar/encoder.rs: correctly sort goodby items
2018-12-29 17:00:48 +01:00
Dietmar Maurer
46b6fbd6ae
use external crate siphasher
...
Because std:#️⃣ :SipHasher is deprecated.
2018-12-28 19:50:07 +01:00
Dietmar Maurer
f0f3029e26
src/catar/encoder.rs: fix bug in file copy
2018-12-28 15:02:42 +01:00
Dietmar Maurer
d05f93215e
src/catar/encoder.rs: fix symlinks
2018-12-28 15:02:42 +01:00
Wolfgang Bumiller
7ebb173352
getopt: cleanup: add trailing commas
...
It is customary in rust to always use trailing commas. (also
suggested by rustfmt)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
2767c5d39b
getopt: cleanup: don't condense 'if' statements too much
...
In a language which enforces curly braces this looks weird
and rustfmt doesn't like it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
7d95c10da0
getopt: whitespace cleanup
...
to make rustfmt happy
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:27:38 +01:00
Dietmar Maurer
a0cc09b5f0
src/catar/encoder.rs: fix binary format, write goodby table
...
We still ned to sort the table (BST) ...
2018-12-28 14:27:00 +01:00
Wolfgang Bumiller
03fb895197
getopt: condense nested match to reduce indentation
...
The `match value` statement is the only thing covering the
entire RawArgument::Option case. `rustfmt` suggests this
more condensed way of writing this case.
See the `git diff -w` of this patch:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 9755af2..4db4579 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,8 +72,7 @@ pub fn parse_arguments<T: AsRef<str>>(
| RawArgument::Separator => {
| break;
| }
|- RawArgument::Option { name, value } => {
|- match value {
|+ RawArgument::Option { name, value } => match value {
| None => {
| let mut want_bool = false;
| let mut can_default = false;
|@@ -125,7 +124,6 @@ pub fn parse_arguments<T: AsRef<str>>(
| data.push((name, v));
| }
| }
|- }
| RawArgument::Argument { value } => {
| rest.push(value);
| }
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:16:58 +01:00
Wolfgang Bumiller
4d0ea9978b
getopt: let parse_arguments() take a slice of AsRef<str>
...
We don't need the content to be owned strings, and we don't
need the list to be a Vec, since we only care about being
able to iterate through and copy out portions of the strings
we need, so take an &[T] where T: AsRef<str>.
This avoids .iter().map(to_string).collect() before calling
parse_arguments().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:05:05 +01:00
Wolfgang Bumiller
2482c095b1
getopt: remove skip logic
...
The 'skip' variable was set to indicate that the "rest of
the args" is to be copied into the 'rest' vec. We can do
this directly and avoid the 'if' case in the loop
altogether.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 13:59:44 +01:00
Wolfgang Bumiller
99da3a073d
getopt: indentation reduction
...
See the `git diff -w` output:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 6548a01..517cc37 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,7 +72,10 @@ pub fn parse_arguments(
| while pos < args.len() {
| if skip {
| rest.push(args[pos].clone());
|- } else {
|+ pos += 1;
|+ continue;
|+ }
|+
| match parse_argument(&args[pos]) {
| RawArgument::Separator => {
| skip = true;
|@@ -135,7 +138,6 @@ pub fn parse_arguments(
| rest.push(value);
| }
| }
|- }
|
| pos += 1;
| }
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 13:54:20 +01:00
Wolfgang Bumiller
ad8a98f7e4
parse_arguments: work with utf8 bytes and reduce indentation
...
We are only caring about '=' and '-' which are single-byte
codepoints, so there's no reason not to work on a byte
slice.
Also, some indentation reduction:
Transform
if (a) {
...
return A;
}
return B;
into
if (!a)
return B;
return A;
and
if (a)
if (b)
foo()
into
if (a && b)
return;
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 13:47:29 +01:00
Dietmar Maurer
248c17af39
src/catar/encoder.rs: write filenames
2018-12-28 11:48:47 +01:00
Dietmar Maurer
2e4ae0e239
src/catar/encoder.rs: write file data
2018-12-28 10:44:12 +01:00
Dietmar Maurer
3192ae968c
src/catar/encoder.rs: write entry and symlink data
2018-12-28 09:55:26 +01:00
Dietmar Maurer
50ea43962e
make map_struct functions public
2018-12-28 08:04:46 +01:00
Dietmar Maurer
d2b03f2397
catar/encoder.rs: improve error handling
2018-12-28 07:45:15 +01:00
Dietmar Maurer
0ff559990c
cleanups, avoid compiler warnings
2018-12-28 07:14:12 +01:00
Dietmar Maurer
45281d4927
catar/encoder: detect symlink and regular files
2018-12-27 14:24:31 +01:00
Dietmar Maurer
fb8365b79f
catar/encoder.rs: first try
2018-12-27 13:15:47 +01:00
Wolfgang Bumiller
fba3437f1e
timer: setup_timeout_handler cannot fail
...
and if it does it panics anyway
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-27 10:36:28 +01:00
Dietmar Maurer
bcd879cfb3
backup-client: allow to specify directories (prepare for catar backup)
2018-12-27 10:11:11 +01:00
Dietmar Maurer
b62b6cadf4
start implementing catar
2018-12-27 09:22:23 +01:00
Dietmar Maurer
dc3de618ed
new helper map_struct and map_struct_mut
2018-12-27 09:20:17 +01:00
Dietmar Maurer
a198d74fc0
cleanup: reduce compiler warnings
2018-12-25 13:29:27 +01:00
Dietmar Maurer
991de6007b
remove outdated comment
2018-12-25 13:24:50 +01:00
Dietmar Maurer
7ee2aa1b94
touch_chunk: use libc::lutimensat
2018-12-25 12:27:25 +01:00
Dietmar Maurer
2c32fdde86
move lookup_datastore() to backup/datastore.rs
2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835
gather usage statistics during garbage collection
2018-12-22 16:58:16 +01:00
Dietmar Maurer
28b96b56e1
open_file_locked: improve error message
2018-12-22 15:59:55 +01:00
Dietmar Maurer
176e4af964
sweep_used_chunks: print percentage
2018-12-22 15:39:05 +01:00
Dietmar Maurer
7b2b40a893
use openssl for faster hashing
2018-12-22 14:31:59 +01:00
Dietmar Maurer
e95950e40a
chunk_store: reduce number of directories
...
Else, scans everything takes too long ...
2018-12-22 14:04:05 +01:00
Dietmar Maurer
15e9b4ed60
src/api3/datastore.rs: new file
2018-12-21 13:38:41 +01:00
Dietmar Maurer
3c140f60cd
use return value (avoid compiler warning)
2018-12-21 12:44:20 +01:00
Dietmar Maurer
03e4753d8e
fix mutability for chunk store
2018-12-21 12:15:26 +01:00
Dietmar Maurer
2d9d143a8f
backup-client: add optional chunk-size parameter
2018-12-21 11:18:08 +01:00
Dietmar Maurer
f0819fe5a5
backup-client: do not start garbage collection after backup
2018-12-21 09:24:55 +01:00
Dietmar Maurer
d62e6e2264
backup-client: use 1M chunks, make chunk_size configurable
2018-12-21 08:36:57 +01:00
Dietmar Maurer
c34eb16651
backup-clinet: specify target file name
2018-12-20 14:09:31 +01:00
Dietmar Maurer
c2d9b3914a
add test code to access static global state
2018-12-20 14:03:42 +01:00
Dietmar Maurer
fe0e04c69c
use proxmox-backup as package name
2018-12-20 10:45:14 +01:00
Dietmar Maurer
f7dd683b46
fix tests
2018-12-19 13:54:22 +01:00
Dietmar Maurer
277fc5a3ce
try to improve error messages
2018-12-19 13:40:26 +01:00
Dietmar Maurer
eae8aa3aa9
cleanups: avoid compiler warnings
2018-12-19 12:49:23 +01:00
Dietmar Maurer
6c20a13d3c
cleanup - avoid too much indentation
2018-12-19 12:40:26 +01:00
Dietmar Maurer
1c43c56b06
cleanup
2018-12-19 12:30:37 +01:00
Dietmar Maurer
2bf5f6b220
ChunkStore::sweep_used_chunks - use openat to speedup directory walks
2018-12-19 12:20:01 +01:00
Wolfgang Bumiller
1628a4c731
use timers with a signal for file locking
...
* rename lock_file -> open_file_locked,
* add lock_file as a function working on already-opened
files
* change timeout types to std::time::Duration
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-19 11:23:00 +01:00
Wolfgang Bumiller
8cf6e764ad
add tools::timer module
...
This wraps POSIX timers via timer_create(2).
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-19 11:23:00 +01:00
Dietmar Maurer
365bb90f17
move lock_file to tools.rs
2018-12-19 10:02:24 +01:00
Dietmar Maurer
08481a0b60
impl sweep_used_chunks, first try
2018-12-19 09:51:33 +01:00