be9b39e17a
src/pxar/sequential_decoder.rs: major refactoring of restore_sequential()
...
This splits the functionality of restore_sequential() into several smaller
functions in order to allow to reuse them when restoring by seeking based on
the goodbye table offsets.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:45:50 +02:00
48a4509c67
src/pxar/encoder.rs: fix bug with incorrect offset stored in goodbye table
...
Commit cd7dc87903
2019-07-05 06:39:39 +02:00
7dcbe051e9
pxar: Refactor SequentialDecoder to store the callback function within the struct
...
Reduces the number of arguments for the function calls within the decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:38:22 +02:00
b46c3fad03
src/backup/fixed_index.rs: use correct size
...
We need to consider that the last chunk may have smaller size.
2019-07-04 15:13:22 +02:00
9cc88a7c00
src/client/http_client.rs - finish: hold reference to avoid early connection close
2019-07-04 15:11:23 +02:00
5e04ec707d
src/api2/backup/environment.rs: improve chunk alignment/size checks
2019-07-04 13:40:43 +02:00
a24e3993e0
src/backup/chunk_store.rs: coding style fixes
2019-07-04 11:39:10 +02:00
e4c2fbf170
src/backup/chunk_store.rs: additionally log chunk count
2019-07-04 11:27:11 +02:00
9850bcdf19
src/backup/chunk_store.rs: improve error reporting
2019-07-04 11:21:54 +02:00
a3f3e91da2
backup/chunk_store: rework chunk iterator
...
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-04 10:55:17 +02:00
99168f43e6
src/client/http_client.rs: s/set_recv_buf_size/set_recv_buffer_size/
2019-07-04 10:51:18 +02:00
c698636a5d
src/tools/fs.rs - read_dir: return nix::Result
2019-07-04 10:47:16 +02:00
a57360983b
src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item
2019-07-04 09:26:44 +02:00
0f0a35b390
src/backup/dynamic_index.rs: lock file inside new, code cleanup
2019-07-04 08:17:30 +02:00
a7c72ad9eb
src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store
2019-07-04 08:13:16 +02:00
a660978c9a
src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats
2019-07-04 07:57:43 +02:00
86eda3eb0d
src/bin/proxmox-backup-client.rs: implement restore using BackupReader
2019-07-03 15:45:12 +02:00
b2a6897982
src/client/remote_chunk_reader.rs: remove dead code
2019-07-03 14:56:24 +02:00
7f99bf691a
src/client/remote_chunk_reader.rs: implement remote chunk reader
2019-07-03 14:39:13 +02:00
4f6aaf542c
src/client/http_client.rs: avoid compiler warning
2019-07-03 14:36:02 +02:00
3467cd91e9
src/client/http_client.rs: return Arc<Self>
...
Avoid to call canceller for each clone on drop ...
2019-07-03 14:26:07 +02:00
1e0784841c
src/backup/read_chunk.rs: use &mut self
2019-07-03 14:25:18 +02:00
0d70535a36
src/pxar/encoder.rs: refactor simple match statements to map_err()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:36 +02:00
6e1c31163b
src/pxar/sequential_decoder.rs: refactoring of simple match statements to map_err()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:24 +02:00
9d227350a3
src/pxar/sequential_decoder.rs: Cleanup and fix incorrect error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:16:59 +02:00
80af046794
src/server/rest.rs: avoid unwrap
2019-07-03 12:00:43 +02:00
7fb4f5642a
src/server/rest.rs: log peer address, use hyper MakeService
2019-07-03 11:54:35 +02:00
6d1f61b208
use hyper/tokio-openssl instead of hyper/tokio-tls
...
This exposes the complete SSL setup. And download is much faster
now (600MB/s instead of 130MB/s)!
2019-07-02 13:36:28 +02:00
74d0a6bc23
src/bin/proxmox-backup-proxy.rs: set socket options for max. performance
2019-07-02 10:48:58 +02:00
93f8bb0445
src/bin/h2s-client.rs: avoid compiler warnings
2019-07-02 08:59:33 +02:00
877dd11133
src/backup/{fixed, dynamic}_index.rs: improve error messages
2019-07-02 08:56:56 +02:00
5c593a4d56
src/backup/read_chunk.rs: use Arc for DataStore
2019-07-02 08:49:16 +02:00
b850673634
src/backup/read_chunk.rs: move read chunk trait into extra file
...
And implement LocalChunkReader.
2019-07-02 08:22:29 +02:00
8fad30a4b1
use latest hyper master branch to include socket buffer size fix
2019-07-02 07:09:46 +02:00
f35197f449
replace tools::vec with proxmox::tools::vec
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 11:03:25 +02:00
d0162d53d3
tools: remove io module
...
now completely replaced by proxmox::tools::io.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:12 +02:00
9b2b627fe0
update remaining users of tools::io::ops
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:03 +02:00
9110a69bd8
tools: remove read/write
...
now completely replaced by
proxmox::tools::io::{ReadExt, WriteExt}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:42 +02:00
ca3c3ce997
daemon: remove last use of tools::read/write
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:13 +02:00
5485b579a1
backup: use new proxmox::tools::io traits
...
This replaces all `write_value` calls with `write_le_value`
calls!
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:37:02 +02:00
dc9775d1b0
src/bin/h2s-client.rs: increase tokio tcp stream read buffer size
...
This improves download speed by factor 10.
2019-06-29 18:09:50 +02:00
34f7131160
src/bin/h2s-client.rs: avoid hyper, use h2 directly
...
But performance is still bad.
2019-06-29 15:58:18 +02:00
5ed2277f0e
src/bin/h2s-server.rs: test to reproduce slow h2 download
...
Simplified code, use with h2s-client.rs.
2019-06-29 14:57:54 +02:00
17243003b3
src/bin/download-speed.rs: h2 download speedtest for our server
...
I get about 30MB/s (much too slow)
2019-06-29 13:43:10 +02:00
a2b29b68d4
src/client/http_client.rs: use maximal h2 window sizes
2019-06-29 11:05:36 +02:00
872debdefa
src/bin/h2client.rs: tune h2 client parameters to get reasonable speed
...
We now get about 2.5GB/s (not really good, but better)
2019-06-29 10:41:24 +02:00
38294bcc56
src/client.rs: make module put_to_stream public
2019-06-29 10:40:43 +02:00
fded1f3154
add code to test H2 speed
...
I currently get about 1MB/s, which is ways too slow (must be a bug)?
2019-06-29 10:08:12 +02:00
d48a9955a5
src/backup/dynamic_index.rs: introduce ReadChunk trait
2019-06-28 16:35:00 +02:00
09d7dc5024
src/api2/reader.rs: add download chunk api
2019-06-28 16:27:01 +02:00
fcf5dea56d
src/client/http_client.rs: increase h2 window size, set tcp nodelay option
...
To increase h2 download speed.
2019-06-28 16:00:58 +02:00
81a6ce6fde
src/backup/chunk_store.rs: new method chunk_path()
...
Returns the absolute path.
2019-06-28 15:48:09 +02:00
fcfb84fedf
file download: avoid unnecessary copy
2019-06-28 07:07:52 +02:00
984a7c3502
src/client/http_client.rs - h2 download: implement flow control
2019-06-28 07:02:43 +02:00
dd066d28e2
src/api2/reader.rs: implement backup reader protocol
2019-06-27 09:01:41 +02:00
42a87f7b96
src/server/h2service.rs: implement generic h2 service
2019-06-26 17:38:33 +02:00
fb0470837b
src/client/http_client.rs: split out code to start new h2 connections
2019-06-26 12:09:18 +02:00
b208da8393
src/backup/data_{chunk,blob}.rs: add verify_crc method
2019-06-26 09:54:25 +02:00
fef44d4f78
src/bin/proxmox-backup-client.rs: decode blobs on download
...
No more need for that clumsy blob cli (removed).
2019-06-26 09:18:59 +02:00
c2b945341c
src/client/http_client.rs - download: use generic Write type, return writer.
...
Make it possible to write int Vec<u8>.
2019-06-26 09:17:13 +02:00
bb8231409e
src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json
...
Use the KeyConfig serialization with kdf = None.
2019-06-26 07:32:34 +02:00
3031e44c58
src/bin/proxmox-backup-client.rs: remove stale download file, start blob cli
2019-06-25 13:00:37 +02:00
f14a8c9a85
src/api2/admin/datastore.rs: use correct path for download
2019-06-25 12:59:36 +02:00
81820b0d4d
src/client/http_client.rs - download: fix error handling
2019-06-25 12:43:55 +02:00
45db6f89e1
src/bin/proxmox-backup-client.rs: implement download command
2019-06-25 11:17:24 +02:00
9e47c0a56e
src/api2/admin/datastore.rs: implement generic download method
...
We simply allow to download raw binary data.
2019-06-25 10:16:59 +02:00
da7d67213e
src/api2/admin/datastore/pxar.rs: remove pxar upload api
...
Upload needs to be done using the backup protocol.
2019-06-25 09:45:35 +02:00
dd8e744f9a
src/api2/admin/datastore.rs: use a worker log log prune actions
2019-06-25 08:12:25 +02:00
6899dbfb47
src/bin/proxmox-backup-client.rs: strip .blob extension
2019-06-25 07:11:02 +02:00
c6d203bbd3
src/backup/backup_info.rs: also list .blob files
2019-06-25 07:07:45 +02:00
045e8a568d
src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key
2019-06-25 07:00:07 +02:00
79bc7345e8
src/cli/command.rs: fix ReST output format
2019-06-25 06:19:51 +02:00
9f46c7de4b
src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup
2019-06-24 13:56:37 +02:00
cd7dc87903
pxar: impl .pxarexclude parsing and exclude matching
...
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.
Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-24 10:26:20 +02:00
46bd880041
src/api2/backup/environment.rs: new helper add_blob()
2019-06-24 09:35:37 +02:00
cb08ac3efe
src/api2/backup.rs: replace upload_config with upload_blob
2019-06-23 11:44:30 +02:00
a38c5d4d12
src/backup/data_blob.rs: implement from_raw()
2019-06-23 09:35:44 +02:00
6f083b7a92
src/api2/backup/upload_chunk.rs: compute crc on upload
2019-06-23 09:04:42 +02:00
ba01828d38
try to use use proxmox::tools::io::ops::ReadExtOps
2019-06-22 16:29:10 +02:00
27042ce637
src/backup/file_formats.rs: improve docs
2019-06-22 15:45:36 +02:00
9f83e0f7ee
src/backup/crypt_config.rs: simplify decode interface
2019-06-22 13:24:29 +02:00
0066c6d972
src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead
2019-06-22 13:02:53 +02:00
ee8a7e8080
src/backup/crypt_config.rs: implement encrypt_to (output to writer)
2019-06-22 12:25:04 +02:00
991abfa8b4
src/backup/file_formats.rs: split out file format data
2019-06-22 09:12:25 +02:00
b7f4f27d6c
add crc field for binary blobs formats
2019-06-21 18:49:04 +02:00
a7f67a9a9c
depend on crc32fast
...
We can use this to compute fast checksums to test file integrity (not crypto safe).
2019-06-21 15:58:00 +02:00
3025b3a53c
src/backup/data_blob.rs: new file format for binary blobs
2019-06-21 11:32:07 +02:00
077a8cae63
src/backup/crypt_config.rs: make code more flexible
...
Allow to pass magic numbers as parameters.
2019-06-21 10:50:48 +02:00
c38266c18d
renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs
2019-06-21 09:51:18 +02:00
3ea8bfc9c0
src/bin/proxmox-backup-client.rs: add import-master-pubkey command
2019-06-21 07:34:17 +02:00
37c5a17505
src/bin/proxmox-backup-client.rs: helper to generate RSA master key
2019-06-21 07:12:19 +02:00
ab44acff57
src/backup/key_derivation.rs: add modified field to key file
2019-06-20 07:58:33 +02:00
6d0983dbe1
src/bin/proxmox-backup-client.rs: add keyfile parameter for backup
2019-06-19 17:16:41 +02:00
181f097af9
src/bin/proxmox-backup-client.rs - key API: pass kdf parameter
...
Allow to store keys without password.
2019-06-18 18:21:01 +02:00
9b06db4563
src/bin/proxmox-backup-client.rs: use file system path to reference keys
2019-06-18 17:38:50 +02:00
091540298e
src/bin/proxmox-backup-client.rs - create_key: always read password from tty
2019-06-18 17:27:52 +02:00
826f309bf5
src/backup/key_derivation.rs: move kdf code into separate file
2019-06-18 11:17:22 +02:00
11515438cc
Cargo.toml: use serde feature derive
2019-06-18 06:23:25 +02:00
ac71623483
src/bin/proxmox-backup-client.rs: implement "key change-passphrase"
2019-06-17 13:10:00 +02:00
f2401311b0
src/bin/proxmox-backup-client.rs: start key management
2019-06-17 10:33:24 +02:00
a7dd483097
src/backup/*_index.rs: used generated magic numbers
2019-06-14 14:58:37 +02:00
bffd40d6b7
src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools
2019-06-14 11:40:04 +02:00
00388226e6
src/api2/backup/environment.rs: log index checksums
2019-06-14 10:36:20 +02:00
9335d74eba
src/backup/fixed_index.rs: compute checksum over the index
2019-06-14 10:34:29 +02:00
16ff6b7cd8
src/backup/dynamic_index.rs: compute checksum over the index
2019-06-14 10:33:59 +02:00
36075475a6
src/api2/backup/environment.rs: log duplicate chunks
2019-06-14 07:12:30 +02:00
fa148dbd11
src/api2/backup/upload_chunk.rs: verify chunk data if possible
2019-06-14 06:23:46 +02:00
0f37577092
src/bin/cipherbench.rs: add test for zstd compression
2019-06-13 17:24:57 +02:00
9399c98f82
src/bin/cipherbench.rs: cipher speed test with large blocks
2019-06-13 17:16:43 +02:00
f640f23ac5
src/backup/data_chunk.rs: fix typo
2019-06-13 16:42:55 +02:00
dba72d5016
src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:56:50 +02:00
68ab376ae9
src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:47:54 +02:00
f98ac774ee
backup: Add support for client side encryption
...
first try ...
2019-06-13 11:47:23 +02:00
51929e4532
src/backup/data_chunk.rs - decode: make crypt_config optional
2019-06-13 11:44:38 +02:00
046bcb9773
src/backup/crypt_setup.rs: add comment for possible improvements
2019-06-13 07:35:59 +02:00
bd0e3c7cfa
src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs
2019-06-13 07:35:02 +02:00
6762db70d6
src/api2/types.rs: define CHUNK_DIGEST_SCHEMA
2019-06-13 07:30:42 +02:00
b595cb9d2c
src/backup/data_chunk.rs: Data Chunk builder
...
Make handling encoded chunks easier.
2019-06-12 13:37:01 +02:00
ef27200cec
src/backup/crypt_setup.rs: fix encoder for uncompressed chunks
2019-06-10 09:56:06 +02:00
74792b95b2
src/backup/crypt_setup.rs: allow compressed and uncompressed chunks
2019-06-10 08:27:35 +02:00
9e0187a203
src/backup/crypt_setup.rs: remove unused CryptData struct
2019-06-10 07:55:36 +02:00
bec8498ae3
src/backup/crypt_setup.rs: avoid useless allocations
2019-06-09 11:44:17 +02:00
c20b17b460
src/backup.rs: define magic numbers for chunk store file format
2019-06-08 10:34:20 +02:00
48b4b40b57
src/backup/crypt_setup.rs: crypto helpers
2019-06-08 09:56:51 +02:00
dd5495d6dc
tree-wide: use 'dyn' for all trait objects
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-07 13:13:48 +02:00
e993db91cd
pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder
...
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:20 +02:00
1f319e766e
src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:06 +02:00
9b77388f2a
src/pxar/encoder.rs: minor refactoring
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:55 +02:00
a4cc8eb7c5
src/pxar/encoder.rs: check if ioctl supported by filesystem
...
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:31 +02:00
4800df7ba6
src/api2/admin/datastore.rs: remove test-upload api
2019-06-07 07:52:11 +02:00
113898d7e9
src/api2/admin/datastore/upload.rs: remove unused upload api
2019-06-07 07:39:41 +02:00
4247fccb0f
src/client/http_client.rs: automatically close connection in finish
2019-06-05 11:41:50 +02:00
cb4426b348
make h2 client connection cancellable
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-05 09:45:59 +02:00
986bef16be
src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1
...
So that we can include it in static doc strings.
2019-06-05 08:41:20 +02:00
c9ec0956cf
src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1
2019-06-05 08:12:13 +02:00
44c54845b3
src/tools/xattr.rs: fix test when run as root
2019-06-05 07:57:42 +02:00
7773ccc11f
src/api2.rs: move backup api to /backup
2019-06-05 07:23:21 +02:00
97eeea3b4a
src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:29:02 +02:00
1ce2f7cfb9
src/pxar/encoder.rs: style fix
2019-06-05 06:27:39 +02:00
e7b60a16c8
pxar: impl storing/dumping/restoring of quota project ids
...
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:26:02 +02:00
7312ab9eef
src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID
...
Defines the types needed to store/restore quota project ids in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:23:06 +02:00
042babe4e7
src/tools/fs.rs: impl ioctl calls to get/set fsxattr
...
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:22:49 +02:00
22e5aa1d3e
src/bin/dump-backup-api.rs: helper to generate backup API docs
2019-06-04 13:12:42 +02:00
5e0f305142
src/api_schema/format.rs: depend on textwrap, new wrap_text helper
2019-06-04 12:42:02 +02:00
339ddfcbfa
src/cli/command.rs: move doc generator code to src/api_schema/format.rs
2019-06-04 12:32:22 +02:00
ef39bf95b3
src/client/http_client.rs - upload_config: improve error message
2019-06-03 10:53:09 +02:00
ec8a9bb953
src/bin/proxmox-backup-client.rs: implement upload_config
2019-06-03 10:39:44 +02:00
39d6846e1e
src/api2/admin/datastore/backup.rs: implement config file upload
2019-06-03 09:45:14 +02:00
1c0472e88f
avoid compiler warnings
2019-06-03 06:43:47 +02:00
36898ffce6
src/backup/chunk_stream.rs: add optional chunk_size parameter
2019-05-30 13:28:24 +02:00
49ef316bcd
src/bin/proxmox-backup-client.rs: backup_image() - pass correct chunk size
2019-05-30 12:52:12 +02:00
202cadc29e
src/bin/proxmox-backup-client.rs: remove unused code
2019-05-30 12:47:35 +02:00
6af905c18f
src/bin/proxmox-backup-client.rs: implement image/block-device upload
2019-05-30 12:46:01 +02:00
eb1804c567
src/bin/proxmox-backup-client.rs: use std lib instead of nix
2019-05-30 12:19:23 +02:00
6e907505aa
src/bin/proxmox-backup-client.rs: fix file type test
2019-05-30 12:08:34 +02:00
96482891ae
src/api2/admin/datastore/backup/environment.rs: log upload statistics
2019-05-30 09:21:19 +02:00
642322b433
src/api2/admin/datastore/backup.rs: use separate api entry points for chunk upload
...
So that we can provide better statistics (patches follows).
2019-05-30 09:21:19 +02:00
390e83c9b9
typo fix and doc improvement
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:57:28 +02:00
7fb49397d5
doc fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:56:14 +02:00
382609b04e
tools: add Cancellable, start a futures submodule
...
To make a future cancellable, use:
let (future, canceller) =
crate::tools::futures::cancellable(future);
Proceed with using `future` as usual, `canceller` is
clonable and can cancel the future via the `.cancel()`
method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
0d32d71fb7
tools: add AsyncMutex::new_locked
...
Allows creating a pre-locked mutex, returning the mutex and
a guard.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
e6389f4e75
src/api2/admin/datastore/backup/environment.rs: do not allow empty backups (no files)
2019-05-29 10:38:57 +02:00
39e60bd6fa
src/api2/admin/datastore/backup.rs: improve debug logging
2019-05-29 10:17:38 +02:00
4ae6551c78
src/api_schema/router.rs: disable debug println
2019-05-29 09:42:16 +02:00
a42d1f55c3
src/api2/admin/datastore/backup.rs: add debug flag
2019-05-29 09:35:21 +02:00
976595e1a9
src/backup/dynamic_index.rs: split class DynamicIndexWriter
2019-05-29 08:49:57 +02:00
cb0708dd46
src/backup/fixed_index.rs: remove ChunkStat from struct
2019-05-29 07:08:34 +02:00
01af11f340
src/backup/fixed_index.rs: improve consistency checks
2019-05-29 07:02:45 +02:00
9b38443314
pxar: change logic and impl all command line flags for xattrs/fcaps/acls
...
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-29 06:51:20 +02:00
60e589a111
src/api2/admin/datastore/backup.rs: log sucessful finish call
2019-05-28 12:55:22 +02:00
c4ff3dcefd
src/bin/proxmox-backup-client.rs: use new BackupClient to upload data
2019-05-28 10:12:44 +02:00
10241c20ea
src/tools.rs: implement image_size helper
2019-05-28 09:35:08 +02:00
006f3ff407
src/api2/admin/datastore/backup/environment.rs: add more consistency checks
2019-05-28 09:21:31 +02:00
3dc5b2a203
src/client/http_client.rs: use start of chunk as offset
2019-05-28 09:21:31 +02:00
8de20e5c1f
src/client/http_client.rs: code cleanup
2019-05-28 08:53:27 +02:00
fc14b84937
src/backup/fixed_index.rs: improve bound checks inside add_digest
2019-05-28 08:47:03 +02:00
e1225de467
src/backup/fixed_index.rs: store index_length to avoid code duplication
2019-05-28 08:27:57 +02:00
e3d525feb7
src/api2/admin/datastore/backup.rs: fix typo
2019-05-28 07:13:19 +02:00
af309d4d5c
src/bin/pxar.rs: set archive permissions to 0o640
...
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:24:47 +02:00
ab87f167f1
src/pxar/encoder.rs: Refactor file stat
...
Introduce helper functions to check file stats
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:23:49 +02:00
578b601159
src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
...
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:22:16 +02:00
a42fa400ee
src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api
2019-05-28 06:18:55 +02:00
29ae5c86a2
src/backup/fixed_index.rs: make chunk_size public
2019-05-28 06:18:24 +02:00
7dd1bcac58
src/client/http_client.rs: correctly parse error response in download_chunk_list
2019-05-27 09:35:30 +02:00
a1e7cff3ea
src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk
2019-05-27 07:57:43 +02:00
771953f9eb
src/api2/admin/datastore/backup.rs: use smaller window sizes
2019-05-27 07:53:40 +02:00
174ad378d8
src/client/http_client.rs: fix request pipelining
2019-05-27 07:24:32 +02:00
624362226e
src/client/merge_known_chunks.rs: use MergedChunkInfo as input
2019-05-26 10:52:56 +02:00
05cba08c9c
src/client/http_client.rs: append chunks in correct order
...
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
2019-05-26 09:57:34 +02:00
417cb0731e
src/api2/admin/datastore/backup.rs: verify chunk offset
...
This clearly shows that the current approach does not work - seems we get
chunks out of order.
2019-05-26 08:49:29 +02:00
34114e2606
src/server/rest.rs: correctly verify json parameters
2019-05-24 09:10:59 +02:00
eb2bdd1bcc
src/client/http_client.rs: encode parameters inside body
...
To avoid problems with long uris.
2019-05-24 08:32:55 +02:00
9bb675ec23
src/client/merge_known_chunks.rs: reduce RAM usage
...
There is no need to store chunk data for known chunks.
2019-05-24 07:42:58 +02:00
ecb21b4794
src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough
2019-05-24 07:36:09 +02:00
2dbba78b98
src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:27:24 +02:00
6a19448004
src/pxar/encoder.rs: impl support to store ACLs in pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:21:33 +02:00
7cfaade7c8
src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:16:55 +02:00
b344461b33
pxar: pass feature_flags to encoder/decoder instead of individual flags
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
687407741b
src/tools/xattr.rs: add test for is_valid_xattr_name()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
357e4614e2
Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
c05a8c8d18
doc-test fixup
...
cargo test by default compiles and runs all code snippets
found in the documentation...
oops...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 15:22:33 +02:00
cc84a830c5
RFC: schema: make enums static
...
I don't see a reason to allow these to be dynamically
modifiable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 12:38:51 +02:00
aa1b2e04fe
src/client/merge_known_chunks.rs: merge known chunks
...
To decrease the number of api calls required...
2019-05-23 12:31:09 +02:00
91320f0879
src/client/http_client.rs: use ChunkInfo streams
...
This will make out of order uploads possible...
2019-05-23 12:31:09 +02:00
8ea3b1d188
tree-wide: use the new vec/io tools modules
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
9cdda3f7c7
tools: add helpful vector and read operations
...
After importing the I/O ops trait via:
use crate::tools::io::ops::*;
Instead of:
let mut buffer = vec![0u8; 65536];
file.read_exact(&mut buffer)?;
use:
let buffer = file.read_exact_allocated(65536)?;
After importing the vector helpers via:
use crate::tools::vec::{self, ops::*};
For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
let mut buffer = vec![0u8; len];
use:
let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
8bea85b42e
src/api2/admin/datastore/backup.rs: verify file size and chunk count on close
2019-05-23 08:50:36 +02:00
1179e15842
src/server/rest.rs: improve error handling
2019-05-23 08:15:32 +02:00
907f5bb0d8
src/api2/admin/datastore/backup/service.rs: improve error logging
2019-05-23 08:05:39 +02:00
32e1bf4260
src/bin/upload-speed.rs: code cleanup
2019-05-23 07:20:05 +02:00
0d9bab05ab
src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:50 +02:00
4add706092
src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:32 +02:00
54b14ad741
src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:09 +02:00
1201360def
src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:39:16 +02:00
8bc8791338
src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:37:37 +02:00
a45a123950
src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
...
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:34:36 +02:00
d6f204eda6
src/client/http_client.rs: re-order code - public functions first
2019-05-23 06:28:10 +02:00
9af37c8f0e
src/client/http_client.rs: move low level H2 code into separate class
2019-05-22 17:28:25 +02:00
d2c48afc6e
src/client/http_client.rs: code cleanup
2019-05-22 13:24:33 +02:00
6ab34afa88
src/client/http_client.rs: cleanup h2 backup client
2019-05-22 13:05:51 +02:00
a55fcd740f
src/api2/admin/datastore/backup.rs: suse POST for finish() api
2019-05-22 13:04:58 +02:00
de6742c04f
src/bin/h2test.rs: removed
...
This test does no work anymore.
2019-05-22 11:14:56 +02:00
5a0b521809
src/api2/admin/datastore/backup.rs: remove test callbacks
2019-05-22 10:55:27 +02:00
de61bc9264
src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr
...
As git does not preserve xattrs they have to be set before they can be read.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-22 10:01:10 +02:00
553610b43e
src/client/http_client.rs: implement download_chunk_list
2019-05-22 09:46:02 +02:00
0cc0fffd1e
src/backup/chunk_stream.rs: use BytesMut in ChunkStream
2019-05-22 09:39:02 +02:00
c052be5c86
src/backup/chunk_stream.rs: use more generics
2019-05-22 09:18:05 +02:00
169c0e060f
src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut)
2019-05-22 09:05:35 +02:00
3be3f3dcc4
src/backup/chunk_stream.rs: fix behaviour for large input buffers
2019-05-22 08:05:27 +02:00
02fa54ff3c
src/backup/chunk_stream.rs: assemble data with correct order
2019-05-22 07:44:51 +02:00
d0afeaa831
src/api2/admin/datastore/backup.rs: improve error message
2019-05-22 07:43:54 +02:00
11867a2b54
src/tools.rs: fix hex_to_digest
2019-05-21 13:42:13 +02:00
7f3d2ffab9
src/backup/index.rs: rename ChunkListReader into DigestListEncoder
2019-05-21 12:28:44 +02:00
ee53955f52
src/backup/index.rs: implement DigestListDecoder
2019-05-21 12:21:22 +02:00
a95849321c
src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks
2019-05-21 09:54:11 +02:00
40f4e198a8
src/backup/dynamic_index.rs: add chunk_info method
2019-05-21 09:52:19 +02:00
596182056b
src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:15:30 +02:00
b303057a19
src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:06:26 +02:00
bee8d8ea78
src/tools/xattr.rs: add functions name_store and security_capability
...
These functions allow to check if the name for xattrs/fcaps are valid.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:05:31 +02:00
a09c0e38d8
src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks
2019-05-20 18:05:10 +02:00
97f03eff13
src/client/pxar_backup_stream.rs: fix docu
2019-05-20 17:29:00 +02:00
82ab72304e
src/client/http_client.rs: implement upload_stream
2019-05-20 14:19:24 +02:00
2698e8a514
src/client/pxar_backup_stream.rs: close pipe inside drop
2019-05-20 11:20:33 +02:00
f2269d8f99
src/tools.rs: implement hex_to_digest
2019-05-19 12:15:46 +02:00
751030aa7c
src/bin/test_chunk_speed2.rs: test max chunk size
2019-05-19 11:09:02 +02:00
ff77dbbea3
Revert "src/backup/chunk_stream.rs: simplify code"
...
This reverts commit e058744d8f
2019-05-19 11:05:56 +02:00
178ac042c4
src/api_schema/schema.rs: improve error messages
2019-05-19 10:18:51 +02:00
0c516b123d
src/client/pxar_backup_stream.rs: use WrappedStreamReader
...
to make it fully async ...
2019-05-18 15:13:31 +02:00
c6e28b66c6
src/client/pxar_backup_stream.rs: use 1M pipe buffer size
2019-05-18 12:57:43 +02:00
6276448519
src/bin/test_chunk_speed2.rs: test chunker speed with real data
2019-05-18 11:59:17 +02:00
e058744d8f
src/backup/chunk_stream.rs: simplify code
2019-05-18 10:55:46 +02:00
8a7cc7565f
src/backup/chunk_stream.rs: implement fixed size chunk stream
2019-05-18 10:55:46 +02:00
9c9ad941c0
src/client/http_client.rs: improve speed test
...
pipeline responses to a separate async channel ...
2019-05-18 10:55:46 +02:00
2dcdd3b427
src/tools/xattrs.rs: impl libc wrapper functions to get/set xattrs and fcaps
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:21 +02:00
de2016d58a
src/pxar/format_definitions.rs: add format definitions for xattrs and fcaps
...
In addition to the format definition, the traits needed for sorting of xattr
entries by name are derived.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:13 +02:00
dec1ff18b8
src/tools/acl.rs: impl add_entry_full for ACL and set_permissions for ACLEntry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:10:06 +02:00
adec8ea263
src/api2/admin/datastore/backup.rs: add speedtest api, improve upload speed
...
We need to disable tcp Nagle algorythm (set_nodelay), and use larger window size for http2
2019-05-16 10:24:23 +02:00
372724afea
src/api2/admin/datastore/backup.rs: implement finish_backup
2019-05-15 12:58:55 +02:00
bb105f9dae
src/api2/admin/datastore/backup.rs: improve logging
2019-05-15 10:09:46 +02:00
25ddc15fe0
src/client/pipe_to_stream.rs: add missing file
2019-05-15 09:07:19 +02:00
a20772525f
src/api2/admin/datastore/backup.rs: implement close for dynamic writers
2019-05-15 07:58:05 +02:00
4e93f8c164
src/api2/types.rs: add schema/format for file names
2019-05-15 07:14:08 +02:00
e3dbd41b20
src/client/http_client.rs: add flow control for h2 upload
2019-05-14 14:54:21 +02:00
97f22ce57d
src/client/http_client.rs: implement h2 upload
...
without flow control so far ...
2019-05-14 12:58:59 +02:00
dafc27aeff
src/backup/chunk_stream.rs: async chunk stream
2019-05-14 10:05:29 +02:00
c0e22141ed
src/client/http_client.rs: derive Clone for H2Client
2019-05-14 09:00:24 +02:00
25e205a1d3
src/tools.rs: add mod acl to tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:34:24 +02:00
d22096ef4b
src/tools/acl.rs: impl POSIX access control list (ACL) manipulation tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:28:42 +02:00
1133fe9a63
src/server/rest.rs: do not log 1xx status codes as errors
2019-05-14 06:23:22 +02:00
850ac6d0f0
src/client/http_client.rs: directly return H2Client on upgrade
2019-05-13 16:44:59 +02:00
10130cf48c
src/client/http_client.rs: cleanup, remove debug println
2019-05-13 12:11:18 +02:00
b57cb26406
src/client/http_client.rs: implement http2 client wrapper
2019-05-13 10:27:22 +02:00
9e391bb7f0
src/client/http_client.rs: allow to pass parameters (encode them as query string)
2019-05-13 09:12:03 +02:00
fbb798f6d2
src/api2/admin/datastore/backup.rs: ignore errors from last_backup
2019-05-13 07:06:56 +02:00
d36113665b
src/api2/admin/datastore/backup.rs: implement chunk index download from previous backup
2019-05-11 16:05:50 +02:00
457531e7fe
src/backup/index.rs: new ChunkListReader
2019-05-11 16:01:42 +02:00
af1d68185c
src/api2/admin/datastore/backup.rs: add missing 'archive-name' parameter to schema
2019-05-11 12:25:15 +02:00
6b95c7df07
src/api2/admin/datastore/backup/environment.rs: do not store path
2019-05-11 12:13:44 +02:00
51a4f63ff7
src/backup/datastore.rs: move last_backup to BackupInfo
2019-05-11 12:07:09 +02:00
b02a52e3a5
src/backup/datastore.rs: impl last_backup helper
2019-05-11 11:36:46 +02:00
c09775011a
src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups
2019-05-11 10:19:34 +02:00
f9578f3c79
backup-api: allow to create DynamicIndexWriter and add chunks to to
2019-05-10 10:25:40 +02:00
35a2d8a6a6
src/backup/backup_info.rs: Derive Clone
2019-05-10 06:59:23 +02:00
0506d1ed04
upload_chunk.rs: add comment about possible unnecessary code
2019-05-10 06:15:25 +02:00
5f42ef009a
cleanup: remove unused code
2019-05-09 18:11:39 +02:00
b4b63e520d
handle_async_api_request: put rpcenv into a Box
...
So that we can pass rpcenv into futures.
2019-05-09 18:01:24 +02:00
2c41096a4b
src/api2/admin/datastore/backup/upload_chunk.rs: fix maximal chunk size
2019-05-09 13:29:50 +02:00
78b5191550
src/server/formatter.rs: further cleanups and renaming ...
2019-05-09 13:28:26 +02:00
d55037e471
src/server/formatter.rs: rename format_result to format_data
...
To avoid confusions with Rust Result type.
2019-05-09 13:15:15 +02:00
21ee7912fa
src/api2/admin/datastore/backup.rs: implement upload chunk
2019-05-09 13:06:09 +02:00
4ebf0eabb0
src/api2/types.rs: use separate file for schema definitions
2019-05-09 07:44:09 +02:00
bd1507c4fa
src/api2/admin/datastore/backup/service.rs: move service code into extra file
2019-05-08 17:36:19 +02:00
0aadd40b4f
src/api2/admin/datastore/backup.rs: add required parameters, cleanups
2019-05-08 13:17:23 +02:00
d95ced64ad
src/api2/admin/datastore/backup/environment.rs: use separate file
2019-05-08 12:41:58 +02:00
ca60c371e2
api: rename h2upload to backup
2019-05-08 12:35:15 +02:00
d906b1124f
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 11:36:43 +02:00
58c8d7d91f
rc/api2/admin/datastore/h2upload.rs: implement BackupEnvironment
...
To pass arbitrary information/state to api methods.
2019-05-08 11:26:54 +02:00
f757b30efc
src/server/rest.rs: use generics to pass RpcEnvironment
2019-05-08 11:09:01 +02:00
2edc341b29
src/tools.rs: Add AsAyn Trait
2019-05-08 11:07:36 +02:00
090ac9f79a
src/api2/admin/datastore/h2upload.rs: avoid unwrap
2019-05-08 08:21:18 +02:00
4e3da4b366
src/api2/admin/datastore/h2upload.rs: do not log client address
2019-05-08 08:15:55 +02:00
e3a44552ee
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 08:12:49 +02:00
a66ab8ae5e
src/api2/admin/datastore/h2upload.rs: make worker abortable
2019-05-08 07:14:40 +02:00
72375ce6c6
src/api2/admin/datastore/h2upload.rs: log to worker task
2019-05-07 14:07:00 +02:00
d9bd06eae8
src/api2/admin/datastore/h2upload.rs: use WorkerTask
2019-05-07 13:42:00 +02:00
52cf506e48
src/api2/admin/datastore/h2upload.rs: implement async test api call
2019-05-07 12:26:55 +02:00
92ac375aff
src/api2/admin/datastore/h2upload.rs: improve http2 service
2019-05-07 11:24:44 +02:00
78f7e225e4
src/server/rest.rs: make handle_(a)sync_api_request public
2019-05-07 11:23:52 +02:00
75097495ad
RestEnvironment: derive Clone
2019-05-07 11:09:18 +02:00
01bf3b7b5f
move find_method to Router class
2019-05-07 11:08:30 +02:00
3578d99f3e
move normalize_path to tools::normalize_uri_path
2019-05-07 09:44:34 +02:00
96e95fc179
src/api2/admin/datastore/h2upload.rs: test if version < 2
...
We can only upgrade http 1.x connections.
2019-05-06 10:29:34 +02:00
45cdce069e
src/client/http_client.rs: request password at creation time
2019-04-30 11:44:35 +02:00
e45afdff9c
src/server/state.rs: use new BroadcastData helper
2019-04-30 10:21:48 +02:00
824b5ee4ee
src/tools/broadcast_future.rs: cleanup, decompose into two classes
...
In order to make it more usable.
2019-04-30 08:57:20 +02:00
d82ed9b0f5
tools: add AsyncMutex
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-29 14:19:03 +02:00
3a9bea3918
src/bin/h2test.rs: test h2 upgrade
2019-04-29 11:59:18 +02:00
cf639a47de
rc/client/http_client.rs: add experimental h2 upgrade code
2019-04-29 11:57:58 +02:00
5a2df00004
src/client/http_client.rs: cleanup, make login fully async
2019-04-28 10:55:03 +02:00
3dceb9b304
src/tools/broadcast_future.rs: add new constructor new_oneshot()
...
To simplify usage. Also added a test case.
2019-04-27 10:56:49 +02:00
152764ec15
src/api2/admin/datastore/h2upload.rs: add test code for http2
2019-04-27 08:57:35 +02:00
490be29ed6
src/tools/broadcast_future.rs: new helper class
2019-04-26 17:57:57 +02:00
e888155712
proxy: allow .key/.pem files instead of a pkcs12/.pfx file
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-26 10:47:33 +02:00
8b7c0def84
src/tools/procfs.rs: clean up tab indentation and use whitespaces instead
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-25 13:14:52 +02:00
5e5eed5c3b
use double-fork for reload
...
To ensure the new process' parent is pid 1, so systemd won't
complain about supervising a process it does not own.
Fixes the following log spam on reloads:
Apr 25 10:50:54 deb-dev systemd[1]: proxmox-backup.service: Supervising process 1625 which is not our child. We'll most likely not notice when it exits.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
efd1536eb7
add a wrapper around nix::unistd::pipe2
...
Using O_CLOEXEC by default, and returning Fd handles to
ensure they get dropped on bail!() or panic!() if the RawFds
aren't used yet.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
897982e237
tools: add read/write convenience helpers
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
d98c9a7a38
use service Type=notify
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:33 +00:00
9c351a36c4
tools/daemon: add sd_notify wrapper
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:28 +00:00
d8871f2f18
src/tools/procfs.rs: avoid useless allocations
2019-04-24 09:35:15 +02:00
ae694f44ef
src/tools/procfs.rs: add tests for read_proc_net_route() and read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:52 +02:00
ec8db837fb
src/tools/procfs.rs: refactor read_proc_net_route() and read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:32 +02:00
eb46c3d1ec
src/tools/procfs.rs: implement read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-23 14:31:23 +02:00
6358b68236
cleanup: avoid compiler warnings
2019-04-23 12:17:24 +02:00
e06ffb10d1
src/api_schema/router.rs: fix typo (subdira -> subdir)
2019-04-16 12:20:17 +02:00
13f1cc17ea
src/api_schema/router.rs: implement list_subdirs() helper
2019-04-16 12:07:02 +02:00
062d4916ff
api_schema: allow generic api handler functions
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-16 11:22:23 +02:00
286f0d4099
src/tools/procfs.rs: implement read_proc_net_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-15 11:13:16 +02:00
1a6c9415ae
src/api2/node/services.rs: add proxmox-backup-proxy service
2019-04-15 09:48:02 +02:00
75bc49bed7
src/server/worker_task.rs: implement abort channel
2019-04-15 09:38:05 +02:00
912524752e
src/api2/admin/datastore/pxar.rs: create worker task for file upload
2019-04-15 08:38:14 +02:00
1be71fb02b
src/server/worker_task.rs_ do not write ERROR: prefix into task index
2019-04-15 08:37:13 +02:00
418def7a78
src/server/worker_task.rs: move common code into log_result
...
Make new() pub
2019-04-15 07:49:03 +02:00
37b87869b5
src/server/upid.rs: check for illegal characters
2019-04-15 07:47:51 +02:00
d2981e2738
src/tools/procfs.rs: implement read_proc_net_dev()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:59 +02:00
13abfdc57a
src/tools/procfs.rs: implement read_memory_usage()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:01 +02:00
3005ef48b6
src/tools/procfs.rs: Rename variable and struct to be more consistent
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:34 +02:00
cc93bacb72
src/tools/procfs.rs: simplify read_cpuinfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:14 +02:00
f2b99c34f7
src/api2/admin/datastore.rs: implement API to return last GC status
2019-04-11 12:04:25 +02:00
59961b8954
src/server/command_socket.rs: check control socket permissions
2019-04-11 10:51:59 +02:00
ba70040dbf
src/server/worker_task.rs: change task list sort order - latest on top
2019-04-11 09:43:33 +02:00
d3f4c08f33
src/server/worker_task.rs: catch panics in worker threads
2019-04-11 07:56:06 +02:00
620dccf1a1
tools/daemon: dup the TcpListener file descriptor
...
Now that we let hyper shutdown gracefully we need an owned
version of the listening socket to prevent it from closing
before running the reload preparations.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
d96bb7f163
tools: add Fd helper
...
stores a raw file descriptor with a drop handler for
safekeeping in closures
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
1cd33633aa
/src/tools: Cleanup tempfiles in folder
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:56:22 +02:00
a665dea128
src/api2/node/tasks.rs: implement stop_task api
2019-04-10 13:55:05 +02:00
98a181f055
src/server/worker_task.rs: log message when we call request_abort()
2019-04-10 13:54:28 +02:00
2b6a3e1396
src/tools/procfs.rs: implement read_cpuinfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:11:11 +02:00
321070b4fa
src/server/worker_task.rs: implement abort_worker (via command_socket)
2019-04-10 12:42:24 +02:00
cfb2d3c17c
src/server/command_socket.rs: correctly handle/spawn handle parallel connections
2019-04-10 11:05:00 +02:00
9b002cbc5f
src/server/worker_task.rs: use abstract socket
2019-04-10 09:03:17 +02:00
5f550fd99f
start hyper server using with_graceful_shutdown()
...
Without, hyper keeps some futures running, and the server does not
correctly shutdown.
2019-04-10 08:24:32 +02:00
7f2078de59
src/tools/procfs.rs: avoid compiler warning
2019-04-10 06:40:29 +02:00
1f6fbf1c57
src/tools/procfs.rs: implement read_meminfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-09 15:21:49 +02:00
c360bd73d3
src/api2/node/tasks.rs: fix task status api
2019-04-09 15:12:20 +02:00
0bfd87bc5e
src/server/worker_task.rs: speedup upid_read_status()
2019-04-09 15:11:48 +02:00
d8d40dd0c2
src/api2/node.rs: add node parameter
2019-04-09 14:43:30 +02:00
7ae0bf52dd
src/server/command_socket.rs: code cleanup - fix error message
2019-04-09 12:47:42 +02:00
d607b8861b
src/server/worker_task.rs: implement task control socket
2019-04-09 12:15:06 +02:00
e201753629
src/server/command_socket.rs: implement auto_remove flag
...
Remove the socket file on close.
2019-04-09 11:47:23 +02:00
634132fe10
src/server/upid.rs: moved code into separate file
2019-04-08 18:43:26 +02:00
78a39e0517
src/server/command_socket.rs: simple command socket
2019-04-08 17:59:39 +02:00
e3f41f2165
src/tools/daemon.rs: use new ServerState handler
2019-04-08 14:00:23 +02:00
7a630df709
implement server state/signal handling, depend on tokio-signal
2019-04-08 13:59:07 +02:00
5a12c0e2fb
src/api2/node/tasks.rs: start upid API
2019-04-07 14:36:57 +02:00
b75b968147
src/server/worker_task.rs: implement is_active()
2019-04-07 14:36:25 +02:00
4494d07819
src/server/worker_task.rs: make log_path a method on UPID
2019-04-07 13:51:46 +02:00
d2a2e02bb7
src/api2/node/tasks.rs: implement userfilter
2019-04-07 13:17:19 +02:00
3c3bee2e32
src/api2/node/tasks.rs: correctly handle start parameter
2019-04-07 12:41:24 +02:00
063ca5be77
src/api2/node/tasks.rs: implement list_tasks
2019-04-07 12:18:58 +02:00
d4b59ae0b8
src/backup/datastore.rs: pass WorkerTask to GC
...
So that we can log to the task log file.
2019-04-06 17:57:38 +02:00
35950380a9
src/server/worker_task.rs: carefully handle file permissions
2019-04-06 17:53:12 +02:00
1619a72063
src/tools.rs: implement create_dir_chown
...
Combinded mkdir with chown.
2019-04-06 16:50:23 +02:00
eea8131952
src/tools.rs: implement file_set_contents_full()
2019-04-06 13:53:43 +02:00
0f778e0652
src/api2/admin/datastore.rs: use new WorkerTask::new_thread()
2019-04-06 11:27:23 +02:00
244abab769
src/tools/file_logger.rs: fix test
2019-04-06 11:24:37 +02:00
660c684640
src/server/worker_task.rs: return upid_str
2019-04-06 11:23:53 +02:00
93aebb38bc
src/server/worker_task.rs: implement read_task_list()
2019-04-06 10:17:11 +02:00
4a36a2c999
src/server/worker_task.rs: improve UPID format description
2019-04-06 09:36:50 +02:00
882594c5e9
src/server.rs: improve crate layout
2019-04-06 09:17:25 +02:00
4b01c983f0
src/server/worker_task.rs: implement persistent task list
2019-04-05 12:37:35 +02:00
a2474d328f
src/tools/procfs.rs: implement read_proc_uptime and read_proc_uptime_ticks
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-05 06:22:21 +02:00
08daa81af6
procfs: implement check_process_running and check_process_running_pstart
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-04 13:54:17 +02:00
9a0d0ff774
server/worker_task.rs: implement UPID parser
2019-04-04 13:49:35 +02:00
49cf9f3d0b
src/tools.rs: file_get_json() - add new default parameter
2019-04-04 12:24:18 +02:00
79ad7b7ec2
src/tools/procfs.rs: split regex into several lines
2019-04-04 09:25:19 +02:00
479f6e40ec
src/server/worker_task.rs: implement task registry
2019-04-04 08:58:39 +02:00
52ddc2522d
src/tools/procfs.rs: use libc::pid_t
2019-04-04 08:05:43 +02:00
d3d7ab1273
src/tools/file_logger.rs: new - accept AsRef<Path>
2019-04-03 14:13:33 +02:00
3c2012f97a
src/tools/procfs.rs: start helper tools to read procfs
2019-04-03 13:39:20 +02:00
3489936ebb
src/tools/file_logger.rs: change timestamp format to rfc3339
2019-04-03 08:58:43 +02:00
92da93b245
abort GC on server shutdown
2019-04-01 12:13:02 +02:00
9136f857cc
add global var to indicate server shutdown requests
2019-04-01 12:05:11 +02:00
3e6a7deeaa
src/api2/admin/datastore.rs: start GC in a separate thread
2019-04-01 12:02:46 +02:00
ad281d1bd9
www/DataStoreConfig.js: add button to start GC
2019-04-01 08:08:34 +02:00
49d123ee3a
src/server/rest.rs: use formatter to encode errors
2019-04-01 08:04:12 +02:00
7d4ef127cb
src/server/rest.rs: generate csrf token if we have a valid ticket
...
This is important if the user reloads the browser page.
2019-04-01 07:52:30 +02:00
11861a482d
src/backup/chunk_store.rs: fix GC
...
Added option to get oldest_writer timestamp from ProcessLocker.
2019-03-31 17:21:36 +02:00
d85987aeeb
fix last commit: the filename var was not ment to be removed, sorry
2019-03-31 16:16:14 +02:00
15a77c4c2e
src/backup/chunk_store.rs: avoid create/unlink race
2019-03-31 10:03:01 +02:00
c374f05499
src/backup.rs: improve GC problem description
2019-03-31 09:44:35 +02:00
c8ec450e37
src/backup.rs: start explaining different GC algorithm
2019-03-30 17:21:40 +01:00
8a4757342c
src/backup.rs: describe the garbage collection problem
2019-03-30 16:26:52 +01:00
10eea49d81
src/backup/fixed_index.rs: store reference to file
...
Keep it open and locked while index is in use.
2019-03-27 10:17:03 +01:00
c597a92c81
index file readers: aquire shared lock
2019-03-27 10:11:14 +01:00
0465218953
src/backup.rs: add documentation about ChunkStore locking
2019-03-22 10:14:50 +01:00
43b1303398
datastore: use new ProcessLocker
...
To make sure only one process runs garbage collection while having active writers.
2019-03-22 09:42:15 +01:00
abfc001f25
src/tools/process_locker.rs: new() - use generic path parameter
2019-03-22 09:41:32 +01:00
a650f50301
src/tools/process_locker.rs: implement inter-process reader-writer locks
2019-03-22 08:05:51 +01:00
eb90c9e372
tools: style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-20 09:57:13 +01:00
e04d1bfadc
Cleanup of if statement with unwraps by a more compact match statement
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-03-20 09:49:18 +01:00
8346f0d59b
src/server/rest.rs: correctly extract content type
2019-03-19 12:50:15 +01:00
a690ecacf0
daemon: simplify daemon creation
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 12:12:54 +01:00
e4311382d9
tools: daemon: rename some structs
...
Reloadable resources are now 'Reloadable' instead of
'ReexecContinue'.
The struct handling the reload is a 'Reloader', not a
'ReexecStore'.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 11:09:46 +01:00
5e7bc50af1
add reload support to api daemon
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
a2ca713701
add reload support to proxy
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
af70c1811b
tools: implement ReexecContinue for tokio's TcpListener
...
This is the only thing we currently need to keep alive for
reloads.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
4422ba2c7f
tools: daemon: add a default signalfd helper
...
Proxy and daemon for now just want to handle reload via
`SIGHUP`, so provide a helper creating the signalfd stream
doing that - this is simply a filtered stream which passes
the remaining signals through, so it can be used exactly
like the signalfd stream could before to add more signals.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
dce94d0e09
tools: add daemon helpers
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
ff7049d481
tools: add fd_change_cloexec helper
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
1c042cdc6c
proxy: listener error handling fixup
...
If the listening socket goes into some error state we'll get
std::io::Errors rather than higher level errors from the
native_tls::TlsAcceptor, those are usually fatal. (Ran into
this after performing a shutdown() on the file descriptor,
after which the future just endlessly loops in accept().)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
60864b1b56
src/pxar.rs: improve docu
2019-03-18 12:27:30 +01:00
515688d1f1
avoid compiler warnings
2019-03-18 10:00:58 +01:00
ba10f2b0d4
src/bin/dynamic-files.rs: test case for file shrink during backup
2019-03-16 14:22:14 +01:00
af572aaa4c
pxar: implement hardlinks
...
So we are no longer compatible with catar ...
2019-03-16 11:02:12 +01:00
812dd935ce
src/pxar/encoder.rs: detect hardlinks
2019-03-15 12:19:51 +01:00
896e803feb
src/pxar/sequential_decoder.rs: dump_entry - fix FIFO/SOCKET handling
2019-03-15 12:15:38 +01:00
7dfa17c7cc
src/pxar/binary_search_tree.rs: fix test
2019-03-15 11:34:31 +01:00
9307279faf
cleanup: remove dead code
2019-03-15 10:18:28 +01:00
46cfe7ab7e
src/pxar/decoder.rs: implement seekable decoder
2019-03-15 09:36:05 +01:00
0e21b87be3
renamed: src/pxar/decoder.rs -> src/pxar/sequential_decoder.rs
2019-03-15 08:36:02 +01:00
3dbfe5b142
simplify pxar module structure
2019-03-15 08:24:32 +01:00
1ef46b81d3
src/bin/pxar.rs: implement extract
2019-03-15 08:03:44 +01:00
02c93163cf
src/pxar/decoder.rs: correctly call callback()
2019-03-15 08:02:04 +01:00
5defa71bf9
rename PxarBackupWriter into PxarDecodeWriter
2019-03-15 07:20:22 +01:00
b005ed12d5
src/client/http_client.rs: avoid compiler warning
2019-03-15 07:16:26 +01:00
9e26abf16b
src/cli/command.rs: fix iterator usage
2019-03-15 07:12:40 +01:00
f98746ba6c
src/pxar/decoder.rs: improve docu
2019-03-15 07:12:11 +01:00
0b2d7fdc81
src/pxar/decoder.rs: avoid compiler warnings
2019-03-14 17:45:47 +01:00
37940aa1f9
src/bin/pxar.rs: implement dump/list
2019-03-14 17:43:11 +01:00
40360fde6e
src/pxar/decoder.rs: implement dump_archive
...
use this to debug archive problems ...
2019-03-14 13:10:27 +01:00
8968258b66
rename catar into pxar
...
To avoid confusion with the casync implementation.
2019-03-14 10:54:09 +01:00
7c4dd94670
rename catar into pxar
...
Seems impossible to stay compatible with the original catar specification,
so we change the name to avoid confusions.
2019-03-14 10:14:32 +01:00
806500cd09
src/tools.rs: improve file name completion
...
Use current working directory by default.
2019-03-14 08:17:35 +01:00
08dc340a12
src/bin/proxmox-backup-client.rs: complete archive names
2019-03-14 08:09:35 +01:00
6f90a6a764
protocol: cleanup finish_backup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-13 14:16:17 +01:00
b238851832
src/bin/proxmox-backup-client.rs: implement completion for snapshots
2019-03-13 13:31:29 +01:00
4981134708
src/bin/proxmox-backup-client.rs: implement chunk-size completion
2019-03-13 12:26:01 +01:00
98f0b97293
src/bin/proxmox-backup-client.rs: cleanup
2019-03-13 12:17:39 +01:00
024f11bb55
src/bin/proxmox-backup-client.rs: implement completion for backup groups
2019-03-13 11:56:37 +01:00
edd3c8c605
src/client/backup_repo.rs: implement FromStr trait
2019-03-13 10:09:39 +01:00
874acb7039
src/client/backup_repo.rs: implement Display trait
2019-03-13 09:57:36 +01:00
d0a03d40ce
src/bin/proxmox-backup-client.rs: implement repository completions
2019-03-13 09:47:12 +01:00
bff110308b
src/bin/proxmox-backup-client.rs: fix BACKUPSPEC_REGEX - correctly specify non-capture group
...
And improve bash completion.
2019-03-13 07:23:59 +01:00
1a71509aad
src/cli/command.rs: fix completion - never display argument parameters as option
2019-03-12 14:53:41 +01:00
496a67846f
src/cli/command.rs: pass parsed parameters to completion function
2019-03-12 14:39:51 +01:00
b5fa28251c
src/cli/getopts.rs: factor out reusable code
2019-03-12 14:38:47 +01:00
e39efdbd33
src/cli/command.rs: use a HashMap to store fixed param as key/value pairs
2019-03-12 12:54:16 +01:00
f4e4d58327
src/cli/command.rs: improve completions
2019-03-12 12:07:45 +01:00
4ba7b6099c
signalfd cleanup
...
- add comment about the boxed file descriptor (and others)
- reexport SigSet and Signal for convenience
- remove Result wrapper (only used once)
- enforce drop order of the PollEvented2 struct with respect
to the boxed fd
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-12 11:07:09 +01:00
f54c19989c
tools: add async signalfd handler
...
This provides a Stream<Item = siginfo> via nix' signalfd,
by wrapping it in tokio's PollEvented2 struct to allow
polling via tokio's event loop.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-12 08:04:28 +01:00
d5c34d98c0
src/catar/decoder.rs: simplify public restore API
2019-03-11 14:31:01 +01:00
8e39232acc
src/bin/proxmox-backup-client.rs: strip .didx file extensions
2019-03-11 10:54:02 +01:00
74cdb52107
src/bin/proxmox-backup-client.rs: include file extension in backupspec
2019-03-11 10:27:28 +01:00
fc2bf37ead
src/catar/decoder.rs: cleanup style
2019-03-11 08:14:28 +01:00
656b23e107
src/catar/decoder.rs: dir_mkdirat - add create_new flag
2019-03-11 07:27:45 +01:00
e9c9409a99
src/catar/decoder.rs: implement real sequential decoder (no Seek)
...
I copied the old code to src/catar/inspector.rs. Will modify
that later to implement a random access decoder ...
2019-03-08 16:55:54 +01:00
eed6db3923
src/bin/proxmox-backup-client.rs: impl option all-file-systems
2019-03-08 09:36:40 +01:00
06178f13fa
move chunker into proxmox_protocol
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-08 09:29:47 +01:00
e3c30c50a6
src/bin/catar.rs: implement all-file-systems option
2019-03-08 09:28:12 +01:00
65092b1e27
src/catar/encoder.rs: log skipped mount points
2019-03-08 09:25:00 +01:00
6b64d25d64
src/catar/encoder.rs: remove debug output
2019-03-08 08:23:07 +01:00
2689810c92
src/bin/catar.rs: implement verbose flags
2019-03-08 08:20:56 +01:00
79c281fdac
src/catar/encoder.rs: use simply flag "all_file_systems"
2019-03-08 08:14:26 +01:00
25e77d3877
src/cli/command.rs: print_bash_completion - avoid crash
2019-03-07 12:14:26 +01:00
141f062e08
src/backup/chunk_store.rs: use zstd compression insteadf of lz4
...
Provides better compressionm rate, and is still fast.
2019-03-07 11:42:59 +01:00
d2690f74bb
src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end
2019-03-07 10:40:01 +01:00
2ae7d196b9
src/bin/proxmox-backup-client.rs: correctly strip file extension
2019-03-06 11:18:46 +01:00
9f91249351
src/bin/proxmox-backup-client.rs: impl restore
2019-03-06 11:02:08 +01:00
01a1342364
src/api2/admin/datastore.rs: impl list_snapshot_files
2019-03-06 11:02:08 +01:00
58e99e138e
src/backup/backup_info.rs: new helper to list files
2019-03-06 11:02:08 +01:00
dd79265a8a
src/api2/admin/datastore/catar.rs: use archive_name directly
2019-03-06 11:02:08 +01:00
a6b7551311
src/client/http_client.rs: impl download
2019-03-06 11:02:08 +01:00
6716f30bbc
add protocol test client
...
This uses futures for everything which is mostly useful as a
test to see if the protocol crate's non-blocking I/O support
can handle it...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:26:12 +01:00
e2d007f76e
api2/admin/datastore: add a backup protocol test api path
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:26:12 +01:00
e3062f87b1
fixed_index: split add_chunk/add_digest
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
c3bb97e59c
fixed_index: impl Send for FixedIndexWriter
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
3b7ade9e78
backup/datastore: expose insert_chunk method
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
91ebea8262
router: add upgrade method
...
This is just an Async GET method, but semantically
different, so it gets its own router method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
80822b95b1
src/bin/proxmox-backup-client.rs: sort list results
2019-03-06 07:20:51 +01:00
fba30411fb
src/bin/proxmox-backup-client.rs: allow to set backup-id
2019-03-06 07:02:52 +01:00
21ea0158b4
src/client/http_client.rs: remove debug statement
2019-03-06 06:45:51 +01:00
ba3a60b2a4
src/client/http_client.rs: store/load ticket in xdg runtime dir, depend on crate xdg
2019-03-05 12:56:21 +01:00
f8f94534d4
src/api2/access.rs: authenticate_user() - add ticket login
2019-03-05 12:56:21 +01:00
e5662b04e9
src/tools/ticket.rs: define const TICKET_LIFETIME
2019-03-05 12:56:21 +01:00
53cafb5925
src/tools.rs: impl file_get_json()
2019-03-05 12:56:21 +01:00
391d310741
backup_info: BackupDir shortcut
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 10:15:06 +01:00
93b49ce38a
backup_info: use two independent Into<String> types
...
there's no reason to force them to be the same input type
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 09:13:08 +01:00
3ec3ec3f9a
src/bin/proxmox-backup-client.rs: print backup duration
2019-03-05 08:11:40 +01:00
d57474e055
src/backup/backup_info.rs: improve docs
2019-03-05 07:28:13 +01:00
b3483782d1
src/backup/backup_info.rs: move code into separate file
...
Also changed create_backup_dir() parameters - uses &BackupDir now.
2019-03-05 07:18:12 +01:00
2b01a22507
src/backup/datastore.rs: add helper to sort backup lists
2019-03-04 18:20:57 +01:00
875fb1c01a
src/backup/datastore.rs: use unix epoch to create DateTime
...
To make sure that we have a timestamp without nanosecond.
2019-03-04 17:58:22 +01:00
38f8815925
src/backup/datastore.rs: use new() function to create structs
2019-03-04 17:44:27 +01:00
9b492eb256
src/backup/datastore.rs: protect BackupDir fields, impl new()
2019-03-04 13:51:36 +01:00
1e9a94e579
src/backup/datastore.rs: protect BackupGroup fields, impl new()
2019-03-04 13:38:23 +01:00
cdebd467e6
src/bin/proxmox-backup-client.rs: simply log to stdout
2019-03-04 09:38:34 +01:00
f72dfd2d10
src/backup/datastore.rs: make sure timestamp does not contains nanoseconds
...
Because we want simple, readable file names.
2019-03-04 09:37:10 +01:00
219ef0e6ca
src/catar/encoder.rs: add verbose flags
2019-03-04 08:01:09 +01:00
e0b8ce6424
src/catar/encoder.rs: skip RAMFS
...
unless explicitly selected.
2019-03-04 06:48:47 +01:00
597a92039e
src/bin/proxmox-backup-client.rs: rename create into backup
2019-03-03 11:46:53 +01:00
6f62c9240a
src/api2/admin/datastore.rs: imp delete_snapshot
2019-03-03 11:29:00 +01:00
af926291e0
avoid compiler warning
2019-03-03 09:52:06 +01:00
e77a02ed87
catar api: cleanup parameter names
2019-03-03 09:48:32 +01:00
421c9ba1d9
src/bin/proxmox-backup-client.rs: use new json_object_to_query
2019-03-03 09:44:25 +01:00
f5f13ebc5a
src/tools.rs: new helper json_object_to_query
2019-03-03 08:51:37 +01:00
49dc074007
parameter name cleanups (use '-' instead of '_')
2019-03-02 16:34:10 +01:00
ad20d19877
src/api2/admin/datastore.rs: rename get_group_list to list_groups, cleanups
2019-03-02 16:28:36 +01:00
184f17afad
src/api2/admin/datastore.rs: impl list_snapshots
2019-03-02 16:20:50 +01:00
af53186e6a
src/tools.rs: add simply string join
2019-03-02 16:15:45 +01:00
812c6f8716
src/api2/admin/datastore.rs: impl list_backup_groups
2019-03-02 11:29:05 +01:00
e909522f06
src/backup/datastore.rs: use timestamp with zone offset
2019-03-01 17:36:40 +01:00
dc4c09faae
src/backup/datastore.rs: split BackupDir
2019-03-01 16:26:35 +01:00
38b0dfa511
src/backup/datastore.rs: split BackupInfo
2019-03-01 16:26:35 +01:00
dcd15d10bb
backup/index: remove IndexIterator
...
removed the need for it, to avoid copies
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-01 13:29:36 +01:00
17d6979afd
src/bin/proxmox-backup-client.rs: generate a log file
2019-03-01 09:35:41 +01:00
3b151414f8
src/tools/file_logger.rs: class to log into files
2019-03-01 09:34:29 +01:00
5a2892f0ae
src/bin/proxmox-backup-client.rs: use chrono::DateTime
2019-03-01 06:59:32 +01:00
51144821f9
src/bin/proxmox-backup-client.rs: login before starting backup
2019-03-01 06:48:41 +01:00
541b908ea4
src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch
2019-02-28 17:03:48 +01:00
b55cee9227
src/cli/command.rs: generate_usage_str() - fix option separator (newline)
2019-02-28 16:20:00 +01:00
c4c5174000
src/cli/command.rs: get_schema_type_text() - display min/max
2019-02-28 16:11:49 +01:00
df67070b1c
client/backup_repo: allow ip addresses for BackupRepository
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
5de2bced2d
datastore: add generic open_index
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
7bc1d72778
add IndexFile trait
...
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.
Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
5be4065baf
make FixedIndexReader Send and update comments
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
fded74d0c7
style: BufferedReader -> BufferedRead
...
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
8f57971714
src/api2/admin/datastore.rs: implement prune
2019-02-28 12:51:27 +01:00
83b7db0209
src/api2/admin/datastore.rs: start prune api
...
just a dummy for now.
2019-02-27 16:53:17 +01:00
0ffbccce5e
rc/server/rest.rs: allow to pass parameters as application/json
2019-02-27 12:37:53 +01:00
f98ea63de2
src/bin/proxmox-backup-client.rs: fix bash completion
2019-02-27 12:37:53 +01:00
8731e40a7f
backup/datastore: create_backup_dir: show if path existed
...
To enable asserting the creation of a new backup rather than
adding to an existing one.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-27 10:15:36 +01:00
25f1650b71
src/bin/proxmox-backup-client.rs: minor code cleanup
2019-02-27 07:21:57 +01:00
9bc3ddb822
src/cli/command.rs: improve generate_usage_str
...
correctly format positional array arguments ({<name>}).
2019-02-26 12:40:51 +01:00
ae0be2dda9
src/bin/proxmox-backup-client.rs: allow to upload more that one directory
2019-02-26 12:27:28 +01:00
a477d6889a
src/client/http_client.rs: login once, store and reuse ticket/token
2019-02-26 11:59:10 +01:00
d02c2ed701
src/cli/getopts.rs: cleanup, avoid compiler warning
2019-02-26 11:55:06 +01:00
f8dfbb45c0
src/tools.rs: new helper required_array_param()
2019-02-26 08:48:17 +01:00
aaba6faace
src/cli/getopts.rs: support positional array arguments
2019-02-26 08:47:34 +01:00
f33fa27307
src/cli/command.rs: improve error message
2019-02-26 08:46:36 +01:00
92a173feb0
src/api_schema/schema.rs: improve ParameterError display function
2019-02-26 07:11:24 +01:00
f661f37427
backup/dynamic_index: split out add_chunk
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:50:45 +01:00
9ac6ec868a
backup/chunk_store: split insert_chunk
...
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
3a50ddd0c9
backup/chunk_store: mark ChunkIterator as fused
...
Ranges are FusedIterators and it can be useful to propagate
that property to users.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
eff25ecaf2
backup/chunk_store: make percentage output optional
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
d59397e60e
backup/datastore: expose chunk_store.get_chunk_iterator
...
will be used by the protocol handler to iterate over
existing chunks
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
8eb1203bda
src/backup/chunk_stat.rs: log write speed
2019-02-25 13:21:00 +01:00
7e3365554e
src/backup/chunk_stat.rs: new struct to track chunk statistics
2019-02-25 12:52:10 +01:00
798f7fa065
src/backup/chunk_store.rs: return info about compressed chunk size
...
So that we can generate better statistics ..
2019-02-25 11:36:05 +01:00
78216a5ab1
src/backup/chunk_store.rs: use lz4 for compression
...
And depend on lz4 lib now.
2019-02-25 10:00:28 +01:00
793b0f4d77
src/cli/command.rs: improve help command completion
2019-02-23 18:07:32 +01:00
6949d91564
src/cli/command.rs: cleanup, avoid array manipulation
2019-02-23 17:50:33 +01:00
9d78d57960
src/cli/command.rs: implement bash completion for help
2019-02-23 17:38:10 +01:00
698d9d4402
src/cli/command.rs: add help command
2019-02-23 15:10:48 +01:00
8b6dd2240d
src/cli/command.rs::allow to generate ReST docu
2019-02-23 11:29:18 +01:00
2f3f2bb77f
src/cli/command.rs: improve usage imformation
2019-02-22 17:40:37 +01:00
f252ca654c
src/backup/chunker.rs: simplify break test
...
To make it a little bit faster ...
2019-02-22 13:02:11 +01:00
6adb40eefc
src/cli/command.rs: start implementing generate_usage_str()
2019-02-22 12:02:59 +01:00
33256db6dd
src/cli/command.rs: simplify cli handler
2019-02-22 11:15:10 +01:00
61834c1cbf
src/backup/chunker.rs: another try to speedup the chunker
2019-02-22 11:15:10 +01:00
532a69eb04
src/cli/getopts.rs: improve docs
2019-02-21 09:36:52 +01:00
4de0e142a0
cleanup cli library structure
2019-02-21 09:08:03 +01:00
56458d9764
http_client: on a tty, read password if no env var is set
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-20 15:03:34 +01:00
c9b296f117
tools: add tty helper module
...
with read_password() and stdin_isatty() functions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-20 14:43:07 +01:00
8cc0d6aff3
src/bin/proxmox-backup-client.rs: implement garbage-collection sub-command
2019-02-20 14:10:45 +01:00
81da38c143
src/client/http_client.rs: implement post
...
without parameters for now...
2019-02-20 14:09:55 +01:00
8c75372b79
src/backup/datastore.rs: list_backups() - include list of index files
2019-02-20 13:37:44 +01:00
14691fc1f3
src/bin/test_chunk_size.rs: avoid compilöer warnings, exit after 1GB
2019-02-20 12:05:28 +01:00
1604ec2654
src/bin/test_chunk_size.rs: another chunker test - gather some stats
2019-02-20 11:50:10 +01:00
fa4ebf0fce
src/backup/chunker.rs: simplify code
2019-02-20 11:17:19 +01:00
1b9c959c32
src/backup/chunker.rs: change window size back to 48
...
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
2019-02-20 11:01:09 +01:00
247cdbce72
src/api2/admin/datastore/catar.rs: allow to configure chunk-size
2019-02-19 15:19:12 +01:00
c584aa21fd
src/backup/chunker.rs: start() - correctly store hash in self.h
2019-02-19 12:40:12 +01:00
cd1598302a
src/backup/chunker.rs: use 64 byte chunker window size
...
This results in faster code.
2019-02-19 12:27:37 +01:00
9a3e869cf5
src/backup/chunker.rs: simplify buzzhash start code
2019-02-19 12:17:17 +01:00
1c7a88aeff
UploadCaTar: currectly write all data using write_all()
2019-02-18 18:31:02 +01:00
4d2881dd4c
src/backup/chunker.rs: correctly compute window index 'idx'
...
And add a simple test.
2019-02-18 16:59:23 +01:00
a4a5c78c01
src/client/http_client.rs: send csrf token
2019-02-18 16:59:23 +01:00
fc7f03523c
cleanup
...
Error::from is already a function taking 1 parameter,
there's no need to wrap it with `|e| Error::from(e)`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-18 13:21:27 +01:00
8fdef1a81c
fix typo: confi{g}dir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-18 13:11:42 +01:00
Christian Ebner
Dietmar Maurer
Wolfgang Bumiller