Commit Graph

1428 Commits

Author SHA1 Message Date
Dietmar Maurer dc9775d1b0 src/bin/h2s-client.rs: increase tokio tcp stream read buffer size
This improves download speed by factor 10.
2019-06-29 18:09:50 +02:00
Dietmar Maurer 34f7131160 src/bin/h2s-client.rs: avoid hyper, use h2 directly
But performance is still bad.
2019-06-29 15:58:18 +02:00
Dietmar Maurer 5ed2277f0e src/bin/h2s-server.rs: test to reproduce slow h2 download
Simplified code, use with h2s-client.rs.
2019-06-29 14:57:54 +02:00
Dietmar Maurer 17243003b3 src/bin/download-speed.rs: h2 download speedtest for our server
I get about 30MB/s (much too slow)
2019-06-29 13:43:10 +02:00
Dietmar Maurer a2b29b68d4 src/client/http_client.rs: use maximal h2 window sizes 2019-06-29 11:05:36 +02:00
Dietmar Maurer 872debdefa src/bin/h2client.rs: tune h2 client parameters to get reasonable speed
We now get about 2.5GB/s (not really good, but better)
2019-06-29 10:41:24 +02:00
Dietmar Maurer 38294bcc56 src/client.rs: make module put_to_stream public 2019-06-29 10:40:43 +02:00
Dietmar Maurer fded1f3154 add code to test H2 speed
I currently get about 1MB/s, which is ways too slow (must be a bug)?
2019-06-29 10:08:12 +02:00
Dietmar Maurer d48a9955a5 src/backup/dynamic_index.rs: introduce ReadChunk trait 2019-06-28 16:35:00 +02:00
Dietmar Maurer 09d7dc5024 src/api2/reader.rs: add download chunk api 2019-06-28 16:27:01 +02:00
Dietmar Maurer fcf5dea56d src/client/http_client.rs: increase h2 window size, set tcp nodelay option
To increase h2 download speed.
2019-06-28 16:00:58 +02:00
Dietmar Maurer 81a6ce6fde src/backup/chunk_store.rs: new method chunk_path()
Returns the absolute path.
2019-06-28 15:48:09 +02:00
Dietmar Maurer fcfb84fedf file download: avoid unnecessary copy 2019-06-28 07:07:52 +02:00
Dietmar Maurer 984a7c3502 src/client/http_client.rs - h2 download: implement flow control 2019-06-28 07:02:43 +02:00
Dietmar Maurer dd066d28e2 src/api2/reader.rs: implement backup reader protocol 2019-06-27 09:01:41 +02:00
Dietmar Maurer 42a87f7b96 src/server/h2service.rs: implement generic h2 service 2019-06-26 17:38:33 +02:00
Dietmar Maurer fb0470837b src/client/http_client.rs: split out code to start new h2 connections 2019-06-26 12:09:18 +02:00
Dietmar Maurer b208da8393 src/backup/data_{chunk,blob}.rs: add verify_crc method 2019-06-26 09:54:25 +02:00
Dietmar Maurer fef44d4f78 src/bin/proxmox-backup-client.rs: decode blobs on download
No more need for that clumsy blob cli (removed).
2019-06-26 09:18:59 +02:00
Dietmar Maurer c2b945341c src/client/http_client.rs - download: use generic Write type, return writer.
Make it possible to write int Vec<u8>.
2019-06-26 09:17:13 +02:00
Dietmar Maurer bb8231409e src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json
Use the KeyConfig serialization with kdf = None.
2019-06-26 07:32:34 +02:00
Dietmar Maurer 3031e44c58 src/bin/proxmox-backup-client.rs: remove stale download file, start blob cli 2019-06-25 13:00:37 +02:00
Dietmar Maurer f14a8c9a85 src/api2/admin/datastore.rs: use correct path for download 2019-06-25 12:59:36 +02:00
Dietmar Maurer 81820b0d4d src/client/http_client.rs - download: fix error handling 2019-06-25 12:43:55 +02:00
Dietmar Maurer 45db6f89e1 src/bin/proxmox-backup-client.rs: implement download command 2019-06-25 11:17:24 +02:00
Dietmar Maurer 9e47c0a56e src/api2/admin/datastore.rs: implement generic download method
We simply allow to download raw binary data.
2019-06-25 10:16:59 +02:00
Dietmar Maurer da7d67213e src/api2/admin/datastore/pxar.rs: remove pxar upload api
Upload needs to be done using the backup protocol.
2019-06-25 09:45:35 +02:00
Dietmar Maurer dd8e744f9a src/api2/admin/datastore.rs: use a worker log log prune actions 2019-06-25 08:12:25 +02:00
Dietmar Maurer 6899dbfb47 src/bin/proxmox-backup-client.rs: strip .blob extension 2019-06-25 07:11:02 +02:00
Dietmar Maurer c6d203bbd3 src/backup/backup_info.rs: also list .blob files 2019-06-25 07:07:45 +02:00
Dietmar Maurer 045e8a568d src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key 2019-06-25 07:00:07 +02:00
Dietmar Maurer 79bc7345e8 src/cli/command.rs: fix ReST output format 2019-06-25 06:19:51 +02:00
Dietmar Maurer 9f46c7de4b src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup 2019-06-24 13:56:37 +02:00
Christian Ebner cd7dc87903 pxar: impl .pxarexclude parsing and exclude matching
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.

Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-24 10:26:20 +02:00
Dietmar Maurer 46bd880041 src/api2/backup/environment.rs: new helper add_blob() 2019-06-24 09:35:37 +02:00
Dietmar Maurer cb08ac3efe src/api2/backup.rs: replace upload_config with upload_blob 2019-06-23 11:44:30 +02:00
Dietmar Maurer a38c5d4d12 src/backup/data_blob.rs: implement from_raw() 2019-06-23 09:35:44 +02:00
Dietmar Maurer 6f083b7a92 src/api2/backup/upload_chunk.rs: compute crc on upload 2019-06-23 09:04:42 +02:00
Dietmar Maurer ba01828d38 try to use use proxmox::tools::io::ops::ReadExtOps 2019-06-22 16:29:10 +02:00
Dietmar Maurer 27042ce637 src/backup/file_formats.rs: improve docs 2019-06-22 15:45:36 +02:00
Dietmar Maurer 9f83e0f7ee src/backup/crypt_config.rs: simplify decode interface 2019-06-22 13:24:29 +02:00
Dietmar Maurer 0066c6d972 src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead 2019-06-22 13:02:53 +02:00
Dietmar Maurer ee8a7e8080 src/backup/crypt_config.rs: implement encrypt_to (output to writer) 2019-06-22 12:25:04 +02:00
Dietmar Maurer 991abfa8b4 src/backup/file_formats.rs: split out file format data 2019-06-22 09:12:25 +02:00
Dietmar Maurer b7f4f27d6c add crc field for binary blobs formats 2019-06-21 18:49:04 +02:00
Dietmar Maurer a7f67a9a9c depend on crc32fast
We can use this to compute fast checksums to test file integrity (not crypto safe).
2019-06-21 15:58:00 +02:00
Dietmar Maurer 3025b3a53c src/backup/data_blob.rs: new file format for binary blobs 2019-06-21 11:32:07 +02:00
Dietmar Maurer 077a8cae63 src/backup/crypt_config.rs: make code more flexible
Allow to pass magic numbers as parameters.
2019-06-21 10:50:48 +02:00
Dietmar Maurer c38266c18d renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs 2019-06-21 09:51:18 +02:00
Dietmar Maurer 3ea8bfc9c0 src/bin/proxmox-backup-client.rs: add import-master-pubkey command 2019-06-21 07:34:17 +02:00
Dietmar Maurer 37c5a17505 src/bin/proxmox-backup-client.rs: helper to generate RSA master key 2019-06-21 07:12:19 +02:00
Dietmar Maurer ab44acff57 src/backup/key_derivation.rs: add modified field to key file 2019-06-20 07:58:33 +02:00
Dietmar Maurer 6d0983dbe1 src/bin/proxmox-backup-client.rs: add keyfile parameter for backup 2019-06-19 17:16:41 +02:00
Dietmar Maurer 181f097af9 src/bin/proxmox-backup-client.rs - key API: pass kdf parameter
Allow to store keys without password.
2019-06-18 18:21:01 +02:00
Dietmar Maurer 9b06db4563 src/bin/proxmox-backup-client.rs: use file system path to reference keys 2019-06-18 17:38:50 +02:00
Dietmar Maurer 091540298e src/bin/proxmox-backup-client.rs - create_key: always read password from tty 2019-06-18 17:27:52 +02:00
Dietmar Maurer 826f309bf5 src/backup/key_derivation.rs: move kdf code into separate file 2019-06-18 11:17:22 +02:00
Dietmar Maurer 11515438cc Cargo.toml: use serde feature derive 2019-06-18 06:23:25 +02:00
Dietmar Maurer ac71623483 src/bin/proxmox-backup-client.rs: implement "key change-passphrase" 2019-06-17 13:10:00 +02:00
Dietmar Maurer f2401311b0 src/bin/proxmox-backup-client.rs: start key management 2019-06-17 10:33:24 +02:00
Dietmar Maurer a7dd483097 src/backup/*_index.rs: used generated magic numbers 2019-06-14 14:58:37 +02:00
Dietmar Maurer bffd40d6b7 src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools 2019-06-14 11:40:04 +02:00
Dietmar Maurer 00388226e6 src/api2/backup/environment.rs: log index checksums 2019-06-14 10:36:20 +02:00
Dietmar Maurer 9335d74eba src/backup/fixed_index.rs: compute checksum over the index 2019-06-14 10:34:29 +02:00
Dietmar Maurer 16ff6b7cd8 src/backup/dynamic_index.rs: compute checksum over the index 2019-06-14 10:33:59 +02:00
Dietmar Maurer 36075475a6 src/api2/backup/environment.rs: log duplicate chunks 2019-06-14 07:12:30 +02:00
Dietmar Maurer fa148dbd11 src/api2/backup/upload_chunk.rs: verify chunk data if possible 2019-06-14 06:23:46 +02:00
Dietmar Maurer 0f37577092 src/bin/cipherbench.rs: add test for zstd compression 2019-06-13 17:24:57 +02:00
Dietmar Maurer 9399c98f82 src/bin/cipherbench.rs: cipher speed test with large blocks 2019-06-13 17:16:43 +02:00
Dietmar Maurer f640f23ac5 src/backup/data_chunk.rs: fix typo 2019-06-13 16:42:55 +02:00
Dietmar Maurer dba72d5016 src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:56:50 +02:00
Dietmar Maurer 68ab376ae9 src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:47:54 +02:00
Dietmar Maurer f98ac774ee backup: Add support for client side encryption
first try ...
2019-06-13 11:47:23 +02:00
Dietmar Maurer 51929e4532 src/backup/data_chunk.rs - decode: make crypt_config optional 2019-06-13 11:44:38 +02:00
Dietmar Maurer 046bcb9773 src/backup/crypt_setup.rs: add comment for possible improvements 2019-06-13 07:35:59 +02:00
Dietmar Maurer bd0e3c7cfa src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs 2019-06-13 07:35:02 +02:00
Dietmar Maurer 6762db70d6 src/api2/types.rs: define CHUNK_DIGEST_SCHEMA 2019-06-13 07:30:42 +02:00
Dietmar Maurer b595cb9d2c src/backup/data_chunk.rs: Data Chunk builder
Make handling encoded chunks easier.
2019-06-12 13:37:01 +02:00
Dietmar Maurer ef27200cec src/backup/crypt_setup.rs: fix encoder for uncompressed chunks 2019-06-10 09:56:06 +02:00
Dietmar Maurer 74792b95b2 src/backup/crypt_setup.rs: allow compressed and uncompressed chunks 2019-06-10 08:27:35 +02:00
Dietmar Maurer 9e0187a203 src/backup/crypt_setup.rs: remove unused CryptData struct 2019-06-10 07:55:36 +02:00
Dietmar Maurer bec8498ae3 src/backup/crypt_setup.rs: avoid useless allocations 2019-06-09 11:44:17 +02:00
Dietmar Maurer c20b17b460 src/backup.rs: define magic numbers for chunk store file format 2019-06-08 10:34:20 +02:00
Dietmar Maurer 48b4b40b57 src/backup/crypt_setup.rs: crypto helpers 2019-06-08 09:56:51 +02:00
Wolfgang Bumiller dd5495d6dc tree-wide: use 'dyn' for all trait objects
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-07 13:13:48 +02:00
Christian Ebner e993db91cd pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:20 +02:00
Christian Ebner 1f319e766e src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:06 +02:00
Christian Ebner 9b77388f2a src/pxar/encoder.rs: minor refactoring
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:55 +02:00
Christian Ebner a4cc8eb7c5 src/pxar/encoder.rs: check if ioctl supported by filesystem
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:31 +02:00
Dietmar Maurer 4800df7ba6 src/api2/admin/datastore.rs: remove test-upload api 2019-06-07 07:52:11 +02:00
Dietmar Maurer 113898d7e9 src/api2/admin/datastore/upload.rs: remove unused upload api 2019-06-07 07:39:41 +02:00
Dietmar Maurer 4247fccb0f src/client/http_client.rs: automatically close connection in finish 2019-06-05 11:41:50 +02:00
Wolfgang Bumiller cb4426b348 make h2 client connection cancellable
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-05 09:45:59 +02:00
Dietmar Maurer 986bef16be src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1
So that we can include it in static doc strings.
2019-06-05 08:41:20 +02:00
Dietmar Maurer c9ec0956cf src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1 2019-06-05 08:12:13 +02:00
Dietmar Maurer 44c54845b3 src/tools/xattr.rs: fix test when run as root 2019-06-05 07:57:42 +02:00
Dietmar Maurer 7773ccc11f src/api2.rs: move backup api to /backup 2019-06-05 07:23:21 +02:00
Christian Ebner 97eeea3b4a src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:29:02 +02:00
Dietmar Maurer 1ce2f7cfb9 src/pxar/encoder.rs: style fix 2019-06-05 06:27:39 +02:00
Christian Ebner e7b60a16c8 pxar: impl storing/dumping/restoring of quota project ids
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:26:02 +02:00
Christian Ebner 7312ab9eef src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID
Defines the types needed to store/restore quota project ids in pxar

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:23:06 +02:00
Christian Ebner 042babe4e7 src/tools/fs.rs: impl ioctl calls to get/set fsxattr
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:22:49 +02:00
Dietmar Maurer 22e5aa1d3e src/bin/dump-backup-api.rs: helper to generate backup API docs 2019-06-04 13:12:42 +02:00
Dietmar Maurer 5e0f305142 src/api_schema/format.rs: depend on textwrap, new wrap_text helper 2019-06-04 12:42:02 +02:00
Dietmar Maurer 339ddfcbfa src/cli/command.rs: move doc generator code to src/api_schema/format.rs 2019-06-04 12:32:22 +02:00
Dietmar Maurer ef39bf95b3 src/client/http_client.rs - upload_config: improve error message 2019-06-03 10:53:09 +02:00
Dietmar Maurer ec8a9bb953 src/bin/proxmox-backup-client.rs: implement upload_config 2019-06-03 10:39:44 +02:00
Dietmar Maurer 39d6846e1e src/api2/admin/datastore/backup.rs: implement config file upload 2019-06-03 09:45:14 +02:00
Dietmar Maurer 1c0472e88f avoid compiler warnings 2019-06-03 06:43:47 +02:00
Dietmar Maurer 36898ffce6 src/backup/chunk_stream.rs: add optional chunk_size parameter 2019-05-30 13:28:24 +02:00
Dietmar Maurer 49ef316bcd src/bin/proxmox-backup-client.rs: backup_image() - pass correct chunk size 2019-05-30 12:52:12 +02:00
Dietmar Maurer 202cadc29e src/bin/proxmox-backup-client.rs: remove unused code 2019-05-30 12:47:35 +02:00
Dietmar Maurer 6af905c18f src/bin/proxmox-backup-client.rs: implement image/block-device upload 2019-05-30 12:46:01 +02:00
Dietmar Maurer eb1804c567 src/bin/proxmox-backup-client.rs: use std lib instead of nix 2019-05-30 12:19:23 +02:00
Dietmar Maurer 6e907505aa src/bin/proxmox-backup-client.rs: fix file type test 2019-05-30 12:08:34 +02:00
Dietmar Maurer 96482891ae src/api2/admin/datastore/backup/environment.rs: log upload statistics 2019-05-30 09:21:19 +02:00
Dietmar Maurer 642322b433 src/api2/admin/datastore/backup.rs: use separate api entry points for chunk upload
So that we can provide better statistics (patches follows).
2019-05-30 09:21:19 +02:00
Wolfgang Bumiller 390e83c9b9 typo fix and doc improvement
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:57:28 +02:00
Wolfgang Bumiller 7fb49397d5 doc fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:56:14 +02:00
Wolfgang Bumiller 382609b04e tools: add Cancellable, start a futures submodule
To make a future cancellable, use:
let (future, canceller) =
    crate::tools::futures::cancellable(future);

Proceed with using `future` as usual, `canceller` is
clonable and can cancel the future via the `.cancel()`
method.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Wolfgang Bumiller 0d32d71fb7 tools: add AsyncMutex::new_locked
Allows creating a pre-locked mutex, returning the mutex and
a guard.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Dietmar Maurer e6389f4e75 src/api2/admin/datastore/backup/environment.rs: do not allow empty backups (no files) 2019-05-29 10:38:57 +02:00
Dietmar Maurer 39e60bd6fa src/api2/admin/datastore/backup.rs: improve debug logging 2019-05-29 10:17:38 +02:00
Dietmar Maurer 4ae6551c78 src/api_schema/router.rs: disable debug println 2019-05-29 09:42:16 +02:00
Dietmar Maurer a42d1f55c3 src/api2/admin/datastore/backup.rs: add debug flag 2019-05-29 09:35:21 +02:00
Dietmar Maurer 976595e1a9 src/backup/dynamic_index.rs: split class DynamicIndexWriter 2019-05-29 08:49:57 +02:00
Dietmar Maurer cb0708dd46 src/backup/fixed_index.rs: remove ChunkStat from struct 2019-05-29 07:08:34 +02:00
Dietmar Maurer 01af11f340 src/backup/fixed_index.rs: improve consistency checks 2019-05-29 07:02:45 +02:00
Christian Ebner 9b38443314 pxar: change logic and impl all command line flags for xattrs/fcaps/acls
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-29 06:51:20 +02:00
Dietmar Maurer 60e589a111 src/api2/admin/datastore/backup.rs: log sucessful finish call 2019-05-28 12:55:22 +02:00
Dietmar Maurer c4ff3dcefd src/bin/proxmox-backup-client.rs: use new BackupClient to upload data 2019-05-28 10:12:44 +02:00
Dietmar Maurer 10241c20ea src/tools.rs: implement image_size helper 2019-05-28 09:35:08 +02:00
Dietmar Maurer 006f3ff407 src/api2/admin/datastore/backup/environment.rs: add more consistency checks 2019-05-28 09:21:31 +02:00
Dietmar Maurer 3dc5b2a203 src/client/http_client.rs: use start of chunk as offset 2019-05-28 09:21:31 +02:00
Dietmar Maurer 8de20e5c1f src/client/http_client.rs: code cleanup 2019-05-28 08:53:27 +02:00
Dietmar Maurer fc14b84937 src/backup/fixed_index.rs: improve bound checks inside add_digest 2019-05-28 08:47:03 +02:00
Dietmar Maurer e1225de467 src/backup/fixed_index.rs: store index_length to avoid code duplication 2019-05-28 08:27:57 +02:00
Dietmar Maurer e3d525feb7 src/api2/admin/datastore/backup.rs: fix typo 2019-05-28 07:13:19 +02:00
Christian Ebner af309d4d5c src/bin/pxar.rs: set archive permissions to 0o640
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:24:47 +02:00
Christian Ebner ab87f167f1 src/pxar/encoder.rs: Refactor file stat
Introduce helper functions to check file stats

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:23:49 +02:00
Christian Ebner 578b601159 src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:22:16 +02:00
Dietmar Maurer a42fa400ee src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api 2019-05-28 06:18:55 +02:00
Dietmar Maurer 29ae5c86a2 src/backup/fixed_index.rs: make chunk_size public 2019-05-28 06:18:24 +02:00
Dietmar Maurer 7dd1bcac58 src/client/http_client.rs: correctly parse error response in download_chunk_list 2019-05-27 09:35:30 +02:00
Dietmar Maurer a1e7cff3ea src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk 2019-05-27 07:57:43 +02:00
Dietmar Maurer 771953f9eb src/api2/admin/datastore/backup.rs: use smaller window sizes 2019-05-27 07:53:40 +02:00
Dietmar Maurer 174ad378d8 src/client/http_client.rs: fix request pipelining 2019-05-27 07:24:32 +02:00
Dietmar Maurer 624362226e src/client/merge_known_chunks.rs: use MergedChunkInfo as input 2019-05-26 10:52:56 +02:00
Dietmar Maurer 05cba08c9c src/client/http_client.rs: append chunks in correct order
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
2019-05-26 09:57:34 +02:00
Dietmar Maurer 417cb0731e src/api2/admin/datastore/backup.rs: verify chunk offset
This clearly shows that the current approach does not work - seems we get
chunks out of order.
2019-05-26 08:49:29 +02:00
Dietmar Maurer 34114e2606 src/server/rest.rs: correctly verify json parameters 2019-05-24 09:10:59 +02:00
Dietmar Maurer eb2bdd1bcc src/client/http_client.rs: encode parameters inside body
To avoid problems with long uris.
2019-05-24 08:32:55 +02:00
Dietmar Maurer 9bb675ec23 src/client/merge_known_chunks.rs: reduce RAM usage
There is no need to store chunk data for known chunks.
2019-05-24 07:42:58 +02:00
Dietmar Maurer ecb21b4794 src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough 2019-05-24 07:36:09 +02:00
Christian Ebner 2dbba78b98 src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:27:24 +02:00
Christian Ebner 6a19448004 src/pxar/encoder.rs: impl support to store ACLs in pxar achives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:21:33 +02:00
Christian Ebner 7cfaade7c8 src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:16:55 +02:00
Christian Ebner b344461b33 pxar: pass feature_flags to encoder/decoder instead of individual flags
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner 687407741b src/tools/xattr.rs: add test for is_valid_xattr_name()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner 357e4614e2 Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Wolfgang Bumiller c05a8c8d18 doc-test fixup
cargo test by default compiles and runs all code snippets
found in the documentation...

oops...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 15:22:33 +02:00
Wolfgang Bumiller cc84a830c5 RFC: schema: make enums static
I don't see a reason to allow these to be dynamically
modifiable.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 12:38:51 +02:00
Dietmar Maurer aa1b2e04fe src/client/merge_known_chunks.rs: merge known chunks
To decrease the number of api calls required...
2019-05-23 12:31:09 +02:00
Dietmar Maurer 91320f0879 src/client/http_client.rs: use ChunkInfo streams
This will make out of order uploads possible...
2019-05-23 12:31:09 +02:00
Wolfgang Bumiller 8ea3b1d188 tree-wide: use the new vec/io tools modules
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Wolfgang Bumiller 9cdda3f7c7 tools: add helpful vector and read operations
After importing the I/O ops trait via:
    use crate::tools::io::ops::*;

Instead of:
    let mut buffer = vec![0u8; 65536];
    file.read_exact(&mut buffer)?;
use:
    let buffer = file.read_exact_allocated(65536)?;

After importing the vector helpers via:
    use crate::tools::vec::{self, ops::*};

For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
    let mut buffer = vec![0u8; len];
use:
    let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Dietmar Maurer 8bea85b42e src/api2/admin/datastore/backup.rs: verify file size and chunk count on close 2019-05-23 08:50:36 +02:00
Dietmar Maurer 1179e15842 src/server/rest.rs: improve error handling 2019-05-23 08:15:32 +02:00
Dietmar Maurer 907f5bb0d8 src/api2/admin/datastore/backup/service.rs: improve error logging 2019-05-23 08:05:39 +02:00
Dietmar Maurer 32e1bf4260 src/bin/upload-speed.rs: code cleanup 2019-05-23 07:20:05 +02:00
Christian Ebner 0d9bab05ab src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:50 +02:00
Christian Ebner 4add706092 src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:32 +02:00
Christian Ebner 54b14ad741 src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:09 +02:00
Christian Ebner 1201360def src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:39:16 +02:00
Christian Ebner 8bc8791338 src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:37:37 +02:00
Christian Ebner a45a123950 src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:34:36 +02:00
Dietmar Maurer d6f204eda6 src/client/http_client.rs: re-order code - public functions first 2019-05-23 06:28:10 +02:00
Dietmar Maurer 9af37c8f0e src/client/http_client.rs: move low level H2 code into separate class 2019-05-22 17:28:25 +02:00
Dietmar Maurer d2c48afc6e src/client/http_client.rs: code cleanup 2019-05-22 13:24:33 +02:00
Dietmar Maurer 6ab34afa88 src/client/http_client.rs: cleanup h2 backup client 2019-05-22 13:05:51 +02:00
Dietmar Maurer a55fcd740f src/api2/admin/datastore/backup.rs: suse POST for finish() api 2019-05-22 13:04:58 +02:00
Dietmar Maurer de6742c04f src/bin/h2test.rs: removed
This test does no work anymore.
2019-05-22 11:14:56 +02:00
Dietmar Maurer 5a0b521809 src/api2/admin/datastore/backup.rs: remove test callbacks 2019-05-22 10:55:27 +02:00
Christian Ebner de61bc9264 src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr
As git does not preserve xattrs they have to be set before they can be read.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-22 10:01:10 +02:00
Dietmar Maurer 553610b43e src/client/http_client.rs: implement download_chunk_list 2019-05-22 09:46:02 +02:00
Dietmar Maurer 0cc0fffd1e src/backup/chunk_stream.rs: use BytesMut in ChunkStream 2019-05-22 09:39:02 +02:00
Dietmar Maurer c052be5c86 src/backup/chunk_stream.rs: use more generics 2019-05-22 09:18:05 +02:00
Dietmar Maurer 169c0e060f src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut) 2019-05-22 09:05:35 +02:00
Dietmar Maurer 3be3f3dcc4 src/backup/chunk_stream.rs: fix behaviour for large input buffers 2019-05-22 08:05:27 +02:00
Dietmar Maurer 02fa54ff3c src/backup/chunk_stream.rs: assemble data with correct order 2019-05-22 07:44:51 +02:00
Dietmar Maurer d0afeaa831 src/api2/admin/datastore/backup.rs: improve error message 2019-05-22 07:43:54 +02:00
Dietmar Maurer 11867a2b54 src/tools.rs: fix hex_to_digest 2019-05-21 13:42:13 +02:00
Dietmar Maurer 7f3d2ffab9 src/backup/index.rs: rename ChunkListReader into DigestListEncoder 2019-05-21 12:28:44 +02:00
Dietmar Maurer ee53955f52 src/backup/index.rs: implement DigestListDecoder 2019-05-21 12:21:22 +02:00
Dietmar Maurer a95849321c src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks 2019-05-21 09:54:11 +02:00
Dietmar Maurer 40f4e198a8 src/backup/dynamic_index.rs: add chunk_info method 2019-05-21 09:52:19 +02:00
Christian Ebner 596182056b src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:15:30 +02:00
Christian Ebner b303057a19 src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:06:26 +02:00
Christian Ebner bee8d8ea78 src/tools/xattr.rs: add functions name_store and security_capability
These functions allow to check if the name for xattrs/fcaps are valid.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:05:31 +02:00
Dietmar Maurer a09c0e38d8 src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks 2019-05-20 18:05:10 +02:00
Dietmar Maurer 97f03eff13 src/client/pxar_backup_stream.rs: fix docu 2019-05-20 17:29:00 +02:00
Dietmar Maurer 82ab72304e src/client/http_client.rs: implement upload_stream 2019-05-20 14:19:24 +02:00
Dietmar Maurer 2698e8a514 src/client/pxar_backup_stream.rs: close pipe inside drop 2019-05-20 11:20:33 +02:00
Dietmar Maurer f2269d8f99 src/tools.rs: implement hex_to_digest 2019-05-19 12:15:46 +02:00
Dietmar Maurer 751030aa7c src/bin/test_chunk_speed2.rs: test max chunk size 2019-05-19 11:09:02 +02:00
Dietmar Maurer ff77dbbea3 Revert "src/backup/chunk_stream.rs: simplify code"
This reverts commit e058744d8f.
The optimization was wrong, and produces wrong chunk boundaries.
2019-05-19 11:05:56 +02:00
Dietmar Maurer 178ac042c4 src/api_schema/schema.rs: improve error messages 2019-05-19 10:18:51 +02:00
Dietmar Maurer 0c516b123d src/client/pxar_backup_stream.rs: use WrappedStreamReader
to make it fully async ...
2019-05-18 15:13:31 +02:00
Dietmar Maurer c6e28b66c6 src/client/pxar_backup_stream.rs: use 1M pipe buffer size 2019-05-18 12:57:43 +02:00
Dietmar Maurer 6276448519 src/bin/test_chunk_speed2.rs: test chunker speed with real data 2019-05-18 11:59:17 +02:00
Dietmar Maurer e058744d8f src/backup/chunk_stream.rs: simplify code 2019-05-18 10:55:46 +02:00
Dietmar Maurer 8a7cc7565f src/backup/chunk_stream.rs: implement fixed size chunk stream 2019-05-18 10:55:46 +02:00
Dietmar Maurer 9c9ad941c0 src/client/http_client.rs: improve speed test
pipeline responses to a separate async channel ...
2019-05-18 10:55:46 +02:00
Christian Ebner 2dcdd3b427 src/tools/xattrs.rs: impl libc wrapper functions to get/set xattrs and fcaps
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:21 +02:00
Christian Ebner de2016d58a src/pxar/format_definitions.rs: add format definitions for xattrs and fcaps
In addition to the format definition, the traits needed for sorting of xattr
entries by name are derived.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:13 +02:00
Christian Ebner dec1ff18b8 src/tools/acl.rs: impl add_entry_full for ACL and set_permissions for ACLEntry
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:10:06 +02:00
Dietmar Maurer adec8ea263 src/api2/admin/datastore/backup.rs: add speedtest api, improve upload speed
We need to disable tcp Nagle algorythm (set_nodelay), and use larger window size for http2
2019-05-16 10:24:23 +02:00
Dietmar Maurer 372724afea src/api2/admin/datastore/backup.rs: implement finish_backup 2019-05-15 12:58:55 +02:00
Dietmar Maurer bb105f9dae src/api2/admin/datastore/backup.rs: improve logging 2019-05-15 10:09:46 +02:00
Dietmar Maurer 25ddc15fe0 src/client/pipe_to_stream.rs: add missing file 2019-05-15 09:07:19 +02:00
Dietmar Maurer a20772525f src/api2/admin/datastore/backup.rs: implement close for dynamic writers 2019-05-15 07:58:05 +02:00
Dietmar Maurer 4e93f8c164 src/api2/types.rs: add schema/format for file names 2019-05-15 07:14:08 +02:00
Dietmar Maurer e3dbd41b20 src/client/http_client.rs: add flow control for h2 upload 2019-05-14 14:54:21 +02:00
Dietmar Maurer 97f22ce57d src/client/http_client.rs: implement h2 upload
without flow control so far ...
2019-05-14 12:58:59 +02:00
Dietmar Maurer dafc27aeff src/backup/chunk_stream.rs: async chunk stream 2019-05-14 10:05:29 +02:00
Dietmar Maurer c0e22141ed src/client/http_client.rs: derive Clone for H2Client 2019-05-14 09:00:24 +02:00
Christian Ebner 25e205a1d3 src/tools.rs: add mod acl to tools
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:34:24 +02:00
Christian Ebner d22096ef4b src/tools/acl.rs: impl POSIX access control list (ACL) manipulation tools
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:28:42 +02:00
Dietmar Maurer 1133fe9a63 src/server/rest.rs: do not log 1xx status codes as errors 2019-05-14 06:23:22 +02:00
Dietmar Maurer 850ac6d0f0 src/client/http_client.rs: directly return H2Client on upgrade 2019-05-13 16:44:59 +02:00
Dietmar Maurer 10130cf48c src/client/http_client.rs: cleanup, remove debug println 2019-05-13 12:11:18 +02:00
Dietmar Maurer b57cb26406 src/client/http_client.rs: implement http2 client wrapper 2019-05-13 10:27:22 +02:00
Dietmar Maurer 9e391bb7f0 src/client/http_client.rs: allow to pass parameters (encode them as query string) 2019-05-13 09:12:03 +02:00
Dietmar Maurer fbb798f6d2 src/api2/admin/datastore/backup.rs: ignore errors from last_backup 2019-05-13 07:06:56 +02:00
Dietmar Maurer d36113665b src/api2/admin/datastore/backup.rs: implement chunk index download from previous backup 2019-05-11 16:05:50 +02:00
Dietmar Maurer 457531e7fe src/backup/index.rs: new ChunkListReader 2019-05-11 16:01:42 +02:00
Dietmar Maurer af1d68185c src/api2/admin/datastore/backup.rs: add missing 'archive-name' parameter to schema 2019-05-11 12:25:15 +02:00
Dietmar Maurer 6b95c7df07 src/api2/admin/datastore/backup/environment.rs: do not store path 2019-05-11 12:13:44 +02:00
Dietmar Maurer 51a4f63ff7 src/backup/datastore.rs: move last_backup to BackupInfo 2019-05-11 12:07:09 +02:00
Dietmar Maurer b02a52e3a5 src/backup/datastore.rs: impl last_backup helper 2019-05-11 11:36:46 +02:00
Dietmar Maurer c09775011a src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups 2019-05-11 10:19:34 +02:00
Dietmar Maurer f9578f3c79 backup-api: allow to create DynamicIndexWriter and add chunks to to 2019-05-10 10:25:40 +02:00
Dietmar Maurer 35a2d8a6a6 src/backup/backup_info.rs: Derive Clone 2019-05-10 06:59:23 +02:00
Dietmar Maurer 0506d1ed04 upload_chunk.rs: add comment about possible unnecessary code 2019-05-10 06:15:25 +02:00
Dietmar Maurer 5f42ef009a cleanup: remove unused code 2019-05-09 18:11:39 +02:00
Dietmar Maurer b4b63e520d handle_async_api_request: put rpcenv into a Box
So that we can pass rpcenv into futures.
2019-05-09 18:01:24 +02:00
Dietmar Maurer 2c41096a4b src/api2/admin/datastore/backup/upload_chunk.rs: fix maximal chunk size 2019-05-09 13:29:50 +02:00
Dietmar Maurer 78b5191550 src/server/formatter.rs: further cleanups and renaming ... 2019-05-09 13:28:26 +02:00
Dietmar Maurer d55037e471 src/server/formatter.rs: rename format_result to format_data
To avoid confusions with Rust Result type.
2019-05-09 13:15:15 +02:00
Dietmar Maurer 21ee7912fa src/api2/admin/datastore/backup.rs: implement upload chunk 2019-05-09 13:06:09 +02:00
Dietmar Maurer 4ebf0eabb0 src/api2/types.rs: use separate file for schema definitions 2019-05-09 07:44:09 +02:00
Dietmar Maurer bd1507c4fa src/api2/admin/datastore/backup/service.rs: move service code into extra file 2019-05-08 17:36:19 +02:00
Dietmar Maurer 0aadd40b4f src/api2/admin/datastore/backup.rs: add required parameters, cleanups 2019-05-08 13:17:23 +02:00
Dietmar Maurer d95ced64ad src/api2/admin/datastore/backup/environment.rs: use separate file 2019-05-08 12:41:58 +02:00
Dietmar Maurer ca60c371e2 api: rename h2upload to backup 2019-05-08 12:35:15 +02:00
Dietmar Maurer d906b1124f src/api2/admin/datastore/h2upload.rs: code cleanups 2019-05-08 11:36:43 +02:00
Dietmar Maurer 58c8d7d91f rc/api2/admin/datastore/h2upload.rs: implement BackupEnvironment
To pass arbitrary information/state to api methods.
2019-05-08 11:26:54 +02:00
Dietmar Maurer f757b30efc src/server/rest.rs: use generics to pass RpcEnvironment 2019-05-08 11:09:01 +02:00
Dietmar Maurer 2edc341b29 src/tools.rs: Add AsAyn Trait 2019-05-08 11:07:36 +02:00
Dietmar Maurer 090ac9f79a src/api2/admin/datastore/h2upload.rs: avoid unwrap 2019-05-08 08:21:18 +02:00
Dietmar Maurer 4e3da4b366 src/api2/admin/datastore/h2upload.rs: do not log client address 2019-05-08 08:15:55 +02:00
Dietmar Maurer e3a44552ee src/api2/admin/datastore/h2upload.rs: code cleanups 2019-05-08 08:12:49 +02:00
Dietmar Maurer a66ab8ae5e src/api2/admin/datastore/h2upload.rs: make worker abortable 2019-05-08 07:14:40 +02:00
Dietmar Maurer 72375ce6c6 src/api2/admin/datastore/h2upload.rs: log to worker task 2019-05-07 14:07:00 +02:00
Dietmar Maurer d9bd06eae8 src/api2/admin/datastore/h2upload.rs: use WorkerTask 2019-05-07 13:42:00 +02:00
Dietmar Maurer 52cf506e48 src/api2/admin/datastore/h2upload.rs: implement async test api call 2019-05-07 12:26:55 +02:00
Dietmar Maurer 92ac375aff src/api2/admin/datastore/h2upload.rs: improve http2 service 2019-05-07 11:24:44 +02:00
Dietmar Maurer 78f7e225e4 src/server/rest.rs: make handle_(a)sync_api_request public 2019-05-07 11:23:52 +02:00
Dietmar Maurer 75097495ad RestEnvironment: derive Clone 2019-05-07 11:09:18 +02:00
Dietmar Maurer 01bf3b7b5f move find_method to Router class 2019-05-07 11:08:30 +02:00
Dietmar Maurer 3578d99f3e move normalize_path to tools::normalize_uri_path 2019-05-07 09:44:34 +02:00
Dietmar Maurer 96e95fc179 src/api2/admin/datastore/h2upload.rs: test if version < 2
We can only upgrade http 1.x connections.
2019-05-06 10:29:34 +02:00
Dietmar Maurer 45cdce069e src/client/http_client.rs: request password at creation time 2019-04-30 11:44:35 +02:00
Dietmar Maurer e45afdff9c src/server/state.rs: use new BroadcastData helper 2019-04-30 10:21:48 +02:00
Dietmar Maurer 824b5ee4ee src/tools/broadcast_future.rs: cleanup, decompose into two classes
In order to make it more usable.
2019-04-30 08:57:20 +02:00
Wolfgang Bumiller d82ed9b0f5 tools: add AsyncMutex
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-29 14:19:03 +02:00
Dietmar Maurer 3a9bea3918 src/bin/h2test.rs: test h2 upgrade 2019-04-29 11:59:18 +02:00
Dietmar Maurer cf639a47de rc/client/http_client.rs: add experimental h2 upgrade code 2019-04-29 11:57:58 +02:00
Dietmar Maurer 5a2df00004 src/client/http_client.rs: cleanup, make login fully async 2019-04-28 10:55:03 +02:00
Dietmar Maurer 3dceb9b304 src/tools/broadcast_future.rs: add new constructor new_oneshot()
To simplify usage. Also added a test case.
2019-04-27 10:56:49 +02:00
Dietmar Maurer 152764ec15 src/api2/admin/datastore/h2upload.rs: add test code for http2 2019-04-27 08:57:35 +02:00
Dietmar Maurer 490be29ed6 src/tools/broadcast_future.rs: new helper class 2019-04-26 17:57:57 +02:00
Wolfgang Bumiller e888155712 proxy: allow .key/.pem files instead of a pkcs12/.pfx file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-26 10:47:33 +02:00
Christian Ebner 8b7c0def84 src/tools/procfs.rs: clean up tab indentation and use whitespaces instead
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-25 13:14:52 +02:00
Wolfgang Bumiller 5e5eed5c3b use double-fork for reload
To ensure the new process' parent is pid 1, so systemd won't
complain about supervising a process it does not own.

Fixes the following log spam on reloads:
Apr 25 10:50:54 deb-dev systemd[1]: proxmox-backup.service: Supervising process 1625 which is not our child. We'll most likely not notice when it exits.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
Wolfgang Bumiller efd1536eb7 add a wrapper around nix::unistd::pipe2
Using O_CLOEXEC by default, and returning Fd handles to
ensure they get dropped on bail!() or panic!() if the RawFds
aren't used yet.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
Wolfgang Bumiller 897982e237 tools: add read/write convenience helpers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
Wolfgang Bumiller d98c9a7a38 use service Type=notify
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:33 +00:00
Wolfgang Bumiller 9c351a36c4 tools/daemon: add sd_notify wrapper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:28 +00:00
Dietmar Maurer d8871f2f18 src/tools/procfs.rs: avoid useless allocations 2019-04-24 09:35:15 +02:00
Christian Ebner ae694f44ef src/tools/procfs.rs: add tests for read_proc_net_route() and read_proc_net_ipv6_route()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:52 +02:00
Christian Ebner ec8db837fb src/tools/procfs.rs: refactor read_proc_net_route() and read_proc_net_ipv6_route()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:32 +02:00
Christian Ebner eb46c3d1ec src/tools/procfs.rs: implement read_proc_net_ipv6_route()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-23 14:31:23 +02:00
Dietmar Maurer 6358b68236 cleanup: avoid compiler warnings 2019-04-23 12:17:24 +02:00
Dietmar Maurer e06ffb10d1 src/api_schema/router.rs: fix typo (subdira -> subdir) 2019-04-16 12:20:17 +02:00
Dietmar Maurer 13f1cc17ea src/api_schema/router.rs: implement list_subdirs() helper 2019-04-16 12:07:02 +02:00
Wolfgang Bumiller 062d4916ff api_schema: allow generic api handler functions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-16 11:22:23 +02:00
Christian Ebner 286f0d4099 src/tools/procfs.rs: implement read_proc_net_route()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-15 11:13:16 +02:00
Dietmar Maurer 1a6c9415ae src/api2/node/services.rs: add proxmox-backup-proxy service 2019-04-15 09:48:02 +02:00
Dietmar Maurer 75bc49bed7 src/server/worker_task.rs: implement abort channel 2019-04-15 09:38:05 +02:00
Dietmar Maurer 912524752e src/api2/admin/datastore/pxar.rs: create worker task for file upload 2019-04-15 08:38:14 +02:00
Dietmar Maurer 1be71fb02b src/server/worker_task.rs_ do not write ERROR: prefix into task index 2019-04-15 08:37:13 +02:00
Dietmar Maurer 418def7a78 src/server/worker_task.rs: move common code into log_result
Make new() pub
2019-04-15 07:49:03 +02:00
Dietmar Maurer 37b87869b5 src/server/upid.rs: check for illegal characters 2019-04-15 07:47:51 +02:00
Christian Ebner d2981e2738 src/tools/procfs.rs: implement read_proc_net_dev()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:59 +02:00
Christian Ebner 13abfdc57a src/tools/procfs.rs: implement read_memory_usage()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:01 +02:00
Christian Ebner 3005ef48b6 src/tools/procfs.rs: Rename variable and struct to be more consistent
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:34 +02:00
Christian Ebner cc93bacb72 src/tools/procfs.rs: simplify read_cpuinfo()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:14 +02:00
Dietmar Maurer f2b99c34f7 src/api2/admin/datastore.rs: implement API to return last GC status 2019-04-11 12:04:25 +02:00
Dietmar Maurer 59961b8954 src/server/command_socket.rs: check control socket permissions 2019-04-11 10:51:59 +02:00
Dietmar Maurer ba70040dbf src/server/worker_task.rs: change task list sort order - latest on top 2019-04-11 09:43:33 +02:00
Dietmar Maurer d3f4c08f33 src/server/worker_task.rs: catch panics in worker threads 2019-04-11 07:56:06 +02:00
Wolfgang Bumiller 620dccf1a1 tools/daemon: dup the TcpListener file descriptor
Now that we let hyper shutdown gracefully we need an owned
version of the listening socket to prevent it from closing
before running the reload preparations.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
Wolfgang Bumiller d96bb7f163 tools: add Fd helper
stores a raw file descriptor with a drop handler for
safekeeping in closures

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
Christian Ebner 1cd33633aa /src/tools: Cleanup tempfiles in folder
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:56:22 +02:00
Dietmar Maurer a665dea128 src/api2/node/tasks.rs: implement stop_task api 2019-04-10 13:55:05 +02:00
Dietmar Maurer 98a181f055 src/server/worker_task.rs: log message when we call request_abort() 2019-04-10 13:54:28 +02:00
Christian Ebner 2b6a3e1396 src/tools/procfs.rs: implement read_cpuinfo()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:11:11 +02:00
Dietmar Maurer 321070b4fa src/server/worker_task.rs: implement abort_worker (via command_socket) 2019-04-10 12:42:24 +02:00
Dietmar Maurer cfb2d3c17c src/server/command_socket.rs: correctly handle/spawn handle parallel connections 2019-04-10 11:05:00 +02:00
Dietmar Maurer 9b002cbc5f src/server/worker_task.rs: use abstract socket 2019-04-10 09:03:17 +02:00
Dietmar Maurer 5f550fd99f start hyper server using with_graceful_shutdown()
Without, hyper keeps some futures running, and the server does not
correctly shutdown.
2019-04-10 08:24:32 +02:00
Dietmar Maurer 7f2078de59 src/tools/procfs.rs: avoid compiler warning 2019-04-10 06:40:29 +02:00
Christian Ebner 1f6fbf1c57 src/tools/procfs.rs: implement read_meminfo()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-09 15:21:49 +02:00
Dietmar Maurer c360bd73d3 src/api2/node/tasks.rs: fix task status api 2019-04-09 15:12:20 +02:00
Dietmar Maurer 0bfd87bc5e src/server/worker_task.rs: speedup upid_read_status() 2019-04-09 15:11:48 +02:00
Dietmar Maurer d8d40dd0c2 src/api2/node.rs: add node parameter 2019-04-09 14:43:30 +02:00
Dietmar Maurer 7ae0bf52dd src/server/command_socket.rs: code cleanup - fix error message 2019-04-09 12:47:42 +02:00
Dietmar Maurer d607b8861b src/server/worker_task.rs: implement task control socket 2019-04-09 12:15:06 +02:00
Dietmar Maurer e201753629 src/server/command_socket.rs: implement auto_remove flag
Remove the socket file on close.
2019-04-09 11:47:23 +02:00
Dietmar Maurer 634132fe10 src/server/upid.rs: moved code into separate file 2019-04-08 18:43:26 +02:00
Dietmar Maurer 78a39e0517 src/server/command_socket.rs: simple command socket 2019-04-08 17:59:39 +02:00
Dietmar Maurer e3f41f2165 src/tools/daemon.rs: use new ServerState handler 2019-04-08 14:00:23 +02:00
Dietmar Maurer 7a630df709 implement server state/signal handling, depend on tokio-signal 2019-04-08 13:59:07 +02:00
Dietmar Maurer 5a12c0e2fb src/api2/node/tasks.rs: start upid API 2019-04-07 14:36:57 +02:00
Dietmar Maurer b75b968147 src/server/worker_task.rs: implement is_active() 2019-04-07 14:36:25 +02:00
Dietmar Maurer 4494d07819 src/server/worker_task.rs: make log_path a method on UPID 2019-04-07 13:51:46 +02:00
Dietmar Maurer d2a2e02bb7 src/api2/node/tasks.rs: implement userfilter 2019-04-07 13:17:19 +02:00
Dietmar Maurer 3c3bee2e32 src/api2/node/tasks.rs: correctly handle start parameter 2019-04-07 12:41:24 +02:00
Dietmar Maurer 063ca5be77 src/api2/node/tasks.rs: implement list_tasks 2019-04-07 12:18:58 +02:00
Dietmar Maurer d4b59ae0b8 src/backup/datastore.rs: pass WorkerTask to GC
So that we can log to the task log file.
2019-04-06 17:57:38 +02:00
Dietmar Maurer 35950380a9 src/server/worker_task.rs: carefully handle file permissions 2019-04-06 17:53:12 +02:00
Dietmar Maurer 1619a72063 src/tools.rs: implement create_dir_chown
Combinded mkdir with chown.
2019-04-06 16:50:23 +02:00
Dietmar Maurer eea8131952 src/tools.rs: implement file_set_contents_full() 2019-04-06 13:53:43 +02:00
Dietmar Maurer 0f778e0652 src/api2/admin/datastore.rs: use new WorkerTask::new_thread() 2019-04-06 11:27:23 +02:00
Dietmar Maurer 244abab769 src/tools/file_logger.rs: fix test 2019-04-06 11:24:37 +02:00
Dietmar Maurer 660c684640 src/server/worker_task.rs: return upid_str 2019-04-06 11:23:53 +02:00
Dietmar Maurer 93aebb38bc src/server/worker_task.rs: implement read_task_list() 2019-04-06 10:17:11 +02:00
Dietmar Maurer 4a36a2c999 src/server/worker_task.rs: improve UPID format description 2019-04-06 09:36:50 +02:00
Dietmar Maurer 882594c5e9 src/server.rs: improve crate layout 2019-04-06 09:17:25 +02:00
Dietmar Maurer 4b01c983f0 src/server/worker_task.rs: implement persistent task list 2019-04-05 12:37:35 +02:00
Christian Ebner a2474d328f src/tools/procfs.rs: implement read_proc_uptime and read_proc_uptime_ticks
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-05 06:22:21 +02:00
Christian Ebner 08daa81af6 procfs: implement check_process_running and check_process_running_pstart
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-04 13:54:17 +02:00
Dietmar Maurer 9a0d0ff774 server/worker_task.rs: implement UPID parser 2019-04-04 13:49:35 +02:00
Dietmar Maurer 49cf9f3d0b src/tools.rs: file_get_json() - add new default parameter 2019-04-04 12:24:18 +02:00
Dietmar Maurer 79ad7b7ec2 src/tools/procfs.rs: split regex into several lines 2019-04-04 09:25:19 +02:00
Dietmar Maurer 479f6e40ec src/server/worker_task.rs: implement task registry 2019-04-04 08:58:39 +02:00
Dietmar Maurer 52ddc2522d src/tools/procfs.rs: use libc::pid_t 2019-04-04 08:05:43 +02:00
Dietmar Maurer d3d7ab1273 src/tools/file_logger.rs: new - accept AsRef<Path> 2019-04-03 14:13:33 +02:00
Dietmar Maurer 3c2012f97a src/tools/procfs.rs: start helper tools to read procfs 2019-04-03 13:39:20 +02:00
Dietmar Maurer 3489936ebb src/tools/file_logger.rs: change timestamp format to rfc3339 2019-04-03 08:58:43 +02:00
Dietmar Maurer 92da93b245 abort GC on server shutdown 2019-04-01 12:13:02 +02:00
Dietmar Maurer 9136f857cc add global var to indicate server shutdown requests 2019-04-01 12:05:11 +02:00
Dietmar Maurer 3e6a7deeaa src/api2/admin/datastore.rs: start GC in a separate thread 2019-04-01 12:02:46 +02:00
Dietmar Maurer ad281d1bd9 www/DataStoreConfig.js: add button to start GC 2019-04-01 08:08:34 +02:00
Dietmar Maurer 49d123ee3a src/server/rest.rs: use formatter to encode errors 2019-04-01 08:04:12 +02:00
Dietmar Maurer 7d4ef127cb src/server/rest.rs: generate csrf token if we have a valid ticket
This is important if the user reloads the browser page.
2019-04-01 07:52:30 +02:00
Dietmar Maurer 11861a482d src/backup/chunk_store.rs: fix GC
Added option to get oldest_writer timestamp from ProcessLocker.
2019-03-31 17:21:36 +02:00
Dietmar Maurer d85987aeeb fix last commit: the filename var was not ment to be removed, sorry 2019-03-31 16:16:14 +02:00
Dietmar Maurer 15a77c4c2e src/backup/chunk_store.rs: avoid create/unlink race 2019-03-31 10:03:01 +02:00
Dietmar Maurer c374f05499 src/backup.rs: improve GC problem description 2019-03-31 09:44:35 +02:00
Dietmar Maurer c8ec450e37 src/backup.rs: start explaining different GC algorithm 2019-03-30 17:21:40 +01:00
Dietmar Maurer 8a4757342c src/backup.rs: describe the garbage collection problem 2019-03-30 16:26:52 +01:00
Dietmar Maurer 10eea49d81 src/backup/fixed_index.rs: store reference to file
Keep it open and locked while index is in use.
2019-03-27 10:17:03 +01:00
Dietmar Maurer c597a92c81 index file readers: aquire shared lock 2019-03-27 10:11:14 +01:00
Dietmar Maurer 0465218953 src/backup.rs: add documentation about ChunkStore locking 2019-03-22 10:14:50 +01:00
Dietmar Maurer 43b1303398 datastore: use new ProcessLocker
To make sure only one process runs garbage collection while having active writers.
2019-03-22 09:42:15 +01:00
Dietmar Maurer abfc001f25 src/tools/process_locker.rs: new() - use generic path parameter 2019-03-22 09:41:32 +01:00
Dietmar Maurer a650f50301 src/tools/process_locker.rs: implement inter-process reader-writer locks 2019-03-22 08:05:51 +01:00
Wolfgang Bumiller eb90c9e372 tools: style fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-20 09:57:13 +01:00
Christian Ebner e04d1bfadc Cleanup of if statement with unwraps by a more compact match statement
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-03-20 09:49:18 +01:00
Dietmar Maurer 8346f0d59b src/server/rest.rs: correctly extract content type 2019-03-19 12:50:15 +01:00
Wolfgang Bumiller a690ecacf0 daemon: simplify daemon creation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 12:12:54 +01:00
Wolfgang Bumiller e4311382d9 tools: daemon: rename some structs
Reloadable resources are now 'Reloadable' instead of
'ReexecContinue'.

The struct handling the reload is a 'Reloader', not a
'ReexecStore'.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 11:09:46 +01:00
Wolfgang Bumiller 5e7bc50af1 add reload support to api daemon
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller a2ca713701 add reload support to proxy
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller af70c1811b tools: implement ReexecContinue for tokio's TcpListener
This is the only thing we currently need to keep alive for
reloads.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller 4422ba2c7f tools: daemon: add a default signalfd helper
Proxy and daemon for now just want to handle reload via
`SIGHUP`, so provide a helper creating the signalfd stream
doing that - this is simply a filtered stream which passes
the remaining signals through, so it can be used exactly
like the signalfd stream could before to add more signals.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller dce94d0e09 tools: add daemon helpers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller ff7049d481 tools: add fd_change_cloexec helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller 1c042cdc6c proxy: listener error handling fixup
If the listening socket goes into some error state we'll get
std::io::Errors rather than higher level errors from the
native_tls::TlsAcceptor, those are usually fatal. (Ran into
this after performing a shutdown() on the file descriptor,
after which the future just endlessly loops in accept().)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Dietmar Maurer 60864b1b56 src/pxar.rs: improve docu 2019-03-18 12:27:30 +01:00
Dietmar Maurer 515688d1f1 avoid compiler warnings 2019-03-18 10:00:58 +01:00
Dietmar Maurer ba10f2b0d4 src/bin/dynamic-files.rs: test case for file shrink during backup 2019-03-16 14:22:14 +01:00
Dietmar Maurer af572aaa4c pxar: implement hardlinks
So we are no longer compatible with catar ...
2019-03-16 11:02:12 +01:00
Dietmar Maurer 812dd935ce src/pxar/encoder.rs: detect hardlinks 2019-03-15 12:19:51 +01:00
Dietmar Maurer 896e803feb src/pxar/sequential_decoder.rs: dump_entry - fix FIFO/SOCKET handling 2019-03-15 12:15:38 +01:00
Dietmar Maurer 7dfa17c7cc src/pxar/binary_search_tree.rs: fix test 2019-03-15 11:34:31 +01:00
Dietmar Maurer 9307279faf cleanup: remove dead code 2019-03-15 10:18:28 +01:00
Dietmar Maurer 46cfe7ab7e src/pxar/decoder.rs: implement seekable decoder 2019-03-15 09:36:05 +01:00