tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,913 Commits 2 Branches 54 Tags 14 MiB
247a8ca5bb
Commit Graph

1676 Commits

Author SHA1 Message Date
Dietmar Maurer
5ed2277f0e src/bin/h2s-server.rs: test to reproduce slow h2 download 
Simplified code, use with h2s-client.rs.
 2019-06-29 14:57:54 +02:00
Dietmar Maurer
17243003b3 src/bin/download-speed.rs: h2 download speedtest for our server 
I get about 30MB/s (much too slow)
 2019-06-29 13:43:10 +02:00
Dietmar Maurer
a2b29b68d4 src/client/http_client.rs: use maximal h2 window sizes 2019-06-29 11:05:36 +02:00
Dietmar Maurer
872debdefa src/bin/h2client.rs: tune h2 client parameters to get reasonable speed 
We now get about 2.5GB/s (not really good, but better)
 2019-06-29 10:41:24 +02:00
Dietmar Maurer
38294bcc56 src/client.rs: make module put_to_stream public 2019-06-29 10:40:43 +02:00
Dietmar Maurer
fded1f3154 add code to test H2 speed 
I currently get about 1MB/s, which is ways too slow (must be a bug)?
 2019-06-29 10:08:12 +02:00
Dietmar Maurer
d48a9955a5 src/backup/dynamic_index.rs: introduce ReadChunk trait 2019-06-28 16:35:00 +02:00
Dietmar Maurer
09d7dc5024 src/api2/reader.rs: add download chunk api 2019-06-28 16:27:01 +02:00
Dietmar Maurer
fcf5dea56d src/client/http_client.rs: increase h2 window size, set tcp nodelay option 
To increase h2 download speed.
 2019-06-28 16:00:58 +02:00
Dietmar Maurer
81a6ce6fde src/backup/chunk_store.rs: new method chunk_path() 
Returns the absolute path.
 2019-06-28 15:48:09 +02:00
Dietmar Maurer
fcfb84fedf file download: avoid unnecessary copy 2019-06-28 07:07:52 +02:00
Dietmar Maurer
984a7c3502 src/client/http_client.rs - h2 download: implement flow control 2019-06-28 07:02:43 +02:00
Dietmar Maurer
dd066d28e2 src/api2/reader.rs: implement backup reader protocol 2019-06-27 09:01:41 +02:00
Dietmar Maurer
42a87f7b96 src/server/h2service.rs: implement generic h2 service 2019-06-26 17:38:33 +02:00
Dietmar Maurer
fb0470837b src/client/http_client.rs: split out code to start new h2 connections 2019-06-26 12:09:18 +02:00
Dietmar Maurer
b208da8393 src/backup/data_{chunk,blob}.rs: add verify_crc method 2019-06-26 09:54:25 +02:00
Dietmar Maurer
fef44d4f78 src/bin/proxmox-backup-client.rs: decode blobs on download 
No more need for that clumsy blob cli (removed).
 2019-06-26 09:18:59 +02:00
Dietmar Maurer
c2b945341c src/client/http_client.rs - download: use generic Write type, return writer. 
Make it possible to write int Vec<u8>.
 2019-06-26 09:17:13 +02:00
Dietmar Maurer
bb8231409e src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json 
Use the KeyConfig serialization with kdf = None.
 2019-06-26 07:32:34 +02:00
Dietmar Maurer
3031e44c58 src/bin/proxmox-backup-client.rs: remove stale download file, start blob cli 2019-06-25 13:00:37 +02:00
Dietmar Maurer
f14a8c9a85 src/api2/admin/datastore.rs: use correct path for download 2019-06-25 12:59:36 +02:00
Dietmar Maurer
81820b0d4d src/client/http_client.rs - download: fix error handling 2019-06-25 12:43:55 +02:00
Dietmar Maurer
45db6f89e1 src/bin/proxmox-backup-client.rs: implement download command 2019-06-25 11:17:24 +02:00
Dietmar Maurer
9e47c0a56e src/api2/admin/datastore.rs: implement generic download method 
We simply allow to download raw binary data.
 2019-06-25 10:16:59 +02:00
Dietmar Maurer
da7d67213e src/api2/admin/datastore/pxar.rs: remove pxar upload api 
Upload needs to be done using the backup protocol.
 2019-06-25 09:45:35 +02:00
Dietmar Maurer
dd8e744f9a src/api2/admin/datastore.rs: use a worker log log prune actions 2019-06-25 08:12:25 +02:00
Dietmar Maurer
6899dbfb47 src/bin/proxmox-backup-client.rs: strip .blob extension 2019-06-25 07:11:02 +02:00
Dietmar Maurer
c6d203bbd3 src/backup/backup_info.rs: also list .blob files 2019-06-25 07:07:45 +02:00
Dietmar Maurer
045e8a568d src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key 2019-06-25 07:00:07 +02:00
Dietmar Maurer
79bc7345e8 src/cli/command.rs: fix ReST output format 2019-06-25 06:19:51 +02:00
Dietmar Maurer
9f46c7de4b src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup 2019-06-24 13:56:37 +02:00
Christian Ebner
cd7dc87903 pxar: impl .pxarexclude parsing and exclude matching 
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.

Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-24 10:26:20 +02:00
Dietmar Maurer
46bd880041 src/api2/backup/environment.rs: new helper add_blob() 2019-06-24 09:35:37 +02:00
Dietmar Maurer
cb08ac3efe src/api2/backup.rs: replace upload_config with upload_blob 2019-06-23 11:44:30 +02:00
Dietmar Maurer
a38c5d4d12 src/backup/data_blob.rs: implement from_raw() 2019-06-23 09:35:44 +02:00
Dietmar Maurer
6f083b7a92 src/api2/backup/upload_chunk.rs: compute crc on upload 2019-06-23 09:04:42 +02:00
Dietmar Maurer
ba01828d38 try to use use proxmox::tools::io::ops::ReadExtOps 2019-06-22 16:29:10 +02:00
Dietmar Maurer
27042ce637 src/backup/file_formats.rs: improve docs 2019-06-22 15:45:36 +02:00
Dietmar Maurer
9f83e0f7ee src/backup/crypt_config.rs: simplify decode interface 2019-06-22 13:24:29 +02:00
Dietmar Maurer
0066c6d972 src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead 2019-06-22 13:02:53 +02:00
Dietmar Maurer
ee8a7e8080 src/backup/crypt_config.rs: implement encrypt_to (output to writer) 2019-06-22 12:25:04 +02:00
Dietmar Maurer
991abfa8b4 src/backup/file_formats.rs: split out file format data 2019-06-22 09:12:25 +02:00
Dietmar Maurer
b7f4f27d6c add crc field for binary blobs formats 2019-06-21 18:49:04 +02:00
Dietmar Maurer
a7f67a9a9c depend on crc32fast 
We can use this to compute fast checksums to test file integrity (not crypto safe).
 2019-06-21 15:58:00 +02:00
Dietmar Maurer
3025b3a53c src/backup/data_blob.rs: new file format for binary blobs 2019-06-21 11:32:07 +02:00
Dietmar Maurer
077a8cae63 src/backup/crypt_config.rs: make code more flexible 
Allow to pass magic numbers as parameters.
 2019-06-21 10:50:48 +02:00
Dietmar Maurer
c38266c18d renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs 2019-06-21 09:51:18 +02:00
Dietmar Maurer
3ea8bfc9c0 src/bin/proxmox-backup-client.rs: add import-master-pubkey command 2019-06-21 07:34:17 +02:00
Dietmar Maurer
37c5a17505 src/bin/proxmox-backup-client.rs: helper to generate RSA master key 2019-06-21 07:12:19 +02:00
Dietmar Maurer
ab44acff57 src/backup/key_derivation.rs: add modified field to key file 2019-06-20 07:58:33 +02:00
Dietmar Maurer
6d0983dbe1 src/bin/proxmox-backup-client.rs: add keyfile parameter for backup 2019-06-19 17:16:41 +02:00
Dietmar Maurer
181f097af9 src/bin/proxmox-backup-client.rs - key API: pass kdf parameter 
Allow to store keys without password.
 2019-06-18 18:21:01 +02:00
Dietmar Maurer
9b06db4563 src/bin/proxmox-backup-client.rs: use file system path to reference keys 2019-06-18 17:38:50 +02:00
Dietmar Maurer
091540298e src/bin/proxmox-backup-client.rs - create_key: always read password from tty 2019-06-18 17:27:52 +02:00
Dietmar Maurer
826f309bf5 src/backup/key_derivation.rs: move kdf code into separate file 2019-06-18 11:17:22 +02:00
Dietmar Maurer
11515438cc Cargo.toml: use serde feature derive 2019-06-18 06:23:25 +02:00
Dietmar Maurer
ac71623483 src/bin/proxmox-backup-client.rs: implement "key change-passphrase" 2019-06-17 13:10:00 +02:00
Dietmar Maurer
f2401311b0 src/bin/proxmox-backup-client.rs: start key management 2019-06-17 10:33:24 +02:00
Dietmar Maurer
a7dd483097 src/backup/*_index.rs: used generated magic numbers 2019-06-14 14:58:37 +02:00
Dietmar Maurer
bffd40d6b7 src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools 2019-06-14 11:40:04 +02:00
Dietmar Maurer
00388226e6 src/api2/backup/environment.rs: log index checksums 2019-06-14 10:36:20 +02:00
Dietmar Maurer
9335d74eba src/backup/fixed_index.rs: compute checksum over the index 2019-06-14 10:34:29 +02:00
Dietmar Maurer
16ff6b7cd8 src/backup/dynamic_index.rs: compute checksum over the index 2019-06-14 10:33:59 +02:00
Dietmar Maurer
36075475a6 src/api2/backup/environment.rs: log duplicate chunks 2019-06-14 07:12:30 +02:00
Dietmar Maurer
fa148dbd11 src/api2/backup/upload_chunk.rs: verify chunk data if possible 2019-06-14 06:23:46 +02:00
Dietmar Maurer
0f37577092 src/bin/cipherbench.rs: add test for zstd compression 2019-06-13 17:24:57 +02:00
Dietmar Maurer
9399c98f82 src/bin/cipherbench.rs: cipher speed test with large blocks 2019-06-13 17:16:43 +02:00
Dietmar Maurer
f640f23ac5 src/backup/data_chunk.rs: fix typo 2019-06-13 16:42:55 +02:00
Dietmar Maurer
dba72d5016 src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:56:50 +02:00
Dietmar Maurer
68ab376ae9 src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:47:54 +02:00
Dietmar Maurer
f98ac774ee backup: Add support for client side encryption 
first try ...
 2019-06-13 11:47:23 +02:00
Dietmar Maurer
51929e4532 src/backup/data_chunk.rs - decode: make crypt_config optional 2019-06-13 11:44:38 +02:00
Dietmar Maurer
046bcb9773 src/backup/crypt_setup.rs: add comment for possible improvements 2019-06-13 07:35:59 +02:00
Dietmar Maurer
bd0e3c7cfa src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs 2019-06-13 07:35:02 +02:00
Dietmar Maurer
6762db70d6 src/api2/types.rs: define CHUNK_DIGEST_SCHEMA 2019-06-13 07:30:42 +02:00
Dietmar Maurer
b595cb9d2c src/backup/data_chunk.rs: Data Chunk builder 
Make handling encoded chunks easier.
 2019-06-12 13:37:01 +02:00
Dietmar Maurer
ef27200cec src/backup/crypt_setup.rs: fix encoder for uncompressed chunks 2019-06-10 09:56:06 +02:00
Dietmar Maurer
74792b95b2 src/backup/crypt_setup.rs: allow compressed and uncompressed chunks 2019-06-10 08:27:35 +02:00
Dietmar Maurer
9e0187a203 src/backup/crypt_setup.rs: remove unused CryptData struct 2019-06-10 07:55:36 +02:00
Dietmar Maurer
bec8498ae3 src/backup/crypt_setup.rs: avoid useless allocations 2019-06-09 11:44:17 +02:00
Dietmar Maurer
c20b17b460 src/backup.rs: define magic numbers for chunk store file format 2019-06-08 10:34:20 +02:00
Dietmar Maurer
48b4b40b57 src/backup/crypt_setup.rs: crypto helpers 2019-06-08 09:56:51 +02:00
Wolfgang Bumiller
dd5495d6dc tree-wide: use 'dyn' for all trait objects 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-06-07 13:13:48 +02:00
Christian Ebner
e993db91cd pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder 
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:08:20 +02:00
Christian Ebner
1f319e766e src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:08:06 +02:00
Christian Ebner
9b77388f2a src/pxar/encoder.rs: minor refactoring 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:07:55 +02:00
Christian Ebner
a4cc8eb7c5 src/pxar/encoder.rs: check if ioctl supported by filesystem 
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:07:31 +02:00
Dietmar Maurer
4800df7ba6 src/api2/admin/datastore.rs: remove test-upload api 2019-06-07 07:52:11 +02:00
Dietmar Maurer
113898d7e9 src/api2/admin/datastore/upload.rs: remove unused upload api 2019-06-07 07:39:41 +02:00
Dietmar Maurer
4247fccb0f src/client/http_client.rs: automatically close connection in finish 2019-06-05 11:41:50 +02:00
Wolfgang Bumiller
cb4426b348 make h2 client connection cancellable 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-06-05 09:45:59 +02:00
Dietmar Maurer
986bef16be src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1 
So that we can include it in static doc strings.
 2019-06-05 08:41:20 +02:00
Dietmar Maurer
c9ec0956cf src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1 2019-06-05 08:12:13 +02:00
Dietmar Maurer
44c54845b3 src/tools/xattr.rs: fix test when run as root 2019-06-05 07:57:42 +02:00
Dietmar Maurer
7773ccc11f src/api2.rs: move backup api to /backup 2019-06-05 07:23:21 +02:00
Christian Ebner
97eeea3b4a src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:29:02 +02:00
Dietmar Maurer
1ce2f7cfb9 src/pxar/encoder.rs: style fix 2019-06-05 06:27:39 +02:00
Christian Ebner
e7b60a16c8 pxar: impl storing/dumping/restoring of quota project ids 
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:26:02 +02:00
Christian Ebner
7312ab9eef src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID 
Defines the types needed to store/restore quota project ids in pxar

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:23:06 +02:00
Christian Ebner
042babe4e7 src/tools/fs.rs: impl ioctl calls to get/set fsxattr 
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:22:49 +02:00
Dietmar Maurer
22e5aa1d3e src/bin/dump-backup-api.rs: helper to generate backup API docs 2019-06-04 13:12:42 +02:00
Dietmar Maurer
5e0f305142 src/api_schema/format.rs: depend on textwrap, new wrap_text helper 2019-06-04 12:42:02 +02:00
Dietmar Maurer
339ddfcbfa src/cli/command.rs: move doc generator code to src/api_schema/format.rs 2019-06-04 12:32:22 +02:00
Dietmar Maurer
ef39bf95b3 src/client/http_client.rs - upload_config: improve error message 2019-06-03 10:53:09 +02:00
Dietmar Maurer
ec8a9bb953 src/bin/proxmox-backup-client.rs: implement upload_config 2019-06-03 10:39:44 +02:00
Dietmar Maurer
39d6846e1e src/api2/admin/datastore/backup.rs: implement config file upload 2019-06-03 09:45:14 +02:00
Dietmar Maurer
1c0472e88f avoid compiler warnings 2019-06-03 06:43:47 +02:00
Dietmar Maurer
36898ffce6 src/backup/chunk_stream.rs: add optional chunk_size parameter 2019-05-30 13:28:24 +02:00
Dietmar Maurer
49ef316bcd src/bin/proxmox-backup-client.rs: backup_image() - pass correct chunk size 2019-05-30 12:52:12 +02:00
Dietmar Maurer
202cadc29e src/bin/proxmox-backup-client.rs: remove unused code 2019-05-30 12:47:35 +02:00
Dietmar Maurer
6af905c18f src/bin/proxmox-backup-client.rs: implement image/block-device upload 2019-05-30 12:46:01 +02:00
Dietmar Maurer
eb1804c567 src/bin/proxmox-backup-client.rs: use std lib instead of nix 2019-05-30 12:19:23 +02:00
Dietmar Maurer
6e907505aa src/bin/proxmox-backup-client.rs: fix file type test 2019-05-30 12:08:34 +02:00
Dietmar Maurer
96482891ae src/api2/admin/datastore/backup/environment.rs: log upload statistics 2019-05-30 09:21:19 +02:00
Dietmar Maurer
642322b433 src/api2/admin/datastore/backup.rs: use separate api entry points for chunk upload 
So that we can provide better statistics (patches follows).
 2019-05-30 09:21:19 +02:00
Wolfgang Bumiller
390e83c9b9 typo fix and doc improvement 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-29 11:57:28 +02:00
Wolfgang Bumiller
7fb49397d5 doc fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-29 11:56:14 +02:00
Wolfgang Bumiller
382609b04e tools: add Cancellable, start a futures submodule 
To make a future cancellable, use:
let (future, canceller) =
    crate::tools::futures::cancellable(future);

Proceed with using `future` as usual, `canceller` is
clonable and can cancel the future via the `.cancel()`
method.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-29 11:18:29 +02:00
Wolfgang Bumiller
0d32d71fb7 tools: add AsyncMutex::new_locked 
Allows creating a pre-locked mutex, returning the mutex and
a guard.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-29 11:18:29 +02:00
Dietmar Maurer
e6389f4e75 src/api2/admin/datastore/backup/environment.rs: do not allow empty backups (no files) 2019-05-29 10:38:57 +02:00
Dietmar Maurer
39e60bd6fa src/api2/admin/datastore/backup.rs: improve debug logging 2019-05-29 10:17:38 +02:00
Dietmar Maurer
4ae6551c78 src/api_schema/router.rs: disable debug println 2019-05-29 09:42:16 +02:00
Dietmar Maurer
a42d1f55c3 src/api2/admin/datastore/backup.rs: add debug flag 2019-05-29 09:35:21 +02:00
Dietmar Maurer
976595e1a9 src/backup/dynamic_index.rs: split class DynamicIndexWriter 2019-05-29 08:49:57 +02:00
Dietmar Maurer
cb0708dd46 src/backup/fixed_index.rs: remove ChunkStat from struct 2019-05-29 07:08:34 +02:00
Dietmar Maurer
01af11f340 src/backup/fixed_index.rs: improve consistency checks 2019-05-29 07:02:45 +02:00
Christian Ebner
9b38443314 pxar: change logic and impl all command line flags for xattrs/fcaps/acls 
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-29 06:51:20 +02:00
Dietmar Maurer
60e589a111 src/api2/admin/datastore/backup.rs: log sucessful finish call 2019-05-28 12:55:22 +02:00
Dietmar Maurer
c4ff3dcefd src/bin/proxmox-backup-client.rs: use new BackupClient to upload data 2019-05-28 10:12:44 +02:00
Dietmar Maurer
10241c20ea src/tools.rs: implement image_size helper 2019-05-28 09:35:08 +02:00
Dietmar Maurer
006f3ff407 src/api2/admin/datastore/backup/environment.rs: add more consistency checks 2019-05-28 09:21:31 +02:00
Dietmar Maurer
3dc5b2a203 src/client/http_client.rs: use start of chunk as offset 2019-05-28 09:21:31 +02:00
Dietmar Maurer
8de20e5c1f src/client/http_client.rs: code cleanup 2019-05-28 08:53:27 +02:00
Dietmar Maurer
fc14b84937 src/backup/fixed_index.rs: improve bound checks inside add_digest 2019-05-28 08:47:03 +02:00
Dietmar Maurer
e1225de467 src/backup/fixed_index.rs: store index_length to avoid code duplication 2019-05-28 08:27:57 +02:00
Dietmar Maurer
e3d525feb7 src/api2/admin/datastore/backup.rs: fix typo 2019-05-28 07:13:19 +02:00
Christian Ebner
af309d4d5c src/bin/pxar.rs: set archive permissions to 0o640 
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-28 06:24:47 +02:00
Christian Ebner
ab87f167f1 src/pxar/encoder.rs: Refactor file stat 
Introduce helper functions to check file stats

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-28 06:23:49 +02:00
Christian Ebner
578b601159 src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs. 
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-28 06:22:16 +02:00
Dietmar Maurer
a42fa400ee src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api 2019-05-28 06:18:55 +02:00
Dietmar Maurer
29ae5c86a2 src/backup/fixed_index.rs: make chunk_size public 2019-05-28 06:18:24 +02:00
Dietmar Maurer
7dd1bcac58 src/client/http_client.rs: correctly parse error response in download_chunk_list 2019-05-27 09:35:30 +02:00
Dietmar Maurer
a1e7cff3ea src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk 2019-05-27 07:57:43 +02:00
Dietmar Maurer
771953f9eb src/api2/admin/datastore/backup.rs: use smaller window sizes 2019-05-27 07:53:40 +02:00
Dietmar Maurer
174ad378d8 src/client/http_client.rs: fix request pipelining 2019-05-27 07:24:32 +02:00
Dietmar Maurer
624362226e src/client/merge_known_chunks.rs: use MergedChunkInfo as input 2019-05-26 10:52:56 +02:00
Dietmar Maurer
05cba08c9c src/client/http_client.rs: append chunks in correct order 
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
 2019-05-26 09:57:34 +02:00
Dietmar Maurer
417cb0731e src/api2/admin/datastore/backup.rs: verify chunk offset 
This clearly shows that the current approach does not work - seems we get
chunks out of order.
 2019-05-26 08:49:29 +02:00
Dietmar Maurer
34114e2606 src/server/rest.rs: correctly verify json parameters 2019-05-24 09:10:59 +02:00
Dietmar Maurer
eb2bdd1bcc src/client/http_client.rs: encode parameters inside body 
To avoid problems with long uris.
 2019-05-24 08:32:55 +02:00
Dietmar Maurer
9bb675ec23 src/client/merge_known_chunks.rs: reduce RAM usage 
There is no need to store chunk data for known chunks.
 2019-05-24 07:42:58 +02:00
Dietmar Maurer
ecb21b4794 src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough 2019-05-24 07:36:09 +02:00
Christian Ebner
2dbba78b98 src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-24 06:27:24 +02:00
Christian Ebner
6a19448004 src/pxar/encoder.rs: impl support to store ACLs in pxar achives 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-24 06:21:33 +02:00
Christian Ebner
7cfaade7c8 src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-24 06:16:55 +02:00
Christian Ebner
b344461b33 pxar: pass feature_flags to encoder/decoder instead of individual flags 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-24 06:11:36 +02:00
Christian Ebner
687407741b src/tools/xattr.rs: add test for is_valid_xattr_name() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-24 06:11:36 +02:00
Christian Ebner
357e4614e2 Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-24 06:11:36 +02:00
Wolfgang Bumiller
c05a8c8d18 doc-test fixup 
cargo test by default compiles and runs all code snippets
found in the documentation...

oops...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-23 15:22:33 +02:00
Wolfgang Bumiller
cc84a830c5 RFC: schema: make enums static 
I don't see a reason to allow these to be dynamically
modifiable.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-23 12:38:51 +02:00
Dietmar Maurer
aa1b2e04fe src/client/merge_known_chunks.rs: merge known chunks 
To decrease the number of api calls required...
 2019-05-23 12:31:09 +02:00
Dietmar Maurer
91320f0879 src/client/http_client.rs: use ChunkInfo streams 
This will make out of order uploads possible...
 2019-05-23 12:31:09 +02:00
Wolfgang Bumiller
8ea3b1d188 tree-wide: use the new vec/io tools modules 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-23 09:49:10 +02:00
Wolfgang Bumiller
9cdda3f7c7 tools: add helpful vector and read operations 
After importing the I/O ops trait via:
    use crate::tools::io::ops::*;

Instead of:
    let mut buffer = vec![0u8; 65536];
    file.read_exact(&mut buffer)?;
use:
    let buffer = file.read_exact_allocated(65536)?;

After importing the vector helpers via:
    use crate::tools::vec::{self, ops::*};

For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
    let mut buffer = vec![0u8; len];
use:
    let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-05-23 09:49:10 +02:00
Dietmar Maurer
8bea85b42e src/api2/admin/datastore/backup.rs: verify file size and chunk count on close 2019-05-23 08:50:36 +02:00
Dietmar Maurer
1179e15842 src/server/rest.rs: improve error handling 2019-05-23 08:15:32 +02:00
Dietmar Maurer
907f5bb0d8 src/api2/admin/datastore/backup/service.rs: improve error logging 2019-05-23 08:05:39 +02:00
Dietmar Maurer
32e1bf4260 src/bin/upload-speed.rs: code cleanup 2019-05-23 07:20:05 +02:00
Christian Ebner
0d9bab05ab src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-23 06:40:50 +02:00
Christian Ebner
4add706092 src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-23 06:40:32 +02:00
Christian Ebner
54b14ad741 src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-23 06:40:09 +02:00
Christian Ebner
1201360def src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-23 06:39:16 +02:00
Christian Ebner
8bc8791338 src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-23 06:37:37 +02:00
Christian Ebner
a45a123950 src/pxar/encoder: Use encoder feature_flags, do not store them within the archive. 
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-23 06:34:36 +02:00
Dietmar Maurer
d6f204eda6 src/client/http_client.rs: re-order code - public functions first 2019-05-23 06:28:10 +02:00
Dietmar Maurer
9af37c8f0e src/client/http_client.rs: move low level H2 code into separate class 2019-05-22 17:28:25 +02:00
Dietmar Maurer
d2c48afc6e src/client/http_client.rs: code cleanup 2019-05-22 13:24:33 +02:00
Dietmar Maurer
6ab34afa88 src/client/http_client.rs: cleanup h2 backup client 2019-05-22 13:05:51 +02:00
Dietmar Maurer
a55fcd740f src/api2/admin/datastore/backup.rs: suse POST for finish() api 2019-05-22 13:04:58 +02:00
Dietmar Maurer
de6742c04f src/bin/h2test.rs: removed 
This test does no work anymore.
 2019-05-22 11:14:56 +02:00
Dietmar Maurer
5a0b521809 src/api2/admin/datastore/backup.rs: remove test callbacks 2019-05-22 10:55:27 +02:00
Christian Ebner
de61bc9264 src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr 
As git does not preserve xattrs they have to be set before they can be read.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-22 10:01:10 +02:00
Dietmar Maurer
553610b43e src/client/http_client.rs: implement download_chunk_list 2019-05-22 09:46:02 +02:00
Dietmar Maurer
0cc0fffd1e src/backup/chunk_stream.rs: use BytesMut in ChunkStream 2019-05-22 09:39:02 +02:00
Dietmar Maurer
c052be5c86 src/backup/chunk_stream.rs: use more generics 2019-05-22 09:18:05 +02:00
Dietmar Maurer
169c0e060f src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut) 2019-05-22 09:05:35 +02:00
Dietmar Maurer
3be3f3dcc4 src/backup/chunk_stream.rs: fix behaviour for large input buffers 2019-05-22 08:05:27 +02:00
Dietmar Maurer
02fa54ff3c src/backup/chunk_stream.rs: assemble data with correct order 2019-05-22 07:44:51 +02:00
Dietmar Maurer
d0afeaa831 src/api2/admin/datastore/backup.rs: improve error message 2019-05-22 07:43:54 +02:00
Dietmar Maurer
11867a2b54 src/tools.rs: fix hex_to_digest 2019-05-21 13:42:13 +02:00
Dietmar Maurer
7f3d2ffab9 src/backup/index.rs: rename ChunkListReader into DigestListEncoder 2019-05-21 12:28:44 +02:00
Dietmar Maurer
ee53955f52 src/backup/index.rs: implement DigestListDecoder 2019-05-21 12:21:22 +02:00
Dietmar Maurer
a95849321c src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks 2019-05-21 09:54:11 +02:00
Dietmar Maurer
40f4e198a8 src/backup/dynamic_index.rs: add chunk_info method 2019-05-21 09:52:19 +02:00
Christian Ebner
596182056b src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-21 06:15:30 +02:00
Christian Ebner
b303057a19 src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-21 06:06:26 +02:00
Christian Ebner
bee8d8ea78 src/tools/xattr.rs: add functions name_store and security_capability 
These functions allow to check if the name for xattrs/fcaps are valid.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-21 06:05:31 +02:00
Dietmar Maurer
a09c0e38d8 src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks 2019-05-20 18:05:10 +02:00
Dietmar Maurer
97f03eff13 src/client/pxar_backup_stream.rs: fix docu 2019-05-20 17:29:00 +02:00
Dietmar Maurer
82ab72304e src/client/http_client.rs: implement upload_stream 2019-05-20 14:19:24 +02:00
Dietmar Maurer
2698e8a514 src/client/pxar_backup_stream.rs: close pipe inside drop 2019-05-20 11:20:33 +02:00
Dietmar Maurer
f2269d8f99 src/tools.rs: implement hex_to_digest 2019-05-19 12:15:46 +02:00
Dietmar Maurer
751030aa7c src/bin/test_chunk_speed2.rs: test max chunk size 2019-05-19 11:09:02 +02:00
Dietmar Maurer
ff77dbbea3 Revert "src/backup/chunk_stream.rs: simplify code" 
This reverts commit e058744d8f.
The optimization was wrong, and produces wrong chunk boundaries.
 2019-05-19 11:05:56 +02:00
Dietmar Maurer
178ac042c4 src/api_schema/schema.rs: improve error messages 2019-05-19 10:18:51 +02:00
Dietmar Maurer
0c516b123d src/client/pxar_backup_stream.rs: use WrappedStreamReader 
to make it fully async ...
 2019-05-18 15:13:31 +02:00
Dietmar Maurer
c6e28b66c6 src/client/pxar_backup_stream.rs: use 1M pipe buffer size 2019-05-18 12:57:43 +02:00
Dietmar Maurer
6276448519 src/bin/test_chunk_speed2.rs: test chunker speed with real data 2019-05-18 11:59:17 +02:00
Dietmar Maurer
e058744d8f src/backup/chunk_stream.rs: simplify code 2019-05-18 10:55:46 +02:00
Dietmar Maurer
8a7cc7565f src/backup/chunk_stream.rs: implement fixed size chunk stream 2019-05-18 10:55:46 +02:00
Dietmar Maurer
9c9ad941c0 src/client/http_client.rs: improve speed test 
pipeline responses to a separate async channel ...
 2019-05-18 10:55:46 +02:00
Christian Ebner
2dcdd3b427 src/tools/xattrs.rs: impl libc wrapper functions to get/set xattrs and fcaps 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-16 11:14:21 +02:00
Christian Ebner
de2016d58a src/pxar/format_definitions.rs: add format definitions for xattrs and fcaps 
In addition to the format definition, the traits needed for sorting of xattr
entries by name are derived.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-16 11:14:13 +02:00
Christian Ebner
dec1ff18b8 src/tools/acl.rs: impl add_entry_full for ACL and set_permissions for ACLEntry 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-16 11:10:06 +02:00
Dietmar Maurer
adec8ea263 src/api2/admin/datastore/backup.rs: add speedtest api, improve upload speed 
We need to disable tcp Nagle algorythm (set_nodelay), and use larger window size for http2
 2019-05-16 10:24:23 +02:00
Dietmar Maurer
372724afea src/api2/admin/datastore/backup.rs: implement finish_backup 2019-05-15 12:58:55 +02:00
Dietmar Maurer
bb105f9dae src/api2/admin/datastore/backup.rs: improve logging 2019-05-15 10:09:46 +02:00
Dietmar Maurer
25ddc15fe0 src/client/pipe_to_stream.rs: add missing file 2019-05-15 09:07:19 +02:00
Dietmar Maurer
a20772525f src/api2/admin/datastore/backup.rs: implement close for dynamic writers 2019-05-15 07:58:05 +02:00
Dietmar Maurer
4e93f8c164 src/api2/types.rs: add schema/format for file names 2019-05-15 07:14:08 +02:00
Dietmar Maurer
e3dbd41b20 src/client/http_client.rs: add flow control for h2 upload 2019-05-14 14:54:21 +02:00
Dietmar Maurer
97f22ce57d src/client/http_client.rs: implement h2 upload 
without flow control so far ...
 2019-05-14 12:58:59 +02:00
Dietmar Maurer
dafc27aeff src/backup/chunk_stream.rs: async chunk stream 2019-05-14 10:05:29 +02:00
Dietmar Maurer
c0e22141ed src/client/http_client.rs: derive Clone for H2Client 2019-05-14 09:00:24 +02:00
Christian Ebner
25e205a1d3 src/tools.rs: add mod acl to tools 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-14 06:34:24 +02:00
Christian Ebner
d22096ef4b src/tools/acl.rs: impl POSIX access control list (ACL) manipulation tools 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-05-14 06:28:42 +02:00
Dietmar Maurer
1133fe9a63 src/server/rest.rs: do not log 1xx status codes as errors 2019-05-14 06:23:22 +02:00
Dietmar Maurer
850ac6d0f0 src/client/http_client.rs: directly return H2Client on upgrade 2019-05-13 16:44:59 +02:00
Dietmar Maurer
10130cf48c src/client/http_client.rs: cleanup, remove debug println 2019-05-13 12:11:18 +02:00
Dietmar Maurer
b57cb26406 src/client/http_client.rs: implement http2 client wrapper 2019-05-13 10:27:22 +02:00
Dietmar Maurer
9e391bb7f0 src/client/http_client.rs: allow to pass parameters (encode them as query string) 2019-05-13 09:12:03 +02:00
Dietmar Maurer
fbb798f6d2 src/api2/admin/datastore/backup.rs: ignore errors from last_backup 2019-05-13 07:06:56 +02:00
Dietmar Maurer
d36113665b src/api2/admin/datastore/backup.rs: implement chunk index download from previous backup 2019-05-11 16:05:50 +02:00
Dietmar Maurer
457531e7fe src/backup/index.rs: new ChunkListReader 2019-05-11 16:01:42 +02:00
Dietmar Maurer
af1d68185c src/api2/admin/datastore/backup.rs: add missing 'archive-name' parameter to schema 2019-05-11 12:25:15 +02:00
Dietmar Maurer
6b95c7df07 src/api2/admin/datastore/backup/environment.rs: do not store path 2019-05-11 12:13:44 +02:00
Dietmar Maurer
51a4f63ff7 src/backup/datastore.rs: move last_backup to BackupInfo 2019-05-11 12:07:09 +02:00
Dietmar Maurer
b02a52e3a5 src/backup/datastore.rs: impl last_backup helper 2019-05-11 11:36:46 +02:00
Dietmar Maurer
c09775011a src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups 2019-05-11 10:19:34 +02:00
Dietmar Maurer
f9578f3c79 backup-api: allow to create DynamicIndexWriter and add chunks to to 2019-05-10 10:25:40 +02:00
Dietmar Maurer
35a2d8a6a6 src/backup/backup_info.rs: Derive Clone 2019-05-10 06:59:23 +02:00
Dietmar Maurer
0506d1ed04 upload_chunk.rs: add comment about possible unnecessary code 2019-05-10 06:15:25 +02:00
Dietmar Maurer
5f42ef009a cleanup: remove unused code 2019-05-09 18:11:39 +02:00
Dietmar Maurer
b4b63e520d handle_async_api_request: put rpcenv into a Box 
So that we can pass rpcenv into futures.
 2019-05-09 18:01:24 +02:00
Dietmar Maurer
2c41096a4b src/api2/admin/datastore/backup/upload_chunk.rs: fix maximal chunk size 2019-05-09 13:29:50 +02:00
Dietmar Maurer
78b5191550 src/server/formatter.rs: further cleanups and renaming ... 2019-05-09 13:28:26 +02:00
Dietmar Maurer
d55037e471 src/server/formatter.rs: rename format_result to format_data 
To avoid confusions with Rust Result type.
 2019-05-09 13:15:15 +02:00
Dietmar Maurer
21ee7912fa src/api2/admin/datastore/backup.rs: implement upload chunk 2019-05-09 13:06:09 +02:00
Dietmar Maurer
4ebf0eabb0 src/api2/types.rs: use separate file for schema definitions 2019-05-09 07:44:09 +02:00
Dietmar Maurer
bd1507c4fa src/api2/admin/datastore/backup/service.rs: move service code into extra file 2019-05-08 17:36:19 +02:00
Dietmar Maurer
0aadd40b4f src/api2/admin/datastore/backup.rs: add required parameters, cleanups 2019-05-08 13:17:23 +02:00
Dietmar Maurer
d95ced64ad src/api2/admin/datastore/backup/environment.rs: use separate file 2019-05-08 12:41:58 +02:00
Dietmar Maurer
ca60c371e2 api: rename h2upload to backup 2019-05-08 12:35:15 +02:00
Dietmar Maurer
d906b1124f src/api2/admin/datastore/h2upload.rs: code cleanups 2019-05-08 11:36:43 +02:00
Dietmar Maurer
58c8d7d91f rc/api2/admin/datastore/h2upload.rs: implement BackupEnvironment 
To pass arbitrary information/state to api methods.
 2019-05-08 11:26:54 +02:00
Dietmar Maurer
f757b30efc src/server/rest.rs: use generics to pass RpcEnvironment 2019-05-08 11:09:01 +02:00
Dietmar Maurer
2edc341b29 src/tools.rs: Add AsAyn Trait 2019-05-08 11:07:36 +02:00
Dietmar Maurer
090ac9f79a src/api2/admin/datastore/h2upload.rs: avoid unwrap 2019-05-08 08:21:18 +02:00
Dietmar Maurer
4e3da4b366 src/api2/admin/datastore/h2upload.rs: do not log client address 2019-05-08 08:15:55 +02:00
Dietmar Maurer
e3a44552ee src/api2/admin/datastore/h2upload.rs: code cleanups 2019-05-08 08:12:49 +02:00
Dietmar Maurer
a66ab8ae5e src/api2/admin/datastore/h2upload.rs: make worker abortable 2019-05-08 07:14:40 +02:00
Dietmar Maurer
72375ce6c6 src/api2/admin/datastore/h2upload.rs: log to worker task 2019-05-07 14:07:00 +02:00
Dietmar Maurer
d9bd06eae8 src/api2/admin/datastore/h2upload.rs: use WorkerTask 2019-05-07 13:42:00 +02:00
Dietmar Maurer
52cf506e48 src/api2/admin/datastore/h2upload.rs: implement async test api call 2019-05-07 12:26:55 +02:00
Dietmar Maurer
92ac375aff src/api2/admin/datastore/h2upload.rs: improve http2 service 2019-05-07 11:24:44 +02:00
Dietmar Maurer
78f7e225e4 src/server/rest.rs: make handle_(a)sync_api_request public 2019-05-07 11:23:52 +02:00
Dietmar Maurer
75097495ad RestEnvironment: derive Clone 2019-05-07 11:09:18 +02:00
Dietmar Maurer
01bf3b7b5f move find_method to Router class 2019-05-07 11:08:30 +02:00
Dietmar Maurer
3578d99f3e move normalize_path to tools::normalize_uri_path 2019-05-07 09:44:34 +02:00
Dietmar Maurer
96e95fc179 src/api2/admin/datastore/h2upload.rs: test if version < 2 
We can only upgrade http 1.x connections.
 2019-05-06 10:29:34 +02:00
Dietmar Maurer
45cdce069e src/client/http_client.rs: request password at creation time 2019-04-30 11:44:35 +02:00
Dietmar Maurer
e45afdff9c src/server/state.rs: use new BroadcastData helper 2019-04-30 10:21:48 +02:00
Dietmar Maurer
824b5ee4ee src/tools/broadcast_future.rs: cleanup, decompose into two classes 
In order to make it more usable.
 2019-04-30 08:57:20 +02:00
Wolfgang Bumiller
d82ed9b0f5 tools: add AsyncMutex 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-29 14:19:03 +02:00
Dietmar Maurer
3a9bea3918 src/bin/h2test.rs: test h2 upgrade 2019-04-29 11:59:18 +02:00
Dietmar Maurer
cf639a47de rc/client/http_client.rs: add experimental h2 upgrade code 2019-04-29 11:57:58 +02:00
Dietmar Maurer
5a2df00004 src/client/http_client.rs: cleanup, make login fully async 2019-04-28 10:55:03 +02:00
Dietmar Maurer
3dceb9b304 src/tools/broadcast_future.rs: add new constructor new_oneshot() 
To simplify usage. Also added a test case.
 2019-04-27 10:56:49 +02:00
Dietmar Maurer
152764ec15 src/api2/admin/datastore/h2upload.rs: add test code for http2 2019-04-27 08:57:35 +02:00
Dietmar Maurer
490be29ed6 src/tools/broadcast_future.rs: new helper class 2019-04-26 17:57:57 +02:00
Wolfgang Bumiller
e888155712 proxy: allow .key/.pem files instead of a pkcs12/.pfx file 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-26 10:47:33 +02:00
Christian Ebner
8b7c0def84 src/tools/procfs.rs: clean up tab indentation and use whitespaces instead 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-25 13:14:52 +02:00
Wolfgang Bumiller
5e5eed5c3b use double-fork for reload 
To ensure the new process' parent is pid 1, so systemd won't
complain about supervising a process it does not own.

Fixes the following log spam on reloads:
Apr 25 10:50:54 deb-dev systemd[1]: proxmox-backup.service: Supervising process 1625 which is not our child. We'll most likely not notice when it exits.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-25 11:02:12 +00:00
Wolfgang Bumiller
efd1536eb7 add a wrapper around nix::unistd::pipe2 
Using O_CLOEXEC by default, and returning Fd handles to
ensure they get dropped on bail!() or panic!() if the RawFds
aren't used yet.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-25 11:02:12 +00:00
Wolfgang Bumiller
897982e237 tools: add read/write convenience helpers 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-25 11:02:12 +00:00
Wolfgang Bumiller
d98c9a7a38 use service Type=notify 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-25 11:01:33 +00:00
Wolfgang Bumiller
9c351a36c4 tools/daemon: add sd_notify wrapper 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-25 11:01:28 +00:00
Dietmar Maurer
d8871f2f18 src/tools/procfs.rs: avoid useless allocations 2019-04-24 09:35:15 +02:00
Christian Ebner
ae694f44ef src/tools/procfs.rs: add tests for read_proc_net_route() and read_proc_net_ipv6_route() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-24 06:43:52 +02:00
Christian Ebner
ec8db837fb src/tools/procfs.rs: refactor read_proc_net_route() and read_proc_net_ipv6_route() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-24 06:43:32 +02:00
Christian Ebner
eb46c3d1ec src/tools/procfs.rs: implement read_proc_net_ipv6_route() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-23 14:31:23 +02:00
Dietmar Maurer
6358b68236 cleanup: avoid compiler warnings 2019-04-23 12:17:24 +02:00
Dietmar Maurer
e06ffb10d1 src/api_schema/router.rs: fix typo (subdira -> subdir) 2019-04-16 12:20:17 +02:00
Dietmar Maurer
13f1cc17ea src/api_schema/router.rs: implement list_subdirs() helper 2019-04-16 12:07:02 +02:00
Wolfgang Bumiller
062d4916ff api_schema: allow generic api handler functions 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-16 11:22:23 +02:00
Christian Ebner
286f0d4099 src/tools/procfs.rs: implement read_proc_net_route() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-15 11:13:16 +02:00
Dietmar Maurer
1a6c9415ae src/api2/node/services.rs: add proxmox-backup-proxy service 2019-04-15 09:48:02 +02:00
Dietmar Maurer
75bc49bed7 src/server/worker_task.rs: implement abort channel 2019-04-15 09:38:05 +02:00
Dietmar Maurer
912524752e src/api2/admin/datastore/pxar.rs: create worker task for file upload 2019-04-15 08:38:14 +02:00
Dietmar Maurer
1be71fb02b src/server/worker_task.rs_ do not write ERROR: prefix into task index 2019-04-15 08:37:13 +02:00
Dietmar Maurer
418def7a78 src/server/worker_task.rs: move common code into log_result 
Make new() pub
 2019-04-15 07:49:03 +02:00
Dietmar Maurer
37b87869b5 src/server/upid.rs: check for illegal characters 2019-04-15 07:47:51 +02:00
Christian Ebner
d2981e2738 src/tools/procfs.rs: implement read_proc_net_dev() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-13 11:52:59 +02:00
Christian Ebner
13abfdc57a src/tools/procfs.rs: implement read_memory_usage() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-13 11:52:01 +02:00
Christian Ebner
3005ef48b6 src/tools/procfs.rs: Rename variable and struct to be more consistent 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-12 12:13:34 +02:00
Christian Ebner
cc93bacb72 src/tools/procfs.rs: simplify read_cpuinfo() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-12 12:13:14 +02:00
Dietmar Maurer
f2b99c34f7 src/api2/admin/datastore.rs: implement API to return last GC status 2019-04-11 12:04:25 +02:00
Dietmar Maurer
59961b8954 src/server/command_socket.rs: check control socket permissions 2019-04-11 10:51:59 +02:00
Dietmar Maurer
ba70040dbf src/server/worker_task.rs: change task list sort order - latest on top 2019-04-11 09:43:33 +02:00
Dietmar Maurer
d3f4c08f33 src/server/worker_task.rs: catch panics in worker threads 2019-04-11 07:56:06 +02:00
Wolfgang Bumiller
620dccf1a1 tools/daemon: dup the TcpListener file descriptor 
Now that we let hyper shutdown gracefully we need an owned
version of the listening socket to prevent it from closing
before running the reload preparations.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-10 15:20:10 +02:00
Wolfgang Bumiller
d96bb7f163 tools: add Fd helper 
stores a raw file descriptor with a drop handler for
safekeeping in closures

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-04-10 15:20:10 +02:00
Christian Ebner
1cd33633aa /src/tools: Cleanup tempfiles in folder 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-10 13:56:22 +02:00
Dietmar Maurer
a665dea128 src/api2/node/tasks.rs: implement stop_task api 2019-04-10 13:55:05 +02:00
Dietmar Maurer
98a181f055 src/server/worker_task.rs: log message when we call request_abort() 2019-04-10 13:54:28 +02:00
Christian Ebner
2b6a3e1396 src/tools/procfs.rs: implement read_cpuinfo() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-10 13:11:11 +02:00
Dietmar Maurer
321070b4fa src/server/worker_task.rs: implement abort_worker (via command_socket) 2019-04-10 12:42:24 +02:00
Dietmar Maurer
cfb2d3c17c src/server/command_socket.rs: correctly handle/spawn handle parallel connections 2019-04-10 11:05:00 +02:00
Dietmar Maurer
9b002cbc5f src/server/worker_task.rs: use abstract socket 2019-04-10 09:03:17 +02:00
Dietmar Maurer
5f550fd99f start hyper server using with_graceful_shutdown() 
Without, hyper keeps some futures running, and the server does not
correctly shutdown.
 2019-04-10 08:24:32 +02:00
Dietmar Maurer
7f2078de59 src/tools/procfs.rs: avoid compiler warning 2019-04-10 06:40:29 +02:00
Christian Ebner
1f6fbf1c57 src/tools/procfs.rs: implement read_meminfo() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-09 15:21:49 +02:00
Dietmar Maurer
c360bd73d3 src/api2/node/tasks.rs: fix task status api 2019-04-09 15:12:20 +02:00
Dietmar Maurer
0bfd87bc5e src/server/worker_task.rs: speedup upid_read_status() 2019-04-09 15:11:48 +02:00
Dietmar Maurer
d8d40dd0c2 src/api2/node.rs: add node parameter 2019-04-09 14:43:30 +02:00
Dietmar Maurer
7ae0bf52dd src/server/command_socket.rs: code cleanup - fix error message 2019-04-09 12:47:42 +02:00
Dietmar Maurer
d607b8861b src/server/worker_task.rs: implement task control socket 2019-04-09 12:15:06 +02:00
Dietmar Maurer
e201753629 src/server/command_socket.rs: implement auto_remove flag 
Remove the socket file on close.
 2019-04-09 11:47:23 +02:00
Dietmar Maurer
634132fe10 src/server/upid.rs: moved code into separate file 2019-04-08 18:43:26 +02:00
Dietmar Maurer
78a39e0517 src/server/command_socket.rs: simple command socket 2019-04-08 17:59:39 +02:00
Dietmar Maurer
e3f41f2165 src/tools/daemon.rs: use new ServerState handler 2019-04-08 14:00:23 +02:00
Dietmar Maurer
7a630df709 implement server state/signal handling, depend on tokio-signal 2019-04-08 13:59:07 +02:00
Dietmar Maurer
5a12c0e2fb src/api2/node/tasks.rs: start upid API 2019-04-07 14:36:57 +02:00
Dietmar Maurer
b75b968147 src/server/worker_task.rs: implement is_active() 2019-04-07 14:36:25 +02:00
Dietmar Maurer
4494d07819 src/server/worker_task.rs: make log_path a method on UPID 2019-04-07 13:51:46 +02:00
Dietmar Maurer
d2a2e02bb7 src/api2/node/tasks.rs: implement userfilter 2019-04-07 13:17:19 +02:00
Dietmar Maurer
3c3bee2e32 src/api2/node/tasks.rs: correctly handle start parameter 2019-04-07 12:41:24 +02:00
Dietmar Maurer
063ca5be77 src/api2/node/tasks.rs: implement list_tasks 2019-04-07 12:18:58 +02:00
Dietmar Maurer
d4b59ae0b8 src/backup/datastore.rs: pass WorkerTask to GC 
So that we can log to the task log file.
 2019-04-06 17:57:38 +02:00
Dietmar Maurer
35950380a9 src/server/worker_task.rs: carefully handle file permissions 2019-04-06 17:53:12 +02:00
Dietmar Maurer
1619a72063 src/tools.rs: implement create_dir_chown 
Combinded mkdir with chown.
 2019-04-06 16:50:23 +02:00
Dietmar Maurer
eea8131952 src/tools.rs: implement file_set_contents_full() 2019-04-06 13:53:43 +02:00
Dietmar Maurer
0f778e0652 src/api2/admin/datastore.rs: use new WorkerTask::new_thread() 2019-04-06 11:27:23 +02:00
Dietmar Maurer
244abab769 src/tools/file_logger.rs: fix test 2019-04-06 11:24:37 +02:00
Dietmar Maurer
660c684640 src/server/worker_task.rs: return upid_str 2019-04-06 11:23:53 +02:00
Dietmar Maurer
93aebb38bc src/server/worker_task.rs: implement read_task_list() 2019-04-06 10:17:11 +02:00
Dietmar Maurer
4a36a2c999 src/server/worker_task.rs: improve UPID format description 2019-04-06 09:36:50 +02:00
Dietmar Maurer
882594c5e9 src/server.rs: improve crate layout 2019-04-06 09:17:25 +02:00
Dietmar Maurer
4b01c983f0 src/server/worker_task.rs: implement persistent task list 2019-04-05 12:37:35 +02:00
Christian Ebner
a2474d328f src/tools/procfs.rs: implement read_proc_uptime and read_proc_uptime_ticks 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-05 06:22:21 +02:00
Christian Ebner
08daa81af6 procfs: implement check_process_running and check_process_running_pstart 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-04-04 13:54:17 +02:00
Dietmar Maurer
9a0d0ff774 server/worker_task.rs: implement UPID parser 2019-04-04 13:49:35 +02:00
Dietmar Maurer
49cf9f3d0b src/tools.rs: file_get_json() - add new default parameter 2019-04-04 12:24:18 +02:00
Dietmar Maurer
79ad7b7ec2 src/tools/procfs.rs: split regex into several lines 2019-04-04 09:25:19 +02:00
Dietmar Maurer
479f6e40ec src/server/worker_task.rs: implement task registry 2019-04-04 08:58:39 +02:00
Dietmar Maurer
52ddc2522d src/tools/procfs.rs: use libc::pid_t 2019-04-04 08:05:43 +02:00
Dietmar Maurer
d3d7ab1273 src/tools/file_logger.rs: new - accept AsRef<Path> 2019-04-03 14:13:33 +02:00
Dietmar Maurer
3c2012f97a src/tools/procfs.rs: start helper tools to read procfs 2019-04-03 13:39:20 +02:00
Dietmar Maurer
3489936ebb src/tools/file_logger.rs: change timestamp format to rfc3339 2019-04-03 08:58:43 +02:00
Dietmar Maurer
92da93b245 abort GC on server shutdown 2019-04-01 12:13:02 +02:00
Dietmar Maurer
9136f857cc add global var to indicate server shutdown requests 2019-04-01 12:05:11 +02:00
Dietmar Maurer
3e6a7deeaa src/api2/admin/datastore.rs: start GC in a separate thread 2019-04-01 12:02:46 +02:00
Dietmar Maurer
ad281d1bd9 www/DataStoreConfig.js: add button to start GC 2019-04-01 08:08:34 +02:00
Dietmar Maurer
49d123ee3a src/server/rest.rs: use formatter to encode errors 2019-04-01 08:04:12 +02:00
Dietmar Maurer
7d4ef127cb src/server/rest.rs: generate csrf token if we have a valid ticket 
This is important if the user reloads the browser page.
 2019-04-01 07:52:30 +02:00
Dietmar Maurer
11861a482d src/backup/chunk_store.rs: fix GC 
Added option to get oldest_writer timestamp from ProcessLocker.
 2019-03-31 17:21:36 +02:00
Dietmar Maurer
d85987aeeb fix last commit: the filename var was not ment to be removed, sorry 2019-03-31 16:16:14 +02:00
Dietmar Maurer
15a77c4c2e src/backup/chunk_store.rs: avoid create/unlink race 2019-03-31 10:03:01 +02:00
Dietmar Maurer
c374f05499 src/backup.rs: improve GC problem description 2019-03-31 09:44:35 +02:00
Dietmar Maurer
c8ec450e37 src/backup.rs: start explaining different GC algorithm 2019-03-30 17:21:40 +01:00
Dietmar Maurer
8a4757342c src/backup.rs: describe the garbage collection problem 2019-03-30 16:26:52 +01:00
Dietmar Maurer
10eea49d81 src/backup/fixed_index.rs: store reference to file 
Keep it open and locked while index is in use.
 2019-03-27 10:17:03 +01:00
Dietmar Maurer
c597a92c81 index file readers: aquire shared lock 2019-03-27 10:11:14 +01:00
Dietmar Maurer
0465218953 src/backup.rs: add documentation about ChunkStore locking 2019-03-22 10:14:50 +01:00
Dietmar Maurer
43b1303398 datastore: use new ProcessLocker 
To make sure only one process runs garbage collection while having active writers.
 2019-03-22 09:42:15 +01:00
Dietmar Maurer
abfc001f25 src/tools/process_locker.rs: new() - use generic path parameter 2019-03-22 09:41:32 +01:00
Dietmar Maurer
a650f50301 src/tools/process_locker.rs: implement inter-process reader-writer locks 2019-03-22 08:05:51 +01:00
Wolfgang Bumiller
eb90c9e372 tools: style fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-20 09:57:13 +01:00
Christian Ebner
e04d1bfadc Cleanup of if statement with unwraps by a more compact match statement 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-03-20 09:49:18 +01:00
Dietmar Maurer
8346f0d59b src/server/rest.rs: correctly extract content type 2019-03-19 12:50:15 +01:00
Wolfgang Bumiller
a690ecacf0 daemon: simplify daemon creation 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-19 12:12:54 +01:00
Wolfgang Bumiller
e4311382d9 tools: daemon: rename some structs 
Reloadable resources are now 'Reloadable' instead of
'ReexecContinue'.

The struct handling the reload is a 'Reloader', not a
'ReexecStore'.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-19 11:09:46 +01:00
Wolfgang Bumiller
5e7bc50af1 add reload support to api daemon 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
a2ca713701 add reload support to proxy 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
af70c1811b tools: implement ReexecContinue for tokio's TcpListener 
This is the only thing we currently need to keep alive for
reloads.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
4422ba2c7f tools: daemon: add a default signalfd helper 
Proxy and daemon for now just want to handle reload via
`SIGHUP`, so provide a helper creating the signalfd stream
doing that - this is simply a filtered stream which passes
the remaining signals through, so it can be used exactly
like the signalfd stream could before to add more signals.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
dce94d0e09 tools: add daemon helpers 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
ff7049d481 tools: add fd_change_cloexec helper 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-18 12:29:18 +01:00
Wolfgang Bumiller
1c042cdc6c proxy: listener error handling fixup 
If the listening socket goes into some error state we'll get
std::io::Errors rather than higher level errors from the
native_tls::TlsAcceptor, those are usually fatal. (Ran into
this after performing a shutdown() on the file descriptor,
after which the future just endlessly loops in accept().)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-18 12:29:18 +01:00
Dietmar Maurer
60864b1b56 src/pxar.rs: improve docu 2019-03-18 12:27:30 +01:00
Dietmar Maurer
515688d1f1 avoid compiler warnings 2019-03-18 10:00:58 +01:00
Dietmar Maurer
ba10f2b0d4 src/bin/dynamic-files.rs: test case for file shrink during backup 2019-03-16 14:22:14 +01:00
Dietmar Maurer
af572aaa4c pxar: implement hardlinks 
So we are no longer compatible with catar ...
 2019-03-16 11:02:12 +01:00
Dietmar Maurer
812dd935ce src/pxar/encoder.rs: detect hardlinks 2019-03-15 12:19:51 +01:00
Dietmar Maurer
896e803feb src/pxar/sequential_decoder.rs: dump_entry - fix FIFO/SOCKET handling 2019-03-15 12:15:38 +01:00
Dietmar Maurer
7dfa17c7cc src/pxar/binary_search_tree.rs: fix test 2019-03-15 11:34:31 +01:00
Dietmar Maurer
9307279faf cleanup: remove dead code 2019-03-15 10:18:28 +01:00
Dietmar Maurer
46cfe7ab7e src/pxar/decoder.rs: implement seekable decoder 2019-03-15 09:36:05 +01:00
Dietmar Maurer
0e21b87be3 renamed: src/pxar/decoder.rs -> src/pxar/sequential_decoder.rs 2019-03-15 08:36:02 +01:00
Dietmar Maurer
3dbfe5b142 simplify pxar module structure 2019-03-15 08:24:32 +01:00
Dietmar Maurer
1ef46b81d3 src/bin/pxar.rs: implement extract 2019-03-15 08:03:44 +01:00
Dietmar Maurer
02c93163cf src/pxar/decoder.rs: correctly call callback() 2019-03-15 08:02:04 +01:00
Dietmar Maurer
5defa71bf9 rename PxarBackupWriter into PxarDecodeWriter 2019-03-15 07:20:22 +01:00
Dietmar Maurer
b005ed12d5 src/client/http_client.rs: avoid compiler warning 2019-03-15 07:16:26 +01:00
Dietmar Maurer
9e26abf16b src/cli/command.rs: fix iterator usage 2019-03-15 07:12:40 +01:00
Dietmar Maurer
f98746ba6c src/pxar/decoder.rs: improve docu 2019-03-15 07:12:11 +01:00
Dietmar Maurer
0b2d7fdc81 src/pxar/decoder.rs: avoid compiler warnings 2019-03-14 17:45:47 +01:00
Dietmar Maurer
37940aa1f9 src/bin/pxar.rs: implement dump/list 2019-03-14 17:43:11 +01:00
Dietmar Maurer
40360fde6e src/pxar/decoder.rs: implement dump_archive 
use this to debug archive problems ...
 2019-03-14 13:10:27 +01:00
Dietmar Maurer
8968258b66 rename catar into pxar 
To avoid confusion with the casync implementation.
 2019-03-14 10:54:09 +01:00
Dietmar Maurer
7c4dd94670 rename catar into pxar 
Seems impossible to stay compatible with the original catar specification,
so we change the name to avoid confusions.
 2019-03-14 10:14:32 +01:00
Dietmar Maurer
806500cd09 src/tools.rs: improve file name completion 
Use current working directory by default.
 2019-03-14 08:17:35 +01:00
Dietmar Maurer
08dc340a12 src/bin/proxmox-backup-client.rs: complete archive names 2019-03-14 08:09:35 +01:00
Wolfgang Bumiller
6f90a6a764 protocol: cleanup finish_backup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-13 14:16:17 +01:00
Dietmar Maurer
b238851832 src/bin/proxmox-backup-client.rs: implement completion for snapshots 2019-03-13 13:31:29 +01:00
Dietmar Maurer
4981134708 src/bin/proxmox-backup-client.rs: implement chunk-size completion 2019-03-13 12:26:01 +01:00
Dietmar Maurer
98f0b97293 src/bin/proxmox-backup-client.rs: cleanup 2019-03-13 12:17:39 +01:00
Dietmar Maurer
024f11bb55 src/bin/proxmox-backup-client.rs: implement completion for backup groups 2019-03-13 11:56:37 +01:00
Dietmar Maurer
edd3c8c605 src/client/backup_repo.rs: implement FromStr trait 2019-03-13 10:09:39 +01:00
Dietmar Maurer
874acb7039 src/client/backup_repo.rs: implement Display trait 2019-03-13 09:57:36 +01:00
Dietmar Maurer
d0a03d40ce src/bin/proxmox-backup-client.rs: implement repository completions 2019-03-13 09:47:12 +01:00
Dietmar Maurer
bff110308b src/bin/proxmox-backup-client.rs: fix BACKUPSPEC_REGEX - correctly specify non-capture group 
And improve bash completion.
 2019-03-13 07:23:59 +01:00
Dietmar Maurer
1a71509aad src/cli/command.rs: fix completion - never display argument parameters as option 2019-03-12 14:53:41 +01:00
Dietmar Maurer
496a67846f src/cli/command.rs: pass parsed parameters to completion function 2019-03-12 14:39:51 +01:00
Dietmar Maurer
b5fa28251c src/cli/getopts.rs: factor out reusable code 2019-03-12 14:38:47 +01:00
Dietmar Maurer
e39efdbd33 src/cli/command.rs: use a HashMap to store fixed param as key/value pairs 2019-03-12 12:54:16 +01:00
Dietmar Maurer
f4e4d58327 src/cli/command.rs: improve completions 2019-03-12 12:07:45 +01:00
Wolfgang Bumiller
4ba7b6099c signalfd cleanup 
- add comment about the boxed file descriptor (and others)
- reexport SigSet and Signal for convenience
- remove Result wrapper (only used once)
- enforce drop order of the PollEvented2 struct with respect
  to the boxed fd

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-12 11:07:09 +01:00
Wolfgang Bumiller
f54c19989c tools: add async signalfd handler 
This provides a Stream<Item = siginfo> via nix' signalfd,
by wrapping it in tokio's PollEvented2 struct to allow
polling via tokio's event loop.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-12 08:04:28 +01:00
Dietmar Maurer
d5c34d98c0 src/catar/decoder.rs: simplify public restore API 2019-03-11 14:31:01 +01:00
Dietmar Maurer
8e39232acc src/bin/proxmox-backup-client.rs: strip .didx file extensions 2019-03-11 10:54:02 +01:00
Dietmar Maurer
74cdb52107 src/bin/proxmox-backup-client.rs: include file extension in backupspec 2019-03-11 10:27:28 +01:00
Dietmar Maurer
fc2bf37ead src/catar/decoder.rs: cleanup style 2019-03-11 08:14:28 +01:00
Dietmar Maurer
656b23e107 src/catar/decoder.rs: dir_mkdirat - add create_new flag 2019-03-11 07:27:45 +01:00
Dietmar Maurer
e9c9409a99 src/catar/decoder.rs: implement real sequential decoder (no Seek) 
I copied the old code to src/catar/inspector.rs. Will modify
that later to implement a random access decoder ...
 2019-03-08 16:55:54 +01:00
Dietmar Maurer
eed6db3923 src/bin/proxmox-backup-client.rs: impl option all-file-systems 2019-03-08 09:36:40 +01:00
Wolfgang Bumiller
06178f13fa move chunker into proxmox_protocol 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-08 09:29:47 +01:00
Dietmar Maurer
e3c30c50a6 src/bin/catar.rs: implement all-file-systems option 2019-03-08 09:28:12 +01:00
Dietmar Maurer
65092b1e27 src/catar/encoder.rs: log skipped mount points 2019-03-08 09:25:00 +01:00
Dietmar Maurer
6b64d25d64 src/catar/encoder.rs: remove debug output 2019-03-08 08:23:07 +01:00
Dietmar Maurer
2689810c92 src/bin/catar.rs: implement verbose flags 2019-03-08 08:20:56 +01:00
Dietmar Maurer
79c281fdac src/catar/encoder.rs: use simply flag "all_file_systems" 2019-03-08 08:14:26 +01:00
Dietmar Maurer
25e77d3877 src/cli/command.rs: print_bash_completion - avoid crash 2019-03-07 12:14:26 +01:00
Dietmar Maurer
141f062e08 src/backup/chunk_store.rs: use zstd compression insteadf of lz4 
Provides better compressionm rate, and is still fast.
 2019-03-07 11:42:59 +01:00
Dietmar Maurer
d2690f74bb src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end 2019-03-07 10:40:01 +01:00
Dietmar Maurer
2ae7d196b9 src/bin/proxmox-backup-client.rs: correctly strip file extension 2019-03-06 11:18:46 +01:00
Dietmar Maurer
9f91249351 src/bin/proxmox-backup-client.rs: impl restore 2019-03-06 11:02:08 +01:00
Dietmar Maurer
01a1342364 src/api2/admin/datastore.rs: impl list_snapshot_files 2019-03-06 11:02:08 +01:00
Dietmar Maurer
58e99e138e src/backup/backup_info.rs: new helper to list files 2019-03-06 11:02:08 +01:00
Dietmar Maurer
dd79265a8a src/api2/admin/datastore/catar.rs: use archive_name directly 2019-03-06 11:02:08 +01:00
Dietmar Maurer
a6b7551311 src/client/http_client.rs: impl download 2019-03-06 11:02:08 +01:00
Wolfgang Bumiller
6716f30bbc add protocol test client 
This uses futures for everything which is mostly useful as a
test to see if the protocol crate's non-blocking I/O support
can handle it...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:26:12 +01:00
Wolfgang Bumiller
e2d007f76e api2/admin/datastore: add a backup protocol test api path 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:26:12 +01:00
Wolfgang Bumiller
e3062f87b1 fixed_index: split add_chunk/add_digest 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
c3bb97e59c fixed_index: impl Send for FixedIndexWriter 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
3b7ade9e78 backup/datastore: expose insert_chunk method 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Wolfgang Bumiller
91ebea8262 router: add upgrade method 
This is just an Async GET method, but semantically
different, so it gets its own router method.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-06 10:25:20 +01:00
Dietmar Maurer
80822b95b1 src/bin/proxmox-backup-client.rs: sort list results 2019-03-06 07:20:51 +01:00
Dietmar Maurer
fba30411fb src/bin/proxmox-backup-client.rs: allow to set backup-id 2019-03-06 07:02:52 +01:00
Dietmar Maurer
21ea0158b4 src/client/http_client.rs: remove debug statement 2019-03-06 06:45:51 +01:00
Dietmar Maurer
ba3a60b2a4 src/client/http_client.rs: store/load ticket in xdg runtime dir, depend on crate xdg 2019-03-05 12:56:21 +01:00
Dietmar Maurer
f8f94534d4 src/api2/access.rs: authenticate_user() - add ticket login 2019-03-05 12:56:21 +01:00
Dietmar Maurer
e5662b04e9 src/tools/ticket.rs: define const TICKET_LIFETIME 2019-03-05 12:56:21 +01:00
Dietmar Maurer
53cafb5925 src/tools.rs: impl file_get_json() 2019-03-05 12:56:21 +01:00
Wolfgang Bumiller
391d310741 backup_info: BackupDir shortcut 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-05 10:15:06 +01:00
Wolfgang Bumiller
93b49ce38a backup_info: use two independent Into<String> types 
there's no reason to force them to be the same input type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-05 09:13:08 +01:00
Dietmar Maurer
3ec3ec3f9a src/bin/proxmox-backup-client.rs: print backup duration 2019-03-05 08:11:40 +01:00
Dietmar Maurer
d57474e055 src/backup/backup_info.rs: improve docs 2019-03-05 07:28:13 +01:00
Dietmar Maurer
b3483782d1 src/backup/backup_info.rs: move code into separate file 
Also changed create_backup_dir() parameters - uses &BackupDir now.
 2019-03-05 07:18:12 +01:00
Dietmar Maurer
2b01a22507 src/backup/datastore.rs: add helper to sort backup lists 2019-03-04 18:20:57 +01:00
Dietmar Maurer
875fb1c01a src/backup/datastore.rs: use unix epoch to create DateTime 
To make sure that we have a timestamp without nanosecond.
 2019-03-04 17:58:22 +01:00
Dietmar Maurer
38f8815925 src/backup/datastore.rs: use new() function to create structs 2019-03-04 17:44:27 +01:00
Dietmar Maurer
9b492eb256 src/backup/datastore.rs: protect BackupDir fields, impl new() 2019-03-04 13:51:36 +01:00
Dietmar Maurer
1e9a94e579 src/backup/datastore.rs: protect BackupGroup fields, impl new() 2019-03-04 13:38:23 +01:00
Dietmar Maurer
cdebd467e6 src/bin/proxmox-backup-client.rs: simply log to stdout 2019-03-04 09:38:34 +01:00
Dietmar Maurer
f72dfd2d10 src/backup/datastore.rs: make sure timestamp does not contains nanoseconds 
Because we want simple, readable file names.
 2019-03-04 09:37:10 +01:00
Dietmar Maurer
219ef0e6ca src/catar/encoder.rs: add verbose flags 2019-03-04 08:01:09 +01:00
Dietmar Maurer
e0b8ce6424 src/catar/encoder.rs: skip RAMFS 
unless explicitly selected.
 2019-03-04 06:48:47 +01:00
Dietmar Maurer
597a92039e src/bin/proxmox-backup-client.rs: rename create into backup 2019-03-03 11:46:53 +01:00
Dietmar Maurer
6f62c9240a src/api2/admin/datastore.rs: imp delete_snapshot 2019-03-03 11:29:00 +01:00
Dietmar Maurer
af926291e0 avoid compiler warning 2019-03-03 09:52:06 +01:00
Dietmar Maurer
e77a02ed87 catar api: cleanup parameter names 2019-03-03 09:48:32 +01:00
Dietmar Maurer
421c9ba1d9 src/bin/proxmox-backup-client.rs: use new json_object_to_query 2019-03-03 09:44:25 +01:00
Dietmar Maurer
f5f13ebc5a src/tools.rs: new helper json_object_to_query 2019-03-03 08:51:37 +01:00
Dietmar Maurer
49dc074007 parameter name cleanups (use '-' instead of '_') 2019-03-02 16:34:10 +01:00
Dietmar Maurer
ad20d19877 src/api2/admin/datastore.rs: rename get_group_list to list_groups, cleanups 2019-03-02 16:28:36 +01:00
Dietmar Maurer
184f17afad src/api2/admin/datastore.rs: impl list_snapshots 2019-03-02 16:20:50 +01:00
Dietmar Maurer
af53186e6a src/tools.rs: add simply string join 2019-03-02 16:15:45 +01:00
Dietmar Maurer
812c6f8716 src/api2/admin/datastore.rs: impl list_backup_groups 2019-03-02 11:29:05 +01:00
Dietmar Maurer
e909522f06 src/backup/datastore.rs: use timestamp with zone offset 2019-03-01 17:36:40 +01:00
Dietmar Maurer
dc4c09faae src/backup/datastore.rs: split BackupDir 2019-03-01 16:26:35 +01:00
Dietmar Maurer
38b0dfa511 src/backup/datastore.rs: split BackupInfo 2019-03-01 16:26:35 +01:00
Wolfgang Bumiller
dcd15d10bb backup/index: remove IndexIterator 
removed the need for it, to avoid copies

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-01 13:29:36 +01:00
Dietmar Maurer
17d6979afd src/bin/proxmox-backup-client.rs: generate a log file 2019-03-01 09:35:41 +01:00
Dietmar Maurer
3b151414f8 src/tools/file_logger.rs: class to log into files 2019-03-01 09:34:29 +01:00
Dietmar Maurer
5a2892f0ae src/bin/proxmox-backup-client.rs: use chrono::DateTime 2019-03-01 06:59:32 +01:00
Dietmar Maurer
51144821f9 src/bin/proxmox-backup-client.rs: login before starting backup 2019-03-01 06:48:41 +01:00
Dietmar Maurer
541b908ea4 src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch 2019-02-28 17:03:48 +01:00
Dietmar Maurer
b55cee9227 src/cli/command.rs: generate_usage_str() - fix option separator (newline) 2019-02-28 16:20:00 +01:00
Dietmar Maurer
c4c5174000 src/cli/command.rs: get_schema_type_text() - display min/max 2019-02-28 16:11:49 +01:00
Wolfgang Bumiller
df67070b1c client/backup_repo: allow ip addresses for BackupRepository 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5de2bced2d datastore: add generic open_index 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
7bc1d72778 add IndexFile trait 
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.

Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5be4065baf make FixedIndexReader Send and update comments 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
fded74d0c7 style: BufferedReader -> BufferedRead 
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Dietmar Maurer
8f57971714 src/api2/admin/datastore.rs: implement prune 2019-02-28 12:51:27 +01:00
Dietmar Maurer
83b7db0209 src/api2/admin/datastore.rs: start prune api 
just a dummy for now.
 2019-02-27 16:53:17 +01:00
Dietmar Maurer
0ffbccce5e rc/server/rest.rs: allow to pass parameters as application/json 2019-02-27 12:37:53 +01:00
Dietmar Maurer
f98ea63de2 src/bin/proxmox-backup-client.rs: fix bash completion 2019-02-27 12:37:53 +01:00
Wolfgang Bumiller
8731e40a7f backup/datastore: create_backup_dir: show if path existed 
To enable asserting the creation of a new backup rather than
adding to an existing one.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-27 10:15:36 +01:00
Dietmar Maurer
25f1650b71 src/bin/proxmox-backup-client.rs: minor code cleanup 2019-02-27 07:21:57 +01:00
Dietmar Maurer
9bc3ddb822 src/cli/command.rs: improve generate_usage_str 
correctly format positional array arguments ({<name>}).
 2019-02-26 12:40:51 +01:00
Dietmar Maurer
ae0be2dda9 src/bin/proxmox-backup-client.rs: allow to upload more that one directory 2019-02-26 12:27:28 +01:00
Dietmar Maurer
a477d6889a src/client/http_client.rs: login once, store and reuse ticket/token 2019-02-26 11:59:10 +01:00
Dietmar Maurer
d02c2ed701 src/cli/getopts.rs: cleanup, avoid compiler warning 2019-02-26 11:55:06 +01:00
Dietmar Maurer
f8dfbb45c0 src/tools.rs: new helper required_array_param() 2019-02-26 08:48:17 +01:00
Dietmar Maurer
aaba6faace src/cli/getopts.rs: support positional array arguments 2019-02-26 08:47:34 +01:00
Dietmar Maurer
f33fa27307 src/cli/command.rs: improve error message 2019-02-26 08:46:36 +01:00
Dietmar Maurer
92a173feb0 src/api_schema/schema.rs: improve ParameterError display function 2019-02-26 07:11:24 +01:00
Wolfgang Bumiller
f661f37427 backup/dynamic_index: split out add_chunk 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:50:45 +01:00
Wolfgang Bumiller
9ac6ec868a backup/chunk_store: split insert_chunk 
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
3a50ddd0c9 backup/chunk_store: mark ChunkIterator as fused 
Ranges are FusedIterators and it can be useful to propagate
that property to users.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
eff25ecaf2 backup/chunk_store: make percentage output optional 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
d59397e60e backup/datastore: expose chunk_store.get_chunk_iterator 
will be used by the protocol handler to iterate over
existing chunks

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Dietmar Maurer
8eb1203bda src/backup/chunk_stat.rs: log write speed 2019-02-25 13:21:00 +01:00
Dietmar Maurer
7e3365554e src/backup/chunk_stat.rs: new struct to track chunk statistics 2019-02-25 12:52:10 +01:00
Dietmar Maurer
798f7fa065 src/backup/chunk_store.rs: return info about compressed chunk size 
So that we can generate better statistics ..
 2019-02-25 11:36:05 +01:00
Dietmar Maurer
78216a5ab1 src/backup/chunk_store.rs: use lz4 for compression 
And depend on lz4 lib now.
 2019-02-25 10:00:28 +01:00
Dietmar Maurer
793b0f4d77 src/cli/command.rs: improve help command completion 2019-02-23 18:07:32 +01:00
Dietmar Maurer
6949d91564 src/cli/command.rs: cleanup, avoid array manipulation 2019-02-23 17:50:33 +01:00
Dietmar Maurer
9d78d57960 src/cli/command.rs: implement bash completion for help 2019-02-23 17:38:10 +01:00
Dietmar Maurer
698d9d4402 src/cli/command.rs: add help command 2019-02-23 15:10:48 +01:00
Dietmar Maurer
8b6dd2240d src/cli/command.rs::allow to generate ReST docu 2019-02-23 11:29:18 +01:00
Dietmar Maurer
2f3f2bb77f src/cli/command.rs: improve usage imformation 2019-02-22 17:40:37 +01:00
Dietmar Maurer
f252ca654c src/backup/chunker.rs: simplify break test 
To make it a little bit faster ...
 2019-02-22 13:02:11 +01:00
Dietmar Maurer
6adb40eefc src/cli/command.rs: start implementing generate_usage_str() 2019-02-22 12:02:59 +01:00
Dietmar Maurer
33256db6dd src/cli/command.rs: simplify cli handler 2019-02-22 11:15:10 +01:00
Dietmar Maurer
61834c1cbf src/backup/chunker.rs: another try to speedup the chunker 2019-02-22 11:15:10 +01:00
Dietmar Maurer
532a69eb04 src/cli/getopts.rs: improve docs 2019-02-21 09:36:52 +01:00
Dietmar Maurer
4de0e142a0 cleanup cli library structure 2019-02-21 09:08:03 +01:00
Wolfgang Bumiller
56458d9764 http_client: on a tty, read password if no env var is set 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-20 15:03:34 +01:00
Wolfgang Bumiller
c9b296f117 tools: add tty helper module 
with read_password() and stdin_isatty() functions

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-20 14:43:07 +01:00
Dietmar Maurer
8cc0d6aff3 src/bin/proxmox-backup-client.rs: implement garbage-collection sub-command 2019-02-20 14:10:45 +01:00
Dietmar Maurer
81da38c143 src/client/http_client.rs: implement post 
without parameters for now...
 2019-02-20 14:09:55 +01:00
Dietmar Maurer
8c75372b79 src/backup/datastore.rs: list_backups() - include list of index files 2019-02-20 13:37:44 +01:00
Dietmar Maurer
14691fc1f3 src/bin/test_chunk_size.rs: avoid compilöer warnings, exit after 1GB 2019-02-20 12:05:28 +01:00
Dietmar Maurer
1604ec2654 src/bin/test_chunk_size.rs: another chunker test - gather some stats 2019-02-20 11:50:10 +01:00
Dietmar Maurer
fa4ebf0fce src/backup/chunker.rs: simplify code 2019-02-20 11:17:19 +01:00
Dietmar Maurer
1b9c959c32 src/backup/chunker.rs: change window size back to 48 
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
 2019-02-20 11:01:09 +01:00
Dietmar Maurer
247cdbce72 src/api2/admin/datastore/catar.rs: allow to configure chunk-size 2019-02-19 15:19:12 +01:00
Dietmar Maurer
c584aa21fd src/backup/chunker.rs: start() - correctly store hash in self.h 2019-02-19 12:40:12 +01:00
Dietmar Maurer
cd1598302a src/backup/chunker.rs: use 64 byte chunker window size 
This results in faster code.
 2019-02-19 12:27:37 +01:00
Dietmar Maurer
9a3e869cf5 src/backup/chunker.rs: simplify buzzhash start code 2019-02-19 12:17:17 +01:00
Dietmar Maurer
1c7a88aeff UploadCaTar: currectly write all data using write_all() 2019-02-18 18:31:02 +01:00
Dietmar Maurer
4d2881dd4c src/backup/chunker.rs: correctly compute window index 'idx' 
And add a simple test.
 2019-02-18 16:59:23 +01:00
Dietmar Maurer
a4a5c78c01 src/client/http_client.rs: send csrf token 2019-02-18 16:59:23 +01:00
Wolfgang Bumiller
fc7f03523c cleanup 
Error::from is already a function taking 1 parameter,
there's no need to wrap it with `|e| Error::from(e)`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-18 13:21:27 +01:00
Wolfgang Bumiller
8fdef1a81c fix typo: confi{g}dir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-18 13:11:42 +01:00
Dietmar Maurer
1cb99c234e src/server/rest.rs: correctly insert NoLogExtension() 2019-02-18 06:54:12 +01:00
Dietmar Maurer
591f570b22 src/client/http_client.rs: trim path 2019-02-18 06:24:28 +01:00
Dietmar Maurer
7f1685230a src/server/rest.rs: get_index() include username and CSRF token 
When we have an valid ticket. Also delay get_index() if called with
an invalid ticket.
 2019-02-17 19:28:32 +01:00
Dietmar Maurer
141de8374a src/server/rest.rs: factor our normalize_path() 2019-02-17 17:31:53 +01:00
Dietmar Maurer
8225aa2ff6 src/server/rest.rs: improve logs for unauthorized request 2019-02-17 17:18:44 +01:00
Dietmar Maurer
54d15ab5c5 src/api_schema/schema.rs: improve docu 2019-02-17 12:13:06 +01:00
Dietmar Maurer
51ebd07974 src/api_schema/router.rs: add docu for RpcEnvironmentType 2019-02-17 11:59:18 +01:00
Dietmar Maurer
d21c2710da api_schema/router.rs: improve doc 2019-02-17 10:30:41 +01:00
Dietmar Maurer
ef2f2efbcc improve api_schema module structure 2019-02-17 10:16:33 +01:00
Dietmar Maurer
dc9a007b11 rename src/api to src/api_schema 2019-02-17 09:59:20 +01:00
Dietmar Maurer
5ddf8cb10a cleanup auth code, verify CSRF prevention token 2019-02-16 15:52:55 +01:00
Dietmar Maurer
95aee7e032 tools/common_regex.rs: add more docu 2019-02-16 12:36:01 +01:00
Dietmar Maurer
99be8844c7 buildcfg.rs: improve doc 2019-02-16 12:31:39 +01:00
Dietmar Maurer
a8f268afbb fix configuration dir permission 
Carefully set and check permissions ...
 2019-02-16 12:19:13 +01:00
Dietmar Maurer
aada2a9719 config/datastore.rs: remove unused code 2019-02-16 10:10:45 +01:00
Dietmar Maurer
d11594db4c config/datastore.rs: return empty config if file does not exist 2019-02-16 10:06:08 +01:00
Dietmar Maurer
12400210a2 tools::file_get_contents - improve error message 2019-02-16 09:42:00 +01:00
Dietmar Maurer
6235a41862 tools::file_read_firstline - improve error message 2019-02-16 09:36:29 +01:00
Dietmar Maurer
17ed456c2e change proxy user from www-data to backup 2019-02-16 09:29:04 +01:00
Dietmar Maurer
5d14eb6a76 tools.rs: new helper to get uid/gid for the sepcified system user. 
Or is there an easier way to get that info?
 2019-02-15 15:33:12 +01:00
Dietmar Maurer
f0a6112451 backup/datastore.rs: improve error reporting 2019-02-15 14:33:03 +01:00
Dietmar Maurer
9b50c16103 section_config.rs: simplify parser by using new try_block macro 2019-02-15 12:13:15 +01:00
Dietmar Maurer
fe651dd6e5 tools.rs: fix doc tests 2019-02-15 11:33:51 +01:00
Dietmar Maurer
e80503d2a6 tools.rs: document try_block macro 2019-02-15 11:20:45 +01:00
Dietmar Maurer
f0dbba8cbe tools.rs: new try_block macro 2019-02-15 10:59:12 +01:00
Wolfgang Bumiller
1d77b6cf6b update to pam 0.7 (renamed from pam-auth) 
It now supports custom conversation methods, so instead of
new() we ask for a default authenticator taking a password
via with_password(). Since the password is now handled by
the now separate conversation handler, `set_credentials()`
is now called on the handler we get via `.get_handler()`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-15 10:34:40 +01:00
Dietmar Maurer
d4736445fe server/rest.rs: add method to log message 2019-02-15 10:16:12 +01:00
Dietmar Maurer
f02e6fc478 api2/admin/datastore/catar.rs: simplify/fix debug message 2019-02-15 10:04:37 +01:00
Dietmar Maurer
44c00c0dfd server/rest.rs: log full error messages 2019-02-15 09:55:12 +01:00
Dietmar Maurer
7171b3e079 server/rest.rs: use a protocol extension to avoid double log 
Instead of modifying the response header itself.
 2019-02-14 16:04:24 +01:00
Dietmar Maurer
7e03988caf avoid double logging of proxied requests 2019-02-14 13:28:41 +01:00
Dietmar Maurer
af6f80d3db Revert "backup/datastore: fn open is a private implementation detail" 
This reverts commit 3dacd9a660.

I currently use this in some test ...
 2019-02-14 13:13:49 +01:00
Dietmar Maurer
78a1fa6722 src/server/rest.rs: log failed requests 2019-02-14 13:07:34 +01:00
Wolfgang Bumiller
2cc90617dd tools/fs: mark ReadDir as Send 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
62f2422f6a backup/chunk_store: verify chunk file names 
get_chunk_iterator() should skip over files which aren't an
actual chunk

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
3dacd9a660 backup/datastore: fn open is a private implementation detail 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f backup/chunk_store: open base dir in get_chunk_iterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Dietmar Maurer
db4868ef4f src/client/backup_repo.rs - improve docu 2019-02-14 11:17:08 +01:00
Dietmar Maurer
151c6ce27d src/client.rs: improve library structure and docu 2019-02-14 11:11:39 +01:00
Wolfgang Bumiller
dd1bfe6c12 tools/timer: add module documentation 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:13:42 +01:00
Wolfgang Bumiller
c7f481b642 backup: replace ChunkIterator 
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
121f18efb7 tools: use scan_subdir iterator for scandir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
32286b039c tools/fs: add file_name_utf8_unchecked shortcut 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
806d7a6a35 tools/fs: change scan_subdir to use filter_file_name_regex 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
53e1e7ca72 tools/fs: add filter_file_name_regex to FileIterOps 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
23fba9d777 tools/fs: add FileIterOps trait and FileTypeFilter 
FileIterOps will provide operations on file entry iterators.
For now this provides a file type filter.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
1db416150d tools/fs: let ReadDir iterator yield ReadDirEntry 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
b12505da69 tools/fs: add ReadDirEntry helper 
This wraps nix::dir::Entry with the parent and additionally
exposes the parent directory file descriptor since we need
it to perform openat().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
a25f863afd tools/fs: add scan_subdir helper 
This filters the results of read_subdir with a regex.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
b4d5787de9 tools: start fs submodule, add read_subdir 
This creates an iterator relative to a RawFd via
nix::dir::Dir over nix::dir::Entrys.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Dietmar Maurer
0dffe3f99a src/client/http_client.rs: try to login 
use an environment var to store passphrase (PBS_PASSWORD)
 2019-02-13 14:31:43 +01:00
Dietmar Maurer
33d64b81e0 src/bin/proxmox-backup-client.rs: define and use BackupRepository 2019-02-13 12:30:52 +01:00
Wolfgang Bumiller
4a3f65172c client: use hyper-tls for now 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-13 11:03:09 +01:00
Wolfgang Bumiller
42d6e4fb05 tools: borrow: implement Deref for Tied 
Eg. if you have an x: Tied<Dir, Iterator> now you can simply
call x.next() as it'll dereference to the Iterator.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-13 11:03:09 +01:00
Dietmar Maurer
541a302224 api2/admin/datastore/catar.rs: use nested file extensions 
i.e. archive.catar.didx
 2019-02-13 10:37:00 +01:00
Dietmar Maurer
e5064ba607 simplify backup lib structure (pub use xxx:*), improve doc 2019-02-12 14:13:31 +01:00
Dietmar Maurer
d78345bcfc src/backup.rs - improve doc 2019-02-12 13:27:11 +01:00
Dietmar Maurer
93d5d77952 rename ArchiveIndex to DynamicIndex 
also changed the file extension from .aidx to .didx
 2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd rename ImageIndex to FixedIndex 
also changed the file extension fron .iidx to .fidx
 2019-02-12 11:50:45 +01:00
Wolfgang Bumiller
82bc0ad40c backup: fixup warnings and readd lost file type check 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:43:43 +01:00
Wolfgang Bumiller
9349d2a1d2 backup: cleanup long lines 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
fdd71f5286 backup: use ChunkIterator for sweep_unused_chunks 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
c9e7f4dc8b backup: add optional progress callback to ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
4070096c6a backup: add ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
6ed25cbe3a tools: add Tied value 
Tie two dependent values together, such as a nix::Dir with
its nix::dir::Iter<'a> where 'a is tied to the Dir's
lifetime, making it otherwise impossible to return them or
store them in a struct.

Alternatively we could try the 'rental' crate.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
9ac1045c7b bin: manager: fixup datastore parameter name 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Dietmar Maurer
8f8d5a420c avoid compiler warning 2019-02-11 15:12:01 +01:00
Dietmar Maurer
aa5a4060ab bin/proxmox-backup-api.rs: improve error handling 2019-02-11 15:11:42 +01:00
Dietmar Maurer
4223d9f800 bin/proxmox-backup-proxy.rs: improve error handling 2019-02-11 14:43:26 +01:00
Wolfgang Bumiller
ace9e3531a access: use proxmox-backup-auth for pam 
allows customization via /etc/pam.d/proxmox-backup-auth

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-07 13:40:22 +01:00
Wolfgang Bumiller
13f8310cac api2: time: use /etc/localtime as fallback 
in case there is no /etc/timezone

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-05 14:14:56 +01:00
Wolfgang Bumiller
a7cac14ed9 tools: use std::fs::read for file_get_contents 
Until we add the extra options like we have in pve, just use
the already existing (since 1.26) shortcut.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:35:32 +01:00
Wolfgang Bumiller
e937034c86 remove some rather inconvenient debug output 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:34:38 +01:00
Wolfgang Bumiller
9d02e5c38a proxy: use <configdir>/server.pfx for tls 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:19:32 +01:00
Wolfgang Bumiller
9f4962d396 introduce buildcfg module and PROXMOX_CONFIGDIR 
buildcfg.rs should contain convenience variables or macros
for using build-time configured variables

For now we replace hardcoded "/etc/proxmox-backup/<foo>"
with configdir!("<foo>").

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:13:08 +01:00
Wolfgang Bumiller
0d176f3681 proxy: use TLS via tokio-tls 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 14:56:07 +01:00
Wolfgang Bumiller
e0a5d1ca2f tests: chunk_store: cleanup .testdir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
a4f9188300 tests: import lazy_static as needed 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
6285b251e7 don't hardcode /usr/share paths 
Add a defines.mk file defining the standard default layout
for installation (PREFIX, DATAROOTDIR, LIBDIR etc.).

Add a $(JSDIR) variable and use env!("JSDIR") in place of
the hardcoded /usr/share/javascript/proxmox-backup path.

Have defines.mk include an *optional* local.mak to override
JSDIR to www/ during local development.

Add `make install` targets and remove debian/install.

Note that service files are now in $libdir/proxmox-backup/
as there's usually no point in starting them from the
command line, so /usr/sbin makes no sense.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
8e4df103dd remove useless command option from daemons 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Dietmar Maurer
6278365fd9 improve ApiMethod docu 2019-02-01 10:38:07 +01:00
Dietmar Maurer
4b2cdeb9a6 implement relead_timezone flag 2019-02-01 10:04:46 +01:00
Dietmar Maurer
c82bc1a1f9 depend on pam-auth, use pam for root login 
Please use username "root@pam" for login via GUI.
 2019-02-01 09:30:50 +01:00
Dietmar Maurer
a154a8e8a4 delay unauthorized request (rate limit) 2019-01-31 14:34:21 +01:00
Dietmar Maurer
5996577ab6 move http error class to router.rs 2019-01-31 13:22:30 +01:00
Dietmar Maurer
b9903d6331 server/rest.rs: verify auth cookie 2019-01-31 12:22:00 +01:00
Dietmar Maurer
f484eed3c2 Utils.js: fix cookie handling 
Use unsecure cookie foör testing.
 2019-01-31 10:08:08 +01:00
Dietmar Maurer
9f49fe1d5d avoid compiler warnings 2019-01-30 18:25:37 +01:00
Dietmar Maurer
34f956bc25 api2/access.rs: add ticket api 2019-01-30 15:16:10 +01:00
Wolfgang Bumiller
1bf446a33e fix timer_delete call 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-30 12:04:20 +01:00
Dietmar Maurer
cf671670a0 auth_helpers.rs: add timestamp to csrf token 2019-01-29 17:50:03 +01:00
Dietmar Maurer
1e76cbc6a2 auth_helpers.rs: implement assemble_csrf_prevention_token 2019-01-29 17:41:45 +01:00