tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,811 Commits 2 Branches 54 Tags 14 MiB
9c18e93522
Commit Graph

1620 Commits

Author SHA1 Message Date
Wolfgang Bumiller
3c0facc787 src/client/remote_chunk_reader.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
369a87e3a2 src/client/pxar_backup_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
2107bb40c1 src/client/pipe_to_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
5b3911995b src/client/merge_known_chunks.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
a6782ca10b src/client/http_client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
8630557918 src/bin/upload-speed.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
cab6816969 src/bin/test_chunk_speed2.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
fda5797b8a src/bin/proxmox-backup-proxy.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
e9722f8bde src/bin/proxmox-backup-client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
e76ac3a499 src/bin/proxmox-backup-api.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
e235c8f719 src/bin/h2s-server.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
74be6dc9b7 src/bin/h2server.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
55d8a631fc src/bin/h2s-client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
15d0e4a3bd src/bin/h2client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
b9203d87f4 src/bin/download-speed.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
745e652a7f src/backup/index.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
32bef1e2d1 src/backup/chunk_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
f2d9b73c91 src/api_schema/router.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
ffb6434485 src/api2/reader.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
7622005574 src/api2/backup/upload_chunk.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
59b2baa0f6 src/api2/backup.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
6be147b78c src/api2/admin/datastore.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
5c1130df9f implement Sync for DynamicIndexReader and FixedIndexReader 
hyper's wrap_stream now needs this

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
236761a3e6 drop src/storage/futures.rs 
it's unused and uses a tokio::Task I don't want to lookup
the std-future conversion for...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Dietmar Maurer
02fcf372e4 rc/bin/proxmox-backup-client.rs - restore: always download index.json.blob 2019-09-02 14:14:32 +02:00
Dietmar Maurer
0d9862803b src/bin/proxmox-backup-client.rs - reatore: avoid loading (large) blobs into memory 2019-09-02 14:13:31 +02:00
Christian Ebner
66c8eb9383 src/client/http_client.rs: Refactor handling Option and Result types 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-29 09:15:16 +02:00
Wolfgang Bumiller
40f8680e37 tools: remove SignalFd 
We don't use it anymore and it'll just be more work when
switching to async-await.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 13:44:53 +02:00
Wolfgang Bumiller
d21f8a5b5b backup/dynamic_index: explicitly zero-initialize the header 
The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 11:42:35 +02:00
Wolfgang Bumiller
990b930f22 backup/dynamic_index: use static assertion, fix size 
The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 11:42:35 +02:00
Christian Ebner
bcb664cb69 pxar/fuse: add pxar cli flag to set single- or multi-threaded session loop 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:43:14 +02:00
Christian Ebner
de2791868a src/pxar/sequentail_decoder.rs: followup: cleanup S_IFMT masking 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:42:22 +02:00
Christian Ebner
9cb9e1b7db src/pxar/sequential_decoder.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:42:19 +02:00
Christian Ebner
dac88033ba src/pxar/dir_stack.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:39:14 +02:00
Christian Ebner
b69169e7be src/pxar/fuse.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:37:58 +02:00
Christian Ebner
d792dc3cd2 src/pxar/match_pattern.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:37:56 +02:00
Christian Ebner
4ea22b68d9 src/pxar/format_definition.rs: style fixup 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:37:53 +02:00
Wolfgang Bumiller
7d83440c60 remove proxmox-protocol subcrate 
AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)

Moved the chunker back into src/backup/chunker.rs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 14:08:25 +02:00
Wolfgang Bumiller
7a57cb77e1 more formatting & use statement fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 13:50:27 +02:00
Christian Ebner
35dfd3ceb7 src/pxar/encoder.rs: style fixups by rustfmt 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 13:29:11 +02:00
Christian Ebner
1af30bc2fe (pxar: replace deprecated std::mem::uninitialized() 
... and use std::mem::MaybeUninit or proxmox::tools::vec::uninitialized() instead.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 13:27:25 +02:00
Wolfgang Bumiller
5e58e1bb7d {dynamic,fixed}_index: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:35:36 +02:00
Wolfgang Bumiller
f93b55b057 src/backup/index: style fixup & unsafe copy removal 
We can use the safe .copy_from_slice alternative in this
case.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:31:45 +02:00
Wolfgang Bumiller
fa17b1ce2a src/backup/index: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:27:35 +02:00
Wolfgang Bumiller
f3a8d1d7e0 api2/node/time: replace mem::uninitialized 
and fixup use statements

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:14:41 +02:00
Wolfgang Bumiller
48b85e8e3b tools: tty: replace mem::uninitialized and style fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:14:19 +02:00
Wolfgang Bumiller
b528fddfc6 tools: timer: style fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:07:56 +02:00
Wolfgang Bumiller
b7f44ace92 timer: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:07:10 +02:00
Wolfgang Bumiller
6100071f4e tools: style & use statement fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 10:57:56 +02:00
Wolfgang Bumiller
5c20e2da6b src/config: more style fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 14:14:26 +02:00
Wolfgang Bumiller
91640ab567 src/config: style fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 14:11:07 +02:00
Wolfgang Bumiller
928650c4eb src/config: use statement cleanup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 14:10:50 +02:00
Wolfgang Bumiller
2ec979e4ed /var/run -> /run 
/var/run is considered deprecated and for instance in
systemd unit files lintian complains...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 12:28:24 +02:00
Wolfgang Bumiller
35238e234d switch from create_dir_chown to create_path 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 12:26:10 +02:00
Christian Ebner
e5471b4836 pxar/decoder: avoid failing on hardlinks and return corresponding target file entry 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
[Fixup: remove unnecessary 'mut']
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 09:32:52 +02:00
Christian Ebner
58262f40f7 src/pxar/decoder.rs: get correct stats for root directory 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 09:31:41 +02:00
Christian Ebner
7d26720e79 src/pxar/decoder.rs: move GOODBYE_ITEM_SIZE from function to module scope. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 09:31:39 +02:00
Wolfgang Bumiller
367f002ee2 use statement cleanup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-16 12:28:40 +02:00
Dietmar Maurer
9d135fe617 src/backup/catalog_blob.rs: moved catalog impl. from pxar 
And avoid loading catalog into memory.
 2019-08-16 12:27:17 +02:00
Wolfgang Bumiller
9025312aa6 avoid lifetimes in blob reader/writer 2019-08-16 10:07:24 +02:00
Dietmar Maurer
71d08e00b7 src/backup/data_blob_reader.rs: fix compiler warning 2019-08-16 08:10:39 +02:00
Christian Ebner
ac0b435344 src/pxar/fuse.rs: fix compiler warnings 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-14 15:14:57 +02:00
Dietmar Maurer
a84ef4c205 src/bin/proxmox-backup-client.rs: avoid loading catalog into memory 
We can use the new DataBlobReader instead.
 2019-08-14 15:07:28 +02:00
Dietmar Maurer
b791804f4b src/backup/data_blob_reader.rs: impl for Read instead of BufRead 2019-08-14 14:56:01 +02:00
Dietmar Maurer
39a4df61d6 improve docs 2019-08-14 14:08:27 +02:00
Dietmar Maurer
018d11bb84 src/backup/data_blob.rs: move parts into single files 2019-08-14 13:24:41 +02:00
Dietmar Maurer
e27c5a559f src/backup/data_blob.rs: avoid excessive stack usage 2019-08-14 12:44:08 +02:00
Dietmar Maurer
c638542b4b src/backup/file_formats.rs: moved header_size 2019-08-14 12:35:53 +02:00
Christian Ebner
41d08aa6de src/pxar/fuse.rs: enable libfuse debug output in verbose mode 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-14 08:59:15 +02:00
Christian Ebner
742e64ea0b src/pxar/fuse.rs: cleanup callback interface and store decoder within session context 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-14 08:58:44 +02:00
Christian Ebner
7750b7f2b7 pxar: decoder: take ownership of underlying reader 
By taking ownership it is easier to move the decoder into another struct,
e.g. into a session context in fuse.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-14 08:55:08 +02:00
Christian Ebner
f50b4fd6a0 src/pxar/decoder.rs: cleanup s/CaDirectoryEntry/DirectoryEntry 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-14 08:54:08 +02:00
Dietmar Maurer
6082216343 src/backup/data_blob.rs - CryptReader/CryptWriter: fix buffer size handling 2019-08-13 15:23:15 +02:00
Dietmar Maurer
548c9489d7 src/backup/data_blob.rs - DataBlobReader: impl compressed, encrypted blobs 2019-08-13 13:17:08 +02:00
Dietmar Maurer
2aa0bfff59 src/backup/data_blob.rs - DataBlobReader: implement reader for encrtypted blobs 2019-08-13 13:17:08 +02:00
Dietmar Maurer
e9a385a78e src/backup/data_blob.rs - DataBlobReader: impl compressed, signed blobs 2019-08-12 17:47:25 +02:00
Dietmar Maurer
4bfa147eaf src/backup/data_blob.rs - DataBlobReader: impl. reader for signed blobs 2019-08-12 17:41:25 +02:00
Dietmar Maurer
09785b2795 src/backup/data_blob.rs - DataBlobReader: start serious impl. 2019-08-12 17:20:41 +02:00
Dietmar Maurer
a32bd8a516 src/backup/data_blob.rs - DataBlobWriter: fix decryption 2019-08-12 11:57:29 +02:00
Dietmar Maurer
5d15cb49b4 src/backup/data_blob.rs - compute_crc: start after blob header 2019-08-12 11:32:26 +02:00
Dietmar Maurer
1cbdfd19a2 src/backup/file_formats.rs; use const instzead of static 2019-08-12 11:20:21 +02:00
Dietmar Maurer
18be4ec24a src/backup/data_blob.rs - DataBlobWriter: fix magic number for uncompressed blobs 2019-08-12 10:46:05 +02:00
Dietmar Maurer
5622a3fcdb src/backup/data_blob.rs - DataBlobWriter: impl. compressed encrypted blobs 2019-08-12 10:16:42 +02:00
Dietmar Maurer
f4942e9ffd src/backup/data_blob.rs - DataBlobWriter: impl. encrypted blobs 2019-08-12 10:08:00 +02:00
Dietmar Maurer
c57ec43a53 src/backup/crypt_config.rs: new helper data_crypter to get openssl Crypter 2019-08-12 10:06:51 +02:00
Dietmar Maurer
7776becf5d src/backup/data_blob.rs - DataBlobWriter: impl. compressed signed blobs 2019-08-12 09:01:08 +02:00
Dietmar Maurer
706638f803 src/backup/data_blob.rs: impl. ChecksumWriter 
To correctly compute crc,hmac of final data.
 2019-08-12 08:53:53 +02:00
Dietmar Maurer
f796351c54 src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs 2019-08-12 07:34:41 +02:00
Dietmar Maurer
cb0eea29d9 src/backup/crypt_config.rs: new method to get hmac signer 2019-08-12 07:33:15 +02:00
Dietmar Maurer
a762ce54fd src/backup/data_blob.rs: make DataBlobWriter more generic 
Allow to write compressed and uncompressed blobs.
 2019-08-11 12:14:04 +02:00
Dietmar Maurer
1f26fdef19 src/backup/data_blob.rs: impl. simple reader/writer 
To avoid loading blob into memory.
 2019-08-11 11:32:36 +02:00
Dietmar Maurer
e240d8be0b src/bin/proxmox-backup-client.rs: implement login/logout 2019-08-10 09:12:17 +02:00
Christian Ebner
f71e8cc96f pxar: add command 'mount' to cli of pxar 
Allows to mount an archive to a specified mountpoint via the cli.
Once the archive is mounted, the process is send to the background.
By passing the --verbose flag, the process is kept in foreground and
debug output is provided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-09 16:46:16 +02:00
Dietmar Maurer
02ba8a5d11 pxar: add fuse module and expose its pub functionality. 2019-08-09 16:45:13 +02:00
Christian Ebner
c50f87442c pxar: add basic code for FUSE implementation. 
This adds the basic code in order to create a fuse session and mount an archive.
It adds libfuse3-3 as runtime dependency and libfuse3-dev as build dependency.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-09 16:42:31 +02:00
Dietmar Maurer
781ac11c6a src/backup/data_blob.rs: define const MAX_BLOB_SIZE 2019-08-09 11:49:06 +02:00
Dietmar Maurer
9049a8cfe6 src/bin/proxmox-backup-client.rs: implement catalog command 2019-08-09 11:36:33 +02:00
Dietmar Maurer
7926a3a1cf src/bin/proxmox-backup-client.rs: add archive filename to catalog 2019-08-09 11:05:56 +02:00
Dietmar Maurer
863be2e6e2 src/backup/data_blob.rs: allow largert blob sizes (128MB) 
Need this for catalog files.
 2019-08-09 10:22:56 +02:00
Dietmar Maurer
3fb53e07b4 src/bin/proxmox-backup-client.rs: avoid compiler warning 2019-08-09 10:22:32 +02:00
Dietmar Maurer
543a260ff9 src/bin/proxmox-backup-client.rs: new completion helper for snapshots 2019-08-09 10:08:45 +02:00
Dietmar Maurer
2761d6a4f3 src/pxar/encoder.rs: use BackupCatalogWriter 2019-08-09 09:46:49 +02:00
Dietmar Maurer
aea0815d32 src/pxar/catalog.rs: new catalog helper 2019-08-09 09:41:43 +02:00
Dietmar Maurer
977eeb24f6 src/bin/proxmox-backup-client.rs: avoid doubled file extensions (.blob.blob) in index.json 2019-08-07 10:23:08 +02:00
Dietmar Maurer
4af0ee055a api: always use complete file names (including add exctensions) 2019-08-07 10:10:14 +02:00
Dietmar Maurer
d7c243977d src/api2/admin/datastore.rs - list_snapshot_files: list all files 
Add files that are not mentioned in index.json.
 2019-08-07 09:16:14 +02:00
Dietmar Maurer
34066e2eeb src/api2/backup/environment.rs - add_blob: always verify blob crc 2019-08-07 09:15:13 +02:00
Dietmar Maurer
eecb23560b src/backup/data_blob.rs - encode: always compute crc 2019-08-07 08:30:27 +02:00
Dietmar Maurer
38a6cddad7 src/backup/backup_info.rs: add BackupInfo::new constructor 2019-08-07 08:27:52 +02:00
Dietmar Maurer
c4f025ebbf src/bin/proxmox-backup-client.rs: fix remote filename completion 2019-08-07 06:48:18 +02:00
Dietmar Maurer
dbed4c8cd7 src/client/http_client.rs: compute checksums for chunk streams 2019-08-06 12:23:06 +02:00
Dietmar Maurer
bd15e96dd9 src/client/http_client.rs: minor code cleanup 2019-08-06 11:46:15 +02:00
Dietmar Maurer
f889b15867 src/backup/data_blob.rs: always compute crc 2019-08-06 11:42:14 +02:00
Dietmar Maurer
c807d231eb src/client/http_client.rs: compute file checksums 2019-08-06 11:26:06 +02:00
Dietmar Maurer
a17a0e7a9f src/api2/admin/datastore.rs: extract backup size from index.json 2019-08-06 10:56:21 +02:00
Christian Ebner
eecb182845 pxar: add initial docs for MatchPattern 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-06 09:03:02 +02:00
Christian Ebner
43e892d293 pxar: add error handling for MatchPattern::matches_filename() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-06 09:02:44 +02:00
Dietmar Maurer
8c70e3eb18 src/api2/admin/datastore.rs - files: return data from index.json 2019-08-05 13:22:19 +02:00
Dietmar Maurer
f69adc81a6 use new proxmox::tools::nodename 2019-08-03 17:06:23 +02:00
Dietmar Maurer
7f66c29e5c remove src/tools/common_regex.rs, use proxmox::tools::common_regex instead 2019-08-03 16:44:31 +02:00
Dietmar Maurer
fd329bdcd7 remove src/tools/procfs.rs, use proxmox::sys::linux::procfs instead 2019-08-03 16:26:44 +02:00
Dietmar Maurer
779e4c90b1 src/tools/vec.rs: remove unused file 
We already moved that to proxmox::tools
 2019-08-03 16:11:47 +02:00
Dietmar Maurer
e18a6c9ee5 update to nix 0.14, use code from proxmox:tools 2019-08-03 13:05:38 +02:00
Dietmar Maurer
b86f263ced use new proxmox::sys::linux::magic 2019-08-03 09:28:57 +02:00
Christian Ebner
4ea0f7d95b pxar: remove flag to change digest algorithm 
The flag CA_FORMAT_SHA512_256 is used to switch between sha512 and sha256 to
calculate digest in casync.

As we use sha256, we can get rid of this flag for now.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-03 08:57:53 +02:00
Christian Ebner
4d142ea79e pxar: cleanup: refactor and rename exclude pattern 
The original name PxarExcludePattern makes no sense anymore as the patterns are
also used to match filenames during restore of the archive.

Therefore, exclude_pattern.rs is moved to match_pattern.rs and PxarExcludePattern
rename to MatchPattern.
Further, since it makes more sense the MatchTypes are now declared as None,
Positive, Negative, PartialPositive or PartialNegative, as this makes more sense
and seems more readable.
Positive matches are those without '!' prefix, Negatives with '!' prefix.

This makes also the filename matching in the encoder/decoder more intuitive and
the logic was adapted accordingly.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-03 08:52:32 +02:00
Christian Ebner
fe076c8259 pxar: cleanup: s/PxarDirBuf/PxarDirStack/g and move dir_buffer.rs to dir_stack.rs 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-03 08:52:10 +02:00
Christian Ebner
ead7546a96 pxar: change uid and gid to u32 instead of u64 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-03 08:51:51 +02:00
Christian Ebner
5e50c606b0 pxar: cleanup: s/CA_FORMAT/PXAR/g and s/CaFormat/Pxar/g 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-03 08:51:33 +02:00
Christian Ebner
47651f9530 pxar: cleanup: move feature flags to src/pxar/flags.rs and omit CA_FORMAT prefix on all of them 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-03 08:51:06 +02:00
Christian Ebner
91451c4355 pxar: cleanup: move filesystem related magic numbers to tools/fs/magic.rs 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-03 08:48:12 +02:00
Dietmar Maurer
b335f5b713 src/bin/proxmox-backup-client.rs: sign index.json 2019-08-02 10:00:15 +02:00
Dietmar Maurer
69ecd8d5b0 src/backup/data_blob.rs: implement signed blobs 2019-08-02 09:56:01 +02:00
Dietmar Maurer
93205f942a src/backup/crypt_config.rs: new compute_auth_tag helper 2019-08-02 08:55:37 +02:00
Dietmar Maurer
c68d2170d5 src/backup/crypt_config.rs: fix typo 2019-08-02 08:29:40 +02:00
Christian Ebner
81a9905e0a pxar: implement feature flag support for device nodes, fifos and sockets 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:55:17 +02:00
Christian Ebner
8abc95a145 pxar: remove some unused feature flags from format definition 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:54:46 +02:00
Christian Ebner
3a905cac2b src/pxar/sequential_decoder.rs: remove of obsolete function dir_mkdirat 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:52:49 +02:00
Christian Ebner
beffac999f src/pxar/sequentail_decoder.rs: fix issue when restoring with glob pattern. 
Partial extraction of an archive with a glob pattern, e.g. '**/*.conf' lead to
the unexpected behaviour of restoring all partially matched directories (in this
example all of them).

This patch fixes this unexpected behaviour by only restoring those directories
were the directory or one of its sub-items fully matched the pattern and should
therefore be restored.

To achive this behavoiur, directory metadata is pushed onto a stack and restored
on demand.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:49:47 +02:00
Christian Ebner
51ac99c314 src/pxar/exclude_pattern.rs: add Clone and Copy trait to MatchType 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:49:47 +02:00
Christian Ebner
6a584cfd76 pxar: add PxarDir and PxarDirBuf to buffer directory metadata 
In order to restore only directories when some of their content fully matched
a match pattern on partial restores, these directories and their metadata are
pushed onto this buffer and only restored successivley on demand.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:45:12 +02:00
Dietmar Maurer
27e6e180b6 src/pxar/format_definition.rs - PxarAttributes: derive Default 
And remove unnecessary impl new().
 2019-08-02 07:16:10 +02:00
Christian Ebner
a92e6c9011 src/pxar/sequential_decoder.rs: split read and restore of attributes into separate functions 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:11:38 +02:00
Christian Ebner
e60cfbcdd0 src/pxar/format_definition.rs: add struct PxarAttributes 
This structure contains all the attributes allowing to easily store those within
a e.g. dir buffer.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 07:10:22 +02:00
Christian Ebner
2be3eff5c3 pxar: avoid taking ownership of xattrs, fcaps and quota_projid on corresponding restore functions. 
By borrowing these objects we preserve the functionality but make sure
that ownership doesn't change, avoiding problems when contained within other
structs such as e.g. a buffer storing these attributes.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-08-02 06:44:00 +02:00
Dietmar Maurer
2c3891d1c3 src/bin/proxmox-backup-client.rs: upload backup index.json 
The plan is to use this file to verify the backup content.
 2019-08-01 12:39:02 +02:00
Dietmar Maurer
953d5e1531 src/tools.rs - file_set_contents_full: only call fchmod when we pass permissions 
So that we can use it to write into /etc/pve/ (which does not support chmod).
 2019-07-31 12:44:27 +02:00
Dietmar Maurer
cec17a3ec7 src/bin/proxmox-backup-client.rs - files: allow --repository parameter 2019-07-31 11:45:55 +02:00
Dietmar Maurer
52c171e49d src/bin/proxmox-backup-client.rs: implement files command 2019-07-31 11:23:53 +02:00
Dietmar Maurer
43a406fd3f src/bin/proxmox-backup-client.rs: supress useless output from prune command 2019-07-31 10:15:16 +02:00
Christian Ebner
9731c8b862 pxar: add support for storing/restoring the quota project id on ZFS 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-30 09:40:48 +02:00
Christian Ebner
6a87910949 pxar: implement allow_existing_dirs for pxar decoder 
By default, restoring an archive will fail if files with the same filename
already exist in the target directory.

By setting the allow_existing_dirs flag, the restore will not fail if an
existing directory is encountered.
The metadata (permissions, acls, ...) of the existing directory will be set
to the ones from the archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-30 09:40:30 +02:00
Dietmar Maurer
46d5aa0a09 src/bin/proxmox-backup-client.rs: add restore flag --allow-existing-dirs 
Not jet implemented ...
 2019-07-29 12:49:15 +02:00
Dietmar Maurer
ea7a7ef299 src/bin/proxmox-backup-client.rs: fixup - correctly pass prune parameters 2019-07-27 09:24:23 +02:00
Dietmar Maurer
9fdc3ef46f src/api2/admin/datastore.rs: pass backup type/id (group) to prune 
IMHO, prune over all backup groups is a bit dangerous, considering
that more than one user might use a datastore.
 2019-07-27 08:49:14 +02:00
Dietmar Maurer
7a6cfbd98d src/bin/proxmox-backup-client.rs: correctly compute duration 
And format time with SecondsFormat::Secs
 2019-07-26 11:13:49 +02:00
Dietmar Maurer
bbf9e7e951 src/api2/backup.rs: cleanup schema definitions 2019-07-26 09:07:29 +02:00
Dietmar Maurer
ca5d0b61ca src/api2/backup.rs: new required backup-time parameter 
The client should pass the time as parameter.
 2019-07-25 13:44:01 +02:00
Dietmar Maurer
e128d4e84f src/api2/admin/datastore.rs: backup logs may not be written twice 2019-07-25 12:29:04 +02:00
Dietmar Maurer
ef3254c086 src/api2/admin/datastore.rs: use correct .blob extension for log 2019-07-25 12:22:50 +02:00
Dietmar Maurer
ec34f7eba0 src/bin/proxmox-backup-client.rs: implement upload-log 2019-07-25 12:18:15 +02:00
Dietmar Maurer
04512d3068 src/client/http_client.rs: allow upload with parameters 2019-07-25 12:17:35 +02:00
Dietmar Maurer
07ee223590 src/api2/admin/datastore.rs: add api to upload backup client log file 2019-07-25 12:15:30 +02:00
Dietmar Maurer
79679c2d6e src/bin/proxmox-backup-client.rs: allow to uploag .log files (as binary blobs) 2019-07-24 13:07:02 +02:00
Dietmar Maurer
5b72c9b4f7 src/pxar/encoder.rs: add new skip_lost_and_found parameter 2019-07-24 12:21:25 +02:00
Dietmar Maurer
6c3c9bceb5 pxar_backup_stream.rs: limit lock scope to avoid blocking forever 2019-07-24 11:30:43 +02:00
Dietmar Maurer
684233aa3b remove debug output 2019-07-24 09:33:52 +02:00
Dietmar Maurer
5be106eeae src/client/pxar_backup_stream.rs: correctly pass errors to stream 2019-07-24 09:24:35 +02:00
Dietmar Maurer
2eeaacb974 src/pxar/encoder.rs: allow to pass list of devices 
For better mount point include control...
 2019-07-24 08:11:59 +02:00
Dietmar Maurer
fa5d6977dd Revert previous, commit, use UTC RFC3339 without timezone (Z) 
We now have human readable data/time, and names are still sortable.
 2019-07-22 11:49:30 +02:00
Dietmar Maurer
10c2a21cfe src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339 
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
 2019-07-22 07:57:03 +02:00
Christian Ebner
1fdef50f52 src/pxar/sequential_decoder.rs: Call callback only on restored paths. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-19 11:58:51 +02:00
Dietmar Maurer
40a13369a3 src/bin/pxar.rs - extract: print archive name with --verbose flag 2019-07-19 07:01:57 +02:00
Dietmar Maurer
bbd4365c61 src/bin/pxar.rs: remove wrong debug message 2019-07-19 07:00:17 +02:00
Christian Ebner
a0ec687cd7 src/bin/pxar.rs: allow to pass paths and match patterns as args to pxar extract 
To improve usability it is now possible to directly pass paths or match patterns
as arguments to pxar extract to partially restore an archive.
The patterns provided via CLI are appended to the ones read from file by the
--files-from option in order to have priority over those.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-19 06:50:39 +02:00
Dietmar Maurer
9aa3f68278 src/bin/proxmox-backup-client.rs: code cleanup 2019-07-18 13:16:32 +02:00
Dietmar Maurer
f6ede796ca src/bin/proxmox-backup-client.rs - status: only print result[data] 2019-07-18 09:52:11 +02:00
Dietmar Maurer
15c847f136 src/api2/admin/datastore.rs: allow to list all snapshots 2019-07-18 09:11:43 +02:00
Dietmar Maurer
0f73ee73bb src/cli/command.rs: set exit status on error 2019-07-17 13:40:10 +02:00
Christian Ebner
129dda4729 src/bin/pxar.rs: add --files-from parameter to pxar extract 
Allows to pass a list of match pattern to the pxar extract call by reading them
from file.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 12:20:29 +02:00
Christian Ebner
fa7e957ca3 pxar: Adapt SequentialDecoder::restore() calls to pass empty match pattern for full restore 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 12:19:31 +02:00
Christian Ebner
d32c2e4d27 src/pxar/sequential_decoder.rs: adapt code for partial restore by match pattern 
Allows to partially restore an archive by passing match patterns to the restore
function.
The whole restore is performed in sequential, therefore the whole archive has to
be read.
By wrapping the RawFd into an Option it can be controlled if the corresponding
part is restored (in case of Some(fd)) or if the Reader reads over it
without restore (in case of None).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 12:11:37 +02:00
Christian Ebner
4902291673 src/pxar/sequential_decoder.rs: implement match_filename to match include pattern for partial restores 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 12:07:54 +02:00
Christian Ebner
894cd49ac4 src/pxar/sequential_decoder.rs: Also check for nul bytes when reading the filename 
Check if the filename does not contain invalid nul byes when reading it from the
archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 12:04:41 +02:00
Christian Ebner
0b39674895 src/pxar/sequential_decoder.rs: remove old comments and refactoring 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 09:46:02 +02:00
Christian Ebner
9f8fcdd050 src/pxar/encoder.rs: refactoring of match_exclude_pattern() 
match_exclude_pattern() does not need a '&mut self' reference to the encoder,
move it therefore out of the impl.
Further, this patch contains some naming and formatting cosmetics.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 09:45:19 +02:00
Christian Ebner
3ff4ef28e6 src/pxar/encoder.rs: remove unneeded dir_count 
dir_count was used to track the number of directory entries to store in the
archive and bail if the maximum is exceeded.
As the number of entries is equally obtained from the list of the filenames to
include, use that one instead.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 09:44:59 +02:00
Christian Ebner
a771f90745 src/pxar/exclude_pattern.rs: do not return partial matches for files 
Partial matches make only sense for directories, files are always leafs of the
tree. Take this into account in order to avoid restoring of files which only
matched the front of a match pattern.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 09:44:06 +02:00
Christian Ebner
bdf0d82ced src/bin/pxar.rs: Make pxar extract target optional 
In order to improve usablity, the target on archive extraction will be the
current working directory by default.
A different target can be provided via the optional --target <PATH> parameter.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-17 09:40:02 +02:00
Dietmar Maurer
34a816cc7b src/bin/proxmox-backup-client.rs: add output-fromat options, implement status api 2019-07-16 13:35:25 +02:00
Dietmar Maurer
0eecf38fbf src/api2/admin/datastore.rs: add status api call 2019-07-16 13:34:38 +02:00
Dietmar Maurer
25500cfa26 src/cli/command.rs: add helpers/schema to format output 2019-07-16 11:46:58 +02:00
Dietmar Maurer
aad2ee4947 src/bin/pxar.rs: only print archive name in verbose mode 2019-07-16 08:38:02 +02:00
Dietmar Maurer
6b9a071028 src/bin/pxar.rs: add --verbose parameter to list (replace dump) 2019-07-16 08:31:40 +02:00
Dietmar Maurer
2665cef703 src/bin/proxmox-backup-client.rs: make repository parameter optional 
And allow to read it from $ENV{PBS_REPOSITORY}
 2019-07-16 07:30:04 +02:00
Christian Ebner
3f5192730e src/pxar/format_definition.rs: fix typo 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-15 10:49:34 +02:00
Dietmar Maurer
9eae781ab8 src/bin/pxar.rs - extract: allow to read from stdin 2019-07-05 12:39:25 +02:00
Dietmar Maurer
c6c9e093d0 src/bin/pxar.rs - dump: allow to read from stdin 2019-07-05 12:32:15 +02:00
Dietmar Maurer
bf1252619a src/bin/proxmox-backup-client.rs - restore: allow to pipe output to stdout 2019-07-05 12:14:50 +02:00
Dietmar Maurer
f8100e9623 src/bin/proxmox-backup-client.rs: remove download cli, do everything in restore 2019-07-05 11:36:45 +02:00
Dietmar Maurer
f4bf7dfcc7 src/client/remote_chunk_reader.rs: implement simple caching 2019-07-05 10:42:46 +02:00
Dietmar Maurer
afb4cd28be src/backup/fixed_index.rs: implement BufferedFixedReader 2019-07-05 09:19:56 +02:00
Christian Ebner
be9b39e17a src/pxar/sequential_decoder.rs: major refactoring of restore_sequential() 
This splits the functionality of restore_sequential() into several smaller
functions in order to allow to reuse them when restoring by seeking based on
the goodbye table offsets.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-05 06:45:50 +02:00
Christian Ebner
48a4509c67 src/pxar/encoder.rs: fix bug with incorrect offset stored in goodbye table 
Commit cd7dc87903 introduced the special treatment
for .pxarexclude files when stored in the archive.
The incorrect placement of a code snipplet from this path leads to an incorrect
offset and size stored in the goodbye table.
This fix places the start to the correct position, restoring the previously
correct behaviour.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-05 06:39:39 +02:00
Christian Ebner
7dcbe051e9 pxar: Refactor SequentialDecoder to store the callback function within the struct 
Reduces the number of arguments for the function calls within the decoder.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-05 06:38:22 +02:00
Dietmar Maurer
b46c3fad03 src/backup/fixed_index.rs: use correct size 
We need to consider that the last chunk may have smaller size.
 2019-07-04 15:13:22 +02:00
Dietmar Maurer
9cc88a7c00 src/client/http_client.rs - finish: hold reference to avoid early connection close 2019-07-04 15:11:23 +02:00
Dietmar Maurer
5e04ec707d src/api2/backup/environment.rs: improve chunk alignment/size checks 2019-07-04 13:40:43 +02:00
Dietmar Maurer
a24e3993e0 src/backup/chunk_store.rs: coding style fixes 2019-07-04 11:39:10 +02:00
Dietmar Maurer
e4c2fbf170 src/backup/chunk_store.rs: additionally log chunk count 2019-07-04 11:27:11 +02:00
Dietmar Maurer
9850bcdf19 src/backup/chunk_store.rs: improve error reporting 2019-07-04 11:21:54 +02:00
Wolfgang Bumiller
a3f3e91da2 backup/chunk_store: rework chunk iterator 
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-04 10:55:17 +02:00
Dietmar Maurer
99168f43e6 src/client/http_client.rs: s/set_recv_buf_size/set_recv_buffer_size/ 2019-07-04 10:51:18 +02:00
Dietmar Maurer
c698636a5d src/tools/fs.rs - read_dir: return nix::Result 2019-07-04 10:47:16 +02:00
Dietmar Maurer
a57360983b src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item 2019-07-04 09:26:44 +02:00
Dietmar Maurer
0f0a35b390 src/backup/dynamic_index.rs: lock file inside new, code cleanup 2019-07-04 08:17:30 +02:00
Dietmar Maurer
a7c72ad9eb src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store 2019-07-04 08:13:16 +02:00
Dietmar Maurer
a660978c9a src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats 2019-07-04 07:57:43 +02:00
Dietmar Maurer
86eda3eb0d src/bin/proxmox-backup-client.rs: implement restore using BackupReader 2019-07-03 15:45:12 +02:00
Dietmar Maurer
b2a6897982 src/client/remote_chunk_reader.rs: remove dead code 2019-07-03 14:56:24 +02:00
Dietmar Maurer
7f99bf691a src/client/remote_chunk_reader.rs: implement remote chunk reader 2019-07-03 14:39:13 +02:00
Dietmar Maurer
4f6aaf542c src/client/http_client.rs: avoid compiler warning 2019-07-03 14:36:02 +02:00
Dietmar Maurer
3467cd91e9 src/client/http_client.rs: return Arc<Self> 
Avoid to call canceller for each clone on drop ...
 2019-07-03 14:26:07 +02:00
Dietmar Maurer
1e0784841c src/backup/read_chunk.rs: use &mut self 2019-07-03 14:25:18 +02:00
Christian Ebner
0d70535a36 src/pxar/encoder.rs: refactor simple match statements to map_err() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-03 12:18:36 +02:00
Christian Ebner
6e1c31163b src/pxar/sequential_decoder.rs: refactoring of simple match statements to map_err() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-03 12:18:24 +02:00
Christian Ebner
9d227350a3 src/pxar/sequential_decoder.rs: Cleanup and fix incorrect error message 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-07-03 12:16:59 +02:00
Dietmar Maurer
80af046794 src/server/rest.rs: avoid unwrap 2019-07-03 12:00:43 +02:00
Dietmar Maurer
7fb4f5642a src/server/rest.rs: log peer address, use hyper MakeService 2019-07-03 11:54:35 +02:00
Dietmar Maurer
6d1f61b208 use hyper/tokio-openssl instead of hyper/tokio-tls 
This exposes the complete SSL setup. And download is much faster
now (600MB/s instead of 130MB/s)!
 2019-07-02 13:36:28 +02:00
Dietmar Maurer
74d0a6bc23 src/bin/proxmox-backup-proxy.rs: set socket options for max. performance 2019-07-02 10:48:58 +02:00
Dietmar Maurer
93f8bb0445 src/bin/h2s-client.rs: avoid compiler warnings 2019-07-02 08:59:33 +02:00
Dietmar Maurer
877dd11133 src/backup/{fixed, dynamic}_index.rs: improve error messages 2019-07-02 08:56:56 +02:00
Dietmar Maurer
5c593a4d56 src/backup/read_chunk.rs: use Arc for DataStore 2019-07-02 08:49:16 +02:00
Dietmar Maurer
b850673634 src/backup/read_chunk.rs: move read chunk trait into extra file 
And implement LocalChunkReader.
 2019-07-02 08:22:29 +02:00
Dietmar Maurer
8fad30a4b1 use latest hyper master branch to include socket buffer size fix 2019-07-02 07:09:46 +02:00
Wolfgang Bumiller
f35197f449 replace tools::vec with proxmox::tools::vec 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 11:03:25 +02:00
Wolfgang Bumiller
d0162d53d3 tools: remove io module 
now completely replaced by proxmox::tools::io.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:44:12 +02:00
Wolfgang Bumiller
9b2b627fe0 update remaining users of tools::io::ops 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:44:03 +02:00
Wolfgang Bumiller
9110a69bd8 tools: remove read/write 
now completely replaced by
proxmox::tools::io::{ReadExt, WriteExt}

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:39:42 +02:00
Wolfgang Bumiller
ca3c3ce997 daemon: remove last use of tools::read/write 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:39:13 +02:00
Wolfgang Bumiller
5485b579a1 backup: use new proxmox::tools::io traits 
This replaces all `write_value` calls with `write_le_value`
calls!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-07-01 10:37:02 +02:00
Dietmar Maurer
dc9775d1b0 src/bin/h2s-client.rs: increase tokio tcp stream read buffer size 
This improves download speed by factor 10.
 2019-06-29 18:09:50 +02:00
Dietmar Maurer
34f7131160 src/bin/h2s-client.rs: avoid hyper, use h2 directly 
But performance is still bad.
 2019-06-29 15:58:18 +02:00
Dietmar Maurer
5ed2277f0e src/bin/h2s-server.rs: test to reproduce slow h2 download 
Simplified code, use with h2s-client.rs.
 2019-06-29 14:57:54 +02:00
Dietmar Maurer
17243003b3 src/bin/download-speed.rs: h2 download speedtest for our server 
I get about 30MB/s (much too slow)
 2019-06-29 13:43:10 +02:00
Dietmar Maurer
a2b29b68d4 src/client/http_client.rs: use maximal h2 window sizes 2019-06-29 11:05:36 +02:00
Dietmar Maurer
872debdefa src/bin/h2client.rs: tune h2 client parameters to get reasonable speed 
We now get about 2.5GB/s (not really good, but better)
 2019-06-29 10:41:24 +02:00
Dietmar Maurer
38294bcc56 src/client.rs: make module put_to_stream public 2019-06-29 10:40:43 +02:00
Dietmar Maurer
fded1f3154 add code to test H2 speed 
I currently get about 1MB/s, which is ways too slow (must be a bug)?
 2019-06-29 10:08:12 +02:00
Dietmar Maurer
d48a9955a5 src/backup/dynamic_index.rs: introduce ReadChunk trait 2019-06-28 16:35:00 +02:00
Dietmar Maurer
09d7dc5024 src/api2/reader.rs: add download chunk api 2019-06-28 16:27:01 +02:00
Dietmar Maurer
fcf5dea56d src/client/http_client.rs: increase h2 window size, set tcp nodelay option 
To increase h2 download speed.
 2019-06-28 16:00:58 +02:00
Dietmar Maurer
81a6ce6fde src/backup/chunk_store.rs: new method chunk_path() 
Returns the absolute path.
 2019-06-28 15:48:09 +02:00
Dietmar Maurer
fcfb84fedf file download: avoid unnecessary copy 2019-06-28 07:07:52 +02:00
Dietmar Maurer
984a7c3502 src/client/http_client.rs - h2 download: implement flow control 2019-06-28 07:02:43 +02:00
Dietmar Maurer
dd066d28e2 src/api2/reader.rs: implement backup reader protocol 2019-06-27 09:01:41 +02:00
Dietmar Maurer
42a87f7b96 src/server/h2service.rs: implement generic h2 service 2019-06-26 17:38:33 +02:00
Dietmar Maurer
fb0470837b src/client/http_client.rs: split out code to start new h2 connections 2019-06-26 12:09:18 +02:00
Dietmar Maurer
b208da8393 src/backup/data_{chunk,blob}.rs: add verify_crc method 2019-06-26 09:54:25 +02:00
Dietmar Maurer
fef44d4f78 src/bin/proxmox-backup-client.rs: decode blobs on download 
No more need for that clumsy blob cli (removed).
 2019-06-26 09:18:59 +02:00
Dietmar Maurer
c2b945341c src/client/http_client.rs - download: use generic Write type, return writer. 
Make it possible to write int Vec<u8>.
 2019-06-26 09:17:13 +02:00
Dietmar Maurer
bb8231409e src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json 
Use the KeyConfig serialization with kdf = None.
 2019-06-26 07:32:34 +02:00
Dietmar Maurer
3031e44c58 src/bin/proxmox-backup-client.rs: remove stale download file, start blob cli 2019-06-25 13:00:37 +02:00
Dietmar Maurer
f14a8c9a85 src/api2/admin/datastore.rs: use correct path for download 2019-06-25 12:59:36 +02:00
Dietmar Maurer
81820b0d4d src/client/http_client.rs - download: fix error handling 2019-06-25 12:43:55 +02:00
Dietmar Maurer
45db6f89e1 src/bin/proxmox-backup-client.rs: implement download command 2019-06-25 11:17:24 +02:00
Dietmar Maurer
9e47c0a56e src/api2/admin/datastore.rs: implement generic download method 
We simply allow to download raw binary data.
 2019-06-25 10:16:59 +02:00
Dietmar Maurer
da7d67213e src/api2/admin/datastore/pxar.rs: remove pxar upload api 
Upload needs to be done using the backup protocol.
 2019-06-25 09:45:35 +02:00
Dietmar Maurer
dd8e744f9a src/api2/admin/datastore.rs: use a worker log log prune actions 2019-06-25 08:12:25 +02:00
Dietmar Maurer
6899dbfb47 src/bin/proxmox-backup-client.rs: strip .blob extension 2019-06-25 07:11:02 +02:00
Dietmar Maurer
c6d203bbd3 src/backup/backup_info.rs: also list .blob files 2019-06-25 07:07:45 +02:00
Dietmar Maurer
045e8a568d src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key 2019-06-25 07:00:07 +02:00
Dietmar Maurer
79bc7345e8 src/cli/command.rs: fix ReST output format 2019-06-25 06:19:51 +02:00
Dietmar Maurer
9f46c7de4b src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup 2019-06-24 13:56:37 +02:00
Christian Ebner
cd7dc87903 pxar: impl .pxarexclude parsing and exclude matching 
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.

Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-24 10:26:20 +02:00
Dietmar Maurer
46bd880041 src/api2/backup/environment.rs: new helper add_blob() 2019-06-24 09:35:37 +02:00
Dietmar Maurer
cb08ac3efe src/api2/backup.rs: replace upload_config with upload_blob 2019-06-23 11:44:30 +02:00
Dietmar Maurer
a38c5d4d12 src/backup/data_blob.rs: implement from_raw() 2019-06-23 09:35:44 +02:00
Dietmar Maurer
6f083b7a92 src/api2/backup/upload_chunk.rs: compute crc on upload 2019-06-23 09:04:42 +02:00
Dietmar Maurer
ba01828d38 try to use use proxmox::tools::io::ops::ReadExtOps 2019-06-22 16:29:10 +02:00
Dietmar Maurer
27042ce637 src/backup/file_formats.rs: improve docs 2019-06-22 15:45:36 +02:00
Dietmar Maurer
9f83e0f7ee src/backup/crypt_config.rs: simplify decode interface 2019-06-22 13:24:29 +02:00
Dietmar Maurer
0066c6d972 src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead 2019-06-22 13:02:53 +02:00
Dietmar Maurer
ee8a7e8080 src/backup/crypt_config.rs: implement encrypt_to (output to writer) 2019-06-22 12:25:04 +02:00
Dietmar Maurer
991abfa8b4 src/backup/file_formats.rs: split out file format data 2019-06-22 09:12:25 +02:00
Dietmar Maurer
b7f4f27d6c add crc field for binary blobs formats 2019-06-21 18:49:04 +02:00
Dietmar Maurer
a7f67a9a9c depend on crc32fast 
We can use this to compute fast checksums to test file integrity (not crypto safe).
 2019-06-21 15:58:00 +02:00
Dietmar Maurer
3025b3a53c src/backup/data_blob.rs: new file format for binary blobs 2019-06-21 11:32:07 +02:00
Dietmar Maurer
077a8cae63 src/backup/crypt_config.rs: make code more flexible 
Allow to pass magic numbers as parameters.
 2019-06-21 10:50:48 +02:00
Dietmar Maurer
c38266c18d renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs 2019-06-21 09:51:18 +02:00
Dietmar Maurer
3ea8bfc9c0 src/bin/proxmox-backup-client.rs: add import-master-pubkey command 2019-06-21 07:34:17 +02:00
Dietmar Maurer
37c5a17505 src/bin/proxmox-backup-client.rs: helper to generate RSA master key 2019-06-21 07:12:19 +02:00
Dietmar Maurer
ab44acff57 src/backup/key_derivation.rs: add modified field to key file 2019-06-20 07:58:33 +02:00
Dietmar Maurer
6d0983dbe1 src/bin/proxmox-backup-client.rs: add keyfile parameter for backup 2019-06-19 17:16:41 +02:00
Dietmar Maurer
181f097af9 src/bin/proxmox-backup-client.rs - key API: pass kdf parameter 
Allow to store keys without password.
 2019-06-18 18:21:01 +02:00
Dietmar Maurer
9b06db4563 src/bin/proxmox-backup-client.rs: use file system path to reference keys 2019-06-18 17:38:50 +02:00
Dietmar Maurer
091540298e src/bin/proxmox-backup-client.rs - create_key: always read password from tty 2019-06-18 17:27:52 +02:00
Dietmar Maurer
826f309bf5 src/backup/key_derivation.rs: move kdf code into separate file 2019-06-18 11:17:22 +02:00
Dietmar Maurer
11515438cc Cargo.toml: use serde feature derive 2019-06-18 06:23:25 +02:00
Dietmar Maurer
ac71623483 src/bin/proxmox-backup-client.rs: implement "key change-passphrase" 2019-06-17 13:10:00 +02:00
Dietmar Maurer
f2401311b0 src/bin/proxmox-backup-client.rs: start key management 2019-06-17 10:33:24 +02:00
Dietmar Maurer
a7dd483097 src/backup/*_index.rs: used generated magic numbers 2019-06-14 14:58:37 +02:00
Dietmar Maurer
bffd40d6b7 src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools 2019-06-14 11:40:04 +02:00
Dietmar Maurer
00388226e6 src/api2/backup/environment.rs: log index checksums 2019-06-14 10:36:20 +02:00
Dietmar Maurer
9335d74eba src/backup/fixed_index.rs: compute checksum over the index 2019-06-14 10:34:29 +02:00
Dietmar Maurer
16ff6b7cd8 src/backup/dynamic_index.rs: compute checksum over the index 2019-06-14 10:33:59 +02:00
Dietmar Maurer
36075475a6 src/api2/backup/environment.rs: log duplicate chunks 2019-06-14 07:12:30 +02:00
Dietmar Maurer
fa148dbd11 src/api2/backup/upload_chunk.rs: verify chunk data if possible 2019-06-14 06:23:46 +02:00
Dietmar Maurer
0f37577092 src/bin/cipherbench.rs: add test for zstd compression 2019-06-13 17:24:57 +02:00
Dietmar Maurer
9399c98f82 src/bin/cipherbench.rs: cipher speed test with large blocks 2019-06-13 17:16:43 +02:00
Dietmar Maurer
f640f23ac5 src/backup/data_chunk.rs: fix typo 2019-06-13 16:42:55 +02:00
Dietmar Maurer
dba72d5016 src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:56:50 +02:00
Dietmar Maurer
68ab376ae9 src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed 2019-06-13 12:47:54 +02:00
Dietmar Maurer
f98ac774ee backup: Add support for client side encryption 
first try ...
 2019-06-13 11:47:23 +02:00
Dietmar Maurer
51929e4532 src/backup/data_chunk.rs - decode: make crypt_config optional 2019-06-13 11:44:38 +02:00
Dietmar Maurer
046bcb9773 src/backup/crypt_setup.rs: add comment for possible improvements 2019-06-13 07:35:59 +02:00
Dietmar Maurer
bd0e3c7cfa src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs 2019-06-13 07:35:02 +02:00
Dietmar Maurer
6762db70d6 src/api2/types.rs: define CHUNK_DIGEST_SCHEMA 2019-06-13 07:30:42 +02:00
Dietmar Maurer
b595cb9d2c src/backup/data_chunk.rs: Data Chunk builder 
Make handling encoded chunks easier.
 2019-06-12 13:37:01 +02:00
Dietmar Maurer
ef27200cec src/backup/crypt_setup.rs: fix encoder for uncompressed chunks 2019-06-10 09:56:06 +02:00
Dietmar Maurer
74792b95b2 src/backup/crypt_setup.rs: allow compressed and uncompressed chunks 2019-06-10 08:27:35 +02:00
Dietmar Maurer
9e0187a203 src/backup/crypt_setup.rs: remove unused CryptData struct 2019-06-10 07:55:36 +02:00
Dietmar Maurer
bec8498ae3 src/backup/crypt_setup.rs: avoid useless allocations 2019-06-09 11:44:17 +02:00
Dietmar Maurer
c20b17b460 src/backup.rs: define magic numbers for chunk store file format 2019-06-08 10:34:20 +02:00
Dietmar Maurer
48b4b40b57 src/backup/crypt_setup.rs: crypto helpers 2019-06-08 09:56:51 +02:00
Wolfgang Bumiller
dd5495d6dc tree-wide: use 'dyn' for all trait objects 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-06-07 13:13:48 +02:00
Christian Ebner
e993db91cd pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder 
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:08:20 +02:00
Christian Ebner
1f319e766e src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:08:06 +02:00
Christian Ebner
9b77388f2a src/pxar/encoder.rs: minor refactoring 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:07:55 +02:00
Christian Ebner
a4cc8eb7c5 src/pxar/encoder.rs: check if ioctl supported by filesystem 
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-07 11:07:31 +02:00
Dietmar Maurer
4800df7ba6 src/api2/admin/datastore.rs: remove test-upload api 2019-06-07 07:52:11 +02:00
Dietmar Maurer
113898d7e9 src/api2/admin/datastore/upload.rs: remove unused upload api 2019-06-07 07:39:41 +02:00
Dietmar Maurer
4247fccb0f src/client/http_client.rs: automatically close connection in finish 2019-06-05 11:41:50 +02:00
Wolfgang Bumiller
cb4426b348 make h2 client connection cancellable 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-06-05 09:45:59 +02:00
Dietmar Maurer
986bef16be src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1 
So that we can include it in static doc strings.
 2019-06-05 08:41:20 +02:00
Dietmar Maurer
c9ec0956cf src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1 2019-06-05 08:12:13 +02:00
Dietmar Maurer
44c54845b3 src/tools/xattr.rs: fix test when run as root 2019-06-05 07:57:42 +02:00
Dietmar Maurer
7773ccc11f src/api2.rs: move backup api to /backup 2019-06-05 07:23:21 +02:00
Christian Ebner
97eeea3b4a src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:29:02 +02:00
Dietmar Maurer
1ce2f7cfb9 src/pxar/encoder.rs: style fix 2019-06-05 06:27:39 +02:00
Christian Ebner
e7b60a16c8 pxar: impl storing/dumping/restoring of quota project ids 
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:26:02 +02:00
Christian Ebner
7312ab9eef src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID 
Defines the types needed to store/restore quota project ids in pxar

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:23:06 +02:00
Christian Ebner
042babe4e7 src/tools/fs.rs: impl ioctl calls to get/set fsxattr 
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-06-05 06:22:49 +02:00
Dietmar Maurer
22e5aa1d3e src/bin/dump-backup-api.rs: helper to generate backup API docs 2019-06-04 13:12:42 +02:00
Dietmar Maurer
5e0f305142 src/api_schema/format.rs: depend on textwrap, new wrap_text helper 2019-06-04 12:42:02 +02:00
Dietmar Maurer
339ddfcbfa src/cli/command.rs: move doc generator code to src/api_schema/format.rs 2019-06-04 12:32:22 +02:00
Dietmar Maurer
ef39bf95b3 src/client/http_client.rs - upload_config: improve error message 2019-06-03 10:53:09 +02:00
Dietmar Maurer
ec8a9bb953 src/bin/proxmox-backup-client.rs: implement upload_config 2019-06-03 10:39:44 +02:00
Dietmar Maurer
39d6846e1e src/api2/admin/datastore/backup.rs: implement config file upload 2019-06-03 09:45:14 +02:00