Wolfgang Bumiller
aa4110cc64
src/server/state.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
91e4587343
src/server/rest.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
b9b7f7ec1f
src/server/h2service.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
160fc8147f
src/server/command_socket.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
3c0facc787
src/client/remote_chunk_reader.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
369a87e3a2
src/client/pxar_backup_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
2107bb40c1
src/client/pipe_to_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
5b3911995b
src/client/merge_known_chunks.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
a6782ca10b
src/client/http_client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
8630557918
src/bin/upload-speed.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
cab6816969
src/bin/test_chunk_speed2.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
fda5797b8a
src/bin/proxmox-backup-proxy.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
e9722f8bde
src/bin/proxmox-backup-client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
e76ac3a499
src/bin/proxmox-backup-api.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
e235c8f719
src/bin/h2s-server.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
74be6dc9b7
src/bin/h2server.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
55d8a631fc
src/bin/h2s-client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
15d0e4a3bd
src/bin/h2client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
b9203d87f4
src/bin/download-speed.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
745e652a7f
src/backup/index.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
32bef1e2d1
src/backup/chunk_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
f2d9b73c91
src/api_schema/router.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
ffb6434485
src/api2/reader.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
7622005574
src/api2/backup/upload_chunk.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
59b2baa0f6
src/api2/backup.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
6be147b78c
src/api2/admin/datastore.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
5c1130df9f
implement Sync for DynamicIndexReader and FixedIndexReader
...
hyper's wrap_stream now needs this
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
236761a3e6
drop src/storage/futures.rs
...
it's unused and uses a tokio::Task I don't want to lookup
the std-future conversion for...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Dietmar Maurer
02fcf372e4
rc/bin/proxmox-backup-client.rs - restore: always download index.json.blob
2019-09-02 14:14:32 +02:00
Dietmar Maurer
0d9862803b
src/bin/proxmox-backup-client.rs - reatore: avoid loading (large) blobs into memory
2019-09-02 14:13:31 +02:00
Christian Ebner
66c8eb9383
src/client/http_client.rs: Refactor handling Option and Result types
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-29 09:15:16 +02:00
Wolfgang Bumiller
40f8680e37
tools: remove SignalFd
...
We don't use it anymore and it'll just be more work when
switching to async-await.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 13:44:53 +02:00
Wolfgang Bumiller
d21f8a5b5b
backup/dynamic_index: explicitly zero-initialize the header
...
The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
Wolfgang Bumiller
990b930f22
backup/dynamic_index: use static assertion, fix size
...
The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
Christian Ebner
bcb664cb69
pxar/fuse: add pxar cli flag to set single- or multi-threaded session loop
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:43:14 +02:00
Christian Ebner
de2791868a
src/pxar/sequentail_decoder.rs: followup: cleanup S_IFMT masking
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:42:22 +02:00
Christian Ebner
9cb9e1b7db
src/pxar/sequential_decoder.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:42:19 +02:00
Christian Ebner
dac88033ba
src/pxar/dir_stack.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:39:14 +02:00
Christian Ebner
b69169e7be
src/pxar/fuse.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:58 +02:00
Christian Ebner
d792dc3cd2
src/pxar/match_pattern.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:56 +02:00
Christian Ebner
4ea22b68d9
src/pxar/format_definition.rs: style fixup
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:53 +02:00
Wolfgang Bumiller
7d83440c60
remove proxmox-protocol subcrate
...
AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)
Moved the chunker back into src/backup/chunker.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 14:08:25 +02:00
Wolfgang Bumiller
7a57cb77e1
more formatting & use statement fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:50:27 +02:00
Christian Ebner
35dfd3ceb7
src/pxar/encoder.rs: style fixups by rustfmt
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:29:11 +02:00
Christian Ebner
1af30bc2fe
(pxar: replace deprecated std::mem::uninitialized()
...
... and use std::mem::MaybeUninit or proxmox::tools::vec::uninitialized() instead.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:27:25 +02:00
Wolfgang Bumiller
5e58e1bb7d
{dynamic,fixed}_index: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:35:36 +02:00
Wolfgang Bumiller
f93b55b057
src/backup/index: style fixup & unsafe copy removal
...
We can use the safe .copy_from_slice alternative in this
case.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:31:45 +02:00
Wolfgang Bumiller
fa17b1ce2a
src/backup/index: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:27:35 +02:00
Wolfgang Bumiller
f3a8d1d7e0
api2/node/time: replace mem::uninitialized
...
and fixup use statements
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:14:41 +02:00
Wolfgang Bumiller
48b85e8e3b
tools: tty: replace mem::uninitialized and style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:14:19 +02:00
Wolfgang Bumiller
b528fddfc6
tools: timer: style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:07:56 +02:00
Wolfgang Bumiller
b7f44ace92
timer: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:07:10 +02:00
Wolfgang Bumiller
6100071f4e
tools: style & use statement fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 10:57:56 +02:00
Wolfgang Bumiller
5c20e2da6b
src/config: more style fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:14:26 +02:00
Wolfgang Bumiller
91640ab567
src/config: style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:11:07 +02:00
Wolfgang Bumiller
928650c4eb
src/config: use statement cleanup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:10:50 +02:00
Wolfgang Bumiller
2ec979e4ed
/var/run -> /run
...
/var/run is considered deprecated and for instance in
systemd unit files lintian complains...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 12:28:24 +02:00
Wolfgang Bumiller
35238e234d
switch from create_dir_chown to create_path
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 12:26:10 +02:00
Christian Ebner
e5471b4836
pxar/decoder: avoid failing on hardlinks and return corresponding target file entry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
[Fixup: remove unnecessary 'mut']
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:32:52 +02:00
Christian Ebner
58262f40f7
src/pxar/decoder.rs: get correct stats for root directory
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:31:41 +02:00
Christian Ebner
7d26720e79
src/pxar/decoder.rs: move GOODBYE_ITEM_SIZE from function to module scope.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:31:39 +02:00
Wolfgang Bumiller
367f002ee2
use statement cleanup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-16 12:28:40 +02:00
Dietmar Maurer
9d135fe617
src/backup/catalog_blob.rs: moved catalog impl. from pxar
...
And avoid loading catalog into memory.
2019-08-16 12:27:17 +02:00
Wolfgang Bumiller
9025312aa6
avoid lifetimes in blob reader/writer
2019-08-16 10:07:24 +02:00
Dietmar Maurer
71d08e00b7
src/backup/data_blob_reader.rs: fix compiler warning
2019-08-16 08:10:39 +02:00
Christian Ebner
ac0b435344
src/pxar/fuse.rs: fix compiler warnings
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 15:14:57 +02:00
Dietmar Maurer
a84ef4c205
src/bin/proxmox-backup-client.rs: avoid loading catalog into memory
...
We can use the new DataBlobReader instead.
2019-08-14 15:07:28 +02:00
Dietmar Maurer
b791804f4b
src/backup/data_blob_reader.rs: impl for Read instead of BufRead
2019-08-14 14:56:01 +02:00
Dietmar Maurer
39a4df61d6
improve docs
2019-08-14 14:08:27 +02:00
Dietmar Maurer
018d11bb84
src/backup/data_blob.rs: move parts into single files
2019-08-14 13:24:41 +02:00
Dietmar Maurer
e27c5a559f
src/backup/data_blob.rs: avoid excessive stack usage
2019-08-14 12:44:08 +02:00
Dietmar Maurer
c638542b4b
src/backup/file_formats.rs: moved header_size
2019-08-14 12:35:53 +02:00
Christian Ebner
41d08aa6de
src/pxar/fuse.rs: enable libfuse debug output in verbose mode
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:59:15 +02:00
Christian Ebner
742e64ea0b
src/pxar/fuse.rs: cleanup callback interface and store decoder within session context
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:58:44 +02:00
Christian Ebner
7750b7f2b7
pxar: decoder: take ownership of underlying reader
...
By taking ownership it is easier to move the decoder into another struct,
e.g. into a session context in fuse.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:55:08 +02:00
Christian Ebner
f50b4fd6a0
src/pxar/decoder.rs: cleanup s/CaDirectoryEntry/DirectoryEntry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:54:08 +02:00
Dietmar Maurer
6082216343
src/backup/data_blob.rs - CryptReader/CryptWriter: fix buffer size handling
2019-08-13 15:23:15 +02:00
Dietmar Maurer
548c9489d7
src/backup/data_blob.rs - DataBlobReader: impl compressed, encrypted blobs
2019-08-13 13:17:08 +02:00
Dietmar Maurer
2aa0bfff59
src/backup/data_blob.rs - DataBlobReader: implement reader for encrtypted blobs
2019-08-13 13:17:08 +02:00
Dietmar Maurer
e9a385a78e
src/backup/data_blob.rs - DataBlobReader: impl compressed, signed blobs
2019-08-12 17:47:25 +02:00
Dietmar Maurer
4bfa147eaf
src/backup/data_blob.rs - DataBlobReader: impl. reader for signed blobs
2019-08-12 17:41:25 +02:00
Dietmar Maurer
09785b2795
src/backup/data_blob.rs - DataBlobReader: start serious impl.
2019-08-12 17:20:41 +02:00
Dietmar Maurer
a32bd8a516
src/backup/data_blob.rs - DataBlobWriter: fix decryption
2019-08-12 11:57:29 +02:00
Dietmar Maurer
5d15cb49b4
src/backup/data_blob.rs - compute_crc: start after blob header
2019-08-12 11:32:26 +02:00
Dietmar Maurer
1cbdfd19a2
src/backup/file_formats.rs; use const instzead of static
2019-08-12 11:20:21 +02:00
Dietmar Maurer
18be4ec24a
src/backup/data_blob.rs - DataBlobWriter: fix magic number for uncompressed blobs
2019-08-12 10:46:05 +02:00
Dietmar Maurer
5622a3fcdb
src/backup/data_blob.rs - DataBlobWriter: impl. compressed encrypted blobs
2019-08-12 10:16:42 +02:00
Dietmar Maurer
f4942e9ffd
src/backup/data_blob.rs - DataBlobWriter: impl. encrypted blobs
2019-08-12 10:08:00 +02:00
Dietmar Maurer
c57ec43a53
src/backup/crypt_config.rs: new helper data_crypter to get openssl Crypter
2019-08-12 10:06:51 +02:00
Dietmar Maurer
7776becf5d
src/backup/data_blob.rs - DataBlobWriter: impl. compressed signed blobs
2019-08-12 09:01:08 +02:00
Dietmar Maurer
706638f803
src/backup/data_blob.rs: impl. ChecksumWriter
...
To correctly compute crc,hmac of final data.
2019-08-12 08:53:53 +02:00
Dietmar Maurer
f796351c54
src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs
2019-08-12 07:34:41 +02:00
Dietmar Maurer
cb0eea29d9
src/backup/crypt_config.rs: new method to get hmac signer
2019-08-12 07:33:15 +02:00
Dietmar Maurer
a762ce54fd
src/backup/data_blob.rs: make DataBlobWriter more generic
...
Allow to write compressed and uncompressed blobs.
2019-08-11 12:14:04 +02:00
Dietmar Maurer
1f26fdef19
src/backup/data_blob.rs: impl. simple reader/writer
...
To avoid loading blob into memory.
2019-08-11 11:32:36 +02:00
Dietmar Maurer
e240d8be0b
src/bin/proxmox-backup-client.rs: implement login/logout
2019-08-10 09:12:17 +02:00
Christian Ebner
f71e8cc96f
pxar: add command 'mount' to cli of pxar
...
Allows to mount an archive to a specified mountpoint via the cli.
Once the archive is mounted, the process is send to the background.
By passing the --verbose flag, the process is kept in foreground and
debug output is provided.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-09 16:46:16 +02:00
Dietmar Maurer
02ba8a5d11
pxar: add fuse module and expose its pub functionality.
2019-08-09 16:45:13 +02:00
Christian Ebner
c50f87442c
pxar: add basic code for FUSE implementation.
...
This adds the basic code in order to create a fuse session and mount an archive.
It adds libfuse3-3 as runtime dependency and libfuse3-dev as build dependency.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-09 16:42:31 +02:00
Dietmar Maurer
781ac11c6a
src/backup/data_blob.rs: define const MAX_BLOB_SIZE
2019-08-09 11:49:06 +02:00
Dietmar Maurer
9049a8cfe6
src/bin/proxmox-backup-client.rs: implement catalog command
2019-08-09 11:36:33 +02:00
Dietmar Maurer
7926a3a1cf
src/bin/proxmox-backup-client.rs: add archive filename to catalog
2019-08-09 11:05:56 +02:00
Dietmar Maurer
863be2e6e2
src/backup/data_blob.rs: allow largert blob sizes (128MB)
...
Need this for catalog files.
2019-08-09 10:22:56 +02:00
Dietmar Maurer
3fb53e07b4
src/bin/proxmox-backup-client.rs: avoid compiler warning
2019-08-09 10:22:32 +02:00
Dietmar Maurer
543a260ff9
src/bin/proxmox-backup-client.rs: new completion helper for snapshots
2019-08-09 10:08:45 +02:00
Dietmar Maurer
2761d6a4f3
src/pxar/encoder.rs: use BackupCatalogWriter
2019-08-09 09:46:49 +02:00
Dietmar Maurer
aea0815d32
src/pxar/catalog.rs: new catalog helper
2019-08-09 09:41:43 +02:00
Dietmar Maurer
977eeb24f6
src/bin/proxmox-backup-client.rs: avoid doubled file extensions (.blob.blob) in index.json
2019-08-07 10:23:08 +02:00
Dietmar Maurer
4af0ee055a
api: always use complete file names (including add exctensions)
2019-08-07 10:10:14 +02:00
Dietmar Maurer
d7c243977d
src/api2/admin/datastore.rs - list_snapshot_files: list all files
...
Add files that are not mentioned in index.json.
2019-08-07 09:16:14 +02:00
Dietmar Maurer
34066e2eeb
src/api2/backup/environment.rs - add_blob: always verify blob crc
2019-08-07 09:15:13 +02:00
Dietmar Maurer
eecb23560b
src/backup/data_blob.rs - encode: always compute crc
2019-08-07 08:30:27 +02:00
Dietmar Maurer
38a6cddad7
src/backup/backup_info.rs: add BackupInfo::new constructor
2019-08-07 08:27:52 +02:00
Dietmar Maurer
c4f025ebbf
src/bin/proxmox-backup-client.rs: fix remote filename completion
2019-08-07 06:48:18 +02:00
Dietmar Maurer
dbed4c8cd7
src/client/http_client.rs: compute checksums for chunk streams
2019-08-06 12:23:06 +02:00
Dietmar Maurer
bd15e96dd9
src/client/http_client.rs: minor code cleanup
2019-08-06 11:46:15 +02:00
Dietmar Maurer
f889b15867
src/backup/data_blob.rs: always compute crc
2019-08-06 11:42:14 +02:00
Dietmar Maurer
c807d231eb
src/client/http_client.rs: compute file checksums
2019-08-06 11:26:06 +02:00
Dietmar Maurer
a17a0e7a9f
src/api2/admin/datastore.rs: extract backup size from index.json
2019-08-06 10:56:21 +02:00
Christian Ebner
eecb182845
pxar: add initial docs for MatchPattern
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-06 09:03:02 +02:00
Christian Ebner
43e892d293
pxar: add error handling for MatchPattern::matches_filename()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-06 09:02:44 +02:00
Dietmar Maurer
8c70e3eb18
src/api2/admin/datastore.rs - files: return data from index.json
2019-08-05 13:22:19 +02:00
Dietmar Maurer
f69adc81a6
use new proxmox::tools::nodename
2019-08-03 17:06:23 +02:00
Dietmar Maurer
7f66c29e5c
remove src/tools/common_regex.rs, use proxmox::tools::common_regex instead
2019-08-03 16:44:31 +02:00
Dietmar Maurer
fd329bdcd7
remove src/tools/procfs.rs, use proxmox::sys::linux::procfs instead
2019-08-03 16:26:44 +02:00
Dietmar Maurer
779e4c90b1
src/tools/vec.rs: remove unused file
...
We already moved that to proxmox::tools
2019-08-03 16:11:47 +02:00
Dietmar Maurer
e18a6c9ee5
update to nix 0.14, use code from proxmox:tools
2019-08-03 13:05:38 +02:00
Dietmar Maurer
b86f263ced
use new proxmox::sys::linux::magic
2019-08-03 09:28:57 +02:00
Christian Ebner
4ea0f7d95b
pxar: remove flag to change digest algorithm
...
The flag CA_FORMAT_SHA512_256 is used to switch between sha512 and sha256 to
calculate digest in casync.
As we use sha256, we can get rid of this flag for now.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:57:53 +02:00
Christian Ebner
4d142ea79e
pxar: cleanup: refactor and rename exclude pattern
...
The original name PxarExcludePattern makes no sense anymore as the patterns are
also used to match filenames during restore of the archive.
Therefore, exclude_pattern.rs is moved to match_pattern.rs and PxarExcludePattern
rename to MatchPattern.
Further, since it makes more sense the MatchTypes are now declared as None,
Positive, Negative, PartialPositive or PartialNegative, as this makes more sense
and seems more readable.
Positive matches are those without '!' prefix, Negatives with '!' prefix.
This makes also the filename matching in the encoder/decoder more intuitive and
the logic was adapted accordingly.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:52:32 +02:00
Christian Ebner
fe076c8259
pxar: cleanup: s/PxarDirBuf/PxarDirStack/g and move dir_buffer.rs to dir_stack.rs
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:52:10 +02:00
Christian Ebner
ead7546a96
pxar: change uid and gid to u32 instead of u64
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:51 +02:00
Christian Ebner
5e50c606b0
pxar: cleanup: s/CA_FORMAT/PXAR/g and s/CaFormat/Pxar/g
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:33 +02:00
Christian Ebner
47651f9530
pxar: cleanup: move feature flags to src/pxar/flags.rs and omit CA_FORMAT prefix on all of them
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:06 +02:00
Christian Ebner
91451c4355
pxar: cleanup: move filesystem related magic numbers to tools/fs/magic.rs
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:48:12 +02:00
Dietmar Maurer
b335f5b713
src/bin/proxmox-backup-client.rs: sign index.json
2019-08-02 10:00:15 +02:00
Dietmar Maurer
69ecd8d5b0
src/backup/data_blob.rs: implement signed blobs
2019-08-02 09:56:01 +02:00
Dietmar Maurer
93205f942a
src/backup/crypt_config.rs: new compute_auth_tag helper
2019-08-02 08:55:37 +02:00
Dietmar Maurer
c68d2170d5
src/backup/crypt_config.rs: fix typo
2019-08-02 08:29:40 +02:00
Christian Ebner
81a9905e0a
pxar: implement feature flag support for device nodes, fifos and sockets
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:55:17 +02:00
Christian Ebner
8abc95a145
pxar: remove some unused feature flags from format definition
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:54:46 +02:00
Christian Ebner
3a905cac2b
src/pxar/sequential_decoder.rs: remove of obsolete function dir_mkdirat
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:52:49 +02:00
Christian Ebner
beffac999f
src/pxar/sequentail_decoder.rs: fix issue when restoring with glob pattern.
...
Partial extraction of an archive with a glob pattern, e.g. '**/*.conf' lead to
the unexpected behaviour of restoring all partially matched directories (in this
example all of them).
This patch fixes this unexpected behaviour by only restoring those directories
were the directory or one of its sub-items fully matched the pattern and should
therefore be restored.
To achive this behavoiur, directory metadata is pushed onto a stack and restored
on demand.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:49:47 +02:00
Christian Ebner
51ac99c314
src/pxar/exclude_pattern.rs: add Clone and Copy trait to MatchType
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:49:47 +02:00
Christian Ebner
6a584cfd76
pxar: add PxarDir and PxarDirBuf to buffer directory metadata
...
In order to restore only directories when some of their content fully matched
a match pattern on partial restores, these directories and their metadata are
pushed onto this buffer and only restored successivley on demand.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:45:12 +02:00
Dietmar Maurer
27e6e180b6
src/pxar/format_definition.rs - PxarAttributes: derive Default
...
And remove unnecessary impl new().
2019-08-02 07:16:10 +02:00
Christian Ebner
a92e6c9011
src/pxar/sequential_decoder.rs: split read and restore of attributes into separate functions
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:11:38 +02:00
Christian Ebner
e60cfbcdd0
src/pxar/format_definition.rs: add struct PxarAttributes
...
This structure contains all the attributes allowing to easily store those within
a e.g. dir buffer.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:10:22 +02:00
Christian Ebner
2be3eff5c3
pxar: avoid taking ownership of xattrs, fcaps and quota_projid on corresponding restore functions.
...
By borrowing these objects we preserve the functionality but make sure
that ownership doesn't change, avoiding problems when contained within other
structs such as e.g. a buffer storing these attributes.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 06:44:00 +02:00
Dietmar Maurer
2c3891d1c3
src/bin/proxmox-backup-client.rs: upload backup index.json
...
The plan is to use this file to verify the backup content.
2019-08-01 12:39:02 +02:00
Dietmar Maurer
953d5e1531
src/tools.rs - file_set_contents_full: only call fchmod when we pass permissions
...
So that we can use it to write into /etc/pve/ (which does not support chmod).
2019-07-31 12:44:27 +02:00
Dietmar Maurer
cec17a3ec7
src/bin/proxmox-backup-client.rs - files: allow --repository parameter
2019-07-31 11:45:55 +02:00
Dietmar Maurer
52c171e49d
src/bin/proxmox-backup-client.rs: implement files command
2019-07-31 11:23:53 +02:00
Dietmar Maurer
43a406fd3f
src/bin/proxmox-backup-client.rs: supress useless output from prune command
2019-07-31 10:15:16 +02:00
Christian Ebner
9731c8b862
pxar: add support for storing/restoring the quota project id on ZFS
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-30 09:40:48 +02:00
Christian Ebner
6a87910949
pxar: implement allow_existing_dirs for pxar decoder
...
By default, restoring an archive will fail if files with the same filename
already exist in the target directory.
By setting the allow_existing_dirs flag, the restore will not fail if an
existing directory is encountered.
The metadata (permissions, acls, ...) of the existing directory will be set
to the ones from the archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-30 09:40:30 +02:00
Dietmar Maurer
46d5aa0a09
src/bin/proxmox-backup-client.rs: add restore flag --allow-existing-dirs
...
Not jet implemented ...
2019-07-29 12:49:15 +02:00
Dietmar Maurer
ea7a7ef299
src/bin/proxmox-backup-client.rs: fixup - correctly pass prune parameters
2019-07-27 09:24:23 +02:00
Dietmar Maurer
9fdc3ef46f
src/api2/admin/datastore.rs: pass backup type/id (group) to prune
...
IMHO, prune over all backup groups is a bit dangerous, considering
that more than one user might use a datastore.
2019-07-27 08:49:14 +02:00
Dietmar Maurer
7a6cfbd98d
src/bin/proxmox-backup-client.rs: correctly compute duration
...
And format time with SecondsFormat::Secs
2019-07-26 11:13:49 +02:00
Dietmar Maurer
bbf9e7e951
src/api2/backup.rs: cleanup schema definitions
2019-07-26 09:07:29 +02:00
Dietmar Maurer
ca5d0b61ca
src/api2/backup.rs: new required backup-time parameter
...
The client should pass the time as parameter.
2019-07-25 13:44:01 +02:00
Dietmar Maurer
e128d4e84f
src/api2/admin/datastore.rs: backup logs may not be written twice
2019-07-25 12:29:04 +02:00
Dietmar Maurer
ef3254c086
src/api2/admin/datastore.rs: use correct .blob extension for log
2019-07-25 12:22:50 +02:00
Dietmar Maurer
ec34f7eba0
src/bin/proxmox-backup-client.rs: implement upload-log
2019-07-25 12:18:15 +02:00
Dietmar Maurer
04512d3068
src/client/http_client.rs: allow upload with parameters
2019-07-25 12:17:35 +02:00
Dietmar Maurer
07ee223590
src/api2/admin/datastore.rs: add api to upload backup client log file
2019-07-25 12:15:30 +02:00
Dietmar Maurer
79679c2d6e
src/bin/proxmox-backup-client.rs: allow to uploag .log files (as binary blobs)
2019-07-24 13:07:02 +02:00
Dietmar Maurer
5b72c9b4f7
src/pxar/encoder.rs: add new skip_lost_and_found parameter
2019-07-24 12:21:25 +02:00
Dietmar Maurer
6c3c9bceb5
pxar_backup_stream.rs: limit lock scope to avoid blocking forever
2019-07-24 11:30:43 +02:00
Dietmar Maurer
684233aa3b
remove debug output
2019-07-24 09:33:52 +02:00
Dietmar Maurer
5be106eeae
src/client/pxar_backup_stream.rs: correctly pass errors to stream
2019-07-24 09:24:35 +02:00
Dietmar Maurer
2eeaacb974
src/pxar/encoder.rs: allow to pass list of devices
...
For better mount point include control...
2019-07-24 08:11:59 +02:00
Dietmar Maurer
fa5d6977dd
Revert previous, commit, use UTC RFC3339 without timezone (Z)
...
We now have human readable data/time, and names are still sortable.
2019-07-22 11:49:30 +02:00
Dietmar Maurer
10c2a21cfe
src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339
...
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
2019-07-22 07:57:03 +02:00
Christian Ebner
1fdef50f52
src/pxar/sequential_decoder.rs: Call callback only on restored paths.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-19 11:58:51 +02:00
Dietmar Maurer
40a13369a3
src/bin/pxar.rs - extract: print archive name with --verbose flag
2019-07-19 07:01:57 +02:00
Dietmar Maurer
bbd4365c61
src/bin/pxar.rs: remove wrong debug message
2019-07-19 07:00:17 +02:00
Christian Ebner
a0ec687cd7
src/bin/pxar.rs: allow to pass paths and match patterns as args to pxar extract
...
To improve usability it is now possible to directly pass paths or match patterns
as arguments to pxar extract to partially restore an archive.
The patterns provided via CLI are appended to the ones read from file by the
--files-from option in order to have priority over those.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-19 06:50:39 +02:00
Dietmar Maurer
9aa3f68278
src/bin/proxmox-backup-client.rs: code cleanup
2019-07-18 13:16:32 +02:00
Dietmar Maurer
f6ede796ca
src/bin/proxmox-backup-client.rs - status: only print result[data]
2019-07-18 09:52:11 +02:00
Dietmar Maurer
15c847f136
src/api2/admin/datastore.rs: allow to list all snapshots
2019-07-18 09:11:43 +02:00
Dietmar Maurer
0f73ee73bb
src/cli/command.rs: set exit status on error
2019-07-17 13:40:10 +02:00
Christian Ebner
129dda4729
src/bin/pxar.rs: add --files-from parameter to pxar extract
...
Allows to pass a list of match pattern to the pxar extract call by reading them
from file.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:20:29 +02:00
Christian Ebner
fa7e957ca3
pxar: Adapt SequentialDecoder::restore() calls to pass empty match pattern for full restore
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:19:31 +02:00
Christian Ebner
d32c2e4d27
src/pxar/sequential_decoder.rs: adapt code for partial restore by match pattern
...
Allows to partially restore an archive by passing match patterns to the restore
function.
The whole restore is performed in sequential, therefore the whole archive has to
be read.
By wrapping the RawFd into an Option it can be controlled if the corresponding
part is restored (in case of Some(fd)) or if the Reader reads over it
without restore (in case of None).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:11:37 +02:00
Christian Ebner
4902291673
src/pxar/sequential_decoder.rs: implement match_filename to match include pattern for partial restores
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:07:54 +02:00
Christian Ebner
894cd49ac4
src/pxar/sequential_decoder.rs: Also check for nul bytes when reading the filename
...
Check if the filename does not contain invalid nul byes when reading it from the
archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:04:41 +02:00
Christian Ebner
0b39674895
src/pxar/sequential_decoder.rs: remove old comments and refactoring
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:46:02 +02:00
Christian Ebner
9f8fcdd050
src/pxar/encoder.rs: refactoring of match_exclude_pattern()
...
match_exclude_pattern() does not need a '&mut self' reference to the encoder,
move it therefore out of the impl.
Further, this patch contains some naming and formatting cosmetics.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:45:19 +02:00
Christian Ebner
3ff4ef28e6
src/pxar/encoder.rs: remove unneeded dir_count
...
dir_count was used to track the number of directory entries to store in the
archive and bail if the maximum is exceeded.
As the number of entries is equally obtained from the list of the filenames to
include, use that one instead.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:44:59 +02:00
Christian Ebner
a771f90745
src/pxar/exclude_pattern.rs: do not return partial matches for files
...
Partial matches make only sense for directories, files are always leafs of the
tree. Take this into account in order to avoid restoring of files which only
matched the front of a match pattern.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:44:06 +02:00
Christian Ebner
bdf0d82ced
src/bin/pxar.rs: Make pxar extract target optional
...
In order to improve usablity, the target on archive extraction will be the
current working directory by default.
A different target can be provided via the optional --target <PATH> parameter.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:40:02 +02:00
Dietmar Maurer
34a816cc7b
src/bin/proxmox-backup-client.rs: add output-fromat options, implement status api
2019-07-16 13:35:25 +02:00
Dietmar Maurer
0eecf38fbf
src/api2/admin/datastore.rs: add status api call
2019-07-16 13:34:38 +02:00
Dietmar Maurer
25500cfa26
src/cli/command.rs: add helpers/schema to format output
2019-07-16 11:46:58 +02:00
Dietmar Maurer
aad2ee4947
src/bin/pxar.rs: only print archive name in verbose mode
2019-07-16 08:38:02 +02:00
Dietmar Maurer
6b9a071028
src/bin/pxar.rs: add --verbose parameter to list (replace dump)
2019-07-16 08:31:40 +02:00
Dietmar Maurer
2665cef703
src/bin/proxmox-backup-client.rs: make repository parameter optional
...
And allow to read it from $ENV{PBS_REPOSITORY}
2019-07-16 07:30:04 +02:00
Christian Ebner
3f5192730e
src/pxar/format_definition.rs: fix typo
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-15 10:49:34 +02:00
Dietmar Maurer
9eae781ab8
src/bin/pxar.rs - extract: allow to read from stdin
2019-07-05 12:39:25 +02:00
Dietmar Maurer
c6c9e093d0
src/bin/pxar.rs - dump: allow to read from stdin
2019-07-05 12:32:15 +02:00
Dietmar Maurer
bf1252619a
src/bin/proxmox-backup-client.rs - restore: allow to pipe output to stdout
2019-07-05 12:14:50 +02:00
Dietmar Maurer
f8100e9623
src/bin/proxmox-backup-client.rs: remove download cli, do everything in restore
2019-07-05 11:36:45 +02:00
Dietmar Maurer
f4bf7dfcc7
src/client/remote_chunk_reader.rs: implement simple caching
2019-07-05 10:42:46 +02:00
Dietmar Maurer
afb4cd28be
src/backup/fixed_index.rs: implement BufferedFixedReader
2019-07-05 09:19:56 +02:00
Christian Ebner
be9b39e17a
src/pxar/sequential_decoder.rs: major refactoring of restore_sequential()
...
This splits the functionality of restore_sequential() into several smaller
functions in order to allow to reuse them when restoring by seeking based on
the goodbye table offsets.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:45:50 +02:00
Christian Ebner
48a4509c67
src/pxar/encoder.rs: fix bug with incorrect offset stored in goodbye table
...
Commit cd7dc87903
introduced the special treatment
for .pxarexclude files when stored in the archive.
The incorrect placement of a code snipplet from this path leads to an incorrect
offset and size stored in the goodbye table.
This fix places the start to the correct position, restoring the previously
correct behaviour.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:39:39 +02:00
Christian Ebner
7dcbe051e9
pxar: Refactor SequentialDecoder to store the callback function within the struct
...
Reduces the number of arguments for the function calls within the decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:38:22 +02:00
Dietmar Maurer
b46c3fad03
src/backup/fixed_index.rs: use correct size
...
We need to consider that the last chunk may have smaller size.
2019-07-04 15:13:22 +02:00
Dietmar Maurer
9cc88a7c00
src/client/http_client.rs - finish: hold reference to avoid early connection close
2019-07-04 15:11:23 +02:00
Dietmar Maurer
5e04ec707d
src/api2/backup/environment.rs: improve chunk alignment/size checks
2019-07-04 13:40:43 +02:00
Dietmar Maurer
a24e3993e0
src/backup/chunk_store.rs: coding style fixes
2019-07-04 11:39:10 +02:00
Dietmar Maurer
e4c2fbf170
src/backup/chunk_store.rs: additionally log chunk count
2019-07-04 11:27:11 +02:00
Dietmar Maurer
9850bcdf19
src/backup/chunk_store.rs: improve error reporting
2019-07-04 11:21:54 +02:00
Wolfgang Bumiller
a3f3e91da2
backup/chunk_store: rework chunk iterator
...
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-04 10:55:17 +02:00
Dietmar Maurer
99168f43e6
src/client/http_client.rs: s/set_recv_buf_size/set_recv_buffer_size/
2019-07-04 10:51:18 +02:00
Dietmar Maurer
c698636a5d
src/tools/fs.rs - read_dir: return nix::Result
2019-07-04 10:47:16 +02:00
Dietmar Maurer
a57360983b
src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item
2019-07-04 09:26:44 +02:00
Dietmar Maurer
0f0a35b390
src/backup/dynamic_index.rs: lock file inside new, code cleanup
2019-07-04 08:17:30 +02:00
Dietmar Maurer
a7c72ad9eb
src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store
2019-07-04 08:13:16 +02:00
Dietmar Maurer
a660978c9a
src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats
2019-07-04 07:57:43 +02:00
Dietmar Maurer
86eda3eb0d
src/bin/proxmox-backup-client.rs: implement restore using BackupReader
2019-07-03 15:45:12 +02:00
Dietmar Maurer
b2a6897982
src/client/remote_chunk_reader.rs: remove dead code
2019-07-03 14:56:24 +02:00
Dietmar Maurer
7f99bf691a
src/client/remote_chunk_reader.rs: implement remote chunk reader
2019-07-03 14:39:13 +02:00
Dietmar Maurer
4f6aaf542c
src/client/http_client.rs: avoid compiler warning
2019-07-03 14:36:02 +02:00
Dietmar Maurer
3467cd91e9
src/client/http_client.rs: return Arc<Self>
...
Avoid to call canceller for each clone on drop ...
2019-07-03 14:26:07 +02:00
Dietmar Maurer
1e0784841c
src/backup/read_chunk.rs: use &mut self
2019-07-03 14:25:18 +02:00
Christian Ebner
0d70535a36
src/pxar/encoder.rs: refactor simple match statements to map_err()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:36 +02:00
Christian Ebner
6e1c31163b
src/pxar/sequential_decoder.rs: refactoring of simple match statements to map_err()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:24 +02:00
Christian Ebner
9d227350a3
src/pxar/sequential_decoder.rs: Cleanup and fix incorrect error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:16:59 +02:00
Dietmar Maurer
80af046794
src/server/rest.rs: avoid unwrap
2019-07-03 12:00:43 +02:00
Dietmar Maurer
7fb4f5642a
src/server/rest.rs: log peer address, use hyper MakeService
2019-07-03 11:54:35 +02:00
Dietmar Maurer
6d1f61b208
use hyper/tokio-openssl instead of hyper/tokio-tls
...
This exposes the complete SSL setup. And download is much faster
now (600MB/s instead of 130MB/s)!
2019-07-02 13:36:28 +02:00
Dietmar Maurer
74d0a6bc23
src/bin/proxmox-backup-proxy.rs: set socket options for max. performance
2019-07-02 10:48:58 +02:00
Dietmar Maurer
93f8bb0445
src/bin/h2s-client.rs: avoid compiler warnings
2019-07-02 08:59:33 +02:00
Dietmar Maurer
877dd11133
src/backup/{fixed, dynamic}_index.rs: improve error messages
2019-07-02 08:56:56 +02:00
Dietmar Maurer
5c593a4d56
src/backup/read_chunk.rs: use Arc for DataStore
2019-07-02 08:49:16 +02:00
Dietmar Maurer
b850673634
src/backup/read_chunk.rs: move read chunk trait into extra file
...
And implement LocalChunkReader.
2019-07-02 08:22:29 +02:00
Dietmar Maurer
8fad30a4b1
use latest hyper master branch to include socket buffer size fix
2019-07-02 07:09:46 +02:00
Wolfgang Bumiller
f35197f449
replace tools::vec with proxmox::tools::vec
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 11:03:25 +02:00
Wolfgang Bumiller
d0162d53d3
tools: remove io module
...
now completely replaced by proxmox::tools::io.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:12 +02:00
Wolfgang Bumiller
9b2b627fe0
update remaining users of tools::io::ops
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:03 +02:00
Wolfgang Bumiller
9110a69bd8
tools: remove read/write
...
now completely replaced by
proxmox::tools::io::{ReadExt, WriteExt}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:42 +02:00
Wolfgang Bumiller
ca3c3ce997
daemon: remove last use of tools::read/write
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:13 +02:00
Wolfgang Bumiller
5485b579a1
backup: use new proxmox::tools::io traits
...
This replaces all `write_value` calls with `write_le_value`
calls!
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:37:02 +02:00
Dietmar Maurer
dc9775d1b0
src/bin/h2s-client.rs: increase tokio tcp stream read buffer size
...
This improves download speed by factor 10.
2019-06-29 18:09:50 +02:00
Dietmar Maurer
34f7131160
src/bin/h2s-client.rs: avoid hyper, use h2 directly
...
But performance is still bad.
2019-06-29 15:58:18 +02:00
Dietmar Maurer
5ed2277f0e
src/bin/h2s-server.rs: test to reproduce slow h2 download
...
Simplified code, use with h2s-client.rs.
2019-06-29 14:57:54 +02:00
Dietmar Maurer
17243003b3
src/bin/download-speed.rs: h2 download speedtest for our server
...
I get about 30MB/s (much too slow)
2019-06-29 13:43:10 +02:00
Dietmar Maurer
a2b29b68d4
src/client/http_client.rs: use maximal h2 window sizes
2019-06-29 11:05:36 +02:00
Dietmar Maurer
872debdefa
src/bin/h2client.rs: tune h2 client parameters to get reasonable speed
...
We now get about 2.5GB/s (not really good, but better)
2019-06-29 10:41:24 +02:00
Dietmar Maurer
38294bcc56
src/client.rs: make module put_to_stream public
2019-06-29 10:40:43 +02:00
Dietmar Maurer
fded1f3154
add code to test H2 speed
...
I currently get about 1MB/s, which is ways too slow (must be a bug)?
2019-06-29 10:08:12 +02:00
Dietmar Maurer
d48a9955a5
src/backup/dynamic_index.rs: introduce ReadChunk trait
2019-06-28 16:35:00 +02:00
Dietmar Maurer
09d7dc5024
src/api2/reader.rs: add download chunk api
2019-06-28 16:27:01 +02:00
Dietmar Maurer
fcf5dea56d
src/client/http_client.rs: increase h2 window size, set tcp nodelay option
...
To increase h2 download speed.
2019-06-28 16:00:58 +02:00
Dietmar Maurer
81a6ce6fde
src/backup/chunk_store.rs: new method chunk_path()
...
Returns the absolute path.
2019-06-28 15:48:09 +02:00
Dietmar Maurer
fcfb84fedf
file download: avoid unnecessary copy
2019-06-28 07:07:52 +02:00
Dietmar Maurer
984a7c3502
src/client/http_client.rs - h2 download: implement flow control
2019-06-28 07:02:43 +02:00
Dietmar Maurer
dd066d28e2
src/api2/reader.rs: implement backup reader protocol
2019-06-27 09:01:41 +02:00
Dietmar Maurer
42a87f7b96
src/server/h2service.rs: implement generic h2 service
2019-06-26 17:38:33 +02:00
Dietmar Maurer
fb0470837b
src/client/http_client.rs: split out code to start new h2 connections
2019-06-26 12:09:18 +02:00
Dietmar Maurer
b208da8393
src/backup/data_{chunk,blob}.rs: add verify_crc method
2019-06-26 09:54:25 +02:00
Dietmar Maurer
fef44d4f78
src/bin/proxmox-backup-client.rs: decode blobs on download
...
No more need for that clumsy blob cli (removed).
2019-06-26 09:18:59 +02:00
Dietmar Maurer
c2b945341c
src/client/http_client.rs - download: use generic Write type, return writer.
...
Make it possible to write int Vec<u8>.
2019-06-26 09:17:13 +02:00
Dietmar Maurer
bb8231409e
src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json
...
Use the KeyConfig serialization with kdf = None.
2019-06-26 07:32:34 +02:00
Dietmar Maurer
3031e44c58
src/bin/proxmox-backup-client.rs: remove stale download file, start blob cli
2019-06-25 13:00:37 +02:00
Dietmar Maurer
f14a8c9a85
src/api2/admin/datastore.rs: use correct path for download
2019-06-25 12:59:36 +02:00
Dietmar Maurer
81820b0d4d
src/client/http_client.rs - download: fix error handling
2019-06-25 12:43:55 +02:00
Dietmar Maurer
45db6f89e1
src/bin/proxmox-backup-client.rs: implement download command
2019-06-25 11:17:24 +02:00
Dietmar Maurer
9e47c0a56e
src/api2/admin/datastore.rs: implement generic download method
...
We simply allow to download raw binary data.
2019-06-25 10:16:59 +02:00
Dietmar Maurer
da7d67213e
src/api2/admin/datastore/pxar.rs: remove pxar upload api
...
Upload needs to be done using the backup protocol.
2019-06-25 09:45:35 +02:00
Dietmar Maurer
dd8e744f9a
src/api2/admin/datastore.rs: use a worker log log prune actions
2019-06-25 08:12:25 +02:00
Dietmar Maurer
6899dbfb47
src/bin/proxmox-backup-client.rs: strip .blob extension
2019-06-25 07:11:02 +02:00
Dietmar Maurer
c6d203bbd3
src/backup/backup_info.rs: also list .blob files
2019-06-25 07:07:45 +02:00
Dietmar Maurer
045e8a568d
src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key
2019-06-25 07:00:07 +02:00
Dietmar Maurer
79bc7345e8
src/cli/command.rs: fix ReST output format
2019-06-25 06:19:51 +02:00
Dietmar Maurer
9f46c7de4b
src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup
2019-06-24 13:56:37 +02:00
Christian Ebner
cd7dc87903
pxar: impl .pxarexclude parsing and exclude matching
...
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.
Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-24 10:26:20 +02:00
Dietmar Maurer
46bd880041
src/api2/backup/environment.rs: new helper add_blob()
2019-06-24 09:35:37 +02:00
Dietmar Maurer
cb08ac3efe
src/api2/backup.rs: replace upload_config with upload_blob
2019-06-23 11:44:30 +02:00
Dietmar Maurer
a38c5d4d12
src/backup/data_blob.rs: implement from_raw()
2019-06-23 09:35:44 +02:00
Dietmar Maurer
6f083b7a92
src/api2/backup/upload_chunk.rs: compute crc on upload
2019-06-23 09:04:42 +02:00
Dietmar Maurer
ba01828d38
try to use use proxmox::tools::io::ops::ReadExtOps
2019-06-22 16:29:10 +02:00
Dietmar Maurer
27042ce637
src/backup/file_formats.rs: improve docs
2019-06-22 15:45:36 +02:00
Dietmar Maurer
9f83e0f7ee
src/backup/crypt_config.rs: simplify decode interface
2019-06-22 13:24:29 +02:00
Dietmar Maurer
0066c6d972
src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead
2019-06-22 13:02:53 +02:00
Dietmar Maurer
ee8a7e8080
src/backup/crypt_config.rs: implement encrypt_to (output to writer)
2019-06-22 12:25:04 +02:00
Dietmar Maurer
991abfa8b4
src/backup/file_formats.rs: split out file format data
2019-06-22 09:12:25 +02:00
Dietmar Maurer
b7f4f27d6c
add crc field for binary blobs formats
2019-06-21 18:49:04 +02:00
Dietmar Maurer
a7f67a9a9c
depend on crc32fast
...
We can use this to compute fast checksums to test file integrity (not crypto safe).
2019-06-21 15:58:00 +02:00
Dietmar Maurer
3025b3a53c
src/backup/data_blob.rs: new file format for binary blobs
2019-06-21 11:32:07 +02:00
Dietmar Maurer
077a8cae63
src/backup/crypt_config.rs: make code more flexible
...
Allow to pass magic numbers as parameters.
2019-06-21 10:50:48 +02:00
Dietmar Maurer
c38266c18d
renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs
2019-06-21 09:51:18 +02:00
Dietmar Maurer
3ea8bfc9c0
src/bin/proxmox-backup-client.rs: add import-master-pubkey command
2019-06-21 07:34:17 +02:00
Dietmar Maurer
37c5a17505
src/bin/proxmox-backup-client.rs: helper to generate RSA master key
2019-06-21 07:12:19 +02:00
Dietmar Maurer
ab44acff57
src/backup/key_derivation.rs: add modified field to key file
2019-06-20 07:58:33 +02:00
Dietmar Maurer
6d0983dbe1
src/bin/proxmox-backup-client.rs: add keyfile parameter for backup
2019-06-19 17:16:41 +02:00
Dietmar Maurer
181f097af9
src/bin/proxmox-backup-client.rs - key API: pass kdf parameter
...
Allow to store keys without password.
2019-06-18 18:21:01 +02:00
Dietmar Maurer
9b06db4563
src/bin/proxmox-backup-client.rs: use file system path to reference keys
2019-06-18 17:38:50 +02:00
Dietmar Maurer
091540298e
src/bin/proxmox-backup-client.rs - create_key: always read password from tty
2019-06-18 17:27:52 +02:00
Dietmar Maurer
826f309bf5
src/backup/key_derivation.rs: move kdf code into separate file
2019-06-18 11:17:22 +02:00
Dietmar Maurer
11515438cc
Cargo.toml: use serde feature derive
2019-06-18 06:23:25 +02:00
Dietmar Maurer
ac71623483
src/bin/proxmox-backup-client.rs: implement "key change-passphrase"
2019-06-17 13:10:00 +02:00
Dietmar Maurer
f2401311b0
src/bin/proxmox-backup-client.rs: start key management
2019-06-17 10:33:24 +02:00
Dietmar Maurer
a7dd483097
src/backup/*_index.rs: used generated magic numbers
2019-06-14 14:58:37 +02:00
Dietmar Maurer
bffd40d6b7
src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools
2019-06-14 11:40:04 +02:00
Dietmar Maurer
00388226e6
src/api2/backup/environment.rs: log index checksums
2019-06-14 10:36:20 +02:00
Dietmar Maurer
9335d74eba
src/backup/fixed_index.rs: compute checksum over the index
2019-06-14 10:34:29 +02:00
Dietmar Maurer
16ff6b7cd8
src/backup/dynamic_index.rs: compute checksum over the index
2019-06-14 10:33:59 +02:00
Dietmar Maurer
36075475a6
src/api2/backup/environment.rs: log duplicate chunks
2019-06-14 07:12:30 +02:00
Dietmar Maurer
fa148dbd11
src/api2/backup/upload_chunk.rs: verify chunk data if possible
2019-06-14 06:23:46 +02:00
Dietmar Maurer
0f37577092
src/bin/cipherbench.rs: add test for zstd compression
2019-06-13 17:24:57 +02:00
Dietmar Maurer
9399c98f82
src/bin/cipherbench.rs: cipher speed test with large blocks
2019-06-13 17:16:43 +02:00
Dietmar Maurer
f640f23ac5
src/backup/data_chunk.rs: fix typo
2019-06-13 16:42:55 +02:00
Dietmar Maurer
dba72d5016
src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:56:50 +02:00
Dietmar Maurer
68ab376ae9
src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:47:54 +02:00
Dietmar Maurer
f98ac774ee
backup: Add support for client side encryption
...
first try ...
2019-06-13 11:47:23 +02:00
Dietmar Maurer
51929e4532
src/backup/data_chunk.rs - decode: make crypt_config optional
2019-06-13 11:44:38 +02:00
Dietmar Maurer
046bcb9773
src/backup/crypt_setup.rs: add comment for possible improvements
2019-06-13 07:35:59 +02:00
Dietmar Maurer
bd0e3c7cfa
src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs
2019-06-13 07:35:02 +02:00
Dietmar Maurer
6762db70d6
src/api2/types.rs: define CHUNK_DIGEST_SCHEMA
2019-06-13 07:30:42 +02:00
Dietmar Maurer
b595cb9d2c
src/backup/data_chunk.rs: Data Chunk builder
...
Make handling encoded chunks easier.
2019-06-12 13:37:01 +02:00
Dietmar Maurer
ef27200cec
src/backup/crypt_setup.rs: fix encoder for uncompressed chunks
2019-06-10 09:56:06 +02:00
Dietmar Maurer
74792b95b2
src/backup/crypt_setup.rs: allow compressed and uncompressed chunks
2019-06-10 08:27:35 +02:00
Dietmar Maurer
9e0187a203
src/backup/crypt_setup.rs: remove unused CryptData struct
2019-06-10 07:55:36 +02:00
Dietmar Maurer
bec8498ae3
src/backup/crypt_setup.rs: avoid useless allocations
2019-06-09 11:44:17 +02:00
Dietmar Maurer
c20b17b460
src/backup.rs: define magic numbers for chunk store file format
2019-06-08 10:34:20 +02:00
Dietmar Maurer
48b4b40b57
src/backup/crypt_setup.rs: crypto helpers
2019-06-08 09:56:51 +02:00
Wolfgang Bumiller
dd5495d6dc
tree-wide: use 'dyn' for all trait objects
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-07 13:13:48 +02:00
Christian Ebner
e993db91cd
pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder
...
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:20 +02:00
Christian Ebner
1f319e766e
src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:06 +02:00
Christian Ebner
9b77388f2a
src/pxar/encoder.rs: minor refactoring
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:55 +02:00
Christian Ebner
a4cc8eb7c5
src/pxar/encoder.rs: check if ioctl supported by filesystem
...
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:31 +02:00
Dietmar Maurer
4800df7ba6
src/api2/admin/datastore.rs: remove test-upload api
2019-06-07 07:52:11 +02:00
Dietmar Maurer
113898d7e9
src/api2/admin/datastore/upload.rs: remove unused upload api
2019-06-07 07:39:41 +02:00
Dietmar Maurer
4247fccb0f
src/client/http_client.rs: automatically close connection in finish
2019-06-05 11:41:50 +02:00
Wolfgang Bumiller
cb4426b348
make h2 client connection cancellable
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-05 09:45:59 +02:00
Dietmar Maurer
986bef16be
src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1
...
So that we can include it in static doc strings.
2019-06-05 08:41:20 +02:00
Dietmar Maurer
c9ec0956cf
src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1
2019-06-05 08:12:13 +02:00
Dietmar Maurer
44c54845b3
src/tools/xattr.rs: fix test when run as root
2019-06-05 07:57:42 +02:00
Dietmar Maurer
7773ccc11f
src/api2.rs: move backup api to /backup
2019-06-05 07:23:21 +02:00
Christian Ebner
97eeea3b4a
src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:29:02 +02:00
Dietmar Maurer
1ce2f7cfb9
src/pxar/encoder.rs: style fix
2019-06-05 06:27:39 +02:00
Christian Ebner
e7b60a16c8
pxar: impl storing/dumping/restoring of quota project ids
...
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:26:02 +02:00
Christian Ebner
7312ab9eef
src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID
...
Defines the types needed to store/restore quota project ids in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:23:06 +02:00
Christian Ebner
042babe4e7
src/tools/fs.rs: impl ioctl calls to get/set fsxattr
...
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:22:49 +02:00
Dietmar Maurer
22e5aa1d3e
src/bin/dump-backup-api.rs: helper to generate backup API docs
2019-06-04 13:12:42 +02:00
Dietmar Maurer
5e0f305142
src/api_schema/format.rs: depend on textwrap, new wrap_text helper
2019-06-04 12:42:02 +02:00
Dietmar Maurer
339ddfcbfa
src/cli/command.rs: move doc generator code to src/api_schema/format.rs
2019-06-04 12:32:22 +02:00
Dietmar Maurer
ef39bf95b3
src/client/http_client.rs - upload_config: improve error message
2019-06-03 10:53:09 +02:00
Dietmar Maurer
ec8a9bb953
src/bin/proxmox-backup-client.rs: implement upload_config
2019-06-03 10:39:44 +02:00
Dietmar Maurer
39d6846e1e
src/api2/admin/datastore/backup.rs: implement config file upload
2019-06-03 09:45:14 +02:00
Dietmar Maurer
1c0472e88f
avoid compiler warnings
2019-06-03 06:43:47 +02:00
Dietmar Maurer
36898ffce6
src/backup/chunk_stream.rs: add optional chunk_size parameter
2019-05-30 13:28:24 +02:00
Dietmar Maurer
49ef316bcd
src/bin/proxmox-backup-client.rs: backup_image() - pass correct chunk size
2019-05-30 12:52:12 +02:00
Dietmar Maurer
202cadc29e
src/bin/proxmox-backup-client.rs: remove unused code
2019-05-30 12:47:35 +02:00
Dietmar Maurer
6af905c18f
src/bin/proxmox-backup-client.rs: implement image/block-device upload
2019-05-30 12:46:01 +02:00
Dietmar Maurer
eb1804c567
src/bin/proxmox-backup-client.rs: use std lib instead of nix
2019-05-30 12:19:23 +02:00
Dietmar Maurer
6e907505aa
src/bin/proxmox-backup-client.rs: fix file type test
2019-05-30 12:08:34 +02:00
Dietmar Maurer
96482891ae
src/api2/admin/datastore/backup/environment.rs: log upload statistics
2019-05-30 09:21:19 +02:00
Dietmar Maurer
642322b433
src/api2/admin/datastore/backup.rs: use separate api entry points for chunk upload
...
So that we can provide better statistics (patches follows).
2019-05-30 09:21:19 +02:00
Wolfgang Bumiller
390e83c9b9
typo fix and doc improvement
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:57:28 +02:00
Wolfgang Bumiller
7fb49397d5
doc fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:56:14 +02:00
Wolfgang Bumiller
382609b04e
tools: add Cancellable, start a futures submodule
...
To make a future cancellable, use:
let (future, canceller) =
crate::tools::futures::cancellable(future);
Proceed with using `future` as usual, `canceller` is
clonable and can cancel the future via the `.cancel()`
method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Wolfgang Bumiller
0d32d71fb7
tools: add AsyncMutex::new_locked
...
Allows creating a pre-locked mutex, returning the mutex and
a guard.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
Dietmar Maurer
e6389f4e75
src/api2/admin/datastore/backup/environment.rs: do not allow empty backups (no files)
2019-05-29 10:38:57 +02:00
Dietmar Maurer
39e60bd6fa
src/api2/admin/datastore/backup.rs: improve debug logging
2019-05-29 10:17:38 +02:00
Dietmar Maurer
4ae6551c78
src/api_schema/router.rs: disable debug println
2019-05-29 09:42:16 +02:00
Dietmar Maurer
a42d1f55c3
src/api2/admin/datastore/backup.rs: add debug flag
2019-05-29 09:35:21 +02:00
Dietmar Maurer
976595e1a9
src/backup/dynamic_index.rs: split class DynamicIndexWriter
2019-05-29 08:49:57 +02:00
Dietmar Maurer
cb0708dd46
src/backup/fixed_index.rs: remove ChunkStat from struct
2019-05-29 07:08:34 +02:00
Dietmar Maurer
01af11f340
src/backup/fixed_index.rs: improve consistency checks
2019-05-29 07:02:45 +02:00
Christian Ebner
9b38443314
pxar: change logic and impl all command line flags for xattrs/fcaps/acls
...
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-29 06:51:20 +02:00
Dietmar Maurer
60e589a111
src/api2/admin/datastore/backup.rs: log sucessful finish call
2019-05-28 12:55:22 +02:00
Dietmar Maurer
c4ff3dcefd
src/bin/proxmox-backup-client.rs: use new BackupClient to upload data
2019-05-28 10:12:44 +02:00
Dietmar Maurer
10241c20ea
src/tools.rs: implement image_size helper
2019-05-28 09:35:08 +02:00
Dietmar Maurer
006f3ff407
src/api2/admin/datastore/backup/environment.rs: add more consistency checks
2019-05-28 09:21:31 +02:00
Dietmar Maurer
3dc5b2a203
src/client/http_client.rs: use start of chunk as offset
2019-05-28 09:21:31 +02:00
Dietmar Maurer
8de20e5c1f
src/client/http_client.rs: code cleanup
2019-05-28 08:53:27 +02:00
Dietmar Maurer
fc14b84937
src/backup/fixed_index.rs: improve bound checks inside add_digest
2019-05-28 08:47:03 +02:00
Dietmar Maurer
e1225de467
src/backup/fixed_index.rs: store index_length to avoid code duplication
2019-05-28 08:27:57 +02:00
Dietmar Maurer
e3d525feb7
src/api2/admin/datastore/backup.rs: fix typo
2019-05-28 07:13:19 +02:00
Christian Ebner
af309d4d5c
src/bin/pxar.rs: set archive permissions to 0o640
...
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:24:47 +02:00
Christian Ebner
ab87f167f1
src/pxar/encoder.rs: Refactor file stat
...
Introduce helper functions to check file stats
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:23:49 +02:00
Christian Ebner
578b601159
src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
...
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:22:16 +02:00
Dietmar Maurer
a42fa400ee
src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api
2019-05-28 06:18:55 +02:00
Dietmar Maurer
29ae5c86a2
src/backup/fixed_index.rs: make chunk_size public
2019-05-28 06:18:24 +02:00
Dietmar Maurer
7dd1bcac58
src/client/http_client.rs: correctly parse error response in download_chunk_list
2019-05-27 09:35:30 +02:00
Dietmar Maurer
a1e7cff3ea
src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk
2019-05-27 07:57:43 +02:00
Dietmar Maurer
771953f9eb
src/api2/admin/datastore/backup.rs: use smaller window sizes
2019-05-27 07:53:40 +02:00
Dietmar Maurer
174ad378d8
src/client/http_client.rs: fix request pipelining
2019-05-27 07:24:32 +02:00
Dietmar Maurer
624362226e
src/client/merge_known_chunks.rs: use MergedChunkInfo as input
2019-05-26 10:52:56 +02:00
Dietmar Maurer
05cba08c9c
src/client/http_client.rs: append chunks in correct order
...
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
2019-05-26 09:57:34 +02:00
Dietmar Maurer
417cb0731e
src/api2/admin/datastore/backup.rs: verify chunk offset
...
This clearly shows that the current approach does not work - seems we get
chunks out of order.
2019-05-26 08:49:29 +02:00
Dietmar Maurer
34114e2606
src/server/rest.rs: correctly verify json parameters
2019-05-24 09:10:59 +02:00
Dietmar Maurer
eb2bdd1bcc
src/client/http_client.rs: encode parameters inside body
...
To avoid problems with long uris.
2019-05-24 08:32:55 +02:00
Dietmar Maurer
9bb675ec23
src/client/merge_known_chunks.rs: reduce RAM usage
...
There is no need to store chunk data for known chunks.
2019-05-24 07:42:58 +02:00
Dietmar Maurer
ecb21b4794
src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough
2019-05-24 07:36:09 +02:00
Christian Ebner
2dbba78b98
src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:27:24 +02:00
Christian Ebner
6a19448004
src/pxar/encoder.rs: impl support to store ACLs in pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:21:33 +02:00
Christian Ebner
7cfaade7c8
src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:16:55 +02:00
Christian Ebner
b344461b33
pxar: pass feature_flags to encoder/decoder instead of individual flags
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner
687407741b
src/tools/xattr.rs: add test for is_valid_xattr_name()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner
357e4614e2
Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Wolfgang Bumiller
c05a8c8d18
doc-test fixup
...
cargo test by default compiles and runs all code snippets
found in the documentation...
oops...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 15:22:33 +02:00
Wolfgang Bumiller
cc84a830c5
RFC: schema: make enums static
...
I don't see a reason to allow these to be dynamically
modifiable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 12:38:51 +02:00
Dietmar Maurer
aa1b2e04fe
src/client/merge_known_chunks.rs: merge known chunks
...
To decrease the number of api calls required...
2019-05-23 12:31:09 +02:00
Dietmar Maurer
91320f0879
src/client/http_client.rs: use ChunkInfo streams
...
This will make out of order uploads possible...
2019-05-23 12:31:09 +02:00
Wolfgang Bumiller
8ea3b1d188
tree-wide: use the new vec/io tools modules
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Wolfgang Bumiller
9cdda3f7c7
tools: add helpful vector and read operations
...
After importing the I/O ops trait via:
use crate::tools::io::ops::*;
Instead of:
let mut buffer = vec![0u8; 65536];
file.read_exact(&mut buffer)?;
use:
let buffer = file.read_exact_allocated(65536)?;
After importing the vector helpers via:
use crate::tools::vec::{self, ops::*};
For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
let mut buffer = vec![0u8; len];
use:
let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Dietmar Maurer
8bea85b42e
src/api2/admin/datastore/backup.rs: verify file size and chunk count on close
2019-05-23 08:50:36 +02:00
Dietmar Maurer
1179e15842
src/server/rest.rs: improve error handling
2019-05-23 08:15:32 +02:00
Dietmar Maurer
907f5bb0d8
src/api2/admin/datastore/backup/service.rs: improve error logging
2019-05-23 08:05:39 +02:00
Dietmar Maurer
32e1bf4260
src/bin/upload-speed.rs: code cleanup
2019-05-23 07:20:05 +02:00
Christian Ebner
0d9bab05ab
src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:50 +02:00
Christian Ebner
4add706092
src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:32 +02:00
Christian Ebner
54b14ad741
src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:09 +02:00
Christian Ebner
1201360def
src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:39:16 +02:00
Christian Ebner
8bc8791338
src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:37:37 +02:00
Christian Ebner
a45a123950
src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
...
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:34:36 +02:00
Dietmar Maurer
d6f204eda6
src/client/http_client.rs: re-order code - public functions first
2019-05-23 06:28:10 +02:00
Dietmar Maurer
9af37c8f0e
src/client/http_client.rs: move low level H2 code into separate class
2019-05-22 17:28:25 +02:00
Dietmar Maurer
d2c48afc6e
src/client/http_client.rs: code cleanup
2019-05-22 13:24:33 +02:00
Dietmar Maurer
6ab34afa88
src/client/http_client.rs: cleanup h2 backup client
2019-05-22 13:05:51 +02:00
Dietmar Maurer
a55fcd740f
src/api2/admin/datastore/backup.rs: suse POST for finish() api
2019-05-22 13:04:58 +02:00
Dietmar Maurer
de6742c04f
src/bin/h2test.rs: removed
...
This test does no work anymore.
2019-05-22 11:14:56 +02:00
Dietmar Maurer
5a0b521809
src/api2/admin/datastore/backup.rs: remove test callbacks
2019-05-22 10:55:27 +02:00
Christian Ebner
de61bc9264
src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr
...
As git does not preserve xattrs they have to be set before they can be read.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-22 10:01:10 +02:00
Dietmar Maurer
553610b43e
src/client/http_client.rs: implement download_chunk_list
2019-05-22 09:46:02 +02:00
Dietmar Maurer
0cc0fffd1e
src/backup/chunk_stream.rs: use BytesMut in ChunkStream
2019-05-22 09:39:02 +02:00
Dietmar Maurer
c052be5c86
src/backup/chunk_stream.rs: use more generics
2019-05-22 09:18:05 +02:00
Dietmar Maurer
169c0e060f
src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut)
2019-05-22 09:05:35 +02:00
Dietmar Maurer
3be3f3dcc4
src/backup/chunk_stream.rs: fix behaviour for large input buffers
2019-05-22 08:05:27 +02:00
Dietmar Maurer
02fa54ff3c
src/backup/chunk_stream.rs: assemble data with correct order
2019-05-22 07:44:51 +02:00
Dietmar Maurer
d0afeaa831
src/api2/admin/datastore/backup.rs: improve error message
2019-05-22 07:43:54 +02:00
Dietmar Maurer
11867a2b54
src/tools.rs: fix hex_to_digest
2019-05-21 13:42:13 +02:00
Dietmar Maurer
7f3d2ffab9
src/backup/index.rs: rename ChunkListReader into DigestListEncoder
2019-05-21 12:28:44 +02:00
Dietmar Maurer
ee53955f52
src/backup/index.rs: implement DigestListDecoder
2019-05-21 12:21:22 +02:00
Dietmar Maurer
a95849321c
src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks
2019-05-21 09:54:11 +02:00
Dietmar Maurer
40f4e198a8
src/backup/dynamic_index.rs: add chunk_info method
2019-05-21 09:52:19 +02:00
Christian Ebner
596182056b
src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:15:30 +02:00
Christian Ebner
b303057a19
src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:06:26 +02:00
Christian Ebner
bee8d8ea78
src/tools/xattr.rs: add functions name_store and security_capability
...
These functions allow to check if the name for xattrs/fcaps are valid.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:05:31 +02:00
Dietmar Maurer
a09c0e38d8
src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks
2019-05-20 18:05:10 +02:00
Dietmar Maurer
97f03eff13
src/client/pxar_backup_stream.rs: fix docu
2019-05-20 17:29:00 +02:00
Dietmar Maurer
82ab72304e
src/client/http_client.rs: implement upload_stream
2019-05-20 14:19:24 +02:00
Dietmar Maurer
2698e8a514
src/client/pxar_backup_stream.rs: close pipe inside drop
2019-05-20 11:20:33 +02:00
Dietmar Maurer
f2269d8f99
src/tools.rs: implement hex_to_digest
2019-05-19 12:15:46 +02:00