Commit Graph

1606 Commits

Author SHA1 Message Date
Dietmar Maurer 9d456cf8ec src/backup/catalog_blob.rs: use write_all() instead of write()
To correctly handle interrupted system calls.
2019-09-05 11:40:48 +02:00
Dietmar Maurer 1a7a0e74c0 src/client/http_client.rs: use async for more functions 2019-09-04 13:48:16 +02:00
Dietmar Maurer 96f5e80abb src/client/http_client.rs: use async for login() 2019-09-04 10:01:46 +02:00
Dietmar Maurer 9d35dbbb8f src/client/http_client.rs: use async for credentials 2019-09-04 09:57:29 +02:00
Dietmar Maurer d2267b112d src/bin/proxmox-backup-client.rs: verify blob/catlog checksums 2019-09-04 08:47:14 +02:00
Christian Ebner 70b03a07c8 src/pxar/sequential_decoder.rs: make functions needed in non-sequential decoder accessible from within crate
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:15:05 +02:00
Christian Ebner f92e8266ae src/pxar/format_definitions.rs: move header type in PxarHeader and hash in PxarGoodbyeItem to top of struct
By this it is possible to read and check just the first u64 of the corresponding
structs in order to identify the items.
This is needed for the fuse implementation in order to get entries based on the
archive offset, used as inode.
Directories are referenced by the offset to the goodbye tail while other items
are referenced by the offset of the filename followed by the entry.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:14:30 +02:00
Christian Ebner 75933d1e83 src/pxar/fuse.rs: add function to map offset to inode and vice versa
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:13:59 +02:00
Christian Ebner c7fee396c6 src/pxar/decoder.rs: add getter functions to obtain start/end offsets for directories
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:10:56 +02:00
Christian Ebner 3626ac611f src/pxar/decoder.rs: rustfmt fixups
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:09:17 +02:00
Christian Ebner 3fa71727ee src/bin/proxmox-backup-client.rs: refactoring of match statement
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:08:00 +02:00
Dietmar Maurer a40220c056 src/bin/proxmox-backup-client.rs: code cleanup - factor out some functions 2019-09-03 13:31:13 +02:00
Dietmar Maurer 36493d4d7e src/backup.rs: define INDEX_BLOB_NAME here 2019-09-03 13:15:44 +02:00
Dietmar Maurer df65bd3da6 src/bin/proxmox-backup-client.rs - restore: verify checksums 2019-09-03 13:12:16 +02:00
Dietmar Maurer 0a51fe0011 src/backup/fixed_index.rs: new helper to compute checksum and file size 2019-09-03 13:11:45 +02:00
Dietmar Maurer 82c85a21a1 src/backup/dynamic_index.rs: new helper to compute checksum and file size 2019-09-03 13:11:09 +02:00
Dietmar Maurer a3e032b754 src/client/http_client.rs: fix dynamic index checksum 2019-09-03 13:10:18 +02:00
Wolfgang Bumiller c77563146b comment fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-03 11:17:33 +02:00
Wolfgang Bumiller daef93f481 tools: add tokio::main() replacement
to deal with block_on() not allowing blocking()

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-03 11:16:30 +02:00
Wolfgang Bumiller 66fbf5bad0 add WrappedReaderStream test
Mostly to just document the rt.block_on() issue with
blocking().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-03 10:05:45 +02:00
Wolfgang Bumiller 2b92971fba merge fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:22:36 +02:00
Wolfgang Bumiller 083ff3fd5d update to tokio 0.2.0-alpha.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 1434f4f8df drop hyper_openssl
We can use .await now, which means the whole connection
state machine doesn't need to be typed out as "types"
anymore, so, at least until hyper_openssl gets updated to
proper dependencies, let's drop it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 556eb70ea3 add generic either-or AsyncRead/Write type
The HttpsConnector will use this. Instead of implementing a
specialized MaybeTlsStream, this is simply a generic "either
this or that kind of Async Read/Write type".

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller b30415d244 src/server/state.rs: update to tokio alpha.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller aadcf7c621 src/server/rest.rs: use tokio::timer::delay
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 0cdb8d9c5b src/tools/wrapped_reader_stream.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller e668912a99 src/tools/futures.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 0f5856acca src/tools/daemon.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 56e351c911 src/tools/broadcast_future.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 627bb7d114 src/tools/async_mutex.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 75fef4b463 src/server/worker_task.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller aa4110cc64 src/server/state.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 91e4587343 src/server/rest.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller b9b7f7ec1f src/server/h2service.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 160fc8147f src/server/command_socket.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 3c0facc787 src/client/remote_chunk_reader.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 369a87e3a2 src/client/pxar_backup_stream.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 2107bb40c1 src/client/pipe_to_stream.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 5b3911995b src/client/merge_known_chunks.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller a6782ca10b src/client/http_client.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller 8630557918 src/bin/upload-speed.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller cab6816969 src/bin/test_chunk_speed2.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller fda5797b8a src/bin/proxmox-backup-proxy.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller e9722f8bde src/bin/proxmox-backup-client.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller e76ac3a499 src/bin/proxmox-backup-api.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller e235c8f719 src/bin/h2s-server.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 74be6dc9b7 src/bin/h2server.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 55d8a631fc src/bin/h2s-client.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 15d0e4a3bd src/bin/h2client.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller b9203d87f4 src/bin/download-speed.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 745e652a7f src/backup/index.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 32bef1e2d1 src/backup/chunk_stream.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller f2d9b73c91 src/api_schema/router.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller ffb6434485 src/api2/reader.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 7622005574 src/api2/backup/upload_chunk.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 59b2baa0f6 src/api2/backup.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 6be147b78c src/api2/admin/datastore.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 5c1130df9f implement Sync for DynamicIndexReader and FixedIndexReader
hyper's wrap_stream now needs this

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Wolfgang Bumiller 236761a3e6 drop src/storage/futures.rs
it's unused and uses a tokio::Task I don't want to lookup
the std-future conversion for...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
Dietmar Maurer 02fcf372e4 rc/bin/proxmox-backup-client.rs - restore: always download index.json.blob 2019-09-02 14:14:32 +02:00
Dietmar Maurer 0d9862803b src/bin/proxmox-backup-client.rs - reatore: avoid loading (large) blobs into memory 2019-09-02 14:13:31 +02:00
Christian Ebner 66c8eb9383 src/client/http_client.rs: Refactor handling Option and Result types
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-29 09:15:16 +02:00
Wolfgang Bumiller 40f8680e37 tools: remove SignalFd
We don't use it anymore and it'll just be more work when
switching to async-await.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 13:44:53 +02:00
Wolfgang Bumiller d21f8a5b5b backup/dynamic_index: explicitly zero-initialize the header
The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
Wolfgang Bumiller 990b930f22 backup/dynamic_index: use static assertion, fix size
The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
Christian Ebner bcb664cb69 pxar/fuse: add pxar cli flag to set single- or multi-threaded session loop
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:43:14 +02:00
Christian Ebner de2791868a src/pxar/sequentail_decoder.rs: followup: cleanup S_IFMT masking
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:42:22 +02:00
Christian Ebner 9cb9e1b7db src/pxar/sequential_decoder.rs: style fixups
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:42:19 +02:00
Christian Ebner dac88033ba src/pxar/dir_stack.rs: style fixups
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:39:14 +02:00
Christian Ebner b69169e7be src/pxar/fuse.rs: style fixups
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:58 +02:00
Christian Ebner d792dc3cd2 src/pxar/match_pattern.rs: style fixups
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:56 +02:00
Christian Ebner 4ea22b68d9 src/pxar/format_definition.rs: style fixup
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:53 +02:00
Wolfgang Bumiller 7d83440c60 remove proxmox-protocol subcrate
AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)

Moved the chunker back into src/backup/chunker.rs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 14:08:25 +02:00
Wolfgang Bumiller 7a57cb77e1 more formatting & use statement fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:50:27 +02:00
Christian Ebner 35dfd3ceb7 src/pxar/encoder.rs: style fixups by rustfmt
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:29:11 +02:00
Christian Ebner 1af30bc2fe (pxar: replace deprecated std::mem::uninitialized()
... and use std::mem::MaybeUninit or proxmox::tools::vec::uninitialized() instead.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:27:25 +02:00
Wolfgang Bumiller 5e58e1bb7d {dynamic,fixed}_index: replace mem::uninitialized
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:35:36 +02:00
Wolfgang Bumiller f93b55b057 src/backup/index: style fixup & unsafe copy removal
We can use the safe .copy_from_slice alternative in this
case.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:31:45 +02:00
Wolfgang Bumiller fa17b1ce2a src/backup/index: replace mem::uninitialized
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:27:35 +02:00
Wolfgang Bumiller f3a8d1d7e0 api2/node/time: replace mem::uninitialized
and fixup use statements

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:14:41 +02:00
Wolfgang Bumiller 48b85e8e3b tools: tty: replace mem::uninitialized and style fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:14:19 +02:00
Wolfgang Bumiller b528fddfc6 tools: timer: style fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:07:56 +02:00
Wolfgang Bumiller b7f44ace92 timer: replace mem::uninitialized
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:07:10 +02:00
Wolfgang Bumiller 6100071f4e tools: style & use statement fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 10:57:56 +02:00
Wolfgang Bumiller 5c20e2da6b src/config: more style fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:14:26 +02:00
Wolfgang Bumiller 91640ab567 src/config: style fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:11:07 +02:00
Wolfgang Bumiller 928650c4eb src/config: use statement cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:10:50 +02:00
Wolfgang Bumiller 2ec979e4ed /var/run -> /run
/var/run is considered deprecated and for instance in
systemd unit files lintian complains...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 12:28:24 +02:00
Wolfgang Bumiller 35238e234d switch from create_dir_chown to create_path
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 12:26:10 +02:00
Christian Ebner e5471b4836 pxar/decoder: avoid failing on hardlinks and return corresponding target file entry
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
[Fixup: remove unnecessary 'mut']
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:32:52 +02:00
Christian Ebner 58262f40f7 src/pxar/decoder.rs: get correct stats for root directory
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:31:41 +02:00
Christian Ebner 7d26720e79 src/pxar/decoder.rs: move GOODBYE_ITEM_SIZE from function to module scope.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:31:39 +02:00
Wolfgang Bumiller 367f002ee2 use statement cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-16 12:28:40 +02:00
Dietmar Maurer 9d135fe617 src/backup/catalog_blob.rs: moved catalog impl. from pxar
And avoid loading catalog into memory.
2019-08-16 12:27:17 +02:00
Wolfgang Bumiller 9025312aa6 avoid lifetimes in blob reader/writer 2019-08-16 10:07:24 +02:00
Dietmar Maurer 71d08e00b7 src/backup/data_blob_reader.rs: fix compiler warning 2019-08-16 08:10:39 +02:00
Christian Ebner ac0b435344 src/pxar/fuse.rs: fix compiler warnings
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 15:14:57 +02:00
Dietmar Maurer a84ef4c205 src/bin/proxmox-backup-client.rs: avoid loading catalog into memory
We can use the new DataBlobReader instead.
2019-08-14 15:07:28 +02:00
Dietmar Maurer b791804f4b src/backup/data_blob_reader.rs: impl for Read instead of BufRead 2019-08-14 14:56:01 +02:00
Dietmar Maurer 39a4df61d6 improve docs 2019-08-14 14:08:27 +02:00
Dietmar Maurer 018d11bb84 src/backup/data_blob.rs: move parts into single files 2019-08-14 13:24:41 +02:00
Dietmar Maurer e27c5a559f src/backup/data_blob.rs: avoid excessive stack usage 2019-08-14 12:44:08 +02:00
Dietmar Maurer c638542b4b src/backup/file_formats.rs: moved header_size 2019-08-14 12:35:53 +02:00
Christian Ebner 41d08aa6de src/pxar/fuse.rs: enable libfuse debug output in verbose mode
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:59:15 +02:00
Christian Ebner 742e64ea0b src/pxar/fuse.rs: cleanup callback interface and store decoder within session context
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:58:44 +02:00
Christian Ebner 7750b7f2b7 pxar: decoder: take ownership of underlying reader
By taking ownership it is easier to move the decoder into another struct,
e.g. into a session context in fuse.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:55:08 +02:00
Christian Ebner f50b4fd6a0 src/pxar/decoder.rs: cleanup s/CaDirectoryEntry/DirectoryEntry
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:54:08 +02:00
Dietmar Maurer 6082216343 src/backup/data_blob.rs - CryptReader/CryptWriter: fix buffer size handling 2019-08-13 15:23:15 +02:00
Dietmar Maurer 548c9489d7 src/backup/data_blob.rs - DataBlobReader: impl compressed, encrypted blobs 2019-08-13 13:17:08 +02:00
Dietmar Maurer 2aa0bfff59 src/backup/data_blob.rs - DataBlobReader: implement reader for encrtypted blobs 2019-08-13 13:17:08 +02:00
Dietmar Maurer e9a385a78e src/backup/data_blob.rs - DataBlobReader: impl compressed, signed blobs 2019-08-12 17:47:25 +02:00
Dietmar Maurer 4bfa147eaf src/backup/data_blob.rs - DataBlobReader: impl. reader for signed blobs 2019-08-12 17:41:25 +02:00
Dietmar Maurer 09785b2795 src/backup/data_blob.rs - DataBlobReader: start serious impl. 2019-08-12 17:20:41 +02:00
Dietmar Maurer a32bd8a516 src/backup/data_blob.rs - DataBlobWriter: fix decryption 2019-08-12 11:57:29 +02:00
Dietmar Maurer 5d15cb49b4 src/backup/data_blob.rs - compute_crc: start after blob header 2019-08-12 11:32:26 +02:00
Dietmar Maurer 1cbdfd19a2 src/backup/file_formats.rs; use const instzead of static 2019-08-12 11:20:21 +02:00
Dietmar Maurer 18be4ec24a src/backup/data_blob.rs - DataBlobWriter: fix magic number for uncompressed blobs 2019-08-12 10:46:05 +02:00
Dietmar Maurer 5622a3fcdb src/backup/data_blob.rs - DataBlobWriter: impl. compressed encrypted blobs 2019-08-12 10:16:42 +02:00
Dietmar Maurer f4942e9ffd src/backup/data_blob.rs - DataBlobWriter: impl. encrypted blobs 2019-08-12 10:08:00 +02:00
Dietmar Maurer c57ec43a53 src/backup/crypt_config.rs: new helper data_crypter to get openssl Crypter 2019-08-12 10:06:51 +02:00
Dietmar Maurer 7776becf5d src/backup/data_blob.rs - DataBlobWriter: impl. compressed signed blobs 2019-08-12 09:01:08 +02:00
Dietmar Maurer 706638f803 src/backup/data_blob.rs: impl. ChecksumWriter
To correctly compute crc,hmac of final data.
2019-08-12 08:53:53 +02:00
Dietmar Maurer f796351c54 src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs 2019-08-12 07:34:41 +02:00
Dietmar Maurer cb0eea29d9 src/backup/crypt_config.rs: new method to get hmac signer 2019-08-12 07:33:15 +02:00
Dietmar Maurer a762ce54fd src/backup/data_blob.rs: make DataBlobWriter more generic
Allow to write compressed and uncompressed blobs.
2019-08-11 12:14:04 +02:00
Dietmar Maurer 1f26fdef19 src/backup/data_blob.rs: impl. simple reader/writer
To avoid loading blob into memory.
2019-08-11 11:32:36 +02:00
Dietmar Maurer e240d8be0b src/bin/proxmox-backup-client.rs: implement login/logout 2019-08-10 09:12:17 +02:00
Christian Ebner f71e8cc96f pxar: add command 'mount' to cli of pxar
Allows to mount an archive to a specified mountpoint via the cli.
Once the archive is mounted, the process is send to the background.
By passing the --verbose flag, the process is kept in foreground and
debug output is provided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-09 16:46:16 +02:00
Dietmar Maurer 02ba8a5d11 pxar: add fuse module and expose its pub functionality. 2019-08-09 16:45:13 +02:00
Christian Ebner c50f87442c pxar: add basic code for FUSE implementation.
This adds the basic code in order to create a fuse session and mount an archive.
It adds libfuse3-3 as runtime dependency and libfuse3-dev as build dependency.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-09 16:42:31 +02:00
Dietmar Maurer 781ac11c6a src/backup/data_blob.rs: define const MAX_BLOB_SIZE 2019-08-09 11:49:06 +02:00
Dietmar Maurer 9049a8cfe6 src/bin/proxmox-backup-client.rs: implement catalog command 2019-08-09 11:36:33 +02:00
Dietmar Maurer 7926a3a1cf src/bin/proxmox-backup-client.rs: add archive filename to catalog 2019-08-09 11:05:56 +02:00
Dietmar Maurer 863be2e6e2 src/backup/data_blob.rs: allow largert blob sizes (128MB)
Need this for catalog files.
2019-08-09 10:22:56 +02:00
Dietmar Maurer 3fb53e07b4 src/bin/proxmox-backup-client.rs: avoid compiler warning 2019-08-09 10:22:32 +02:00
Dietmar Maurer 543a260ff9 src/bin/proxmox-backup-client.rs: new completion helper for snapshots 2019-08-09 10:08:45 +02:00
Dietmar Maurer 2761d6a4f3 src/pxar/encoder.rs: use BackupCatalogWriter 2019-08-09 09:46:49 +02:00
Dietmar Maurer aea0815d32 src/pxar/catalog.rs: new catalog helper 2019-08-09 09:41:43 +02:00
Dietmar Maurer 977eeb24f6 src/bin/proxmox-backup-client.rs: avoid doubled file extensions (.blob.blob) in index.json 2019-08-07 10:23:08 +02:00
Dietmar Maurer 4af0ee055a api: always use complete file names (including add exctensions) 2019-08-07 10:10:14 +02:00
Dietmar Maurer d7c243977d src/api2/admin/datastore.rs - list_snapshot_files: list all files
Add files that are not mentioned in index.json.
2019-08-07 09:16:14 +02:00
Dietmar Maurer 34066e2eeb src/api2/backup/environment.rs - add_blob: always verify blob crc 2019-08-07 09:15:13 +02:00
Dietmar Maurer eecb23560b src/backup/data_blob.rs - encode: always compute crc 2019-08-07 08:30:27 +02:00
Dietmar Maurer 38a6cddad7 src/backup/backup_info.rs: add BackupInfo::new constructor 2019-08-07 08:27:52 +02:00
Dietmar Maurer c4f025ebbf src/bin/proxmox-backup-client.rs: fix remote filename completion 2019-08-07 06:48:18 +02:00
Dietmar Maurer dbed4c8cd7 src/client/http_client.rs: compute checksums for chunk streams 2019-08-06 12:23:06 +02:00
Dietmar Maurer bd15e96dd9 src/client/http_client.rs: minor code cleanup 2019-08-06 11:46:15 +02:00
Dietmar Maurer f889b15867 src/backup/data_blob.rs: always compute crc 2019-08-06 11:42:14 +02:00
Dietmar Maurer c807d231eb src/client/http_client.rs: compute file checksums 2019-08-06 11:26:06 +02:00
Dietmar Maurer a17a0e7a9f src/api2/admin/datastore.rs: extract backup size from index.json 2019-08-06 10:56:21 +02:00
Christian Ebner eecb182845 pxar: add initial docs for MatchPattern
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-06 09:03:02 +02:00
Christian Ebner 43e892d293 pxar: add error handling for MatchPattern::matches_filename()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-06 09:02:44 +02:00
Dietmar Maurer 8c70e3eb18 src/api2/admin/datastore.rs - files: return data from index.json 2019-08-05 13:22:19 +02:00
Dietmar Maurer f69adc81a6 use new proxmox::tools::nodename 2019-08-03 17:06:23 +02:00
Dietmar Maurer 7f66c29e5c remove src/tools/common_regex.rs, use proxmox::tools::common_regex instead 2019-08-03 16:44:31 +02:00
Dietmar Maurer fd329bdcd7 remove src/tools/procfs.rs, use proxmox::sys::linux::procfs instead 2019-08-03 16:26:44 +02:00
Dietmar Maurer 779e4c90b1 src/tools/vec.rs: remove unused file
We already moved that to proxmox::tools
2019-08-03 16:11:47 +02:00
Dietmar Maurer e18a6c9ee5 update to nix 0.14, use code from proxmox:tools 2019-08-03 13:05:38 +02:00
Dietmar Maurer b86f263ced use new proxmox::sys::linux::magic 2019-08-03 09:28:57 +02:00
Christian Ebner 4ea0f7d95b pxar: remove flag to change digest algorithm
The flag CA_FORMAT_SHA512_256 is used to switch between sha512 and sha256 to
calculate digest in casync.

As we use sha256, we can get rid of this flag for now.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:57:53 +02:00
Christian Ebner 4d142ea79e pxar: cleanup: refactor and rename exclude pattern
The original name PxarExcludePattern makes no sense anymore as the patterns are
also used to match filenames during restore of the archive.

Therefore, exclude_pattern.rs is moved to match_pattern.rs and PxarExcludePattern
rename to MatchPattern.
Further, since it makes more sense the MatchTypes are now declared as None,
Positive, Negative, PartialPositive or PartialNegative, as this makes more sense
and seems more readable.
Positive matches are those without '!' prefix, Negatives with '!' prefix.

This makes also the filename matching in the encoder/decoder more intuitive and
the logic was adapted accordingly.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:52:32 +02:00
Christian Ebner fe076c8259 pxar: cleanup: s/PxarDirBuf/PxarDirStack/g and move dir_buffer.rs to dir_stack.rs
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:52:10 +02:00
Christian Ebner ead7546a96 pxar: change uid and gid to u32 instead of u64
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:51 +02:00
Christian Ebner 5e50c606b0 pxar: cleanup: s/CA_FORMAT/PXAR/g and s/CaFormat/Pxar/g
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:33 +02:00
Christian Ebner 47651f9530 pxar: cleanup: move feature flags to src/pxar/flags.rs and omit CA_FORMAT prefix on all of them
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:06 +02:00
Christian Ebner 91451c4355 pxar: cleanup: move filesystem related magic numbers to tools/fs/magic.rs
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:48:12 +02:00
Dietmar Maurer b335f5b713 src/bin/proxmox-backup-client.rs: sign index.json 2019-08-02 10:00:15 +02:00
Dietmar Maurer 69ecd8d5b0 src/backup/data_blob.rs: implement signed blobs 2019-08-02 09:56:01 +02:00
Dietmar Maurer 93205f942a src/backup/crypt_config.rs: new compute_auth_tag helper 2019-08-02 08:55:37 +02:00
Dietmar Maurer c68d2170d5 src/backup/crypt_config.rs: fix typo 2019-08-02 08:29:40 +02:00
Christian Ebner 81a9905e0a pxar: implement feature flag support for device nodes, fifos and sockets
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:55:17 +02:00
Christian Ebner 8abc95a145 pxar: remove some unused feature flags from format definition
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:54:46 +02:00
Christian Ebner 3a905cac2b src/pxar/sequential_decoder.rs: remove of obsolete function dir_mkdirat
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:52:49 +02:00
Christian Ebner beffac999f src/pxar/sequentail_decoder.rs: fix issue when restoring with glob pattern.
Partial extraction of an archive with a glob pattern, e.g. '**/*.conf' lead to
the unexpected behaviour of restoring all partially matched directories (in this
example all of them).

This patch fixes this unexpected behaviour by only restoring those directories
were the directory or one of its sub-items fully matched the pattern and should
therefore be restored.

To achive this behavoiur, directory metadata is pushed onto a stack and restored
on demand.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:49:47 +02:00
Christian Ebner 51ac99c314 src/pxar/exclude_pattern.rs: add Clone and Copy trait to MatchType
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:49:47 +02:00
Christian Ebner 6a584cfd76 pxar: add PxarDir and PxarDirBuf to buffer directory metadata
In order to restore only directories when some of their content fully matched
a match pattern on partial restores, these directories and their metadata are
pushed onto this buffer and only restored successivley on demand.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:45:12 +02:00
Dietmar Maurer 27e6e180b6 src/pxar/format_definition.rs - PxarAttributes: derive Default
And remove unnecessary impl new().
2019-08-02 07:16:10 +02:00
Christian Ebner a92e6c9011 src/pxar/sequential_decoder.rs: split read and restore of attributes into separate functions
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:11:38 +02:00
Christian Ebner e60cfbcdd0 src/pxar/format_definition.rs: add struct PxarAttributes
This structure contains all the attributes allowing to easily store those within
a e.g. dir buffer.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:10:22 +02:00
Christian Ebner 2be3eff5c3 pxar: avoid taking ownership of xattrs, fcaps and quota_projid on corresponding restore functions.
By borrowing these objects we preserve the functionality but make sure
that ownership doesn't change, avoiding problems when contained within other
structs such as e.g. a buffer storing these attributes.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 06:44:00 +02:00
Dietmar Maurer 2c3891d1c3 src/bin/proxmox-backup-client.rs: upload backup index.json
The plan is to use this file to verify the backup content.
2019-08-01 12:39:02 +02:00
Dietmar Maurer 953d5e1531 src/tools.rs - file_set_contents_full: only call fchmod when we pass permissions
So that we can use it to write into /etc/pve/ (which does not support chmod).
2019-07-31 12:44:27 +02:00
Dietmar Maurer cec17a3ec7 src/bin/proxmox-backup-client.rs - files: allow --repository parameter 2019-07-31 11:45:55 +02:00
Dietmar Maurer 52c171e49d src/bin/proxmox-backup-client.rs: implement files command 2019-07-31 11:23:53 +02:00
Dietmar Maurer 43a406fd3f src/bin/proxmox-backup-client.rs: supress useless output from prune command 2019-07-31 10:15:16 +02:00
Christian Ebner 9731c8b862 pxar: add support for storing/restoring the quota project id on ZFS
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-30 09:40:48 +02:00
Christian Ebner 6a87910949 pxar: implement allow_existing_dirs for pxar decoder
By default, restoring an archive will fail if files with the same filename
already exist in the target directory.

By setting the allow_existing_dirs flag, the restore will not fail if an
existing directory is encountered.
The metadata (permissions, acls, ...) of the existing directory will be set
to the ones from the archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-30 09:40:30 +02:00
Dietmar Maurer 46d5aa0a09 src/bin/proxmox-backup-client.rs: add restore flag --allow-existing-dirs
Not jet implemented ...
2019-07-29 12:49:15 +02:00
Dietmar Maurer ea7a7ef299 src/bin/proxmox-backup-client.rs: fixup - correctly pass prune parameters 2019-07-27 09:24:23 +02:00
Dietmar Maurer 9fdc3ef46f src/api2/admin/datastore.rs: pass backup type/id (group) to prune
IMHO, prune over all backup groups is a bit dangerous, considering
that more than one user might use a datastore.
2019-07-27 08:49:14 +02:00
Dietmar Maurer 7a6cfbd98d src/bin/proxmox-backup-client.rs: correctly compute duration
And format time with SecondsFormat::Secs
2019-07-26 11:13:49 +02:00
Dietmar Maurer bbf9e7e951 src/api2/backup.rs: cleanup schema definitions 2019-07-26 09:07:29 +02:00
Dietmar Maurer ca5d0b61ca src/api2/backup.rs: new required backup-time parameter
The client should pass the time as parameter.
2019-07-25 13:44:01 +02:00
Dietmar Maurer e128d4e84f src/api2/admin/datastore.rs: backup logs may not be written twice 2019-07-25 12:29:04 +02:00
Dietmar Maurer ef3254c086 src/api2/admin/datastore.rs: use correct .blob extension for log 2019-07-25 12:22:50 +02:00
Dietmar Maurer ec34f7eba0 src/bin/proxmox-backup-client.rs: implement upload-log 2019-07-25 12:18:15 +02:00
Dietmar Maurer 04512d3068 src/client/http_client.rs: allow upload with parameters 2019-07-25 12:17:35 +02:00
Dietmar Maurer 07ee223590 src/api2/admin/datastore.rs: add api to upload backup client log file 2019-07-25 12:15:30 +02:00
Dietmar Maurer 79679c2d6e src/bin/proxmox-backup-client.rs: allow to uploag .log files (as binary blobs) 2019-07-24 13:07:02 +02:00
Dietmar Maurer 5b72c9b4f7 src/pxar/encoder.rs: add new skip_lost_and_found parameter 2019-07-24 12:21:25 +02:00
Dietmar Maurer 6c3c9bceb5 pxar_backup_stream.rs: limit lock scope to avoid blocking forever 2019-07-24 11:30:43 +02:00
Dietmar Maurer 684233aa3b remove debug output 2019-07-24 09:33:52 +02:00
Dietmar Maurer 5be106eeae src/client/pxar_backup_stream.rs: correctly pass errors to stream 2019-07-24 09:24:35 +02:00
Dietmar Maurer 2eeaacb974 src/pxar/encoder.rs: allow to pass list of devices
For better mount point include control...
2019-07-24 08:11:59 +02:00
Dietmar Maurer fa5d6977dd Revert previous, commit, use UTC RFC3339 without timezone (Z)
We now have human readable data/time, and names are still sortable.
2019-07-22 11:49:30 +02:00
Dietmar Maurer 10c2a21cfe src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
2019-07-22 07:57:03 +02:00
Christian Ebner 1fdef50f52 src/pxar/sequential_decoder.rs: Call callback only on restored paths.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-19 11:58:51 +02:00
Dietmar Maurer 40a13369a3 src/bin/pxar.rs - extract: print archive name with --verbose flag 2019-07-19 07:01:57 +02:00
Dietmar Maurer bbd4365c61 src/bin/pxar.rs: remove wrong debug message 2019-07-19 07:00:17 +02:00
Christian Ebner a0ec687cd7 src/bin/pxar.rs: allow to pass paths and match patterns as args to pxar extract
To improve usability it is now possible to directly pass paths or match patterns
as arguments to pxar extract to partially restore an archive.
The patterns provided via CLI are appended to the ones read from file by the
--files-from option in order to have priority over those.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-19 06:50:39 +02:00
Dietmar Maurer 9aa3f68278 src/bin/proxmox-backup-client.rs: code cleanup 2019-07-18 13:16:32 +02:00
Dietmar Maurer f6ede796ca src/bin/proxmox-backup-client.rs - status: only print result[data] 2019-07-18 09:52:11 +02:00
Dietmar Maurer 15c847f136 src/api2/admin/datastore.rs: allow to list all snapshots 2019-07-18 09:11:43 +02:00
Dietmar Maurer 0f73ee73bb src/cli/command.rs: set exit status on error 2019-07-17 13:40:10 +02:00
Christian Ebner 129dda4729 src/bin/pxar.rs: add --files-from parameter to pxar extract
Allows to pass a list of match pattern to the pxar extract call by reading them
from file.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:20:29 +02:00
Christian Ebner fa7e957ca3 pxar: Adapt SequentialDecoder::restore() calls to pass empty match pattern for full restore
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:19:31 +02:00
Christian Ebner d32c2e4d27 src/pxar/sequential_decoder.rs: adapt code for partial restore by match pattern
Allows to partially restore an archive by passing match patterns to the restore
function.
The whole restore is performed in sequential, therefore the whole archive has to
be read.
By wrapping the RawFd into an Option it can be controlled if the corresponding
part is restored (in case of Some(fd)) or if the Reader reads over it
without restore (in case of None).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:11:37 +02:00
Christian Ebner 4902291673 src/pxar/sequential_decoder.rs: implement match_filename to match include pattern for partial restores
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:07:54 +02:00
Christian Ebner 894cd49ac4 src/pxar/sequential_decoder.rs: Also check for nul bytes when reading the filename
Check if the filename does not contain invalid nul byes when reading it from the
archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:04:41 +02:00
Christian Ebner 0b39674895 src/pxar/sequential_decoder.rs: remove old comments and refactoring
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:46:02 +02:00
Christian Ebner 9f8fcdd050 src/pxar/encoder.rs: refactoring of match_exclude_pattern()
match_exclude_pattern() does not need a '&mut self' reference to the encoder,
move it therefore out of the impl.
Further, this patch contains some naming and formatting cosmetics.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:45:19 +02:00
Christian Ebner 3ff4ef28e6 src/pxar/encoder.rs: remove unneeded dir_count
dir_count was used to track the number of directory entries to store in the
archive and bail if the maximum is exceeded.
As the number of entries is equally obtained from the list of the filenames to
include, use that one instead.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:44:59 +02:00
Christian Ebner a771f90745 src/pxar/exclude_pattern.rs: do not return partial matches for files
Partial matches make only sense for directories, files are always leafs of the
tree. Take this into account in order to avoid restoring of files which only
matched the front of a match pattern.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:44:06 +02:00
Christian Ebner bdf0d82ced src/bin/pxar.rs: Make pxar extract target optional
In order to improve usablity, the target on archive extraction will be the
current working directory by default.
A different target can be provided via the optional --target <PATH> parameter.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:40:02 +02:00
Dietmar Maurer 34a816cc7b src/bin/proxmox-backup-client.rs: add output-fromat options, implement status api 2019-07-16 13:35:25 +02:00
Dietmar Maurer 0eecf38fbf src/api2/admin/datastore.rs: add status api call 2019-07-16 13:34:38 +02:00
Dietmar Maurer 25500cfa26 src/cli/command.rs: add helpers/schema to format output 2019-07-16 11:46:58 +02:00
Dietmar Maurer aad2ee4947 src/bin/pxar.rs: only print archive name in verbose mode 2019-07-16 08:38:02 +02:00
Dietmar Maurer 6b9a071028 src/bin/pxar.rs: add --verbose parameter to list (replace dump) 2019-07-16 08:31:40 +02:00
Dietmar Maurer 2665cef703 src/bin/proxmox-backup-client.rs: make repository parameter optional
And allow to read it from $ENV{PBS_REPOSITORY}
2019-07-16 07:30:04 +02:00
Christian Ebner 3f5192730e src/pxar/format_definition.rs: fix typo
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-15 10:49:34 +02:00
Dietmar Maurer 9eae781ab8 src/bin/pxar.rs - extract: allow to read from stdin 2019-07-05 12:39:25 +02:00
Dietmar Maurer c6c9e093d0 src/bin/pxar.rs - dump: allow to read from stdin 2019-07-05 12:32:15 +02:00
Dietmar Maurer bf1252619a src/bin/proxmox-backup-client.rs - restore: allow to pipe output to stdout 2019-07-05 12:14:50 +02:00
Dietmar Maurer f8100e9623 src/bin/proxmox-backup-client.rs: remove download cli, do everything in restore 2019-07-05 11:36:45 +02:00
Dietmar Maurer f4bf7dfcc7 src/client/remote_chunk_reader.rs: implement simple caching 2019-07-05 10:42:46 +02:00
Dietmar Maurer afb4cd28be src/backup/fixed_index.rs: implement BufferedFixedReader 2019-07-05 09:19:56 +02:00
Christian Ebner be9b39e17a src/pxar/sequential_decoder.rs: major refactoring of restore_sequential()
This splits the functionality of restore_sequential() into several smaller
functions in order to allow to reuse them when restoring by seeking based on
the goodbye table offsets.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:45:50 +02:00
Christian Ebner 48a4509c67 src/pxar/encoder.rs: fix bug with incorrect offset stored in goodbye table
Commit cd7dc87903 introduced the special treatment
for .pxarexclude files when stored in the archive.
The incorrect placement of a code snipplet from this path leads to an incorrect
offset and size stored in the goodbye table.
This fix places the start to the correct position, restoring the previously
correct behaviour.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:39:39 +02:00
Christian Ebner 7dcbe051e9 pxar: Refactor SequentialDecoder to store the callback function within the struct
Reduces the number of arguments for the function calls within the decoder.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:38:22 +02:00
Dietmar Maurer b46c3fad03 src/backup/fixed_index.rs: use correct size
We need to consider that the last chunk may have smaller size.
2019-07-04 15:13:22 +02:00
Dietmar Maurer 9cc88a7c00 src/client/http_client.rs - finish: hold reference to avoid early connection close 2019-07-04 15:11:23 +02:00
Dietmar Maurer 5e04ec707d src/api2/backup/environment.rs: improve chunk alignment/size checks 2019-07-04 13:40:43 +02:00
Dietmar Maurer a24e3993e0 src/backup/chunk_store.rs: coding style fixes 2019-07-04 11:39:10 +02:00
Dietmar Maurer e4c2fbf170 src/backup/chunk_store.rs: additionally log chunk count 2019-07-04 11:27:11 +02:00
Dietmar Maurer 9850bcdf19 src/backup/chunk_store.rs: improve error reporting 2019-07-04 11:21:54 +02:00
Wolfgang Bumiller a3f3e91da2 backup/chunk_store: rework chunk iterator
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-04 10:55:17 +02:00
Dietmar Maurer 99168f43e6 src/client/http_client.rs: s/set_recv_buf_size/set_recv_buffer_size/ 2019-07-04 10:51:18 +02:00
Dietmar Maurer c698636a5d src/tools/fs.rs - read_dir: return nix::Result 2019-07-04 10:47:16 +02:00
Dietmar Maurer a57360983b src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item 2019-07-04 09:26:44 +02:00
Dietmar Maurer 0f0a35b390 src/backup/dynamic_index.rs: lock file inside new, code cleanup 2019-07-04 08:17:30 +02:00
Dietmar Maurer a7c72ad9eb src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store 2019-07-04 08:13:16 +02:00
Dietmar Maurer a660978c9a src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats 2019-07-04 07:57:43 +02:00
Dietmar Maurer 86eda3eb0d src/bin/proxmox-backup-client.rs: implement restore using BackupReader 2019-07-03 15:45:12 +02:00
Dietmar Maurer b2a6897982 src/client/remote_chunk_reader.rs: remove dead code 2019-07-03 14:56:24 +02:00
Dietmar Maurer 7f99bf691a src/client/remote_chunk_reader.rs: implement remote chunk reader 2019-07-03 14:39:13 +02:00
Dietmar Maurer 4f6aaf542c src/client/http_client.rs: avoid compiler warning 2019-07-03 14:36:02 +02:00
Dietmar Maurer 3467cd91e9 src/client/http_client.rs: return Arc<Self>
Avoid to call canceller for each clone on drop ...
2019-07-03 14:26:07 +02:00
Dietmar Maurer 1e0784841c src/backup/read_chunk.rs: use &mut self 2019-07-03 14:25:18 +02:00
Christian Ebner 0d70535a36 src/pxar/encoder.rs: refactor simple match statements to map_err()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:36 +02:00
Christian Ebner 6e1c31163b src/pxar/sequential_decoder.rs: refactoring of simple match statements to map_err()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:24 +02:00
Christian Ebner 9d227350a3 src/pxar/sequential_decoder.rs: Cleanup and fix incorrect error message
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:16:59 +02:00
Dietmar Maurer 80af046794 src/server/rest.rs: avoid unwrap 2019-07-03 12:00:43 +02:00
Dietmar Maurer 7fb4f5642a src/server/rest.rs: log peer address, use hyper MakeService 2019-07-03 11:54:35 +02:00
Dietmar Maurer 6d1f61b208 use hyper/tokio-openssl instead of hyper/tokio-tls
This exposes the complete SSL setup. And download is much faster
now (600MB/s instead of 130MB/s)!
2019-07-02 13:36:28 +02:00
Dietmar Maurer 74d0a6bc23 src/bin/proxmox-backup-proxy.rs: set socket options for max. performance 2019-07-02 10:48:58 +02:00
Dietmar Maurer 93f8bb0445 src/bin/h2s-client.rs: avoid compiler warnings 2019-07-02 08:59:33 +02:00
Dietmar Maurer 877dd11133 src/backup/{fixed, dynamic}_index.rs: improve error messages 2019-07-02 08:56:56 +02:00
Dietmar Maurer 5c593a4d56 src/backup/read_chunk.rs: use Arc for DataStore 2019-07-02 08:49:16 +02:00
Dietmar Maurer b850673634 src/backup/read_chunk.rs: move read chunk trait into extra file
And implement LocalChunkReader.
2019-07-02 08:22:29 +02:00
Dietmar Maurer 8fad30a4b1 use latest hyper master branch to include socket buffer size fix 2019-07-02 07:09:46 +02:00
Wolfgang Bumiller f35197f449 replace tools::vec with proxmox::tools::vec
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 11:03:25 +02:00
Wolfgang Bumiller d0162d53d3 tools: remove io module
now completely replaced by proxmox::tools::io.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:12 +02:00
Wolfgang Bumiller 9b2b627fe0 update remaining users of tools::io::ops
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:03 +02:00
Wolfgang Bumiller 9110a69bd8 tools: remove read/write
now completely replaced by
proxmox::tools::io::{ReadExt, WriteExt}

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:42 +02:00
Wolfgang Bumiller ca3c3ce997 daemon: remove last use of tools::read/write
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:13 +02:00
Wolfgang Bumiller 5485b579a1 backup: use new proxmox::tools::io traits
This replaces all `write_value` calls with `write_le_value`
calls!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:37:02 +02:00
Dietmar Maurer dc9775d1b0 src/bin/h2s-client.rs: increase tokio tcp stream read buffer size
This improves download speed by factor 10.
2019-06-29 18:09:50 +02:00
Dietmar Maurer 34f7131160 src/bin/h2s-client.rs: avoid hyper, use h2 directly
But performance is still bad.
2019-06-29 15:58:18 +02:00
Dietmar Maurer 5ed2277f0e src/bin/h2s-server.rs: test to reproduce slow h2 download
Simplified code, use with h2s-client.rs.
2019-06-29 14:57:54 +02:00
Dietmar Maurer 17243003b3 src/bin/download-speed.rs: h2 download speedtest for our server
I get about 30MB/s (much too slow)
2019-06-29 13:43:10 +02:00
Dietmar Maurer a2b29b68d4 src/client/http_client.rs: use maximal h2 window sizes 2019-06-29 11:05:36 +02:00
Dietmar Maurer 872debdefa src/bin/h2client.rs: tune h2 client parameters to get reasonable speed
We now get about 2.5GB/s (not really good, but better)
2019-06-29 10:41:24 +02:00
Dietmar Maurer 38294bcc56 src/client.rs: make module put_to_stream public 2019-06-29 10:40:43 +02:00
Dietmar Maurer fded1f3154 add code to test H2 speed
I currently get about 1MB/s, which is ways too slow (must be a bug)?
2019-06-29 10:08:12 +02:00
Dietmar Maurer d48a9955a5 src/backup/dynamic_index.rs: introduce ReadChunk trait 2019-06-28 16:35:00 +02:00
Dietmar Maurer 09d7dc5024 src/api2/reader.rs: add download chunk api 2019-06-28 16:27:01 +02:00
Dietmar Maurer fcf5dea56d src/client/http_client.rs: increase h2 window size, set tcp nodelay option
To increase h2 download speed.
2019-06-28 16:00:58 +02:00
Dietmar Maurer 81a6ce6fde src/backup/chunk_store.rs: new method chunk_path()
Returns the absolute path.
2019-06-28 15:48:09 +02:00
Dietmar Maurer fcfb84fedf file download: avoid unnecessary copy 2019-06-28 07:07:52 +02:00
Dietmar Maurer 984a7c3502 src/client/http_client.rs - h2 download: implement flow control 2019-06-28 07:02:43 +02:00
Dietmar Maurer dd066d28e2 src/api2/reader.rs: implement backup reader protocol 2019-06-27 09:01:41 +02:00
Dietmar Maurer 42a87f7b96 src/server/h2service.rs: implement generic h2 service 2019-06-26 17:38:33 +02:00
Dietmar Maurer fb0470837b src/client/http_client.rs: split out code to start new h2 connections 2019-06-26 12:09:18 +02:00
Dietmar Maurer b208da8393 src/backup/data_{chunk,blob}.rs: add verify_crc method 2019-06-26 09:54:25 +02:00
Dietmar Maurer fef44d4f78 src/bin/proxmox-backup-client.rs: decode blobs on download
No more need for that clumsy blob cli (removed).
2019-06-26 09:18:59 +02:00
Dietmar Maurer c2b945341c src/client/http_client.rs - download: use generic Write type, return writer.
Make it possible to write int Vec<u8>.
2019-06-26 09:17:13 +02:00
Dietmar Maurer bb8231409e src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json
Use the KeyConfig serialization with kdf = None.
2019-06-26 07:32:34 +02:00
Dietmar Maurer 3031e44c58 src/bin/proxmox-backup-client.rs: remove stale download file, start blob cli 2019-06-25 13:00:37 +02:00