tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,811 Commits 2 Branches 54 Tags 14 MiB
9c18e93522
Commit Graph

1620 Commits

Author SHA1 Message Date
Dietmar Maurer
5d92935e03 src/backup/catalog.rs - dump_dir: make sure we parsed the whole data block 2019-11-09 13:21:20 +01:00
Dietmar Maurer
12a1975af9 src/backup/catalog.rs: store u64 compressed 
This way catalogs get about 1/3 smaller.
 2019-11-09 13:07:47 +01:00
Dietmar Maurer
bb19af7302 src/client/backup_reader.rs: factor out download_blob() helper 2019-11-08 13:16:41 +01:00
Dietmar Maurer
7205050059 src/client/backup_reader.rs: factor out download_fixed_index() helper 2019-11-08 12:51:52 +01:00
Dietmar Maurer
c3d84a2281 src/client/backup_reader.rs: factor out download_dynamic_index() helper 2019-11-08 12:51:52 +01:00
Dietmar Maurer
89245fb530 renamed: src/backup/catalog_blob.rs -> src/backup/catalog.rs 2019-11-08 10:41:00 +01:00
Dietmar Maurer
bf6e321744 change catalog format, use dynamic index to store catalog. 
In order to remove size restriction of a single blob.
 2019-11-08 10:35:48 +01:00
Dietmar Maurer
e016f9ff2f src/client/backup_writer.rs: rename force_close to cancel 
And do not consume self.
 2019-11-08 10:34:16 +01:00
Dietmar Maurer
0997967d65 do no restrict index names too much 2019-11-08 10:28:14 +01:00
Christian Ebner
e66621182b src/pxar/decoder.rs: impl lookup for Decoder 
Allows to lookup an entry in a directory based on the provided `DirectoryEntry`.
This is needed to navigate the filesystem based on `DirectoryEntry`s and similar
to the find_goodbye_entry() function in src/pxar/fuse.rs

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-06 10:00:41 +01:00
Christian Ebner
fb2554de29 src/pxar/decoder.rs: fix wrong filename check for Decoder::restore() 
As it turns out the original implementation was correct and the start in
`DirectoryEntry` points to the `PxarEntry` and not as wrongly stated to the
filename.
This reverts the incorrect code and adds comments to the fields clarifying this.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-06 09:59:35 +01:00
Christian Ebner
33ad183a40 src/pxar/decoder.rs: allow to pass match pattern to restore 
In order to partially restore the archive starting from the directory provided as
parameter.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-05 14:19:09 +01:00
Christian Ebner
5e8d600c71 src/pxar/decoder.rs: fix issue with restore 
`Decoder::restore()` calls the `SequentialDecoder::restore()` which expects to
encounter a `PxarEntry` at first. But the start of `DirectoryEntry` points to the
filename (except for the root dir), so skip over it.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-05 14:18:29 +01:00
Christian Ebner
9708b2d98c src/pxar/fuse.rs: refactor Iterator: use find() instead of position() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-11-04 13:59:16 +01:00
Christian Ebner
b82a101c86 catalog: store current writer position in CatalogBlobWriter 
Counts the bytes written by the CatalogBlobWriter in order to obtain the
stream position, needed to get offset to reference catalog items.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-31 09:11:51 +01:00
Dietmar Maurer
62ee2eb405 avoid some clippy warnings 2019-10-26 11:42:05 +02:00
Dietmar Maurer
834a2f95a0 avoid some clippy warnings 2019-10-25 18:44:51 +02:00
Dietmar Maurer
f58f426e86 src/pxar/binary_search_tree.rs: allow(clippy::many_single_char_names) 2019-10-25 18:22:19 +02:00
Dietmar Maurer
11377a47bb avoid some clippy warnings 2019-10-25 18:04:37 +02:00
Dietmar Maurer
9fe2f639b8 allow(clippy::cast_ptr_alignment) 2019-10-25 18:03:55 +02:00
Dietmar Maurer
cd69d36b45 src/backup/dynamic_index.rs: use write_all() 2019-10-25 17:22:41 +02:00
Christian Ebner
8abd12f1f3 pxar: decoder: cleanup un-needed Decoder::get_dir() 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-23 08:02:23 +02:00
Christian Ebner
9c25c0330a pxar: fuse: optimize fuse readdir callback 
The expensive call to Decoder::read_directory_entry() can be omitted as
Decoder::attributes() returns all the information the fuse response needs.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-23 08:02:03 +02:00
Dietmar Maurer
ffe171c5db src/pxar/encoder.rs: avoid duplicate check for MAX_DIRECTORY_ENTRIES 
Simply by reordering code.
 2019-10-23 07:55:56 +02:00
Christian Ebner
62d123e50a pxar: add functionality to pass exclude MatchPatterns on create 
This exposes the option to pass a list of exclude MatchPattern via the
'--exclude' option.
The list is encoded as file '.pxarexclude-cli' in the archives root directory.
If such a file is present in the filesystem, it is skipped and not included in
the archive in order to avoid conflicting information.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-23 07:31:49 +02:00
Christian Ebner
ba8165c607 pxar: encoder: add encode_pxar_exclude_cli() function. 
This provides the functionality needed to encode MatchPatterns passed on the cli
in the root directory.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-23 07:30:30 +02:00
Dietmar Maurer
920243b12d pxar: simplify MatchPattern::to_bytes 2019-10-23 07:29:02 +02:00
Christian Ebner
e50a90e010 pxar: Implement to_bytes() for MatchPattern in order to write them to file. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-23 07:21:11 +02:00
Dietmar Maurer
a8f10f849e src/bin/proxmox-backup-client.rs: fix parameters for get_encryption_key_password 2019-10-17 10:28:21 +02:00
Christian Ebner
40c9ff9d81 src/pxar/encoder.rs: include .pxarexclude files into the catalog 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-17 10:25:20 +02:00
Christian Ebner
20feec9fac pxar: fuse: include nsec information in stat 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-10-17 10:24:13 +02:00
Dietmar Maurer
649c5ee7b1 load_and_decrtypt_key: allow function closures to readf passphrase 2019-10-16 13:23:52 +02:00
Dietmar Maurer
60ffa9649c src/backup/manifest.rs: fix property names (use '-' instead of '_') 2019-10-14 12:28:27 +02:00
Dietmar Maurer
4c9f753c56 src/backup/data_blob.rs: define create_zero_chunk as member function 
and rename it to build_zero_chunk()
 2019-10-14 10:58:26 +02:00
Dietmar Maurer
2745151d93 src/backup/data_blob.rs: add create_zero_chunk() helper 2019-10-14 10:44:46 +02:00
Dietmar Maurer
f06b820ac0 src/backup/manifest.rs: add verify_file 2019-10-13 10:09:12 +02:00
Dietmar Maurer
511a47bd73 cleanup 2019-10-13 09:39:21 +02:00
Dietmar Maurer
e17d5d864d src/backup/manifest.rs: improve parser (try_from) 2019-10-13 08:39:49 +02:00
Dietmar Maurer
59e9ba01c6 src/backup/manifest.rs: new class to generate/parse index.json 2019-10-12 17:58:08 +02:00
Dietmar Maurer
ad6e5a6f51 rename INDEX_BLOB_NAME to MANIFEST_BLOB_NAME 2019-10-12 15:53:27 +02:00
Dietmar Maurer
296c50ba3d src/client/backup_reader.rs: add download_manifest 
Also add crypt_config as member variable.
 2019-10-12 15:50:26 +02:00
Dietmar Maurer
cf9271e27c src/client/backup_writer.rs: move BackupClient code into extra file 
And rename it to BackupWriter.
 2019-10-12 13:53:11 +02:00
Dietmar Maurer
913acb41fd src/client/backup_reader.rs: add documentation 2019-10-12 13:26:47 +02:00
Dietmar Maurer
9e490a7479 src/client/backup_reader.rs: split BackupReader code into separate file 2019-10-12 12:57:08 +02:00
Dietmar Maurer
fd04ca7a5a src/bin/proxmox-backup-client.rs: pass verbose flag to dump_image 
And print some useful status to stderr.
 2019-10-09 12:04:02 +02:00
Dietmar Maurer
2c2097ee75 fix typo in error message 2019-10-07 12:48:56 +02:00
Dietmar Maurer
f7aa6f15d3 src/api2/reader.rs: use tokio::fs::read() to read file async 2019-10-07 12:24:06 +02:00
Dietmar Maurer
7123ff7d43 src/backup/data_blob.rs: avoid Arc<CryptConfig> 
use simple reference instead.
 2019-10-07 11:41:35 +02:00
Dietmar Maurer
4ee8f53d07 remove DataChunk file format - use DataBlob instead 2019-10-06 10:31:06 +02:00
Dietmar Maurer
c26d4b4573 cleanup: remove dead code 2019-10-05 16:10:41 +02:00
Dietmar Maurer
88892ea801 src/bin/proxmox-backup-client.rs: cleanup restore 
- write directly to /dev/stdout (instead of using stdout line writer)
- avoid BufferedFixedReader
 2019-10-05 11:52:12 +02:00
Dietmar Maurer
62e05bb3b3 src/client/remote_chunk_reader.rs: pass reference to download_chunk 
Make sure we do not move data around.
 2019-10-05 11:45:20 +02:00
Dietmar Maurer
c0b1b14c05 src/api2/reader.rs - download_chunk: use blocking io 
Turns out to be twice as fast as tokio async io...
 2019-10-05 11:41:19 +02:00
Dietmar Maurer
ff01c1e393 src/client/http_client.rs: store raw data/blob size in index.json 2019-10-02 10:54:06 +02:00
Dietmar Maurer
f16aea687a src/client/http_client.rs: new helper send_upload_request() 2019-09-28 18:22:48 +02:00
Dietmar Maurer
9e603e254c src/backup/fixed_index.rs: do not use offset for checksum 2019-09-23 11:30:20 +02:00
Dietmar Maurer
fb6026b66b backup writer api: verify checksum inside close 2019-09-23 10:56:53 +02:00
Dietmar Maurer
c8c4051aa7 src/client/http_client.rs - H2Client::upload: fix parameter order 2019-09-20 13:19:55 +02:00
Dietmar Maurer
f011dba0f7 src/client/http_client.rs: add method parameter to H2 upload 2019-09-20 12:40:23 +02:00
Dietmar Maurer
792a70b9e1 src/client/http_client.rs: add content type parameter to H2 upload/requets_builder 2019-09-20 12:26:21 +02:00
Dietmar Maurer
a55b2975db src/client/http_client.rs: cleanup (rename parameter) 2019-09-20 11:57:03 +02:00
Dietmar Maurer
32efac1cce remove trailing whitespace 2019-09-20 08:12:31 +02:00
Christian Ebner
70235f72f4 src/bin/proxmox-backup-client.rs: add functionality to mount pxar archive. 
This adds the basic functionality to mount an archive stored in a repository
locally via fuse.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-20 08:01:01 +02:00
Christian Ebner
2fa91f5221 src/bin/pxar.rs: adapt to new deamonize flag 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-20 08:00:20 +02:00
Christian Ebner
d388e244f5 src/pxar/fuse.rs: make verbose flag independent from deamonize by adding this as new flag 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-20 08:00:04 +02:00
Christian Ebner
e717e9a3bf src/pxar/fuse.rs: refactor Session::new() 
and add Session::from_decoder() in order to be able to create a fuse session
with a `Decoder` given as argument.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-20 07:59:51 +02:00
Christian Ebner
946b72a6b2 src/pxar/fuse.rs: introduce Context in order to add caching. 
This patch introduces `Context` to hold the decoder, ino_offset and caches for
the attributes and the goodbye table.
By caching, certain callbacks can be handled without the need to read additional
data via the decoder, which improves performance.
The searching of the goodbye table is refactored as well, avoiding recursive
function calls in case of a hash collision.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-20 07:56:40 +02:00
Christian Ebner
1c93182371 src/pxar/decoder.rs: fix bug in decoder read 
`offset` points to the `PXAR_FILENAME`, therefore read the filename before
the `PXAR_ENTRY`.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-20 07:54:16 +02:00
Dietmar Maurer
2aba16bde1 src/backup/data_chunk.rs: add into_raw() 2019-09-19 14:24:17 +02:00
Dietmar Maurer
15bb1bba89 BackupClient: add upload() function 2019-09-19 13:59:24 +02:00
Dietmar Maurer
6d4df36c3c make download_chunk_list() public 2019-09-19 13:17:23 +02:00
Dietmar Maurer
cc2ce4a92f src/client/http_client.rs: add password parameter to HttpClient::new() 2019-09-16 12:35:23 +02:00
Christian Ebner
0e20b336e1 pxar: fuse: avoid possible hash collision in lookup by additional checking against filename 
The hash of the filename in the goodbye table items allows to quickly compare to
a hashed filename.
Unfortunately, a matching hash is no garantee for matching filenames as hash
collisions are possible.
This patch fixes such possible collisions by further checking the filenames once
a matching hash has been found.
This introduces no significant extra cost (except for the filename comparison)
for cases with matching hashes, as the lookup call has to seek and read the file
attributes (including the filename) anyway.
In cases with hash collision, the next matching item is read and treaded
analogously (what means we need at least one extra seek).
As collisions should be not that frequent, this should be an acceptable penalty.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-16 10:22:22 +02:00
Wolfgang Bumiller
ba5e67475a clippy: fix access to unaligned buffer in dynamic reader 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:50:45 +02:00
Wolfgang Bumiller
b526bd1408 backup: cleanup unused imports 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:46:46 +02:00
Wolfgang Bumiller
4dc79bb1d9 clippy: fix access to unaligned buffer 
via the newly introduced ReadExt::read_host_value_boxed()
from proxmox::tools::io.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:45:44 +02:00
Wolfgang Bumiller
ccdf3ad1ec backup: use static assertion in fixed index reader 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:43:25 +02:00
Wolfgang Bumiller
34a3845b01 clippy: don't use clone a double reference 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 14:02:30 +02:00
Wolfgang Bumiller
b21851cc5c clippy: collapse identical if branches 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 13:57:58 +02:00
Wolfgang Bumiller
299a2f18b2 fixup variable naming 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 13:57:23 +02:00
Wolfgang Bumiller
175eeb870d clippy: use write_all in file logger 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 13:56:09 +02:00
Wolfgang Bumiller
5a54935bc9 clippy: fix pointer to temporary value 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 13:55:56 +02:00
Wolfgang Bumiller
bd430c225b clippy: remove a loop{} which never actually loops 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 12:31:39 +02:00
Wolfgang Bumiller
44fed91e17 more clippy fixups 
mostly indentation changes, view with `-w`

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 12:23:12 +02:00
Wolfgang Bumiller
653b1ca10e some clippy fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-11 12:14:05 +02:00
Christian Ebner
03310bea12 src/pxar/fuse.rs: s/Buf/ReplyBuf/ and s/BufState/ReplyBufState/ 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-11 11:27:18 +02:00
Christian Ebner
48cc1b8234 src/pxar/fuse.rs: refactor stat and fix i-node mapping 
The functionality of stat is split into smaller sub-functions, which allows
to reuse them more flexible, as the code flow is similar but not always the same.
By this, the ugly and incorrect re-setting of the i-node in the lookup callback
function is avoided.
The correct i-node is now calculated beforehand and stat simply creates a
`libc::stat` struct from the provided parameters.
Also, this fixes incorrect i-node assignments in the readdir callback function.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-11 11:26:55 +02:00
Christian Ebner
22eaa905a4 src/pxar/fuse.rs: impl readdir callback for fuse 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 11:38:58 +02:00
Christian Ebner
1652192f4d src/pxar/fuse.rs: impl readlink callback for fuse 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 11:38:48 +02:00
Christian Ebner
3710b23db5 src/pxar/decoder.rs: read_link has to read filename before entry. 
The offset points to the start of the item, which is the filename

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 11:36:04 +02:00
Christian Ebner
7eb9848ba7 src/pxar/fuse.rs: impl opendir callback for fuse 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 11:35:13 +02:00
Christian Ebner
2c5fd378d3 src/pxar/fuse.rs: impl read callback for fuse 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 11:34:55 +02:00
Christian Ebner
97e4db9dd5 src/pxar/fuse.rs: impl open callback for fuse 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 11:34:18 +02:00
Christian Ebner
b00689254a src/pxar/fuse.rs: impl lookup callback for fuse 
The lookup call checks if the given filename is found in the directory referenced
by the i-node by calclulating the filenames hash and looking it up in the
directories goodbye table.
If found, the entries parameters are returned.
In order to be able to lookup the parent offset by a given file offset in the
readdir callback, this also stores the corresponding values in a HashMap.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 10:54:23 +02:00
Christian Ebner
1443927c02 src/pxar/decoder.rs: correctly handle hardlinks in Decoders attribute function 
If the attributes are then returned by seeking to the new offset.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 10:46:39 +02:00
Christian Ebner
d00097a0e6 src/pxar/decoder.rs: split functionality of list_dir into list_dir and goodbye_table 
In order to read the contents of the goodbye table while keeping the
functionality of list_dir in place as is.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 10:46:29 +02:00
Christian Ebner
ac12570e99 src/pxar/decoder.rs: make read_directory_entry usable by fuse 
... and thereby allow it to read a single directory entry based on the
start and end archive offsets.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 10:32:48 +02:00
Christian Ebner
3f7f4f34be src/pxar/decoder.rs: remove DirectoryEntry start_offset and end_offset functions 
... as they are not needed with the latest iteration of the fuse callback
function impl (which never made it into the repos).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 10:30:45 +02:00
Christian Ebner
ac30bc3326 src/pxar/decoder.rs: correctly read, check and skip filename 
The previous implementation simply skipped over `size` bytes, which is not
correct as size includes also the header.
By relying on `SequentailDecoder`s read_filename function, this is correctly
taken care of plus some more integrity checks.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-10 10:30:21 +02:00
Dietmar Maurer
4a7de56e2f avoid injecting ENV vars from Makefile 
So that we can run "cargo build" without setting vars manually.
 2019-09-09 10:51:08 +02:00
Christian Ebner
d21ae955a6 src/pxar/fuse.rs: impl getattr callback for fuse 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-06 12:24:14 +02:00
Christian Ebner
ec04ea81f2 src/pxar/fuse.rs: Refactor run_in_context and remove inode_to_offset and offset_to_inode 
as their functionality is used only once so it makes more sense to keep them
inline.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-06 12:22:53 +02:00
Christian Ebner
8413083fc5 src/pxar/fuse.rs: Improve comments, add rustfmt skip tag and reafctor 
Also, removes an unused println statement in the decoder callback function and
fixes a typo.
Further, use ABI compatible Option<&T> for FFI to avoid use of raw pointers.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-06 12:22:17 +02:00
Dietmar Maurer
54a5a885a3 src/client/http_client.rs: use async for upload_speedtest() 2019-09-06 08:55:47 +02:00
Christian Ebner
bbd055bf45 src/pxar/decoder.rs: impl functionality needed for fuse implementation 
Implements functions attributes, open, read, read_link and get_dir
to be used by the fuse implementation which uses file offsets within the archive
as inodes to reference the archives items.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-06 07:18:14 +02:00
Christian Ebner
132cb0d0db src/pxar/decoder.rs: Add comment for unsolved issue and refactor if statement. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-06 07:16:31 +02:00
Dietmar Maurer
2a05048b93 src/client/http_client.rs: simplify send_request 2019-09-05 15:07:37 +02:00
Dietmar Maurer
9edd3bf1b8 src/client/http_client.rs: use async for h2api_response() 2019-09-05 14:56:52 +02:00
Dietmar Maurer
c18fddf80f src/client/http_client.rs - download_chunk_list: simplify async code 2019-09-05 14:16:45 +02:00
Dietmar Maurer
44f59dc731 src/client/http_client.rs - H2Client: fix error message in download() 2019-09-05 13:26:06 +02:00
Dietmar Maurer
c2a5a9f353 src/client/http_client.rs: use async for download_chunk_list 2019-09-05 13:12:03 +02:00
Dietmar Maurer
2f831baec0 src/client/http_client.rs - BackupClient: use async 2019-09-05 12:55:22 +02:00
Dietmar Maurer
2a1e6d7dea src/client/http_client.rs - BackupClient: use async 2019-09-05 11:42:12 +02:00
Dietmar Maurer
d4a085e564 src/client/http_client.rs - BackupReader: use async 2019-09-05 11:42:05 +02:00
Dietmar Maurer
9d456cf8ec src/backup/catalog_blob.rs: use write_all() instead of write() 
To correctly handle interrupted system calls.
 2019-09-05 11:40:48 +02:00
Dietmar Maurer
1a7a0e74c0 src/client/http_client.rs: use async for more functions 2019-09-04 13:48:16 +02:00
Dietmar Maurer
96f5e80abb src/client/http_client.rs: use async for login() 2019-09-04 10:01:46 +02:00
Dietmar Maurer
9d35dbbb8f src/client/http_client.rs: use async for credentials 2019-09-04 09:57:29 +02:00
Dietmar Maurer
d2267b112d src/bin/proxmox-backup-client.rs: verify blob/catlog checksums 2019-09-04 08:47:14 +02:00
Christian Ebner
70b03a07c8 src/pxar/sequential_decoder.rs: make functions needed in non-sequential decoder accessible from within crate 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-04 06:15:05 +02:00
Christian Ebner
f92e8266ae src/pxar/format_definitions.rs: move header type in PxarHeader and hash in PxarGoodbyeItem to top of struct 
By this it is possible to read and check just the first u64 of the corresponding
structs in order to identify the items.
This is needed for the fuse implementation in order to get entries based on the
archive offset, used as inode.
Directories are referenced by the offset to the goodbye tail while other items
are referenced by the offset of the filename followed by the entry.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-04 06:14:30 +02:00
Christian Ebner
75933d1e83 src/pxar/fuse.rs: add function to map offset to inode and vice versa 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-04 06:13:59 +02:00
Christian Ebner
c7fee396c6 src/pxar/decoder.rs: add getter functions to obtain start/end offsets for directories 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-04 06:10:56 +02:00
Christian Ebner
3626ac611f src/pxar/decoder.rs: rustfmt fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-04 06:09:17 +02:00
Christian Ebner
3fa71727ee src/bin/proxmox-backup-client.rs: refactoring of match statement 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2019-09-04 06:08:00 +02:00
Dietmar Maurer
a40220c056 src/bin/proxmox-backup-client.rs: code cleanup - factor out some functions 2019-09-03 13:31:13 +02:00
Dietmar Maurer
36493d4d7e src/backup.rs: define INDEX_BLOB_NAME here 2019-09-03 13:15:44 +02:00
Dietmar Maurer
df65bd3da6 src/bin/proxmox-backup-client.rs - restore: verify checksums 2019-09-03 13:12:16 +02:00
Dietmar Maurer
0a51fe0011 src/backup/fixed_index.rs: new helper to compute checksum and file size 2019-09-03 13:11:45 +02:00
Dietmar Maurer
82c85a21a1 src/backup/dynamic_index.rs: new helper to compute checksum and file size 2019-09-03 13:11:09 +02:00
Dietmar Maurer
a3e032b754 src/client/http_client.rs: fix dynamic index checksum 2019-09-03 13:10:18 +02:00
Wolfgang Bumiller
c77563146b comment fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-03 11:17:33 +02:00
Wolfgang Bumiller
daef93f481 tools: add tokio::main() replacement 
to deal with block_on() not allowing blocking()

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-03 11:16:30 +02:00
Wolfgang Bumiller
66fbf5bad0 add WrappedReaderStream test 
Mostly to just document the rt.block_on() issue with
blocking().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-03 10:05:45 +02:00
Wolfgang Bumiller
2b92971fba merge fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:22:36 +02:00
Wolfgang Bumiller
083ff3fd5d update to tokio 0.2.0-alpha.4 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
1434f4f8df drop hyper_openssl 
We can use .await now, which means the whole connection
state machine doesn't need to be typed out as "types"
anymore, so, at least until hyper_openssl gets updated to
proper dependencies, let's drop it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
556eb70ea3 add generic either-or AsyncRead/Write type 
The HttpsConnector will use this. Instead of implementing a
specialized MaybeTlsStream, this is simply a generic "either
this or that kind of Async Read/Write type".

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
b30415d244 src/server/state.rs: update to tokio alpha.2 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
aadcf7c621 src/server/rest.rs: use tokio::timer::delay 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
0cdb8d9c5b src/tools/wrapped_reader_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
e668912a99 src/tools/futures.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
0f5856acca src/tools/daemon.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
56e351c911 src/tools/broadcast_future.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
627bb7d114 src/tools/async_mutex.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
75fef4b463 src/server/worker_task.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
aa4110cc64 src/server/state.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
91e4587343 src/server/rest.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
b9b7f7ec1f src/server/h2service.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
160fc8147f src/server/command_socket.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
3c0facc787 src/client/remote_chunk_reader.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
369a87e3a2 src/client/pxar_backup_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
2107bb40c1 src/client/pipe_to_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
5b3911995b src/client/merge_known_chunks.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
a6782ca10b src/client/http_client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
8630557918 src/bin/upload-speed.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
cab6816969 src/bin/test_chunk_speed2.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
fda5797b8a src/bin/proxmox-backup-proxy.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
e9722f8bde src/bin/proxmox-backup-client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:21:26 +02:00
Wolfgang Bumiller
e76ac3a499 src/bin/proxmox-backup-api.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
e235c8f719 src/bin/h2s-server.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
74be6dc9b7 src/bin/h2server.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
55d8a631fc src/bin/h2s-client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
15d0e4a3bd src/bin/h2client.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
b9203d87f4 src/bin/download-speed.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
745e652a7f src/backup/index.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
32bef1e2d1 src/backup/chunk_stream.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
f2d9b73c91 src/api_schema/router.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
ffb6434485 src/api2/reader.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
7622005574 src/api2/backup/upload_chunk.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
59b2baa0f6 src/api2/backup.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
6be147b78c src/api2/admin/datastore.rs: switch to async 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
5c1130df9f implement Sync for DynamicIndexReader and FixedIndexReader 
hyper's wrap_stream now needs this

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Wolfgang Bumiller
236761a3e6 drop src/storage/futures.rs 
it's unused and uses a tokio::Task I don't want to lookup
the std-future conversion for...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-09-02 15:17:58 +02:00
Dietmar Maurer
02fcf372e4 rc/bin/proxmox-backup-client.rs - restore: always download index.json.blob 2019-09-02 14:14:32 +02:00
Dietmar Maurer
0d9862803b src/bin/proxmox-backup-client.rs - reatore: avoid loading (large) blobs into memory 2019-09-02 14:13:31 +02:00
Christian Ebner
66c8eb9383 src/client/http_client.rs: Refactor handling Option and Result types 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-29 09:15:16 +02:00
Wolfgang Bumiller
40f8680e37 tools: remove SignalFd 
We don't use it anymore and it'll just be more work when
switching to async-await.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 13:44:53 +02:00
Wolfgang Bumiller
d21f8a5b5b backup/dynamic_index: explicitly zero-initialize the header 
The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 11:42:35 +02:00
Wolfgang Bumiller
990b930f22 backup/dynamic_index: use static assertion, fix size 
The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 11:42:35 +02:00
Christian Ebner
bcb664cb69 pxar/fuse: add pxar cli flag to set single- or multi-threaded session loop 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:43:14 +02:00
Christian Ebner
de2791868a src/pxar/sequentail_decoder.rs: followup: cleanup S_IFMT masking 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:42:22 +02:00
Christian Ebner
9cb9e1b7db src/pxar/sequential_decoder.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:42:19 +02:00
Christian Ebner
dac88033ba src/pxar/dir_stack.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:39:14 +02:00
Christian Ebner
b69169e7be src/pxar/fuse.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:37:58 +02:00
Christian Ebner
d792dc3cd2 src/pxar/match_pattern.rs: style fixups 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:37:56 +02:00
Christian Ebner
4ea22b68d9 src/pxar/format_definition.rs: style fixup 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-23 09:37:53 +02:00
Wolfgang Bumiller
7d83440c60 remove proxmox-protocol subcrate 
AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)

Moved the chunker back into src/backup/chunker.rs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 14:08:25 +02:00
Wolfgang Bumiller
7a57cb77e1 more formatting & use statement fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 13:50:27 +02:00
Christian Ebner
35dfd3ceb7 src/pxar/encoder.rs: style fixups by rustfmt 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 13:29:11 +02:00
Christian Ebner
1af30bc2fe (pxar: replace deprecated std::mem::uninitialized() 
... and use std::mem::MaybeUninit or proxmox::tools::vec::uninitialized() instead.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 13:27:25 +02:00
Wolfgang Bumiller
5e58e1bb7d {dynamic,fixed}_index: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:35:36 +02:00
Wolfgang Bumiller
f93b55b057 src/backup/index: style fixup & unsafe copy removal 
We can use the safe .copy_from_slice alternative in this
case.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:31:45 +02:00
Wolfgang Bumiller
fa17b1ce2a src/backup/index: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:27:35 +02:00
Wolfgang Bumiller
f3a8d1d7e0 api2/node/time: replace mem::uninitialized 
and fixup use statements

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:14:41 +02:00
Wolfgang Bumiller
48b85e8e3b tools: tty: replace mem::uninitialized and style fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:14:19 +02:00
Wolfgang Bumiller
b528fddfc6 tools: timer: style fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:07:56 +02:00
Wolfgang Bumiller
b7f44ace92 timer: replace mem::uninitialized 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 11:07:10 +02:00
Wolfgang Bumiller
6100071f4e tools: style & use statement fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-22 10:57:56 +02:00
Wolfgang Bumiller
5c20e2da6b src/config: more style fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-08-21 14:14:26 +02:00