Commit Graph

1398 Commits

Author SHA1 Message Date
Dietmar Maurer e3d525feb7 src/api2/admin/datastore/backup.rs: fix typo 2019-05-28 07:13:19 +02:00
Christian Ebner af309d4d5c src/bin/pxar.rs: set archive permissions to 0o640
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:24:47 +02:00
Christian Ebner ab87f167f1 src/pxar/encoder.rs: Refactor file stat
Introduce helper functions to check file stats

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:23:49 +02:00
Christian Ebner 578b601159 src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:22:16 +02:00
Dietmar Maurer a42fa400ee src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api 2019-05-28 06:18:55 +02:00
Dietmar Maurer 29ae5c86a2 src/backup/fixed_index.rs: make chunk_size public 2019-05-28 06:18:24 +02:00
Dietmar Maurer 7dd1bcac58 src/client/http_client.rs: correctly parse error response in download_chunk_list 2019-05-27 09:35:30 +02:00
Dietmar Maurer a1e7cff3ea src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk 2019-05-27 07:57:43 +02:00
Dietmar Maurer 771953f9eb src/api2/admin/datastore/backup.rs: use smaller window sizes 2019-05-27 07:53:40 +02:00
Dietmar Maurer 174ad378d8 src/client/http_client.rs: fix request pipelining 2019-05-27 07:24:32 +02:00
Dietmar Maurer 624362226e src/client/merge_known_chunks.rs: use MergedChunkInfo as input 2019-05-26 10:52:56 +02:00
Dietmar Maurer 05cba08c9c src/client/http_client.rs: append chunks in correct order
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
2019-05-26 09:57:34 +02:00
Dietmar Maurer 417cb0731e src/api2/admin/datastore/backup.rs: verify chunk offset
This clearly shows that the current approach does not work - seems we get
chunks out of order.
2019-05-26 08:49:29 +02:00
Dietmar Maurer 34114e2606 src/server/rest.rs: correctly verify json parameters 2019-05-24 09:10:59 +02:00
Dietmar Maurer eb2bdd1bcc src/client/http_client.rs: encode parameters inside body
To avoid problems with long uris.
2019-05-24 08:32:55 +02:00
Dietmar Maurer 9bb675ec23 src/client/merge_known_chunks.rs: reduce RAM usage
There is no need to store chunk data for known chunks.
2019-05-24 07:42:58 +02:00
Dietmar Maurer ecb21b4794 src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough 2019-05-24 07:36:09 +02:00
Christian Ebner 2dbba78b98 src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:27:24 +02:00
Christian Ebner 6a19448004 src/pxar/encoder.rs: impl support to store ACLs in pxar achives
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:21:33 +02:00
Christian Ebner 7cfaade7c8 src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:16:55 +02:00
Christian Ebner b344461b33 pxar: pass feature_flags to encoder/decoder instead of individual flags
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner 687407741b src/tools/xattr.rs: add test for is_valid_xattr_name()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Christian Ebner 357e4614e2 Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
Wolfgang Bumiller c05a8c8d18 doc-test fixup
cargo test by default compiles and runs all code snippets
found in the documentation...

oops...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 15:22:33 +02:00
Wolfgang Bumiller cc84a830c5 RFC: schema: make enums static
I don't see a reason to allow these to be dynamically
modifiable.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 12:38:51 +02:00
Dietmar Maurer aa1b2e04fe src/client/merge_known_chunks.rs: merge known chunks
To decrease the number of api calls required...
2019-05-23 12:31:09 +02:00
Dietmar Maurer 91320f0879 src/client/http_client.rs: use ChunkInfo streams
This will make out of order uploads possible...
2019-05-23 12:31:09 +02:00
Wolfgang Bumiller 8ea3b1d188 tree-wide: use the new vec/io tools modules
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Wolfgang Bumiller 6f0069ae1e buildsys: make valgrind=1 to enable valgrind support
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Wolfgang Bumiller 9cdda3f7c7 tools: add helpful vector and read operations
After importing the I/O ops trait via:
    use crate::tools::io::ops::*;

Instead of:
    let mut buffer = vec![0u8; 65536];
    file.read_exact(&mut buffer)?;
use:
    let buffer = file.read_exact_allocated(65536)?;

After importing the vector helpers via:
    use crate::tools::vec::{self, ops::*};

For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
    let mut buffer = vec![0u8; len];
use:
    let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
Dietmar Maurer 8bea85b42e src/api2/admin/datastore/backup.rs: verify file size and chunk count on close 2019-05-23 08:50:36 +02:00
Dietmar Maurer 1179e15842 src/server/rest.rs: improve error handling 2019-05-23 08:15:32 +02:00
Dietmar Maurer 907f5bb0d8 src/api2/admin/datastore/backup/service.rs: improve error logging 2019-05-23 08:05:39 +02:00
Dietmar Maurer 32e1bf4260 src/bin/upload-speed.rs: code cleanup 2019-05-23 07:20:05 +02:00
Christian Ebner 238dd1b7ab tests/catar.rs: Pass dummy xattr/fcaps feature_flags to encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:41:03 +02:00
Christian Ebner 0d9bab05ab src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:50 +02:00
Christian Ebner 4add706092 src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:32 +02:00
Christian Ebner 54b14ad741 src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:09 +02:00
Christian Ebner 1201360def src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:39:16 +02:00
Christian Ebner 8bc8791338 src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:37:37 +02:00
Christian Ebner a45a123950 src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:34:36 +02:00
Christian Ebner fe61cc1099 tests/pxar.rs: Make sure the correct pxar binary is run in the tests.
If compiled in debug mode, the debug binary should be tested, in release mode
the release binary has to be invoked.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:31:37 +02:00
Christian Ebner ec56f07bec Makefile: make the tests folder available in the build folder on 'make deb'
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:31:06 +02:00
Dietmar Maurer d6f204eda6 src/client/http_client.rs: re-order code - public functions first 2019-05-23 06:28:10 +02:00
Dietmar Maurer 9af37c8f0e src/client/http_client.rs: move low level H2 code into separate class 2019-05-22 17:28:25 +02:00
Dietmar Maurer d2c48afc6e src/client/http_client.rs: code cleanup 2019-05-22 13:24:33 +02:00
Dietmar Maurer 6ab34afa88 src/client/http_client.rs: cleanup h2 backup client 2019-05-22 13:05:51 +02:00
Dietmar Maurer a55fcd740f src/api2/admin/datastore/backup.rs: suse POST for finish() api 2019-05-22 13:04:58 +02:00
Dietmar Maurer de6742c04f src/bin/h2test.rs: removed
This test does no work anymore.
2019-05-22 11:14:56 +02:00
Dietmar Maurer 5a0b521809 src/api2/admin/datastore/backup.rs: remove test callbacks 2019-05-22 10:55:27 +02:00