Dietmar Maurer
ff3d3100d4
backup/datastore.rs: try to create useful directory layout
...
store/type/ID/TIME/xxx.yyy
2019-01-18 12:01:37 +01:00
Dietmar Maurer
0d38dcb43c
cleanup nodename()
2019-01-18 10:13:45 +01:00
Dietmar Maurer
74a69302a7
tools.rs: new helper nodename()
2019-01-18 09:58:15 +01:00
Dietmar Maurer
5a778d92b3
api3/admin/datastore.rs: rename parameter "name" to "store"
2019-01-18 08:33:11 +01:00
Dietmar Maurer
244d9b17a8
bin/proxmox-backup-client.rs: implement file name completions - first try ...
2019-01-17 14:24:20 +01:00
Dietmar Maurer
83bdac1e3b
api3/admin/datastore/upload_catar.rs: verify content type ("application/x-proxmox-backup-catar")
2019-01-17 12:43:29 +01:00
Dietmar Maurer
06aeb76a5c
client/catar_backup_stream.rs: remove debug code
2019-01-17 12:16:36 +01:00
Dietmar Maurer
0fe5d605e7
tools.rs: new helper required_string_param()
...
To extract parameters from Value.
2019-01-17 12:14:02 +01:00
Dietmar Maurer
0ee0ad5bf3
api/router.rs: return Result in upload handler
2019-01-17 12:03:38 +01:00
Dietmar Maurer
23bb8780d4
proxmox-backup-client: implement upload to server
2019-01-17 11:38:22 +01:00
Dietmar Maurer
597641fdd5
client/http_client.rs: new helper class
2019-01-17 11:38:03 +01:00
Dietmar Maurer
264f52cf70
api3/admin/datastore.rs: change api patch for upload_catar
2019-01-17 09:16:50 +01:00
Wolfgang Bumiller
37b98249f4
upload: don't return NotReady
...
We need to kick (poll) the stream's next element, so simply
reiterate, the try_read!() will then return NotReady from
the next stream item for us.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-16 15:17:26 +01:00
Wolfgang Bumiller
1dcd52e5df
cleanup previous commit...
...
This kind of notation is weird...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-16 14:19:13 +01:00
Wolfgang Bumiller
6a50b77e72
listen on :: instead of 0.0.0.0
...
Since this supports both ipv6 & ipv4 unless v6only is
explicitly enabled.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-16 14:18:16 +01:00
Dietmar Maurer
cf16af2ab3
server/rest.rs: correctly pass query/url parameters
2019-01-16 13:58:36 +01:00
Dietmar Maurer
2085142ed4
reduce compiler warnings
2019-01-16 10:19:49 +01:00
Dietmar Maurer
e8edbbd49c
client/catar_backup_stream.rs: new helper for catar uploads to server
2019-01-16 10:15:39 +01:00
Dietmar Maurer
150f1bd8f6
backup/datastore.rs: use Arc for chunk_store references
2019-01-15 12:36:16 +01:00
Dietmar Maurer
1629d2ad7b
api3/admin/datastore/upload_catar.rs: implement upload future
2019-01-15 11:38:26 +01:00
Dietmar Maurer
7e21da6e23
api/router.rs: allow different types of api methods
...
Added a prototype for file/backup uploads.
2019-01-14 12:26:04 +01:00
Dietmar Maurer
b41d1aab8c
catar: define all feature flags
2019-01-12 16:49:02 +01:00
Dietmar Maurer
4c0fd4871e
catar/encoder.rs: allow to pass list of devices
...
We can use this to implement the --one-file-system like options. This
is a bit more convenient, because this way we can select to include
multiple devices (which is required by container backups).
2019-01-12 15:43:20 +01:00
Dietmar Maurer
a8a0495612
catar/encoder.rs: disable backup of virtual file systems
2019-01-12 11:56:53 +01:00
Dietmar Maurer
d7bfac8605
catar/encoder.rs: only test fat attribute on MSDOS and FUSE file systems
2019-01-12 10:28:26 +01:00
Dietmar Maurer
13d98013b2
/catar/encoder.rs: do not store data from virtual file systems
2019-01-12 10:20:08 +01:00
Dietmar Maurer
80881f60ef
catar/encoder.rs: add helpers to test file system type
2019-01-12 09:18:21 +01:00
Dietmar Maurer
1c4804cf67
catar/encoder.rs: pass magic fs number to encode_dir
2019-01-12 08:51:44 +01:00
Dietmar Maurer
dce0ebd505
catar/encoder.rs: use O_DIRECTORY to open dirs
2019-01-11 16:53:17 +01:00
Dietmar Maurer
490683ec93
catar: encode/decode sockets and fifos
2019-01-11 13:26:05 +01:00
Dietmar Maurer
20e2043a08
catar/decoder.rs: corrently pass mode to mknodat
2019-01-11 13:12:55 +01:00
Dietmar Maurer
a7e3713122
catar: encode/decode devices
2019-01-11 12:22:00 +01:00
Dietmar Maurer
1bdb3130f7
catar/format_definition.rs: add more feature flags
2019-01-11 10:44:57 +01:00
Dietmar Maurer
5c76c2f331
catar/encoder.rs: store fat file system attributes
2019-01-11 10:18:22 +01:00
Dietmar Maurer
8c1dfa6c72
catar/encoder.rs: cleanups
2019-01-11 10:01:51 +01:00
Dietmar Maurer
4f6892eb74
catar/encoder.rs: encode linux fs attributes, cleanup encoder
...
Try to avoid duplicate stat calls (assume file tree is stable during backup).
2019-01-11 09:20:10 +01:00
Dietmar Maurer
ddbdf80d51
avoid compiler warnings
2019-01-11 08:41:33 +01:00
Dietmar Maurer
373ef4a504
backup/archive_index.rs: correctly check for errors
2019-01-10 11:19:54 +01:00
Dietmar Maurer
23f68e53b4
catar/decoder.rs: restore mode, uid, gid and mtime
2019-01-10 09:02:50 +01:00
Dietmar Maurer
25f60394d2
catar/decoder.rs: avoid slashes in file names
2019-01-10 09:02:50 +01:00
Dietmar Maurer
9b1bb5a277
catar/decoder.rs: impl restore dirs, files, and symlinks
2019-01-10 09:01:36 +01:00
Wolfgang Bumiller
1ed86a0bae
rest: rename utf-8-checked 'bytes' to 'utf8'
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-08 14:22:43 +01:00
Wolfgang Bumiller
9fec7ae593
rest: don't copy the body
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-08 14:21:54 +01:00
Wolfgang Bumiller
d9fbcc4631
tools/timer: fixup doc formatting
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-08 14:11:10 +01:00
Dietmar Maurer
676bf71ad4
catar/decoder.rs: restore_sequential - read payload
2019-01-08 09:26:07 +01:00
Dietmar Maurer
c6a0d3fc6b
catar/decoder.rs: improve restore_sequential
2019-01-07 19:18:59 +01:00
Dietmar Maurer
b6ebfb8dab
catar/decoder.rs: start implementing sequential decoder, cleanups
...
And use [derive(Endian)] for endian conversion.
2019-01-07 19:07:03 +01:00
Dietmar Maurer
02c7d8e522
catar: implement create
2019-01-07 13:25:41 +01:00
Dietmar Maurer
e86c49247e
bin/catar.rs: add list command
2019-01-06 17:42:23 +01:00
Dietmar Maurer
3d8c24ecf8
catar/decoder.rs: start implementation
2019-01-06 17:27:57 +01:00
Dietmar Maurer
4b864ad436
format_definition.rs: remove wrong docs
2019-01-06 17:27:22 +01:00
Dietmar Maurer
4624fe29cc
impl Read for BufferedArchiveReader
2019-01-06 10:04:45 +01:00
Dietmar Maurer
343370504b
impl Seek for BufferedArchiveReader
2019-01-06 09:35:39 +01:00
Dietmar Maurer
318564ac03
buffered_read: return empty buffer on eof
2019-01-06 09:17:28 +01:00
Dietmar Maurer
0a72e26704
define and use BufferedReader trait
2019-01-05 17:28:20 +01:00
Dietmar Maurer
1006acd626
remove dead code
2019-01-05 16:55:21 +01:00
Dietmar Maurer
51b499db74
tools.rs: improve docs
2019-01-05 16:53:28 +01:00
Dietmar Maurer
39c6bd86cc
backup/archive_index.rs: implement BufferedArchiveReader
...
Implement relativly fast random read using binary search.
2019-01-05 14:47:56 +01:00
Dietmar Maurer
060c4811a0
backup/chunk_store.rs: resize buffer by powers of two
2019-01-05 11:34:49 +01:00
Dietmar Maurer
9409255aae
catar/encoder.rs: code cleanup
...
Note: Rust Vec resize uses len(), not capacity()!
2019-01-04 17:23:01 +01:00
Dietmar Maurer
df9973e8f5
backup/chunk_store.rs: avoid allocation in read_chunk
2019-01-04 17:16:56 +01:00
Dietmar Maurer
7394ca3e95
white space cleanups
2019-01-04 12:51:43 +01:00
Dietmar Maurer
96df2fb44b
backup/chunk_store.rs: implement read_chunk()
2019-01-04 12:50:54 +01:00
Dietmar Maurer
691c89a0fb
proxmox-backup-manager: add garbage collection cli
2019-01-04 11:34:32 +01:00
Dietmar Maurer
07b4694a33
cleanup crate names
2019-01-04 10:49:52 +01:00
Dietmar Maurer
4bdba5f4f1
rename backup-client to proxmox-backup-client
2019-01-04 10:41:37 +01:00
Dietmar Maurer
a689f6df0e
rename pbs to proxmox-backup-manager
2019-01-04 10:39:32 +01:00
Dietmar Maurer
bc616633d5
backup/chunk_store.rs: only use one directory level
2019-01-04 10:35:22 +01:00
Dietmar Maurer
030cc2a903
move admin API to /admin/ subfolder
2019-01-04 10:32:27 +01:00
Dietmar Maurer
b663789bdb
backup/archive_index.rs: pass correct size to unmap
2019-01-04 09:28:41 +01:00
Dietmar Maurer
48d0d35688
backup/image_index.rs: use little endian
2019-01-04 08:45:45 +01:00
Wolfgang Bumiller
4968bc3ac0
cli::command: wrap usage errors in a UsageError
...
So we can distinguish them and show usage output
conditionally.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-03 16:08:51 +01:00
Dietmar Maurer
c7fa487250
catar/encoder.rs: make sure we write all data with write_all()
2019-01-03 15:47:32 +01:00
Dietmar Maurer
050f8a6355
chunker.rs: make sure chunk_size_avg is a power of two
2019-01-03 15:10:54 +01:00
Dietmar Maurer
ea4ea34baf
improve chunker speed (avoid modulo)
2019-01-03 14:35:19 +01:00
Dietmar Maurer
3329ae8c2e
add benchmark for chunker
...
We get about 300MB/s - so this is still not fast enough
Note: This is quite slow with debug target, so please compile with --release
2019-01-02 19:18:53 +01:00
Dietmar Maurer
0b8e75ed16
image_index.rs: verify file size
2019-01-02 18:14:02 +01:00
Dietmar Maurer
77703d95aa
implement garbage collection for .aidx files
2019-01-02 14:27:04 +01:00
Dietmar Maurer
a360f6fa2d
image_index.rs: verify header magic/version
2019-01-02 13:13:13 +01:00
Dietmar Maurer
44b3f62b42
fix typo
2019-01-02 12:56:04 +01:00
Dietmar Maurer
594fa52016
disable debug code
2019-01-02 12:55:18 +01:00
Dietmar Maurer
5032b57b44
archive_index.rs: correctly write .aidx file
2019-01-02 12:54:40 +01:00
Dietmar Maurer
580dc84bac
image_index.rs: print stats
2019-01-02 12:53:49 +01:00
Dietmar Maurer
5e7a09be0d
src/backup/archive_index.rs: use close() instead of flush()
...
Also pass a reference to the encoder.
2019-01-02 11:02:56 +01:00
Dietmar Maurer
94a882e900
src/backup/archive_index.rs: implement flush()
2018-12-31 18:01:07 +01:00
Dietmar Maurer
0433db1949
src/backup/archive_index.rs: first try
2018-12-31 17:30:08 +01:00
Dietmar Maurer
cbdd8c54ae
create backup mod in backup.rs, improve docu
2018-12-31 16:08:04 +01:00
Dietmar Maurer
cb4412b18e
move chunker.rs to backup/
2018-12-31 15:46:16 +01:00
Dietmar Maurer
1c287cb19a
rc/catar/chunker.rs: first try
2018-12-31 13:01:06 +01:00
Dietmar Maurer
bf205f94c5
avoid compiler warnings
2018-12-31 10:54:25 +01:00
Dietmar Maurer
1b0dc9f680
src/catar/encoder.rs: correctly sort goodbye items by hash key
2018-12-31 10:11:28 +01:00
Dietmar Maurer
c60d34bdbf
new catar binary
...
currently used for debugging
2018-12-30 18:02:46 +01:00
Dietmar Maurer
e75eac73ca
fix doc syntax
2018-12-30 17:43:53 +01:00
Dietmar Maurer
0866748de6
catar cleanups ...
2018-12-30 17:32:52 +01:00
Dietmar Maurer
6cd28d200e
binary_search_tree.rs: fix docs
2018-12-30 15:34:43 +01:00
Dietmar Maurer
389e562524
src/catar/encoder.rs: auto-resize file_copy_buffer, limit number of dirs
2018-12-30 14:09:59 +01:00
Dietmar Maurer
4fa71e0573
improve catar docs
2018-12-30 13:47:27 +01:00
Dietmar Maurer
48147efd12
add a comment about posible improvements
2018-12-29 19:43:25 +01:00
Dietmar Maurer
0b78833d8e
src/catar/binary_search_tree.rs: add regression tests and fix one bug
2018-12-29 18:32:03 +01:00
Dietmar Maurer
b17d7149d2
src/catar/binary_search_tree.rs: improve docu
2018-12-29 17:38:50 +01:00
Dietmar Maurer
985567fb45
src/catar/encoder.rs: cleanup, factor out write_goodbye_table
2018-12-29 17:26:32 +01:00
Dietmar Maurer
95bd5dfec7
src/catar/encoder.rs: correctly sort goodby items
2018-12-29 17:00:48 +01:00
Dietmar Maurer
46b6fbd6ae
use external crate siphasher
...
Because std:#️⃣ :SipHasher is deprecated.
2018-12-28 19:50:07 +01:00
Dietmar Maurer
f0f3029e26
src/catar/encoder.rs: fix bug in file copy
2018-12-28 15:02:42 +01:00
Dietmar Maurer
d05f93215e
src/catar/encoder.rs: fix symlinks
2018-12-28 15:02:42 +01:00
Wolfgang Bumiller
7ebb173352
getopt: cleanup: add trailing commas
...
It is customary in rust to always use trailing commas. (also
suggested by rustfmt)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
2767c5d39b
getopt: cleanup: don't condense 'if' statements too much
...
In a language which enforces curly braces this looks weird
and rustfmt doesn't like it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
7d95c10da0
getopt: whitespace cleanup
...
to make rustfmt happy
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:27:38 +01:00
Dietmar Maurer
a0cc09b5f0
src/catar/encoder.rs: fix binary format, write goodby table
...
We still ned to sort the table (BST) ...
2018-12-28 14:27:00 +01:00
Wolfgang Bumiller
03fb895197
getopt: condense nested match to reduce indentation
...
The `match value` statement is the only thing covering the
entire RawArgument::Option case. `rustfmt` suggests this
more condensed way of writing this case.
See the `git diff -w` of this patch:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 9755af2..4db4579 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,8 +72,7 @@ pub fn parse_arguments<T: AsRef<str>>(
| RawArgument::Separator => {
| break;
| }
|- RawArgument::Option { name, value } => {
|- match value {
|+ RawArgument::Option { name, value } => match value {
| None => {
| let mut want_bool = false;
| let mut can_default = false;
|@@ -125,7 +124,6 @@ pub fn parse_arguments<T: AsRef<str>>(
| data.push((name, v));
| }
| }
|- }
| RawArgument::Argument { value } => {
| rest.push(value);
| }
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:16:58 +01:00
Wolfgang Bumiller
4d0ea9978b
getopt: let parse_arguments() take a slice of AsRef<str>
...
We don't need the content to be owned strings, and we don't
need the list to be a Vec, since we only care about being
able to iterate through and copy out portions of the strings
we need, so take an &[T] where T: AsRef<str>.
This avoids .iter().map(to_string).collect() before calling
parse_arguments().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 14:05:05 +01:00
Wolfgang Bumiller
2482c095b1
getopt: remove skip logic
...
The 'skip' variable was set to indicate that the "rest of
the args" is to be copied into the 'rest' vec. We can do
this directly and avoid the 'if' case in the loop
altogether.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 13:59:44 +01:00
Wolfgang Bumiller
99da3a073d
getopt: indentation reduction
...
See the `git diff -w` output:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 6548a01..517cc37 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,7 +72,10 @@ pub fn parse_arguments(
| while pos < args.len() {
| if skip {
| rest.push(args[pos].clone());
|- } else {
|+ pos += 1;
|+ continue;
|+ }
|+
| match parse_argument(&args[pos]) {
| RawArgument::Separator => {
| skip = true;
|@@ -135,7 +138,6 @@ pub fn parse_arguments(
| rest.push(value);
| }
| }
|- }
|
| pos += 1;
| }
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 13:54:20 +01:00
Wolfgang Bumiller
ad8a98f7e4
parse_arguments: work with utf8 bytes and reduce indentation
...
We are only caring about '=' and '-' which are single-byte
codepoints, so there's no reason not to work on a byte
slice.
Also, some indentation reduction:
Transform
if (a) {
...
return A;
}
return B;
into
if (!a)
return B;
return A;
and
if (a)
if (b)
foo()
into
if (a && b)
return;
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-28 13:47:29 +01:00
Dietmar Maurer
248c17af39
src/catar/encoder.rs: write filenames
2018-12-28 11:48:47 +01:00
Dietmar Maurer
2e4ae0e239
src/catar/encoder.rs: write file data
2018-12-28 10:44:12 +01:00
Dietmar Maurer
3192ae968c
src/catar/encoder.rs: write entry and symlink data
2018-12-28 09:55:26 +01:00
Dietmar Maurer
50ea43962e
make map_struct functions public
2018-12-28 08:04:46 +01:00
Dietmar Maurer
d2b03f2397
catar/encoder.rs: improve error handling
2018-12-28 07:45:15 +01:00
Dietmar Maurer
0ff559990c
cleanups, avoid compiler warnings
2018-12-28 07:14:12 +01:00
Dietmar Maurer
45281d4927
catar/encoder: detect symlink and regular files
2018-12-27 14:24:31 +01:00
Dietmar Maurer
fb8365b79f
catar/encoder.rs: first try
2018-12-27 13:15:47 +01:00
Wolfgang Bumiller
fba3437f1e
timer: setup_timeout_handler cannot fail
...
and if it does it panics anyway
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2018-12-27 10:36:28 +01:00
Dietmar Maurer
bcd879cfb3
backup-client: allow to specify directories (prepare for catar backup)
2018-12-27 10:11:11 +01:00
Dietmar Maurer
b62b6cadf4
start implementing catar
2018-12-27 09:22:23 +01:00
Dietmar Maurer
dc3de618ed
new helper map_struct and map_struct_mut
2018-12-27 09:20:17 +01:00
Dietmar Maurer
a198d74fc0
cleanup: reduce compiler warnings
2018-12-25 13:29:27 +01:00
Dietmar Maurer
991de6007b
remove outdated comment
2018-12-25 13:24:50 +01:00
Dietmar Maurer
7ee2aa1b94
touch_chunk: use libc::lutimensat
2018-12-25 12:27:25 +01:00
Dietmar Maurer
2c32fdde86
move lookup_datastore() to backup/datastore.rs
2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835
gather usage statistics during garbage collection
2018-12-22 16:58:16 +01:00
Dietmar Maurer
28b96b56e1
open_file_locked: improve error message
2018-12-22 15:59:55 +01:00
Dietmar Maurer
176e4af964
sweep_used_chunks: print percentage
2018-12-22 15:39:05 +01:00
Dietmar Maurer
7b2b40a893
use openssl for faster hashing
2018-12-22 14:31:59 +01:00
Dietmar Maurer
e95950e40a
chunk_store: reduce number of directories
...
Else, scans everything takes too long ...
2018-12-22 14:04:05 +01:00
Dietmar Maurer
15e9b4ed60
src/api3/datastore.rs: new file
2018-12-21 13:38:41 +01:00
Dietmar Maurer
3c140f60cd
use return value (avoid compiler warning)
2018-12-21 12:44:20 +01:00
Dietmar Maurer
03e4753d8e
fix mutability for chunk store
2018-12-21 12:15:26 +01:00
Dietmar Maurer
2d9d143a8f
backup-client: add optional chunk-size parameter
2018-12-21 11:18:08 +01:00
Dietmar Maurer
f0819fe5a5
backup-client: do not start garbage collection after backup
2018-12-21 09:24:55 +01:00
Dietmar Maurer
d62e6e2264
backup-client: use 1M chunks, make chunk_size configurable
2018-12-21 08:36:57 +01:00
Dietmar Maurer
c34eb16651
backup-clinet: specify target file name
2018-12-20 14:09:31 +01:00
Dietmar Maurer
c2d9b3914a
add test code to access static global state
2018-12-20 14:03:42 +01:00
Dietmar Maurer
fe0e04c69c
use proxmox-backup as package name
2018-12-20 10:45:14 +01:00
Dietmar Maurer
f7dd683b46
fix tests
2018-12-19 13:54:22 +01:00
Dietmar Maurer
277fc5a3ce
try to improve error messages
2018-12-19 13:40:26 +01:00
Dietmar Maurer
eae8aa3aa9
cleanups: avoid compiler warnings
2018-12-19 12:49:23 +01:00
Dietmar Maurer
6c20a13d3c
cleanup - avoid too much indentation
2018-12-19 12:40:26 +01:00
Dietmar Maurer
1c43c56b06
cleanup
2018-12-19 12:30:37 +01:00
Dietmar Maurer
2bf5f6b220
ChunkStore::sweep_used_chunks - use openat to speedup directory walks
2018-12-19 12:20:01 +01:00