tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
2,953 Commits 2 Branches 54 Tags 14 MiB
a3eb7b2cea
Commit Graph

2444 Commits

Author SHA1 Message Date
Wolfgang Bumiller
37b98249f4 upload: don't return NotReady 
We need to kick (poll) the stream's next element, so simply
reiterate, the try_read!() will then return NotReady from
the next stream item for us.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-16 15:17:26 +01:00
Wolfgang Bumiller
1dcd52e5df cleanup previous commit... 
This kind of notation is weird...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-16 14:19:13 +01:00
Wolfgang Bumiller
6a50b77e72 listen on :: instead of 0.0.0.0 
Since this supports both ipv6 & ipv4 unless v6only is
explicitly enabled.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-16 14:18:16 +01:00
Dietmar Maurer
cf16af2ab3 server/rest.rs: correctly pass query/url parameters 2019-01-16 13:58:36 +01:00
Dietmar Maurer
2085142ed4 reduce compiler warnings 2019-01-16 10:19:49 +01:00
Dietmar Maurer
e8edbbd49c client/catar_backup_stream.rs: new helper for catar uploads to server 2019-01-16 10:15:39 +01:00
Dietmar Maurer
150f1bd8f6 backup/datastore.rs: use Arc for chunk_store references 2019-01-15 12:36:16 +01:00
Dietmar Maurer
1629d2ad7b api3/admin/datastore/upload_catar.rs: implement upload future 2019-01-15 11:38:26 +01:00
Dietmar Maurer
7e21da6e23 api/router.rs: allow different types of api methods 
Added a prototype for file/backup uploads.
 2019-01-14 12:26:04 +01:00
Dietmar Maurer
b41d1aab8c catar: define all feature flags 2019-01-12 16:49:02 +01:00
Dietmar Maurer
4c0fd4871e catar/encoder.rs: allow to pass list of devices 
We can use this to implement the --one-file-system like options. This
is a bit more convenient, because this way we can select to include
multiple devices (which is required by container backups).
 2019-01-12 15:43:20 +01:00
Dietmar Maurer
a8a0495612 catar/encoder.rs: disable backup of virtual file systems 2019-01-12 11:56:53 +01:00
Dietmar Maurer
d7bfac8605 catar/encoder.rs: only test fat attribute on MSDOS and FUSE file systems 2019-01-12 10:28:26 +01:00
Dietmar Maurer
13d98013b2 /catar/encoder.rs: do not store data from virtual file systems 2019-01-12 10:20:08 +01:00
Dietmar Maurer
80881f60ef catar/encoder.rs: add helpers to test file system type 2019-01-12 09:18:21 +01:00
Dietmar Maurer
1c4804cf67 catar/encoder.rs: pass magic fs number to encode_dir 2019-01-12 08:51:44 +01:00
Dietmar Maurer
dce0ebd505 catar/encoder.rs: use O_DIRECTORY to open dirs 2019-01-11 16:53:17 +01:00
Dietmar Maurer
490683ec93 catar: encode/decode sockets and fifos 2019-01-11 13:26:05 +01:00
Dietmar Maurer
20e2043a08 catar/decoder.rs: corrently pass mode to mknodat 2019-01-11 13:12:55 +01:00
Dietmar Maurer
a7e3713122 catar: encode/decode devices 2019-01-11 12:22:00 +01:00
Dietmar Maurer
1bdb3130f7 catar/format_definition.rs: add more feature flags 2019-01-11 10:44:57 +01:00
Dietmar Maurer
5c76c2f331 catar/encoder.rs: store fat file system attributes 2019-01-11 10:18:22 +01:00
Dietmar Maurer
8c1dfa6c72 catar/encoder.rs: cleanups 2019-01-11 10:01:51 +01:00
Dietmar Maurer
4f6892eb74 catar/encoder.rs: encode linux fs attributes, cleanup encoder 
Try to avoid duplicate stat calls (assume file tree is stable during backup).
 2019-01-11 09:20:10 +01:00
Dietmar Maurer
ddbdf80d51 avoid compiler warnings 2019-01-11 08:41:33 +01:00
Dietmar Maurer
373ef4a504 backup/archive_index.rs: correctly check for errors 2019-01-10 11:19:54 +01:00
Dietmar Maurer
23f68e53b4 catar/decoder.rs: restore mode, uid, gid and mtime 2019-01-10 09:02:50 +01:00
Dietmar Maurer
25f60394d2 catar/decoder.rs: avoid slashes in file names 2019-01-10 09:02:50 +01:00
Dietmar Maurer
9b1bb5a277 catar/decoder.rs: impl restore dirs, files, and symlinks 2019-01-10 09:01:36 +01:00
Wolfgang Bumiller
1ed86a0bae rest: rename utf-8-checked 'bytes' to 'utf8' 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-08 14:22:43 +01:00
Wolfgang Bumiller
9fec7ae593 rest: don't copy the body 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-08 14:21:54 +01:00
Wolfgang Bumiller
d9fbcc4631 tools/timer: fixup doc formatting 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-08 14:11:10 +01:00
Dietmar Maurer
676bf71ad4 catar/decoder.rs: restore_sequential - read payload 2019-01-08 09:26:07 +01:00
Dietmar Maurer
c6a0d3fc6b catar/decoder.rs: improve restore_sequential 2019-01-07 19:18:59 +01:00
Dietmar Maurer
b6ebfb8dab catar/decoder.rs: start implementing sequential decoder, cleanups 
And use [derive(Endian)] for endian conversion.
 2019-01-07 19:07:03 +01:00
Dietmar Maurer
02c7d8e522 catar: implement create 2019-01-07 13:25:41 +01:00
Dietmar Maurer
e86c49247e bin/catar.rs: add list command 2019-01-06 17:42:23 +01:00
Dietmar Maurer
3d8c24ecf8 catar/decoder.rs: start implementation 2019-01-06 17:27:57 +01:00
Dietmar Maurer
4b864ad436 format_definition.rs: remove wrong docs 2019-01-06 17:27:22 +01:00
Dietmar Maurer
4624fe29cc impl Read for BufferedArchiveReader 2019-01-06 10:04:45 +01:00
Dietmar Maurer
343370504b impl Seek for BufferedArchiveReader 2019-01-06 09:35:39 +01:00
Dietmar Maurer
318564ac03 buffered_read: return empty buffer on eof 2019-01-06 09:17:28 +01:00
Dietmar Maurer
0a72e26704 define and use BufferedReader trait 2019-01-05 17:28:20 +01:00
Dietmar Maurer
1006acd626 remove dead code 2019-01-05 16:55:21 +01:00
Dietmar Maurer
51b499db74 tools.rs: improve docs 2019-01-05 16:53:28 +01:00
Dietmar Maurer
39c6bd86cc backup/archive_index.rs: implement BufferedArchiveReader 
Implement relativly fast random read using binary search.
 2019-01-05 14:47:56 +01:00
Dietmar Maurer
060c4811a0 backup/chunk_store.rs: resize buffer by powers of two 2019-01-05 11:34:49 +01:00
Dietmar Maurer
9409255aae catar/encoder.rs: code cleanup 
Note: Rust Vec resize uses len(), not capacity()!
 2019-01-04 17:23:01 +01:00
Dietmar Maurer
df9973e8f5 backup/chunk_store.rs: avoid allocation in read_chunk 2019-01-04 17:16:56 +01:00
Dietmar Maurer
7394ca3e95 white space cleanups 2019-01-04 12:51:43 +01:00
Dietmar Maurer
96df2fb44b backup/chunk_store.rs: implement read_chunk() 2019-01-04 12:50:54 +01:00
Dietmar Maurer
691c89a0fb proxmox-backup-manager: add garbage collection cli 2019-01-04 11:34:32 +01:00
Dietmar Maurer
07b4694a33 cleanup crate names 2019-01-04 10:49:52 +01:00
Dietmar Maurer
4bdba5f4f1 rename backup-client to proxmox-backup-client 2019-01-04 10:41:37 +01:00
Dietmar Maurer
a689f6df0e rename pbs to proxmox-backup-manager 2019-01-04 10:39:32 +01:00
Dietmar Maurer
bc616633d5 backup/chunk_store.rs: only use one directory level 2019-01-04 10:35:22 +01:00
Dietmar Maurer
030cc2a903 move admin API to /admin/ subfolder 2019-01-04 10:32:27 +01:00
Dietmar Maurer
b663789bdb backup/archive_index.rs: pass correct size to unmap 2019-01-04 09:28:41 +01:00
Dietmar Maurer
48d0d35688 backup/image_index.rs: use little endian 2019-01-04 08:45:45 +01:00
Wolfgang Bumiller
4968bc3ac0 cli::command: wrap usage errors in a UsageError 
So we can distinguish them and show usage output
conditionally.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-03 16:08:51 +01:00
Dietmar Maurer
c7fa487250 catar/encoder.rs: make sure we write all data with write_all() 2019-01-03 15:47:32 +01:00
Dietmar Maurer
050f8a6355 chunker.rs: make sure chunk_size_avg is a power of two 2019-01-03 15:10:54 +01:00
Dietmar Maurer
ea4ea34baf improve chunker speed (avoid modulo) 2019-01-03 14:35:19 +01:00
Dietmar Maurer
3329ae8c2e add benchmark for chunker 
We get about 300MB/s - so this is still not fast enough

Note: This is quite slow with debug target, so please compile with --release
 2019-01-02 19:18:53 +01:00
Dietmar Maurer
0b8e75ed16 image_index.rs: verify file size 2019-01-02 18:14:02 +01:00
Dietmar Maurer
77703d95aa implement garbage collection for .aidx files 2019-01-02 14:27:04 +01:00
Dietmar Maurer
a360f6fa2d image_index.rs: verify header magic/version 2019-01-02 13:13:13 +01:00
Dietmar Maurer
44b3f62b42 fix typo 2019-01-02 12:56:04 +01:00
Dietmar Maurer
594fa52016 disable debug code 2019-01-02 12:55:18 +01:00
Dietmar Maurer
5032b57b44 archive_index.rs: correctly write .aidx file 2019-01-02 12:54:40 +01:00
Dietmar Maurer
580dc84bac image_index.rs: print stats 2019-01-02 12:53:49 +01:00
Dietmar Maurer
5e7a09be0d src/backup/archive_index.rs: use close() instead of flush() 
Also pass a reference to the encoder.
 2019-01-02 11:02:56 +01:00
Dietmar Maurer
94a882e900 src/backup/archive_index.rs: implement flush() 2018-12-31 18:01:07 +01:00
Dietmar Maurer
0433db1949 src/backup/archive_index.rs: first try 2018-12-31 17:30:08 +01:00
Dietmar Maurer
cbdd8c54ae create backup mod in backup.rs, improve docu 2018-12-31 16:08:04 +01:00
Dietmar Maurer
cb4412b18e move chunker.rs to backup/ 2018-12-31 15:46:16 +01:00
Dietmar Maurer
1c287cb19a rc/catar/chunker.rs: first try 2018-12-31 13:01:06 +01:00
Dietmar Maurer
bf205f94c5 avoid compiler warnings 2018-12-31 10:54:25 +01:00
Dietmar Maurer
1b0dc9f680 src/catar/encoder.rs: correctly sort goodbye items by hash key 2018-12-31 10:11:28 +01:00
Dietmar Maurer
c60d34bdbf new catar binary 
currently used for debugging
 2018-12-30 18:02:46 +01:00
Dietmar Maurer
e75eac73ca fix doc syntax 2018-12-30 17:43:53 +01:00
Dietmar Maurer
0866748de6 catar cleanups ... 2018-12-30 17:32:52 +01:00
Dietmar Maurer
6cd28d200e binary_search_tree.rs: fix docs 2018-12-30 15:34:43 +01:00
Dietmar Maurer
389e562524 src/catar/encoder.rs: auto-resize file_copy_buffer, limit number of dirs 2018-12-30 14:09:59 +01:00
Dietmar Maurer
4fa71e0573 improve catar docs 2018-12-30 13:47:27 +01:00
Dietmar Maurer
48147efd12 add a comment about posible improvements 2018-12-29 19:43:25 +01:00
Dietmar Maurer
0b78833d8e src/catar/binary_search_tree.rs: add regression tests and fix one bug 2018-12-29 18:32:03 +01:00
Dietmar Maurer
b17d7149d2 src/catar/binary_search_tree.rs: improve docu 2018-12-29 17:38:50 +01:00
Dietmar Maurer
985567fb45 src/catar/encoder.rs: cleanup, factor out write_goodbye_table 2018-12-29 17:26:32 +01:00
Dietmar Maurer
95bd5dfec7 src/catar/encoder.rs: correctly sort goodby items 2018-12-29 17:00:48 +01:00
Dietmar Maurer
46b6fbd6ae use external crate siphasher 
Because std:#️⃣:SipHasher is deprecated.
 2018-12-28 19:50:07 +01:00
Dietmar Maurer
f0f3029e26 src/catar/encoder.rs: fix bug in file copy 2018-12-28 15:02:42 +01:00
Dietmar Maurer
d05f93215e src/catar/encoder.rs: fix symlinks 2018-12-28 15:02:42 +01:00
Wolfgang Bumiller
7ebb173352 getopt: cleanup: add trailing commas 
It is customary in rust to always use trailing commas. (also
suggested by rustfmt)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
2767c5d39b getopt: cleanup: don't condense 'if' statements too much 
In a language which enforces curly braces this looks weird
and rustfmt doesn't like it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
7d95c10da0 getopt: whitespace cleanup 
to make rustfmt happy

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:27:38 +01:00
Dietmar Maurer
a0cc09b5f0 src/catar/encoder.rs: fix binary format, write goodby table 
We still ned to sort the table (BST) ...
 2018-12-28 14:27:00 +01:00
Wolfgang Bumiller
03fb895197 getopt: condense nested match to reduce indentation 
The `match value` statement is the only thing covering the
entire RawArgument::Option case. `rustfmt` suggests this
more condensed way of writing this case.

See the `git diff -w` of this patch:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 9755af2..4db4579 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,8 +72,7 @@ pub fn parse_arguments<T: AsRef<str>>(
|             RawArgument::Separator => {
|                 break;
|             }
|-            RawArgument::Option { name, value } => {
|-                match value {
|+            RawArgument::Option { name, value } => match value {
|                 None => {
|                     let mut want_bool = false;
|                     let mut can_default = false;
|@@ -125,7 +124,6 @@ pub fn parse_arguments<T: AsRef<str>>(
|                     data.push((name, v));
|                 }
|             }
|-            }
|             RawArgument::Argument { value } => {
|                 rest.push(value);
|             }

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:16:58 +01:00
Wolfgang Bumiller
4d0ea9978b getopt: let parse_arguments() take a slice of AsRef<str> 
We don't need the content to be owned strings, and we don't
need the list to be a Vec, since we only care about being
able to iterate through and copy out portions of the strings
we need, so take an &[T] where T: AsRef<str>.

This avoids .iter().map(to_string).collect() before calling
parse_arguments().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:05:05 +01:00
Wolfgang Bumiller
2482c095b1 getopt: remove skip logic 
The 'skip' variable was set to indicate that the "rest of
the args" is to be copied into the 'rest' vec. We can do
this directly and avoid the 'if' case in the loop
altogether.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 13:59:44 +01:00
Wolfgang Bumiller
99da3a073d getopt: indentation reduction 
See the `git diff -w` output:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 6548a01..517cc37 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,7 +72,10 @@ pub fn parse_arguments(
|     while pos < args.len() {
|         if skip {
|             rest.push(args[pos].clone());
|-        } else {
|+            pos += 1;
|+            continue;
|+        }
|+
|         match parse_argument(&args[pos]) {
|             RawArgument::Separator => {
|                 skip = true;
|@@ -135,7 +138,6 @@ pub fn parse_arguments(
|                 rest.push(value);
|             }
|         }
|-        }
|
|         pos += 1;
|     }

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 13:54:20 +01:00
Wolfgang Bumiller
ad8a98f7e4 parse_arguments: work with utf8 bytes and reduce indentation 
We are only caring about '=' and '-' which are single-byte
codepoints, so there's no reason not to work on a byte
slice.

Also, some indentation reduction:

Transform
    if (a) {
        ...
        return A;
    }
    return B;
into
    if (!a)
        return B;
    return A;

and
    if (a)
        if (b)
            foo()
into
    if (a && b)
        return;

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 13:47:29 +01:00
Dietmar Maurer
248c17af39 src/catar/encoder.rs: write filenames 2018-12-28 11:48:47 +01:00
Dietmar Maurer
2e4ae0e239 src/catar/encoder.rs: write file data 2018-12-28 10:44:12 +01:00
Dietmar Maurer
3192ae968c src/catar/encoder.rs: write entry and symlink data 2018-12-28 09:55:26 +01:00
Dietmar Maurer
50ea43962e make map_struct functions public 2018-12-28 08:04:46 +01:00
Dietmar Maurer
d2b03f2397 catar/encoder.rs: improve error handling 2018-12-28 07:45:15 +01:00
Dietmar Maurer
0ff559990c cleanups, avoid compiler warnings 2018-12-28 07:14:12 +01:00
Dietmar Maurer
45281d4927 catar/encoder: detect symlink and regular files 2018-12-27 14:24:31 +01:00
Dietmar Maurer
fb8365b79f catar/encoder.rs: first try 2018-12-27 13:15:47 +01:00
Wolfgang Bumiller
fba3437f1e timer: setup_timeout_handler cannot fail 
and if it does it panics anyway

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-27 10:36:28 +01:00
Dietmar Maurer
bcd879cfb3 backup-client: allow to specify directories (prepare for catar backup) 2018-12-27 10:11:11 +01:00
Dietmar Maurer
b62b6cadf4 start implementing catar 2018-12-27 09:22:23 +01:00
Dietmar Maurer
dc3de618ed new helper map_struct and map_struct_mut 2018-12-27 09:20:17 +01:00
Dietmar Maurer
a198d74fc0 cleanup: reduce compiler warnings 2018-12-25 13:29:27 +01:00
Dietmar Maurer
991de6007b remove outdated comment 2018-12-25 13:24:50 +01:00
Dietmar Maurer
7ee2aa1b94 touch_chunk: use libc::lutimensat 2018-12-25 12:27:25 +01:00
Dietmar Maurer
2c32fdde86 move lookup_datastore() to backup/datastore.rs 2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835 gather usage statistics during garbage collection 2018-12-22 16:58:16 +01:00
Dietmar Maurer
28b96b56e1 open_file_locked: improve error message 2018-12-22 15:59:55 +01:00
Dietmar Maurer
176e4af964 sweep_used_chunks: print percentage 2018-12-22 15:39:05 +01:00
Dietmar Maurer
7b2b40a893 use openssl for faster hashing 2018-12-22 14:31:59 +01:00
Dietmar Maurer
e95950e40a chunk_store: reduce number of directories 
Else, scans everything takes too long ...
 2018-12-22 14:04:05 +01:00
Dietmar Maurer
15e9b4ed60 src/api3/datastore.rs: new file 2018-12-21 13:38:41 +01:00
Dietmar Maurer
3c140f60cd use return value (avoid compiler warning) 2018-12-21 12:44:20 +01:00
Dietmar Maurer
03e4753d8e fix mutability for chunk store 2018-12-21 12:15:26 +01:00
Dietmar Maurer
2d9d143a8f backup-client: add optional chunk-size parameter 2018-12-21 11:18:08 +01:00
Dietmar Maurer
f0819fe5a5 backup-client: do not start garbage collection after backup 2018-12-21 09:24:55 +01:00
Dietmar Maurer
d62e6e2264 backup-client: use 1M chunks, make chunk_size configurable 2018-12-21 08:36:57 +01:00
Dietmar Maurer
c34eb16651 backup-clinet: specify target file name 2018-12-20 14:09:31 +01:00
Dietmar Maurer
c2d9b3914a add test code to access static global state 2018-12-20 14:03:42 +01:00
Dietmar Maurer
fe0e04c69c use proxmox-backup as package name 2018-12-20 10:45:14 +01:00
Dietmar Maurer
f7dd683b46 fix tests 2018-12-19 13:54:22 +01:00
Dietmar Maurer
277fc5a3ce try to improve error messages 2018-12-19 13:40:26 +01:00
Dietmar Maurer
eae8aa3aa9 cleanups: avoid compiler warnings 2018-12-19 12:49:23 +01:00
Dietmar Maurer
6c20a13d3c cleanup - avoid too much indentation 2018-12-19 12:40:26 +01:00
Dietmar Maurer
1c43c56b06 cleanup 2018-12-19 12:30:37 +01:00
Dietmar Maurer
2bf5f6b220 ChunkStore::sweep_used_chunks - use openat to speedup directory walks 2018-12-19 12:20:01 +01:00
Wolfgang Bumiller
1628a4c731 use timers with a signal for file locking 
* rename lock_file -> open_file_locked,
* add lock_file as a function working on already-opened
  files
* change timeout types to std::time::Duration

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-19 11:23:00 +01:00
Wolfgang Bumiller
8cf6e764ad add tools::timer module 
This wraps POSIX timers via timer_create(2).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-19 11:23:00 +01:00
Dietmar Maurer
365bb90f17 move lock_file to tools.rs 2018-12-19 10:02:24 +01:00
Dietmar Maurer
08481a0b60 impl sweep_used_chunks, first try 2018-12-19 09:51:33 +01:00
Dietmar Maurer
6ea3a0b7fc move sweep_used_chunks to chunk_store 2018-12-18 11:18:55 +01:00
Dietmar Maurer
3d5c11e5d2 start impl garbage collection 2018-12-18 11:06:03 +01:00
Dietmar Maurer
529de6c7a3 start implementing DataStore 2018-12-17 13:00:39 +01:00
Dietmar Maurer
4818c8b6f7 start implementing ImageIndexReader 2018-12-16 14:44:44 +01:00
Dietmar Maurer
a27a3ee4ba avoid compiler warnings 2018-12-16 13:57:59 +01:00
Dietmar Maurer
af3e7d7551 fix chunk store file paths 2018-12-16 13:52:16 +01:00
Dietmar Maurer
0cd9d4206f improve error messages 2018-12-16 13:43:19 +01:00
Dietmar Maurer
4fbb72a8b4 image_index.rs: implement atomic write/rename 2018-12-16 13:39:21 +01:00
Dietmar Maurer
798881a68a rebane ImageIndex to ImageIndexWriter 2018-12-16 11:51:16 +01:00
Dietmar Maurer
5e5b7f1c2d use u64 to store time 2018-12-16 11:48:03 +01:00
Dietmar Maurer
d13e3745b3 image_index.rs: write header 2018-12-15 17:05:49 +01:00
Dietmar Maurer
606ce64bb0 image_index.rs:write idx of chunks (use mmap) 2018-12-15 14:51:05 +01:00
Dietmar Maurer
f8838fe913 backup-client: add bash completion for datastore names 2018-12-15 11:24:39 +01:00
Dietmar Maurer
5f0c2d5652 remove debug println 2018-12-15 11:16:27 +01:00
Dietmar Maurer
43eeef28d1 move file_chunker to tools 2018-12-15 11:14:41 +01:00
Dietmar Maurer
48e8689505 file_chunker: use Read trait 2018-12-15 11:08:49 +01:00
Dietmar Maurer
fc8b19368a backup-client: pass chunk_size as parameter to file_chunker 2018-12-15 10:58:48 +01:00
root
a914a77448 backup-client: read file by chunks 2018-12-14 13:39:41 +01:00
root
ff5d37074d bin/backup-client.rs: nbew tool to play with chunk store 2018-12-14 08:28:56 +01:00
root
15b64d4606 api3/config/datastore.rs: create datastore files 2018-12-13 14:41:14 +01:00
Dietmar Maurer
7f4e639b37 improve completion 2018-12-12 13:00:58 +01:00
Dietmar Maurer
e3d40bb8cd complete_datastore_name: write more compact code 2018-12-12 12:41:59 +01:00
Dietmar Maurer
30d2e99c77 implement completion functions 2018-12-12 12:19:26 +01:00
Dietmar Maurer
91643d9012 section_config.rs: remove debug output 2018-12-12 12:19:26 +01:00
Dietmar Maurer
d64d80d20a add comment 2018-12-12 11:23:04 +01:00
Dietmar Maurer
5ba6968922 add comment to make intensions more clear 2018-12-12 11:21:00 +01:00
Dietmar Maurer
ca9caffa23 cleanup parameter types 2018-12-12 10:37:03 +01:00
Dietmar Maurer
38555b5072 implement completion for enums 2018-12-12 08:33:50 +01:00
Dietmar Maurer
2f0258959b improve completion 2018-12-12 08:18:38 +01:00
Dietmar Maurer
f46403cc65 improve bash completion 2018-12-12 07:51:43 +01:00
Dietmar Maurer
b6e8dd39aa impl bash completion: first try 2018-12-11 14:21:05 +01:00
Dietmar Maurer
c21de379ed install bash completion scripts 
Completion itself is not implemented.
 2018-12-11 12:54:10 +01:00
Dietmar Maurer
6460764dbb cli/command.rs: add new type CliCommandMap 2018-12-11 11:31:36 +01:00
Dietmar Maurer
bf7f103944 implement builder pattern for CliCommand 2018-12-11 11:12:13 +01:00
Dietmar Maurer
baed30b702 cli/command.rs: implement prefix match 2018-12-10 18:13:55 +01:00
Dietmar Maurer
8f62336b0f impl From for CommandLineInterface 2018-12-10 13:51:10 +01:00
Dietmar Maurer
9f6ab1fc58 rename CmdDef to CommandLineInterface 2018-12-10 13:40:10 +01:00
Dietmar Maurer
b7329c8a1a src/cli/command.rs: move code into separate file 2018-12-10 13:36:52 +01:00
Dietmar Maurer
211fabd795 try to support nested cli command definitions 2018-12-10 13:28:38 +01:00
Dietmar Maurer
34d3ba52eb api3/config/datastore.rs: implement delete 2018-12-09 16:52:32 +01:00
Dietmar Maurer
1a7bc3dd5e file_set_contents: correctly use perm parameter 2018-12-09 16:41:54 +01:00
Dietmar Maurer
f12f8ff1a6 implement tools::file_set_contents 2018-12-09 16:37:48 +01:00
Dietmar Maurer
652c11900d api3/config/datastore.rs: impl create 2018-12-09 12:51:31 +01:00
Dietmar Maurer
ea0b8b6ec0 bin/pbs-datastore.rs: start implementing cli commands 2018-12-09 11:59:32 +01:00
Dietmar Maurer
5b34c2607d rename entry to item 2018-12-09 10:25:56 +01:00
Dietmar Maurer
b65eaac62d api3/config/datastore.rs: return real data 2018-12-09 10:23:19 +01:00
Dietmar Maurer
6349ed60b4 config/datastore.rs - open file for reading 2018-12-09 10:22:39 +01:00
Dietmar Maurer
ce55dbbcc6 simplify code 2018-12-09 09:42:17 +01:00
Dietmar Maurer
461e62fcdf remove some compiler warnings 2018-12-08 14:55:54 +01:00
Dietmar Maurer
567713b4c3 rename data_store to datastore 2018-12-08 14:51:08 +01:00
Dietmar Maurer
6ce50400c5 cleanup api3 structure 2018-12-08 14:44:55 +01:00
Dietmar Maurer
678d72df6b add data_store configuration 2018-12-08 13:58:45 +01:00
Dietmar Maurer
391a2e43ff chunk_store.insert_chunk: return information about existing chunks 2018-12-08 11:25:11 +01:00
Dietmar Maurer
12bb93b3ad code cleanup 2018-12-08 11:20:02 +01:00
Dietmar Maurer
022fb421b2 lock with timeout 
clumsy, but I have node idea how to make it better
 2018-12-08 10:57:09 +01:00
Dietmar Maurer
2989f6bf20 improve error messages 2018-12-08 08:38:42 +01:00
Dietmar Maurer
b8d4766a4d code cleanup 2018-12-08 08:21:20 +01:00
Dietmar Maurer
45773720ab depend on nix, use flock 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2018-12-07 18:14:07 +01:00
Dietmar Maurer
c5d82e5f34 chunk_store: use mutex 2018-12-07 16:12:45 +01:00
Dietmar Maurer
128b37feea impl insert_chunk 2018-12-07 14:44:56 +01:00
Dietmar Maurer
35cf5daa53 start experiments with simple chunk store 2018-12-07 10:51:53 +01:00
Dietmar Maurer
b22751b20c getopts.rs: fix test 2018-12-07 09:02:24 +01:00
Dietmar Maurer
a4b7c3f2df more cleanups 2018-12-06 10:41:57 +01:00
Dietmar Maurer
d3369f819d code cleanup 2018-12-06 10:23:45 +01:00
Dietmar Maurer
c578fcd9e2 simplify formatter code 2018-12-05 18:22:56 +01:00
Dietmar Maurer
4b26195c6a extract version from environment 2018-12-05 14:40:45 +01:00
Dietmar Maurer
95b492add8 api: add /version 2018-12-05 14:03:15 +01:00
Dietmar Maurer
1571873d7b add output formatter 2018-12-05 12:43:22 +01:00
Dietmar Maurer
083418619e move src/api/server.rs -> src/server/rest.rs 2018-12-05 10:16:23 +01:00
Dietmar Maurer
5c7a1b15c8 start the GUI 2018-12-04 17:53:10 +01:00
Dietmar Maurer
826bb98286 set content type for static file download 2018-12-02 11:00:52 +01:00
Dietmar Maurer
8adbdb0a54 fix file download, listen to 0.0.0.0 2018-12-01 15:21:25 +01:00
Dietmar Maurer
f4c514c13d remove www/pbs-index.html.tt, hardcode into rust for now 2018-12-01 13:37:49 +01:00
Dietmar Maurer
e3871182c5 rename bin to src/bin/api-test-server.rs 2018-11-30 12:48:45 +01:00
Dietmar Maurer
73f29c34d7 improve command line parser 2018-11-30 12:10:26 +01:00
Dietmar Maurer
d42398666c improve error message 2018-11-30 12:07:27 +01:00
Dietmar Maurer
bfb1d69abc fix compiler warnings, add storage/config.rs 2018-11-30 11:15:26 +01:00
Dietmar Maurer
9131286246 remove missleading comment 2018-11-30 08:47:26 +01:00
Dietmar Maurer
a27b905c28 use VecDeque instead of LinkedList 2018-11-29 17:28:28 +01:00
Dietmar Maurer
c6ed6cac12 SectionConfig::write - return raw string 2018-11-29 16:47:40 +01:00
Dietmar Maurer
ae3a512dd9 SectionConfig::write - verify data 2018-11-29 13:18:15 +01:00
Dietmar Maurer
aeb4c8217b schema: implement verify_json 2018-11-29 13:06:43 +01:00
Dietmar Maurer
9c1b42d28e simplify var names 2018-11-29 09:33:27 +01:00
Dietmar Maurer
e189c93b8c section_config: add write - first try 2018-11-28 14:09:35 +01:00
Dietmar Maurer
92fb0784d4 section_config: add new struct SectionConfigData 2018-11-28 08:25:26 +01:00
Dietmar Maurer
e3a2217050 avoid clone 2018-11-27 14:10:16 +01:00
Dietmar Maurer
ee7fc4335a SectionConfig::parse - return Value 2018-11-27 12:54:40 +01:00
Dietmar Maurer
826698d56f SectionConfig: add schema for ID property 2018-11-27 12:34:56 +01:00
Dietmar Maurer
bdb631da88 section_config: improve parser 2018-11-27 11:55:21 +01:00
Dietmar Maurer
1fa897cfed section_config: parse section content 2018-11-27 09:01:36 +01:00
Dietmar Maurer
803b95a01d SectionConfigPlugin: add schema 2018-11-27 08:41:25 +01:00
Dietmar Maurer
b943ed8d2b section_config: further parser improvements 2018-11-26 18:22:07 +01:00
Dietmar Maurer
0d97734e92 use is_empty() to check for empty string 2018-11-26 18:02:29 +01:00
Dietmar Maurer
dd193a61d0 section_config: improve parser 2018-11-26 18:01:24 +01:00
Dietmar Maurer
07d3fa7289 add some inline documentation 2018-11-24 17:52:43 +01:00
Dietmar Maurer
6656ba9d32 StringSchema: add method to check string length 2018-11-24 12:37:05 +01:00
Dietmar Maurer
ab672c124e ArraySchema: add method to check array length 2018-11-24 12:33:44 +01:00
Dietmar Maurer
ed04ce8093 parse_property_string: allow arrays 
So that we can build simple lists like "1,2,3,4"
 2018-11-24 12:21:56 +01:00
Dietmar Maurer
84669dc18d rename test case 2018-11-24 11:21:01 +01:00
Dietmar Maurer
b436389d47 parse_property_string: first try 2018-11-24 11:20:17 +01:00
Dietmar Maurer
a23dd9bb42 add test for VerifyFn 2018-11-24 08:30:23 +01:00
Dietmar Maurer
715f22fb4a remove outdated test code 2018-11-24 08:15:08 +01:00
Dietmar Maurer
757d9b4769 ArraySchema: Add min_length and max_length 2018-11-24 08:12:23 +01:00
Dietmar Maurer
3e1497fb75 remove description from ApiMethod 
We use the description from parameters ObjectSchema instead.
 2018-11-23 17:02:50 +01:00
Dietmar Maurer
768e01091a implement Into Arc<Schema> 2018-11-23 13:18:41 +01:00
Dietmar Maurer
8a41cfdd55 implement ArraySchema 2018-11-23 12:05:55 +01:00
Dietmar Maurer
7edeec7b06 remove parameter macro, implement ObjectSchema builder 
And store "optional" attribute inside properties hash.
 2018-11-23 11:34:15 +01:00
Dietmar Maurer
82df76fff0 start using builder pattern instead of macros 2018-11-23 09:55:38 +01:00
Dietmar Maurer
dc454ef096 implement schema registry 2018-11-22 11:23:49 +01:00
Dietmar Maurer
00c908df85 use Arc pointer to properties 2018-11-20 17:20:50 +01:00
Dietmar Maurer
e471d69946 section_config.rs: add some test code 2018-11-19 08:07:22 +01:00
Dietmar Maurer
222454221c add section_config.rs 2018-11-19 06:47:39 +01:00
Dietmar Maurer
948c74f4ca move future test code into extra file 2018-11-19 06:41:50 +01:00
Dietmar Maurer
0f52af5b23 implement simply test future for storage operations 2018-11-18 15:01:19 +01:00
Dietmar Maurer
be45ccd203 fix getopts when called without any argument 2018-11-18 14:57:18 +01:00
Dietmar Maurer
a19f223dea impl argument parameters 2018-11-18 10:09:13 +01:00
Dietmar Maurer
c78bcf07ad getopts: more tests 2018-11-18 09:34:43 +01:00
Dietmar Maurer
62c40162ee improve getopt parser 2018-11-18 08:55:21 +01:00
Dietmar Maurer
a653882dd9 use ObjectSchema for parameters 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2018-11-18 08:46:26 +01:00
Dietmar Maurer
5d9f0eae6b another testcase for boolean argument parser 2018-11-17 11:29:42 +01:00
Dietmar Maurer
2770fbf3c3 getopt parser fixes and cleanups 2018-11-17 11:28:26 +01:00
Dietmar Maurer
7556cab45e getopts: allow "-option" and "--option" 2018-11-17 10:02:35 +01:00
Dietmar Maurer
0c9ce2bbf0 getopts: improve error handling 2018-11-17 09:57:26 +01:00
Dietmar Maurer
ffff48fcfe cleanup variable names 2018-11-16 16:47:23 +01:00
Dietmar Maurer
845901f41d implement getopt parser - first try 2018-11-16 13:14:11 +01:00
Dietmar Maurer
c548fa250a router: no need to use Fn (fn also works for static closures) 2018-11-16 11:12:00 +01:00
Dietmar Maurer
5206d1d951 fix typo 2018-11-16 09:16:37 +01:00
Dietmar Maurer
e7ea17deb0 handle uri parameters correctly 2018-11-16 09:15:33 +01:00
Dietmar Maurer
484eb9ac19 Router: allow to set match parameter name 2018-11-16 08:29:23 +01:00
Dietmar Maurer
2ab1f7119c Router: implement missing builder methods 2018-11-16 08:24:10 +01:00
Dietmar Maurer
fc937099b3 remove stale macro 2018-11-16 07:35:19 +01:00
Dietmar Maurer
fe476ce884 allow closure handlers 2018-11-15 17:47:59 +01:00
Dietmar Maurer
cbbeff1a6d add missing file 2018-11-15 17:07:55 +01:00
Dietmar Maurer
f17db0abfd cleanup module names 2018-11-15 17:07:10 +01:00
Dietmar Maurer
c0262d7480 rename Jss to Schema 2018-11-15 16:56:28 +01:00
Dietmar Maurer
426a48b04f fix parameter macro for empty object 2018-11-15 16:48:36 +01:00
Dietmar Maurer
8a7005321b improve router 2018-11-15 13:28:15 +01:00
Dietmar Maurer
e63e99d646 rename MethodInfo to Router 2018-11-15 11:46:13 +01:00
Dietmar Maurer
10be1d290a avoid macros 2018-11-15 10:57:52 +01:00
Dietmar Maurer
4dcf43d266 rename ApiError into HttpError 2018-11-15 10:42:01 +01:00
Dietmar Maurer
279ecfdfb4 remove unneeded lifetime annotations 2018-11-15 10:25:59 +01:00
Dietmar Maurer
f0b10921b8 reorder content 2018-11-15 10:18:01 +01:00
Dietmar Maurer
16b48b81c5 move ApiConfig into extra file 2018-11-15 10:14:08 +01:00
Dietmar Maurer
9bc17e8dcd move server code to api_server.rs, implement Service and NewService 2018-11-15 10:05:37 +01:00
Dietmar Maurer
08bd8d476a remove more test code, cleanups 2018-11-14 17:32:17 +01:00
Dietmar Maurer
98d82428b0 remove asnyc test code 2018-11-14 14:39:01 +01:00
Dietmar Maurer
5599e2633c correctly use Arc pointer 2018-11-14 14:28:54 +01:00
Dietmar Maurer
1a53be1443 factor out server code into api_server.rs 2018-11-14 13:22:33 +01:00
Dietmar Maurer
a9696f7beb use Arc for ApiServer 2018-11-14 11:57:01 +01:00
Dietmar Maurer
2c10fd5cb1 simplify code 2018-11-14 11:38:26 +01:00
Dietmar Maurer
324a5bd095 remove service implementation, use service_fn again 2018-11-13 14:47:03 +01:00
Dietmar Maurer
a974251ee6 improve error handling, use a Service 2018-11-13 12:36:56 +01:00
Dietmar Maurer
035cce945e remove static directory cache 2018-11-12 14:11:04 +01:00
Dietmar Maurer
198fab6f2a santtize paths in handle_request 2018-11-12 13:19:53 +01:00
Dietmar Maurer
a0efdca18b code cleanup 2018-11-11 17:19:24 +01:00
Dietmar Maurer
78d0783b00 handle_static_file_download: optimize small files 
Avoid chuncked transfer for small files.
 2018-11-11 17:10:42 +01:00
Dietmar Maurer
579fbe7dc8 implement static file download 
Fully async using futures and stream.
 2018-11-11 15:54:05 +01:00
Dietmar Maurer
cec9f02ebc limit allowed body size 2018-11-11 13:24:14 +01:00
Dietmar Maurer
3cdec2a010 use header::CONTENT_TYPE 2018-11-11 12:55:30 +01:00
Dietmar Maurer
ef1f0e655a add code to create directory name cache 2018-11-10 17:17:41 +01:00
Dietmar Maurer
3c1d7afc42 another way to read files using tokio_codec 2018-11-10 15:12:45 +01:00
Dietmar Maurer
805aec1572 play around with async tasks 2018-11-10 12:06:39 +01:00
Dietmar Maurer
6639c14bd9 use better http status codes 2018-11-10 10:32:25 +01:00
Dietmar Maurer
260c1ee8c6 cleanup futures 2018-11-10 10:00:48 +01:00
Dietmar Maurer
c819ec8dde call api method and return result 2018-11-09 13:48:57 +01:00
Dietmar Maurer
b82472c0d2 use futures - first try 2018-11-09 12:14:18 +01:00
Dietmar Maurer
4beaacb6f4 code cleanup 2018-11-07 13:58:09 +01:00
Dietmar Maurer
a908292008 implement string enum validation 2018-11-07 13:25:47 +01:00
Dietmar Maurer
0a33b12851 try to reduce space usage in ApiStringFormat by using Box 2018-11-07 13:01:14 +01:00
Dietmar Maurer
abbf40fc2e define a separate enum ApiStringFormat 2018-11-07 12:55:33 +01:00
Dietmar Maurer
d46287e083 implement string regex matcher 2018-11-07 12:35:52 +01:00
Dietmar Maurer
144f0f5eda remove unnecessary test 2018-11-07 12:16:16 +01:00
Dietmar Maurer
7b5316ddfa improve error messages 2018-11-07 12:14:52 +01:00
Dietmar Maurer
7c32470aea implement some string tests 2018-11-07 12:11:09 +01:00
Dietmar Maurer
08a13bb456 implement integer schema tests 2018-11-07 11:55:08 +01:00
Dietmar Maurer
e72677bf50 pass ApiMethod to handler 2018-11-07 11:06:37 +01:00
Dietmar Maurer
ab9e6de21c test for required parameters 2018-11-06 14:18:13 +01:00
Dietmar Maurer
b8bcc7630c try to write unit test 2018-11-06 13:58:05 +01:00
Dietmar Maurer
cfa5886c33 implement simple schema parser 2018-11-06 13:10:10 +01:00
Dietmar Maurer
42d569c186 move url related code back to main.rs 2018-11-06 09:35:11 +01:00
Dietmar Maurer
6d77fb4063 move parse_query into json_schema 2018-11-05 15:20:27 +01:00
Dietmar Maurer
0dde2f04d0 avoid static references 2018-11-03 15:10:21 +01:00
Dietmar Maurer
331ab992d7 use const for default object initializers 2018-11-03 10:50:24 +01:00
Dietmar Maurer
504b359724 another way to initialize the api tree 2018-11-03 10:42:48 +01:00
Dietmar Maurer
0e6967d648 move test code 2018-11-03 09:08:01 +01:00
Dietmar Maurer
18671ca530 convert find_method_info function into a method 2018-11-02 10:01:47 +01:00
Dietmar Maurer
8741bc2079 use reference to Jss inside PropertyMap 2018-11-02 09:50:26 +01:00
Dietmar Maurer
016aa6cbab define macro for propertymap 2018-11-02 09:44:18 +01:00
Dietmar Maurer
79fa17d986 use macro to declare static methodinfo items 2018-11-01 18:10:36 +01:00
Dietmar Maurer
bfcba4fd86 new subdirmap macro, cleanups 2018-11-01 15:41:08 +01:00
Dietmar Maurer
763220cefa use edition 2018 (beta) 
This is expected to be stable by the end of the year.
 2018-11-01 14:42:27 +01:00
Dietmar Maurer
1ac3e212c2 avoid warning about unused vars 2018-11-01 14:19:02 +01:00
Dietmar Maurer
f1c0021436 remove hardcoded static lifetime 2018-11-01 14:16:41 +01:00
Dietmar Maurer
89feb6acdf use refs in ApiMethod 2018-11-01 13:57:53 +01:00
Dietmar Maurer
b90dcc34bb cleanup lifetime in find_method_info 2018-11-01 13:54:04 +01:00
Dietmar Maurer
3cd244b9fa static_map: remove starnge type borrow 2018-11-01 13:45:10 +01:00
Dietmar Maurer
d11f14f77d do not use phf 2018-11-01 13:05:45 +01:00
Dietmar Maurer
28e47cea55 start implementing request handler 2018-11-01 11:30:49 +01:00
Dietmar Maurer
886e5ce8f7 split code into several files 2018-10-31 10:42:14 +01:00
Dietmar Maurer
22f0adf26a try to define an static api 2018-10-30 14:06:37 +01:00
Dietmar Maurer
d6a4ba7192 move code into lib.rs 2018-10-30 10:40:05 +01:00
Dietmar Maurer
d8d978ebc4 initial version 2018-10-30 10:04:30 +01:00