tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
1,042 Commits 2 Branches 54 Tags 14 MiB
7dd1bcac58
Commit Graph

935 Commits

Author SHA1 Message Date
Dietmar Maurer
38b0dfa511 src/backup/datastore.rs: split BackupInfo 2019-03-01 16:26:35 +01:00
Wolfgang Bumiller
dcd15d10bb backup/index: remove IndexIterator 
removed the need for it, to avoid copies

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-03-01 13:29:36 +01:00
Dietmar Maurer
17d6979afd src/bin/proxmox-backup-client.rs: generate a log file 2019-03-01 09:35:41 +01:00
Dietmar Maurer
3b151414f8 src/tools/file_logger.rs: class to log into files 2019-03-01 09:34:29 +01:00
Dietmar Maurer
5a2892f0ae src/bin/proxmox-backup-client.rs: use chrono::DateTime 2019-03-01 06:59:32 +01:00
Dietmar Maurer
51144821f9 src/bin/proxmox-backup-client.rs: login before starting backup 2019-03-01 06:48:41 +01:00
Dietmar Maurer
541b908ea4 src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch 2019-02-28 17:03:48 +01:00
Dietmar Maurer
b55cee9227 src/cli/command.rs: generate_usage_str() - fix option separator (newline) 2019-02-28 16:20:00 +01:00
Dietmar Maurer
c4c5174000 src/cli/command.rs: get_schema_type_text() - display min/max 2019-02-28 16:11:49 +01:00
Wolfgang Bumiller
df67070b1c client/backup_repo: allow ip addresses for BackupRepository 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5de2bced2d datastore: add generic open_index 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
7bc1d72778 add IndexFile trait 
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.

Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
5be4065baf make FixedIndexReader Send and update comments 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Wolfgang Bumiller
fded74d0c7 style: BufferedReader -> BufferedRead 
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-28 15:26:40 +01:00
Dietmar Maurer
8f57971714 src/api2/admin/datastore.rs: implement prune 2019-02-28 12:51:27 +01:00
Dietmar Maurer
83b7db0209 src/api2/admin/datastore.rs: start prune api 
just a dummy for now.
 2019-02-27 16:53:17 +01:00
Dietmar Maurer
0ffbccce5e rc/server/rest.rs: allow to pass parameters as application/json 2019-02-27 12:37:53 +01:00
Dietmar Maurer
f98ea63de2 src/bin/proxmox-backup-client.rs: fix bash completion 2019-02-27 12:37:53 +01:00
Wolfgang Bumiller
8731e40a7f backup/datastore: create_backup_dir: show if path existed 
To enable asserting the creation of a new backup rather than
adding to an existing one.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-27 10:15:36 +01:00
Dietmar Maurer
25f1650b71 src/bin/proxmox-backup-client.rs: minor code cleanup 2019-02-27 07:21:57 +01:00
Dietmar Maurer
9bc3ddb822 src/cli/command.rs: improve generate_usage_str 
correctly format positional array arguments ({<name>}).
 2019-02-26 12:40:51 +01:00
Dietmar Maurer
ae0be2dda9 src/bin/proxmox-backup-client.rs: allow to upload more that one directory 2019-02-26 12:27:28 +01:00
Dietmar Maurer
a477d6889a src/client/http_client.rs: login once, store and reuse ticket/token 2019-02-26 11:59:10 +01:00
Dietmar Maurer
d02c2ed701 src/cli/getopts.rs: cleanup, avoid compiler warning 2019-02-26 11:55:06 +01:00
Dietmar Maurer
f8dfbb45c0 src/tools.rs: new helper required_array_param() 2019-02-26 08:48:17 +01:00
Dietmar Maurer
aaba6faace src/cli/getopts.rs: support positional array arguments 2019-02-26 08:47:34 +01:00
Dietmar Maurer
f33fa27307 src/cli/command.rs: improve error message 2019-02-26 08:46:36 +01:00
Dietmar Maurer
92a173feb0 src/api_schema/schema.rs: improve ParameterError display function 2019-02-26 07:11:24 +01:00
Wolfgang Bumiller
f661f37427 backup/dynamic_index: split out add_chunk 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:50:45 +01:00
Wolfgang Bumiller
9ac6ec868a backup/chunk_store: split insert_chunk 
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
3a50ddd0c9 backup/chunk_store: mark ChunkIterator as fused 
Ranges are FusedIterators and it can be useful to propagate
that property to users.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
eff25ecaf2 backup/chunk_store: make percentage output optional 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Wolfgang Bumiller
d59397e60e backup/datastore: expose chunk_store.get_chunk_iterator 
will be used by the protocol handler to iterate over
existing chunks

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-25 13:44:29 +01:00
Dietmar Maurer
8eb1203bda src/backup/chunk_stat.rs: log write speed 2019-02-25 13:21:00 +01:00
Dietmar Maurer
7e3365554e src/backup/chunk_stat.rs: new struct to track chunk statistics 2019-02-25 12:52:10 +01:00
Dietmar Maurer
798f7fa065 src/backup/chunk_store.rs: return info about compressed chunk size 
So that we can generate better statistics ..
 2019-02-25 11:36:05 +01:00
Dietmar Maurer
78216a5ab1 src/backup/chunk_store.rs: use lz4 for compression 
And depend on lz4 lib now.
 2019-02-25 10:00:28 +01:00
Dietmar Maurer
793b0f4d77 src/cli/command.rs: improve help command completion 2019-02-23 18:07:32 +01:00
Dietmar Maurer
6949d91564 src/cli/command.rs: cleanup, avoid array manipulation 2019-02-23 17:50:33 +01:00
Dietmar Maurer
9d78d57960 src/cli/command.rs: implement bash completion for help 2019-02-23 17:38:10 +01:00
Dietmar Maurer
698d9d4402 src/cli/command.rs: add help command 2019-02-23 15:10:48 +01:00
Dietmar Maurer
8b6dd2240d src/cli/command.rs::allow to generate ReST docu 2019-02-23 11:29:18 +01:00
Dietmar Maurer
2f3f2bb77f src/cli/command.rs: improve usage imformation 2019-02-22 17:40:37 +01:00
Dietmar Maurer
f252ca654c src/backup/chunker.rs: simplify break test 
To make it a little bit faster ...
 2019-02-22 13:02:11 +01:00
Dietmar Maurer
6adb40eefc src/cli/command.rs: start implementing generate_usage_str() 2019-02-22 12:02:59 +01:00
Dietmar Maurer
33256db6dd src/cli/command.rs: simplify cli handler 2019-02-22 11:15:10 +01:00
Dietmar Maurer
61834c1cbf src/backup/chunker.rs: another try to speedup the chunker 2019-02-22 11:15:10 +01:00
Dietmar Maurer
532a69eb04 src/cli/getopts.rs: improve docs 2019-02-21 09:36:52 +01:00
Dietmar Maurer
4de0e142a0 cleanup cli library structure 2019-02-21 09:08:03 +01:00
Wolfgang Bumiller
56458d9764 http_client: on a tty, read password if no env var is set 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-20 15:03:34 +01:00
Wolfgang Bumiller
c9b296f117 tools: add tty helper module 
with read_password() and stdin_isatty() functions

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-20 14:43:07 +01:00
Dietmar Maurer
8cc0d6aff3 src/bin/proxmox-backup-client.rs: implement garbage-collection sub-command 2019-02-20 14:10:45 +01:00
Dietmar Maurer
81da38c143 src/client/http_client.rs: implement post 
without parameters for now...
 2019-02-20 14:09:55 +01:00
Dietmar Maurer
8c75372b79 src/backup/datastore.rs: list_backups() - include list of index files 2019-02-20 13:37:44 +01:00
Dietmar Maurer
14691fc1f3 src/bin/test_chunk_size.rs: avoid compilöer warnings, exit after 1GB 2019-02-20 12:05:28 +01:00
Dietmar Maurer
1604ec2654 src/bin/test_chunk_size.rs: another chunker test - gather some stats 2019-02-20 11:50:10 +01:00
Dietmar Maurer
fa4ebf0fce src/backup/chunker.rs: simplify code 2019-02-20 11:17:19 +01:00
Dietmar Maurer
1b9c959c32 src/backup/chunker.rs: change window size back to 48 
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
 2019-02-20 11:01:09 +01:00
Dietmar Maurer
247cdbce72 src/api2/admin/datastore/catar.rs: allow to configure chunk-size 2019-02-19 15:19:12 +01:00
Dietmar Maurer
c584aa21fd src/backup/chunker.rs: start() - correctly store hash in self.h 2019-02-19 12:40:12 +01:00
Dietmar Maurer
cd1598302a src/backup/chunker.rs: use 64 byte chunker window size 
This results in faster code.
 2019-02-19 12:27:37 +01:00
Dietmar Maurer
9a3e869cf5 src/backup/chunker.rs: simplify buzzhash start code 2019-02-19 12:17:17 +01:00
Dietmar Maurer
1c7a88aeff UploadCaTar: currectly write all data using write_all() 2019-02-18 18:31:02 +01:00
Dietmar Maurer
4d2881dd4c src/backup/chunker.rs: correctly compute window index 'idx' 
And add a simple test.
 2019-02-18 16:59:23 +01:00
Dietmar Maurer
a4a5c78c01 src/client/http_client.rs: send csrf token 2019-02-18 16:59:23 +01:00
Wolfgang Bumiller
fc7f03523c cleanup 
Error::from is already a function taking 1 parameter,
there's no need to wrap it with `|e| Error::from(e)`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-18 13:21:27 +01:00
Wolfgang Bumiller
8fdef1a81c fix typo: confi{g}dir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-18 13:11:42 +01:00
Dietmar Maurer
1cb99c234e src/server/rest.rs: correctly insert NoLogExtension() 2019-02-18 06:54:12 +01:00
Dietmar Maurer
591f570b22 src/client/http_client.rs: trim path 2019-02-18 06:24:28 +01:00
Dietmar Maurer
7f1685230a src/server/rest.rs: get_index() include username and CSRF token 
When we have an valid ticket. Also delay get_index() if called with
an invalid ticket.
 2019-02-17 19:28:32 +01:00
Dietmar Maurer
141de8374a src/server/rest.rs: factor our normalize_path() 2019-02-17 17:31:53 +01:00
Dietmar Maurer
8225aa2ff6 src/server/rest.rs: improve logs for unauthorized request 2019-02-17 17:18:44 +01:00
Dietmar Maurer
54d15ab5c5 src/api_schema/schema.rs: improve docu 2019-02-17 12:13:06 +01:00
Dietmar Maurer
51ebd07974 src/api_schema/router.rs: add docu for RpcEnvironmentType 2019-02-17 11:59:18 +01:00
Dietmar Maurer
d21c2710da api_schema/router.rs: improve doc 2019-02-17 10:30:41 +01:00
Dietmar Maurer
ef2f2efbcc improve api_schema module structure 2019-02-17 10:16:33 +01:00
Dietmar Maurer
dc9a007b11 rename src/api to src/api_schema 2019-02-17 09:59:20 +01:00
Dietmar Maurer
5ddf8cb10a cleanup auth code, verify CSRF prevention token 2019-02-16 15:52:55 +01:00
Dietmar Maurer
95aee7e032 tools/common_regex.rs: add more docu 2019-02-16 12:36:01 +01:00
Dietmar Maurer
99be8844c7 buildcfg.rs: improve doc 2019-02-16 12:31:39 +01:00
Dietmar Maurer
a8f268afbb fix configuration dir permission 
Carefully set and check permissions ...
 2019-02-16 12:19:13 +01:00
Dietmar Maurer
aada2a9719 config/datastore.rs: remove unused code 2019-02-16 10:10:45 +01:00
Dietmar Maurer
d11594db4c config/datastore.rs: return empty config if file does not exist 2019-02-16 10:06:08 +01:00
Dietmar Maurer
12400210a2 tools::file_get_contents - improve error message 2019-02-16 09:42:00 +01:00
Dietmar Maurer
6235a41862 tools::file_read_firstline - improve error message 2019-02-16 09:36:29 +01:00
Dietmar Maurer
17ed456c2e change proxy user from www-data to backup 2019-02-16 09:29:04 +01:00
Dietmar Maurer
5d14eb6a76 tools.rs: new helper to get uid/gid for the sepcified system user. 
Or is there an easier way to get that info?
 2019-02-15 15:33:12 +01:00
Dietmar Maurer
f0a6112451 backup/datastore.rs: improve error reporting 2019-02-15 14:33:03 +01:00
Dietmar Maurer
9b50c16103 section_config.rs: simplify parser by using new try_block macro 2019-02-15 12:13:15 +01:00
Dietmar Maurer
fe651dd6e5 tools.rs: fix doc tests 2019-02-15 11:33:51 +01:00
Dietmar Maurer
e80503d2a6 tools.rs: document try_block macro 2019-02-15 11:20:45 +01:00
Dietmar Maurer
f0dbba8cbe tools.rs: new try_block macro 2019-02-15 10:59:12 +01:00
Wolfgang Bumiller
1d77b6cf6b update to pam 0.7 (renamed from pam-auth) 
It now supports custom conversation methods, so instead of
new() we ask for a default authenticator taking a password
via with_password(). Since the password is now handled by
the now separate conversation handler, `set_credentials()`
is now called on the handler we get via `.get_handler()`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-15 10:34:40 +01:00
Dietmar Maurer
d4736445fe server/rest.rs: add method to log message 2019-02-15 10:16:12 +01:00
Dietmar Maurer
f02e6fc478 api2/admin/datastore/catar.rs: simplify/fix debug message 2019-02-15 10:04:37 +01:00
Dietmar Maurer
44c00c0dfd server/rest.rs: log full error messages 2019-02-15 09:55:12 +01:00
Dietmar Maurer
7171b3e079 server/rest.rs: use a protocol extension to avoid double log 
Instead of modifying the response header itself.
 2019-02-14 16:04:24 +01:00
Dietmar Maurer
7e03988caf avoid double logging of proxied requests 2019-02-14 13:28:41 +01:00
Dietmar Maurer
af6f80d3db Revert "backup/datastore: fn open is a private implementation detail" 
This reverts commit 3dacd9a660.

I currently use this in some test ...
 2019-02-14 13:13:49 +01:00
Dietmar Maurer
78a1fa6722 src/server/rest.rs: log failed requests 2019-02-14 13:07:34 +01:00
Wolfgang Bumiller
2cc90617dd tools/fs: mark ReadDir as Send 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
62f2422f6a backup/chunk_store: verify chunk file names 
get_chunk_iterator() should skip over files which aren't an
actual chunk

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
3dacd9a660 backup/datastore: fn open is a private implementation detail 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Wolfgang Bumiller
9739aca41f backup/chunk_store: open base dir in get_chunk_iterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 12:37:31 +01:00
Dietmar Maurer
db4868ef4f src/client/backup_repo.rs - improve docu 2019-02-14 11:17:08 +01:00
Dietmar Maurer
151c6ce27d src/client.rs: improve library structure and docu 2019-02-14 11:11:39 +01:00
Wolfgang Bumiller
dd1bfe6c12 tools/timer: add module documentation 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:13:42 +01:00
Wolfgang Bumiller
c7f481b642 backup: replace ChunkIterator 
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
121f18efb7 tools: use scan_subdir iterator for scandir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
32286b039c tools/fs: add file_name_utf8_unchecked shortcut 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
806d7a6a35 tools/fs: change scan_subdir to use filter_file_name_regex 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
53e1e7ca72 tools/fs: add filter_file_name_regex to FileIterOps 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
23fba9d777 tools/fs: add FileIterOps trait and FileTypeFilter 
FileIterOps will provide operations on file entry iterators.
For now this provides a file type filter.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
1db416150d tools/fs: let ReadDir iterator yield ReadDirEntry 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
b12505da69 tools/fs: add ReadDirEntry helper 
This wraps nix::dir::Entry with the parent and additionally
exposes the parent directory file descriptor since we need
it to perform openat().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
a25f863afd tools/fs: add scan_subdir helper 
This filters the results of read_subdir with a regex.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Wolfgang Bumiller
b4d5787de9 tools: start fs submodule, add read_subdir 
This creates an iterator relative to a RawFd via
nix::dir::Dir over nix::dir::Entrys.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-14 10:09:59 +01:00
Dietmar Maurer
0dffe3f99a src/client/http_client.rs: try to login 
use an environment var to store passphrase (PBS_PASSWORD)
 2019-02-13 14:31:43 +01:00
Dietmar Maurer
33d64b81e0 src/bin/proxmox-backup-client.rs: define and use BackupRepository 2019-02-13 12:30:52 +01:00
Wolfgang Bumiller
4a3f65172c client: use hyper-tls for now 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-13 11:03:09 +01:00
Wolfgang Bumiller
42d6e4fb05 tools: borrow: implement Deref for Tied 
Eg. if you have an x: Tied<Dir, Iterator> now you can simply
call x.next() as it'll dereference to the Iterator.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-13 11:03:09 +01:00
Dietmar Maurer
541a302224 api2/admin/datastore/catar.rs: use nested file extensions 
i.e. archive.catar.didx
 2019-02-13 10:37:00 +01:00
Dietmar Maurer
e5064ba607 simplify backup lib structure (pub use xxx:*), improve doc 2019-02-12 14:13:31 +01:00
Dietmar Maurer
d78345bcfc src/backup.rs - improve doc 2019-02-12 13:27:11 +01:00
Dietmar Maurer
93d5d77952 rename ArchiveIndex to DynamicIndex 
also changed the file extension from .aidx to .didx
 2019-02-12 12:05:33 +01:00
Dietmar Maurer
91a905b6dd rename ImageIndex to FixedIndex 
also changed the file extension fron .iidx to .fidx
 2019-02-12 11:50:45 +01:00
Wolfgang Bumiller
82bc0ad40c backup: fixup warnings and readd lost file type check 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:43:43 +01:00
Wolfgang Bumiller
9349d2a1d2 backup: cleanup long lines 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
fdd71f5286 backup: use ChunkIterator for sweep_unused_chunks 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
c9e7f4dc8b backup: add optional progress callback to ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
4070096c6a backup: add ChunkIterator 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
6ed25cbe3a tools: add Tied value 
Tie two dependent values together, such as a nix::Dir with
its nix::dir::Iter<'a> where 'a is tied to the Dir's
lifetime, making it otherwise impossible to return them or
store them in a struct.

Alternatively we could try the 'rental' crate.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Wolfgang Bumiller
9ac1045c7b bin: manager: fixup datastore parameter name 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-12 10:39:11 +01:00
Dietmar Maurer
8f8d5a420c avoid compiler warning 2019-02-11 15:12:01 +01:00
Dietmar Maurer
aa5a4060ab bin/proxmox-backup-api.rs: improve error handling 2019-02-11 15:11:42 +01:00
Dietmar Maurer
4223d9f800 bin/proxmox-backup-proxy.rs: improve error handling 2019-02-11 14:43:26 +01:00
Wolfgang Bumiller
ace9e3531a access: use proxmox-backup-auth for pam 
allows customization via /etc/pam.d/proxmox-backup-auth

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-07 13:40:22 +01:00
Wolfgang Bumiller
13f8310cac api2: time: use /etc/localtime as fallback 
in case there is no /etc/timezone

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-05 14:14:56 +01:00
Wolfgang Bumiller
a7cac14ed9 tools: use std::fs::read for file_get_contents 
Until we add the extra options like we have in pve, just use
the already existing (since 1.26) shortcut.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:35:32 +01:00
Wolfgang Bumiller
e937034c86 remove some rather inconvenient debug output 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:34:38 +01:00
Wolfgang Bumiller
9d02e5c38a proxy: use <configdir>/server.pfx for tls 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:19:32 +01:00
Wolfgang Bumiller
9f4962d396 introduce buildcfg module and PROXMOX_CONFIGDIR 
buildcfg.rs should contain convenience variables or macros
for using build-time configured variables

For now we replace hardcoded "/etc/proxmox-backup/<foo>"
with configdir!("<foo>").

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 15:13:08 +01:00
Wolfgang Bumiller
0d176f3681 proxy: use TLS via tokio-tls 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-04 14:56:07 +01:00
Wolfgang Bumiller
e0a5d1ca2f tests: chunk_store: cleanup .testdir 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
a4f9188300 tests: import lazy_static as needed 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
6285b251e7 don't hardcode /usr/share paths 
Add a defines.mk file defining the standard default layout
for installation (PREFIX, DATAROOTDIR, LIBDIR etc.).

Add a $(JSDIR) variable and use env!("JSDIR") in place of
the hardcoded /usr/share/javascript/proxmox-backup path.

Have defines.mk include an *optional* local.mak to override
JSDIR to www/ during local development.

Add `make install` targets and remove debian/install.

Note that service files are now in $libdir/proxmox-backup/
as there's usually no point in starting them from the
command line, so /usr/sbin makes no sense.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Wolfgang Bumiller
8e4df103dd remove useless command option from daemons 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-02-01 10:41:54 +01:00
Dietmar Maurer
6278365fd9 improve ApiMethod docu 2019-02-01 10:38:07 +01:00
Dietmar Maurer
4b2cdeb9a6 implement relead_timezone flag 2019-02-01 10:04:46 +01:00
Dietmar Maurer
c82bc1a1f9 depend on pam-auth, use pam for root login 
Please use username "root@pam" for login via GUI.
 2019-02-01 09:30:50 +01:00
Dietmar Maurer
a154a8e8a4 delay unauthorized request (rate limit) 2019-01-31 14:34:21 +01:00
Dietmar Maurer
5996577ab6 move http error class to router.rs 2019-01-31 13:22:30 +01:00
Dietmar Maurer
b9903d6331 server/rest.rs: verify auth cookie 2019-01-31 12:22:00 +01:00
Dietmar Maurer
f484eed3c2 Utils.js: fix cookie handling 
Use unsecure cookie foör testing.
 2019-01-31 10:08:08 +01:00
Dietmar Maurer
9f49fe1d5d avoid compiler warnings 2019-01-30 18:25:37 +01:00
Dietmar Maurer
34f956bc25 api2/access.rs: add ticket api 2019-01-30 15:16:10 +01:00
Wolfgang Bumiller
1bf446a33e fix timer_delete call 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-30 12:04:20 +01:00
Dietmar Maurer
cf671670a0 auth_helpers.rs: add timestamp to csrf token 2019-01-29 17:50:03 +01:00
Dietmar Maurer
1e76cbc6a2 auth_helpers.rs: implement assemble_csrf_prevention_token 2019-01-29 17:41:45 +01:00
Dietmar Maurer
d01e2420f7 load auth keys on startup 2019-01-29 17:21:58 +01:00
Dietmar Maurer
6c30068ebf auth_helpers.rs: split code into separate file 2019-01-29 16:55:49 +01:00
Dietmar Maurer
39a90ca6c5 bin/proxmox-backup-api.rs: gererate secret for csrf token 2019-01-29 13:15:33 +01:00
Dietmar Maurer
26027a151e tools/ticket.rs: remove debug statements 2019-01-29 13:04:01 +01:00
Dietmar Maurer
8d04280b29 tools/ticket.rs: add basic ticket support 2019-01-29 12:59:07 +01:00
Dietmar Maurer
c8f3f9b1af server/rest.rs: simplify proxy code 
Only pass neccessary parameters.
 2019-01-28 18:22:16 +01:00
Dietmar Maurer
a859fa95a9 add protected flag for some api methods 2019-01-28 18:16:47 +01:00
Dietmar Maurer
a3da38ddfd server/rest.rs: implement proxy_sync_api_request 2019-01-28 18:06:42 +01:00
Dietmar Maurer
f120483353 server/rest.rs: add proxy_sync_api_request() dummy 2019-01-28 17:30:39 +01:00
Dietmar Maurer
4e4df8e2f0 api/router.rs: add new 'protected' flag to ApiMethod 2019-01-28 17:18:42 +01:00
Dietmar Maurer
5bb3398171 server/formatter.rs: fix extjs error format 2019-01-28 13:44:48 +01:00
Dietmar Maurer
02c7a75520 src/bin/proxmox-backup-proxy.rs: implement unpriviledged server 
We want to run the public server as user www-data. Requests needing
root priviledges needs to be proxied to the proxmox-backup.service, which
now listens to 127.0.0.1:82.
 2019-01-28 13:29:58 +01:00
Dietmar Maurer
48849593ee api2/node/services.rs: implement service commands 2019-01-27 12:40:31 +01:00
Dietmar Maurer
d7d23785f0 RpcEnvironment: implement set_user() and get_user() 2019-01-27 10:52:26 +01:00
Dietmar Maurer
162b979394 RpcEnvironment: add environment type enum RpcEnvironmentType 2019-01-27 10:33:42 +01:00
Dietmar Maurer
e82dad9700 also pass rpcenv to async handlers 2019-01-27 10:18:52 +01:00
Dietmar Maurer
d2ab5f19e2 api2/node/services.rs: start services api 2019-01-26 17:54:18 +01:00
Dietmar Maurer
0f253593c6 move rpc environment implementation to separate files 2019-01-26 15:08:02 +01:00
Dietmar Maurer
6049b71f41 api: pass RpcEnvirnment to api handlers 2019-01-26 14:50:37 +01:00
Dietmar Maurer
d96d82736d use crate log and syslog 2019-01-26 10:56:11 +01:00
Dietmar Maurer
e182ab4a4a api2/node/syslog.rs: improve error handling 2019-01-26 10:23:52 +01:00
Dietmar Maurer
0441886842 api2/node/syslog.rs: implement dump journal 2019-01-25 18:20:51 +01:00
Dietmar Maurer
4f9a726833 api2/node/syslog.rs: add syslog api schema 2019-01-25 17:17:30 +01:00
Dietmar Maurer
1631c54fca tools::get_hardware_address - fix FILENAME, use file_get_contents() 2019-01-25 12:23:47 +01:00
Dietmar Maurer
af2fddea4f api2/node/dns.rs: implement concurrent update protection 2019-01-25 11:38:59 +01:00
Dietmar Maurer
de6b0721fa pi2/node/dns.rs: return file digest 2019-01-25 11:03:41 +01:00
Dietmar Maurer
2296860027 tools.rs: move digest_to_hex() into tools 2019-01-25 10:58:28 +01:00
Dietmar Maurer
53157ca6cf tools.rs: implement file_get_contents() 2019-01-25 10:44:40 +01:00
Dietmar Maurer
46b79b9ee5 api2/node/dns.rs: improve dns api 2019-01-25 10:15:32 +01:00
Dietmar Maurer
332dcc2277 backup/chunk_store.rs: fix test case - use absolute path 2019-01-25 10:14:25 +01:00
Dietmar Maurer
8f973f8181 api2/node/dns.rs: implement resolv.conf parser 2019-01-24 14:59:40 +01:00
Dietmar Maurer
784252dbc5 backup/datastore.rs: use single lazy_static block 2019-01-24 14:58:47 +01:00
Dietmar Maurer
e6ffeb918c api2/node/time.rs: implement set timezone 2019-01-24 12:05:06 +01:00
Dietmar Maurer
cd8b91f78c api2/node/time.rs: add return schema 2019-01-24 11:07:11 +01:00
Dietmar Maurer
447787ab7c tools.rs: implement file_read_firstline 2019-01-24 10:43:30 +01:00
Dietmar Maurer
0463602a79 api2/node/time.rs: implement read api 2019-01-23 15:14:14 +01:00
Dietmar Maurer
b2b3485d5f src/api2/node: start node configuration api 2019-01-23 13:09:17 +01:00
Dietmar Maurer
d15009c0ce server/rest.rs: fake login cookie 2019-01-23 12:49:10 +01:00
Dietmar Maurer
248cb51862 add missing api2.rs 2019-01-22 13:56:27 +01:00
Dietmar Maurer
7e13b2d67f api2/subscription.rs: add subscription api class 2019-01-22 12:50:19 +01:00
Dietmar Maurer
576e3bf252 rename api3 back to api2 
There is no real need to change the path, so using api2 we can reuse
all helpers (like tools from proxmox widget toolkit).
 2019-01-22 12:10:38 +01:00
Dietmar Maurer
1adb353d6f lient/http_client.rs: simplify code 2019-01-22 10:36:04 +01:00
Dietmar Maurer
41c039e143 bin/proxmox-backup-client.rs: implement list command 2019-01-21 18:58:14 +01:00
Dietmar Maurer
1fdb4c6ffa client/http_client.rs: implement get, share common code 2019-01-21 18:56:48 +01:00
Dietmar Maurer
83dbd80bd8 api3/admin/datastore.rs: implement list backups 2019-01-21 11:22:55 +01:00
Dietmar Maurer
7ca80246ea backup/datastore.rs: use DateTime<Utc> to store backup time 2019-01-21 10:51:52 +01:00
Dietmar Maurer
be0084b001 white space fix 2019-01-21 10:32:07 +01:00
Dietmar Maurer
e25736b4c3 backup/datastore.rs: implement list_backups 2019-01-20 18:02:27 +01:00
Dietmar Maurer
cce1676a9e tools.rs: use FnMut for scandir callback 2019-01-20 17:49:11 +01:00
Dietmar Maurer
443f3743bb tools.rs: new scandir() helper 2019-01-20 17:31:43 +01:00
Dietmar Maurer
68469eebb4 backup/chunk_store.rs: verify base path is absolute 2019-01-20 16:49:22 +01:00
Dietmar Maurer
059ca7c3a1 remove crate tokio-codec (seems to be part of tokio now) 2019-01-20 14:28:06 +01:00
Dietmar Maurer
8eeef8492a tools/wrapped_reader_stream.rs: allocate Vec<u8> for buffer 2019-01-20 11:01:18 +01:00
Dietmar Maurer
ff8bdf3b64 tools/wrapped_reader_stream.rs: only implement Stream (without AsyncRead) 
Looks more efficent ...
 2019-01-20 10:41:21 +01:00
Dietmar Maurer
8e89d9cafe tools/wrapped_reader_stream.rs: use tokio_threadpool::blocking 2019-01-20 10:28:51 +01:00
Dietmar Maurer
ce7ba139f1 catar: improve bash completion 2019-01-20 09:55:20 +01:00
Dietmar Maurer
29f34b8e1a /api3/admin/datastore.rs: add missing subdir to directory index 2019-01-20 09:47:32 +01:00
Dietmar Maurer
0b05fd5830 api3/admin/datastore/catar.rs: implement download 2019-01-20 09:39:32 +01:00
Dietmar Maurer
7f0d67cf79 tools/wrapped_reader_stream.rs: new helper class 
Will use that to download catar files.
 2019-01-20 09:38:28 +01:00
Dietmar Maurer
6a4c091616 api3/admin/datastore/catar.rs: start impl. download 2019-01-19 17:27:02 +01:00
Dietmar Maurer
50cfb695ae api/router.rs: rename ApiUploadMethod to ApiAsyncMethod 
We can use this for uploads and downloads ...
 2019-01-19 16:42:43 +01:00
Dietmar Maurer
379ea0edb6 Revert "/api/schema.rs: implement Schema::Option" 
This reverts commit 0a35462c1e.

I am not sure this add much value, and the old approach needs
less memory. If we really need single optional values, we can still
implement such Option while keeping the hash based approach...
 2019-01-19 12:53:32 +01:00
Dietmar Maurer
80f069656d Revert "api/schema.rs: avoid Option(Option( nesting" 
This reverts commit ffdac1af01.
 2019-01-19 12:52:51 +01:00
Dietmar Maurer
ca3d861513 remove unused type PropertyMap 2019-01-18 17:54:06 +01:00
Dietmar Maurer
ffdac1af01 api/schema.rs: avoid Option(Option( nesting 2019-01-18 17:49:19 +01:00
Dietmar Maurer
0a35462c1e /api/schema.rs: implement Schema::Option 2019-01-18 17:40:37 +01:00
Dietmar Maurer
728797d0c1 reduce compiler warnings 2019-01-18 16:50:15 +01:00
Dietmar Maurer
383e857749 move complete_file_name() helper into tools.rs 2019-01-18 13:42:52 +01:00
Dietmar Maurer
73e844dbfe bin/proxmox-backup-client.rs: improve file name completion 2019-01-18 13:40:05 +01:00
Dietmar Maurer
95cea65b04 backup/datastore.rs: list all index files using walkdir crate 2019-01-18 12:24:58 +01:00
Dietmar Maurer
ff3d3100d4 backup/datastore.rs: try to create useful directory layout 
store/type/ID/TIME/xxx.yyy
 2019-01-18 12:01:37 +01:00
Dietmar Maurer
0d38dcb43c cleanup nodename() 2019-01-18 10:13:45 +01:00
Dietmar Maurer
74a69302a7 tools.rs: new helper nodename() 2019-01-18 09:58:15 +01:00
Dietmar Maurer
5a778d92b3 api3/admin/datastore.rs: rename parameter "name" to "store" 2019-01-18 08:33:11 +01:00
Dietmar Maurer
244d9b17a8 bin/proxmox-backup-client.rs: implement file name completions - first try ... 2019-01-17 14:24:20 +01:00
Dietmar Maurer
83bdac1e3b api3/admin/datastore/upload_catar.rs: verify content type ("application/x-proxmox-backup-catar") 2019-01-17 12:43:29 +01:00
Dietmar Maurer
06aeb76a5c client/catar_backup_stream.rs: remove debug code 2019-01-17 12:16:36 +01:00
Dietmar Maurer
0fe5d605e7 tools.rs: new helper required_string_param() 
To extract parameters from Value.
 2019-01-17 12:14:02 +01:00
Dietmar Maurer
0ee0ad5bf3 api/router.rs: return Result in upload handler 2019-01-17 12:03:38 +01:00
Dietmar Maurer
23bb8780d4 proxmox-backup-client: implement upload to server 2019-01-17 11:38:22 +01:00
Dietmar Maurer
597641fdd5 client/http_client.rs: new helper class 2019-01-17 11:38:03 +01:00
Dietmar Maurer
264f52cf70 api3/admin/datastore.rs: change api patch for upload_catar 2019-01-17 09:16:50 +01:00
Wolfgang Bumiller
37b98249f4 upload: don't return NotReady 
We need to kick (poll) the stream's next element, so simply
reiterate, the try_read!() will then return NotReady from
the next stream item for us.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-16 15:17:26 +01:00
Wolfgang Bumiller
1dcd52e5df cleanup previous commit... 
This kind of notation is weird...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-16 14:19:13 +01:00
Wolfgang Bumiller
6a50b77e72 listen on :: instead of 0.0.0.0 
Since this supports both ipv6 & ipv4 unless v6only is
explicitly enabled.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-16 14:18:16 +01:00
Dietmar Maurer
cf16af2ab3 server/rest.rs: correctly pass query/url parameters 2019-01-16 13:58:36 +01:00
Dietmar Maurer
2085142ed4 reduce compiler warnings 2019-01-16 10:19:49 +01:00
Dietmar Maurer
e8edbbd49c client/catar_backup_stream.rs: new helper for catar uploads to server 2019-01-16 10:15:39 +01:00
Dietmar Maurer
150f1bd8f6 backup/datastore.rs: use Arc for chunk_store references 2019-01-15 12:36:16 +01:00
Dietmar Maurer
1629d2ad7b api3/admin/datastore/upload_catar.rs: implement upload future 2019-01-15 11:38:26 +01:00
Dietmar Maurer
7e21da6e23 api/router.rs: allow different types of api methods 
Added a prototype for file/backup uploads.
 2019-01-14 12:26:04 +01:00
Dietmar Maurer
b41d1aab8c catar: define all feature flags 2019-01-12 16:49:02 +01:00
Dietmar Maurer
4c0fd4871e catar/encoder.rs: allow to pass list of devices 
We can use this to implement the --one-file-system like options. This
is a bit more convenient, because this way we can select to include
multiple devices (which is required by container backups).
 2019-01-12 15:43:20 +01:00
Dietmar Maurer
a8a0495612 catar/encoder.rs: disable backup of virtual file systems 2019-01-12 11:56:53 +01:00
Dietmar Maurer
d7bfac8605 catar/encoder.rs: only test fat attribute on MSDOS and FUSE file systems 2019-01-12 10:28:26 +01:00
Dietmar Maurer
13d98013b2 /catar/encoder.rs: do not store data from virtual file systems 2019-01-12 10:20:08 +01:00
Dietmar Maurer
80881f60ef catar/encoder.rs: add helpers to test file system type 2019-01-12 09:18:21 +01:00
Dietmar Maurer
1c4804cf67 catar/encoder.rs: pass magic fs number to encode_dir 2019-01-12 08:51:44 +01:00
Dietmar Maurer
dce0ebd505 catar/encoder.rs: use O_DIRECTORY to open dirs 2019-01-11 16:53:17 +01:00
Dietmar Maurer
490683ec93 catar: encode/decode sockets and fifos 2019-01-11 13:26:05 +01:00
Dietmar Maurer
20e2043a08 catar/decoder.rs: corrently pass mode to mknodat 2019-01-11 13:12:55 +01:00
Dietmar Maurer
a7e3713122 catar: encode/decode devices 2019-01-11 12:22:00 +01:00
Dietmar Maurer
1bdb3130f7 catar/format_definition.rs: add more feature flags 2019-01-11 10:44:57 +01:00
Dietmar Maurer
5c76c2f331 catar/encoder.rs: store fat file system attributes 2019-01-11 10:18:22 +01:00
Dietmar Maurer
8c1dfa6c72 catar/encoder.rs: cleanups 2019-01-11 10:01:51 +01:00
Dietmar Maurer
4f6892eb74 catar/encoder.rs: encode linux fs attributes, cleanup encoder 
Try to avoid duplicate stat calls (assume file tree is stable during backup).
 2019-01-11 09:20:10 +01:00
Dietmar Maurer
ddbdf80d51 avoid compiler warnings 2019-01-11 08:41:33 +01:00
Dietmar Maurer
373ef4a504 backup/archive_index.rs: correctly check for errors 2019-01-10 11:19:54 +01:00
Dietmar Maurer
23f68e53b4 catar/decoder.rs: restore mode, uid, gid and mtime 2019-01-10 09:02:50 +01:00
Dietmar Maurer
25f60394d2 catar/decoder.rs: avoid slashes in file names 2019-01-10 09:02:50 +01:00
Dietmar Maurer
9b1bb5a277 catar/decoder.rs: impl restore dirs, files, and symlinks 2019-01-10 09:01:36 +01:00
Wolfgang Bumiller
1ed86a0bae rest: rename utf-8-checked 'bytes' to 'utf8' 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-08 14:22:43 +01:00
Wolfgang Bumiller
9fec7ae593 rest: don't copy the body 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-08 14:21:54 +01:00
Wolfgang Bumiller
d9fbcc4631 tools/timer: fixup doc formatting 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-08 14:11:10 +01:00
Dietmar Maurer
676bf71ad4 catar/decoder.rs: restore_sequential - read payload 2019-01-08 09:26:07 +01:00
Dietmar Maurer
c6a0d3fc6b catar/decoder.rs: improve restore_sequential 2019-01-07 19:18:59 +01:00
Dietmar Maurer
b6ebfb8dab catar/decoder.rs: start implementing sequential decoder, cleanups 
And use [derive(Endian)] for endian conversion.
 2019-01-07 19:07:03 +01:00
Dietmar Maurer
02c7d8e522 catar: implement create 2019-01-07 13:25:41 +01:00
Dietmar Maurer
e86c49247e bin/catar.rs: add list command 2019-01-06 17:42:23 +01:00
Dietmar Maurer
3d8c24ecf8 catar/decoder.rs: start implementation 2019-01-06 17:27:57 +01:00
Dietmar Maurer
4b864ad436 format_definition.rs: remove wrong docs 2019-01-06 17:27:22 +01:00
Dietmar Maurer
4624fe29cc impl Read for BufferedArchiveReader 2019-01-06 10:04:45 +01:00
Dietmar Maurer
343370504b impl Seek for BufferedArchiveReader 2019-01-06 09:35:39 +01:00
Dietmar Maurer
318564ac03 buffered_read: return empty buffer on eof 2019-01-06 09:17:28 +01:00
Dietmar Maurer
0a72e26704 define and use BufferedReader trait 2019-01-05 17:28:20 +01:00
Dietmar Maurer
1006acd626 remove dead code 2019-01-05 16:55:21 +01:00
Dietmar Maurer
51b499db74 tools.rs: improve docs 2019-01-05 16:53:28 +01:00
Dietmar Maurer
39c6bd86cc backup/archive_index.rs: implement BufferedArchiveReader 
Implement relativly fast random read using binary search.
 2019-01-05 14:47:56 +01:00
Dietmar Maurer
060c4811a0 backup/chunk_store.rs: resize buffer by powers of two 2019-01-05 11:34:49 +01:00
Dietmar Maurer
9409255aae catar/encoder.rs: code cleanup 
Note: Rust Vec resize uses len(), not capacity()!
 2019-01-04 17:23:01 +01:00
Dietmar Maurer
df9973e8f5 backup/chunk_store.rs: avoid allocation in read_chunk 2019-01-04 17:16:56 +01:00
Dietmar Maurer
7394ca3e95 white space cleanups 2019-01-04 12:51:43 +01:00
Dietmar Maurer
96df2fb44b backup/chunk_store.rs: implement read_chunk() 2019-01-04 12:50:54 +01:00
Dietmar Maurer
691c89a0fb proxmox-backup-manager: add garbage collection cli 2019-01-04 11:34:32 +01:00
Dietmar Maurer
07b4694a33 cleanup crate names 2019-01-04 10:49:52 +01:00
Dietmar Maurer
4bdba5f4f1 rename backup-client to proxmox-backup-client 2019-01-04 10:41:37 +01:00
Dietmar Maurer
a689f6df0e rename pbs to proxmox-backup-manager 2019-01-04 10:39:32 +01:00
Dietmar Maurer
bc616633d5 backup/chunk_store.rs: only use one directory level 2019-01-04 10:35:22 +01:00
Dietmar Maurer
030cc2a903 move admin API to /admin/ subfolder 2019-01-04 10:32:27 +01:00
Dietmar Maurer
b663789bdb backup/archive_index.rs: pass correct size to unmap 2019-01-04 09:28:41 +01:00
Dietmar Maurer
48d0d35688 backup/image_index.rs: use little endian 2019-01-04 08:45:45 +01:00
Wolfgang Bumiller
4968bc3ac0 cli::command: wrap usage errors in a UsageError 
So we can distinguish them and show usage output
conditionally.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2019-01-03 16:08:51 +01:00
Dietmar Maurer
c7fa487250 catar/encoder.rs: make sure we write all data with write_all() 2019-01-03 15:47:32 +01:00
Dietmar Maurer
050f8a6355 chunker.rs: make sure chunk_size_avg is a power of two 2019-01-03 15:10:54 +01:00
Dietmar Maurer
ea4ea34baf improve chunker speed (avoid modulo) 2019-01-03 14:35:19 +01:00
Dietmar Maurer
3329ae8c2e add benchmark for chunker 
We get about 300MB/s - so this is still not fast enough

Note: This is quite slow with debug target, so please compile with --release
 2019-01-02 19:18:53 +01:00
Dietmar Maurer
0b8e75ed16 image_index.rs: verify file size 2019-01-02 18:14:02 +01:00
Dietmar Maurer
77703d95aa implement garbage collection for .aidx files 2019-01-02 14:27:04 +01:00
Dietmar Maurer
a360f6fa2d image_index.rs: verify header magic/version 2019-01-02 13:13:13 +01:00
Dietmar Maurer
44b3f62b42 fix typo 2019-01-02 12:56:04 +01:00
Dietmar Maurer
594fa52016 disable debug code 2019-01-02 12:55:18 +01:00
Dietmar Maurer
5032b57b44 archive_index.rs: correctly write .aidx file 2019-01-02 12:54:40 +01:00
Dietmar Maurer
580dc84bac image_index.rs: print stats 2019-01-02 12:53:49 +01:00
Dietmar Maurer
5e7a09be0d src/backup/archive_index.rs: use close() instead of flush() 
Also pass a reference to the encoder.
 2019-01-02 11:02:56 +01:00
Dietmar Maurer
94a882e900 src/backup/archive_index.rs: implement flush() 2018-12-31 18:01:07 +01:00
Dietmar Maurer
0433db1949 src/backup/archive_index.rs: first try 2018-12-31 17:30:08 +01:00
Dietmar Maurer
cbdd8c54ae create backup mod in backup.rs, improve docu 2018-12-31 16:08:04 +01:00
Dietmar Maurer
cb4412b18e move chunker.rs to backup/ 2018-12-31 15:46:16 +01:00
Dietmar Maurer
1c287cb19a rc/catar/chunker.rs: first try 2018-12-31 13:01:06 +01:00
Dietmar Maurer
bf205f94c5 avoid compiler warnings 2018-12-31 10:54:25 +01:00
Dietmar Maurer
1b0dc9f680 src/catar/encoder.rs: correctly sort goodbye items by hash key 2018-12-31 10:11:28 +01:00
Dietmar Maurer
c60d34bdbf new catar binary 
currently used for debugging
 2018-12-30 18:02:46 +01:00
Dietmar Maurer
e75eac73ca fix doc syntax 2018-12-30 17:43:53 +01:00
Dietmar Maurer
0866748de6 catar cleanups ... 2018-12-30 17:32:52 +01:00
Dietmar Maurer
6cd28d200e binary_search_tree.rs: fix docs 2018-12-30 15:34:43 +01:00
Dietmar Maurer
389e562524 src/catar/encoder.rs: auto-resize file_copy_buffer, limit number of dirs 2018-12-30 14:09:59 +01:00
Dietmar Maurer
4fa71e0573 improve catar docs 2018-12-30 13:47:27 +01:00
Dietmar Maurer
48147efd12 add a comment about posible improvements 2018-12-29 19:43:25 +01:00
Dietmar Maurer
0b78833d8e src/catar/binary_search_tree.rs: add regression tests and fix one bug 2018-12-29 18:32:03 +01:00
Dietmar Maurer
b17d7149d2 src/catar/binary_search_tree.rs: improve docu 2018-12-29 17:38:50 +01:00
Dietmar Maurer
985567fb45 src/catar/encoder.rs: cleanup, factor out write_goodbye_table 2018-12-29 17:26:32 +01:00
Dietmar Maurer
95bd5dfec7 src/catar/encoder.rs: correctly sort goodby items 2018-12-29 17:00:48 +01:00
Dietmar Maurer
46b6fbd6ae use external crate siphasher 
Because std:#️⃣:SipHasher is deprecated.
 2018-12-28 19:50:07 +01:00
Dietmar Maurer
f0f3029e26 src/catar/encoder.rs: fix bug in file copy 2018-12-28 15:02:42 +01:00
Dietmar Maurer
d05f93215e src/catar/encoder.rs: fix symlinks 2018-12-28 15:02:42 +01:00
Wolfgang Bumiller
7ebb173352 getopt: cleanup: add trailing commas 
It is customary in rust to always use trailing commas. (also
suggested by rustfmt)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
2767c5d39b getopt: cleanup: don't condense 'if' statements too much 
In a language which enforces curly braces this looks weird
and rustfmt doesn't like it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:27:38 +01:00
Wolfgang Bumiller
7d95c10da0 getopt: whitespace cleanup 
to make rustfmt happy

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:27:38 +01:00
Dietmar Maurer
a0cc09b5f0 src/catar/encoder.rs: fix binary format, write goodby table 
We still ned to sort the table (BST) ...
 2018-12-28 14:27:00 +01:00
Wolfgang Bumiller
03fb895197 getopt: condense nested match to reduce indentation 
The `match value` statement is the only thing covering the
entire RawArgument::Option case. `rustfmt` suggests this
more condensed way of writing this case.

See the `git diff -w` of this patch:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 9755af2..4db4579 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,8 +72,7 @@ pub fn parse_arguments<T: AsRef<str>>(
|             RawArgument::Separator => {
|                 break;
|             }
|-            RawArgument::Option { name, value } => {
|-                match value {
|+            RawArgument::Option { name, value } => match value {
|                 None => {
|                     let mut want_bool = false;
|                     let mut can_default = false;
|@@ -125,7 +124,6 @@ pub fn parse_arguments<T: AsRef<str>>(
|                     data.push((name, v));
|                 }
|             }
|-            }
|             RawArgument::Argument { value } => {
|                 rest.push(value);
|             }

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:16:58 +01:00
Wolfgang Bumiller
4d0ea9978b getopt: let parse_arguments() take a slice of AsRef<str> 
We don't need the content to be owned strings, and we don't
need the list to be a Vec, since we only care about being
able to iterate through and copy out portions of the strings
we need, so take an &[T] where T: AsRef<str>.

This avoids .iter().map(to_string).collect() before calling
parse_arguments().

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 14:05:05 +01:00
Wolfgang Bumiller
2482c095b1 getopt: remove skip logic 
The 'skip' variable was set to indicate that the "rest of
the args" is to be copied into the 'rest' vec. We can do
this directly and avoid the 'if' case in the loop
altogether.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 13:59:44 +01:00
Wolfgang Bumiller
99da3a073d getopt: indentation reduction 
See the `git diff -w` output:
|diff --git a/src/getopts.rs b/src/getopts.rs
|index 6548a01..517cc37 100644
|--- a/src/getopts.rs
|+++ b/src/getopts.rs
|@@ -72,7 +72,10 @@ pub fn parse_arguments(
|     while pos < args.len() {
|         if skip {
|             rest.push(args[pos].clone());
|-        } else {
|+            pos += 1;
|+            continue;
|+        }
|+
|         match parse_argument(&args[pos]) {
|             RawArgument::Separator => {
|                 skip = true;
|@@ -135,7 +138,6 @@ pub fn parse_arguments(
|                 rest.push(value);
|             }
|         }
|-        }
|
|         pos += 1;
|     }

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 13:54:20 +01:00
Wolfgang Bumiller
ad8a98f7e4 parse_arguments: work with utf8 bytes and reduce indentation 
We are only caring about '=' and '-' which are single-byte
codepoints, so there's no reason not to work on a byte
slice.

Also, some indentation reduction:

Transform
    if (a) {
        ...
        return A;
    }
    return B;
into
    if (!a)
        return B;
    return A;

and
    if (a)
        if (b)
            foo()
into
    if (a && b)
        return;

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-28 13:47:29 +01:00
Dietmar Maurer
248c17af39 src/catar/encoder.rs: write filenames 2018-12-28 11:48:47 +01:00
Dietmar Maurer
2e4ae0e239 src/catar/encoder.rs: write file data 2018-12-28 10:44:12 +01:00
Dietmar Maurer
3192ae968c src/catar/encoder.rs: write entry and symlink data 2018-12-28 09:55:26 +01:00
Dietmar Maurer
50ea43962e make map_struct functions public 2018-12-28 08:04:46 +01:00
Dietmar Maurer
d2b03f2397 catar/encoder.rs: improve error handling 2018-12-28 07:45:15 +01:00
Dietmar Maurer
0ff559990c cleanups, avoid compiler warnings 2018-12-28 07:14:12 +01:00
Dietmar Maurer
45281d4927 catar/encoder: detect symlink and regular files 2018-12-27 14:24:31 +01:00
Dietmar Maurer
fb8365b79f catar/encoder.rs: first try 2018-12-27 13:15:47 +01:00
Wolfgang Bumiller
fba3437f1e timer: setup_timeout_handler cannot fail 
and if it does it panics anyway

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-27 10:36:28 +01:00
Dietmar Maurer
bcd879cfb3 backup-client: allow to specify directories (prepare for catar backup) 2018-12-27 10:11:11 +01:00
Dietmar Maurer
b62b6cadf4 start implementing catar 2018-12-27 09:22:23 +01:00
Dietmar Maurer
dc3de618ed new helper map_struct and map_struct_mut 2018-12-27 09:20:17 +01:00
Dietmar Maurer
a198d74fc0 cleanup: reduce compiler warnings 2018-12-25 13:29:27 +01:00
Dietmar Maurer
991de6007b remove outdated comment 2018-12-25 13:24:50 +01:00
Dietmar Maurer
7ee2aa1b94 touch_chunk: use libc::lutimensat 2018-12-25 12:27:25 +01:00
Dietmar Maurer
2c32fdde86 move lookup_datastore() to backup/datastore.rs 2018-12-22 17:37:25 +01:00
Dietmar Maurer
64e53b2835 gather usage statistics during garbage collection 2018-12-22 16:58:16 +01:00
Dietmar Maurer
28b96b56e1 open_file_locked: improve error message 2018-12-22 15:59:55 +01:00
Dietmar Maurer
176e4af964 sweep_used_chunks: print percentage 2018-12-22 15:39:05 +01:00
Dietmar Maurer
7b2b40a893 use openssl for faster hashing 2018-12-22 14:31:59 +01:00
Dietmar Maurer
e95950e40a chunk_store: reduce number of directories 
Else, scans everything takes too long ...
 2018-12-22 14:04:05 +01:00
Dietmar Maurer
15e9b4ed60 src/api3/datastore.rs: new file 2018-12-21 13:38:41 +01:00
Dietmar Maurer
3c140f60cd use return value (avoid compiler warning) 2018-12-21 12:44:20 +01:00
Dietmar Maurer
03e4753d8e fix mutability for chunk store 2018-12-21 12:15:26 +01:00
Dietmar Maurer
2d9d143a8f backup-client: add optional chunk-size parameter 2018-12-21 11:18:08 +01:00
Dietmar Maurer
f0819fe5a5 backup-client: do not start garbage collection after backup 2018-12-21 09:24:55 +01:00
Dietmar Maurer
d62e6e2264 backup-client: use 1M chunks, make chunk_size configurable 2018-12-21 08:36:57 +01:00
Dietmar Maurer
c34eb16651 backup-clinet: specify target file name 2018-12-20 14:09:31 +01:00
Dietmar Maurer
c2d9b3914a add test code to access static global state 2018-12-20 14:03:42 +01:00
Dietmar Maurer
fe0e04c69c use proxmox-backup as package name 2018-12-20 10:45:14 +01:00
Dietmar Maurer
f7dd683b46 fix tests 2018-12-19 13:54:22 +01:00
Dietmar Maurer
277fc5a3ce try to improve error messages 2018-12-19 13:40:26 +01:00
Dietmar Maurer
eae8aa3aa9 cleanups: avoid compiler warnings 2018-12-19 12:49:23 +01:00
Dietmar Maurer
6c20a13d3c cleanup - avoid too much indentation 2018-12-19 12:40:26 +01:00
Dietmar Maurer
1c43c56b06 cleanup 2018-12-19 12:30:37 +01:00
Dietmar Maurer
2bf5f6b220 ChunkStore::sweep_used_chunks - use openat to speedup directory walks 2018-12-19 12:20:01 +01:00
Wolfgang Bumiller
1628a4c731 use timers with a signal for file locking 
* rename lock_file -> open_file_locked,
* add lock_file as a function working on already-opened
  files
* change timeout types to std::time::Duration

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-19 11:23:00 +01:00
Wolfgang Bumiller
8cf6e764ad add tools::timer module 
This wraps POSIX timers via timer_create(2).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2018-12-19 11:23:00 +01:00
Dietmar Maurer
365bb90f17 move lock_file to tools.rs 2018-12-19 10:02:24 +01:00
Dietmar Maurer
08481a0b60 impl sweep_used_chunks, first try 2018-12-19 09:51:33 +01:00
Dietmar Maurer
6ea3a0b7fc move sweep_used_chunks to chunk_store 2018-12-18 11:18:55 +01:00
Dietmar Maurer
3d5c11e5d2 start impl garbage collection 2018-12-18 11:06:03 +01:00
Dietmar Maurer
529de6c7a3 start implementing DataStore 2018-12-17 13:00:39 +01:00
Dietmar Maurer
4818c8b6f7 start implementing ImageIndexReader 2018-12-16 14:44:44 +01:00
Dietmar Maurer
a27a3ee4ba avoid compiler warnings 2018-12-16 13:57:59 +01:00
Dietmar Maurer
af3e7d7551 fix chunk store file paths 2018-12-16 13:52:16 +01:00
Dietmar Maurer
0cd9d4206f improve error messages 2018-12-16 13:43:19 +01:00
Dietmar Maurer
4fbb72a8b4 image_index.rs: implement atomic write/rename 2018-12-16 13:39:21 +01:00
Dietmar Maurer
798881a68a rebane ImageIndex to ImageIndexWriter 2018-12-16 11:51:16 +01:00
Dietmar Maurer
5e5b7f1c2d use u64 to store time 2018-12-16 11:48:03 +01:00
Dietmar Maurer
d13e3745b3 image_index.rs: write header 2018-12-15 17:05:49 +01:00
Dietmar Maurer
606ce64bb0 image_index.rs:write idx of chunks (use mmap) 2018-12-15 14:51:05 +01:00
Dietmar Maurer
f8838fe913 backup-client: add bash completion for datastore names 2018-12-15 11:24:39 +01:00
Dietmar Maurer
5f0c2d5652 remove debug println 2018-12-15 11:16:27 +01:00
Dietmar Maurer
43eeef28d1 move file_chunker to tools 2018-12-15 11:14:41 +01:00
Dietmar Maurer
48e8689505 file_chunker: use Read trait 2018-12-15 11:08:49 +01:00
Dietmar Maurer
fc8b19368a backup-client: pass chunk_size as parameter to file_chunker 2018-12-15 10:58:48 +01:00