e23f586344
src/bin/proxmox-backup-client.rs - status: improve alignment
2020-02-27 11:38:09 +01:00
be2425ff85
src/bin/proxmox-backup-client.rs - status: do not print header
2020-02-27 11:31:05 +01:00
f24fc1166b
src/bin/proxmox-backup-client.rs - list_snapshots: use format_and_print_result_full()
...
Depend on proxmox v0.1.14.
2020-02-27 11:28:44 +01:00
af934f8cf6
tools::lru_cache: Improve access() and insert() by using HashMap::entry().
...
entry() allows to lookup the position where and entry belongs and update/insert
it in the HashMap more efficiently than get_mut() and insert().
Details: https://gankra.github.io/blah/hashbrown-insert/
In addition, use the struct LinkedList and remove the outdated code.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:56:25 +01:00
e3ab9a383c
tools::lru_cache: Separate LinkedList part of LruCache into own struct.
...
In order to keep the separation more aparent and avoid mut borrow conflics.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:55:48 +01:00
75c2ee7bab
tools::lru_cache: Make key generic.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:55:16 +01:00
03f779c6f5
catalog: shell: Improve output of stat command.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:53:17 +01:00
ea5f547fdc
src/bin/proxmox-backup-client.rs - list_snapshot_files: use format_and_print_result_full()
2020-02-26 13:49:47 +01:00
390c5bdde4
src/bin/proxmox-backup-client.rs - status: use format_and_print_result_full
2020-02-26 13:29:00 +01:00
03ac286c80
src/bin/proxmox-backup-manager.rs: use crate::tools::runtime::block_on()
2020-02-26 11:37:38 +01:00
dd04383bb8
src/tools/runtime.rs: call OPENSSL_thread_stop to avoid race with openssl cleanup handlers
2020-02-26 10:38:55 +01:00
ca2dbb8af1
src/client/backup_writer.rs: also print archive size
2020-02-25 14:12:29 +01:00
0bcbb5434e
cleanup unused module
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-25 11:38:27 +01:00
495b238bc1
pxar: Change text of warning for files matched by exclude patterns.
...
In addition to the .pxarexclude files, glob match patterns can be passed to pxar
also via cli parameters.
Therefore the warning is rephrased to be more ambiguous.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-25 10:47:24 +01:00
6da73c823f
src/client/backup_writer.rs: prefix upload stats with archive name
2020-02-24 13:24:46 +01:00
e02c3d461f
src/client/backup_writer.rs: implement verbose options
2020-02-24 12:48:40 +01:00
ee5fe978e2
src/client/backup_writer.rs: reduce verbosity
2020-02-24 11:33:00 +01:00
8ce49a76da
src/backup/backup_info.rs: fix SNAPSHOT_PATH_REGEX and GROUP_PATH_REGEX
2020-02-18 13:16:35 +01:00
9e003074cb
runtime: fix blocking strategy:
...
- do not "double"-block_in_place() (it may not be nested)
- do not call block_in_place() in non-worker threads
is_in_tokio() isn't sufficient, we need to actually know
that we're in a worker-thread, so we do this by remembering
that we're blocking.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-17 09:48:22 +01:00
1283d58ca9
runtime: remove IN_TOKIO thread local variable
...
tokio now has Handle::try_current() allowing us to
generally check for a tokio runtime even if spawned by
someone else
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-13 08:56:38 +01:00
a6e3da987c
src/client/http_client.rs: do not store fingerprints passed via options in cache
2020-02-11 11:54:43 +01:00
1bd6f32b43
src/client/http_client.rs: always convert fingerprint to lowercase
2020-02-11 11:42:03 +01:00
d1c657276a
src/client/http_client.rs: remove useless password_env
2020-02-11 11:42:03 +01:00
a05c0c6ff6
src/bin/proxmox-backup-client.rs: allows to pass fingerprint via env vars
2020-02-11 11:42:03 +01:00
b69b8af26b
s/pbsbanner/proxmox-backup-banner/ + move to /usr/lib
...
We want to avoid pbs if possible and also avoid placing internal
binaries, not intended for human direct use, in /bin or /sbin paths.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-10 14:43:28 +01:00
e2b5e75a3e
pbsbanner: use match and also return error
...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-07 18:30:55 +01:00
274b0c7bb7
add login banner service
...
Modeled after the one from PVE, but using rust instead of perl for
resolving the nodename and writing to /etc/issue
Behavior differs a bit. We write all non-loopback addresses to this
file, as the gui accepts connections from them all, so limiting it to
the first one is not really sensible.
Further an error to resolve, or only getting loopback addresses won't
write out an empty /etc/issue file, but a note about the error at the
place where the address would be displayed.
Named it "pbsbanner", not "proxmox-backup-banner" as it's rather an
internal tool anyway and mirrors pvebanner, pmgbanner
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-06 22:19:22 +01:00
ba050e3788
catalog: shell: Allow two or more successive slashes in path.
...
Two or more successive slashes should be allowed and treated as a single slash.
We also do not treat two successive slashes at the beginning of a path any
different.
Details are found here:
https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap04.html#tag_04_11
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-05 09:32:29 +01:00
9ff9a00572
pxar::fuse: cleanup: Remove unused code.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 13:43:11 +01:00
3e69abef02
pxar::fuse: Refactor lookup in order to cache accessed entries.
...
Context::find_goodbye_entry() is removed and incorporated into the lookup
callback in order to take advantage of the entry_cache and since it is only used
inside this callback.
All entries read on lookup are cached.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 13:43:06 +01:00
63698e720c
pxar: Improve read performance for fuse.
...
By storing the payload start offset in the `DirectoryEntry` and passing this
information to `Decoder::read()`, the payload can be read directly and a repeated
re-reading of the entry information is avoided.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 12:08:03 +01:00
032d3ad80f
src/bin/proxmox-backup-client.rs: do not use api macro for prune
...
So that we can use add_common_prune_prameters!() macro again.
2020-01-31 10:31:36 +01:00
b8a192e3b0
src/config/remote.rs: do not serialize empty option
2020-01-31 09:09:24 +01:00
c43b36bcb6
src/section_config.rs - write: improve error message
2020-01-31 09:08:42 +01:00
501f4fa220
depend on proxmox 0.1.13, use new tty helpers from there
2020-01-31 08:16:00 +01:00
74616f63b0
proxmox-backup-client: add missing prune options to api definition.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-31 07:01:56 +01:00
93fbb4ef0a
update proxmox dependency to 0.1.12
2020-01-31 06:59:17 +01:00
43592f290d
src/bin/proxmox-backup-manager.rs: remove stale debug comment
2020-01-30 17:59:13 +01:00
9e165b5cad
src/bin/proxmox-backup-manager.rs - list remotes: do not use client, call directly
2020-01-30 17:57:37 +01:00
9894469e89
src/bin/proxmox-backup-manager.rs: use new texdt table formatter
2020-01-30 13:31:20 +01:00
8528fce874
src/api2/node/tasks.rs; make list_tasks public
2020-01-30 13:31:20 +01:00
70fa1aa3fc
fixup
2020-01-30 13:31:20 +01:00
5eeea607ae
src/section_config.rs - convert_to_array: add option to skip properties
2020-01-30 13:26:46 +01:00
93a9eb284e
pxar::fuse: Do not return fcaps value on listxattr.
...
listxattr must only return the name list, no extended attribute values.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:09 +01:00
5d2158e89a
pxar::fuse: return ACL names in listxattr if present
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:03 +01:00
72677fb0a5
pxar::fuse: add support to read ACLs.
...
ACLs are stored separately in the pxar archive. This implements the functionality
needed to read the ACLs and return them as extended attributes in the getxattr
callback.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:00 +01:00
0502ce6da3
tools::acl: Add helpers to create extended attribute buffer.
...
This helpers are used to construct the extended attributes values from
the ACLs stored in the pxar archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:34:57 +01:00
bcf0d452c9
pxar: move acl helper functions to src/tools/acl.rs
...
They are not only needed by the pxar::sequential_decoder but also for the fuse
xattr impl, so it makes more sense to have them there.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:34:54 +01:00
6f763ae673
tools: LruCache: add dropcheck marker
...
See https://doc.rust-lang.org/nomicon/phantom-data.html
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:18:27 +01:00
99384f7933
src/api2/types.rs: define and use struct TaskListItem
2020-01-28 11:23:08 +01:00
2c4b303c62
src/api2/node/tasks.rs: use api macro for list_tasks
2020-01-28 11:23:08 +01:00
3e56c4abd7
pxar::fuse: Introduce LRU caching for goodbye table and directory entries.
...
The goodbye table of directory entries is cached in a LRU cache to speed up
subsequent accesses.
This is especially important for directories with many entries, as then the
readdirplus callback is called repeatedly because of the limited reply buffer
size.
`DirectoryEntry`s are cached for subsequent access in their own LRU cache,
independent of the goodbye tables.
In order to avoid borrow conflicts, the `Context` provides a fn as_mut_refs
as well as a fn run_with_context_refs.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-28 10:06:53 +01:00
cc552754c8
pxar::fuse: refactor fn stat
...
By passing `&DirectoryEntry` to stat, the function interface is simplified.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-28 10:06:51 +01:00
d08bc483db
use proxmox 0.1.9 with new cli command helpers
2020-01-27 18:08:43 +01:00
99d863d770
src/client/http_client.rs: new password_env option
2020-01-27 10:42:37 +01:00
5030b7cea4
src/client/http_client.rs: new prefix options
2020-01-27 10:20:55 +01:00
5a74756c15
src/client/http_client.rs: implement fingerprint cache
2020-01-25 15:37:34 +01:00
d59dbeca1b
src/client/http_client.rs: new struct HttpClientOptions
2020-01-25 12:18:00 +01:00
6afbe1d846
src/config/remote.rs: add fingerprint
2020-01-25 09:49:45 +01:00
dcb8db66d9
src/api2/types.rs: add CERT_FINGERPRINT_SHA256_SCHEMA
2020-01-25 09:49:45 +01:00
90c815bfe2
src/bin/proxmox-backup-client.rs: fix another typo in property description
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 13:46:08 +01:00
5eee6d89b6
src/bin/proxmox-backup-client.rs: fix typo in property description
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 13:45:22 +01:00
04350b4c08
tools: add access method to LruCache
...
This will return a mutable reference just like get_mut, but on a cache miss
it will get and insert the missing value via the fetch method provided via the
Cacher trait.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 09:32:09 +01:00
4191018c79
src/api2/admin/datastore.rs - download_file: use BACKUP_ARCHIVE_NAME_SCHEMA for file-name
2020-01-23 13:49:30 +01:00
a92830dc39
src/api2/types.rs: define and use api type GarbageCollectionStatus
2020-01-23 13:40:12 +01:00
dfc58d47ad
src/api2/admin/datastore.rs: use api macro for start_garbage_collection
2020-01-23 13:01:21 +01:00
1dc117bbba
src/api2/types.rs: define and use struct StorageStatus
2020-01-23 12:51:30 +01:00
09b1f7b202
src/api2/types.rs: define struct BackupContent, and use it with list_snapshot_files
2020-01-23 11:17:38 +01:00
51534c8de9
src/backup/manifest.rs: check if manifest contains files
2020-01-23 11:16:12 +01:00
68a6a0ee03
src/api2/admin/datastore.rs: use api macro for delete_snapshot
2020-01-23 10:16:45 +01:00
6abce6c2bb
src/backup/datastore.rs: remove_backup_(group/dir) - return Error instead of io::Error
2020-01-23 10:14:46 +01:00
c425bdc9b2
src/api2/pull.rs: delete fanished snapshots
2020-01-23 09:58:57 +01:00
8a1d68c8b9
src/backup/datastore.rs: improve error messages
2020-01-23 09:58:14 +01:00
18cc66ee85
src/api2/pull.rs: use new Datastore::last_successful_backup helper
...
Do not fail if local group does not exists.
2020-01-22 15:06:28 +01:00
41b373eced
src/backup/datastore.rs: new helpers
...
last_successful_backup: Returns the time of the last successful backup
group_path: Returns the absolute path for a backup_group
snapshot_path: Returns the absolute path for a backup_dir
2020-01-22 15:05:47 +01:00
1ca037d81c
followup: commit all changes
...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-01-22 14:14:52 +01:00
44b65b4ec7
api/datastore: do not allow updating path for now
...
It's a bit dangerous as it points to all the saved backups, so they
would be seemingly lost after updating the path.
Follow our logic from other products, e.g. in PVE we do not allow to
update the backing path/location of a storage either for similar
reasons.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-01-22 13:55:37 +01:00
1685c2e341
tools: LRU cache to use for pxar FUSE implementation.
...
Implements a cache with least recently used cache replacement policy.
Internally the state is tracked by a HashMap (for fast access) and a doubly
linked list (for the access order).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-22 13:11:42 +01:00
f1d99e3f6a
src/bin/proxmox-backup-client.rs: use a std channel to write the catalog
2020-01-22 12:49:08 +01:00
02141b4d9b
src/client/pxar_backup_stream.rs: use a channel instead of a pipe
2020-01-22 11:37:16 +01:00
dcd033a53c
src/tools/std_channel_writer.rs: new StdChannelWriter helper class
2020-01-22 11:33:57 +01:00
9ea4bce444
bump proxmox crate to 0.1.7
...
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).
When we need to depend on "parts" of the crate later on
we'll just have to use features.
The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 13:48:37 +01:00
3f06d6fbbd
use proxmox_backup::tools::runtime::main to run async run_cli_command
2020-01-21 11:34:45 +01:00
077ac6f8b3
runtime: actually park on Pending...
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 10:37:16 +01:00
2d51569396
runtime: custom block_on implementation
...
futures::executor::block_on does not allow nesting, but we
need that...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 10:24:52 +01:00
650e052299
runtime: drop now not required Send and static restrictions
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 14:09:24 +01:00
d973aa827c
introduce new runtime tokio helpers
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 13:12:40 +01:00
aac9dbf635
pxar::fuse: remove open, opendir and releasedir callbacks
...
They are not needed as there is no internal state which needs to be tracked.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-20 12:56:21 +01:00
a8aff3535d
pxar: Include symlink target in DirectoryEntry
...
This allows to read the target path of a symbolic link in the
Decoder::read_directory_entry() function and stores it in the DirectoryEntry.
By this the Decoder::read_link() function becomes obsolete and is therefore
removed.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-20 12:56:10 +01:00
138910bcd4
src/client/pxar_backup_stream.rs: use std: 🧵 :Builder to name the thread
...
To make debugging easier.
2020-01-18 12:21:22 +01:00
4b4eba0b9e
src/api2/pull.rs: implement delete flag for vanished groups
2020-01-17 11:24:55 +01:00
11d89239c3
src/backup/backup_info.rs: new list_groups helper
2020-01-17 10:42:03 +01:00
b31c8019d7
src/api2/types.rs: define and use GroupListItem
2020-01-17 10:17:18 +01:00
2bbbade367
pxar: Refactor fuse and remove unused code.
...
By ambiguously using the Decoder::read_directory_entry() the code is simplified
and reading of the DirectoryEntry is concentrated into Context::run_in_context().
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
95c9460c4a
pxar::fuse: Always use start offset of entries as inode.
...
Previously it was disciminated based on the entry mode.
For directories, the inode was the offset of the corresponding
goodbye tail mark while for all others it was the offset of the filename.
By simply using the start offset as calculated from the corresponding
goodbye table entry (which yields the archive offset of the filename),
the code is simplified and the more ambiguous read_directory_entry()
function can be used.
The disatvantage of this approach is the need to keep track of the
start and end offsets for each entry, as the end offset is needed in
order to access the goodbye table of directory entries.
The root node still has to be treated special, as it's inode is 1 as per fuse
definition and it has no filename as per the pxar file format definition.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
6744440714
pxar::decoder: refactor Decoder::lookup()
...
Use Decoder::read_directory_entry() instead of Decoder::attributes() as this
already returns the needed DirectoryEntry.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
f357390c15
renamed: src/config/remotes.rs -> src/config/remote.rs
...
And use 'remote' instead of 'remotes' everywhere.
2020-01-16 14:32:06 +01:00
d2dd827877
src/backup/catalog.rs - SenderWriter: use tokio::task::block_in_place
...
Make sure we do not block the executor.
2020-01-16 14:24:15 +01:00
94609e2380
src/api2/pull.rs: use reference to remote configuration
2020-01-16 13:55:19 +01:00
625d19621f
src/client/remote_chunk_reader.rs: use tokio::task::block_in_place to avoid blocking
...
Else we block the current executor.
2020-01-16 13:55:19 +01:00
6a0dc4a5bc
bump proxmox crate version
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 11:40:23 +01:00
619495b25e
fix use of deprecated items
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 08:58:54 +01:00
18c0df4c92
import ordering cleanup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 08:51:52 +01:00
ef2d7f2f96
pxar::fuse: remove readdir implementation
...
By not implementing readdir but only readdirplus, the FUSE_CAP_READDIRPLUS flag
is set while the FUSE_CAP_READDIRPLUS_AUTO flag is not set.
Thereby the kernel will issue only readdirplus calls.
Documentation at:
https://libfuse.github.io/doxygen/fuse-3_88_80_2include_2fuse__common_8h.html#a9b90333ad08d0e1c2ed0134d9305ee87
As the expensive part for accessing and reading the attributes is seeking and
decoding each directory entry, it is usefull to force readdirplus calls.
By this a struct `EntryParam` is returned for each entry, therebye avoiding a
subsequent lookup call.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-15 14:44:53 +01:00
3be839c61c
src/bin/proxmox-backup-manager.rs: add completion function for update commands
2020-01-15 12:42:13 +01:00
002a191abf
src/api2/config: check digest
2020-01-15 12:27:05 +01:00
347834df25
src/api2/config: correctly lock files
2020-01-15 11:57:12 +01:00
1c79e1017f
services: fix indentation
...
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-15 10:55:04 +01:00
cbe01dc5c7
tools: add read_and_verify_password helper
...
add a helper to perform some basic checks on password prompts.
- verification (asks for a 2nd time)
- check length
also use the new helper where password input in tty is taken to reduce
duplicate code.
this helper should be used when creating keys, changing passphrases etc.
note: this helper can be extended later on to provide better checks for
password strength.
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-15 10:52:57 +01:00
ea5061979b
pxar::fuse: Avoid macro to generate filler function, use closure instead.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-15 10:25:16 +01:00
f3ec5dae7f
src/api2/config: improve api schema by adding return type
2020-01-14 14:55:58 +01:00
ddc526623f
src/bin/proxmox-backup-manager.rs: add datastore update cli
2020-01-14 14:47:26 +01:00
c5799e4043
src/api2/config/datastore.rs: improve api, implement update and read
2020-01-14 14:45:56 +01:00
08195ac823
src/api2/config/remotes.rs: improve api, implement update and read
2020-01-14 14:20:16 +01:00
16f04b9d79
src/section_config.rs: add lookup_json() helper
2020-01-14 14:19:22 +01:00
71805bb7ec
src/api2/config/remotes.rs: fix variable name -> s/remote/id/
2020-01-14 13:20:13 +01:00
d0187a51a9
src/section_config.rs - convert_to_array: optionally add digest
...
datastore::config() -> also return digest
remotes::config() -> also return digest
2020-01-14 12:57:03 +01:00
4566303b05
src/section_config.rs - write: improve error message
2020-01-14 12:02:46 +01:00
f51420ba1f
src/section_config.rs: do not allow control characters
2020-01-14 11:41:48 +01:00
7e7b781a18
src/api2/types.rs: also define PASSWORD_FORMAT and use it correctly
2020-01-14 11:32:02 +01:00
da4a15a351
src/api2/types.rs: define and use PASSWORD_REGEX
2020-01-14 11:22:42 +01:00
ff620a3d1a
src/api2/types.rs: add regression test for PROXMOX_USER_ID_SCHEMA
2020-01-13 17:26:43 +01:00
ae62c4fe59
src/api2/types.rs: fix regex patterns
2020-01-13 14:38:31 +01:00
163dc16c0b
src/api2/types.rs: define PROXMOX_AUTH_REALM_SCHEMA and PROXMOX_USER_ID_SCHEMA
...
And try to use nbew schemas with config api...
2020-01-13 14:18:19 +01:00
b25f313d66
src/api2/types.rs: define HOSTNAME_FORMAT, DNS_NAME_FORMAT and DNS_NAME_OR_IP_FORMAT
2020-01-13 13:10:52 +01:00
8aea35fcff
src/config/*: add #[serde(skip_serializing_if="Option::is_none")] to optinal comment
2020-01-13 12:14:14 +01:00
454c13edce
src/api2/types.rs: define SINGLE_LINE_COMMENT_SCHEMA
2020-01-13 12:02:13 +01:00
167971ed49
src/api2/types.rs: define REMOTE_ID_SCHEMA here
2020-01-13 11:47:07 +01:00
6de36b94b1
pxar::fuse: impl readdirplus callback
...
readdirplus returns the entries together with their `EntryParam`, so subsequent
lookups for each of the entries are avoided.
In order to reduce code duplication, the code for filling the reply buffer is
moved into a macro.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-13 11:17:34 +01:00
caa201d895
pxar::fuse: Get rid of broken cacheing
...
Remove the current caching of attrs and goodbye tables as it is broken anyway.
This will be replaced with a LRU cache.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-13 11:16:06 +01:00
6d20a29d73
fix typo for function name load_and_decrypt_key
...
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-13 11:14:47 +01:00
50af953e1b
src/section_config.rs: make set_data generic
2020-01-11 11:09:27 +01:00
688fbe07a1
cleanup config api, add remotes config cli interface
2020-01-11 10:42:09 +01:00
9e9bc6525e
src/config/datastore.rs: define DataStoreConfig using api macro
2020-01-11 09:18:42 +01:00
ca44172404
src/api2/config/remotes.rs: cleanup - avoid compiler warnings
2020-01-10 19:25:26 +01:00
c7257c5539
pxar::fuse: refactor HashMap to store child parent inode relation.
...
By moving the HashMap into `Context`, the use of lazy_static as well as the
additional Mutex can be avoided (`Context` is already guarded by a Mutex).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:58 +01:00
90fc97af6a
pxar::decoder::Decoder: include xattrs and payload size in `DirectoryEntry`.
...
By reading and including xattrs and payload size in struct `DirectoryEntry`,
the tuple of return types is avoided and the code is simpler.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:40 +01:00
6fc053ed85
pxar: encoder: limit number of max entries held at once in memory during archive creation.
...
Limit the total number of entries and therefore the approximate memory
consumption instead of doing this on a per directory basis as it was previously.
This makes more sense as it limits not only the width but also the depth of the
directory tree.
Further, instead of hardcoding this value, allow to pass this information as
additional optional parameter 'entires-max'.
By this, creation of the archive with directories containing a large number of
entries is possible.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:08 +01:00
141304d64e
src/api2/config/remotes.rs: new API to configure remotes
2020-01-10 13:28:15 +01:00
331b869d96
src/bin/proxmox-backup-manager.rs: implement shell completion for remote datastore name
2020-01-10 11:58:31 +01:00
eb506c830c
rename src/api2/sync.rs -> src/api2/pull.rs
...
Use "pull" instead of "sync", because this also indicates a
direction (like "sync from").
2020-01-10 11:15:16 +01:00
a81af92f9d
src/section_config.rs: implement generic lookup
2020-01-09 17:35:44 +01:00
0eb0e02451
src/bin/proxmox-backup-manager.rs: implement cli sync command
2020-01-09 14:52:29 +01:00
a575320657
src/config/remotes.rs: implement SectionConfig for remote hosts
2020-01-09 14:51:02 +01:00
ecbc62261c
pxar: encoder: warn on lacking read permissions instead of fail.
...
If during creation of the archive, files/dirs with lacking read permissions are
encountered, the user is displayed a warning and the archive is created without
including the file/dir.
Previously this resulted in an error and the archive creation failed.
In order to implement this also for the .pxarexclude files, the Error type of
MatchPattern::from_file() and MatchPattern::from_line() was adopted accordingly.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-09 10:41:19 +01:00
88cee60bad
pxar: encoder: fix incorrect goodbye table entries and update displayed path.
...
If nodes are excluded by feature flags, they must not appear in the goodbye table.
This is fixed by continuing with the next entry in the for loop.
Further the relative path buffer is now poped in order to correctly display the path.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-09 10:39:41 +01:00
ec906dcf19
src/pxar/encoder.rs: avoid use of macro and initialize Vec with known capacity.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-09 10:38:58 +01:00
71da3d6a1b
src/api2/types.rs: fix SnapshotListItem schema definition
2020-01-09 10:36:19 +01:00
de8ec04123
src/api2/sync.rs: implement remote sync
2020-01-08 14:03:52 +01:00
27c9affb23
src/bin/proxmox-backup-client.rs: cleanup - factor out api_datastore_latest_snapshot()
2020-01-07 15:18:36 +01:00
42af4b8f06
src/bin/proxmox-backup-client.rs: cleanup - factor out api_datastore_list_snapshots()
2020-01-07 15:07:57 +01:00
af9d4afc66
src/bin/proxmox-backup-client.rs: use SnapshotListItem
2020-01-07 13:25:25 +01:00
fc189b198c
src/api2/types.rs: define and use new api type SnapshotListItem
2020-01-07 12:52:52 +01:00
8f14e8fe4c
src/backup/backup_info.rs: implement last_successful_backup()
...
A backup is considered successful if there is a manifest file.
2020-01-06 11:35:22 +01:00
247a8ca5bb
src/backup/manifest.rs: impl TryFrom DataBlob
2020-01-05 16:28:18 +01:00
7759eef552
src/backup/datastore.rs: implement cleanup_backup_dir()
2020-01-05 15:15:12 +01:00
5e1d1bead1
src/client/backup_reader.rs: make compute_file_csum() public
2020-01-05 14:50:54 +01:00
3743dee6de
src/client/http_client.rs: allow access to AuthInfo fields
2020-01-05 10:31:19 +01:00
3758b398cd
src/backup/read_chunk.rs: add read_raw_chunk()
...
Allow to read chunk data without decoding.
2020-01-02 13:29:10 +01:00
2585a8a4e2
src/backup/chunk_store.rs: implement cond_touch_chunk()
...
This will be used by backup sync to test if a chunk already exists.
2020-01-02 13:26:28 +01:00
1cf5178ac5
src/backup/datastore.rs: new helper try_shared_chunk_store_lock()
2020-01-02 11:00:33 +01:00
1e8da0a789
src/backup/manifest.rs: new helper archive_type()
2019-12-31 15:23:41 +01:00
94f085fab5
src/bin/proxmox-backup-proxy.rs: remove unused use clause
2019-12-30 11:50:24 +01:00
22be470d90
src/bin/proxmox-backup-api.rs: create cert here
2019-12-27 13:44:21 +01:00
ca6119551d
src/client/http_client.rs - start_h2_connection: correctly return error during upgrade
2019-12-27 13:41:31 +01:00
a546a8a072
proxy/api: await running workers before shutdown
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-27 12:50:33 +01:00
f8fd5095d8
src/bin/proxmox-backup-proxy.rs: create self signed cert at startup.
2019-12-27 11:21:32 +01:00
afddffc7ec
proxmox-backup-manager.rs: remove wrong fixme
2019-12-27 10:17:06 +01:00
e739a8d8b4
src/bin/proxmox-backup-manager.rs: impl. cert info
2019-12-23 13:06:43 +01:00
550e0d8870
src/bin/proxmox-backup-manager.rs: add cli to generate proxy certificate
...
Without calling external openssl binary.
2019-12-22 17:35:06 +01:00
5e62d19cfb
fix create_datastore parameters:
...
comment was declared as optional but unwrap()ed, causing
panics when not using it
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-20 12:30:25 +01:00
645995634a
src/api2/config/datastore.rs - create: pass uid and gid instead of User
2019-12-20 09:23:58 +01:00
e67770d496
src/backup/chunk_store.rs - create: pass User instead of CreateOptions
2019-12-20 09:11:40 +01:00
bca9093520
api/ui: datastore: allow to set simple comment
...
for now forbid all control characters[0] in the comment value, the
section config writer cannot cope with newlines in the value, it
writes them out literally, allowing "injection" or breaking the whole
config.
In the webinterface use also a textfield, not a textarea.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-19 17:58:01 +01:00
afdcfb5bc9
let ChunkStore::create take CreateOptions
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-19 13:14:49 +01:00
47d4712152
src/bin/proxmox-backup-manager.rs: add task management cli
2019-12-19 11:27:36 +01:00
d6c4a1198a
src/bin/proxmox-backup-client.rs: use async fn
2019-12-19 11:20:59 +01:00
f74a03da1f
remove tools::getpwnam_ugid, impl. crate::backup::backup_user()
...
And use new nix::unistd::User struct.
2019-12-19 10:20:13 +01:00
8bf4559b4e
update to nix 0.16
2019-12-19 09:29:44 +01:00
769f8c9998
src/bin/proxmox-backup-manager.rs: connect to daemon to execute GC
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2019-12-19 08:45:27 +01:00
25de1c8005
src/client/backup_repo.rs: impl. new()
2019-12-19 08:44:38 +01:00
94913f35ca
src/bin/proxmox-backup-client.rs - start_garbage_collection: add missing output-format parameter
2019-12-19 07:57:53 +01:00
868c585219
src/backup/datastore.rs: avoid divide by zero
2019-12-19 07:14:23 +01:00
217170e11e
src/server/worker_task.rs: set worker thread name
...
This gives better logs on thread panic...
2019-12-19 07:07:39 +01:00
7e210bd0b4
src/backup/chunk_store.rs: create lock file with correct owner
2019-12-19 06:55:53 +01:00
2337df7b3d
api2: add very basic 'status/usage' endpoint for nodes
...
For returning the nodes basic "usage status", for now one gets memory
and CPU utilization.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-18 17:24:32 +01:00
bce023c57c
src/tools.rs: fix typo
2019-12-18 14:29:29 +01:00
0b97bc6158
src/backup/chunk_store.rs: use proxmox::tools::fs::create_path
2019-12-18 12:26:43 +01:00
f8ec1473be
src/config/datastore.rs: use backup gid instead of uid
2019-12-18 12:21:44 +01:00
feaa1ad35f
replace file_set_contents with replace_file
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-18 11:16:04 +01:00
a66e7920dc
src/api2/config/datastore.rs: set protected flags for create/delete API
2019-12-18 10:52:45 +01:00
f0db500808
src/api2/config/datastore.rs: fix api docu
2019-12-18 10:48:19 +01:00
424766bc3b
src/config/datastore.rs: change file owner/permissions
...
owner(root) => read and write
group(backup) => read only
2019-12-18 10:41:58 +01:00
c07b458cac
datastore: set correct owner on datastore.cfg
...
since manager runs as root, we have to specify backup:root owner while
saving config.
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-12-17 16:59:10 +01:00
14f1e63067
chunk_store: create parent directories
...
'datastore create storename /path/to/dir/that/may/not/exist' should
work.
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-12-17 15:39:42 +01:00
ebfb2df22b
binary_search_tree: add additional doctest for search_binary_tree_by()
...
Make sure a start indexes larger or equal to the array size results in a return
value of `None`.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 15:37:02 +01:00
48f6d67764
binary_search_tree: fix off by one error in search_binary_tree_by()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 14:57:34 +01:00
38d9a69875
catalog: decoder and shell: fix incorrect logic in find matching
...
The find matching was incorrectly performed starting from the parent directroy
and not as intended from the entries of the parent directory.
Further, the match pattern passed from the catalog shell contains the absolute
path of the search entry point as prefix, so find() must always start from the
archive root. This is because the match pattern has to be stored in the selected
list for a subsequent restore-selected command in the shell.
All matching paths are shown as absolute paths with all contents in the subdir,
equal to what would be restored by the given pattern.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 14:57:34 +01:00
81cc71c0d7
use proxmox-mini-journalreader to display syslog
2019-12-17 14:06:48 +01:00
20197f7cf5
src/api2/node/syslog.rs: use api macro
2019-12-17 13:26:49 +01:00
9c01e73c2f
ui: add "Documentation" button to main view
...
Similar to PVE and PMG, for quick access when one has the basic
webinterface open anyway. Should move to the "proxmoxHelpButton" once
we have an onlineHelp mapping to the docs.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 11:22:34 +01:00
2d694f8f1f
add CSS file for PBS ExtJS6 basic ui
...
some fitting rules copied over from PVE's ext6-pve.css file.
simply place it in the css subfolder where the proxmox-backup-gui.js
file is hosted and add a "css/" alias for that directory, the
formatter gets use the right content type with that.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 11:20:32 +01:00
25cdd0e0a1
catalog_shell: impl find for catalog shell.
...
Implements the find command which allows to find and select files for subsequent
restore.
Files selected for restore are now stored in a Vec instead of a HashSet.
This is needed, since instead of the full paths for each file, selected files are
now identified by a list of match pattern, where ordering matters.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
90dfd0a71d
catalog: impl find() for file search by match pattern.
...
find() iterates over the file tree and matches each node against a list of match
patterns provided at function call.
For each matching node, a callback function with the current directroy stack is
called.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
a333b4ccea
pxar: match_pattern: impl invert() to switch the match type of a MatchPattern.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
48af80b3ab
pxar: match_pattern: impl traits needed for MatchPattern to be comparable.
...
In order to check if two MatchPattern are identical, which will be used to avoid
duplicates.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
255bb86030
pxar: match_pattern: refactor MatchPattern and introduce MatchPatternSlice.
...
The MatchPattern impl heavily used copies and therefore was inefficient regarding
memory management.
This patch intoduces MatchPatternSlice as struct to avoid copies and perform the
same pattern matching functionality.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
dc08934563
drop Cancellable future in favor of abortable
...
futures-0.3 has a futures::future::abortable() function
which does the exact same, returns an Abortable future with
an AbortHandle providing an abort() method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-17 10:52:07 +01:00
8554ac5ec3
fix broadcast_future test case
...
We used to await all the futures via the runtime's shutdown
method, which doesn't exist anymore, so await all the join
handles instead.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-17 10:47:12 +01:00
ba3eb88d95
generate authkey: public part needs to be readable by backup group
...
else the API proxy cannot use it and fails to run..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 10:17:03 +01:00
9c18e93522
handle_static_file_download: move from and_then to await
...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 08:56:55 +01:00
4e8cff1c9d
cipherbench: align/format output nicely
...
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 08:49:34 +01:00
4832f53ab5
cipherbench: be conform with rfc7539, use 12 byte nonce
...
else we get an error from this call, using a 16 byte (128 bit) nonce
is currently only supported by the still in draft
XChaCha20-Poly1305, not the current default specified by RFC 7539[0],
which uses a 12 byte (96 bit) nonce.
Fixes the following error:
> thread 'main' panicked at 'called `Result::unwrap()` on an `Err`
> value: ErrorStack([])', src/libcore/result.rs:1165:5
[0]: https://tools.ietf.org/html/rfc7539
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 08:49:07 +01:00
a47a02ae62
src/bin/proxmox-backup-client.rs: use #[api] macro
2019-12-16 13:49:18 +01:00
78d5436040
client: use api macro for catalog_shell async fn
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-16 11:24:44 +01:00
87c423751a
src/bin/proxmox-backup-client.rs: use ApiHandler::Async for prune and status
2019-12-16 11:00:43 +01:00
8a8a470316
src/bin/proxmox-backup-client.rs: use new ApiHandler::Async
2019-12-16 10:06:26 +01:00
b9799012cf
src/pxar/decoder.rs: require Send
2019-12-16 10:01:51 +01:00
bb084b9c91
api2: update for latest proxmox-api changes
...
- rename ApiFuture into ApiResponseFuture
- impl. ApiHandler::Async
2019-12-16 10:01:51 +01:00
236a396aa1
src/api2/admin/datastore.rs - prune: log retention options
2019-12-14 16:32:16 +01:00
503995c767
src/api2/admin/datastore.rs: fix prune worker id
2019-12-14 16:05:21 +01:00
f374ba4c0b
update tests
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 12:42:02 +01:00
b5c9a706ce
fix cancellable future
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 12:10:04 +01:00
8a1028e0e7
update to url/percent-encoding to 2.1
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:57:01 +01:00
db0cb9ce0b
update a chunk of stuff to the hyper release
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:24:41 +01:00
3f1020b732
src/bin/proxmox-backup-client.rs: implement task stop
2019-12-13 09:43:08 +01:00
1ae5677de0
src/api2/types.rs: use PROXMOX_SAFE_ID_FORMAT instead of FILENAME_FORMAT
2019-12-12 14:15:40 +01:00
d0adf270fb
src/api2/types.rs: introduce PROXMOX_SAFE_ID_REGEX, use it for DATASTORE_SCHEMA
2019-12-12 12:37:11 +01:00
66c49c21c3
src/api2/types.rs: factor out DATASTORE_SCHEMA
2019-12-11 13:45:23 +01:00
567d3e00fb
src/api2/node/tasks.rs: new filter "store" to filter tasks for one store
2019-12-11 12:53:34 +01:00
d3dbe52f37
pxar: match_pattern: refactor match_filename
...
The match_filename() in sequentail_decoder and encoder are moved to be static
functions of MatchPattern.
This allows to reuse the code also in the catalog find implementation.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-11 10:55:01 +01:00
f084505ec5
src/backup/catalog_shell.rs: sort output of list-selected
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-11 10:54:20 +01:00
6d04612955
src/api2/node/tasks.rs: add "running" filter to list only running tasks
2019-12-11 10:52:27 +01:00
8560fe3e5f
src/api2/node/tasks.rs - extract_upid: avoid duplicate error message
2019-12-10 13:45:56 +01:00
5830c20560
src/bin/proxmox-backup-client.rs: add simple task management cli
2019-12-10 13:43:53 +01:00
73e57f244e
src/backup/datastore.rs - garbage-collection: fix average chunk size
2019-12-10 11:25:29 +01:00
d105176f45
src/bin/proxmox-backup-client.rs: cleanup - factor out view_task_result()
2019-12-09 18:00:22 +01:00
48ef3c3327
cli: avoid useless .into()
...
This needs latest proxmox-api updates.
2019-12-09 17:40:34 +01:00
eb7e2ee00b
src/bin/proxmox-backup-client.rs: remove unnecessary .to_owned()
2019-12-09 17:11:48 +01:00
e5f7def40e
src/bin/proxmox-backup-client.rs - garbage-collection: display task log
2019-12-09 13:29:23 +01:00
0ec9e1b010
src/bin/proxmox-backup-client.rs: impl. and use complete_pxar_archive_name
2019-12-09 13:14:32 +01:00
1c6ad6ef90
src/bin/proxmox-backup-client.rs: impl. catalog sub command
...
catalog dump
catalog shell
2019-12-09 13:07:25 +01:00
5a0b484bdd
src/client/task_log.rs: move display_task_log into extra file
2019-12-08 11:27:15 +01:00
3b03abfe14
src/api2/admin/datastore.rs: remove test_prune, add dry-run to normal prune
...
And use display_task_log on the client side.
2019-12-08 10:59:47 +01:00
ec137a99c6
src/bin/proxmox-backup-client.r - display_task_log: use string range instead of as_bytes
2019-12-08 10:25:26 +01:00
b73ff941f6
src/bin/proxmox-backup-client.r - display_task_log: add new strip_date parameter
2019-12-08 10:22:02 +01:00
163e9bbe91
src/bin/proxmox-backup-client.rs: new display task log helper
...
New helper to display worker task logs. Use it for prune (experimental).
2019-12-07 16:11:26 +01:00
6b508dd563
src/server/formatter.rs: impl. new result attribute "active"
2019-12-07 15:29:42 +01:00
102d8d4136
src/backup/prune.rs: implement --keep-hourly
2019-12-07 11:23:33 +01:00
052c4aa160
src/backup/prune.rs: prune - correctly use iso_week year
2019-12-06 19:41:38 +01:00
fa0b950070
pxar: fuse: add support for list- and getxattr
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 16:44:15 +01:00
25ad4cbf63
pxar: fuse: cache goodbye table for each directory on opendir and release it on releasedir
...
Cache not only the goodbye table for the last directory but for each opened
directory.
The opendir fuse callback will fill the cache with the goodbye table and
releasedir will remove it from the cache.
This should reduce the number of chuncks fetched from the server in some cases.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 16:43:59 +01:00
02491b8fc6
pxar: fuse: use binary search instead of linear search in goodbye table.
...
The goodbye table is layed out as binary search tree based on the hash, so use
this to be more efficient when looking up a hash in the table for directories
with a large number of entries.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 16:42:01 +01:00
7074a0b339
src/bin/proxmox-backup-client.rs: white space cleanup
2019-12-06 13:35:59 +01:00
74fa81b89c
src/bin/proxmox-backup-client.rs: implement prune --dry-run
2019-12-06 13:35:25 +01:00
1f604f1a5a
src/api2/admin/datastore.rs: add new api test_prune()
2019-12-06 13:06:56 +01:00
0ab08ac980
src/api2/admin/datastore.rs: cleanup - move schema defininition in front of rust method
2019-12-06 12:36:41 +01:00
1a0678b601
src/api2/admin/datastore.rs: remove prune parameters from status call
...
Not idea why they are added there.
2019-12-06 12:32:53 +01:00
9e3f008804
src/backup/prune.rs: add new helper keeps_something()
2019-12-06 12:28:31 +01:00
d75fbb0a21
src/bin/dump-catalog-shell-cli.rs: generate catalog shell docs
2019-12-06 09:54:13 +01:00
55c3cb69cd
rename catalog_shell_api() into catalog_shell_cli()
2019-12-06 09:42:05 +01:00
ecbaa38fa4
src/backup/catalog_shell.rs: factor out catalog_shell_api()
...
In order to extract documentation ...
2019-12-06 09:35:47 +01:00
cca67dc0c3
src/backup/readline.rs: removed in favor of new readline impl
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 09:03:04 +01:00
6934c6fe77
src/backup/catalog_shell.rs: adapt to use API Schema definition and rustyline
...
This major refactoring of the catalog based shell utilizes the new API macro and
the API Schema as well as rustyline instead of the old GNU readline C API.
The code now has these 3 main components:
* The `Shell` which handles the readline loop via rustyline.
* The shell functions defined via the API macro.
* The `Context` which holds catalog and decoder instances.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 09:02:44 +01:00
9b78352188
src/backup/prune.rs: define new struct PruneOptions
2019-12-06 08:56:27 +01:00
92acbd69f7
src/backup/backup_info.rs: remove old prune code
2019-12-06 08:29:27 +01:00
408434360b
src/backup/prune.rs: add PruneMark::KeepPartial
...
Use separate mark for partial backup - dont mess them up with regular ones.
2019-12-06 08:15:32 +01:00
a8c8366cfa
src/backup/prune.rs: factor out code into remove_incomplete_snapshots()
2019-12-06 08:12:08 +01:00
dc18849156
src/backup/prune.rs: moved prune related code into extra file
2019-12-06 08:06:21 +01:00
6f47dd8a0f
src/backup/backup_info.rs - prune: improve algorythm
2019-12-05 18:41:28 +01:00
8f0b4c1f90
src/api2/admin/datastore.rs - prune: log info about all snapshots
2019-12-05 15:35:11 +01:00
9ce42759ec
src/backup/backup_info.rs - compute_prune_list: remove unfinished backups
2019-12-05 11:18:10 +01:00
2c034f8d0a
src/backup/backup_info.rs: improve prune algorithm
2019-12-05 08:55:19 +01:00
aeeac29bb1
src/backup/backup_info.rs: new compute_prune_list helper
...
So that we can write regression tests for this.
2019-12-04 15:49:11 +01:00
3d229a4a01
Revert "test another api macro use"
...
This reverts commit a9aa52e6a8
2019-12-04 14:18:11 +01:00
4f1e40a29e
src/api2/admin/datastore.rs - read_backup_index: add manifest with size
...
So that the client can display the size of the index file.
2019-12-04 10:03:52 +01:00
a9aa52e6a8
test another api macro use
...
The api macro now supports hyphens in parameter names and
referencing externally defined `Schema`s, so here's an
example.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-03 13:41:40 +01:00
de0aa53174
pxar: decoder: fix incorrect read for hardlinks.
...
Decoders read must check if the file is a hardlink and read data from the
corresponding offset if so.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-03 12:08:23 +01:00
b7e18004a2
pxar: decoder: return original name in attributes()
...
The returned filename should be the one of the file given at the offset, not of
the one the hardlink points to.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-03 12:08:07 +01:00
7eea56ca00
use cli from proxmox::api
2019-12-02 11:56:29 +01:00
524b22df35
src/cli/completion.rs - get_help_completion: fix for "help -" expansion
2019-12-02 11:11:31 +01:00
e8e9bae4d1
src/bin/completion.rs: add echo command
2019-12-02 11:04:40 +01:00
19d7a59236
src/bin/completion.rs: use api macros
2019-12-02 10:57:19 +01:00
f9a51322cb
src/cli/command.rs: use insert_help()
2019-12-02 08:52:08 +01:00
e3e72546dd
src/cli/command.rs: cleanup, remove unused top_def parameter
2019-12-02 08:47:03 +01:00
2348e77edf
src/cli/completion.rs - get_simple_completion: remove all_arg_param parameters
...
Seems useless.
2019-12-02 08:31:03 +01:00
28c855c0a2
src/cli: improve docs
2019-12-01 16:41:49 +01:00
3bf920527c
src/cli/command.rs: make set_help_context private - no need to expose
2019-12-01 12:54:41 +01:00
bca6d9120a
src/cli/command.rs: improve help command description
2019-12-01 12:40:37 +01:00
2f6ecc1ccd
src/cli/completion.rs: moved all completion related code here
2019-12-01 11:44:31 +01:00
a810e05288
src/cli/readline.rs: moved readline related code here
2019-12-01 11:31:35 +01:00
5d64a0d083
src/cli/shellword.rs: moved shellword related code into this file
2019-12-01 11:17:57 +01:00
fb8a00bc00
src/cli/command.rs - get_nested_completion: allow fuzzy match
2019-12-01 11:06:54 +01:00
9498fcf9c5
src/cli/command.rs - get_nested_completion: return nothing if first argument does not match
2019-12-01 10:24:07 +01:00
309e449db2
src/cli/command.rs: improve get_help_completion
2019-12-01 10:18:47 +01:00
b8a82922c7
src/cli/command.rs: handle multilevel sub commands in help
2019-12-01 09:42:51 +01:00
b7bf71dfa1
src/cli/command.rs: fix help completions
2019-11-30 15:43:44 +01:00
7afc1af7ab
src/bin/completion.rs: add history support
2019-11-30 15:08:37 +01:00
4b8573da65
src/cli/command.rs - handle_command: return Result instead of exit()
2019-11-30 15:08:00 +01:00
2b691daf6f
src/cli.rs: add insert_help() method
2019-11-30 14:56:31 +01:00
1201abcffa
src/cli/command.rs: implement help
2019-11-30 13:02:00 +01:00
8423c1fe64
cli/completion: use match statements and .contains()
...
`match` is a bit more readable than the if-else chains,
also replace
space_chars.iter().any(|s| c == *s)
with
space_chars.contains(&c)
which is also more readable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-29 13:37:34 +01:00
e399398444
remove shellwords crate
2019-11-29 12:49:44 +01:00
ca60ac13d9
use rustyline to replace readline
2019-11-29 12:49:44 +01:00
41874331ed
whitespace fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-29 09:52:15 +01:00
2905f2b5e6
update api macro example usage
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-29 09:51:27 +01:00
8b130e71da
src/cli/command.rs: cleanup, make handlers more generic
2019-11-27 14:44:21 +01:00
7b6c41078b
update api macro invocation to new style
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-27 14:19:46 +01:00
314bb35868
catalog_shell: fix several clippy warnings
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-27 10:13:50 +01:00
6486cb853f
first api macro usage test/example
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-27 10:05:37 +01:00
951cf17ee3
catalog_shell: major refactoring of cli command definition and parsing
...
By changing the way shell commands are defined and parsed, this makes it more
straight forward to extend the current functionality.
The readline input is parsed based on the provided command definition and the
given parameters and options are passed to a command specific callback function.
In addition, the provided command definition including its description is used
to generate a help string to display.
The help command shows a list of all supported commands or the help string for
the provided command.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 13:01:00 +01:00
59bc6ad676
catalog_shell: check and return on empty list in order to avoid division by zero
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 12:55:33 +01:00
4e56b2f792
catalog_shell: do not restore full archive if no entries are selected for restore-selected
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 12:55:18 +01:00
12b4098660
src/cli/format.rs: move doc generation code into extra file
2019-11-26 12:12:53 +01:00
2f2b87e659
src/pxar/fuse.rs: add impl Context
2019-11-26 11:16:46 +01:00
eedd1f95eb
src/pxar/fuse.rs: move init and destroy into Session
2019-11-26 11:08:55 +01:00
970687c982
src/pxar/fuse.rs: cleanup - remove setup_session (moved into new)
2019-11-26 11:04:46 +01:00
2a11191072
src/pxar/fuse.rs: rename from_decoder into new, new into from_path
2019-11-26 10:56:41 +01:00
99b5b6cba9
src/pxar/decoder.rs: use trait object as reader
...
So that the Decoder is no longer genertic.
2019-11-26 10:45:11 +01:00
f701d0335e
src/pxar/sequential_decoder.rs: remove callback from new()
...
And use an extra functzion set_callback() to configure that.
Also rewrite pxar/fuse.rs and implement a generic Session (will get
further cleanups with next patches).
2019-11-26 09:56:48 +01:00
49fddd985c
src/cli/command.rs: use static array for arg_param
2019-11-25 11:16:41 +01:00
329d40b50b
rename ApiHandler::Async into ApiHandler::AsyncHttp
2019-11-23 09:03:21 +01:00
70fbac84da
src/server/rest.rs: simplify code
2019-11-22 18:44:14 +01:00
2bbd835b9b
src/server/rest.rs: rename get_request_parameters_async to get_request_parameters
2019-11-22 17:24:16 +01:00
75a5a68944
src/server/rest.rs - only pass ObjectSchema to get_request_parameters_async()
2019-11-22 17:22:07 +01:00
ad51d02aa9
src/server/rest.rs: cleanup async code
2019-11-22 13:02:05 +01:00
be2bb37205
remove api_schema submodule
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-22 09:31:45 +01:00
e57e1cd82d
move src/api_schema/config.rs -> src/server/config.rs
2019-11-22 09:23:03 +01:00
8256b0e417
src/api_schema/registry.rs: remove dead code
2019-11-22 09:04:52 +01:00
3cf73c4e53
src/bin/proxmox-backup-client.rs: impl shell subcommand for client
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 08:22:50 +01:00
f14c96ea38
src/backup/catalog_shell.rs: impl shell to inspect and restore a snapshot via the catalog
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:54:04 +01:00
38446a9551
src/backup/readline.rs: impl wrapper for GNU readline
...
In order to provide the context needed for tab completion via the readline
callback, the needed mut ref is passed via a thread local storage key.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:46:24 +01:00
6dba015043
src/backup/catalog.rs: derive Clone for DirEntry and DirEntryAttribute
...
This is needed in order to explicitly clone the values when needed in the
catalog shell implementation.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:15:39 +01:00
c9f002213e
src/backup/catalog.rs: impl DirEntry::is_directory()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:14:00 +01:00
d3bcc7f261
src/pxar/sequential_decoder.rs: create all directories in target path
...
Create the full target path and not fail if an intermediate directory does not
exist.
This is needed in order to restore multiple archives via the catalog, where the
target should further contain each archive name as subdir.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:11:49 +01:00
b6ca690095
src/pxar/decoder.rs: return also payload size in lookup
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:05:14 +01:00
45cc157f7c
docs: add backup reader protocol
2019-11-22 06:59:37 +01:00
cad540e969
api/compat: remove remaining api_schema references
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:53:15 +01:00
a2479cfa1a
api/compat: drop more compat imports from api_schema.rs
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:36:28 +01:00
cc4ecf80b0
api/compat: drop api_handler submodule
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:18:41 +01:00
4f1963d259
api/comapt: drop schema submodule
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:17:36 +01:00
2194824f77
api: BoxFut -> ApiFuture
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:16:37 +01:00
3d482025b3
move api schema into proxmox::api crate
...
And leave some compat imports in api_schema.rs to get it to
build with minimal changes.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:14:54 +01:00
552c225948
sort all property lookup tables
...
Required, because we use binary sreach to find items.
2019-11-21 13:32:09 +01:00
255f378a1b
use const api definitions
2019-11-21 13:32:09 +01:00
c5946faffd
drop md5 crate dependency
...
we already depend on openssl which also provides md5
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-14 11:19:43 +01:00
f569acc5e2
drop uuid crate dependency
...
proxmox::tools now has a Uuid module using the native
libuuid.
Adds build dependency: libuuid1 (which is a Pre-Depends of
util-linux, so always installed anyway).
Drops uuid + 16 more crate dependencies.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-14 10:22:29 +01:00
b423958d8e
catalog: fix issue with DirEntry start
...
start has to be the parent start - offset
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-12 17:18:13 +01:00
780dd2b0a1
src/backup/catalog.rs: add some docs
2019-11-12 13:23:40 +01:00
6aa906b547
src/backup/catalog.rs - CatalogReader: verify magic number
2019-11-12 12:54:06 +01:00
2ec208aef5
src/backup/catalog.rs - CatalogReader::dump(): use root()
2019-11-12 12:47:21 +01:00
8f24a9ea18
src/backup/catalog.rs - Direntry::parse: abort if callback return false
2019-11-12 11:41:44 +01:00
7d017123fd
src/backup/catalog.rs: impl read_dir() and lookup(), refactor common code
2019-11-12 11:41:44 +01:00
dc9596de45
src/backup/catalog.rs: move CatalogEntryType from src/pxar/catalog.rs
2019-11-12 11:41:44 +01:00
55c0b3cc7c
src/backup/catalog.rs - DirEntry: factor out name attribute
...
And avoid allocations inside parse().
2019-11-12 11:41:44 +01:00
3f1c5b5e65
catalog: impl std::fmt::Display trait for CatalogEntryType
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-12 07:00:08 +01:00
f89359c2f3
src/backup/catalog.rs: clenup - use single impl block
2019-11-10 11:38:31 +01:00
e8d8fdc61d
src/backup/catalog.rs: remove unused code
2019-11-10 11:37:06 +01:00
c74c074bfd
src/backup/catalog.rs: write MAGIC at start of file
2019-11-10 11:36:45 +01:00
265664517a
src/backup/catalog.rs: new parse() helper
2019-11-09 14:37:33 +01:00
5d92935e03
src/backup/catalog.rs - dump_dir: make sure we parsed the whole data block
2019-11-09 13:21:20 +01:00
12a1975af9
src/backup/catalog.rs: store u64 compressed
...
This way catalogs get about 1/3 smaller.
2019-11-09 13:07:47 +01:00
bb19af7302
src/client/backup_reader.rs: factor out download_blob() helper
2019-11-08 13:16:41 +01:00
7205050059
src/client/backup_reader.rs: factor out download_fixed_index() helper
2019-11-08 12:51:52 +01:00
c3d84a2281
src/client/backup_reader.rs: factor out download_dynamic_index() helper
2019-11-08 12:51:52 +01:00
89245fb530
renamed: src/backup/catalog_blob.rs -> src/backup/catalog.rs
2019-11-08 10:41:00 +01:00
bf6e321744
change catalog format, use dynamic index to store catalog.
...
In order to remove size restriction of a single blob.
2019-11-08 10:35:48 +01:00
e016f9ff2f
src/client/backup_writer.rs: rename force_close to cancel
...
And do not consume self.
2019-11-08 10:34:16 +01:00
0997967d65
do no restrict index names too much
2019-11-08 10:28:14 +01:00
e66621182b
src/pxar/decoder.rs: impl lookup for `Decoder`
...
Allows to lookup an entry in a directory based on the provided `DirectoryEntry`.
This is needed to navigate the filesystem based on `DirectoryEntry`s and similar
to the find_goodbye_entry() function in src/pxar/fuse.rs
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-06 10:00:41 +01:00
fb2554de29
src/pxar/decoder.rs: fix wrong filename check for Decoder::restore()
...
As it turns out the original implementation was correct and the start in
`DirectoryEntry` points to the `PxarEntry` and not as wrongly stated to the
filename.
This reverts the incorrect code and adds comments to the fields clarifying this.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-06 09:59:35 +01:00
33ad183a40
src/pxar/decoder.rs: allow to pass match pattern to restore
...
In order to partially restore the archive starting from the directory provided as
parameter.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-05 14:19:09 +01:00
5e8d600c71
src/pxar/decoder.rs: fix issue with restore
...
`Decoder::restore()` calls the `SequentialDecoder::restore()` which expects to
encounter a `PxarEntry` at first. But the start of `DirectoryEntry` points to the
filename (except for the root dir), so skip over it.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-05 14:18:29 +01:00
9708b2d98c
src/pxar/fuse.rs: refactor Iterator: use find() instead of position()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-04 13:59:16 +01:00
b82a101c86
catalog: store current writer position in CatalogBlobWriter
...
Counts the bytes written by the CatalogBlobWriter in order to obtain the
stream position, needed to get offset to reference catalog items.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-31 09:11:51 +01:00
62ee2eb405
avoid some clippy warnings
2019-10-26 11:42:05 +02:00
834a2f95a0
avoid some clippy warnings
2019-10-25 18:44:51 +02:00
f58f426e86
src/pxar/binary_search_tree.rs: allow(clippy::many_single_char_names)
2019-10-25 18:22:19 +02:00
11377a47bb
avoid some clippy warnings
2019-10-25 18:04:37 +02:00
9fe2f639b8
allow(clippy::cast_ptr_alignment)
2019-10-25 18:03:55 +02:00
cd69d36b45
src/backup/dynamic_index.rs: use write_all()
2019-10-25 17:22:41 +02:00
8abd12f1f3
pxar: decoder: cleanup un-needed Decoder::get_dir()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 08:02:23 +02:00
9c25c0330a
pxar: fuse: optimize fuse readdir callback
...
The expensive call to Decoder::read_directory_entry() can be omitted as
Decoder::attributes() returns all the information the fuse response needs.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 08:02:03 +02:00
ffe171c5db
src/pxar/encoder.rs: avoid duplicate check for MAX_DIRECTORY_ENTRIES
...
Simply by reordering code.
2019-10-23 07:55:56 +02:00
62d123e50a
pxar: add functionality to pass exclude MatchPatterns on create
...
This exposes the option to pass a list of exclude MatchPattern via the
'--exclude' option.
The list is encoded as file '.pxarexclude-cli' in the archives root directory.
If such a file is present in the filesystem, it is skipped and not included in
the archive in order to avoid conflicting information.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 07:31:49 +02:00
ba8165c607
pxar: encoder: add encode_pxar_exclude_cli() function.
...
This provides the functionality needed to encode MatchPatterns passed on the cli
in the root directory.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 07:30:30 +02:00
920243b12d
pxar: simplify MatchPattern::to_bytes
2019-10-23 07:29:02 +02:00
e50a90e010
pxar: Implement to_bytes() for MatchPattern in order to write them to file.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 07:21:11 +02:00
a8f10f849e
src/bin/proxmox-backup-client.rs: fix parameters for get_encryption_key_password
2019-10-17 10:28:21 +02:00
40c9ff9d81
src/pxar/encoder.rs: include .pxarexclude files into the catalog
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-17 10:25:20 +02:00
20feec9fac
pxar: fuse: include nsec information in stat
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-17 10:24:13 +02:00
649c5ee7b1
load_and_decrtypt_key: allow function closures to readf passphrase
2019-10-16 13:23:52 +02:00
60ffa9649c
src/backup/manifest.rs: fix property names (use '-' instead of '_')
2019-10-14 12:28:27 +02:00
4c9f753c56
src/backup/data_blob.rs: define create_zero_chunk as member function
...
and rename it to build_zero_chunk()
2019-10-14 10:58:26 +02:00
2745151d93
src/backup/data_blob.rs: add create_zero_chunk() helper
2019-10-14 10:44:46 +02:00
f06b820ac0
src/backup/manifest.rs: add verify_file
2019-10-13 10:09:12 +02:00
511a47bd73
cleanup
2019-10-13 09:39:21 +02:00
e17d5d864d
src/backup/manifest.rs: improve parser (try_from)
2019-10-13 08:39:49 +02:00
59e9ba01c6
src/backup/manifest.rs: new class to generate/parse index.json
2019-10-12 17:58:08 +02:00
ad6e5a6f51
rename INDEX_BLOB_NAME to MANIFEST_BLOB_NAME
2019-10-12 15:53:27 +02:00
296c50ba3d
src/client/backup_reader.rs: add download_manifest
...
Also add crypt_config as member variable.
2019-10-12 15:50:26 +02:00
cf9271e27c
src/client/backup_writer.rs: move BackupClient code into extra file
...
And rename it to BackupWriter.
2019-10-12 13:53:11 +02:00
913acb41fd
src/client/backup_reader.rs: add documentation
2019-10-12 13:26:47 +02:00
9e490a7479
src/client/backup_reader.rs: split BackupReader code into separate file
2019-10-12 12:57:08 +02:00
fd04ca7a5a
src/bin/proxmox-backup-client.rs: pass verbose flag to dump_image
...
And print some useful status to stderr.
2019-10-09 12:04:02 +02:00
2c2097ee75
fix typo in error message
2019-10-07 12:48:56 +02:00
f7aa6f15d3
src/api2/reader.rs: use tokio::fs::read() to read file async
2019-10-07 12:24:06 +02:00
7123ff7d43
src/backup/data_blob.rs: avoid Arc<CryptConfig>
...
use simple reference instead.
2019-10-07 11:41:35 +02:00
4ee8f53d07
remove DataChunk file format - use DataBlob instead
2019-10-06 10:31:06 +02:00
c26d4b4573
cleanup: remove dead code
2019-10-05 16:10:41 +02:00
88892ea801
src/bin/proxmox-backup-client.rs: cleanup restore
...
- write directly to /dev/stdout (instead of using stdout line writer)
- avoid BufferedFixedReader
2019-10-05 11:52:12 +02:00
62e05bb3b3
src/client/remote_chunk_reader.rs: pass reference to download_chunk
...
Make sure we do not move data around.
2019-10-05 11:45:20 +02:00
c0b1b14c05
src/api2/reader.rs - download_chunk: use blocking io
...
Turns out to be twice as fast as tokio async io...
2019-10-05 11:41:19 +02:00
ff01c1e393
src/client/http_client.rs: store raw data/blob size in index.json
2019-10-02 10:54:06 +02:00
f16aea687a
src/client/http_client.rs: new helper send_upload_request()
2019-09-28 18:22:48 +02:00
9e603e254c
src/backup/fixed_index.rs: do not use offset for checksum
2019-09-23 11:30:20 +02:00
fb6026b66b
backup writer api: verify checksum inside close
2019-09-23 10:56:53 +02:00
c8c4051aa7
src/client/http_client.rs - H2Client::upload: fix parameter order
2019-09-20 13:19:55 +02:00
f011dba0f7
src/client/http_client.rs: add method parameter to H2 upload
2019-09-20 12:40:23 +02:00
792a70b9e1
src/client/http_client.rs: add content type parameter to H2 upload/requets_builder
2019-09-20 12:26:21 +02:00
a55b2975db
src/client/http_client.rs: cleanup (rename parameter)
2019-09-20 11:57:03 +02:00
32efac1cce
remove trailing whitespace
2019-09-20 08:12:31 +02:00
70235f72f4
src/bin/proxmox-backup-client.rs: add functionality to mount pxar archive.
...
This adds the basic functionality to mount an archive stored in a repository
locally via fuse.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-20 08:01:01 +02:00
2fa91f5221
src/bin/pxar.rs: adapt to new deamonize flag
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-20 08:00:20 +02:00
d388e244f5
src/pxar/fuse.rs: make verbose flag independent from deamonize by adding this as new flag
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-20 08:00:04 +02:00
e717e9a3bf
src/pxar/fuse.rs: refactor Session::new()
...
and add Session::from_decoder() in order to be able to create a fuse session
with a `Decoder` given as argument.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-20 07:59:51 +02:00
946b72a6b2
src/pxar/fuse.rs: introduce `Context` in order to add caching.
...
This patch introduces `Context` to hold the decoder, ino_offset and caches for
the attributes and the goodbye table.
By caching, certain callbacks can be handled without the need to read additional
data via the decoder, which improves performance.
The searching of the goodbye table is refactored as well, avoiding recursive
function calls in case of a hash collision.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-20 07:56:40 +02:00
1c93182371
src/pxar/decoder.rs: fix bug in decoder read
...
`offset` points to the `PXAR_FILENAME`, therefore read the filename before
the `PXAR_ENTRY`.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-20 07:54:16 +02:00
2aba16bde1
src/backup/data_chunk.rs: add into_raw()
2019-09-19 14:24:17 +02:00
15bb1bba89
BackupClient: add upload() function
2019-09-19 13:59:24 +02:00
6d4df36c3c
make download_chunk_list() public
2019-09-19 13:17:23 +02:00
cc2ce4a92f
src/client/http_client.rs: add password parameter to HttpClient::new()
2019-09-16 12:35:23 +02:00
0e20b336e1
pxar: fuse: avoid possible hash collision in lookup by additional checking against filename
...
The hash of the filename in the goodbye table items allows to quickly compare to
a hashed filename.
Unfortunately, a matching hash is no garantee for matching filenames as hash
collisions are possible.
This patch fixes such possible collisions by further checking the filenames once
a matching hash has been found.
This introduces no significant extra cost (except for the filename comparison)
for cases with matching hashes, as the lookup call has to seek and read the file
attributes (including the filename) anyway.
In cases with hash collision, the next matching item is read and treaded
analogously (what means we need at least one extra seek).
As collisions should be not that frequent, this should be an acceptable penalty.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-16 10:22:22 +02:00
ba5e67475a
clippy: fix access to unaligned buffer in dynamic reader
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:50:45 +02:00
b526bd1408
backup: cleanup unused imports
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:46:46 +02:00
4dc79bb1d9
clippy: fix access to unaligned buffer
...
via the newly introduced ReadExt::read_host_value_boxed()
from proxmox::tools::io.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:45:44 +02:00
ccdf3ad1ec
backup: use static assertion in fixed index reader
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:43:25 +02:00
34a3845b01
clippy: don't use clone a double reference
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 14:02:30 +02:00
b21851cc5c
clippy: collapse identical if branches
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:57:58 +02:00
299a2f18b2
fixup variable naming
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:57:23 +02:00
175eeb870d
clippy: use write_all in file logger
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:56:09 +02:00
5a54935bc9
clippy: fix pointer to temporary value
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:55:56 +02:00
bd430c225b
clippy: remove a loop{} which never actually loops
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 12:31:39 +02:00
44fed91e17
more clippy fixups
...
mostly indentation changes, view with `-w`
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 12:23:12 +02:00
653b1ca10e
some clippy fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 12:14:05 +02:00
03310bea12
src/pxar/fuse.rs: s/Buf/ReplyBuf/ and s/BufState/ReplyBufState/
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-11 11:27:18 +02:00
48cc1b8234
src/pxar/fuse.rs: refactor stat and fix i-node mapping
...
The functionality of stat is split into smaller sub-functions, which allows
to reuse them more flexible, as the code flow is similar but not always the same.
By this, the ugly and incorrect re-setting of the i-node in the lookup callback
function is avoided.
The correct i-node is now calculated beforehand and stat simply creates a
`libc::stat` struct from the provided parameters.
Also, this fixes incorrect i-node assignments in the readdir callback function.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-11 11:26:55 +02:00
22eaa905a4
src/pxar/fuse.rs: impl readdir callback for fuse
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 11:38:58 +02:00
1652192f4d
src/pxar/fuse.rs: impl readlink callback for fuse
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 11:38:48 +02:00
3710b23db5
src/pxar/decoder.rs: read_link has to read filename before entry.
...
The offset points to the start of the item, which is the filename
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 11:36:04 +02:00
7eb9848ba7
src/pxar/fuse.rs: impl opendir callback for fuse
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 11:35:13 +02:00
2c5fd378d3
src/pxar/fuse.rs: impl read callback for fuse
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 11:34:55 +02:00
97e4db9dd5
src/pxar/fuse.rs: impl open callback for fuse
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 11:34:18 +02:00
b00689254a
src/pxar/fuse.rs: impl lookup callback for fuse
...
The lookup call checks if the given filename is found in the directory referenced
by the i-node by calclulating the filenames hash and looking it up in the
directories goodbye table.
If found, the entries parameters are returned.
In order to be able to lookup the parent offset by a given file offset in the
readdir callback, this also stores the corresponding values in a HashMap.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 10:54:23 +02:00
1443927c02
src/pxar/decoder.rs: correctly handle hardlinks in `Decoder`s attribute function
...
If the attributes are then returned by seeking to the new offset.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 10:46:39 +02:00
d00097a0e6
src/pxar/decoder.rs: split functionality of list_dir into list_dir and goodbye_table
...
In order to read the contents of the goodbye table while keeping the
functionality of list_dir in place as is.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 10:46:29 +02:00
ac12570e99
src/pxar/decoder.rs: make read_directory_entry usable by fuse
...
... and thereby allow it to read a single directory entry based on the
start and end archive offsets.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 10:32:48 +02:00
3f7f4f34be
src/pxar/decoder.rs: remove `DirectoryEntry` start_offset and end_offset functions
...
... as they are not needed with the latest iteration of the fuse callback
function impl (which never made it into the repos).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 10:30:45 +02:00
ac30bc3326
src/pxar/decoder.rs: correctly read, check and skip filename
...
The previous implementation simply skipped over `size` bytes, which is not
correct as size includes also the header.
By relying on `SequentailDecoder`s read_filename function, this is correctly
taken care of plus some more integrity checks.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-10 10:30:21 +02:00
4a7de56e2f
avoid injecting ENV vars from Makefile
...
So that we can run "cargo build" without setting vars manually.
2019-09-09 10:51:08 +02:00
d21ae955a6
src/pxar/fuse.rs: impl getattr callback for fuse
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-06 12:24:14 +02:00
ec04ea81f2
src/pxar/fuse.rs: Refactor run_in_context and remove inode_to_offset and offset_to_inode
...
as their functionality is used only once so it makes more sense to keep them
inline.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-06 12:22:53 +02:00
8413083fc5
src/pxar/fuse.rs: Improve comments, add rustfmt skip tag and reafctor
...
Also, removes an unused println statement in the decoder callback function and
fixes a typo.
Further, use ABI compatible Option<&T> for FFI to avoid use of raw pointers.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-06 12:22:17 +02:00
54a5a885a3
src/client/http_client.rs: use async for upload_speedtest()
2019-09-06 08:55:47 +02:00
bbd055bf45
src/pxar/decoder.rs: impl functionality needed for fuse implementation
...
Implements functions attributes, open, read, read_link and get_dir
to be used by the fuse implementation which uses file offsets within the archive
as inodes to reference the archives items.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-06 07:18:14 +02:00
132cb0d0db
src/pxar/decoder.rs: Add comment for unsolved issue and refactor if statement.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-06 07:16:31 +02:00
2a05048b93
src/client/http_client.rs: simplify send_request
2019-09-05 15:07:37 +02:00
9edd3bf1b8
src/client/http_client.rs: use async for h2api_response()
2019-09-05 14:56:52 +02:00
c18fddf80f
src/client/http_client.rs - download_chunk_list: simplify async code
2019-09-05 14:16:45 +02:00
44f59dc731
src/client/http_client.rs - H2Client: fix error message in download()
2019-09-05 13:26:06 +02:00
c2a5a9f353
src/client/http_client.rs: use async for download_chunk_list
2019-09-05 13:12:03 +02:00
2f831baec0
src/client/http_client.rs - BackupClient: use async
2019-09-05 12:55:22 +02:00
2a1e6d7dea
src/client/http_client.rs - BackupClient: use async
2019-09-05 11:42:12 +02:00
d4a085e564
src/client/http_client.rs - BackupReader: use async
2019-09-05 11:42:05 +02:00
9d456cf8ec
src/backup/catalog_blob.rs: use write_all() instead of write()
...
To correctly handle interrupted system calls.
2019-09-05 11:40:48 +02:00
1a7a0e74c0
src/client/http_client.rs: use async for more functions
2019-09-04 13:48:16 +02:00
96f5e80abb
src/client/http_client.rs: use async for login()
2019-09-04 10:01:46 +02:00
9d35dbbb8f
src/client/http_client.rs: use async for credentials
2019-09-04 09:57:29 +02:00
d2267b112d
src/bin/proxmox-backup-client.rs: verify blob/catlog checksums
2019-09-04 08:47:14 +02:00
70b03a07c8
src/pxar/sequential_decoder.rs: make functions needed in non-sequential decoder accessible from within crate
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:15:05 +02:00
f92e8266ae
src/pxar/format_definitions.rs: move header type in PxarHeader and hash in PxarGoodbyeItem to top of struct
...
By this it is possible to read and check just the first u64 of the corresponding
structs in order to identify the items.
This is needed for the fuse implementation in order to get entries based on the
archive offset, used as inode.
Directories are referenced by the offset to the goodbye tail while other items
are referenced by the offset of the filename followed by the entry.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:14:30 +02:00
75933d1e83
src/pxar/fuse.rs: add function to map offset to inode and vice versa
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:13:59 +02:00
c7fee396c6
src/pxar/decoder.rs: add getter functions to obtain start/end offsets for directories
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:10:56 +02:00
3626ac611f
src/pxar/decoder.rs: rustfmt fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:09:17 +02:00
3fa71727ee
src/bin/proxmox-backup-client.rs: refactoring of match statement
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-09-04 06:08:00 +02:00
a40220c056
src/bin/proxmox-backup-client.rs: code cleanup - factor out some functions
2019-09-03 13:31:13 +02:00
36493d4d7e
src/backup.rs: define INDEX_BLOB_NAME here
2019-09-03 13:15:44 +02:00
df65bd3da6
src/bin/proxmox-backup-client.rs - restore: verify checksums
2019-09-03 13:12:16 +02:00
0a51fe0011
src/backup/fixed_index.rs: new helper to compute checksum and file size
2019-09-03 13:11:45 +02:00
82c85a21a1
src/backup/dynamic_index.rs: new helper to compute checksum and file size
2019-09-03 13:11:09 +02:00
a3e032b754
src/client/http_client.rs: fix dynamic index checksum
2019-09-03 13:10:18 +02:00
c77563146b
comment fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-03 11:17:33 +02:00
daef93f481
tools: add tokio::main() replacement
...
to deal with block_on() not allowing blocking()
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-03 11:16:30 +02:00
66fbf5bad0
add WrappedReaderStream test
...
Mostly to just document the rt.block_on() issue with
blocking().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-03 10:05:45 +02:00
2b92971fba
merge fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:22:36 +02:00
083ff3fd5d
update to tokio 0.2.0-alpha.4
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
1434f4f8df
drop hyper_openssl
...
We can use .await now, which means the whole connection
state machine doesn't need to be typed out as "types"
anymore, so, at least until hyper_openssl gets updated to
proper dependencies, let's drop it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
556eb70ea3
add generic either-or AsyncRead/Write type
...
The HttpsConnector will use this. Instead of implementing a
specialized MaybeTlsStream, this is simply a generic "either
this or that kind of Async Read/Write type".
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
b30415d244
src/server/state.rs: update to tokio alpha.2
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
aadcf7c621
src/server/rest.rs: use tokio::timer::delay
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
0cdb8d9c5b
src/tools/wrapped_reader_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
e668912a99
src/tools/futures.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
0f5856acca
src/tools/daemon.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
56e351c911
src/tools/broadcast_future.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
627bb7d114
src/tools/async_mutex.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
75fef4b463
src/server/worker_task.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
aa4110cc64
src/server/state.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
91e4587343
src/server/rest.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
b9b7f7ec1f
src/server/h2service.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
160fc8147f
src/server/command_socket.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
3c0facc787
src/client/remote_chunk_reader.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
369a87e3a2
src/client/pxar_backup_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
2107bb40c1
src/client/pipe_to_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
5b3911995b
src/client/merge_known_chunks.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
a6782ca10b
src/client/http_client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
8630557918
src/bin/upload-speed.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
cab6816969
src/bin/test_chunk_speed2.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
fda5797b8a
src/bin/proxmox-backup-proxy.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
e9722f8bde
src/bin/proxmox-backup-client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
e76ac3a499
src/bin/proxmox-backup-api.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
e235c8f719
src/bin/h2s-server.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
74be6dc9b7
src/bin/h2server.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
55d8a631fc
src/bin/h2s-client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
15d0e4a3bd
src/bin/h2client.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
b9203d87f4
src/bin/download-speed.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
745e652a7f
src/backup/index.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
32bef1e2d1
src/backup/chunk_stream.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
f2d9b73c91
src/api_schema/router.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
ffb6434485
src/api2/reader.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
7622005574
src/api2/backup/upload_chunk.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
59b2baa0f6
src/api2/backup.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
6be147b78c
src/api2/admin/datastore.rs: switch to async
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
5c1130df9f
implement Sync for DynamicIndexReader and FixedIndexReader
...
hyper's wrap_stream now needs this
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
236761a3e6
drop src/storage/futures.rs
...
it's unused and uses a tokio::Task I don't want to lookup
the std-future conversion for...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:17:58 +02:00
02fcf372e4
rc/bin/proxmox-backup-client.rs - restore: always download index.json.blob
2019-09-02 14:14:32 +02:00
0d9862803b
src/bin/proxmox-backup-client.rs - reatore: avoid loading (large) blobs into memory
2019-09-02 14:13:31 +02:00
66c8eb9383
src/client/http_client.rs: Refactor handling Option and Result types
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-29 09:15:16 +02:00
40f8680e37
tools: remove SignalFd
...
We don't use it anymore and it'll just be more work when
switching to async-await.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 13:44:53 +02:00
d21f8a5b5b
backup/dynamic_index: explicitly zero-initialize the header
...
The writer.write_all() call accessed data marked as
undefined to valgrind. Note that we shouldn't write out
uninitialized memory for security reasons anyway.
(note that vec::undefined already did zero-initialize the
data, but also marked it as undefined for valgrind when
compiling with the valgrind feature)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
990b930f22
backup/dynamic_index: use static assertion, fix size
...
The type was sized properly but the number was still wrong,
fixed this.
TODO! Once unions with non-Copy values are stable make this
a `union { full: [u8; 4096], data: TheActualHeader }`;
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 11:42:35 +02:00
bcb664cb69
pxar/fuse: add pxar cli flag to set single- or multi-threaded session loop
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:43:14 +02:00
de2791868a
src/pxar/sequentail_decoder.rs: followup: cleanup S_IFMT masking
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:42:22 +02:00
9cb9e1b7db
src/pxar/sequential_decoder.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:42:19 +02:00
dac88033ba
src/pxar/dir_stack.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:39:14 +02:00
b69169e7be
src/pxar/fuse.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:58 +02:00
d792dc3cd2
src/pxar/match_pattern.rs: style fixups
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:56 +02:00
4ea22b68d9
src/pxar/format_definition.rs: style fixup
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-23 09:37:53 +02:00
7d83440c60
remove proxmox-protocol subcrate
...
AFAICT we have no use for it anymore, its api entry points
are gone. If we do end up needing something from it, it's
still in the git history anyway. (And about two thirds of it
can be made much less awkward by utilizing async-await
anyway, so no love lost there...)
Moved the chunker back into src/backup/chunker.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 14:08:25 +02:00
7a57cb77e1
more formatting & use statement fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:50:27 +02:00
35dfd3ceb7
src/pxar/encoder.rs: style fixups by rustfmt
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:29:11 +02:00
1af30bc2fe
(pxar: replace deprecated std::mem::uninitialized()
...
... and use std::mem::MaybeUninit or proxmox::tools::vec::uninitialized() instead.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 13:27:25 +02:00
5e58e1bb7d
{dynamic,fixed}_index: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:35:36 +02:00
f93b55b057
src/backup/index: style fixup & unsafe copy removal
...
We can use the safe .copy_from_slice alternative in this
case.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:31:45 +02:00
fa17b1ce2a
src/backup/index: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:27:35 +02:00
f3a8d1d7e0
api2/node/time: replace mem::uninitialized
...
and fixup use statements
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:14:41 +02:00
48b85e8e3b
tools: tty: replace mem::uninitialized and style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:14:19 +02:00
b528fddfc6
tools: timer: style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:07:56 +02:00
b7f44ace92
timer: replace mem::uninitialized
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 11:07:10 +02:00
6100071f4e
tools: style & use statement fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-22 10:57:56 +02:00
5c20e2da6b
src/config: more style fixups
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:14:26 +02:00
91640ab567
src/config: style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:11:07 +02:00
928650c4eb
src/config: use statement cleanup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 14:10:50 +02:00
2ec979e4ed
/var/run -> /run
...
/var/run is considered deprecated and for instance in
systemd unit files lintian complains...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 12:28:24 +02:00
35238e234d
switch from create_dir_chown to create_path
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 12:26:10 +02:00
e5471b4836
pxar/decoder: avoid failing on hardlinks and return corresponding target file entry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
[Fixup: remove unnecessary 'mut']
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:32:52 +02:00
58262f40f7
src/pxar/decoder.rs: get correct stats for root directory
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:31:41 +02:00
7d26720e79
src/pxar/decoder.rs: move GOODBYE_ITEM_SIZE from function to module scope.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-21 09:31:39 +02:00
367f002ee2
use statement cleanup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-08-16 12:28:40 +02:00
9d135fe617
src/backup/catalog_blob.rs: moved catalog impl. from pxar
...
And avoid loading catalog into memory.
2019-08-16 12:27:17 +02:00
9025312aa6
avoid lifetimes in blob reader/writer
2019-08-16 10:07:24 +02:00
71d08e00b7
src/backup/data_blob_reader.rs: fix compiler warning
2019-08-16 08:10:39 +02:00
ac0b435344
src/pxar/fuse.rs: fix compiler warnings
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 15:14:57 +02:00
a84ef4c205
src/bin/proxmox-backup-client.rs: avoid loading catalog into memory
...
We can use the new DataBlobReader instead.
2019-08-14 15:07:28 +02:00
b791804f4b
src/backup/data_blob_reader.rs: impl for Read instead of BufRead
2019-08-14 14:56:01 +02:00
39a4df61d6
improve docs
2019-08-14 14:08:27 +02:00
018d11bb84
src/backup/data_blob.rs: move parts into single files
2019-08-14 13:24:41 +02:00
e27c5a559f
src/backup/data_blob.rs: avoid excessive stack usage
2019-08-14 12:44:08 +02:00
c638542b4b
src/backup/file_formats.rs: moved header_size
2019-08-14 12:35:53 +02:00
41d08aa6de
src/pxar/fuse.rs: enable libfuse debug output in verbose mode
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:59:15 +02:00
742e64ea0b
src/pxar/fuse.rs: cleanup callback interface and store decoder within session context
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:58:44 +02:00
7750b7f2b7
pxar: decoder: take ownership of underlying reader
...
By taking ownership it is easier to move the decoder into another struct,
e.g. into a session context in fuse.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:55:08 +02:00
f50b4fd6a0
src/pxar/decoder.rs: cleanup s/CaDirectoryEntry/DirectoryEntry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-14 08:54:08 +02:00
6082216343
src/backup/data_blob.rs - CryptReader/CryptWriter: fix buffer size handling
2019-08-13 15:23:15 +02:00
548c9489d7
src/backup/data_blob.rs - DataBlobReader: impl compressed, encrypted blobs
2019-08-13 13:17:08 +02:00
2aa0bfff59
src/backup/data_blob.rs - DataBlobReader: implement reader for encrtypted blobs
2019-08-13 13:17:08 +02:00
e9a385a78e
src/backup/data_blob.rs - DataBlobReader: impl compressed, signed blobs
2019-08-12 17:47:25 +02:00
4bfa147eaf
src/backup/data_blob.rs - DataBlobReader: impl. reader for signed blobs
2019-08-12 17:41:25 +02:00
09785b2795
src/backup/data_blob.rs - DataBlobReader: start serious impl.
2019-08-12 17:20:41 +02:00
a32bd8a516
src/backup/data_blob.rs - DataBlobWriter: fix decryption
2019-08-12 11:57:29 +02:00
5d15cb49b4
src/backup/data_blob.rs - compute_crc: start after blob header
2019-08-12 11:32:26 +02:00
1cbdfd19a2
src/backup/file_formats.rs; use const instzead of static
2019-08-12 11:20:21 +02:00
18be4ec24a
src/backup/data_blob.rs - DataBlobWriter: fix magic number for uncompressed blobs
2019-08-12 10:46:05 +02:00
5622a3fcdb
src/backup/data_blob.rs - DataBlobWriter: impl. compressed encrypted blobs
2019-08-12 10:16:42 +02:00
f4942e9ffd
src/backup/data_blob.rs - DataBlobWriter: impl. encrypted blobs
2019-08-12 10:08:00 +02:00
c57ec43a53
src/backup/crypt_config.rs: new helper data_crypter to get openssl Crypter
2019-08-12 10:06:51 +02:00
7776becf5d
src/backup/data_blob.rs - DataBlobWriter: impl. compressed signed blobs
2019-08-12 09:01:08 +02:00
706638f803
src/backup/data_blob.rs: impl. ChecksumWriter
...
To correctly compute crc,hmac of final data.
2019-08-12 08:53:53 +02:00
f796351c54
src/backup/data_blob.rs: impl. DataBlobWriter for authenticated blobs
2019-08-12 07:34:41 +02:00
cb0eea29d9
src/backup/crypt_config.rs: new method to get hmac signer
2019-08-12 07:33:15 +02:00
a762ce54fd
src/backup/data_blob.rs: make DataBlobWriter more generic
...
Allow to write compressed and uncompressed blobs.
2019-08-11 12:14:04 +02:00
1f26fdef19
src/backup/data_blob.rs: impl. simple reader/writer
...
To avoid loading blob into memory.
2019-08-11 11:32:36 +02:00
e240d8be0b
src/bin/proxmox-backup-client.rs: implement login/logout
2019-08-10 09:12:17 +02:00
f71e8cc96f
pxar: add command 'mount' to cli of pxar
...
Allows to mount an archive to a specified mountpoint via the cli.
Once the archive is mounted, the process is send to the background.
By passing the --verbose flag, the process is kept in foreground and
debug output is provided.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-09 16:46:16 +02:00
02ba8a5d11
pxar: add fuse module and expose its pub functionality.
2019-08-09 16:45:13 +02:00
c50f87442c
pxar: add basic code for FUSE implementation.
...
This adds the basic code in order to create a fuse session and mount an archive.
It adds libfuse3-3 as runtime dependency and libfuse3-dev as build dependency.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-09 16:42:31 +02:00
781ac11c6a
src/backup/data_blob.rs: define const MAX_BLOB_SIZE
2019-08-09 11:49:06 +02:00
9049a8cfe6
src/bin/proxmox-backup-client.rs: implement catalog command
2019-08-09 11:36:33 +02:00
7926a3a1cf
src/bin/proxmox-backup-client.rs: add archive filename to catalog
2019-08-09 11:05:56 +02:00
863be2e6e2
src/backup/data_blob.rs: allow largert blob sizes (128MB)
...
Need this for catalog files.
2019-08-09 10:22:56 +02:00
3fb53e07b4
src/bin/proxmox-backup-client.rs: avoid compiler warning
2019-08-09 10:22:32 +02:00
543a260ff9
src/bin/proxmox-backup-client.rs: new completion helper for snapshots
2019-08-09 10:08:45 +02:00
2761d6a4f3
src/pxar/encoder.rs: use BackupCatalogWriter
2019-08-09 09:46:49 +02:00
aea0815d32
src/pxar/catalog.rs: new catalog helper
2019-08-09 09:41:43 +02:00
977eeb24f6
src/bin/proxmox-backup-client.rs: avoid doubled file extensions (.blob.blob) in index.json
2019-08-07 10:23:08 +02:00
4af0ee055a
api: always use complete file names (including add exctensions)
2019-08-07 10:10:14 +02:00
d7c243977d
src/api2/admin/datastore.rs - list_snapshot_files: list all files
...
Add files that are not mentioned in index.json.
2019-08-07 09:16:14 +02:00
34066e2eeb
src/api2/backup/environment.rs - add_blob: always verify blob crc
2019-08-07 09:15:13 +02:00
eecb23560b
src/backup/data_blob.rs - encode: always compute crc
2019-08-07 08:30:27 +02:00
38a6cddad7
src/backup/backup_info.rs: add BackupInfo::new constructor
2019-08-07 08:27:52 +02:00
c4f025ebbf
src/bin/proxmox-backup-client.rs: fix remote filename completion
2019-08-07 06:48:18 +02:00
dbed4c8cd7
src/client/http_client.rs: compute checksums for chunk streams
2019-08-06 12:23:06 +02:00
bd15e96dd9
src/client/http_client.rs: minor code cleanup
2019-08-06 11:46:15 +02:00
f889b15867
src/backup/data_blob.rs: always compute crc
2019-08-06 11:42:14 +02:00
c807d231eb
src/client/http_client.rs: compute file checksums
2019-08-06 11:26:06 +02:00
a17a0e7a9f
src/api2/admin/datastore.rs: extract backup size from index.json
2019-08-06 10:56:21 +02:00
eecb182845
pxar: add initial docs for MatchPattern
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-06 09:03:02 +02:00
43e892d293
pxar: add error handling for MatchPattern::matches_filename()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-06 09:02:44 +02:00
8c70e3eb18
src/api2/admin/datastore.rs - files: return data from index.json
2019-08-05 13:22:19 +02:00
f69adc81a6
use new proxmox::tools::nodename
2019-08-03 17:06:23 +02:00
7f66c29e5c
remove src/tools/common_regex.rs, use proxmox::tools::common_regex instead
2019-08-03 16:44:31 +02:00
fd329bdcd7
remove src/tools/procfs.rs, use proxmox::sys::linux::procfs instead
2019-08-03 16:26:44 +02:00
779e4c90b1
src/tools/vec.rs: remove unused file
...
We already moved that to proxmox::tools
2019-08-03 16:11:47 +02:00
e18a6c9ee5
update to nix 0.14, use code from proxmox:tools
2019-08-03 13:05:38 +02:00
b86f263ced
use new proxmox::sys::linux::magic
2019-08-03 09:28:57 +02:00
4ea0f7d95b
pxar: remove flag to change digest algorithm
...
The flag CA_FORMAT_SHA512_256 is used to switch between sha512 and sha256 to
calculate digest in casync.
As we use sha256, we can get rid of this flag for now.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:57:53 +02:00
4d142ea79e
pxar: cleanup: refactor and rename exclude pattern
...
The original name PxarExcludePattern makes no sense anymore as the patterns are
also used to match filenames during restore of the archive.
Therefore, exclude_pattern.rs is moved to match_pattern.rs and PxarExcludePattern
rename to MatchPattern.
Further, since it makes more sense the MatchTypes are now declared as None,
Positive, Negative, PartialPositive or PartialNegative, as this makes more sense
and seems more readable.
Positive matches are those without '!' prefix, Negatives with '!' prefix.
This makes also the filename matching in the encoder/decoder more intuitive and
the logic was adapted accordingly.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:52:32 +02:00
fe076c8259
pxar: cleanup: s/PxarDirBuf/PxarDirStack/g and move dir_buffer.rs to dir_stack.rs
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:52:10 +02:00
ead7546a96
pxar: change uid and gid to u32 instead of u64
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:51 +02:00
5e50c606b0
pxar: cleanup: s/CA_FORMAT/PXAR/g and s/CaFormat/Pxar/g
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:33 +02:00
47651f9530
pxar: cleanup: move feature flags to src/pxar/flags.rs and omit CA_FORMAT prefix on all of them
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:51:06 +02:00
91451c4355
pxar: cleanup: move filesystem related magic numbers to tools/fs/magic.rs
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-03 08:48:12 +02:00
b335f5b713
src/bin/proxmox-backup-client.rs: sign index.json
2019-08-02 10:00:15 +02:00
69ecd8d5b0
src/backup/data_blob.rs: implement signed blobs
2019-08-02 09:56:01 +02:00
93205f942a
src/backup/crypt_config.rs: new compute_auth_tag helper
2019-08-02 08:55:37 +02:00
c68d2170d5
src/backup/crypt_config.rs: fix typo
2019-08-02 08:29:40 +02:00
81a9905e0a
pxar: implement feature flag support for device nodes, fifos and sockets
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:55:17 +02:00
8abc95a145
pxar: remove some unused feature flags from format definition
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:54:46 +02:00
3a905cac2b
src/pxar/sequential_decoder.rs: remove of obsolete function dir_mkdirat
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:52:49 +02:00
beffac999f
src/pxar/sequentail_decoder.rs: fix issue when restoring with glob pattern.
...
Partial extraction of an archive with a glob pattern, e.g. '**/*.conf' lead to
the unexpected behaviour of restoring all partially matched directories (in this
example all of them).
This patch fixes this unexpected behaviour by only restoring those directories
were the directory or one of its sub-items fully matched the pattern and should
therefore be restored.
To achive this behavoiur, directory metadata is pushed onto a stack and restored
on demand.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:49:47 +02:00
51ac99c314
src/pxar/exclude_pattern.rs: add Clone and Copy trait to MatchType
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:49:47 +02:00
6a584cfd76
pxar: add PxarDir and PxarDirBuf to buffer directory metadata
...
In order to restore only directories when some of their content fully matched
a match pattern on partial restores, these directories and their metadata are
pushed onto this buffer and only restored successivley on demand.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:45:12 +02:00
27e6e180b6
src/pxar/format_definition.rs - PxarAttributes: derive Default
...
And remove unnecessary impl new().
2019-08-02 07:16:10 +02:00
a92e6c9011
src/pxar/sequential_decoder.rs: split read and restore of attributes into separate functions
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:11:38 +02:00
e60cfbcdd0
src/pxar/format_definition.rs: add struct PxarAttributes
...
This structure contains all the attributes allowing to easily store those within
a e.g. dir buffer.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 07:10:22 +02:00
2be3eff5c3
pxar: avoid taking ownership of xattrs, fcaps and quota_projid on corresponding restore functions.
...
By borrowing these objects we preserve the functionality but make sure
that ownership doesn't change, avoiding problems when contained within other
structs such as e.g. a buffer storing these attributes.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-08-02 06:44:00 +02:00
2c3891d1c3
src/bin/proxmox-backup-client.rs: upload backup index.json
...
The plan is to use this file to verify the backup content.
2019-08-01 12:39:02 +02:00
953d5e1531
src/tools.rs - file_set_contents_full: only call fchmod when we pass permissions
...
So that we can use it to write into /etc/pve/ (which does not support chmod).
2019-07-31 12:44:27 +02:00
cec17a3ec7
src/bin/proxmox-backup-client.rs - files: allow --repository parameter
2019-07-31 11:45:55 +02:00
52c171e49d
src/bin/proxmox-backup-client.rs: implement files command
2019-07-31 11:23:53 +02:00
43a406fd3f
src/bin/proxmox-backup-client.rs: supress useless output from prune command
2019-07-31 10:15:16 +02:00
9731c8b862
pxar: add support for storing/restoring the quota project id on ZFS
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-30 09:40:48 +02:00
6a87910949
pxar: implement allow_existing_dirs for pxar decoder
...
By default, restoring an archive will fail if files with the same filename
already exist in the target directory.
By setting the allow_existing_dirs flag, the restore will not fail if an
existing directory is encountered.
The metadata (permissions, acls, ...) of the existing directory will be set
to the ones from the archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-30 09:40:30 +02:00
46d5aa0a09
src/bin/proxmox-backup-client.rs: add restore flag --allow-existing-dirs
...
Not jet implemented ...
2019-07-29 12:49:15 +02:00
ea7a7ef299
src/bin/proxmox-backup-client.rs: fixup - correctly pass prune parameters
2019-07-27 09:24:23 +02:00
9fdc3ef46f
src/api2/admin/datastore.rs: pass backup type/id (group) to prune
...
IMHO, prune over all backup groups is a bit dangerous, considering
that more than one user might use a datastore.
2019-07-27 08:49:14 +02:00
7a6cfbd98d
src/bin/proxmox-backup-client.rs: correctly compute duration
...
And format time with SecondsFormat::Secs
2019-07-26 11:13:49 +02:00
bbf9e7e951
src/api2/backup.rs: cleanup schema definitions
2019-07-26 09:07:29 +02:00
ca5d0b61ca
src/api2/backup.rs: new required backup-time parameter
...
The client should pass the time as parameter.
2019-07-25 13:44:01 +02:00
e128d4e84f
src/api2/admin/datastore.rs: backup logs may not be written twice
2019-07-25 12:29:04 +02:00
ef3254c086
src/api2/admin/datastore.rs: use correct .blob extension for log
2019-07-25 12:22:50 +02:00
ec34f7eba0
src/bin/proxmox-backup-client.rs: implement upload-log
2019-07-25 12:18:15 +02:00
04512d3068
src/client/http_client.rs: allow upload with parameters
2019-07-25 12:17:35 +02:00
07ee223590
src/api2/admin/datastore.rs: add api to upload backup client log file
2019-07-25 12:15:30 +02:00
79679c2d6e
src/bin/proxmox-backup-client.rs: allow to uploag .log files (as binary blobs)
2019-07-24 13:07:02 +02:00
5b72c9b4f7
src/pxar/encoder.rs: add new skip_lost_and_found parameter
2019-07-24 12:21:25 +02:00
6c3c9bceb5
pxar_backup_stream.rs: limit lock scope to avoid blocking forever
2019-07-24 11:30:43 +02:00
684233aa3b
remove debug output
2019-07-24 09:33:52 +02:00
5be106eeae
src/client/pxar_backup_stream.rs: correctly pass errors to stream
2019-07-24 09:24:35 +02:00
2eeaacb974
src/pxar/encoder.rs: allow to pass list of devices
...
For better mount point include control...
2019-07-24 08:11:59 +02:00
fa5d6977dd
Revert previous, commit, use UTC RFC3339 without timezone (Z)
...
We now have human readable data/time, and names are still sortable.
2019-07-22 11:49:30 +02:00
10c2a21cfe
src/backup/backup_info.rs: use unix epoch timestamps instead of RFC3339
...
RFC3339 contains additional timezone information, but IMHO this just adds
more confusion. The API also uses timestamps.
2019-07-22 07:57:03 +02:00
1fdef50f52
src/pxar/sequential_decoder.rs: Call callback only on restored paths.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-19 11:58:51 +02:00
40a13369a3
src/bin/pxar.rs - extract: print archive name with --verbose flag
2019-07-19 07:01:57 +02:00
bbd4365c61
src/bin/pxar.rs: remove wrong debug message
2019-07-19 07:00:17 +02:00
a0ec687cd7
src/bin/pxar.rs: allow to pass paths and match patterns as args to pxar extract
...
To improve usability it is now possible to directly pass paths or match patterns
as arguments to pxar extract to partially restore an archive.
The patterns provided via CLI are appended to the ones read from file by the
--files-from option in order to have priority over those.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-19 06:50:39 +02:00
9aa3f68278
src/bin/proxmox-backup-client.rs: code cleanup
2019-07-18 13:16:32 +02:00
f6ede796ca
src/bin/proxmox-backup-client.rs - status: only print result[data]
2019-07-18 09:52:11 +02:00
15c847f136
src/api2/admin/datastore.rs: allow to list all snapshots
2019-07-18 09:11:43 +02:00
0f73ee73bb
src/cli/command.rs: set exit status on error
2019-07-17 13:40:10 +02:00
129dda4729
src/bin/pxar.rs: add --files-from parameter to pxar extract
...
Allows to pass a list of match pattern to the pxar extract call by reading them
from file.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:20:29 +02:00
fa7e957ca3
pxar: Adapt SequentialDecoder::restore() calls to pass empty match pattern for full restore
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:19:31 +02:00
d32c2e4d27
src/pxar/sequential_decoder.rs: adapt code for partial restore by match pattern
...
Allows to partially restore an archive by passing match patterns to the restore
function.
The whole restore is performed in sequential, therefore the whole archive has to
be read.
By wrapping the RawFd into an Option it can be controlled if the corresponding
part is restored (in case of Some(fd)) or if the Reader reads over it
without restore (in case of None).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:11:37 +02:00
4902291673
src/pxar/sequential_decoder.rs: implement match_filename to match include pattern for partial restores
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:07:54 +02:00
894cd49ac4
src/pxar/sequential_decoder.rs: Also check for nul bytes when reading the filename
...
Check if the filename does not contain invalid nul byes when reading it from the
archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 12:04:41 +02:00
0b39674895
src/pxar/sequential_decoder.rs: remove old comments and refactoring
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:46:02 +02:00
9f8fcdd050
src/pxar/encoder.rs: refactoring of match_exclude_pattern()
...
match_exclude_pattern() does not need a '&mut self' reference to the encoder,
move it therefore out of the impl.
Further, this patch contains some naming and formatting cosmetics.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:45:19 +02:00
3ff4ef28e6
src/pxar/encoder.rs: remove unneeded dir_count
...
dir_count was used to track the number of directory entries to store in the
archive and bail if the maximum is exceeded.
As the number of entries is equally obtained from the list of the filenames to
include, use that one instead.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:44:59 +02:00
a771f90745
src/pxar/exclude_pattern.rs: do not return partial matches for files
...
Partial matches make only sense for directories, files are always leafs of the
tree. Take this into account in order to avoid restoring of files which only
matched the front of a match pattern.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:44:06 +02:00
bdf0d82ced
src/bin/pxar.rs: Make pxar extract target optional
...
In order to improve usablity, the target on archive extraction will be the
current working directory by default.
A different target can be provided via the optional --target <PATH> parameter.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-17 09:40:02 +02:00
34a816cc7b
src/bin/proxmox-backup-client.rs: add output-fromat options, implement status api
2019-07-16 13:35:25 +02:00
0eecf38fbf
src/api2/admin/datastore.rs: add status api call
2019-07-16 13:34:38 +02:00
25500cfa26
src/cli/command.rs: add helpers/schema to format output
2019-07-16 11:46:58 +02:00
aad2ee4947
src/bin/pxar.rs: only print archive name in verbose mode
2019-07-16 08:38:02 +02:00
6b9a071028
src/bin/pxar.rs: add --verbose parameter to list (replace dump)
2019-07-16 08:31:40 +02:00
2665cef703
src/bin/proxmox-backup-client.rs: make repository parameter optional
...
And allow to read it from $ENV{PBS_REPOSITORY}
2019-07-16 07:30:04 +02:00
3f5192730e
src/pxar/format_definition.rs: fix typo
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-15 10:49:34 +02:00
9eae781ab8
src/bin/pxar.rs - extract: allow to read from stdin
2019-07-05 12:39:25 +02:00
c6c9e093d0
src/bin/pxar.rs - dump: allow to read from stdin
2019-07-05 12:32:15 +02:00
bf1252619a
src/bin/proxmox-backup-client.rs - restore: allow to pipe output to stdout
2019-07-05 12:14:50 +02:00
f8100e9623
src/bin/proxmox-backup-client.rs: remove download cli, do everything in restore
2019-07-05 11:36:45 +02:00
f4bf7dfcc7
src/client/remote_chunk_reader.rs: implement simple caching
2019-07-05 10:42:46 +02:00
afb4cd28be
src/backup/fixed_index.rs: implement BufferedFixedReader
2019-07-05 09:19:56 +02:00
be9b39e17a
src/pxar/sequential_decoder.rs: major refactoring of restore_sequential()
...
This splits the functionality of restore_sequential() into several smaller
functions in order to allow to reuse them when restoring by seeking based on
the goodbye table offsets.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:45:50 +02:00
48a4509c67
src/pxar/encoder.rs: fix bug with incorrect offset stored in goodbye table
...
Commit cd7dc87903
2019-07-05 06:39:39 +02:00
7dcbe051e9
pxar: Refactor SequentialDecoder to store the callback function within the struct
...
Reduces the number of arguments for the function calls within the decoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-05 06:38:22 +02:00
b46c3fad03
src/backup/fixed_index.rs: use correct size
...
We need to consider that the last chunk may have smaller size.
2019-07-04 15:13:22 +02:00
9cc88a7c00
src/client/http_client.rs - finish: hold reference to avoid early connection close
2019-07-04 15:11:23 +02:00
5e04ec707d
src/api2/backup/environment.rs: improve chunk alignment/size checks
2019-07-04 13:40:43 +02:00
a24e3993e0
src/backup/chunk_store.rs: coding style fixes
2019-07-04 11:39:10 +02:00
e4c2fbf170
src/backup/chunk_store.rs: additionally log chunk count
2019-07-04 11:27:11 +02:00
9850bcdf19
src/backup/chunk_store.rs: improve error reporting
2019-07-04 11:21:54 +02:00
a3f3e91da2
backup/chunk_store: rework chunk iterator
...
We can now use iter::from_fn() which makes for a much nicer
logic. The only thing better is going to be when we can use
generators with `yield`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-04 10:55:17 +02:00
99168f43e6
src/client/http_client.rs: s/set_recv_buf_size/set_recv_buffer_size/
2019-07-04 10:51:18 +02:00
c698636a5d
src/tools/fs.rs - read_dir: return nix::Result
2019-07-04 10:47:16 +02:00
a57360983b
src/backup/chunk_store.rs - get_chunk_iterator: return percentage inside iterator item
2019-07-04 09:26:44 +02:00
0f0a35b390
src/backup/dynamic_index.rs: lock file inside new, code cleanup
2019-07-04 08:17:30 +02:00
a7c72ad9eb
src/backup/fixed_index.rs - FixedIndexReader: remove reference to chunk store
2019-07-04 08:13:16 +02:00
a660978c9a
src/backup/datastore.rs: generic index_mark_used_chunks implementation, improve GC stats
2019-07-04 07:57:43 +02:00
86eda3eb0d
src/bin/proxmox-backup-client.rs: implement restore using BackupReader
2019-07-03 15:45:12 +02:00
b2a6897982
src/client/remote_chunk_reader.rs: remove dead code
2019-07-03 14:56:24 +02:00
7f99bf691a
src/client/remote_chunk_reader.rs: implement remote chunk reader
2019-07-03 14:39:13 +02:00
4f6aaf542c
src/client/http_client.rs: avoid compiler warning
2019-07-03 14:36:02 +02:00
3467cd91e9
src/client/http_client.rs: return Arc<Self>
...
Avoid to call canceller for each clone on drop ...
2019-07-03 14:26:07 +02:00
1e0784841c
src/backup/read_chunk.rs: use &mut self
2019-07-03 14:25:18 +02:00
0d70535a36
src/pxar/encoder.rs: refactor simple match statements to map_err()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:36 +02:00
6e1c31163b
src/pxar/sequential_decoder.rs: refactoring of simple match statements to map_err()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:18:24 +02:00
9d227350a3
src/pxar/sequential_decoder.rs: Cleanup and fix incorrect error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-07-03 12:16:59 +02:00
80af046794
src/server/rest.rs: avoid unwrap
2019-07-03 12:00:43 +02:00
7fb4f5642a
src/server/rest.rs: log peer address, use hyper MakeService
2019-07-03 11:54:35 +02:00
6d1f61b208
use hyper/tokio-openssl instead of hyper/tokio-tls
...
This exposes the complete SSL setup. And download is much faster
now (600MB/s instead of 130MB/s)!
2019-07-02 13:36:28 +02:00
74d0a6bc23
src/bin/proxmox-backup-proxy.rs: set socket options for max. performance
2019-07-02 10:48:58 +02:00
93f8bb0445
src/bin/h2s-client.rs: avoid compiler warnings
2019-07-02 08:59:33 +02:00
877dd11133
src/backup/{fixed, dynamic}_index.rs: improve error messages
2019-07-02 08:56:56 +02:00
5c593a4d56
src/backup/read_chunk.rs: use Arc for DataStore
2019-07-02 08:49:16 +02:00
b850673634
src/backup/read_chunk.rs: move read chunk trait into extra file
...
And implement LocalChunkReader.
2019-07-02 08:22:29 +02:00
8fad30a4b1
use latest hyper master branch to include socket buffer size fix
2019-07-02 07:09:46 +02:00
f35197f449
replace tools::vec with proxmox::tools::vec
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 11:03:25 +02:00
d0162d53d3
tools: remove io module
...
now completely replaced by proxmox::tools::io.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:12 +02:00
9b2b627fe0
update remaining users of tools::io::ops
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:44:03 +02:00
9110a69bd8
tools: remove read/write
...
now completely replaced by
proxmox::tools::io::{ReadExt, WriteExt}
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:42 +02:00
ca3c3ce997
daemon: remove last use of tools::read/write
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:39:13 +02:00
5485b579a1
backup: use new proxmox::tools::io traits
...
This replaces all `write_value` calls with `write_le_value`
calls!
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-07-01 10:37:02 +02:00
dc9775d1b0
src/bin/h2s-client.rs: increase tokio tcp stream read buffer size
...
This improves download speed by factor 10.
2019-06-29 18:09:50 +02:00
34f7131160
src/bin/h2s-client.rs: avoid hyper, use h2 directly
...
But performance is still bad.
2019-06-29 15:58:18 +02:00
5ed2277f0e
src/bin/h2s-server.rs: test to reproduce slow h2 download
...
Simplified code, use with h2s-client.rs.
2019-06-29 14:57:54 +02:00
17243003b3
src/bin/download-speed.rs: h2 download speedtest for our server
...
I get about 30MB/s (much too slow)
2019-06-29 13:43:10 +02:00
a2b29b68d4
src/client/http_client.rs: use maximal h2 window sizes
2019-06-29 11:05:36 +02:00
872debdefa
src/bin/h2client.rs: tune h2 client parameters to get reasonable speed
...
We now get about 2.5GB/s (not really good, but better)
2019-06-29 10:41:24 +02:00
38294bcc56
src/client.rs: make module put_to_stream public
2019-06-29 10:40:43 +02:00
fded1f3154
add code to test H2 speed
...
I currently get about 1MB/s, which is ways too slow (must be a bug)?
2019-06-29 10:08:12 +02:00
d48a9955a5
src/backup/dynamic_index.rs: introduce ReadChunk trait
2019-06-28 16:35:00 +02:00
09d7dc5024
src/api2/reader.rs: add download chunk api
2019-06-28 16:27:01 +02:00
fcf5dea56d
src/client/http_client.rs: increase h2 window size, set tcp nodelay option
...
To increase h2 download speed.
2019-06-28 16:00:58 +02:00
81a6ce6fde
src/backup/chunk_store.rs: new method chunk_path()
...
Returns the absolute path.
2019-06-28 15:48:09 +02:00
fcfb84fedf
file download: avoid unnecessary copy
2019-06-28 07:07:52 +02:00
984a7c3502
src/client/http_client.rs - h2 download: implement flow control
2019-06-28 07:02:43 +02:00
dd066d28e2
src/api2/reader.rs: implement backup reader protocol
2019-06-27 09:01:41 +02:00
42a87f7b96
src/server/h2service.rs: implement generic h2 service
2019-06-26 17:38:33 +02:00
fb0470837b
src/client/http_client.rs: split out code to start new h2 connections
2019-06-26 12:09:18 +02:00
b208da8393
src/backup/data_{chunk,blob}.rs: add verify_crc method
2019-06-26 09:54:25 +02:00
fef44d4f78
src/bin/proxmox-backup-client.rs: decode blobs on download
...
No more need for that clumsy blob cli (removed).
2019-06-26 09:18:59 +02:00
c2b945341c
src/client/http_client.rs - download: use generic Write type, return writer.
...
Make it possible to write int Vec<u8>.
2019-06-26 09:17:13 +02:00
bb8231409e
src/backup/crypt_config.rs - generate_rsa_encoded_key: store as json
...
Use the KeyConfig serialization with kdf = None.
2019-06-26 07:32:34 +02:00
3031e44c58
src/bin/proxmox-backup-client.rs: remove stale download file, start blob cli
2019-06-25 13:00:37 +02:00
f14a8c9a85
src/api2/admin/datastore.rs: use correct path for download
2019-06-25 12:59:36 +02:00
81820b0d4d
src/client/http_client.rs - download: fix error handling
2019-06-25 12:43:55 +02:00
45db6f89e1
src/bin/proxmox-backup-client.rs: implement download command
2019-06-25 11:17:24 +02:00
9e47c0a56e
src/api2/admin/datastore.rs: implement generic download method
...
We simply allow to download raw binary data.
2019-06-25 10:16:59 +02:00
da7d67213e
src/api2/admin/datastore/pxar.rs: remove pxar upload api
...
Upload needs to be done using the backup protocol.
2019-06-25 09:45:35 +02:00
dd8e744f9a
src/api2/admin/datastore.rs: use a worker log log prune actions
2019-06-25 08:12:25 +02:00
6899dbfb47
src/bin/proxmox-backup-client.rs: strip .blob extension
2019-06-25 07:11:02 +02:00
c6d203bbd3
src/backup/backup_info.rs: also list .blob files
2019-06-25 07:07:45 +02:00
045e8a568d
src/bin/proxmox-backup-client.rs: change key name to rsa-encrypted.key
2019-06-25 07:00:07 +02:00
79bc7345e8
src/cli/command.rs: fix ReST output format
2019-06-25 06:19:51 +02:00
9f46c7de4b
src/bin/proxmox-backup-client.rs: upload rsa encoded key after backup
2019-06-24 13:56:37 +02:00
cd7dc87903
pxar: impl .pxarexclude parsing and exclude matching
...
.pxarexclude files allow to exclude or include parts of a subtree by matching
with a glob pattern. The globs are used according to the matches of fnmatch.
In addition '**' can be used to match multiple directories within the path.
Order of the entries matter, as later ones win over previous ones.
As the .pxarexclude files can be placed at any node of the directory hirarchy,
this implies that matching child entries win over parent entries.
The only exception to this behaviour is, when a parent entry already fully
matched the path, thereby excluding the child entries which would match
otherwise.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-24 10:26:20 +02:00
46bd880041
src/api2/backup/environment.rs: new helper add_blob()
2019-06-24 09:35:37 +02:00
cb08ac3efe
src/api2/backup.rs: replace upload_config with upload_blob
2019-06-23 11:44:30 +02:00
a38c5d4d12
src/backup/data_blob.rs: implement from_raw()
2019-06-23 09:35:44 +02:00
6f083b7a92
src/api2/backup/upload_chunk.rs: compute crc on upload
2019-06-23 09:04:42 +02:00
ba01828d38
try to use use proxmox::tools::io::ops::ReadExtOps
2019-06-22 16:29:10 +02:00
27042ce637
src/backup/file_formats.rs: improve docs
2019-06-22 15:45:36 +02:00
9f83e0f7ee
src/backup/crypt_config.rs: simplify decode interface
2019-06-22 13:24:29 +02:00
0066c6d972
src/backup/crypt_config.rs: remove encode_chunk, use encrypt_to instead
2019-06-22 13:02:53 +02:00
ee8a7e8080
src/backup/crypt_config.rs: implement encrypt_to (output to writer)
2019-06-22 12:25:04 +02:00
991abfa8b4
src/backup/file_formats.rs: split out file format data
2019-06-22 09:12:25 +02:00
b7f4f27d6c
add crc field for binary blobs formats
2019-06-21 18:49:04 +02:00
a7f67a9a9c
depend on crc32fast
...
We can use this to compute fast checksums to test file integrity (not crypto safe).
2019-06-21 15:58:00 +02:00
3025b3a53c
src/backup/data_blob.rs: new file format for binary blobs
2019-06-21 11:32:07 +02:00
077a8cae63
src/backup/crypt_config.rs: make code more flexible
...
Allow to pass magic numbers as parameters.
2019-06-21 10:50:48 +02:00
c38266c18d
renamed: src/backup/crypt_setup.rs -> src/backup/crypt_config.rs
2019-06-21 09:51:18 +02:00
3ea8bfc9c0
src/bin/proxmox-backup-client.rs: add import-master-pubkey command
2019-06-21 07:34:17 +02:00
37c5a17505
src/bin/proxmox-backup-client.rs: helper to generate RSA master key
2019-06-21 07:12:19 +02:00
ab44acff57
src/backup/key_derivation.rs: add modified field to key file
2019-06-20 07:58:33 +02:00
6d0983dbe1
src/bin/proxmox-backup-client.rs: add keyfile parameter for backup
2019-06-19 17:16:41 +02:00
181f097af9
src/bin/proxmox-backup-client.rs - key API: pass kdf parameter
...
Allow to store keys without password.
2019-06-18 18:21:01 +02:00
9b06db4563
src/bin/proxmox-backup-client.rs: use file system path to reference keys
2019-06-18 17:38:50 +02:00
091540298e
src/bin/proxmox-backup-client.rs - create_key: always read password from tty
2019-06-18 17:27:52 +02:00
826f309bf5
src/backup/key_derivation.rs: move kdf code into separate file
2019-06-18 11:17:22 +02:00
11515438cc
Cargo.toml: use serde feature derive
2019-06-18 06:23:25 +02:00
ac71623483
src/bin/proxmox-backup-client.rs: implement "key change-passphrase"
2019-06-17 13:10:00 +02:00
f2401311b0
src/bin/proxmox-backup-client.rs: start key management
2019-06-17 10:33:24 +02:00
a7dd483097
src/backup/*_index.rs: used generated magic numbers
2019-06-14 14:58:37 +02:00
bffd40d6b7
src/tools.rs: move hex_to_digest and digest_to_hex to proxmox::tools
2019-06-14 11:40:04 +02:00
00388226e6
src/api2/backup/environment.rs: log index checksums
2019-06-14 10:36:20 +02:00
9335d74eba
src/backup/fixed_index.rs: compute checksum over the index
2019-06-14 10:34:29 +02:00
16ff6b7cd8
src/backup/dynamic_index.rs: compute checksum over the index
2019-06-14 10:33:59 +02:00
36075475a6
src/api2/backup/environment.rs: log duplicate chunks
2019-06-14 07:12:30 +02:00
fa148dbd11
src/api2/backup/upload_chunk.rs: verify chunk data if possible
2019-06-14 06:23:46 +02:00
0f37577092
src/bin/cipherbench.rs: add test for zstd compression
2019-06-13 17:24:57 +02:00
9399c98f82
src/bin/cipherbench.rs: cipher speed test with large blocks
2019-06-13 17:16:43 +02:00
f640f23ac5
src/backup/data_chunk.rs: fix typo
2019-06-13 16:42:55 +02:00
dba72d5016
src/backup/data_chunk.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:56:50 +02:00
68ab376ae9
src/backup/crypt_setup.rs: only use compressed data if it is shorter than uncompressed
2019-06-13 12:47:54 +02:00
f98ac774ee
backup: Add support for client side encryption
...
first try ...
2019-06-13 11:47:23 +02:00
51929e4532
src/backup/data_chunk.rs - decode: make crypt_config optional
2019-06-13 11:44:38 +02:00
046bcb9773
src/backup/crypt_setup.rs: add comment for possible improvements
2019-06-13 07:35:59 +02:00
bd0e3c7cfa
src/backup/data_chunk.rs: Moved ChunkInfo from merge_known_chunks.rs
2019-06-13 07:35:02 +02:00
6762db70d6
src/api2/types.rs: define CHUNK_DIGEST_SCHEMA
2019-06-13 07:30:42 +02:00
b595cb9d2c
src/backup/data_chunk.rs: Data Chunk builder
...
Make handling encoded chunks easier.
2019-06-12 13:37:01 +02:00
ef27200cec
src/backup/crypt_setup.rs: fix encoder for uncompressed chunks
2019-06-10 09:56:06 +02:00
74792b95b2
src/backup/crypt_setup.rs: allow compressed and uncompressed chunks
2019-06-10 08:27:35 +02:00
9e0187a203
src/backup/crypt_setup.rs: remove unused CryptData struct
2019-06-10 07:55:36 +02:00
bec8498ae3
src/backup/crypt_setup.rs: avoid useless allocations
2019-06-09 11:44:17 +02:00
c20b17b460
src/backup.rs: define magic numbers for chunk store file format
2019-06-08 10:34:20 +02:00
48b4b40b57
src/backup/crypt_setup.rs: crypto helpers
2019-06-08 09:56:51 +02:00
dd5495d6dc
tree-wide: use 'dyn' for all trait objects
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-07 13:13:48 +02:00
e993db91cd
pxar: introduce fs_feature_flags obtained from filesystem magic in Encoder
...
Not all filesystems support features such as xattrs,acl,... and trying to get
them is rather expensive.
By getting the supported features based on the filesystem magic and masking the
user set feature flags, unsupported features are excluded rather inexpensively
while encoding the archive.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:20 +02:00
1f319e766e
src/pxar/encoder.rs: refactoring: use ioctl calls from src/tools/fs.rs instead of local
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:08:06 +02:00
9b77388f2a
src/pxar/encoder.rs: minor refactoring
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:55 +02:00
a4cc8eb7c5
src/pxar/encoder.rs: check if ioctl supported by filesystem
...
Reading the quota project id relies on a ioctl call to get fsxattr.
On FUSE filesystems, ioctl calls might not be supported and will fail with
an errno indicating no support.
For these cases, the error is ignored and the default project id is used
(indicated by returning Ok(None)).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-07 11:07:31 +02:00
4800df7ba6
src/api2/admin/datastore.rs: remove test-upload api
2019-06-07 07:52:11 +02:00
113898d7e9
src/api2/admin/datastore/upload.rs: remove unused upload api
2019-06-07 07:39:41 +02:00
4247fccb0f
src/client/http_client.rs: automatically close connection in finish
2019-06-05 11:41:50 +02:00
cb4426b348
make h2 client connection cancellable
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-06-05 09:45:59 +02:00
986bef16be
src/backup.rs; use a macro to define PROXMOX_BACKUP_PROTOCOL_ID_V1
...
So that we can include it in static doc strings.
2019-06-05 08:41:20 +02:00
c9ec0956cf
src/backup.rs: define const PROXMOX_BACKUP_PROTOCOL_ID_V1
2019-06-05 08:12:13 +02:00
44c54845b3
src/tools/xattr.rs: fix test when run as root
2019-06-05 07:57:42 +02:00
7773ccc11f
src/api2.rs: move backup api to /backup
2019-06-05 07:23:21 +02:00
97eeea3b4a
src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:29:02 +02:00
1ce2f7cfb9
src/pxar/encoder.rs: style fix
2019-06-05 06:27:39 +02:00
e7b60a16c8
pxar: impl storing/dumping/restoring of quota project ids
...
Allows to store/dump/restore the quota project id associated with an inode in
order to correctly restore project quotas.
The project id is obtained/set via ioctl calls getting/setting the fsxattr
associated with the given file descriptor.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:26:02 +02:00
7312ab9eef
src/pxar/format_definition.rs: add CA_FORMAT_QUOTA_PROJID and CaFormatQuotaProjID
...
Defines the types needed to store/restore quota project ids in pxar
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:23:06 +02:00
042babe4e7
src/tools/fs.rs: impl ioctl calls to get/set fsxattr
...
This implements fs_ioc_fsgetxattr/fs_ioc_fssetxattr calls in order to read or
write fsxattr for a given file descriptor.
This is needed in order to read or write the quota project id for filesystems
which support project quotas (EXT4/XFS/FUSE).
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:22:49 +02:00
22e5aa1d3e
src/bin/dump-backup-api.rs: helper to generate backup API docs
2019-06-04 13:12:42 +02:00
5e0f305142
src/api_schema/format.rs: depend on textwrap, new wrap_text helper
2019-06-04 12:42:02 +02:00
339ddfcbfa
src/cli/command.rs: move doc generator code to src/api_schema/format.rs
2019-06-04 12:32:22 +02:00
ef39bf95b3
src/client/http_client.rs - upload_config: improve error message
2019-06-03 10:53:09 +02:00
ec8a9bb953
src/bin/proxmox-backup-client.rs: implement upload_config
2019-06-03 10:39:44 +02:00
39d6846e1e
src/api2/admin/datastore/backup.rs: implement config file upload
2019-06-03 09:45:14 +02:00
1c0472e88f
avoid compiler warnings
2019-06-03 06:43:47 +02:00
36898ffce6
src/backup/chunk_stream.rs: add optional chunk_size parameter
2019-05-30 13:28:24 +02:00
49ef316bcd
src/bin/proxmox-backup-client.rs: backup_image() - pass correct chunk size
2019-05-30 12:52:12 +02:00
202cadc29e
src/bin/proxmox-backup-client.rs: remove unused code
2019-05-30 12:47:35 +02:00
6af905c18f
src/bin/proxmox-backup-client.rs: implement image/block-device upload
2019-05-30 12:46:01 +02:00
eb1804c567
src/bin/proxmox-backup-client.rs: use std lib instead of nix
2019-05-30 12:19:23 +02:00
6e907505aa
src/bin/proxmox-backup-client.rs: fix file type test
2019-05-30 12:08:34 +02:00
96482891ae
src/api2/admin/datastore/backup/environment.rs: log upload statistics
2019-05-30 09:21:19 +02:00
642322b433
src/api2/admin/datastore/backup.rs: use separate api entry points for chunk upload
...
So that we can provide better statistics (patches follows).
2019-05-30 09:21:19 +02:00
390e83c9b9
typo fix and doc improvement
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:57:28 +02:00
7fb49397d5
doc fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:56:14 +02:00
382609b04e
tools: add Cancellable, start a futures submodule
...
To make a future cancellable, use:
let (future, canceller) =
crate::tools::futures::cancellable(future);
Proceed with using `future` as usual, `canceller` is
clonable and can cancel the future via the `.cancel()`
method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
0d32d71fb7
tools: add AsyncMutex::new_locked
...
Allows creating a pre-locked mutex, returning the mutex and
a guard.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-29 11:18:29 +02:00
e6389f4e75
src/api2/admin/datastore/backup/environment.rs: do not allow empty backups (no files)
2019-05-29 10:38:57 +02:00
39e60bd6fa
src/api2/admin/datastore/backup.rs: improve debug logging
2019-05-29 10:17:38 +02:00
4ae6551c78
src/api_schema/router.rs: disable debug println
2019-05-29 09:42:16 +02:00
a42d1f55c3
src/api2/admin/datastore/backup.rs: add debug flag
2019-05-29 09:35:21 +02:00
976595e1a9
src/backup/dynamic_index.rs: split class DynamicIndexWriter
2019-05-29 08:49:57 +02:00
cb0708dd46
src/backup/fixed_index.rs: remove ChunkStat from struct
2019-05-29 07:08:34 +02:00
01af11f340
src/backup/fixed_index.rs: improve consistency checks
2019-05-29 07:02:45 +02:00
9b38443314
pxar: change logic and impl all command line flags for xattrs/fcaps/acls
...
Allows to individually set the flags for storing/dumping/restoring of
xattrs/fcaps/acls in the cli of pxar.
Changes logic so that each of them can be threated individually.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-29 06:51:20 +02:00
60e589a111
src/api2/admin/datastore/backup.rs: log sucessful finish call
2019-05-28 12:55:22 +02:00
c4ff3dcefd
src/bin/proxmox-backup-client.rs: use new BackupClient to upload data
2019-05-28 10:12:44 +02:00
10241c20ea
src/tools.rs: implement image_size helper
2019-05-28 09:35:08 +02:00
006f3ff407
src/api2/admin/datastore/backup/environment.rs: add more consistency checks
2019-05-28 09:21:31 +02:00
3dc5b2a203
src/client/http_client.rs: use start of chunk as offset
2019-05-28 09:21:31 +02:00
8de20e5c1f
src/client/http_client.rs: code cleanup
2019-05-28 08:53:27 +02:00
fc14b84937
src/backup/fixed_index.rs: improve bound checks inside add_digest
2019-05-28 08:47:03 +02:00
e1225de467
src/backup/fixed_index.rs: store index_length to avoid code duplication
2019-05-28 08:27:57 +02:00
e3d525feb7
src/api2/admin/datastore/backup.rs: fix typo
2019-05-28 07:13:19 +02:00
af309d4d5c
src/bin/pxar.rs: set archive permissions to 0o640
...
As the archive can contain potentially sensitive data such as key files, it makes
sense to restrict the permissions.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:24:47 +02:00
ab87f167f1
src/pxar/encoder.rs: Refactor file stat
...
Introduce helper functions to check file stats
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:23:49 +02:00
578b601159
src/pxar/encoder.rs: Don't bail if endpoint does not support xattrs.
...
The encoder bailed if a endpoint which did not support xattrs was encountered.
Instead of bailing, we ignore these errors and simply do not store xattrs for
such endpoints.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-28 06:22:16 +02:00
a42fa400ee
src/api2/admin/datastore/backup.rs: implement fixed sized chunk upload api
2019-05-28 06:18:55 +02:00
29ae5c86a2
src/backup/fixed_index.rs: make chunk_size public
2019-05-28 06:18:24 +02:00
7dd1bcac58
src/client/http_client.rs: correctly parse error response in download_chunk_list
2019-05-27 09:35:30 +02:00
a1e7cff3ea
src/api2/admin/datastore/backup/upload_chunk.rs: rename upload_dynamic_chunk into upload_chunk
2019-05-27 07:57:43 +02:00
771953f9eb
src/api2/admin/datastore/backup.rs: use smaller window sizes
2019-05-27 07:53:40 +02:00
174ad378d8
src/client/http_client.rs: fix request pipelining
2019-05-27 07:24:32 +02:00
624362226e
src/client/merge_known_chunks.rs: use MergedChunkInfo as input
2019-05-26 10:52:56 +02:00
05cba08c9c
src/client/http_client.rs: append chunks in correct order
...
We need to wait for response from upload_chunk. After that we can
add the chunk to the index file.
2019-05-26 09:57:34 +02:00
417cb0731e
src/api2/admin/datastore/backup.rs: verify chunk offset
...
This clearly shows that the current approach does not work - seems we get
chunks out of order.
2019-05-26 08:49:29 +02:00
34114e2606
src/server/rest.rs: correctly verify json parameters
2019-05-24 09:10:59 +02:00
eb2bdd1bcc
src/client/http_client.rs: encode parameters inside body
...
To avoid problems with long uris.
2019-05-24 08:32:55 +02:00
9bb675ec23
src/client/merge_known_chunks.rs: reduce RAM usage
...
There is no need to store chunk data for known chunks.
2019-05-24 07:42:58 +02:00
ecb21b4794
src/client/merge_known_chunks.rs: avoid VecDequeu - a single buffer is enough
2019-05-24 07:36:09 +02:00
2dbba78b98
src/pxar/sequential_decoder.rs: impl support to dump/restore ACLs from pxar archives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:27:24 +02:00
6a19448004
src/pxar/encoder.rs: impl support to store ACLs in pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:21:33 +02:00
7cfaade7c8
src/pxar/format_definition.rs: define types and structs needed to impl ACL support in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:16:55 +02:00
b344461b33
pxar: pass feature_flags to encoder/decoder instead of individual flags
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
687407741b
src/tools/xattr.rs: add test for is_valid_xattr_name()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
357e4614e2
Minor refactoring of pxars xattr encoder/decoder, mostly reformatting and renaming.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-24 06:11:36 +02:00
c05a8c8d18
doc-test fixup
...
cargo test by default compiles and runs all code snippets
found in the documentation...
oops...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 15:22:33 +02:00
cc84a830c5
RFC: schema: make enums static
...
I don't see a reason to allow these to be dynamically
modifiable.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 12:38:51 +02:00
aa1b2e04fe
src/client/merge_known_chunks.rs: merge known chunks
...
To decrease the number of api calls required...
2019-05-23 12:31:09 +02:00
91320f0879
src/client/http_client.rs: use ChunkInfo streams
...
This will make out of order uploads possible...
2019-05-23 12:31:09 +02:00
8ea3b1d188
tree-wide: use the new vec/io tools modules
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
9cdda3f7c7
tools: add helpful vector and read operations
...
After importing the I/O ops trait via:
use crate::tools::io::ops::*;
Instead of:
let mut buffer = vec![0u8; 65536];
file.read_exact(&mut buffer)?;
use:
let buffer = file.read_exact_allocated(65536)?;
After importing the vector helpers via:
use crate::tools::vec::{self, ops::*};
For a buffer which *could* be uninitialized but you prefer
zero-initialization anyway for security reasons, instead of:
let mut buffer = vec![0u8; len];
use:
let mut buffer = vec::undefined(len);
which zero-initializes, but, if the `valgrind` feature flag
is enabled, marks the vector as having undefined contents,
so reading from it will cause valgrind errors.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-05-23 09:49:10 +02:00
8bea85b42e
src/api2/admin/datastore/backup.rs: verify file size and chunk count on close
2019-05-23 08:50:36 +02:00
1179e15842
src/server/rest.rs: improve error handling
2019-05-23 08:15:32 +02:00
907f5bb0d8
src/api2/admin/datastore/backup/service.rs: improve error logging
2019-05-23 08:05:39 +02:00
32e1bf4260
src/bin/upload-speed.rs: code cleanup
2019-05-23 07:20:05 +02:00
0d9bab05ab
src/bin/pxar.rs: impl CLI flags to disable archiving/restoring of xattrs and fcaps.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:50 +02:00
4add706092
src/client/: pass dummy flags for xattrs and fcaps to pxar encoder/decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:32 +02:00
54b14ad741
src/pxar/decoder.rs: pass flags for xattrs and fcaps to decoder.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:40:09 +02:00
1201360def
src/pxar/sequential_decoder.rs: Conditional restore xattrs and fcaps if feature_flags is set
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:39:16 +02:00
8bc8791338
src/pxar/format_definitions.rs: do not store the feature_flags in pxar archive.
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:37:37 +02:00
a45a123950
src/pxar/encoder: Use encoder feature_flags, do not store them within the archive.
...
Instead of testing and storing the feature_flags in the pxar archive, set and use
the feature flags within the encoder.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-23 06:34:36 +02:00
d6f204eda6
src/client/http_client.rs: re-order code - public functions first
2019-05-23 06:28:10 +02:00
9af37c8f0e
src/client/http_client.rs: move low level H2 code into separate class
2019-05-22 17:28:25 +02:00
d2c48afc6e
src/client/http_client.rs: code cleanup
2019-05-22 13:24:33 +02:00
6ab34afa88
src/client/http_client.rs: cleanup h2 backup client
2019-05-22 13:05:51 +02:00
a55fcd740f
src/api2/admin/datastore/backup.rs: suse POST for finish() api
2019-05-22 13:04:58 +02:00
de6742c04f
src/bin/h2test.rs: removed
...
This test does no work anymore.
2019-05-22 11:14:56 +02:00
5a0b521809
src/api2/admin/datastore/backup.rs: remove test callbacks
2019-05-22 10:55:27 +02:00
de61bc9264
src/tools/xattr.rs: impl unit tests for fsetattr and fgetattr
...
As git does not preserve xattrs they have to be set before they can be read.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-22 10:01:10 +02:00
553610b43e
src/client/http_client.rs: implement download_chunk_list
2019-05-22 09:46:02 +02:00
0cc0fffd1e
src/backup/chunk_stream.rs: use BytesMut in ChunkStream
2019-05-22 09:39:02 +02:00
c052be5c86
src/backup/chunk_stream.rs: use more generics
2019-05-22 09:18:05 +02:00
169c0e060f
src/backup/chunk_stream.rs: optimize FixedChunkStream (use BytesMut)
2019-05-22 09:05:35 +02:00
3be3f3dcc4
src/backup/chunk_stream.rs: fix behaviour for large input buffers
2019-05-22 08:05:27 +02:00
02fa54ff3c
src/backup/chunk_stream.rs: assemble data with correct order
2019-05-22 07:44:51 +02:00
d0afeaa831
src/api2/admin/datastore/backup.rs: improve error message
2019-05-22 07:43:54 +02:00
11867a2b54
src/tools.rs: fix hex_to_digest
2019-05-21 13:42:13 +02:00
7f3d2ffab9
src/backup/index.rs: rename ChunkListReader into DigestListEncoder
2019-05-21 12:28:44 +02:00
ee53955f52
src/backup/index.rs: implement DigestListDecoder
2019-05-21 12:21:22 +02:00
a95849321c
src/api2/admin/datastore/backup.rs: ignore error after finish, register known_chunks
2019-05-21 09:54:11 +02:00
40f4e198a8
src/backup/dynamic_index.rs: add chunk_info method
2019-05-21 09:52:19 +02:00
596182056b
src/pxar/sequential_decoder.rs: impl functionality to restore xattrs/fcaps from pxar achives
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:15:30 +02:00
b303057a19
src/pxar/encoder.rs: impl functionality to encode xattrs/fcaps in pxar
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:06:26 +02:00
bee8d8ea78
src/tools/xattr.rs: add functions name_store and security_capability
...
These functions allow to check if the name for xattrs/fcaps are valid.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-21 06:05:31 +02:00
a09c0e38d8
src/api2/admin/datastore/backup/environment.rs: register/lookup known chunks
2019-05-20 18:05:10 +02:00
97f03eff13
src/client/pxar_backup_stream.rs: fix docu
2019-05-20 17:29:00 +02:00
82ab72304e
src/client/http_client.rs: implement upload_stream
2019-05-20 14:19:24 +02:00
2698e8a514
src/client/pxar_backup_stream.rs: close pipe inside drop
2019-05-20 11:20:33 +02:00
f2269d8f99
src/tools.rs: implement hex_to_digest
2019-05-19 12:15:46 +02:00
751030aa7c
src/bin/test_chunk_speed2.rs: test max chunk size
2019-05-19 11:09:02 +02:00
ff77dbbea3
Revert "src/backup/chunk_stream.rs: simplify code"
...
This reverts commit e058744d8f
2019-05-19 11:05:56 +02:00
178ac042c4
src/api_schema/schema.rs: improve error messages
2019-05-19 10:18:51 +02:00
0c516b123d
src/client/pxar_backup_stream.rs: use WrappedStreamReader
...
to make it fully async ...
2019-05-18 15:13:31 +02:00
c6e28b66c6
src/client/pxar_backup_stream.rs: use 1M pipe buffer size
2019-05-18 12:57:43 +02:00
6276448519
src/bin/test_chunk_speed2.rs: test chunker speed with real data
2019-05-18 11:59:17 +02:00
e058744d8f
src/backup/chunk_stream.rs: simplify code
2019-05-18 10:55:46 +02:00
8a7cc7565f
src/backup/chunk_stream.rs: implement fixed size chunk stream
2019-05-18 10:55:46 +02:00
9c9ad941c0
src/client/http_client.rs: improve speed test
...
pipeline responses to a separate async channel ...
2019-05-18 10:55:46 +02:00
2dcdd3b427
src/tools/xattrs.rs: impl libc wrapper functions to get/set xattrs and fcaps
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:21 +02:00
de2016d58a
src/pxar/format_definitions.rs: add format definitions for xattrs and fcaps
...
In addition to the format definition, the traits needed for sorting of xattr
entries by name are derived.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:14:13 +02:00
dec1ff18b8
src/tools/acl.rs: impl add_entry_full for ACL and set_permissions for ACLEntry
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-16 11:10:06 +02:00
adec8ea263
src/api2/admin/datastore/backup.rs: add speedtest api, improve upload speed
...
We need to disable tcp Nagle algorythm (set_nodelay), and use larger window size for http2
2019-05-16 10:24:23 +02:00
372724afea
src/api2/admin/datastore/backup.rs: implement finish_backup
2019-05-15 12:58:55 +02:00
bb105f9dae
src/api2/admin/datastore/backup.rs: improve logging
2019-05-15 10:09:46 +02:00
25ddc15fe0
src/client/pipe_to_stream.rs: add missing file
2019-05-15 09:07:19 +02:00
a20772525f
src/api2/admin/datastore/backup.rs: implement close for dynamic writers
2019-05-15 07:58:05 +02:00
4e93f8c164
src/api2/types.rs: add schema/format for file names
2019-05-15 07:14:08 +02:00
e3dbd41b20
src/client/http_client.rs: add flow control for h2 upload
2019-05-14 14:54:21 +02:00
97f22ce57d
src/client/http_client.rs: implement h2 upload
...
without flow control so far ...
2019-05-14 12:58:59 +02:00
dafc27aeff
src/backup/chunk_stream.rs: async chunk stream
2019-05-14 10:05:29 +02:00
c0e22141ed
src/client/http_client.rs: derive Clone for H2Client
2019-05-14 09:00:24 +02:00
25e205a1d3
src/tools.rs: add mod acl to tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:34:24 +02:00
d22096ef4b
src/tools/acl.rs: impl POSIX access control list (ACL) manipulation tools
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-05-14 06:28:42 +02:00
1133fe9a63
src/server/rest.rs: do not log 1xx status codes as errors
2019-05-14 06:23:22 +02:00
850ac6d0f0
src/client/http_client.rs: directly return H2Client on upgrade
2019-05-13 16:44:59 +02:00
10130cf48c
src/client/http_client.rs: cleanup, remove debug println
2019-05-13 12:11:18 +02:00
b57cb26406
src/client/http_client.rs: implement http2 client wrapper
2019-05-13 10:27:22 +02:00
9e391bb7f0
src/client/http_client.rs: allow to pass parameters (encode them as query string)
2019-05-13 09:12:03 +02:00
fbb798f6d2
src/api2/admin/datastore/backup.rs: ignore errors from last_backup
2019-05-13 07:06:56 +02:00
d36113665b
src/api2/admin/datastore/backup.rs: implement chunk index download from previous backup
2019-05-11 16:05:50 +02:00
457531e7fe
src/backup/index.rs: new ChunkListReader
2019-05-11 16:01:42 +02:00
af1d68185c
src/api2/admin/datastore/backup.rs: add missing 'archive-name' parameter to schema
2019-05-11 12:25:15 +02:00
6b95c7df07
src/api2/admin/datastore/backup/environment.rs: do not store path
2019-05-11 12:13:44 +02:00
51a4f63ff7
src/backup/datastore.rs: move last_backup to BackupInfo
2019-05-11 12:07:09 +02:00
b02a52e3a5
src/backup/datastore.rs: impl last_backup helper
2019-05-11 11:36:46 +02:00
c09775011a
src/backup/backup_info.rs: implement list_backup for BackupGroup, cleanups
2019-05-11 10:19:34 +02:00
f9578f3c79
backup-api: allow to create DynamicIndexWriter and add chunks to to
2019-05-10 10:25:40 +02:00
35a2d8a6a6
src/backup/backup_info.rs: Derive Clone
2019-05-10 06:59:23 +02:00
0506d1ed04
upload_chunk.rs: add comment about possible unnecessary code
2019-05-10 06:15:25 +02:00
5f42ef009a
cleanup: remove unused code
2019-05-09 18:11:39 +02:00
b4b63e520d
handle_async_api_request: put rpcenv into a Box
...
So that we can pass rpcenv into futures.
2019-05-09 18:01:24 +02:00
2c41096a4b
src/api2/admin/datastore/backup/upload_chunk.rs: fix maximal chunk size
2019-05-09 13:29:50 +02:00
78b5191550
src/server/formatter.rs: further cleanups and renaming ...
2019-05-09 13:28:26 +02:00
d55037e471
src/server/formatter.rs: rename format_result to format_data
...
To avoid confusions with Rust Result type.
2019-05-09 13:15:15 +02:00
21ee7912fa
src/api2/admin/datastore/backup.rs: implement upload chunk
2019-05-09 13:06:09 +02:00
4ebf0eabb0
src/api2/types.rs: use separate file for schema definitions
2019-05-09 07:44:09 +02:00
bd1507c4fa
src/api2/admin/datastore/backup/service.rs: move service code into extra file
2019-05-08 17:36:19 +02:00
0aadd40b4f
src/api2/admin/datastore/backup.rs: add required parameters, cleanups
2019-05-08 13:17:23 +02:00
d95ced64ad
src/api2/admin/datastore/backup/environment.rs: use separate file
2019-05-08 12:41:58 +02:00
ca60c371e2
api: rename h2upload to backup
2019-05-08 12:35:15 +02:00
d906b1124f
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 11:36:43 +02:00
58c8d7d91f
rc/api2/admin/datastore/h2upload.rs: implement BackupEnvironment
...
To pass arbitrary information/state to api methods.
2019-05-08 11:26:54 +02:00
f757b30efc
src/server/rest.rs: use generics to pass RpcEnvironment
2019-05-08 11:09:01 +02:00
2edc341b29
src/tools.rs: Add AsAyn Trait
2019-05-08 11:07:36 +02:00
090ac9f79a
src/api2/admin/datastore/h2upload.rs: avoid unwrap
2019-05-08 08:21:18 +02:00
4e3da4b366
src/api2/admin/datastore/h2upload.rs: do not log client address
2019-05-08 08:15:55 +02:00
e3a44552ee
src/api2/admin/datastore/h2upload.rs: code cleanups
2019-05-08 08:12:49 +02:00
a66ab8ae5e
src/api2/admin/datastore/h2upload.rs: make worker abortable
2019-05-08 07:14:40 +02:00
72375ce6c6
src/api2/admin/datastore/h2upload.rs: log to worker task
2019-05-07 14:07:00 +02:00
d9bd06eae8
src/api2/admin/datastore/h2upload.rs: use WorkerTask
2019-05-07 13:42:00 +02:00
52cf506e48
src/api2/admin/datastore/h2upload.rs: implement async test api call
2019-05-07 12:26:55 +02:00
92ac375aff
src/api2/admin/datastore/h2upload.rs: improve http2 service
2019-05-07 11:24:44 +02:00
78f7e225e4
src/server/rest.rs: make handle_(a)sync_api_request public
2019-05-07 11:23:52 +02:00
75097495ad
RestEnvironment: derive Clone
2019-05-07 11:09:18 +02:00
01bf3b7b5f
move find_method to Router class
2019-05-07 11:08:30 +02:00
3578d99f3e
move normalize_path to tools::normalize_uri_path
2019-05-07 09:44:34 +02:00
96e95fc179
src/api2/admin/datastore/h2upload.rs: test if version < 2
...
We can only upgrade http 1.x connections.
2019-05-06 10:29:34 +02:00
45cdce069e
src/client/http_client.rs: request password at creation time
2019-04-30 11:44:35 +02:00
e45afdff9c
src/server/state.rs: use new BroadcastData helper
2019-04-30 10:21:48 +02:00
824b5ee4ee
src/tools/broadcast_future.rs: cleanup, decompose into two classes
...
In order to make it more usable.
2019-04-30 08:57:20 +02:00
d82ed9b0f5
tools: add AsyncMutex
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-29 14:19:03 +02:00
3a9bea3918
src/bin/h2test.rs: test h2 upgrade
2019-04-29 11:59:18 +02:00
cf639a47de
rc/client/http_client.rs: add experimental h2 upgrade code
2019-04-29 11:57:58 +02:00
5a2df00004
src/client/http_client.rs: cleanup, make login fully async
2019-04-28 10:55:03 +02:00
3dceb9b304
src/tools/broadcast_future.rs: add new constructor new_oneshot()
...
To simplify usage. Also added a test case.
2019-04-27 10:56:49 +02:00
152764ec15
src/api2/admin/datastore/h2upload.rs: add test code for http2
2019-04-27 08:57:35 +02:00
490be29ed6
src/tools/broadcast_future.rs: new helper class
2019-04-26 17:57:57 +02:00
e888155712
proxy: allow .key/.pem files instead of a pkcs12/.pfx file
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-26 10:47:33 +02:00
8b7c0def84
src/tools/procfs.rs: clean up tab indentation and use whitespaces instead
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-25 13:14:52 +02:00
5e5eed5c3b
use double-fork for reload
...
To ensure the new process' parent is pid 1, so systemd won't
complain about supervising a process it does not own.
Fixes the following log spam on reloads:
Apr 25 10:50:54 deb-dev systemd[1]: proxmox-backup.service: Supervising process 1625 which is not our child. We'll most likely not notice when it exits.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
efd1536eb7
add a wrapper around nix::unistd::pipe2
...
Using O_CLOEXEC by default, and returning Fd handles to
ensure they get dropped on bail!() or panic!() if the RawFds
aren't used yet.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
897982e237
tools: add read/write convenience helpers
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:02:12 +00:00
d98c9a7a38
use service Type=notify
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:33 +00:00
9c351a36c4
tools/daemon: add sd_notify wrapper
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:28 +00:00
d8871f2f18
src/tools/procfs.rs: avoid useless allocations
2019-04-24 09:35:15 +02:00
ae694f44ef
src/tools/procfs.rs: add tests for read_proc_net_route() and read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:52 +02:00
ec8db837fb
src/tools/procfs.rs: refactor read_proc_net_route() and read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-24 06:43:32 +02:00
eb46c3d1ec
src/tools/procfs.rs: implement read_proc_net_ipv6_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-23 14:31:23 +02:00
6358b68236
cleanup: avoid compiler warnings
2019-04-23 12:17:24 +02:00
e06ffb10d1
src/api_schema/router.rs: fix typo (subdira -> subdir)
2019-04-16 12:20:17 +02:00
13f1cc17ea
src/api_schema/router.rs: implement list_subdirs() helper
2019-04-16 12:07:02 +02:00
062d4916ff
api_schema: allow generic api handler functions
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-16 11:22:23 +02:00
286f0d4099
src/tools/procfs.rs: implement read_proc_net_route()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-15 11:13:16 +02:00
1a6c9415ae
src/api2/node/services.rs: add proxmox-backup-proxy service
2019-04-15 09:48:02 +02:00
75bc49bed7
src/server/worker_task.rs: implement abort channel
2019-04-15 09:38:05 +02:00
912524752e
src/api2/admin/datastore/pxar.rs: create worker task for file upload
2019-04-15 08:38:14 +02:00
1be71fb02b
src/server/worker_task.rs_ do not write ERROR: prefix into task index
2019-04-15 08:37:13 +02:00
418def7a78
src/server/worker_task.rs: move common code into log_result
...
Make new() pub
2019-04-15 07:49:03 +02:00
37b87869b5
src/server/upid.rs: check for illegal characters
2019-04-15 07:47:51 +02:00
d2981e2738
src/tools/procfs.rs: implement read_proc_net_dev()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:59 +02:00
13abfdc57a
src/tools/procfs.rs: implement read_memory_usage()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-13 11:52:01 +02:00
3005ef48b6
src/tools/procfs.rs: Rename variable and struct to be more consistent
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:34 +02:00
cc93bacb72
src/tools/procfs.rs: simplify read_cpuinfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-12 12:13:14 +02:00
f2b99c34f7
src/api2/admin/datastore.rs: implement API to return last GC status
2019-04-11 12:04:25 +02:00
59961b8954
src/server/command_socket.rs: check control socket permissions
2019-04-11 10:51:59 +02:00
ba70040dbf
src/server/worker_task.rs: change task list sort order - latest on top
2019-04-11 09:43:33 +02:00
d3f4c08f33
src/server/worker_task.rs: catch panics in worker threads
2019-04-11 07:56:06 +02:00
620dccf1a1
tools/daemon: dup the TcpListener file descriptor
...
Now that we let hyper shutdown gracefully we need an owned
version of the listening socket to prevent it from closing
before running the reload preparations.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
d96bb7f163
tools: add Fd helper
...
stores a raw file descriptor with a drop handler for
safekeeping in closures
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-10 15:20:10 +02:00
1cd33633aa
/src/tools: Cleanup tempfiles in folder
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:56:22 +02:00
a665dea128
src/api2/node/tasks.rs: implement stop_task api
2019-04-10 13:55:05 +02:00
98a181f055
src/server/worker_task.rs: log message when we call request_abort()
2019-04-10 13:54:28 +02:00
2b6a3e1396
src/tools/procfs.rs: implement read_cpuinfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-10 13:11:11 +02:00
321070b4fa
src/server/worker_task.rs: implement abort_worker (via command_socket)
2019-04-10 12:42:24 +02:00
cfb2d3c17c
src/server/command_socket.rs: correctly handle/spawn handle parallel connections
2019-04-10 11:05:00 +02:00
9b002cbc5f
src/server/worker_task.rs: use abstract socket
2019-04-10 09:03:17 +02:00
5f550fd99f
start hyper server using with_graceful_shutdown()
...
Without, hyper keeps some futures running, and the server does not
correctly shutdown.
2019-04-10 08:24:32 +02:00
7f2078de59
src/tools/procfs.rs: avoid compiler warning
2019-04-10 06:40:29 +02:00
1f6fbf1c57
src/tools/procfs.rs: implement read_meminfo()
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-09 15:21:49 +02:00
c360bd73d3
src/api2/node/tasks.rs: fix task status api
2019-04-09 15:12:20 +02:00
0bfd87bc5e
src/server/worker_task.rs: speedup upid_read_status()
2019-04-09 15:11:48 +02:00
d8d40dd0c2
src/api2/node.rs: add node parameter
2019-04-09 14:43:30 +02:00
7ae0bf52dd
src/server/command_socket.rs: code cleanup - fix error message
2019-04-09 12:47:42 +02:00
d607b8861b
src/server/worker_task.rs: implement task control socket
2019-04-09 12:15:06 +02:00
e201753629
src/server/command_socket.rs: implement auto_remove flag
...
Remove the socket file on close.
2019-04-09 11:47:23 +02:00
634132fe10
src/server/upid.rs: moved code into separate file
2019-04-08 18:43:26 +02:00
78a39e0517
src/server/command_socket.rs: simple command socket
2019-04-08 17:59:39 +02:00
e3f41f2165
src/tools/daemon.rs: use new ServerState handler
2019-04-08 14:00:23 +02:00
7a630df709
implement server state/signal handling, depend on tokio-signal
2019-04-08 13:59:07 +02:00
5a12c0e2fb
src/api2/node/tasks.rs: start upid API
2019-04-07 14:36:57 +02:00
b75b968147
src/server/worker_task.rs: implement is_active()
2019-04-07 14:36:25 +02:00
4494d07819
src/server/worker_task.rs: make log_path a method on UPID
2019-04-07 13:51:46 +02:00
d2a2e02bb7
src/api2/node/tasks.rs: implement userfilter
2019-04-07 13:17:19 +02:00
3c3bee2e32
src/api2/node/tasks.rs: correctly handle start parameter
2019-04-07 12:41:24 +02:00
063ca5be77
src/api2/node/tasks.rs: implement list_tasks
2019-04-07 12:18:58 +02:00
d4b59ae0b8
src/backup/datastore.rs: pass WorkerTask to GC
...
So that we can log to the task log file.
2019-04-06 17:57:38 +02:00
35950380a9
src/server/worker_task.rs: carefully handle file permissions
2019-04-06 17:53:12 +02:00
1619a72063
src/tools.rs: implement create_dir_chown
...
Combinded mkdir with chown.
2019-04-06 16:50:23 +02:00
eea8131952
src/tools.rs: implement file_set_contents_full()
2019-04-06 13:53:43 +02:00
0f778e0652
src/api2/admin/datastore.rs: use new WorkerTask::new_thread()
2019-04-06 11:27:23 +02:00
244abab769
src/tools/file_logger.rs: fix test
2019-04-06 11:24:37 +02:00
660c684640
src/server/worker_task.rs: return upid_str
2019-04-06 11:23:53 +02:00
93aebb38bc
src/server/worker_task.rs: implement read_task_list()
2019-04-06 10:17:11 +02:00
4a36a2c999
src/server/worker_task.rs: improve UPID format description
2019-04-06 09:36:50 +02:00
882594c5e9
src/server.rs: improve crate layout
2019-04-06 09:17:25 +02:00
4b01c983f0
src/server/worker_task.rs: implement persistent task list
2019-04-05 12:37:35 +02:00
a2474d328f
src/tools/procfs.rs: implement read_proc_uptime and read_proc_uptime_ticks
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-05 06:22:21 +02:00
08daa81af6
procfs: implement check_process_running and check_process_running_pstart
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-04-04 13:54:17 +02:00
9a0d0ff774
server/worker_task.rs: implement UPID parser
2019-04-04 13:49:35 +02:00
49cf9f3d0b
src/tools.rs: file_get_json() - add new default parameter
2019-04-04 12:24:18 +02:00
79ad7b7ec2
src/tools/procfs.rs: split regex into several lines
2019-04-04 09:25:19 +02:00
479f6e40ec
src/server/worker_task.rs: implement task registry
2019-04-04 08:58:39 +02:00
52ddc2522d
src/tools/procfs.rs: use libc::pid_t
2019-04-04 08:05:43 +02:00
d3d7ab1273
src/tools/file_logger.rs: new - accept AsRef<Path>
2019-04-03 14:13:33 +02:00
3c2012f97a
src/tools/procfs.rs: start helper tools to read procfs
2019-04-03 13:39:20 +02:00
3489936ebb
src/tools/file_logger.rs: change timestamp format to rfc3339
2019-04-03 08:58:43 +02:00
92da93b245
abort GC on server shutdown
2019-04-01 12:13:02 +02:00
9136f857cc
add global var to indicate server shutdown requests
2019-04-01 12:05:11 +02:00
3e6a7deeaa
src/api2/admin/datastore.rs: start GC in a separate thread
2019-04-01 12:02:46 +02:00
ad281d1bd9
www/DataStoreConfig.js: add button to start GC
2019-04-01 08:08:34 +02:00
49d123ee3a
src/server/rest.rs: use formatter to encode errors
2019-04-01 08:04:12 +02:00
7d4ef127cb
src/server/rest.rs: generate csrf token if we have a valid ticket
...
This is important if the user reloads the browser page.
2019-04-01 07:52:30 +02:00
11861a482d
src/backup/chunk_store.rs: fix GC
...
Added option to get oldest_writer timestamp from ProcessLocker.
2019-03-31 17:21:36 +02:00
d85987aeeb
fix last commit: the filename var was not ment to be removed, sorry
2019-03-31 16:16:14 +02:00
15a77c4c2e
src/backup/chunk_store.rs: avoid create/unlink race
2019-03-31 10:03:01 +02:00
c374f05499
src/backup.rs: improve GC problem description
2019-03-31 09:44:35 +02:00
c8ec450e37
src/backup.rs: start explaining different GC algorithm
2019-03-30 17:21:40 +01:00
8a4757342c
src/backup.rs: describe the garbage collection problem
2019-03-30 16:26:52 +01:00
10eea49d81
src/backup/fixed_index.rs: store reference to file
...
Keep it open and locked while index is in use.
2019-03-27 10:17:03 +01:00
c597a92c81
index file readers: aquire shared lock
2019-03-27 10:11:14 +01:00
0465218953
src/backup.rs: add documentation about ChunkStore locking
2019-03-22 10:14:50 +01:00
43b1303398
datastore: use new ProcessLocker
...
To make sure only one process runs garbage collection while having active writers.
2019-03-22 09:42:15 +01:00
abfc001f25
src/tools/process_locker.rs: new() - use generic path parameter
2019-03-22 09:41:32 +01:00
a650f50301
src/tools/process_locker.rs: implement inter-process reader-writer locks
2019-03-22 08:05:51 +01:00
eb90c9e372
tools: style fixup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-20 09:57:13 +01:00
e04d1bfadc
Cleanup of if statement with unwraps by a more compact match statement
...
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-03-20 09:49:18 +01:00
8346f0d59b
src/server/rest.rs: correctly extract content type
2019-03-19 12:50:15 +01:00
a690ecacf0
daemon: simplify daemon creation
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 12:12:54 +01:00
e4311382d9
tools: daemon: rename some structs
...
Reloadable resources are now 'Reloadable' instead of
'ReexecContinue'.
The struct handling the reload is a 'Reloader', not a
'ReexecStore'.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 11:09:46 +01:00
5e7bc50af1
add reload support to api daemon
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
a2ca713701
add reload support to proxy
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
af70c1811b
tools: implement ReexecContinue for tokio's TcpListener
...
This is the only thing we currently need to keep alive for
reloads.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
4422ba2c7f
tools: daemon: add a default signalfd helper
...
Proxy and daemon for now just want to handle reload via
`SIGHUP`, so provide a helper creating the signalfd stream
doing that - this is simply a filtered stream which passes
the remaining signals through, so it can be used exactly
like the signalfd stream could before to add more signals.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
dce94d0e09
tools: add daemon helpers
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
ff7049d481
tools: add fd_change_cloexec helper
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
1c042cdc6c
proxy: listener error handling fixup
...
If the listening socket goes into some error state we'll get
std::io::Errors rather than higher level errors from the
native_tls::TlsAcceptor, those are usually fatal. (Ran into
this after performing a shutdown() on the file descriptor,
after which the future just endlessly loops in accept().)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
60864b1b56
src/pxar.rs: improve docu
2019-03-18 12:27:30 +01:00
515688d1f1
avoid compiler warnings
2019-03-18 10:00:58 +01:00
ba10f2b0d4
src/bin/dynamic-files.rs: test case for file shrink during backup
2019-03-16 14:22:14 +01:00
af572aaa4c
pxar: implement hardlinks
...
So we are no longer compatible with catar ...
2019-03-16 11:02:12 +01:00
812dd935ce
src/pxar/encoder.rs: detect hardlinks
2019-03-15 12:19:51 +01:00
896e803feb
src/pxar/sequential_decoder.rs: dump_entry - fix FIFO/SOCKET handling
2019-03-15 12:15:38 +01:00
7dfa17c7cc
src/pxar/binary_search_tree.rs: fix test
2019-03-15 11:34:31 +01:00
9307279faf
cleanup: remove dead code
2019-03-15 10:18:28 +01:00
46cfe7ab7e
src/pxar/decoder.rs: implement seekable decoder
2019-03-15 09:36:05 +01:00
0e21b87be3
renamed: src/pxar/decoder.rs -> src/pxar/sequential_decoder.rs
2019-03-15 08:36:02 +01:00
3dbfe5b142
simplify pxar module structure
2019-03-15 08:24:32 +01:00
1ef46b81d3
src/bin/pxar.rs: implement extract
2019-03-15 08:03:44 +01:00
02c93163cf
src/pxar/decoder.rs: correctly call callback()
2019-03-15 08:02:04 +01:00
5defa71bf9
rename PxarBackupWriter into PxarDecodeWriter
2019-03-15 07:20:22 +01:00
b005ed12d5
src/client/http_client.rs: avoid compiler warning
2019-03-15 07:16:26 +01:00
9e26abf16b
src/cli/command.rs: fix iterator usage
2019-03-15 07:12:40 +01:00
f98746ba6c
src/pxar/decoder.rs: improve docu
2019-03-15 07:12:11 +01:00
0b2d7fdc81
src/pxar/decoder.rs: avoid compiler warnings
2019-03-14 17:45:47 +01:00
37940aa1f9
src/bin/pxar.rs: implement dump/list
2019-03-14 17:43:11 +01:00
40360fde6e
src/pxar/decoder.rs: implement dump_archive
...
use this to debug archive problems ...
2019-03-14 13:10:27 +01:00
8968258b66
rename catar into pxar
...
To avoid confusion with the casync implementation.
2019-03-14 10:54:09 +01:00
7c4dd94670
rename catar into pxar
...
Seems impossible to stay compatible with the original catar specification,
so we change the name to avoid confusions.
2019-03-14 10:14:32 +01:00
806500cd09
src/tools.rs: improve file name completion
...
Use current working directory by default.
2019-03-14 08:17:35 +01:00
08dc340a12
src/bin/proxmox-backup-client.rs: complete archive names
2019-03-14 08:09:35 +01:00
6f90a6a764
protocol: cleanup finish_backup
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-13 14:16:17 +01:00
b238851832
src/bin/proxmox-backup-client.rs: implement completion for snapshots
2019-03-13 13:31:29 +01:00
4981134708
src/bin/proxmox-backup-client.rs: implement chunk-size completion
2019-03-13 12:26:01 +01:00
98f0b97293
src/bin/proxmox-backup-client.rs: cleanup
2019-03-13 12:17:39 +01:00
024f11bb55
src/bin/proxmox-backup-client.rs: implement completion for backup groups
2019-03-13 11:56:37 +01:00
edd3c8c605
src/client/backup_repo.rs: implement FromStr trait
2019-03-13 10:09:39 +01:00
874acb7039
src/client/backup_repo.rs: implement Display trait
2019-03-13 09:57:36 +01:00
d0a03d40ce
src/bin/proxmox-backup-client.rs: implement repository completions
2019-03-13 09:47:12 +01:00
bff110308b
src/bin/proxmox-backup-client.rs: fix BACKUPSPEC_REGEX - correctly specify non-capture group
...
And improve bash completion.
2019-03-13 07:23:59 +01:00
1a71509aad
src/cli/command.rs: fix completion - never display argument parameters as option
2019-03-12 14:53:41 +01:00
496a67846f
src/cli/command.rs: pass parsed parameters to completion function
2019-03-12 14:39:51 +01:00
b5fa28251c
src/cli/getopts.rs: factor out reusable code
2019-03-12 14:38:47 +01:00
e39efdbd33
src/cli/command.rs: use a HashMap to store fixed param as key/value pairs
2019-03-12 12:54:16 +01:00
f4e4d58327
src/cli/command.rs: improve completions
2019-03-12 12:07:45 +01:00
4ba7b6099c
signalfd cleanup
...
- add comment about the boxed file descriptor (and others)
- reexport SigSet and Signal for convenience
- remove Result wrapper (only used once)
- enforce drop order of the PollEvented2 struct with respect
to the boxed fd
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-12 11:07:09 +01:00
f54c19989c
tools: add async signalfd handler
...
This provides a Stream<Item = siginfo> via nix' signalfd,
by wrapping it in tokio's PollEvented2 struct to allow
polling via tokio's event loop.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-12 08:04:28 +01:00
d5c34d98c0
src/catar/decoder.rs: simplify public restore API
2019-03-11 14:31:01 +01:00
8e39232acc
src/bin/proxmox-backup-client.rs: strip .didx file extensions
2019-03-11 10:54:02 +01:00
74cdb52107
src/bin/proxmox-backup-client.rs: include file extension in backupspec
2019-03-11 10:27:28 +01:00
fc2bf37ead
src/catar/decoder.rs: cleanup style
2019-03-11 08:14:28 +01:00
656b23e107
src/catar/decoder.rs: dir_mkdirat - add create_new flag
2019-03-11 07:27:45 +01:00
e9c9409a99
src/catar/decoder.rs: implement real sequential decoder (no Seek)
...
I copied the old code to src/catar/inspector.rs. Will modify
that later to implement a random access decoder ...
2019-03-08 16:55:54 +01:00
eed6db3923
src/bin/proxmox-backup-client.rs: impl option all-file-systems
2019-03-08 09:36:40 +01:00
06178f13fa
move chunker into proxmox_protocol
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-08 09:29:47 +01:00
e3c30c50a6
src/bin/catar.rs: implement all-file-systems option
2019-03-08 09:28:12 +01:00
65092b1e27
src/catar/encoder.rs: log skipped mount points
2019-03-08 09:25:00 +01:00
6b64d25d64
src/catar/encoder.rs: remove debug output
2019-03-08 08:23:07 +01:00
2689810c92
src/bin/catar.rs: implement verbose flags
2019-03-08 08:20:56 +01:00
79c281fdac
src/catar/encoder.rs: use simply flag "all_file_systems"
2019-03-08 08:14:26 +01:00
25e77d3877
src/cli/command.rs: print_bash_completion - avoid crash
2019-03-07 12:14:26 +01:00
141f062e08
src/backup/chunk_store.rs: use zstd compression insteadf of lz4
...
Provides better compressionm rate, and is still fast.
2019-03-07 11:42:59 +01:00
d2690f74bb
src/backup/chunk_store.rs: read_chunk - clear buffer before calling read_to_end
2019-03-07 10:40:01 +01:00
2ae7d196b9
src/bin/proxmox-backup-client.rs: correctly strip file extension
2019-03-06 11:18:46 +01:00
9f91249351
src/bin/proxmox-backup-client.rs: impl restore
2019-03-06 11:02:08 +01:00
01a1342364
src/api2/admin/datastore.rs: impl list_snapshot_files
2019-03-06 11:02:08 +01:00
58e99e138e
src/backup/backup_info.rs: new helper to list files
2019-03-06 11:02:08 +01:00
dd79265a8a
src/api2/admin/datastore/catar.rs: use archive_name directly
2019-03-06 11:02:08 +01:00
a6b7551311
src/client/http_client.rs: impl download
2019-03-06 11:02:08 +01:00
6716f30bbc
add protocol test client
...
This uses futures for everything which is mostly useful as a
test to see if the protocol crate's non-blocking I/O support
can handle it...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:26:12 +01:00
e2d007f76e
api2/admin/datastore: add a backup protocol test api path
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:26:12 +01:00
e3062f87b1
fixed_index: split add_chunk/add_digest
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
c3bb97e59c
fixed_index: impl Send for FixedIndexWriter
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
3b7ade9e78
backup/datastore: expose insert_chunk method
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
91ebea8262
router: add upgrade method
...
This is just an Async GET method, but semantically
different, so it gets its own router method.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-06 10:25:20 +01:00
80822b95b1
src/bin/proxmox-backup-client.rs: sort list results
2019-03-06 07:20:51 +01:00
fba30411fb
src/bin/proxmox-backup-client.rs: allow to set backup-id
2019-03-06 07:02:52 +01:00
21ea0158b4
src/client/http_client.rs: remove debug statement
2019-03-06 06:45:51 +01:00
ba3a60b2a4
src/client/http_client.rs: store/load ticket in xdg runtime dir, depend on crate xdg
2019-03-05 12:56:21 +01:00
f8f94534d4
src/api2/access.rs: authenticate_user() - add ticket login
2019-03-05 12:56:21 +01:00
e5662b04e9
src/tools/ticket.rs: define const TICKET_LIFETIME
2019-03-05 12:56:21 +01:00
53cafb5925
src/tools.rs: impl file_get_json()
2019-03-05 12:56:21 +01:00
391d310741
backup_info: BackupDir shortcut
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 10:15:06 +01:00
93b49ce38a
backup_info: use two independent Into<String> types
...
there's no reason to force them to be the same input type
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-05 09:13:08 +01:00
3ec3ec3f9a
src/bin/proxmox-backup-client.rs: print backup duration
2019-03-05 08:11:40 +01:00
d57474e055
src/backup/backup_info.rs: improve docs
2019-03-05 07:28:13 +01:00
b3483782d1
src/backup/backup_info.rs: move code into separate file
...
Also changed create_backup_dir() parameters - uses &BackupDir now.
2019-03-05 07:18:12 +01:00
2b01a22507
src/backup/datastore.rs: add helper to sort backup lists
2019-03-04 18:20:57 +01:00
875fb1c01a
src/backup/datastore.rs: use unix epoch to create DateTime
...
To make sure that we have a timestamp without nanosecond.
2019-03-04 17:58:22 +01:00
38f8815925
src/backup/datastore.rs: use new() function to create structs
2019-03-04 17:44:27 +01:00
9b492eb256
src/backup/datastore.rs: protect BackupDir fields, impl new()
2019-03-04 13:51:36 +01:00
1e9a94e579
src/backup/datastore.rs: protect BackupGroup fields, impl new()
2019-03-04 13:38:23 +01:00
cdebd467e6
src/bin/proxmox-backup-client.rs: simply log to stdout
2019-03-04 09:38:34 +01:00
f72dfd2d10
src/backup/datastore.rs: make sure timestamp does not contains nanoseconds
...
Because we want simple, readable file names.
2019-03-04 09:37:10 +01:00
219ef0e6ca
src/catar/encoder.rs: add verbose flags
2019-03-04 08:01:09 +01:00
e0b8ce6424
src/catar/encoder.rs: skip RAMFS
...
unless explicitly selected.
2019-03-04 06:48:47 +01:00
597a92039e
src/bin/proxmox-backup-client.rs: rename create into backup
2019-03-03 11:46:53 +01:00
6f62c9240a
src/api2/admin/datastore.rs: imp delete_snapshot
2019-03-03 11:29:00 +01:00
af926291e0
avoid compiler warning
2019-03-03 09:52:06 +01:00
e77a02ed87
catar api: cleanup parameter names
2019-03-03 09:48:32 +01:00
421c9ba1d9
src/bin/proxmox-backup-client.rs: use new json_object_to_query
2019-03-03 09:44:25 +01:00
f5f13ebc5a
src/tools.rs: new helper json_object_to_query
2019-03-03 08:51:37 +01:00
49dc074007
parameter name cleanups (use '-' instead of '_')
2019-03-02 16:34:10 +01:00
ad20d19877
src/api2/admin/datastore.rs: rename get_group_list to list_groups, cleanups
2019-03-02 16:28:36 +01:00
184f17afad
src/api2/admin/datastore.rs: impl list_snapshots
2019-03-02 16:20:50 +01:00
af53186e6a
src/tools.rs: add simply string join
2019-03-02 16:15:45 +01:00
812c6f8716
src/api2/admin/datastore.rs: impl list_backup_groups
2019-03-02 11:29:05 +01:00
e909522f06
src/backup/datastore.rs: use timestamp with zone offset
2019-03-01 17:36:40 +01:00
dc4c09faae
src/backup/datastore.rs: split BackupDir
2019-03-01 16:26:35 +01:00
38b0dfa511
src/backup/datastore.rs: split BackupInfo
2019-03-01 16:26:35 +01:00
dcd15d10bb
backup/index: remove IndexIterator
...
removed the need for it, to avoid copies
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-01 13:29:36 +01:00
17d6979afd
src/bin/proxmox-backup-client.rs: generate a log file
2019-03-01 09:35:41 +01:00
3b151414f8
src/tools/file_logger.rs: class to log into files
2019-03-01 09:34:29 +01:00
5a2892f0ae
src/bin/proxmox-backup-client.rs: use chrono::DateTime
2019-03-01 06:59:32 +01:00
51144821f9
src/bin/proxmox-backup-client.rs: login before starting backup
2019-03-01 06:48:41 +01:00
541b908ea4
src/backup/datastore.rs: use DateTime<Utc> instead of unix epoch
2019-02-28 17:03:48 +01:00
b55cee9227
src/cli/command.rs: generate_usage_str() - fix option separator (newline)
2019-02-28 16:20:00 +01:00
c4c5174000
src/cli/command.rs: get_schema_type_text() - display min/max
2019-02-28 16:11:49 +01:00
df67070b1c
client/backup_repo: allow ip addresses for BackupRepository
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
5de2bced2d
datastore: add generic open_index
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
7bc1d72778
add IndexFile trait
...
We want to be able to iterate through digests of index
files, but without always having to distinguish between
dynamic and fixed types, so add a trait we can use as a
trait object.
Unfortunately the iterator needs to yield copies as
iterators cannot yield values with life times when
represented as trait objects (Box<dyn Iterator<Item = ?>>)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
5be4065baf
make FixedIndexReader Send and update comments
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
fded74d0c7
style: BufferedReader -> BufferedRead
...
Single-method traits usually use the same name as their
method and aren't usually the 'noun' (which is usually an
implementation of them instead).
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-28 15:26:40 +01:00
8f57971714
src/api2/admin/datastore.rs: implement prune
2019-02-28 12:51:27 +01:00
83b7db0209
src/api2/admin/datastore.rs: start prune api
...
just a dummy for now.
2019-02-27 16:53:17 +01:00
0ffbccce5e
rc/server/rest.rs: allow to pass parameters as application/json
2019-02-27 12:37:53 +01:00
f98ea63de2
src/bin/proxmox-backup-client.rs: fix bash completion
2019-02-27 12:37:53 +01:00
8731e40a7f
backup/datastore: create_backup_dir: show if path existed
...
To enable asserting the creation of a new backup rather than
adding to an existing one.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-27 10:15:36 +01:00
25f1650b71
src/bin/proxmox-backup-client.rs: minor code cleanup
2019-02-27 07:21:57 +01:00
9bc3ddb822
src/cli/command.rs: improve generate_usage_str
...
correctly format positional array arguments ({<name>}).
2019-02-26 12:40:51 +01:00
ae0be2dda9
src/bin/proxmox-backup-client.rs: allow to upload more that one directory
2019-02-26 12:27:28 +01:00
a477d6889a
src/client/http_client.rs: login once, store and reuse ticket/token
2019-02-26 11:59:10 +01:00
d02c2ed701
src/cli/getopts.rs: cleanup, avoid compiler warning
2019-02-26 11:55:06 +01:00
f8dfbb45c0
src/tools.rs: new helper required_array_param()
2019-02-26 08:48:17 +01:00
aaba6faace
src/cli/getopts.rs: support positional array arguments
2019-02-26 08:47:34 +01:00
f33fa27307
src/cli/command.rs: improve error message
2019-02-26 08:46:36 +01:00
92a173feb0
src/api_schema/schema.rs: improve ParameterError display function
2019-02-26 07:11:24 +01:00
f661f37427
backup/dynamic_index: split out add_chunk
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:50:45 +01:00
9ac6ec868a
backup/chunk_store: split insert_chunk
...
The protocol handler will receive chunk data plus a hash
pre-calculated by the client. It will verify the hash before
sending it up to the datastore in order to respond to the
client with an error on a mismatch, so there's no need to
recalculate the hash another time.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
3a50ddd0c9
backup/chunk_store: mark ChunkIterator as fused
...
Ranges are FusedIterators and it can be useful to propagate
that property to users.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
eff25ecaf2
backup/chunk_store: make percentage output optional
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
d59397e60e
backup/datastore: expose chunk_store.get_chunk_iterator
...
will be used by the protocol handler to iterate over
existing chunks
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-25 13:44:29 +01:00
8eb1203bda
src/backup/chunk_stat.rs: log write speed
2019-02-25 13:21:00 +01:00
7e3365554e
src/backup/chunk_stat.rs: new struct to track chunk statistics
2019-02-25 12:52:10 +01:00
798f7fa065
src/backup/chunk_store.rs: return info about compressed chunk size
...
So that we can generate better statistics ..
2019-02-25 11:36:05 +01:00
78216a5ab1
src/backup/chunk_store.rs: use lz4 for compression
...
And depend on lz4 lib now.
2019-02-25 10:00:28 +01:00
793b0f4d77
src/cli/command.rs: improve help command completion
2019-02-23 18:07:32 +01:00
6949d91564
src/cli/command.rs: cleanup, avoid array manipulation
2019-02-23 17:50:33 +01:00
9d78d57960
src/cli/command.rs: implement bash completion for help
2019-02-23 17:38:10 +01:00
698d9d4402
src/cli/command.rs: add help command
2019-02-23 15:10:48 +01:00
8b6dd2240d
src/cli/command.rs::allow to generate ReST docu
2019-02-23 11:29:18 +01:00
2f3f2bb77f
src/cli/command.rs: improve usage imformation
2019-02-22 17:40:37 +01:00
f252ca654c
src/backup/chunker.rs: simplify break test
...
To make it a little bit faster ...
2019-02-22 13:02:11 +01:00
6adb40eefc
src/cli/command.rs: start implementing generate_usage_str()
2019-02-22 12:02:59 +01:00
33256db6dd
src/cli/command.rs: simplify cli handler
2019-02-22 11:15:10 +01:00
61834c1cbf
src/backup/chunker.rs: another try to speedup the chunker
2019-02-22 11:15:10 +01:00
532a69eb04
src/cli/getopts.rs: improve docs
2019-02-21 09:36:52 +01:00
4de0e142a0
cleanup cli library structure
2019-02-21 09:08:03 +01:00
56458d9764
http_client: on a tty, read password if no env var is set
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-20 15:03:34 +01:00
c9b296f117
tools: add tty helper module
...
with read_password() and stdin_isatty() functions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-20 14:43:07 +01:00
8cc0d6aff3
src/bin/proxmox-backup-client.rs: implement garbage-collection sub-command
2019-02-20 14:10:45 +01:00
81da38c143
src/client/http_client.rs: implement post
...
without parameters for now...
2019-02-20 14:09:55 +01:00
8c75372b79
src/backup/datastore.rs: list_backups() - include list of index files
2019-02-20 13:37:44 +01:00
14691fc1f3
src/bin/test_chunk_size.rs: avoid compilöer warnings, exit after 1GB
2019-02-20 12:05:28 +01:00
1604ec2654
src/bin/test_chunk_size.rs: another chunker test - gather some stats
2019-02-20 11:50:10 +01:00
fa4ebf0fce
src/backup/chunker.rs: simplify code
2019-02-20 11:17:19 +01:00
1b9c959c32
src/backup/chunker.rs: change window size back to 48
...
Using 64 is a bad idea, because it always computes hash 0 for
constant data streams.
2019-02-20 11:01:09 +01:00
247cdbce72
src/api2/admin/datastore/catar.rs: allow to configure chunk-size
2019-02-19 15:19:12 +01:00
c584aa21fd
src/backup/chunker.rs: start() - correctly store hash in self.h
2019-02-19 12:40:12 +01:00
cd1598302a
src/backup/chunker.rs: use 64 byte chunker window size
...
This results in faster code.
2019-02-19 12:27:37 +01:00
9a3e869cf5
src/backup/chunker.rs: simplify buzzhash start code
2019-02-19 12:17:17 +01:00
1c7a88aeff
UploadCaTar: currectly write all data using write_all()
2019-02-18 18:31:02 +01:00
4d2881dd4c
src/backup/chunker.rs: correctly compute window index 'idx'
...
And add a simple test.
2019-02-18 16:59:23 +01:00
a4a5c78c01
src/client/http_client.rs: send csrf token
2019-02-18 16:59:23 +01:00
fc7f03523c
cleanup
...
Error::from is already a function taking 1 parameter,
there's no need to wrap it with `|e| Error::from(e)`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-18 13:21:27 +01:00
8fdef1a81c
fix typo: confi{g}dir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-18 13:11:42 +01:00
1cb99c234e
src/server/rest.rs: correctly insert NoLogExtension()
2019-02-18 06:54:12 +01:00
591f570b22
src/client/http_client.rs: trim path
2019-02-18 06:24:28 +01:00
7f1685230a
src/server/rest.rs: get_index() include username and CSRF token
...
When we have an valid ticket. Also delay get_index() if called with
an invalid ticket.
2019-02-17 19:28:32 +01:00
141de8374a
src/server/rest.rs: factor our normalize_path()
2019-02-17 17:31:53 +01:00
8225aa2ff6
src/server/rest.rs: improve logs for unauthorized request
2019-02-17 17:18:44 +01:00
54d15ab5c5
src/api_schema/schema.rs: improve docu
2019-02-17 12:13:06 +01:00
51ebd07974
src/api_schema/router.rs: add docu for RpcEnvironmentType
2019-02-17 11:59:18 +01:00
d21c2710da
api_schema/router.rs: improve doc
2019-02-17 10:30:41 +01:00
ef2f2efbcc
improve api_schema module structure
2019-02-17 10:16:33 +01:00
dc9a007b11
rename src/api to src/api_schema
2019-02-17 09:59:20 +01:00
5ddf8cb10a
cleanup auth code, verify CSRF prevention token
2019-02-16 15:52:55 +01:00
95aee7e032
tools/common_regex.rs: add more docu
2019-02-16 12:36:01 +01:00
99be8844c7
buildcfg.rs: improve doc
2019-02-16 12:31:39 +01:00
a8f268afbb
fix configuration dir permission
...
Carefully set and check permissions ...
2019-02-16 12:19:13 +01:00
aada2a9719
config/datastore.rs: remove unused code
2019-02-16 10:10:45 +01:00
d11594db4c
config/datastore.rs: return empty config if file does not exist
2019-02-16 10:06:08 +01:00
12400210a2
tools::file_get_contents - improve error message
2019-02-16 09:42:00 +01:00
6235a41862
tools::file_read_firstline - improve error message
2019-02-16 09:36:29 +01:00
17ed456c2e
change proxy user from www-data to backup
2019-02-16 09:29:04 +01:00
5d14eb6a76
tools.rs: new helper to get uid/gid for the sepcified system user.
...
Or is there an easier way to get that info?
2019-02-15 15:33:12 +01:00
f0a6112451
backup/datastore.rs: improve error reporting
2019-02-15 14:33:03 +01:00
9b50c16103
section_config.rs: simplify parser by using new try_block macro
2019-02-15 12:13:15 +01:00
fe651dd6e5
tools.rs: fix doc tests
2019-02-15 11:33:51 +01:00
e80503d2a6
tools.rs: document try_block macro
2019-02-15 11:20:45 +01:00
f0dbba8cbe
tools.rs: new try_block macro
2019-02-15 10:59:12 +01:00
1d77b6cf6b
update to pam 0.7 (renamed from pam-auth)
...
It now supports custom conversation methods, so instead of
new() we ask for a default authenticator taking a password
via with_password(). Since the password is now handled by
the now separate conversation handler, `set_credentials()`
is now called on the handler we get via `.get_handler()`.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-15 10:34:40 +01:00
d4736445fe
server/rest.rs: add method to log message
2019-02-15 10:16:12 +01:00
f02e6fc478
api2/admin/datastore/catar.rs: simplify/fix debug message
2019-02-15 10:04:37 +01:00
44c00c0dfd
server/rest.rs: log full error messages
2019-02-15 09:55:12 +01:00
7171b3e079
server/rest.rs: use a protocol extension to avoid double log
...
Instead of modifying the response header itself.
2019-02-14 16:04:24 +01:00
7e03988caf
avoid double logging of proxied requests
2019-02-14 13:28:41 +01:00
af6f80d3db
Revert "backup/datastore: fn open is a private implementation detail"
...
This reverts commit 3dacd9a660
2019-02-14 13:13:49 +01:00
78a1fa6722
src/server/rest.rs: log failed requests
2019-02-14 13:07:34 +01:00
2cc90617dd
tools/fs: mark ReadDir as Send
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
62f2422f6a
backup/chunk_store: verify chunk file names
...
get_chunk_iterator() should skip over files which aren't an
actual chunk
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
3dacd9a660
backup/datastore: fn open is a private implementation detail
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
9739aca41f
backup/chunk_store: open base dir in get_chunk_iterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 12:37:31 +01:00
db4868ef4f
src/client/backup_repo.rs - improve docu
2019-02-14 11:17:08 +01:00
151c6ce27d
src/client.rs: improve library structure and docu
2019-02-14 11:11:39 +01:00
dd1bfe6c12
tools/timer: add module documentation
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:13:42 +01:00
c7f481b642
backup: replace ChunkIterator
...
Instead we just build the iterator using our
tools::fs::read_subdir iterator as follows:
Use the index range (0..0x10000) as an iterator, map() it
to yield a subdirectory iterator for each index, then
flatten it.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
121f18efb7
tools: use scan_subdir iterator for scandir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
32286b039c
tools/fs: add file_name_utf8_unchecked shortcut
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
806d7a6a35
tools/fs: change scan_subdir to use filter_file_name_regex
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
53e1e7ca72
tools/fs: add filter_file_name_regex to FileIterOps
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
23fba9d777
tools/fs: add FileIterOps trait and FileTypeFilter
...
FileIterOps will provide operations on file entry iterators.
For now this provides a file type filter.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
1db416150d
tools/fs: let ReadDir iterator yield ReadDirEntry
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
b12505da69
tools/fs: add ReadDirEntry helper
...
This wraps nix::dir::Entry with the parent and additionally
exposes the parent directory file descriptor since we need
it to perform openat().
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
a25f863afd
tools/fs: add scan_subdir helper
...
This filters the results of read_subdir with a regex.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
b4d5787de9
tools: start fs submodule, add read_subdir
...
This creates an iterator relative to a RawFd via
nix::dir::Dir over nix::dir::Entrys.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-14 10:09:59 +01:00
0dffe3f99a
src/client/http_client.rs: try to login
...
use an environment var to store passphrase (PBS_PASSWORD)
2019-02-13 14:31:43 +01:00
33d64b81e0
src/bin/proxmox-backup-client.rs: define and use BackupRepository
2019-02-13 12:30:52 +01:00
4a3f65172c
client: use hyper-tls for now
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-13 11:03:09 +01:00
42d6e4fb05
tools: borrow: implement Deref for Tied
...
Eg. if you have an x: Tied<Dir, Iterator> now you can simply
call x.next() as it'll dereference to the Iterator.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-13 11:03:09 +01:00
541a302224
api2/admin/datastore/catar.rs: use nested file extensions
...
i.e. archive.catar.didx
2019-02-13 10:37:00 +01:00
e5064ba607
simplify backup lib structure (pub use xxx:*), improve doc
2019-02-12 14:13:31 +01:00
d78345bcfc
src/backup.rs - improve doc
2019-02-12 13:27:11 +01:00
93d5d77952
rename ArchiveIndex to DynamicIndex
...
also changed the file extension from .aidx to .didx
2019-02-12 12:05:33 +01:00
91a905b6dd
rename ImageIndex to FixedIndex
...
also changed the file extension fron .iidx to .fidx
2019-02-12 11:50:45 +01:00
82bc0ad40c
backup: fixup warnings and readd lost file type check
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:43:43 +01:00
9349d2a1d2
backup: cleanup long lines
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
fdd71f5286
backup: use ChunkIterator for sweep_unused_chunks
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
c9e7f4dc8b
backup: add optional progress callback to ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
4070096c6a
backup: add ChunkIterator
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
6ed25cbe3a
tools: add Tied value
...
Tie two dependent values together, such as a nix::Dir with
its nix::dir::Iter<'a> where 'a is tied to the Dir's
lifetime, making it otherwise impossible to return them or
store them in a struct.
Alternatively we could try the 'rental' crate.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
9ac1045c7b
bin: manager: fixup datastore parameter name
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-12 10:39:11 +01:00
8f8d5a420c
avoid compiler warning
2019-02-11 15:12:01 +01:00
aa5a4060ab
bin/proxmox-backup-api.rs: improve error handling
2019-02-11 15:11:42 +01:00
4223d9f800
bin/proxmox-backup-proxy.rs: improve error handling
2019-02-11 14:43:26 +01:00
ace9e3531a
access: use proxmox-backup-auth for pam
...
allows customization via /etc/pam.d/proxmox-backup-auth
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-07 13:40:22 +01:00
13f8310cac
api2: time: use /etc/localtime as fallback
...
in case there is no /etc/timezone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-05 14:14:56 +01:00
a7cac14ed9
tools: use std::fs::read for file_get_contents
...
Until we add the extra options like we have in pve, just use
the already existing (since 1.26) shortcut.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:35:32 +01:00
e937034c86
remove some rather inconvenient debug output
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:34:38 +01:00
9d02e5c38a
proxy: use <configdir>/server.pfx for tls
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:19:32 +01:00
9f4962d396
introduce buildcfg module and PROXMOX_CONFIGDIR
...
buildcfg.rs should contain convenience variables or macros
for using build-time configured variables
For now we replace hardcoded "/etc/proxmox-backup/<foo>"
with configdir!("<foo>").
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:13:08 +01:00
0d176f3681
proxy: use TLS via tokio-tls
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 14:56:07 +01:00
e0a5d1ca2f
tests: chunk_store: cleanup .testdir
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
a4f9188300
tests: import lazy_static as needed
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
6285b251e7
don't hardcode /usr/share paths
...
Add a defines.mk file defining the standard default layout
for installation (PREFIX, DATAROOTDIR, LIBDIR etc.).
Add a $(JSDIR) variable and use env!("JSDIR") in place of
the hardcoded /usr/share/javascript/proxmox-backup path.
Have defines.mk include an *optional* local.mak to override
JSDIR to www/ during local development.
Add `make install` targets and remove debian/install.
Note that service files are now in $libdir/proxmox-backup/
as there's usually no point in starting them from the
command line, so /usr/sbin makes no sense.
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
8e4df103dd
remove useless command option from daemons
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
6278365fd9
improve ApiMethod docu
2019-02-01 10:38:07 +01:00
4b2cdeb9a6
implement relead_timezone flag
2019-02-01 10:04:46 +01:00
c82bc1a1f9
depend on pam-auth, use pam for root login
...
Please use username "root@pam" for login via GUI.
2019-02-01 09:30:50 +01:00
a154a8e8a4
delay unauthorized request (rate limit)
2019-01-31 14:34:21 +01:00
5996577ab6
move http error class to router.rs
2019-01-31 13:22:30 +01:00
b9903d6331
server/rest.rs: verify auth cookie
2019-01-31 12:22:00 +01:00
f484eed3c2
Utils.js: fix cookie handling
...
Use unsecure cookie foör testing.
2019-01-31 10:08:08 +01:00
9f49fe1d5d
avoid compiler warnings
2019-01-30 18:25:37 +01:00
34f956bc25
api2/access.rs: add ticket api
2019-01-30 15:16:10 +01:00
1bf446a33e
fix timer_delete call
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-01-30 12:04:20 +01:00
cf671670a0
auth_helpers.rs: add timestamp to csrf token
2019-01-29 17:50:03 +01:00
1e76cbc6a2
auth_helpers.rs: implement assemble_csrf_prevention_token
2019-01-29 17:41:45 +01:00
d01e2420f7
load auth keys on startup
2019-01-29 17:21:58 +01:00
6c30068ebf
auth_helpers.rs: split code into separate file
2019-01-29 16:55:49 +01:00
39a90ca6c5
bin/proxmox-backup-api.rs: gererate secret for csrf token
2019-01-29 13:15:33 +01:00
26027a151e
tools/ticket.rs: remove debug statements
2019-01-29 13:04:01 +01:00
8d04280b29
tools/ticket.rs: add basic ticket support
2019-01-29 12:59:07 +01:00
c8f3f9b1af
server/rest.rs: simplify proxy code
...
Only pass neccessary parameters.
2019-01-28 18:22:16 +01:00
a859fa95a9
add protected flag for some api methods
2019-01-28 18:16:47 +01:00
a3da38ddfd
server/rest.rs: implement proxy_sync_api_request
2019-01-28 18:06:42 +01:00
f120483353
server/rest.rs: add proxy_sync_api_request() dummy
2019-01-28 17:30:39 +01:00
4e4df8e2f0
api/router.rs: add new 'protected' flag to ApiMethod
2019-01-28 17:18:42 +01:00
5bb3398171
server/formatter.rs: fix extjs error format
2019-01-28 13:44:48 +01:00
02c7a75520
src/bin/proxmox-backup-proxy.rs: implement unpriviledged server
...
We want to run the public server as user www-data. Requests needing
root priviledges needs to be proxied to the proxmox-backup.service, which
now listens to 127.0.0.1:82.
2019-01-28 13:29:58 +01:00
48849593ee
api2/node/services.rs: implement service commands
2019-01-27 12:40:31 +01:00
d7d23785f0
RpcEnvironment: implement set_user() and get_user()
2019-01-27 10:52:26 +01:00
162b979394
RpcEnvironment: add environment type enum RpcEnvironmentType
2019-01-27 10:33:42 +01:00
e82dad9700
also pass rpcenv to async handlers
2019-01-27 10:18:52 +01:00
d2ab5f19e2
api2/node/services.rs: start services api
2019-01-26 17:54:18 +01:00
0f253593c6
move rpc environment implementation to separate files
2019-01-26 15:08:02 +01:00
6049b71f41
api: pass RpcEnvirnment to api handlers
2019-01-26 14:50:37 +01:00
d96d82736d
use crate log and syslog
2019-01-26 10:56:11 +01:00
e182ab4a4a
api2/node/syslog.rs: improve error handling
2019-01-26 10:23:52 +01:00
0441886842
api2/node/syslog.rs: implement dump journal
2019-01-25 18:20:51 +01:00
4f9a726833
api2/node/syslog.rs: add syslog api schema
2019-01-25 17:17:30 +01:00
1631c54fca
tools::get_hardware_address - fix FILENAME, use file_get_contents()
2019-01-25 12:23:47 +01:00
af2fddea4f
api2/node/dns.rs: implement concurrent update protection
2019-01-25 11:38:59 +01:00
de6b0721fa
pi2/node/dns.rs: return file digest
2019-01-25 11:03:41 +01:00
2296860027
tools.rs: move digest_to_hex() into tools
2019-01-25 10:58:28 +01:00
53157ca6cf
tools.rs: implement file_get_contents()
2019-01-25 10:44:40 +01:00
46b79b9ee5
api2/node/dns.rs: improve dns api
2019-01-25 10:15:32 +01:00
332dcc2277
backup/chunk_store.rs: fix test case - use absolute path
2019-01-25 10:14:25 +01:00
8f973f8181
api2/node/dns.rs: implement resolv.conf parser
2019-01-24 14:59:40 +01:00
784252dbc5
backup/datastore.rs: use single lazy_static block
2019-01-24 14:58:47 +01:00
e6ffeb918c
api2/node/time.rs: implement set timezone
2019-01-24 12:05:06 +01:00
cd8b91f78c
api2/node/time.rs: add return schema
2019-01-24 11:07:11 +01:00
447787ab7c
tools.rs: implement file_read_firstline
2019-01-24 10:43:30 +01:00
0463602a79
api2/node/time.rs: implement read api
2019-01-23 15:14:14 +01:00
b2b3485d5f
src/api2/node: start node configuration api
2019-01-23 13:09:17 +01:00
d15009c0ce
server/rest.rs: fake login cookie
2019-01-23 12:49:10 +01:00
248cb51862
add missing api2.rs
2019-01-22 13:56:27 +01:00
7e13b2d67f
api2/subscription.rs: add subscription api class
2019-01-22 12:50:19 +01:00
576e3bf252
rename api3 back to api2
...
There is no real need to change the path, so using api2 we can reuse
all helpers (like tools from proxmox widget toolkit).
2019-01-22 12:10:38 +01:00
1adb353d6f
lient/http_client.rs: simplify code
2019-01-22 10:36:04 +01:00
41c039e143
bin/proxmox-backup-client.rs: implement list command
2019-01-21 18:58:14 +01:00
1fdb4c6ffa
client/http_client.rs: implement get, share common code
2019-01-21 18:56:48 +01:00
83dbd80bd8
api3/admin/datastore.rs: implement list backups
2019-01-21 11:22:55 +01:00
7ca80246ea
backup/datastore.rs: use DateTime<Utc> to store backup time
2019-01-21 10:51:52 +01:00
Dietmar Maurer
Christian Ebner
Wolfgang Bumiller
Thomas Lamprecht
Oguz Bektas