Commit Graph

2062 Commits

Author SHA1 Message Date
Dietmar Maurer 68da20bf62 src/api2/types.rs: define NETWORK_INTERFACE_NAME_SCHEMA 2020-04-21 17:54:52 +02:00
Dietmar Maurer c357260d09 src/config/network.rs: move type definitions to src/api2/types.rs 2020-04-21 17:25:05 +02:00
Dietmar Maurer 7e02d08cd0 rename ConfigMethod to NetworkConfigMethod 2020-04-21 17:17:57 +02:00
Dietmar Maurer ca0e534796 src/api2/config/network.rs: start network configuration api 2020-04-21 14:28:26 +02:00
Dietmar Maurer 904e988667 src/config/network.rs: impleement load/save 2020-04-21 12:55:33 +02:00
Dietmar Maurer 3f129233be src/config/network.rs: add Interface flags 'exists' and 'active' 2020-04-21 11:46:56 +02:00
Dietmar Maurer a9bb491e35 src/config/network.rs: cleanup autostart flag handling 2020-04-21 11:06:22 +02:00
Dietmar Maurer 1ec7f8a0dd src/config/network/helper.rs: new helper get_network_interfaces() 2020-04-21 10:32:54 +02:00
Dietmar Maurer 92310d585c src/config/network.rs: simplify code 2020-04-20 18:10:15 +02:00
Dietmar Maurer f34d4401f7 src/config/network.rs: read/write /etc/network/interfaces
Start implementing a recursive descent parser.
2020-04-20 14:15:57 +02:00
Dietmar Maurer 6e695960ca src/config/cached_user_info.rs: cache it up to 5 seconds 2020-04-18 08:49:20 +02:00
Dietmar Maurer 365f0f720c fix permission tests using non-uri parameters
We nood to do those tests inside the fuction body instead...
2020-04-18 08:23:04 +02:00
Dietmar Maurer a737179eb4 src/config/cached_user_info.rs: new check_privs helper 2020-04-18 08:09:34 +02:00
Dietmar Maurer bb072ba49c src/api2/access.rs: cleanup 2020-04-18 07:28:25 +02:00
Dietmar Maurer ff329f970b src/api2/types.rs: use anyhow::Error in test cases 2020-04-18 07:05:31 +02:00
Wolfgang Bumiller f7d4e4b506 switch from failure to anyhow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Dietmar Maurer 404d78c41e src/api2/pull.rs: add access permission 2020-04-17 15:27:04 +02:00
Dietmar Maurer 1bfc1efa50 src/api2/subscription.rs: add access permissions 2020-04-17 15:14:28 +02:00
Dietmar Maurer 73ce1d1146 src/api2/reader.rs: add access permissions 2020-04-17 15:01:56 +02:00
Dietmar Maurer 70e5f2461d src/api2/config/remote.rs: add access permissions 2020-04-17 14:57:26 +02:00
Dietmar Maurer c0ef209aeb src/api2/config/datastore.rs: impl digest check for delete, add access permissions 2020-04-17 14:51:29 +02:00
Dietmar Maurer 9f9f7eefa3 src/api2/backup.rs: add access permissions 2020-04-17 14:40:20 +02:00
Dietmar Maurer bb34b58910 src/api2/admin/datastore.rs: add access permissions - first try
We need to refine this later (introduce backup owner concept?)
2020-04-17 14:36:27 +02:00
Dietmar Maurer 5972def5ec acl: change path "storage" to "datastore" 2020-04-17 14:15:44 +02:00
Dietmar Maurer aa90ced3bf src/api2/access/role.rs: use schema ACL_ROLE_SCHEMA 2020-04-17 14:14:06 +02:00
Dietmar Maurer ca257c8097 move type defs from src/api2/access/acl.rs to src/api2/types.rs 2020-04-17 14:13:15 +02:00
Dietmar Maurer 3fff55b293 src/api2/access/role.rs: new api to list roles 2020-04-17 14:03:24 +02:00
Dietmar Maurer 4f66423fcc src/api2/access/user.rs: add access permissions 2020-04-17 11:04:36 +02:00
Dietmar Maurer d4f020f4c5 src/api2/access/user.rs: add access permissions 2020-04-17 10:08:45 +02:00
Dietmar Maurer d28ddb8e04 src/api2/access/acl.rs: add access permissions 2020-04-17 10:03:09 +02:00
Dietmar Maurer 83b6a7cf71 src/api2/node/tasks.rs: use api macro, implement access permissions 2020-04-16 17:47:21 +02:00
Dietmar Maurer e4681f9f71 src/api2/node/syslog.rs: add access permissions 2020-04-16 17:08:19 +02:00
Dietmar Maurer b5037fa8ed src/api2/node/status.rs: add access permissions 2020-04-16 17:05:09 +02:00
Dietmar Maurer 9989d2c4e9 src/server/rest.rs: reduce delay for permission error to 500ms 2020-04-16 12:56:34 +02:00
Dietmar Maurer 1cf7bbf412 src/api2/node/services.rs: add access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer 68ed0c629d src/api2/node/journal.rs: add access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer 4b40148caa start impl. access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer 423e656163 src/config/cached_user_info.rs: new helper class 2020-04-16 10:05:16 +02:00
Dietmar Maurer 109d7817cd src/config/user.rs - cached_config: do not store/return digest 2020-04-15 11:35:57 +02:00
Dietmar Maurer 5354511fd0 src/config/acl.rs: implement cached_config 2020-04-15 11:30:47 +02:00
Dietmar Maurer bd098a7f77 src/api2/node/dns.rs: use api macro (cleanup) 2020-04-15 10:09:18 +02:00
Dietmar Maurer 8d048af2bf acl: improve NoAccess handling 2020-04-15 08:11:43 +02:00
Dietmar Maurer 9a328319dd pxar extract: remove pattern from arg_param, add target instead 2020-04-15 06:41:37 +02:00
Dietmar Maurer 7e3d2e5b41 pxar create: remove exclude from arg_param 2020-04-15 06:31:46 +02:00
Dietmar Maurer 9c06f6c292 fix previous commit - use result. 2020-04-14 17:48:10 +02:00
Dietmar Maurer 9f4e47dd93 acl update: check path 2020-04-14 17:23:48 +02:00
Dietmar Maurer d83175dd69 acl update: check if user exist. 2020-04-14 13:46:27 +02:00
Dietmar Maurer 68ccdf09a4 src/config/user.rs: implement user config cache 2020-04-14 13:45:45 +02:00
Dietmar Maurer 9765092ede acl api: implement update 2020-04-14 10:16:49 +02:00
Dietmar Maurer ed3e60ae69 start ACL api 2020-04-13 11:09:44 +02:00
Dietmar Maurer a83eab3c4d acl: use BTreeMap and BTreeSet to avoid sort() 2020-04-12 17:13:53 +02:00
Dietmar Maurer 0815ec7e65 acl: implement roles(), add regression tests. 2020-04-12 13:06:50 +02:00
Dietmar Maurer 5c6cdf9815 add acl config 2020-04-11 12:24:26 +02:00
Dietmar Maurer b88f9c5b1e PASSWORD_SCHEMA: set max_length to 1024 (for tickets) 2020-04-09 13:35:58 +02:00
Dietmar Maurer 879546aff6 api: add default property to domain list 2020-04-09 13:35:08 +02:00
Dietmar Maurer 73b40e9b46 api: correctly sort access subdirmap 2020-04-09 13:34:07 +02:00
Dietmar Maurer 708db4b3ae api: add list_domains 2020-04-09 11:36:45 +02:00
Dietmar Maurer 685e13347e api: move config/user to access/users, implement change_password
To make it similar to the pve api
2020-04-09 10:21:24 +02:00
Dietmar Maurer 7d817b0358 implement auth framework 2020-04-08 14:06:15 +02:00
Dietmar Maurer 579728c641 add user configiguration 2020-04-08 14:06:15 +02:00
Dietmar Maurer cf459b1982 gc: log pending removals 2020-04-06 09:50:40 +02:00
Dietmar Maurer dda7015497 prune api: return a usable result (we run synchronous anyways) 2020-04-01 12:24:28 +02:00
Dietmar Maurer 4299ca727c src/server/rest.rs: use correct formatter 2020-03-26 12:54:20 +01:00
Dominik Csapak 6650a242fb rewrite future select in upgrade_to_backup_protocol using select macro
and handle all ok/err cases with at least logging

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-03-18 11:33:59 +01:00
Dietmar Maurer ca26db1140 delete unused src/storage/config.rs 2020-03-02 13:31:38 +01:00
Dietmar Maurer 90c5239d46 use SectionConfig from proxmox 0.1.18-1 2020-03-02 12:52:11 +01:00
Dietmar Maurer 18deda403d src/bin/proxmox-backup-client.rs - list_groups: render full snapshot path for last-backup column 2020-02-28 09:09:22 +01:00
Dietmar Maurer 6882b3716c src/bin/proxmox-backup-manager.rs: avoid compiler warning 2020-02-28 08:52:12 +01:00
Christian Ebner 32d192a952 catalog: shell: Use the new logic including resolving symlinks for catalog
and remove the old unused code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:51:01 +01:00
Christian Ebner fee5528e59 catalog: shell: introduce new CatalogPathStack to navigate in catalog shell.
This is basically a rewrite of the current logic for navigating the catalog,
but in addition allows to follow symlinks.
Following symlinks introduces the issue that generation of canonical paths
(needed in the actual pxar archive) is more complex, as symlinks have to be
resolved and loops avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:50:43 +01:00
Christian Ebner 4145c36749 catalog: add missing function documentation.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:50:08 +01:00
Christian Ebner c2f9149461 catalog: introduce is_symlink() to check if DirEntry is a symlink.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:49:55 +01:00
Christian Ebner 11ee5c0563 catalog: derive PartEq for equality checks of two DirEntry's.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:49:39 +01:00
Christian Ebner 536683e73b src/backup/dynamic_index.rs: Add LruCache for chunks.
In order to improve non-sequential reads of chunks as e.g. in FUSE.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:47:16 +01:00
Christian Ebner 35ddf0b419 catalog: shell: Introduce clear-selected command.
'clear-selected' allows to clear all the match patterns from the list of
patterns for a subsequent restore.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:40:55 +01:00
Christian Ebner 8e464141cf catalog: shell: Improve list-selected command.
'list-selected' now shows the filenames matching the patterns for a restore
instead of the patterns themselfs.
The patterns can be displayed by passing the '--pattern' flag.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:40:42 +01:00
Christian Ebner 351b913d1e pxar::decoder: Return the correct symlink target size.
This eliminates also repeated calls to readlink in fuse, which occur when the
preallocated buffer to store the symlink target path is to small.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:38:56 +01:00
Christian Ebner 189996cf4a proxmox-backup-client: expose exclude match patterns to cli.
Allows to pass exclude match patterns to the 'backup' command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 07:56:09 +01:00
Dietmar Maurer ac3faaf5c0 src/bin/proxmox-backup-manager.rs: cleanup output handling 2020-02-28 07:42:36 +01:00
Dietmar Maurer 4939255fb4 src/tools/format.rs: move output rendering code to this new file 2020-02-28 07:30:35 +01:00
Dietmar Maurer c81b2b7c70 src/bin/proxmox-backup-client.rs - list_groups: use format_and_print_result_full()
Depend on proxmox 0.1.16-1
2020-02-27 13:32:11 +01:00
Dietmar Maurer c2043614f7 src/bin/proxmox-backup-client.rs: use new output formath helpers from proxmox 0.15.1-1 2020-02-27 12:41:15 +01:00
Dietmar Maurer e23f586344 src/bin/proxmox-backup-client.rs - status: improve alignment 2020-02-27 11:38:09 +01:00
Dietmar Maurer be2425ff85 src/bin/proxmox-backup-client.rs - status: do not print header 2020-02-27 11:31:05 +01:00
Dietmar Maurer f24fc1166b src/bin/proxmox-backup-client.rs - list_snapshots: use format_and_print_result_full()
Depend on proxmox v0.1.14.
2020-02-27 11:28:44 +01:00
Christian Ebner af934f8cf6 tools::lru_cache: Improve access() and insert() by using HashMap::entry().
entry() allows to lookup the position where and entry belongs and update/insert
it in the HashMap more efficiently than get_mut() and insert().
Details: https://gankra.github.io/blah/hashbrown-insert/

In addition, use the struct LinkedList and remove the outdated code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:56:25 +01:00
Christian Ebner e3ab9a383c tools::lru_cache: Separate LinkedList part of LruCache into own struct.
In order to keep the separation more aparent and avoid mut borrow conflics.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:55:48 +01:00
Christian Ebner 75c2ee7bab tools::lru_cache: Make key generic.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:55:16 +01:00
Christian Ebner 03f779c6f5 catalog: shell: Improve output of stat command.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:53:17 +01:00
Dietmar Maurer ea5f547fdc src/bin/proxmox-backup-client.rs - list_snapshot_files: use format_and_print_result_full() 2020-02-26 13:49:47 +01:00
Dietmar Maurer 390c5bdde4 src/bin/proxmox-backup-client.rs - status: use format_and_print_result_full 2020-02-26 13:29:00 +01:00
Dietmar Maurer 03ac286c80 src/bin/proxmox-backup-manager.rs: use crate::tools::runtime::block_on() 2020-02-26 11:37:38 +01:00
Dietmar Maurer dd04383bb8 src/tools/runtime.rs: call OPENSSL_thread_stop to avoid race with openssl cleanup handlers 2020-02-26 10:38:55 +01:00
Dietmar Maurer ca2dbb8af1 src/client/backup_writer.rs: also print archive size 2020-02-25 14:12:29 +01:00
Wolfgang Bumiller 0bcbb5434e cleanup unused module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-25 11:38:27 +01:00
Christian Ebner 495b238bc1 pxar: Change text of warning for files matched by exclude patterns.
In addition to the .pxarexclude files, glob match patterns can be passed to pxar
also via cli parameters.
Therefore the warning is rephrased to be more ambiguous.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-25 10:47:24 +01:00
Dietmar Maurer 6da73c823f src/client/backup_writer.rs: prefix upload stats with archive name 2020-02-24 13:24:46 +01:00
Dietmar Maurer e02c3d461f src/client/backup_writer.rs: implement verbose options 2020-02-24 12:48:40 +01:00
Dietmar Maurer ee5fe978e2 src/client/backup_writer.rs: reduce verbosity 2020-02-24 11:33:00 +01:00
Dietmar Maurer 8ce49a76da src/backup/backup_info.rs: fix SNAPSHOT_PATH_REGEX and GROUP_PATH_REGEX 2020-02-18 13:16:35 +01:00
Wolfgang Bumiller 9e003074cb runtime: fix blocking strategy:
- do not "double"-block_in_place() (it may not be nested)
- do not call block_in_place() in non-worker threads

is_in_tokio() isn't sufficient, we need to actually know
that we're in a worker-thread, so we do this by remembering
that we're blocking.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-17 09:48:22 +01:00
Wolfgang Bumiller 1283d58ca9 runtime: remove IN_TOKIO thread local variable
tokio now has Handle::try_current() allowing us to
generally check for a tokio runtime even if spawned by
someone else

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-13 08:56:38 +01:00
Dietmar Maurer a6e3da987c src/client/http_client.rs: do not store fingerprints passed via options in cache 2020-02-11 11:54:43 +01:00
Dietmar Maurer 1bd6f32b43 src/client/http_client.rs: always convert fingerprint to lowercase 2020-02-11 11:42:03 +01:00
Dietmar Maurer d1c657276a src/client/http_client.rs: remove useless password_env 2020-02-11 11:42:03 +01:00
Dietmar Maurer a05c0c6ff6 src/bin/proxmox-backup-client.rs: allows to pass fingerprint via env vars 2020-02-11 11:42:03 +01:00
Thomas Lamprecht b69b8af26b s/pbsbanner/proxmox-backup-banner/ + move to /usr/lib
We want to avoid pbs if possible and also avoid placing internal
binaries, not intended for human direct use, in /bin or /sbin paths.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-10 14:43:28 +01:00
Thomas Lamprecht e2b5e75a3e pbsbanner: use match and also return error
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-07 18:30:55 +01:00
Thomas Lamprecht 274b0c7bb7 add login banner service
Modeled after the one from PVE, but using rust instead of perl for
resolving the nodename and writing to /etc/issue

Behavior differs a bit. We write all non-loopback addresses to this
file, as the gui accepts connections from them all, so limiting it to
the first one is not really sensible.
Further an error to resolve, or only getting loopback addresses won't
write out an empty /etc/issue file, but a note about the error at the
place where the address would be displayed.

Named it "pbsbanner", not "proxmox-backup-banner" as it's rather an
internal tool anyway and mirrors pvebanner, pmgbanner

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-06 22:19:22 +01:00
Christian Ebner ba050e3788 catalog: shell: Allow two or more successive slashes in path.
Two or more successive slashes should be allowed and treated as a single slash.
We also do not treat two successive slashes at the beginning of a path any
different.

Details are found here:
https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap04.html#tag_04_11

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-05 09:32:29 +01:00
Christian Ebner 9ff9a00572 pxar::fuse: cleanup: Remove unused code.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 13:43:11 +01:00
Christian Ebner 3e69abef02 pxar::fuse: Refactor lookup in order to cache accessed entries.
Context::find_goodbye_entry() is removed and incorporated into the lookup
callback in order to take advantage of the entry_cache and since it is only used
inside this callback.
All entries read on lookup are cached.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 13:43:06 +01:00
Christian Ebner 63698e720c pxar: Improve read performance for fuse.
By storing the payload start offset in the `DirectoryEntry` and passing this
information to `Decoder::read()`, the payload can be read directly and a repeated
re-reading of the entry information is avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 12:08:03 +01:00
Dietmar Maurer 032d3ad80f src/bin/proxmox-backup-client.rs: do not use api macro for prune
So that we can use add_common_prune_prameters!() macro again.
2020-01-31 10:31:36 +01:00
Dietmar Maurer b8a192e3b0 src/config/remote.rs: do not serialize empty option 2020-01-31 09:09:24 +01:00
Dietmar Maurer c43b36bcb6 src/section_config.rs - write: improve error message 2020-01-31 09:08:42 +01:00
Dietmar Maurer 501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Christian Ebner 74616f63b0 proxmox-backup-client: add missing prune options to api definition.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-31 07:01:56 +01:00
Dietmar Maurer 93fbb4ef0a update proxmox dependency to 0.1.12 2020-01-31 06:59:17 +01:00
Dietmar Maurer 43592f290d src/bin/proxmox-backup-manager.rs: remove stale debug comment 2020-01-30 17:59:13 +01:00
Dietmar Maurer 9e165b5cad src/bin/proxmox-backup-manager.rs - list remotes: do not use client, call directly 2020-01-30 17:57:37 +01:00
Dietmar Maurer 9894469e89 src/bin/proxmox-backup-manager.rs: use new texdt table formatter 2020-01-30 13:31:20 +01:00
Dietmar Maurer 8528fce874 src/api2/node/tasks.rs; make list_tasks public 2020-01-30 13:31:20 +01:00
Dietmar Maurer 70fa1aa3fc fixup 2020-01-30 13:31:20 +01:00
Dietmar Maurer 5eeea607ae src/section_config.rs - convert_to_array: add option to skip properties 2020-01-30 13:26:46 +01:00
Christian Ebner 93a9eb284e pxar::fuse: Do not return fcaps value on listxattr.
listxattr must only return the name list, no extended attribute values.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:09 +01:00
Christian Ebner 5d2158e89a pxar::fuse: return ACL names in listxattr if present
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:03 +01:00
Christian Ebner 72677fb0a5 pxar::fuse: add support to read ACLs.
ACLs are stored separately in the pxar archive. This implements the functionality
needed to read the ACLs and return them as extended attributes in the getxattr
callback.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:00 +01:00
Christian Ebner 0502ce6da3 tools::acl: Add helpers to create extended attribute buffer.
This helpers are used to construct the extended attributes values from
the ACLs stored in the pxar archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:34:57 +01:00
Christian Ebner bcf0d452c9 pxar: move acl helper functions to src/tools/acl.rs
They are not only needed by the pxar::sequential_decoder but also for the fuse
xattr impl, so it makes more sense to have them there.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:34:54 +01:00
Wolfgang Bumiller 6f763ae673 tools: LruCache: add dropcheck marker
See https://doc.rust-lang.org/nomicon/phantom-data.html

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:18:27 +01:00
Dietmar Maurer 99384f7933 src/api2/types.rs: define and use struct TaskListItem 2020-01-28 11:23:08 +01:00
Dietmar Maurer 2c4b303c62 src/api2/node/tasks.rs: use api macro for list_tasks 2020-01-28 11:23:08 +01:00
Christian Ebner 3e56c4abd7 pxar::fuse: Introduce LRU caching for goodbye table and directory entries.
The goodbye table of directory entries is cached in a LRU cache to speed up
subsequent accesses.
This is especially important for directories with many entries, as then the
readdirplus callback is called repeatedly because of the limited reply buffer
size.

`DirectoryEntry`s are cached for subsequent access in their own LRU cache,
independent of the goodbye tables.

In order to avoid borrow conflicts, the `Context` provides a fn as_mut_refs
as well as a fn run_with_context_refs.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-28 10:06:53 +01:00
Christian Ebner cc552754c8 pxar::fuse: refactor fn stat
By passing `&DirectoryEntry` to stat, the function interface is simplified.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-28 10:06:51 +01:00
Dietmar Maurer d08bc483db use proxmox 0.1.9 with new cli command helpers 2020-01-27 18:08:43 +01:00
Dietmar Maurer 99d863d770 src/client/http_client.rs: new password_env option 2020-01-27 10:42:37 +01:00
Dietmar Maurer 5030b7cea4 src/client/http_client.rs: new prefix options 2020-01-27 10:20:55 +01:00
Dietmar Maurer 5a74756c15 src/client/http_client.rs: implement fingerprint cache 2020-01-25 15:37:34 +01:00
Dietmar Maurer d59dbeca1b src/client/http_client.rs: new struct HttpClientOptions 2020-01-25 12:18:00 +01:00
Dietmar Maurer 6afbe1d846 src/config/remote.rs: add fingerprint 2020-01-25 09:49:45 +01:00
Dietmar Maurer dcb8db66d9 src/api2/types.rs: add CERT_FINGERPRINT_SHA256_SCHEMA 2020-01-25 09:49:45 +01:00
Christian Ebner 90c815bfe2 src/bin/proxmox-backup-client.rs: fix another typo in property description
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 13:46:08 +01:00
Christian Ebner 5eee6d89b6 src/bin/proxmox-backup-client.rs: fix typo in property description
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 13:45:22 +01:00
Christian Ebner 04350b4c08 tools: add access method to LruCache
This will return a mutable reference just like get_mut, but on a cache miss
it will get and insert the missing value via the fetch method provided via the
Cacher trait.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 09:32:09 +01:00
Dietmar Maurer 4191018c79 src/api2/admin/datastore.rs - download_file: use BACKUP_ARCHIVE_NAME_SCHEMA for file-name 2020-01-23 13:49:30 +01:00
Dietmar Maurer a92830dc39 src/api2/types.rs: define and use api type GarbageCollectionStatus 2020-01-23 13:40:12 +01:00
Dietmar Maurer dfc58d47ad src/api2/admin/datastore.rs: use api macro for start_garbage_collection 2020-01-23 13:01:21 +01:00
Dietmar Maurer 1dc117bbba src/api2/types.rs: define and use struct StorageStatus 2020-01-23 12:51:30 +01:00
Dietmar Maurer 09b1f7b202 src/api2/types.rs: define struct BackupContent, and use it with list_snapshot_files 2020-01-23 11:17:38 +01:00
Dietmar Maurer 51534c8de9 src/backup/manifest.rs: check if manifest contains files 2020-01-23 11:16:12 +01:00
Dietmar Maurer 68a6a0ee03 src/api2/admin/datastore.rs: use api macro for delete_snapshot 2020-01-23 10:16:45 +01:00
Dietmar Maurer 6abce6c2bb src/backup/datastore.rs: remove_backup_(group/dir) - return Error instead of io::Error 2020-01-23 10:14:46 +01:00
Dietmar Maurer c425bdc9b2 src/api2/pull.rs: delete fanished snapshots 2020-01-23 09:58:57 +01:00
Dietmar Maurer 8a1d68c8b9 src/backup/datastore.rs: improve error messages 2020-01-23 09:58:14 +01:00
Dietmar Maurer 18cc66ee85 src/api2/pull.rs: use new Datastore::last_successful_backup helper
Do not fail if local group does not exists.
2020-01-22 15:06:28 +01:00
Dietmar Maurer 41b373eced src/backup/datastore.rs: new helpers
last_successful_backup: Returns the time of the last successful backup
group_path: Returns the absolute path for a backup_group
snapshot_path: Returns the absolute path for a backup_dir
2020-01-22 15:05:47 +01:00
Thomas Lamprecht 1ca037d81c followup: commit all changes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-01-22 14:14:52 +01:00
Thomas Lamprecht 44b65b4ec7 api/datastore: do not allow updating path for now
It's a bit dangerous as it points to all the saved backups, so they
would be seemingly lost after updating the path.
Follow our logic from other products, e.g. in PVE we do not allow to
update the backing path/location of a storage either for similar
reasons.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-01-22 13:55:37 +01:00
Christian Ebner 1685c2e341 tools: LRU cache to use for pxar FUSE implementation.
Implements a cache with least recently used cache replacement policy.
Internally the state is tracked by a HashMap (for fast access) and a doubly
linked list (for the access order).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-22 13:11:42 +01:00
Dietmar Maurer f1d99e3f6a src/bin/proxmox-backup-client.rs: use a std channel to write the catalog 2020-01-22 12:49:08 +01:00
Dietmar Maurer 02141b4d9b src/client/pxar_backup_stream.rs: use a channel instead of a pipe 2020-01-22 11:37:16 +01:00
Dietmar Maurer dcd033a53c src/tools/std_channel_writer.rs: new StdChannelWriter helper class 2020-01-22 11:33:57 +01:00
Wolfgang Bumiller 9ea4bce444 bump proxmox crate to 0.1.7
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).

When we need to depend on "parts" of the crate later on
we'll just have to use features.

The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 13:48:37 +01:00
Dietmar Maurer 3f06d6fbbd use proxmox_backup::tools::runtime::main to run async run_cli_command 2020-01-21 11:34:45 +01:00
Wolfgang Bumiller 077ac6f8b3 runtime: actually park on Pending...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 10:37:16 +01:00
Wolfgang Bumiller 2d51569396 runtime: custom block_on implementation
futures::executor::block_on does not allow nesting, but we
need that...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 10:24:52 +01:00
Wolfgang Bumiller 650e052299 runtime: drop now not required Send and static restrictions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 14:09:24 +01:00
Wolfgang Bumiller d973aa827c introduce new runtime tokio helpers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 13:12:40 +01:00
Christian Ebner aac9dbf635 pxar::fuse: remove open, opendir and releasedir callbacks
They are not needed as there is no internal state which needs to be tracked.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-20 12:56:21 +01:00
Christian Ebner a8aff3535d pxar: Include symlink target in DirectoryEntry
This allows to read the target path of a symbolic link in the
Decoder::read_directory_entry() function and stores it in the DirectoryEntry.
By this the Decoder::read_link() function becomes obsolete and is therefore
removed.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-20 12:56:10 +01:00
Dietmar Maurer 138910bcd4 src/client/pxar_backup_stream.rs: use std:🧵:Builder to name the thread
To make debugging easier.
2020-01-18 12:21:22 +01:00
Dietmar Maurer 4b4eba0b9e src/api2/pull.rs: implement delete flag for vanished groups 2020-01-17 11:24:55 +01:00
Dietmar Maurer 11d89239c3 src/backup/backup_info.rs: new list_groups helper 2020-01-17 10:42:03 +01:00
Dietmar Maurer b31c8019d7 src/api2/types.rs: define and use GroupListItem 2020-01-17 10:17:18 +01:00
Christian Ebner 2bbbade367 pxar: Refactor fuse and remove unused code.
By ambiguously using the Decoder::read_directory_entry() the code is simplified
and reading of the DirectoryEntry is concentrated into Context::run_in_context().

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
Christian Ebner 95c9460c4a pxar::fuse: Always use start offset of entries as inode.
Previously it was disciminated based on the entry mode.
For directories, the inode was the offset of the corresponding
goodbye tail mark while for all others it was the offset of the filename.

By simply using the start offset as calculated from the corresponding
goodbye table entry (which yields the archive offset of the filename),
the code is simplified and the more ambiguous read_directory_entry()
function can be used.
The disatvantage of this approach is the need to keep track of the
start and end offsets for each entry, as the end offset is needed in
order to access the goodbye table of directory entries.
The root node still has to be treated special, as it's inode is 1 as per fuse
definition and it has no filename as per the pxar file format definition.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
Christian Ebner 6744440714 pxar::decoder: refactor Decoder::lookup()
Use Decoder::read_directory_entry() instead of Decoder::attributes() as this
already returns the needed DirectoryEntry.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
Dietmar Maurer f357390c15 renamed: src/config/remotes.rs -> src/config/remote.rs
And use 'remote' instead of 'remotes' everywhere.
2020-01-16 14:32:06 +01:00
Dietmar Maurer d2dd827877 src/backup/catalog.rs - SenderWriter: use tokio::task::block_in_place
Make sure we do not block the executor.
2020-01-16 14:24:15 +01:00
Dietmar Maurer 94609e2380 src/api2/pull.rs: use reference to remote configuration 2020-01-16 13:55:19 +01:00
Dietmar Maurer 625d19621f src/client/remote_chunk_reader.rs: use tokio::task::block_in_place to avoid blocking
Else we block the current executor.
2020-01-16 13:55:19 +01:00
Wolfgang Bumiller 6a0dc4a5bc bump proxmox crate version
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 11:40:23 +01:00
Wolfgang Bumiller 619495b25e fix use of deprecated items
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 08:58:54 +01:00
Wolfgang Bumiller 18c0df4c92 import ordering cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 08:51:52 +01:00
Christian Ebner ef2d7f2f96 pxar::fuse: remove readdir implementation
By not implementing readdir but only readdirplus, the FUSE_CAP_READDIRPLUS flag
is set while the FUSE_CAP_READDIRPLUS_AUTO flag is not set.
Thereby the kernel will issue only readdirplus calls.
Documentation at:
https://libfuse.github.io/doxygen/fuse-3_88_80_2include_2fuse__common_8h.html#a9b90333ad08d0e1c2ed0134d9305ee87

As the expensive part for accessing and reading the attributes is seeking and
decoding each directory entry, it is usefull to force readdirplus calls.
By this a struct `EntryParam` is returned for each entry, therebye avoiding a
subsequent lookup call.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-15 14:44:53 +01:00
Dietmar Maurer 3be839c61c src/bin/proxmox-backup-manager.rs: add completion function for update commands 2020-01-15 12:42:13 +01:00
Dietmar Maurer 002a191abf src/api2/config: check digest 2020-01-15 12:27:05 +01:00
Dietmar Maurer 347834df25 src/api2/config: correctly lock files 2020-01-15 11:57:12 +01:00
Oguz Bektas 1c79e1017f services: fix indentation
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-15 10:55:04 +01:00
Oguz Bektas cbe01dc5c7 tools: add read_and_verify_password helper
add a helper to perform some basic checks on password prompts.
- verification (asks for a 2nd time)
- check length

also use the new helper where password input in tty is taken to reduce
duplicate code.

this helper should be used when creating keys, changing passphrases etc.

note: this helper can be extended later on to provide better checks for
password strength.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-15 10:52:57 +01:00
Christian Ebner ea5061979b pxar::fuse: Avoid macro to generate filler function, use closure instead.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-15 10:25:16 +01:00
Dietmar Maurer f3ec5dae7f src/api2/config: improve api schema by adding return type 2020-01-14 14:55:58 +01:00
Dietmar Maurer ddc526623f src/bin/proxmox-backup-manager.rs: add datastore update cli 2020-01-14 14:47:26 +01:00
Dietmar Maurer c5799e4043 src/api2/config/datastore.rs: improve api, implement update and read 2020-01-14 14:45:56 +01:00
Dietmar Maurer 08195ac823 src/api2/config/remotes.rs: improve api, implement update and read 2020-01-14 14:20:16 +01:00
Dietmar Maurer 16f04b9d79 src/section_config.rs: add lookup_json() helper 2020-01-14 14:19:22 +01:00
Dietmar Maurer 71805bb7ec src/api2/config/remotes.rs: fix variable name -> s/remote/id/ 2020-01-14 13:20:13 +01:00
Dietmar Maurer d0187a51a9 src/section_config.rs - convert_to_array: optionally add digest
datastore::config() -> also return digest
remotes::config() -> also return digest
2020-01-14 12:57:03 +01:00
Dietmar Maurer 4566303b05 src/section_config.rs - write: improve error message 2020-01-14 12:02:46 +01:00