Commit Graph

3952 Commits

Author SHA1 Message Date
Dietmar Maurer 651a61f559 pmtx: implement scan command 2021-01-27 12:40:51 +01:00
Dietmar Maurer 89ccb125d1 tape: use 36 byte Inquiry (recommended size) 2021-01-27 12:35:28 +01:00
Dietmar Maurer 887f1cb90c cleanup: move scan changers API implementation 2021-01-27 09:58:16 +01:00
Dietmar Maurer 16b4d78400 tape: rename retry_command to execute_scsi_command, make retry a flag 2021-01-27 09:34:24 +01:00
Dietmar Maurer ec8d9c6b80 tape: repeat changer scsi command until successful 2021-01-27 08:59:10 +01:00
Dietmar Maurer 49c2d1dcad sgutils2: use sg_get_asc_ascq_str to produce error messages 2021-01-27 06:56:11 +01:00
Dietmar Maurer d0f51651f9 sgutils2: add ASC codes from tandeberg docs 2021-01-26 18:54:08 +01:00
Dietmar Maurer 481ccf16a5 sgutils2: further improve error messages 2021-01-26 15:19:43 +01:00
Dietmar Maurer a223458753 sgutils2: support RequestSense Descriptor format 2021-01-26 13:38:16 +01:00
Dominik Csapak e1740f3f01 tape/changer/mtx: add mtx parser test
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 12:51:26 +01:00
Dominik Csapak 740dc9d1d4 api2/tape/changer: reorganize api
add a changer listing here (copied from api2/config/changer)
and put the status and transfer api calls below that

puts the changer scan into the top level tape api
and removes the (now redundant) info from the config api path

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 12:47:34 +01:00
Dominik Csapak bbf01b644c tape: fix typos
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 12:39:54 +01:00
Dietmar Maurer c81c46c336 sgutils2: improve error messages 2021-01-26 12:24:58 +01:00
Dietmar Maurer c3747b93c8 tape: add new command line tool "pmtx"
Also improve sgutil2 error reporting
2021-01-26 11:57:15 +01:00
Dominik Csapak 6864fd0149 server/worker_task: improve newline handling in upid_read_status
improves upid_read_status with:
* ignore multiple newlines at the end
* remove all code that could panic (array index access)
  the one place where we access with '[pos+1..]' is ok since
  we explicitely test the len of the vector, this is done to
  let rust optimize away the range checks, so it cannot panic

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-01-26 10:48:15 +01:00
Wolfgang Bumiller 340c0bf9e3 pxar: don't clone patterns unnecessarily
The options struct has no Drop handler and is passed by-move
so we can partially move out of it.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-26 10:24:18 +01:00
Fabian Grünbichler 4d104cd4d8 clippy: more misc fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:55 +01:00
Fabian Grünbichler 367c0ff7c6 clippy: allow api functions with many arguments
some of those can be reduced/cleaned up when we have updater support in
the api macro.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:52 +01:00
Fabian Grünbichler 9c26a3d61a verify: factor out common parameters
all the verify methods pass along the following:
- task worker
- datastore
- corrupt and verified chunks

might as well pull that out into a common type, with the added bonus of
now having a single point for construction instead of copying the
default capacaties in three different modules..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:49 +01:00
Fabian Grünbichler 93e3581ce7 derive/impl and use Default for some structs
and revamp HttpClientOptions with two constructors for the common use
cases

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:45 +01:00
Fabian Grünbichler f4e52bb27d authid: make Tokenname(Ref) derive Eq
it's needed to derive Hash, and we always compare Authids or their
Userid components, never just the Tokenname part anyway..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:40 +01:00
Fabian Grünbichler 72064fd0df pxar: extract PxarExtractOptions
same as PxarCreateOptions, but for extraction/restore rather than
create.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:36 +01:00
Fabian Grünbichler 77486a608e pxar: factor out PxarCreateOptions
containing the CLI parameters that are mostly passed-through from the
client to our pxar archive creation wrapper in pxar::create

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:32 +01:00
Fabian Grünbichler e97025ab02 pxar: typedef on_error as ErrorHandler
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:26 +01:00
Fabian Grünbichler e43b9175c0 client: factor out UploadOptions
to reduce function signature complexity.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:18 +01:00
Fabian Grünbichler 9cc1415ef5 systemd/time: extract Time/DateSpec structs
could be pulled up into CalendarEvent if desired..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:13 +01:00
Fabian Grünbichler bd215dc0e4 async index reader: typedef ReadFuture
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:58 +01:00
Fabian Grünbichler 12e874cef0 allow complex Futures in tower_service impl
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:55 +01:00
Fabian Grünbichler 6d233161b0 client: refactor catalog upload spawning
by pulling out Result type into separate struct

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:51 +01:00
Fabian Grünbichler 905a570489 broadcast_future: refactor broadcast/future binding
into its own, private struct.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:48 +01:00
Fabian Grünbichler 432fe44187 report: type-alias function call tuple
to make clippy happy.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:53:43 +01:00
Wolfgang Bumiller 51b938496d tools::sgutils2: name fixup
it's not a box anymore

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 15:05:52 +01:00
Wolfgang Bumiller b7f9b25e4d tools::sgutils2: use NonNull
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:56:10 +01:00
Wolfgang Bumiller fe61280b6b tools::sgutils2: extern 'C' and import ordering
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:54:25 +01:00
Wolfgang Bumiller 68c087d578 tools::sgutils2: don't transmute to a Box
Otherwise we run the drop handler for the scsi pt object AND
the box itself, which shouldn't even work as it should be
doing a double-free (unless the library does some kind of
reference counting in which case this should simply crash
later on?)

anyway, let's make a wrapper simply called `SgPt` containing
the pointer from `construct_scsi_pt_obj()`

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:48:27 +01:00
Wolfgang Bumiller d6bf87cab7 tools::sgutils2: const correctness
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 14:33:45 +01:00
Dietmar Maurer 2b96a43879 tape: cleanup - use ScsiMediaChange trait instead of mtx_status() 2021-01-25 13:25:22 +01:00
Dietmar Maurer 697c41c584 tape: add/use rust scsi changer implementation using libsgutil2 2021-01-25 13:14:07 +01:00
Dietmar Maurer a2379996e6 sgutils2: add scsi_inquiry command 2021-01-25 13:14:07 +01:00
Fabian Grünbichler 29077d95db http-client: further clippy cleanups
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:54 +01:00
Fabian Grünbichler dbd00a57b0 http-client: fix typoed ticket cache condition
which was even copy-pasted once without noticing.

found with clippy.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:51 +01:00
Fabian Grünbichler d08cff51a4 rework GC traversal error handling
the error message don't make sense with an empty default

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:48 +01:00
Fabian Grünbichler 3e461dec1c apt: let api handle optional bool with default
one less FIXME :)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:46 +01:00
Fabian Grünbichler 4d08e25913 clippy: rewrite ifs with identical return values
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:43 +01:00
Fabian Grünbichler 43313c2ee7 clippy: rewrite comparison chains
chunk_stream one can be collapsed, since split == split_to with at set
to buffer.len() anyway.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:39 +01:00
Fabian Grünbichler 81b2a87232 clippy: fix Mutex with unused value
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:36 +01:00
Fabian Grünbichler 3d8cd0ced7 clippy: add is_empty() when len() is implemented
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:32 +01:00
Dietmar Maurer 7c78d54231 sgutils: allow command which does not transfer any data 2021-01-24 15:19:43 +01:00
Dietmar Maurer f9d71e8b17 sgutils2: allow to set custom timeouts 2021-01-24 14:54:30 +01:00
Dietmar Maurer 0107fd323c cleanup: avoid compiler warnings 2021-01-23 17:34:26 +01:00
Dietmar Maurer 979dccc7ec tape: avoid error when clearing encryption key
Simply ignore clear request when sg_spin_data_encryption_caps fails.
Assume those are tapes without hardware encryption support.
2021-01-23 10:20:43 +01:00
Dietmar Maurer bf78f70885 improve code docs in api2
Note: API methos should be declared pub, so that they show up in the generated docu.
2021-01-22 15:57:42 +01:00
Dietmar Maurer 0d916ac531 tape: add media pool config code docs 2021-01-22 12:01:46 +01:00
Dietmar Maurer d4ab407045 tape: add drive config code docs 2021-01-22 11:51:36 +01:00
Dietmar Maurer 45212a8c78 fix mtx parser bug: s/strip_suffix/strip_prefix/ 2021-01-22 11:00:56 +01:00
Dietmar Maurer 64b83c3d70 tape: implement paperkey command for tape encryption keys 2021-01-22 09:56:14 +01:00
Dietmar Maurer 639a6782bd paperkey: move code to src/tools/paperkey.rs 2021-01-22 09:42:59 +01:00
Dietmar Maurer 5f34d69bcc tape: add volume-statistics api/command 2021-01-22 08:45:35 +01:00
Dietmar Maurer 337ff5a3cc tape: add estimated medium wearout to status 2021-01-22 08:06:25 +01:00
Dietmar Maurer 8e6459a818 tape: set encryption key on restore 2021-01-22 07:26:42 +01:00
Dietmar Maurer aff3e16194 tape: add code docs to src/config/tape_encryption_keys.rs 2021-01-21 18:23:07 +01:00
Dietmar Maurer 9372c0787d renamed src/tape/sgutils2.rs -> src/tools/sgutils2.rs 2021-01-21 17:57:17 +01:00
Dietmar Maurer 83fb2da53e tape: move MediaCatalog magic number into struct (doc cleanup) 2021-01-21 17:48:07 +01:00
Dietmar Maurer 645a044bf6 tape: further hierarchy improvements 2021-01-21 17:25:32 +01:00
Dietmar Maurer 37796ff73f tape: change code hierarchy to improve docs 2021-01-21 17:12:01 +01:00
Dietmar Maurer e1fdcb1678 tape: do not export/doc low level libsgutils2 bindings 2021-01-21 16:38:24 +01:00
Dietmar Maurer edda5039d4 tape: improve code docs 2021-01-21 13:19:07 +01:00
Dietmar Maurer 1c86893d95 cleanup: always compute fingerprint in KeyConfig constructors 2021-01-21 11:56:54 +01:00
Dietmar Maurer d543587d34 Merge branch 'master' of ssh://proxdev.maurer-it.com/rust/proxmox-backup 2021-01-21 10:56:52 +01:00
Dietmar Maurer 780bc4cad2 tape: try to set encryption key with read-label command 2021-01-21 10:31:49 +01:00
Dietmar Maurer 18bd6ba13d tape: restore_key - always update key, even if there is already an entry 2021-01-21 10:31:49 +01:00
Dietmar Maurer 4dafc513cc tape: fix file permissions for tape encryptiuon keys 2021-01-21 10:31:49 +01:00
Dietmar Maurer 7acd5c5659 cleanup: remove missleading wording from code docs 2021-01-21 10:31:49 +01:00
Dietmar Maurer 8428063d9e cleanup: KeyConfig::decrypt - show password hint on error 2021-01-21 10:31:49 +01:00
Dietmar Maurer f490dda05a tape: use type Uuid instead of String 2021-01-21 10:31:49 +01:00
Dietmar Maurer 2b191385ea tape: use specialized encryption key per media-set 2021-01-21 10:31:49 +01:00
Dietmar Maurer bc228e5eaf api: add types for UUIDs 2021-01-20 17:16:46 +01:00
Fabian Grünbichler 8be65e34de clippy: replace transmute with &*
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:41:02 +01:00
Fabian Grünbichler d967d8f1a7 clippy: remove drop(&..)
it does nothing.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:41:02 +01:00
Fabian Grünbichler 50deb0d3f8 clippy: use is_null to check for null pointers
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:41:02 +01:00
Fabian Grünbichler 1d928b25fe clippy: remove some unnecessary reference taking
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler f2f81791d1 clippy: fix for_kv_map
and allow it in the one case where the entry loop is intended, but the
code is not yet implemented fully.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 382f10a0cc clippy: fix/allow needless_range_loop
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 0d2133db98 clippy: use while let loops
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 09faa9ee95 clippy: pass &str/&[..] instead of &String/&Vec
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler ccec086e25 clippy: remove unnecessary &mut
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 05725ac9a4 clippy: remove unnecessary `let` binding
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 96b7483138 clippy: remove/replace needless explicit lifetimes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 81281d04a4 clippy: fix/allow identity_op
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler e062ebbc29 clippy: us *_or_else with function calls
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler b92cad0938 clippy: convert single match to if let
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler ea368a06cd clippy: misc. fixes
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 3f48cdb380 clippy: don't pass along unit value
make it explicit. this whole section should probably be re-written with
select!

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 17c7b46a69 clippy: use unwrap_or_default
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler a375df6f4c clippy: use copied/cloned instead of map
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler a3775bb4e8 clippy: shorten assignments
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 1e0c6194b5 clippy: fix option_as_ref_deref
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler a6bd669854 clippy: use matches!
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 6334bdc1c5 clippy: collapse nested ifs
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 3b82f3eea5 clippy: avoid useless format!
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 38556bf60d clippy: remove explicit returns
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler d8d8af9826 clippy: use chars / byte string literals
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 3984a5fd77 clippy: is_some/none/ok/err/empty
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:54 +01:00
Fabian Grünbichler 397356096a clippy: remove needless bool literals
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:23:52 +01:00
Fabian Grünbichler 365915da9a clippy: use strip_prefix instead of manual stripping
it's less error-prone (off-by-one!)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 87152fbac6 clippy: drop redundant 'static lifetime
those declarations are already const/static..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 22a9189ee0 clippy: remove unnecessary closures
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 4428818412 clippy: remove unnecessary clones
and from::<T>(T)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Fabian Grünbichler 47ea98e0e3 clippy: collapse/rework nested ifs
no semantic changes (intended).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-20 16:22:59 +01:00
Dietmar Maurer 6dd0513546 tape: allocate new media set when pool encryption key changes 2021-01-20 15:43:39 +01:00
Dietmar Maurer 8abe51b71d improve code docs 2021-01-20 15:43:19 +01:00
Dietmar Maurer 69b8bc3bfa tape: implemenmt show key
Moved API types Kdf and KeyInfo to src/api2/types/mod.rs.
2021-01-20 15:43:19 +01:00
Dietmar Maurer 301b8aa0a5 tape: implement change-passphrase for tape encryption keys 2021-01-20 15:43:19 +01:00
Dietmar Maurer e5b6c93323 tape: add --kdf parameter to create key api 2021-01-20 15:43:19 +01:00
Dietmar Maurer 9a045790ed cleanup KeyConfig 2021-01-20 15:43:19 +01:00
Dietmar Maurer 82a103c8f9 add "password hint" to KeyConfig 2021-01-20 15:43:19 +01:00
Dietmar Maurer feb1645f37 tape: generate random encryptions keys and store key_config on media 2021-01-19 11:20:07 +01:00
Dietmar Maurer 8ca37d6a65 cleanup: factor out decrypt_key_config 2021-01-19 11:20:07 +01:00
Wolfgang Bumiller 9b6bddb24c tfa: remove/empty description for recovery keys
While the user chosen description is not allowed to be
empty, we do leave it empty for recovery keys, as a "dummy
description" makes little sense...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-18 15:20:39 +01:00
Wolfgang Bumiller ad5cee1d22 tfa: add 'created' timestamp to entries
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-18 14:06:12 +01:00
Wolfgang Bumiller ca1060862e tfa: remember recovery indices
and tell the client which keys are still available rather
than just yes/no/low

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-18 13:51:23 +01:00
Dietmar Maurer 8a0046f519 tape: implement encrypted backup - simple version
This is just a proof of concept, only storing the encryption key fingerprint
inside the media-set label.
2021-01-18 13:38:22 +01:00
Dietmar Maurer 84cbdb35c4 implement FromStr for Fingerprint 2021-01-18 13:38:22 +01:00
Dietmar Maurer 1e93fbb5c1 tape: add encrypt property to media pool configuration 2021-01-18 13:38:22 +01:00
Dietmar Maurer 619554af2b tape: clear encryption key before writing labels
We always write labels unencrypted.
2021-01-18 13:38:22 +01:00
Dietmar Maurer d5a48b5ce4 tape: add hardware encryption key managenent api 2021-01-18 13:38:22 +01:00
Fabian Grünbichler 44de5bcc00 pull: add error context for initial group list call
otherwise the user is confronted with a generic error like "permission
check failed" with no indication that it refers to a request made to the
remote PBS instance..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-18 06:51:05 +01:00
Fabian Grünbichler e2956c605d pull: rustfmt
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-18 06:50:23 +01:00
Dietmar Maurer b22b6c2299 tape: encryption scsi command cleanup 2021-01-16 18:24:04 +01:00
Dietmar Maurer 90950c9c20 tape: add scsi commands to control drive hardware encryption 2021-01-16 15:59:05 +01:00
Dietmar Maurer 0c5b9e7820 tape: sgutils2.rs - add do_out_command()
Make it possible to run commands that writes data.
2021-01-16 15:59:05 +01:00
Wolfgang Bumiller a670b99db1 tfa: add webauthn configuration API entry points
Currently there's not yet a node config and the WA config is
somewhat "tightly coupled" to the user entries in that
changing it can lock them all out, so for now I opted for
fewer reorganization and just use a digest of the
canonicalized config here, and keep it all in the tfa.json
file.

Experimentally using the flatten feature on the methods with
an`Updater` struct similar to what the api macro is supposed
to be able to derive on its own in the future.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-15 15:19:52 +01:00
Wolfgang Bumiller aefd74197a bakckup::manifest: use tools::json for canonical representation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-15 15:19:52 +01:00
Wolfgang Bumiller 9ff747ef50 add tools::json for canonical json generation
moving this from backup::manifest, no functional changes

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-15 15:19:52 +01:00
Dietmar Maurer a08a198577 tape: do not abort backup if tape drive does not support tape-alert-flags 2021-01-15 11:43:17 +01:00
Oguz Bektas 6bbe49aa14 access: restrict password changes on @pam realm to superuser
for behavior consistency with `update_user`

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2021-01-15 08:49:22 +01:00
Oguz Bektas 5aa1019010 access: limit editing pam credentials to superuser
modifying @pam users credentials should be only possible for root@pam,
otherwise it can have unintended consequences.

also enforce the same limit on user creation (except self_service check,
since it makes no sense during user creation)

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2021-01-15 08:49:22 +01:00
Fabian Grünbichler 29a59b380c proxmox 0.10: adapt to moved ParameterSchema
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 0bfcea6a11 cleanup: remove unnecessary 'mut' and '.clone()'
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 89e9134a3f hyper: use new hyper::upgrade
the old Body::on_upgrade method is no more

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler b5a202acb6 tokio 1.0: update to new Signal interface
Signal does not yet re-implement Stream (and is not yet wrapped in
tokio-stream either).

see https://github.com/tokio-rs/tokio/pull/3383

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 0f860f712f tokio 1.0: update to new tokio-openssl interface
connect/accept are now happening on pinned SslStreams

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 7c66701366 tokio 1.0: use ReceiverStream from tokio-stream
to wrap a Receiver in a Stream. this will likely move back into tokio
proper once we have a std Stream..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 585e90c0de tokio: adapt to 1.0 process:Child changes
Child itself is no longer a Future, but it has a new wait() async fn
that does the same thing

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 5c852d5b82 tokio: adapt to 1.0 runtime changes
enter() now returns a guard, and the builder got revamped to make the
choice between MT and current thread explicit.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 484172b5f8 tokio 1.0: AsyncRead/Seek with ReadBuf
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler d148958b67 proxmox 0.10: use tokio::time::timeout directly
TimeoutFutureExt is no more

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 0a8d773ad0 tokio 1.0: delay -> sleep
almost the same thing, new name(s), no longer Unpin

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler 427d90e6c1 update to tokio 1.0
and various related crates

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Thomas Lamprecht 1a0b410554 manager: user/token list: fix rendering 0 (never) expire date
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-01-14 13:59:08 +01:00
Dietmar Maurer 2d50a6192f tape: sg-tape-cmd - add more ways to specify devices 2021-01-14 13:05:26 +01:00
Dietmar Maurer 781da7f6f0 tape: add --inventorize flag to read-label API/CLI 2021-01-14 11:51:23 +01:00
Dietmar Maurer 25e464c5ce tape: MediaPool - allow to allocate free tapes 2021-01-13 14:25:51 +01:00
Dietmar Maurer 8446fbca85 tape: rename changer_id to label_text 2021-01-13 13:26:59 +01:00
Dietmar Maurer 0bce2118e7 tape: improve docu 2021-01-12 16:37:23 +01:00
Dietmar Maurer 6543214dde tape: MediaListEntry - add ctime 2021-01-12 12:01:21 +01:00
Dietmar Maurer a484c9cf96 tape: automatically reload tapes inside autoloader
We always automatically unload tapes to free library slots,
so it should not happen that an ejected tape resides inside the drive.

This is just a safe guard to handle the situation in case it happens ...

You can manually produce the situation by ejecting a tape without unloading:

 mt -f /dev/nst0 eject

Note: Our "proxmox-tape eject" does automatic unload
2021-01-12 09:49:05 +01:00
Dietmar Maurer 5654d8ceba tape: make eject/export more reliable, improve logging 2021-01-12 09:16:16 +01:00
Dietmar Maurer 31cf625af5 tape: improve backup logs 2021-01-11 13:23:12 +01:00
Dietmar Maurer 93be18ffd2 tape: fix tape alert flag values 2021-01-11 13:23:12 +01:00
Wolfgang Bumiller ad0ed40a59 api: return "invalid" as CSRF token for partial tickets
So that old clients don't `unwrap` a `None` value.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:13 +01:00
Wolfgang Bumiller 7ad33e8052 tfa: use UNAUTHORIZED http status in password check
to trigger our 3s delay in the rest handler

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:13 +01:00
Wolfgang Bumiller abfe0c0e70 tfa: fixup for challenge file split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:13 +01:00
Wolfgang Bumiller f22dfb5ece tfa: remove tfa user when a user is deleted
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:10 +01:00
Wolfgang Bumiller 4bda51688b tfa: improve user existence check
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller eab25e2f33 tfa: allow deletion of entries of non-existent users
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 94bd11bae2 typo fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 759af9f00c tfa api: return types and 'pub' structs/methods
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller f58e5132aa tfa: entry access/iteration cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller d831846706 tfa: r#type parameter name
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 1fc9ac0433 tfa: _entry api method name suffix consistency
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:23:03 +01:00
Wolfgang Bumiller 7f066a9b21 proxy: expose qrcodejs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 027ef213aa api: tfa management and login
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller dc1fdd6267 config: add tfa configuration
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 96918252e5 buildcfg: add rundir helper macro
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 014dc5f9d7 tools: add create_run_dir helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Wolfgang Bumiller 59e94227af add tools::serde_filter submodule
can be used to perform filtering at parse time

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-11 10:22:32 +01:00
Dietmar Maurer 6638c034d2 tape: remove unused eject_on_unload method 2021-01-10 16:20:18 +01:00
Dietmar Maurer 04df41cec1 tape: more MediaChange cleanups
Try to provide generic implementation for complex operations:

- unload_to_free_slot
- load_media
- export media
- clean drive
- online_media_changer_ids
2021-01-10 15:32:52 +01:00
Dietmar Maurer 483da89d03 tape: improve export media to directly export from drive, add CLI 2021-01-10 13:44:44 +01:00
Dietmar Maurer c92e3832bf tape: cleanup: s/transfer/transfer_media/, avoid compiler warnings 2021-01-10 12:18:30 +01:00
Dietmar Maurer edb90f6afa tape: backup - implement export-media-set option 2021-01-10 11:59:55 +01:00
Dietmar Maurer 0057f0e580 tape: MediaChange - add transfer, implement export 2021-01-10 11:51:09 +01:00
Dietmar Maurer e6217b8b36 tape: renamed src/tape/changer/linux_tape.rs -> src/tape/changer/mtx.rs 2021-01-10 10:07:40 +01:00
Dietmar Maurer 6fe16039b9 tape: simplify media changer implementation - new struct MtxMediaChanger 2021-01-10 10:02:01 +01:00
Dietmar Maurer 42967bf185 tape: backup - implement --eject-media option 2021-01-09 15:17:03 +01:00
Dietmar Maurer 5843268c47 tape: abort backup when we detect critical tape alert flags 2021-01-09 12:34:00 +01:00
Dietmar Maurer 7273ba3de2 tape: change default media set naming template to "%c" 2021-01-09 10:51:51 +01:00
Dietmar Maurer 0bf1c314da tape: show catalog status in media list 2021-01-09 10:24:48 +01:00
Dietmar Maurer c7926d8e8c tape: split MediaSet into extra file 2021-01-09 08:54:58 +01:00
Dietmar Maurer df69a4fc59 tape: implement drive clean 2021-01-08 11:32:56 +01:00
Dietmar Maurer 25d3965769 tape: correctly skip cleaning tapes (not regular tapes) 2021-01-08 09:16:42 +01:00
Dietmar Maurer 879569d73f tape: changer transfer - make name parameter optional 2021-01-07 17:09:47 +01:00
Dietmar Maurer b63f833d36 tape: fix paramater name - s/slot/source-slot/ 2021-01-07 15:39:25 +01:00
Dietmar Maurer 482c6e33dd tape: changer status command: make changer name optional 2021-01-07 15:12:19 +01:00
Dietmar Maurer 46a1863f88 tape: improve MediaChange trait
We expose the whole MtxStatus, and we can load/store from/to
specified slot numbers.
2021-01-07 14:26:43 +01:00
Dietmar Maurer d5035c5600 tape: mtx_status - consider new export-slots property 2021-01-06 11:53:33 +01:00
Dietmar Maurer 38ae42b11a tape: changer - add export-slot config 2021-01-06 11:06:50 +01:00
Dietmar Maurer c4b2b9ab41 tape: only query volume stats if we can read MAM 2021-01-06 09:20:36 +01:00
Dietmar Maurer ef942e04c2 tape: add function to classify tape-alert-flags 2021-01-05 17:23:30 +01:00
Dietmar Maurer b40ab10d38 tape: add volume_mounts and medium_passes to LinuxDriveAndMediaStatus 2021-01-05 13:43:17 +01:00
Dietmar Maurer f8ccbfdedd tape: implement read_volume_statistics 2021-01-05 12:58:18 +01:00
Dietmar Maurer 470f1c798a tape: status - show thape alert flags 2021-01-04 13:15:30 +01:00
Dietmar Maurer 5c012b392a tape: use LP 12h TapeAlert Response to query tape alert flags 2021-01-04 13:14:02 +01:00
Dietmar Maurer 165b641c1d tape: changer status - show full slots (for cartridge without barcode) 2021-01-04 12:06:05 +01:00
Dietmar Maurer 66e42bec05 tape: further PoolWriter cleanups 2021-01-03 12:08:40 +01:00
Dietmar Maurer c503ea7045 tape: cleanup - rename 'info' to 'media_id'
Second try.
2021-01-03 11:38:00 +01:00
Dietmar Maurer 745ec187ce Revert "tape: cleanup - rename 'info' to 'media_id'"
This reverts commit f046313c0e.

media_id is already use as parameter, so this commit is totally buggy.
2021-01-03 11:14:58 +01:00
Dietmar Maurer f046313c0e tape: cleanup - rename 'info' to 'media_id' 2021-01-03 10:37:42 +01:00
Dietmar Maurer 74595b8821 tape: sg-tape-cmd tape-alert-flags 2021-01-03 10:09:43 +01:00
Dietmar Maurer c9fdd142a4 tape: commit missing file 2021-01-02 13:39:34 +01:00
Dietmar Maurer abaa6d0ac9 tape: decode TapeAlertFlags in cartridge-memory command 2021-01-02 10:55:30 +01:00
Dietmar Maurer cfae8f0656 tape: merge MediaStateDatabase into Inventory 2021-01-01 16:15:13 +01:00
Dietmar Maurer 54f4ecd46a tape: implement MediaPool flag to consider offline media
For standalone tape drives.
2021-01-01 10:03:59 +01:00
Dietmar Maurer b9b4b31284 tape: add basic restore api/command 2020-12-31 10:26:48 +01:00
Dietmar Maurer b4772d1c43 tape: new inventory helper - lookup_media_set_pool 2020-12-31 10:03:17 +01:00
Fabian Grünbichler 08ac90f920 api: allow tokens to list users
their owner, or all if they have the appropriate privileges.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-31 08:29:49 +01:00
Fabian Grünbichler 13f5863561 api: improve error messages for restricted endpoints
the old variant attempted to parse a tokenid as userid and returned the
cryptic parsing error to the client, which is rather confusing.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-31 08:29:09 +01:00
Dietmar Maurer 81764111fe tape: media_change - log all errors 2020-12-30 19:17:18 +01:00
Dietmar Maurer cb022525ff tape: only log to stdout in CLI environment 2020-12-30 19:01:39 +01:00
Dietmar Maurer 75656a78c6 tape: improve inline docu 2020-12-30 17:28:33 +01:00
Dietmar Maurer 284eb5daff tape: cleanup/simplify media_change code 2020-12-30 17:16:57 +01:00
Dietmar Maurer ff58c51919 tape: improve media request/load 2020-12-30 13:09:28 +01:00
Dietmar Maurer 12299b333b tape: set minimal media label length to 2 2020-12-30 10:15:02 +01:00
Dietmar Maurer b017bbc441 tape: add restore code, implement catalog api/command 2020-12-30 09:48:18 +01:00
Dietmar Maurer 9e8c0d2e33 tape: cleanup - remove debug messages 2020-12-30 08:41:30 +01:00
Dietmar Maurer 250c29edd2 tape: correctly sort media api entries 2020-12-29 12:09:51 +01:00
Dietmar Maurer c431659d05 cleanup: remove debug output 2020-12-29 11:59:57 +01:00
Dietmar Maurer a33389c391 tape: implement media content list api 2020-12-29 11:58:26 +01:00
Dietmar Maurer 3460565414 tape: create the MediaCatalog when we label a tape 2020-12-29 10:55:20 +01:00
Dietmar Maurer 26b62138ee cleanup: disable debug message when we detect a stopped worker task 2020-12-29 10:53:16 +01:00
Dietmar Maurer afb0220642 tape: cleanup LinuxDriveStatus - make density optional 2020-12-29 09:10:30 +01:00
Dietmar Maurer 0993923ed5 tape: factor out get_drive_and_media_status 2020-12-29 08:39:06 +01:00
Dietmar Maurer e0362b0d0f tape: correctly parse mtx import/export slots 2020-12-28 13:32:56 +01:00
Dietmar Maurer b27c32821c tape: install new sg-tape-cmd setuid binary 2020-12-28 11:10:25 +01:00
Dietmar Maurer 76b15a035f tape: MediaCatalog: write magic number before content 2020-12-26 11:05:25 +01:00
Dietmar Maurer eb8feb1281 tape: add LTO1 to TapeDensity 2020-12-26 10:48:32 +01:00
Dietmar Maurer fc6ce9835b tape: fix non-rewinding tape device check 2020-12-25 15:38:29 +01:00
Dietmar Maurer 8ae9f4efc2 tape: minor cleanups 2020-12-25 13:45:26 +01:00
Dietmar Maurer c9d13b0fc4 tape: expose check_tape_is_linux_tape_device 2020-12-24 15:51:49 +01:00
Dietmar Maurer bfacc1d8c3 tape: cleanup - factor out open_linux_tape_device 2020-12-24 11:24:45 +01:00
Dietmar Maurer 5ae86dfaa1 tape: return media usage info with status command 2020-12-23 11:24:34 +01:00
Dietmar Maurer dbe7e556b0 tape: implement binding for libsgutils2
So that we can read cartridge memory without calling "sg_raw". In future,
we may need further low level command to control the tape..
2020-12-23 09:44:53 +01:00
Stefan Reiter 4799280ccd http_client: add timeouts for critical connects
Use timeout futures for sections that might hang in certain error
conditions. This is mostly intended to be used as a safeguard, not a
first line of defense - i.e. best-effort avoidance of total hangs.

Not every future used for the HttpClient/H2Client is changed, only those
where a quick response is to be expected. For example, the response
reading futures are left alone, so data transfer is never capped with
timeout, only the initial server connect.

It is also used for upgrading to H2 connections, as that can take a long
time on overloaded servers.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-12-22 13:31:10 +01:00
Dietmar Maurer cb80d900b3 tape: add drive status api 2020-12-22 10:42:22 +01:00
Dietmar Maurer ee01737e87 tape: rename 'mam' api to 'cartridge-memory' 2020-12-22 09:27:34 +01:00
Dietmar Maurer 2012825913 depend on proxmox 0.9.0 2020-12-22 08:52:24 +01:00
Wolfgang Bumiller b2362a1207 adaptions for proxmox 0.9 and proxmox-api-macro 0.3
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-12-22 07:31:05 +01:00
Dietmar Maurer 1e20f819d5 tape: add command to read cartridge memory (MAM)
Thsi add an additional dependency to sg3-utils (small).
2020-12-21 12:12:33 +01:00
Dietmar Maurer 9aa58f0143 cleanup: rename mtfsf into forward_space_count_files 2020-12-18 16:57:49 +01:00
Dietmar Maurer 8835664653 tape: add tape backup api 2020-12-18 15:32:12 +01:00
Dietmar Maurer d37da6b7fc tape: add PoolWriter 2020-12-18 15:27:44 +01:00
Dietmar Maurer b9ee86efe1 tape: use SnapshotReader to create snapshot archive 2020-12-18 12:11:29 +01:00
Dietmar Maurer d108b610fd tape: fix write_media_set_label - move to correct position 2020-12-18 12:11:29 +01:00
Dominik Csapak 0ec79339f7 tools/daemon: improve reload behaviour
it seems that sometimes, the child process signal gets handled
before the parent process signal. Systemd then ignores the
childs signal (finished reloading) and only after going into
reloading state because of the parent. this will never finish.

Instead, wait for the state to change to 'reloading' after sending
that signal in the parent, an only fork afterwards. This way
we ensure that systemd knows about the reloading before actually trying
to do it.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Fabian Ebner <f.ebner@proxmox.com>
2020-12-18 10:30:37 +01:00
Dietmar Maurer 2afdc7f27d tape: MediaPool::with_config() - remove name parameter
Not required, because config already contains the pool name.
2020-12-18 08:14:24 +01:00
Dietmar Maurer 26aa9aca40 tape: return current_file_number as u64 2020-12-18 07:44:50 +01:00
Dominik Csapak 3e2984bcb9 tools/process_locker: Decrement writer count in drop handler
of ProcessLockSharedGuard.

We use a counter to determine if we can unlock the file again, but
we never actually decremented the writer count, so we held the
lock forever.

This fixes the issue that we could not start a garbage collect after
a reload, as long as the old process is still running, even when that
process has no active backup anymore but another long running task
(e.g. file download, terminal, etc.).

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-18 07:15:08 +01:00
Fabian Grünbichler a7a5406c32 acl: rustfmt module
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-18 07:07:01 +01:00
Fabian Grünbichler 4f727a783e acl: reformat privileges
for better readability, and tell rustfmt to leave those definitions
alone.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-18 07:05:45 +01:00
Fabian Grünbichler 23dc68fdea acl: add docs and adapt visibility
document all public things, add some doc links and make some
previously-public things only available for test cases or within the
crate:

previously public, now private:
- AclTreeNode::extract_user_roles (we have extract_roles())
- AclTreeNode::extract_group_roles (same)
- AclTreeNode::delete_group_role (exists on AclTree)
- AclTreeNode::delete_user_role (same)
- AclTreeNode::insert_group_role (same)
- AclTreeNode::insert_user_role (same)
- AclTree::write_config (we have save_config())
- AclTree::load (we have config()/cached_config())

previously public, now crate-internal:
- AclTree::from_raw (only used by tests)
- split_acl_path (used by some test binaries)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-18 07:05:11 +01:00
Dietmar Maurer b532dd00c4 tape: add helper to read snapshot contents
- lock the snapshot for reading
- use openat to open files
- provides an iterator over all chunks
2020-12-17 13:07:52 +01:00
Fabian Grünbichler c01742855a KeyConfig: bail on wrong fingerprint
instead of just logging the error. this should never happen in practice
unless someone is messing with the keyfile, in which case, it's better
to abort.

update tests accordingly (wrong fingerprint should fail, no fingerprint
should get the expected one).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 11:27:06 +01:00
Dietmar Maurer 9c953dd260 tape: add code to write backup snapshot files (without chunks) to tape 2020-12-17 08:28:47 +01:00
Dietmar Maurer 3fbf2d2fcd tape: cleanup MediaCatalog 2020-12-17 08:05:53 +01:00
Fabian Grünbichler e0af222ec3 KeyConfig: always calculate fingerprint
and warn if stored and calculated fingerprint don't match.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:52:55 +01:00
Fabian Grünbichler 73b5011786 KeyConfig: add encrypt/decrypt test
the RSA key and the encryption key itself are hard-coded to avoid
stalling the test runs because of lack of entropy, they have no special
significance otherwise.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:47:45 +01:00
Fabian Grünbichler 7137630d43 client: add 'import-with-master-key' command
to import an encrypted encryption key using a master key.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:46:24 +01:00
Fabian Grünbichler 8acfd15d6e key: move RSA-encryption to KeyConfig
since that is what gets encrypted, and not a CryptConfig.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:43:34 +01:00
Fabian Grünbichler 48fbbfeb7e fix #3197: skip fingerprint check when restoring key
when restoring an encrypted key, the original one is obviously not
available to check the fingerprint with.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:37:54 +01:00
Fabian Grünbichler 9990af3042 master key: store blob name in constant
since we will use it in more than one place.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-17 06:36:06 +01:00
Dietmar Maurer fe6c19383b tape: remove MediaLabelInfo, use MediaId instead
The additional content_uuid was quite useless...
2020-12-16 13:31:32 +01:00
Dietmar Maurer 9839d3f778 tape: improve docu 2020-12-16 12:43:51 +01:00
Dietmar Maurer dd59e3c2a1 tape: improve docu 2020-12-16 12:23:52 +01:00
Dietmar Maurer 0b7432ae09 tape: add chunk archive reader/writer 2020-12-16 12:08:34 +01:00
Dietmar Maurer c1c2c8f635 tape: cleanup MediaLocation type for direct use with API 2020-12-16 10:49:01 +01:00
Dietmar Maurer 42298d5896 tape: add magic number to identify media catalog files 2020-12-16 09:00:14 +01:00
Dietmar Maurer 205e187613 tape: add MediaCatalog implementation 2020-12-15 13:40:49 +01:00
Dietmar Maurer a78348acbb tape: rename DriveLabel to MediaLabel 2020-12-14 17:37:16 +01:00
Dietmar Maurer 410611b4f2 tape: improve file format docu 2020-12-14 17:29:57 +01:00
Dietmar Maurer af07ec8f29 tape: minor code cleanup 2020-12-14 16:56:26 +01:00
Dietmar Maurer 3f803af00b tape: scan - print more debug info 2020-12-14 13:16:18 +01:00
Dietmar Maurer ac461bd651 tape: implement scan command (useful for debug) 2020-12-14 12:55:49 +01:00
Dietmar Maurer ce955e1635 tape: implement eod cli command (debug tool) 2020-12-14 09:56:59 +01:00
Dietmar Maurer e20d008c6a tape: rename cli 'media media-destroy' toö 'media destroy' 2020-12-14 09:30:32 +01:00
Dietmar Maurer fb657d8ee5 tape: implement destroy_media 2020-12-14 08:58:40 +01:00
Dietmar Maurer fba0b77469 tape: add media api 2020-12-14 07:55:57 +01:00
Dietmar Maurer b5c1296eaa tape: make changer get_status async 2020-12-14 07:14:24 +01:00
Dietmar Maurer 065df12872 tape: split api type definitions for changers into extra file 2020-12-13 09:31:02 +01:00
Dietmar Maurer 7e1d4712b8 tape: rename CHANGER_ID_SCHEMA to CHANGER_NAME_SCHEMA 2020-12-13 09:22:08 +01:00
Dietmar Maurer 49c965a497 tape: rename DRIVE_ID_SCHEMA to DRIVE_NAME_SCHEMA 2020-12-13 09:18:16 +01:00
Dietmar Maurer 6fe9aedd0b tape: correctly call Async handler in proxmox-tape 2020-12-12 09:58:47 +01:00
Dietmar Maurer 42cb9bd6a5 tape: avoid executor blocking in changer api 2020-12-12 09:45:08 +01:00
Dietmar Maurer 66dbe5639e tape: avoid executor blocking in drive API
By using tokio::task::spawn_blocking().
2020-12-12 09:20:04 +01:00
Dietmar Maurer 663ef85992 tape: use WorkerTask for erase and rewind 2020-12-11 11:19:33 +01:00
Dietmar Maurer e92c75815b tape: split inventory api
inventory: sync, list labels with uuids,
update_inventory: WorkerTask, updates database
2020-12-11 10:42:29 +01:00
Dietmar Maurer 6dbad5b4b5 tape: run label commands as WorkerTask (threads) 2020-12-11 09:10:22 +01:00
Dietmar Maurer bff7e3f3e4 tape: implement barcode-label-mdedia 2020-12-11 07:50:19 +01:00
Dietmar Maurer 83abc7497d tape: implement inventory command 2020-12-11 07:39:28 +01:00
Dietmar Maurer be1a8c94ae fix build: add missing file 2020-12-10 13:40:20 +01:00
Dietmar Maurer 4606f34353 tape: implement read-label command 2020-12-10 13:20:39 +01:00
Dietmar Maurer 7bb720cb4d tape: implement label command 2020-12-10 12:30:27 +01:00
Dietmar Maurer c4d8542ec1 tape: add media pool handling 2020-12-10 11:41:35 +01:00
Dietmar Maurer 9700d5374a tape: add media pool cli 2020-12-10 11:13:12 +01:00
Dietmar Maurer 05e90d6463 tape: add media pool config api 2020-12-10 10:52:27 +01:00
Dietmar Maurer 55118ca18e tape: correctly sort drive api subdir 2020-12-10 10:09:12 +01:00
Dietmar Maurer f70d8091d3 tape: implement option changer-drive-id 2020-12-10 09:09:06 +01:00
Dietmar Maurer a3c709ef21 tape: cli cleanup - avoid api redefinition 2020-12-10 08:35:11 +01:00
Dietmar Maurer 4917f1e2d4 tape: implement delete property for drive update command 2020-12-10 08:25:46 +01:00
Dietmar Maurer 93829fc680 tape: cleanup load-slot api 2020-12-10 08:04:55 +01:00
Dietmar Maurer 5605ca5619 tape: cli cleanup - rename scana-for-* into scan 2020-12-10 07:58:45 +01:00
Dietmar Maurer e49f0c03d9 tape: implement load-media command 2020-12-10 07:52:56 +01:00
Dietmar Maurer 0098b712a5 tape: implement eject 2020-12-09 17:50:48 +01:00
Dietmar Maurer 5fb694e8c0 tape: implement rewind 2020-12-09 17:43:38 +01:00
Dietmar Maurer 583a68a446 tape: implement erase media 2020-12-09 17:35:31 +01:00
Dietmar Maurer e6604cf391 tape: add command line interface proxmox-tape 2020-12-09 13:00:20 +01:00
Dietmar Maurer 43cfb3c35a tape: do not remove changer while still used 2020-12-09 12:55:54 +01:00
Dietmar Maurer 8a16c571d2 tape: add changer property to drive create api 2020-12-09 12:55:10 +01:00
Dietmar Maurer 314652a499 tape: set protected flag for configuration change api methods 2020-12-09 12:02:55 +01:00
Dietmar Maurer 6b68e5d597 client: move connect_to_localhost into client module 2020-12-09 11:59:50 +01:00
Dietmar Maurer cafd51bf42 tape: add media state database 2020-12-09 11:21:56 +01:00
Wolfgang Bumiller 9b93c62044 remove unused descriptions from api macros
these are now a hard error in the api macro

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-12-09 10:55:18 +01:00
Dietmar Maurer 5d90860688 tape: expose basic tape/changer functionality at api2/tape/ 2020-12-08 15:42:50 +01:00
Dietmar Maurer 5ba83ed099 tape: check digest on config update 2020-12-08 11:24:38 +01:00
Dietmar Maurer 50bf10ad56 tape: add changer configuration API 2020-12-08 09:04:56 +01:00
Dietmar Maurer 16d444c979 tape: add tape drive configuration API 2020-12-07 13:04:32 +01:00
Dietmar Maurer fa9c9be737 tape: add tape device driver 2020-12-07 08:29:22 +01:00
Dietmar Maurer 2e7014e31d tape: add BlockeReader/BlockedWriter streams
This is the basic format used to write data to tapes.
2020-12-06 12:09:55 +01:00
Dietmar Maurer a84050c1f0 tape: add BlockHeader impl 2020-12-06 10:26:24 +01:00
Dietmar Maurer 7c9835465e tape: add helpers to emulate tape read/write behavior 2020-12-06 09:41:16 +01:00
Dietmar Maurer ec00200411 fix bug #3189: fix change_password permission checks, run protected 2020-12-05 16:20:29 +01:00
Dietmar Maurer b107fdb99a tape: add tape changer support using 'mtx' command 2020-12-05 14:54:12 +01:00
Dietmar Maurer 7320e9ff4b tape: add media invenotry 2020-12-05 12:54:15 +01:00
Dietmar Maurer c4d2d54a6d tape: define useful constants 2020-12-05 12:20:46 +01:00
Dietmar Maurer 1142350e8d tape: add media pool config 2020-12-05 11:59:38 +01:00
Dietmar Maurer d735b31345 tape: add tape read trait 2020-12-05 10:54:38 +01:00
Dietmar Maurer e211fee562 tape: add tape write trait 2020-12-05 10:51:34 +01:00
Dietmar Maurer 8c15560b68 tape: add file format definitions 2020-12-05 10:45:08 +01:00
Dietmar Maurer 327e93711f commit missing file: tape api type definitions 2020-12-04 16:00:52 +01:00
Dietmar Maurer a076571470 tape support: add drive configuration 2020-12-04 15:42:32 +01:00
Dominik Csapak 179145dc24 backup/datastore: move manifest locking to /run
this fixes the issue that on some filesystems, you cannot recursively
remove a directory when you hold a lock on a file inside (e.g. nfs/cifs)

it is not really backwards compatible (so during an upgrade, there
could be two daemons have the lock), but since the locking was
broken before (see previous patch) it should not really matter
(also it seems very unlikely that someone will trigger this)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-03 09:56:42 +01:00
Dominik Csapak 6bd0a00c46 backup/datastore: really lock manifest on delete
'lock_manifest' returns a Result<File, Error> so we always got the result,
even when we did not get the lock, but we acted like we had.

bubble the locking error up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-02 14:37:05 +01:00
Dominik Csapak f6e28f4e62 client/pull: log how many groups to pull were found
if no groups were found, the task log was very confusing as it
contained no real information why nothing was synced, e.g.:

 Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
 Sync datastore 'local-datastore' from 'remote/datastore'
 sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
 TASK OK

this patch simply logs how many groups were found and are about to be synced:

 Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
 Sync datastore 'local-datastore' from 'remote/datastore'
 found 0 groups to sync
 sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
 TASK OK

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-02 07:22:50 +01:00
Dietmar Maurer 2260f065d4 cleanup: use extra file for StoreProgress 2020-12-01 06:34:33 +01:00
Dietmar Maurer 6eff8dec4f cleanup: remove unnecessary StoreProgress clone() 2020-12-01 06:29:11 +01:00
Fabian Grünbichler 7e25b9aaaa verify: use same progress as pull
percentage of verified groups, interpolating based on snapshot count
within the group. in most cases, this will also be closer to 'real'
progress since added snapshots (those which will be verified) in active
backup groups will be roughly evenly distributed, while number of total
snapshots per group will be heavily skewed towards those groups which
have existed the longest, even though most of those old snapshots will
only be re-verified very infrequently.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:22:55 +01:00
Fabian Grünbichler f867ef9c4a progress: add format variants
for iterating over a single group, or iterating just on the group level

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:22:12 +01:00
Fabian Grünbichler fc8920e35d pull: factor out interpolated progress
and add group/snapshot count info.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:13:11 +01:00
Fabian Grünbichler 7f3b0f67e7 remove BackupGroup::list_groups
BackupInfo::list_backup_groups is identical code-wise, and makes more
sense as entry point for listing groups.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:09:44 +01:00
Fabian Grünbichler 844660036b gc: don't limit index listing to same filesystem
WalkDir does not follow symlinks by default anyway, and this behaviour
is not documented anywhere. e.g., if a sysadmin mounts 'extra storage'
for some backup group or type (not knowing that only metadata is stored
in those directories), GC will ignore all the indices contained within
and happily garbage collect their chunks..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:07:09 +01:00
Fabian Grünbichler efcac39d34 gc: remove duplicate variable
list_images already returns absolute paths, we don't need to prepend
anything.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:06:51 +01:00
Fabian Grünbichler cb4b721cb0 gc: log index files found outside of expected scheme
for safety reason, GC finds and marks all index files below the
datastore base path. as a result of regular operations, only index files
within the expected scheme of <TYPE>/<ID>/<TIMESTAMP> should exist.

add a small check + warning if the index list contains index files out
side of this expected scheme, so that an admin with shell access can
investigate.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:06:17 +01:00
Fabian Grünbichler 7956877f14 gc: shorten progress messages
we have messages starting the phases anyway, and limit the number of
progress updates so that context remains available at all times.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:04:13 +01:00
Wolfgang Bumiller 43e60ceb41 file logger: remove test.log after test as well
and a doc formatting fixup

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 14:13:21 +01:00
Wolfgang Bumiller b760d8a23f derive PartialEq for Userid
the manual implementation is equivalent

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 14:10:17 +01:00
Wolfgang Bumiller 2c1592263d tiny clippy hint
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 14:03:43 +01:00
Wolfgang Bumiller 616533823c don't enforce Vec and String in tools::join
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:56:59 +01:00
Wolfgang Bumiller 913dddea85 minor cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:56:21 +01:00
Wolfgang Bumiller 3530430365 tools avoid unnecessary copying of parameters/properties
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:53:49 +01:00
Wolfgang Bumiller a4ba60be8f minor cleanups
whitespace, formatting and superfluous lifetime annotations

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 13:47:31 +01:00
Wolfgang Bumiller 99e98f605c network helpers: fix fd leak in get_network_interfaces
This one always leaked.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller 935ee97b17 use fd_change_cloexec helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller 6b9bfd7fe9 minor cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller dd519bbad1 pxar: stricter file descriptor guards
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller 35fe981c7d client: use tools::pipe instead of nix
nix::unistd::pipe returns unguarded RawFds which should be
avoided

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Wolfgang Bumiller b6570abe79 changes for proxmox 0.8
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-30 11:25:53 +01:00
Fabian Grünbichler 490560e0c6 restore: print to STDERR
else restoring to STDOUT is broken..

Reported-by: Dominic Jäger <d.jaeger@proxmox.com>

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-25 14:38:02 +01:00
Dietmar Maurer 0c9209b04c cli: rename command "upload-log" to "snapshot upload-log" 2020-11-25 07:57:39 +01:00
Dietmar Maurer edebd52374 cli: rename command "forget" to "snapshot forget" 2020-11-25 07:57:39 +01:00
Dietmar Maurer 61205f00fb cli: rename command "files" to "snapshot files" 2020-11-25 07:57:39 +01:00
Dietmar Maurer a303e00289 fingerprint: add new() method 2020-11-25 07:57:39 +01:00
Fabian Grünbichler af9f72e9d8 fingerprint: add bytes() accessor
needed for libproxmox-backup-qemu0

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-25 06:34:34 +01:00
Dietmar Maurer 731eeef25b cli: use new alias feature for "snapshots"
Now maps to "snapshot list".
2020-11-24 13:26:43 +01:00
Dominik Csapak a65e3e4bc0 client: add 'snapshot notes show/update' command
to show and update snapshot notes from the cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-24 11:44:19 +01:00
Fabian Grünbichler 035c40e638 list_snapshots: return manifest fingerprint
for display in clients.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:44:55 +01:00
Fabian Grünbichler 79c535955d refactor BackupInfo -> SnapshotListItem helper
before adding more fields to the tuple, let's just create the struct
inside the match arms to improve readability.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:44:55 +01:00
Fabian Grünbichler 8b7f8d3f3d expose previous backup time in backup env
and use this information to add more information to client backup log
and guide the download manifest decision.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:44:55 +01:00
Fabian Grünbichler 23e4e90540 verification: fix message in notification mail
the errors Vec can contain failed groups as well (e.g., if a group has
no or an invalid owner).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:33:20 +01:00
Fabian Grünbichler a4fa3fc241 verification job: log failed dirs
else users have to manually search through a potentially very long task
log to find the entries that are different.. this is the same summary
printed at the end of a manual verify task.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:33:20 +01:00
Dietmar Maurer 81d10c3b37 cleanup: remove dead code 2020-11-24 08:03:00 +01:00
Fabian Grünbichler f1e2904150 paperkey: refactor common code
from formatting functions to main function, and pass along the key data
lines instead of the full string.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:57:21 +01:00
Fabian Grünbichler 23f9503a31 client: check fingerprint after downloading manifest
this is stricter than the check that happened on manifest load, as it
also fails if the manifest is signed but we don't have a key available.

add some additional output at the start of a backup to indicate whether
a previous manifest is available to base the backup on.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:55:12 +01:00
Fabian Grünbichler a0ef68b93c manifest: check fingerprint when loading with key
otherwise loading will run into the signature mismatch which is
technically true, but not the complete picture in this case.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:49:51 +01:00
Fabian Grünbichler 6b127e6ea0 fix #3139: add key fingerprint to manifest
if the manifest is signed/the contained archives/blobs are encrypted.
stored in 'unprotected' area, since there is already a strong binding
between key and manifest via the signature, and this avoids breaking
backwards compatibility for a simple usability improvement.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 07:45:11 +01:00
Dietmar Maurer 5e17dbf2bb cli: cleanup 'key show' - use format_and_print_result_full
We now expose all key derivation functions on the cli, so users can
choose between scrypt or pbkdf2.
2020-11-24 07:32:34 +01:00
Fabian Grünbichler dfb04575ad client: add 'key show' command
for (pretty-)printing a keyfile.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:15:29 +01:00
Fabian Grünbichler 6f2626ae19 client: print key fingerprint and master key
for operations where it makes sense.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:11:26 +01:00
Fabian Grünbichler 37e60ddcde key: add fingerprint to key config
and set/generate it on
- key creation
- key passphrase change
- key decryption if not already set
- key encryption with master key

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:03:46 +01:00
Fabian Grünbichler 05cdc05347 crypt config: add fingerprint mechanism
by computing the ID digest of a hash of a static string.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-23 13:03:16 +01:00
Dominik Csapak 968a0ab261 fix systemd-encoded upid strings in http client
since we systemd-encode parts of the upid string, and those can contain
characters that are invalid in urls (e.g. '\'), we have to percent encode
those

add a 'percent_encode_component' helper, so that we can maybe change
the AsciiSet for all uses at the same time

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-19 11:01:19 +01:00
Stefan Reiter fd19256470 gc: treat .bad files like regular chunks
Simplify the phase 2 code by treating .bad files just like regular
chunks, with the exception of stat logging.

To facilitate, we need to touch .bad files in phase 1. We only do this
under the condition that 1) the original chunk is missing (as before),
and 2) the original chunk is still referenced somewhere (since the code
lives in the error handler for a failed chunk touch, it only gets called
for chunks we expect to be there, i.e. ones that are referenced).

Untouched they will then be cleaned up after 24 hours (or after the last
longer-running task finishes).

Reason 2) is also a fix for .bad files not being cleaned up at all if
the original is no longer referenced anywhere (e.g. a user deleting all
snapshots after seeing some corrupt chunks appear).

cond_touch_path is introduced to touch arbitrary paths in the chunk
store with the same logic as touching chunks.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-18 14:04:49 +01:00
Fabian Grünbichler 1ed022576c api: include store in invalid owner errors
since a group might exist in plenty stores

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:11:24 +01:00
Fabian Grünbichler f6aa7b38bf drop now unused BackupInfo::list_backups
all global backup listing now happens via BackupGroup

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:11:21 +01:00
Fabian Grünbichler fdfcb74d67 api: filter snapshot counts
unprivileged users should only see the counts related to their part of
the datastore.

while we're at it, switch to a list groups, filter groups, count
snapshots approach (like list_snapshots) to speedup calls to this
endpoint when many unprivileged users share a datastore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:05:50 +01:00
Fabian Grünbichler 98afc7b152 api: make expensive parts of datastore status opt-in
used in the PBS GUI, but also for PVE usage queries which don't need all
the extra expensive information..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 11:05:47 +01:00
Fabian Grünbichler 0d08fceeb9 improve group/snapshot listing
by listing groups first, then filtering, then listing group snapshots.

this cuts down the number of openat/getdirents calls for users that just
have a partial view of the datastore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-18 10:37:04 +01:00
Dominik Csapak 3c945d73c2 client/http_client: add put method
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-16 16:59:14 +01:00
Fabian Ebner 58fcbf5ab7 client: expose all-file-systems option
Useful to avoid the need for a long (and possibly changing) list of include-dev
options in certain situations, e.g. nested ZFS file systems. The option is
already implemented and seems to work as expected. The checks for virtual
filesystems are not affected by this option.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-16 16:59:14 +01:00
Fabian Ebner c9097ff801 pxar: avoid including archive root's exclude patterns in .pxarexclude-cli
The patterns from the archive root's .pxarexclude file are already present in
self.patterns when encode_pxarexclude_cli is called. Pass along the number of
CLI patterns and slice accordingly.

Suggested-By: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 13:05:09 +01:00
Wolfgang Bumiller fb01fd3af6 visibility cleanups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:53:50 +01:00
Fabian Ebner fa4bcbcad0 pxar: only generate .pxarexclude-cli if there were CLI parameters
previously a .pxarexclude entry in the root of the archive caused the file to
be generated as well, because the patterns are read before calling
generate_directory_file_list and within the function it wasn't possible to
distinguish between a pattern coming from the CLI and a pattern coming from
archive/root/.pxarexclude

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:18:08 +01:00
Fabian Ebner 189cdb7427 pxar: include .pxarexclude files in the archive
The documentation states:
.pxarexclude files are treated as regular files and will be included in the
backup archive.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:18:06 +01:00
Fabian Ebner 874bd5454d pxar: fix anchored exclusion at archive root
There is no leading slash in an entry's full_path, causing an anchored
exclude at the root level to fail, e.g. having "/name" as the content of the
file archive/root/.pxarexclude didn't match the file archive/root/name

Fix this by prepending a leading slash before matching.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:18:04 +01:00
Wolfgang Bumiller b649887e9a remove unused function
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 11:15:15 +01:00
Thomas Lamprecht 8c62c15f56 follouwp: whitespace cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-12 11:02:45 +01:00
Mira Limbeck 51ac17b56e api: apt/versions: fix running_kernel string for unknown package case
Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-11-12 11:02:20 +01:00
Thomas Lamprecht fc5a012068 manager: versions: non-verbose should actually print server pkg info
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-12 10:28:03 +01:00
Stefan Reiter 5e293f1315 apt: use typed response for get_versions
...and cleanup get_versions for manager CLI.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-12 10:15:32 +01:00
Wolfgang Bumiller 97030c9407 cleanup clippy leftovers
this used to contain a pointer cast, now it doesn't

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 09:43:38 +01:00
Wolfgang Bumiller 5d1d0f5d6c use pin-project to remove more unsafe blocks
we already have it in our dependency tree, so use it

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 09:43:38 +01:00
Thomas Lamprecht 294466ee61 manager: versions: unify printing
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00
Mira Limbeck c100fe9108 add versions command to proxmox-backup-manager
Add the versions command to proxmox-backup-manager with a similar output
to pveversion [-v]. It prints the packages line by line with only the
package name, followed by the version and, for proxmox-backup and
proxmox-backup-server, some additional information (running kernel,
running version).

In addition it supports the optional output-format parameter which can
be used to print the complete data in either json, json-pretty or text
format. If output-format is specified, the --verbose parameter is
ignored and the detailed list of packages is printed.

With the addition of the versions command, the report is extended as
well.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-11-11 18:30:33 +01:00
Thomas Lamprecht e754da3ac2 api: versions: add version also in server package unknown case
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00
Thomas Lamprecht bc1e52bc38 api: versions: rust fmt cleanups
line length limit is 100

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00
Thomas Lamprecht 6f0073bbb5 api: apt update info: do not serialize extra info if none
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 18:30:33 +01:00
Mira Limbeck 2decf85d6e add extra_info field to APTUpdateInfo
Add an optional string field to APTUpdateInfo which can be used for
extra information.

This is used for passing running kernel and running version information
in the versions API call together with proxmox-backup and
proxmox-backup-server.

Signed-off-by: Mira Limbeck <m.limbeck@proxmox.com>
2020-11-11 16:39:11 +01:00
Dominik Csapak 1d8f849457 api2/node/syslog: use 'real_service_name' here also
for now this only does the 'postfix' -> 'postfix@-' conversion,
fixes the issue that we only showed the 'postfix' service syslog
(which is rather empty in a default setup) instead of the instance one

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-11 16:36:42 +01:00
Stoiko Ivanov beb07279b6 log source of encryption key
This patch prints the source of the encryption key when running
operations with proxmox-backup-client.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-11-11 16:35:20 +01:00
Stoiko Ivanov 8c6854c8fd inform user when using default encryption key
Currently if you generate a default encryption key:
`proxmox-backup-client key create --kdf none`

all backup operations which don't explicitly disable encryption will be
encrypted with this key.

I found it quite surprising, that my backups were all encrypted without
me explicitly specfying neither key nor encryption mode

This patch informs the user when the default key is used (and no
crypt-mode is provided explicitly)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-11-11 16:35:20 +01:00
Oguz Bektas 57f472d9bb report: use '$' instead of '#' for showing commands
since some files can contain '#' character for comments. (i.e.,
/etc/hosts)

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-11-11 16:19:37 +01:00
Oguz Bektas 94ffca10a2 report: fix grammar error
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-11-11 16:19:33 +01:00
Dominik Csapak c0026563b0 make user properties deletable
by using our usual pattern for the update call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-11 14:09:40 +01:00
Fabian Grünbichler e411924c7c rest: check for disabled token (user)
when authenticating a token, and not just when authenticating a
user/ticket.

Reported-By: Dominik Jäger <d.jaeger@proxmox.com>

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-11 12:21:29 +01:00
Thomas Lamprecht 06c9059dac daemon: rename method, endless loop, bail on exec error
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-11 10:14:01 +01:00
Dominik Csapak d7c6ad60dd daemon: add hack for sd_notify
sd_notify is not synchronous, iow. it only waits until the message
reaches the queue not until it is processed by systemd

when the process that sent such a message exits before systemd could
process it, it cannot be associated to the correct pid

so in case of reloading, we send a message with 'MAINPID=<newpid>'
to signal that it will change. if now the old process exits before
systemd knows this, it will not accept the 'READY=1' message from the
child, since it rejects the MAINPID change

since there is no (AFAICS) library interface to check the unit status,
we use 'systemctl is-active <SERVICE_NAME>' to check the state until
it is not 'reloading' anymore.

on newer systemd versions, there is 'sd_notify_barrier' which would
allow us to wait for systemd to have all messages from the current
pid to be processed before acknowledging to the child, but on buster
the systemd version is to old...

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-11 09:43:00 +01:00
Wolfgang Bumiller a86e703661 tools::runtime: pin_mut instead of unsafe block
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-10 14:18:45 +01:00
Wolfgang Bumiller 1ecf4e6d20 async_io: require Unpin for EitherStream and HyperAccept
We use it with Unpin types and this way we get rid of a lot
of `unsafe` blocks.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-10 14:18:45 +01:00
Fabian Grünbichler 9f9a661b1a verify: cleanup logging order/messages
otherwise we end up printing warnings before the start message..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-10 14:11:36 +01:00
Fabian Grünbichler 1b1cab8321 verify: log/warn on invalid owner
in order to trigger a notification/make the problem more visible than
just in syslog.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-10 14:11:36 +01:00
Fabian Grünbichler 10052ea644 remote.cfg: rename userid to 'auth-id'
and fixup config file on upgrades accordingly

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-10 13:25:24 +01:00
Fabian Grünbichler 414c23facb fix #3060:: improve get_owner error handling
log invalid owners to system log, and continue with next group just as
if permission checks fail for the following operations:
- verify store with limited permissions
- list store groups
- list store snapshots

all other call sites either handle it correctly already (sync/pull), or
operate on a single group/snapshot and can bubble up the error.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-10 12:58:44 +01:00
Wolfgang Bumiller 5d08c750ef HttpsConnector: include destination on connect errors
for more useful log output
old:
Nov 10 11:50:51 foo pvestatd[3378]: proxmox-backup-client failed: Error: error trying to connect: tcp connect error: No route to host (os error 113)
new:
Nov 10 11:55:21 foo pvestatd[3378]: proxmox-backup-client failed: Error: error trying to connect: error connecting to https://thebackuphost:8007/ - tcp connect error: No route to host (os error 113)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-10 11:58:19 +01:00
Wolfgang Bumiller f3fde36beb client: error context when building HttpClient
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-10 11:58:19 +01:00
Dietmar Maurer 30f73fa2e0 fix bug #3060: continue sync if we cannot aquire the group lock 2020-11-10 11:29:36 +01:00
Dominik Csapak b35eb0a175 api2/status/datastore-usage: add gc-status and history start and delta
so that we can show more info and calculate the points in time for the
history

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-10 10:43:07 +01:00
Fabian Ebner c960d2b501 bail if mount point already exists for directories
similar to what we do for zfs. By bailing before partitioning, the disk is
still considered unused after a failed attempt.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-10 09:25:58 +01:00
Fabian Ebner f5d9f2534b mount zpools created via API under /mnt/datastore
as we do for other file systems

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-11-10 09:25:58 +01:00
Stefan Reiter 137a6ebcad apt: allow changelog retrieval from enterprise repo
If a package is or will be installed from the enterprise repo, retrieve
the changelog from there as well (securely via HTTPS and authenticated
with the subcription key).

Extends the get_string method to take additional headers, in this case
used for 'Authorization'. Hyper does not have built-in basic auth
support AFAICT but it's simple enough to just build the header manually.

Take the opportunity and also set the User-Agent sensibly for GET
requests, just like for POST.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-09 17:28:58 +01:00
Dominik Csapak 6e880f19cc api2/node/tasks: add check_job_store and use it
to easily check the store of a worker_id
this fixes the issue that one could not filter by type 'syncjob' and
datastore simultaneously

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-09 16:37:24 +01:00
Thomas Lamprecht 0eb0c4bd63 proxy: fix log message for auth log rotation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-09 16:34:03 +01:00
Thomas Lamprecht ed2beb334d api: node/apt: add versions call
very basic, based on API/concepts of PVE one.

Still missing, addint an extra_info string option to APTUpdateInfo
and pass along running kernel/PBS version there.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-09 16:31:56 +01:00
Dietmar Maurer 2791318ff1 fix bug #3121: forbid removing used reemotes 2020-11-09 12:48:29 +01:00
Wolfgang Bumiller 47208b4147 pxar: log when skipping mount points
Clippy complains about the number of paramters we have for
create_archive and it really does need to be made somewhat
less awkward and more usable. For now we just log to stderr
as we previously did. Added todo-comments for this.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-09 12:43:16 +01:00
Thomas Lamprecht 38260cddf5 tools apt: include package name in filter data
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-09 08:55:08 +01:00
Dominik Csapak db8a606707 proxmox-backup-proxy: remove unnecessary alias
the basedir is already /usr/share/javascript/proxmox-backup/
so adding a subdir of that as alias is not needed

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-06 18:08:18 +01:00
Thomas Lamprecht ad53c1d6dd api: datastore: allow to set "verify-new" option over API
Until now, one could only set this by editing the configuration file
manually.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-06 17:24:14 +01:00
Wolfgang Bumiller b59c308219 Vec::new is Vec's default default
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-06 14:55:34 +01:00
Fabian Grünbichler 0224c3c273 client: properly complete new-owner
with remote Authids, not local Userids.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-06 14:54:08 +01:00
Fabian Grünbichler dbd45a72c3 tasks: allow access to job tasks
if the user/token could have either configured/manually executed the
task, but it was either executed via the schedule (root@pam) or
another user/token.

without this change, semi-privileged users (that cannot read all tasks
globally, but are DatastoreAdmin) could schedule jobs, but not read
their logs once the schedule executes them. it also makes sense for
multiple such users to see eachothers manually executed jobs, as long as
the privilege level on the datastore (or remote/remote_store/local
store) itself is sufficient.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-06 12:56:06 +01:00
Fabian Grünbichler 4c979d5450 verify: allow unprivileged access to admin API
which is the one used by the GUI.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-06 12:41:41 +01:00
Fabian Grünbichler 35c80d696f verify: fix unprivileged verification jobs
since the store is not a path parameter, we need to do manual instead of
schema checks. also dropping Datastore.Backup here

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-06 12:39:06 +01:00
Dietmar Maurer ad54df3178 get rid of backup@pam 2020-11-06 08:39:30 +01:00
Fabian Grünbichler e0100d618e api: refactor remote client and add remote scan
to allow on-demand scanning of remote datastores accessible for the
configured remote user.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-05 12:56:20 +01:00
Fabian Grünbichler 455e5f7110 types: extract DataStoreListItem
for reuse in remote scan API call

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-05 12:56:20 +01:00
Dietmar Maurer c26c9390ff config: make notify a property string
For example "gc=never,verify=always,sync=error".
2020-11-05 11:35:14 +01:00
Dominik Csapak 9e45e03aef tools/daemon: fix reload with open connections
instead of await'ing the result of 'create_service' directly,
poll it together with the shutdown_future

if we reached that, fork_restart the new daemon, and await
the open future from 'create_service'

this way the old process still handles open connections until they finish,
while we already start a new process that handles new incoming connections

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-05 11:14:56 +01:00
Wolfgang Bumiller e144810d73 pxar: more concise EOF handling
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-05 10:32:48 +01:00
Dominik Csapak 3c2dd8ad05 pxar/create: handle ErrorKind::Interrupted for file reads
they are not an error and we should retry the read

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-05 10:27:36 +01:00
Dominik Csapak 91e3b38da4 pxar/create: fix endless loop for shrinking files
when a file shrunk during backup, we endlessly looped, reading/copying 0 bytes
we already have code that handles shrunk files, but we forgot to
break from the read loop

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-05 10:27:30 +01:00
Thomas Lamprecht fe7bdc9d29 proxy: also rotate auth.log file
no need for triggering re-open here, we always re-open that file.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-04 17:26:34 +01:00
Thomas Lamprecht 546b6a23df proxy: logrotate: do not serialize sending async log-reopen commands
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-04 17:26:34 +01:00
Thomas Lamprecht 4fdf13f95f api: factor out auth logger and use for all API authentication failures
we have information here not available in the access log, especially
if the /api2/extjs formatter is used, which encapsulates errors in a
200 response.

So keep the auth log for now, but extend it use from create ticket
calls to all authentication failures for API calls, this ensures one
can also fail2ban tokens.

Do that logging in a central place, which makes it simple but means
that we do not have the user ID information available to include in
the log.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-04 17:26:34 +01:00
Thomas Lamprecht 385681c9ab worker task: fix passing upid to send command
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-04 17:16:55 +01:00
Thomas Lamprecht be99df2767 log rotate: only add .zst to new file after second rotation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-04 17:16:55 +01:00
Dietmar Maurer f47c1d3a2f proxy: use new datastore notify settings 2020-11-04 11:54:29 +01:00
Dietmar Maurer 6e545d0058 config: allow to configure who receives job notify emails 2020-11-04 11:54:29 +01:00
Dietmar Maurer 42ca9e918a sync: improve log format 2020-11-04 09:10:56 +01:00
Dietmar Maurer ea93bea7bf proxy: log if there are too many open connections 2020-11-04 08:49:35 +01:00
Dietmar Maurer 0081903f7c fix bug #2870: use updated tickets 2020-11-04 08:20:36 +01:00
Dietmar Maurer e1d367df47 proxy: use env PROXMOX_DEBUG to enable/disable debug output
We only print early connection errors when this env var is set.
2020-11-04 06:55:57 +01:00
Dietmar Maurer 71f413cd27 cleanup: use Arc to count open connections 2020-11-04 06:35:44 +01:00
Dietmar Maurer 48aa2b93b7 fix #3106: correctly queue incoming connections 2020-11-04 06:24:42 +01:00
Thomas Lamprecht 2f08ee1fe3 report: add more commands/files to check
add all of our configuration files in /etc/proxmox-backup/ further,
call some ZFS tool to get that status.

Also, use the subscription command form manager, as we often require
more info than the status. Also, adapt formatting a bit.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-03 17:33:16 +01:00
Thomas Lamprecht 93f077c5cf report: avoid lazy_static for command/files/.. definitions
those are not in a hot code path, and it is not really much work to
build them on the go..

It may not matther much, but it is unnecessary. Rust will probably
inline most of it anyway..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-03 17:27:16 +01:00
Thomas Lamprecht 941342f70e manager: report: call method directly, avoid HTTPS request
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-03 17:23:43 +01:00
Hannes Laimer 9a556c8a30 manager: add report cli command
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-11-03 15:16:42 +01:00
Hannes Laimer b0ef9631e6 report: add api endpoint and function to generate report
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-11-03 15:16:42 +01:00
Dominik Csapak c1fa057cce api2/node/tasks: add optional until filter
so that users select specific time ranges with 'since' and 'until'
(e.g. a single day)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-03 11:35:21 +01:00
Dominik Csapak f66565203a api2/status: remove list_task api call
we do not need it anymore, we can do everything with nodes/NODE/tasks
instead

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-03 11:35:21 +01:00
Dominik Csapak a2a7dd1535 api2/node/tasks: add optional since/typefilter/statusfilter
and change all users of the /status/tasks api call to this

with this change we can now delete the /status/tasks api call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-03 11:35:21 +01:00
Dominik Csapak e7dd169fdf api2/node/tasks: change limit behaviour when it is 0
instead of returning 0 elements (which does not really make sense anyway),
change it so that there is no limit anymore (besides usize::MAX)

this is technically a breaking change for the api, but i guess
no one is using limit=0 for anything sensible anyway

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-03 11:35:21 +01:00
Dominik Csapak fa31f4c54c server/worker_task: add tasktype to return the api type of a taskstate
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-03 11:35:21 +01:00
Dietmar Maurer 038ee59960 cleanup: use const_regex, use BACKUP_ID_REGEX for api too 2020-11-03 06:36:50 +01:00
Dominik Csapak e1c1533790 fix #3039: use the same ID regex for info and api
in the api we use PROXMOX_SAFE_ID_REGEX for backup ids, but here
(where we use it to list them) we use a local regex

since the first is a superset of the one used here, simply extend
the local one

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-03 06:25:06 +01:00
Fabian Grünbichler aa64e06540 sync: add access check tests
should cover all the current scenarios. remote server-side checks can't
be meaningfully unit-tested, but they are simple enough so should
hopefully never break.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-02 21:13:24 +01:00
Fabian Grünbichler 18077ac633 user.cfg/user info: add test constructors
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-02 21:13:24 +01:00
Thomas Lamprecht a71a009313 proxy: drop now unused UPID import
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 21:08:38 +01:00
Dominik Csapak b6ba5acd29 proxmox-backup-proxy: use only jobstate for garbage_collection schedule
in case the garbage_collection errors out, we never set the in-memory
state, so if it failed, the last 'good' starttime was considered
for the schedule

this could lead to the job running every minute instead of the
correct schedule

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-02 21:08:38 +01:00
Dominik Csapak 4fdf5ddf5b api2/admin/datastore: start the garbage_collection task with our helper
instead of manually, this has the advantage that we now set
the jobstate correctly and can return with an error if it is
currently running (instead of failing in the task)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-02 21:08:38 +01:00
Dominik Csapak c724f65805 server/gc_job: add 'to_stdout'
we will use this for the manual api call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-02 21:08:38 +01:00
Dominik Csapak 79c9bf55b9 backup/{dynamic, fixed}_index: improve error message for small index files
index files that were smaller than their respective header size,
would fail with

"failed to fill whole buffer"

instead now check explicitely for the size and fail with
"index too small (size)"

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-02 21:08:38 +01:00
Thomas Lamprecht 788d82d9b7 gc: mark_used_chunks: reduce implementation noise
try do reduce some unecessary lines, make match arms more precise so
one can faster see what's actually happening.

Also, avoid
> return Err(format_err!(...))
stuff, just use bail!()

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 21:08:38 +01:00
Dominik Csapak 2f0b92352d garbage collect: improve index error messages
so that in case of a broken index file, the user knows which it is

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-02 20:08:50 +01:00
Thomas Lamprecht b7f2be5137 log rotate task: make task archive limits be binary based
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:53:30 +01:00
Thomas Lamprecht 72aa1834dc log rotate task: adapt internal jobstate ID, set worker one to None for now
as we have only one logrotate task currently..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:53:30 +01:00
Thomas Lamprecht fe4cc5b1a1 server: implement access log rotation with re-open via command socket
re-use the future we already have for task log rotation to trigger
it.

Move the FileLogger in ApiConfig into an Arc, so that we can actually
update it and REST using the new one.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:53:30 +01:00
Thomas Lamprecht 04b053d87e server: write main daemon PID to run directory
so that we can easily get the main PID of the last recently launched
daemon. Will be used to get the control socket of that one for access
lgo rotate in a future patch

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:50:24 +01:00
Thomas Lamprecht b469011fd1 command socket: make create_control_socket private
this is internal for now, use the comanndo socket struct
implementation, and ideally not a new one but the existing ones
created in the proxy and api daemons.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:50:24 +01:00
Thomas Lamprecht a68768cf31 server: use generalized commando socket for worker tasks commands
Allows to extend the use of that socket in the future, e.g., for log
rotate re-open signaling.

To reflect this we use a more general name, and change the commandos
to a more clear namespace.

Both are actually somewhat a breaking change, but the single real
world issue it should be able to cause is, that one won't be able to
stop task from older daemons, which still use the older abstract
socket name format.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:48:04 +01:00
Thomas Lamprecht f3df613cb7 server: add CommandoSocket where multiple users can register commands
This is a preparatory step to replace the task control socket with it
and provide a "reopen log file" command for the rest server.

Kept it simple by disallowing to register new commands after the
socket gets spawned, this avoids the need for locking.

If we really need that we can always wrap it in a Arc<RWLock<..>> or
something like that, or even nicer, register at compile time.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:32:22 +01:00
Thomas Lamprecht 056ee78567 config: network: use error message when parsing netmask failed
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:32:22 +01:00
Thomas Lamprecht 3cd529ea51 tools: file logger: avoid some possible unwraps in log method
writing to a file can explode quite easily.
time formatting to rfc3339 should be more robust, but it has a few
conditions where it could fail, so catch that too (and only really
do it if required).

The writes to stdout are left as is, it normally is redirected to
journal which is in memory, and thus breaks later than most stuff,
and at that point we probably do not care anymore anyway.

It could make sense to actually return a result here..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:32:22 +01:00
Thomas Lamprecht 3aade17125 tools: log rotate: compressing rotated files
We renamed the last one always to a file without compression
extension, even if it was .zst previously. So always add the correct
ending to the new last one, if compress was true.

Further, we cannot detect if there'd be a compression required if we
rotated (renamed) it already to the file with .zst included.

So check on rotation itself if it would be a "no .zst" -> ",zst"
transition, and call compress there.

it really should be OK now *knocking wood*

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 18:35:13 +01:00
Thomas Lamprecht 1dc2fe20dd tools: log rotate: fix file ending for compressed files
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 18:35:13 +01:00
Dietmar Maurer 645a47ff6e config: support netmask when parsing interfaces file 2020-11-02 14:32:35 +01:00
Thomas Lamprecht a9fcbec9dc file logger: allow reopening file
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 10:03:10 +01:00
Thomas Lamprecht 346a488e35 pull out /run and /var/log directory constants to buildcfg
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 10:03:10 +01:00
Dietmar Maurer 3066f56481 notify: add link to server GUI 2020-11-02 09:12:14 +01:00
Dietmar Maurer 07ca4e3609 gc: remove extra empty lines in email notification template 2020-11-02 09:12:14 +01:00
Fabian Grünbichler 59af9ca98e sync: allow sync for non-superusers
by requiring
- Datastore.Backup permission for target datastore
- Remote.Read permission for source remote/datastore
- Datastore.Prune if vanished snapshots should be removed
- Datastore.Modify if another user should own the freshly synced
snapshots

reading a sync job entry only requires knowing about both the source
remote and the target datastore.

note that this does not affect the Authid used to authenticate with the
remote, which of course also needs permissions to access the source
datastore.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-02 07:10:12 +01:00
Fabian Grünbichler f1694b062d fix #2864: add owner option to sync
instead of hard-coding 'backup@pam'. this allows a bit more flexibility
(e.g., syncing to a datastore that can directly be used as restore
source) without overly complicating things.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-02 07:08:05 +01:00
Thomas Lamprecht fa7aceeb15 manager: subscription commands s/delete/remove/
no idea why I added it as "delete", for all other such operations we
use the "remove" sub-command...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-01 13:19:30 +01:00
Thomas Lamprecht 0e16f57e37 apt: sort packages for update notifcation mail
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 22:58:52 +01:00
Thomas Lamprecht bc00289bce add daily update and maintenance task
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 22:51:26 +01:00
Thomas Lamprecht 86d602457a api: apt: implement support to send notification email on new updates
again, base idea copied off PVE, but, we safe the information about
which pending version we send a mail out already in a separate
object, to keep the api return type APTUpdateInfo clean.

This also makes a few things a bit easier, as we can update the
package status without saving/restoring the notify information.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 22:51:26 +01:00
Thomas Lamprecht 33508b1237 api: implement apt pkg cache
based on the idea of PVE

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:42:49 +01:00
Thomas Lamprecht b282557563 api: apt: factor out and improve calling apt update
apt changes some of its state/cache also if it errors out, most of
the time, so we actually want to print both, stderr and stdout.

Further, only warn if its exit code is non-zero, for the same
rationale, it may bring updates available even if it errors (e.g.,
because a future pbs-enterprise repo is additionally configured but
not accessible).

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:59 +01:00
Thomas Lamprecht e6513bd5de api/tools: split out apt helpers from api to own module
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:36 +01:00
Thomas Lamprecht 5911f74096 api types: derive Debug for APTUpdateInfo
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:36 +01:00
Thomas Lamprecht 0bb74e54b1 worker task: drop debug prints
they are not useful anymore, rather noisy

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:36 +01:00
Thomas Lamprecht f254a27071 tools: do not unnecessarily prefix module path
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:36 +01:00
Thomas Lamprecht d0abba3397 trivial: fix typo in comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:36 +01:00
Dietmar Maurer 1143f6ca93 cleanup: fix wording in GC status emails 2020-10-31 07:56:42 +01:00
Fabian Grünbichler 8b600f9965 api: replace auth_id with auth-id
in parameters, and fix up the completion for the ACL update parameter.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:46:19 +01:00
Fabian Grünbichler e4e280183e privs: add some more comments explaining privileges
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:42:30 +01:00
Fabian Grünbichler 2fc45a97a9 privs: remove PRIV_REMOVE_PRUNE
it's not used anywhere, and not needed either until the day we might
implement push syncs.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:42:26 +01:00
Fabian Grünbichler b7ce2e575f verify jobs: add permissions
equivalent to verifying a whole datastore, except for reading job
(entries), which is accessible to regular Datastore.Audit/Backup users
as well.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:36:52 +01:00
Fabian Grünbichler 09f6a24078 verify: introduce & use new Datastore.Verify privilege
for verifying a whole datastore. Datastore.Backup now allows verifying
only backups owned by the triggering user.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:36:52 +01:00
Fabian Grünbichler b728a69e7d privs: use Datastore.Modify|Backup to set backup notes
Datastore.Backup is limited to owned groups, as usual.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:36:52 +01:00
Fabian Grünbichler 1401f4be5f privs: allow reading notes with Datastore.Audit
they are returned when reading the manifest, which just requires
Datastore.Audit as well. Datastore.Read is for reading backup contents,
not metadata.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:36:52 +01:00
Wolfgang Bumiller e4a864bd21 impl From<Authid> for Userid
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-30 15:19:07 +01:00
Thomas Lamprecht 625a56b75e server/rest: accept also = as token separator
Like we do in Proxmox VE

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-30 13:34:26 +01:00
Thomas Lamprecht 6d8a1ac9e4 server/rest: user constants for HTTP headers
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-30 13:33:36 +01:00
Fabian Grünbichler 362739054e api tokens: add authorization method
and properly decode secret (which is a no-op with the current scheme).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 13:15:14 +01:00
Thomas Lamprecht 2762481cc8 proxmox-backup-manager: add subscription commands
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-30 13:03:58 +01:00
Thomas Lamprecht 652506e6b8 api: define subscription module and methods as public
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-30 13:03:58 +01:00
Thomas Lamprecht 926d253126 api: define subscription key schema and use it
nicer to have the correct regex checked in parameter verification
already

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-30 12:57:14 +01:00
Fabian Grünbichler 1cd951c93e proxy: fix warnings
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 12:49:43 +01:00
Dietmar Maurer 3b707fbb8f proxy: split out code to run garbage collection job 2020-10-30 11:01:45 +01:00
Dietmar Maurer b15751bf55 check_schedule cleanup: use &str instead of String
This way we can avoid many clone() calls.
2020-10-30 09:49:50 +01:00
Hannes Laimer 82c05b41fa proxy: extract commonly used logic for scheduling into new function
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-30 09:49:50 +01:00
Hannes Laimer b8d9079835 proxy: move prune logic into new file
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-30 09:49:50 +01:00
Wolfgang Bumiller 906ef6c5bd api2/access/user: fix return type schema
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-29 15:20:10 +01:00
Wolfgang Bumiller ea1853a17b api2/access/user: drop Option, treat empty Vec as None
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-29 15:17:54 +01:00
Wolfgang Bumiller 221177ba41 fixup hardcoded paths
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-29 15:15:17 +01:00
Fabian Grünbichler b2da7fbd1c acls: allow viewing/editing user's token ACLs
even for otherwise unprivileged users.

since effective privileges of an API token are always intersected with
those of their owning user, this does not allow an unprivileged user to
elevate their privileges in practice, but avoids the need to involve a
privileged user to deploy API tokens.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler 4615325f9e manager: add user permissions command
useful for debugging complex ACL setups.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler 2156dec5a9 manager: add token commands
to generate, list and delete tokens. adding them to ACLs already works
out of the box.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler 16245d540c tasks: allow unpriv users to read their tokens' tasks
and tighten down the return schema while we're at it.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler bff8557298 owner checks: handle backups owned by API tokens
a user should be allowed to read/list/overwrite backups owned by their
own tokens, but a token should not be able to read/list/overwrite
backups owned by their owning user.

when changing ownership of a backup group, a user should be able to
transfer ownership to/from their own tokens if the backup is owned by
them (or one of their tokens).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler 34aa8e13b6 client/remote: allow using ApiToken + secret
in place of user + password.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler babab85b56 api: add permissions endpoint
and adapt privilege calculation to return propagate flag

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler 6746bbb1a2 api: allow listing users + tokens
since it's not possible to extend existing structs, UserWithTokens
duplicates most of user::User.. to avoid duplicating user::ApiToken as
well, this returns full API token IDs, not just the token name part.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler 942078c40b api: add API token endpoints
beneath the user endpoint.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler c30816c1f8 REST: extract and handle API tokens
and refactor handling of headers in the REST server while we're at it.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler e6dc35acb8 replace Userid with Authid
in most generic places. this is accompanied by a change in
RpcEnvironment to purposefully break existing call sites.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:11:39 +01:00
Fabian Grünbichler f8adf8f83f config: add token.shadow file
containing pairs of token ids and hashed secret values.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:11:39 +01:00
Fabian Grünbichler e0538349e2 api: add Authid as wrapper around Userid
with an optional Tokenname, appended with '!' as delimiter in the string
representation like for PVE.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:11:39 +01:00
Thomas Lamprecht b6563f48ad GC: improve task logs
Make it more clear that removed files are chunks (not indexes or
something like that, user cannot know that we do not touch them here)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-29 14:47:39 +01:00
Thomas Lamprecht 932390bd46 GC: fix logging leftover bad chunks
fixes commit b4fb262335, which copied
over the "Removed bad files:" block, but only adapted the log text,
not the actual variable.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-29 14:40:29 +01:00
Dominik Csapak 264779e704 server/worker_task: simplify task log writing
instead of prerotating 1000 tasks
(which resulted in 2 writes each time an active worker was finished)
simply append finished tasks to the archive (which will be rotated)

page cache should be good enough so that we can get the task logs fast

since existing installations might have an 'index' file, we
still have to read tasks from there, but only if it exists

this simplifies the TaskListInfoIterator a good amount

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-29 12:41:20 +01:00
Thomas Lamprecht 7f3d91003c worker task: remove debug print, faster modulo
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-29 12:35:33 +01:00
Dominik Csapak 14e0862509 api: datstore status: introduce proper structs and restore compatibility
by moving the properties of the storage status out again to the top
level object

also introduce proper structs for the types used, to get type-safety
and better documentation for the api calls

this changes the backup counts from an array of [groups,snapshots] to
an object/struct with { groups, snapshots } and include 'other' types
(though we do not have any at this moment)

this way it is better documented

this also adapts the ui code to cope with the api changes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-29 12:31:27 +01:00
Dietmar Maurer 9e733dae48 send sync job status emails 2020-10-29 12:22:50 +01:00
Dietmar Maurer bfea476be2 schedule_datastore_sync_jobs: remove unneccessary clone() 2020-10-29 12:22:41 +01:00
Dietmar Maurer 385cf2bd9d send_job_status_mail: corectly escape html characters 2020-10-29 11:22:08 +01:00
Dietmar Maurer d6373f3525 garbage_collection: log deduplication factor 2020-10-29 11:13:01 +01:00
Dietmar Maurer b4fb262335 garbage_collection: log bad chunks (still_bad value) 2020-10-29 10:24:31 +01:00
Dominik Csapak 5499bd3dee fix #2998: encode mtime as i64 instead of u64
saves files mtime as i64 instead of u64 which enables backup of
files with negative mtime

the catalog_decode_i64 is compatible to encoded u64 values (if < 2^63)
but not reverse, so all "old" catalogs can be read with the new
decoder, but catalogs that contain negative mtimes will decode wrongly
on older clients

also remove the arbitrary maximum value of 2^63 - 1 for
encode_u64 (we just use up to 10 bytes now) and correctly
decode them and update the comments accordingly

adds also test for i64 encode/decode and for compatibility between
u64 encode and i64 decode

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-29 08:51:10 +01:00
Dietmar Maurer d771a608f5 verify: directly pass manifest to filter function
In order to avoid loading the manifest twice during verify.
2020-10-29 07:59:19 +01:00
Thomas Lamprecht f9beae9cc9 client: adapt to change datastroe status return schema
fixes commit 16f9f244cf which extended
the return schema of the status API but did not adapted the client
status command to that.

Simply define our own tiny return schema and use that.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-28 22:59:40 +01:00
Thomas Lamprecht eef18365e8 tools: socket: fix typo in comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-28 21:26:11 +01:00
Dominik Csapak 8b4f4d9ee4 tools/logrotate: fix compression logic
we never actually compressed any files, since we only looked at
the extension:
* if it was 'zst' (which was always true for newly rotated files), we
  would not compress it
* even if it was not 'zst', we compressed it inplace, never adding '.zst'
  (possibly compressing them multiple times as zstd)

now we add new rotated files simply as '.X' and add a 'target' to the
compress fn, where we rename it to (but now we have to unlink the source
path)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-28 18:50:16 +01:00
Thomas Lamprecht b4b14dc16e do_verification_job: fix "never-reverify" and refactor/comment
commit a4915dfc2b made a wrong fix, as
it did not observed that the last expressions was done under the
invariant that we had a last verification result, because if none
could be loaded we already returned true (include).

It thus broke the case for "never re-verify", which is important when
using multiple schedules, a more high frequent one for new,
unverified snapshots, and a low frequency to re-verify older snapshots,
e.g., monthly.

Fix this case again, rework the code to avoid this easy to oversee
invariant. Use a nested match to better express the implication of
each setting, and add some comments.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-28 16:12:09 +01:00
Thomas Lamprecht c4a45ec744 document verify job structs
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-28 15:32:28 +01:00
Thomas Lamprecht 5428f5ca29 do verification: always verify if manifest load fails
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-28 14:11:44 +01:00
Dietmar Maurer 328df3b507 verify: avoid generics and use &dyn Fn() for filter 2020-10-28 13:19:21 +01:00
Dietmar Maurer a4915dfc2b verify: improve code reuse, fix filter function
Try to reuse verify_all_backups(), because this function has better
logging and well defined snaphot order.
2020-10-28 12:58:15 +01:00
Dietmar Maurer d642802d8c jobstate: fix doctest 2020-10-28 10:52:16 +01:00
Dietmar Maurer a20fcab060 fix compile warning 2020-10-28 10:47:30 +01:00
Dietmar Maurer b9e7bcc272 send notification mails for GC and verify jobs 2020-10-28 10:44:23 +01:00
Dietmar Maurer acc3d9df5a src/server/verify_job.rs: add missing file 2020-10-28 07:58:07 +01:00
Dietmar Maurer 1298618a83 move jobstate to server 2020-10-28 07:37:01 +01:00
Dominik Csapak d58e6313e1 api/{verify, syncjobs}: add optional datastore parameter
to limit the lists to the given datastores

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Dominik Csapak 16f9f244cf admin/datastore: add more info to status call
add also the snapshot counts as well as the status of the last garbage
collection

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Dominik Csapak b683fd589c backup/datastore: save garbage collection status to disk
and load it again when opening it

this way we can persist the status of the last garbage collect across
daemon reloads and reboots

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Dominik Csapak a2285525be backup/datastore: count still bad chunks for the status
we want to show the user that there are still bad chunks after a garbage
collection

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht f23497b088 apt auth: add newline to the end
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht b57b3c9bfc hack: workaround unused code warning until proxmox-api-macro bump
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht 72c0e102ff tools: get_hardware_address: better error handling
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht 7b22fb257f implement subscription handling and api
mostly modelled after PVE

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht 2e201e7da6 tools: http: add simple general post method
This is intended for when the server needs to do requests on
arbitrary, non PBS, external HTTP resources.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht ee89416319 api: disks: cleanup use statement
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht 52fe9e8ece get_hardware_address: must be uppercased
we're a bit strict here what we accept, rather than changing that
lets do it like PVE/PMG and uppercase.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-26 20:18:28 +01:00
Thomas Lamprecht eed1bae554 api: add world accessible ping dummy endpoint
This is indented to be used for the PVE storage library, replacing
the missuse of the much more expensive status API call.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-24 19:12:14 +02:00
Fabian Grünbichler 6eb41487ce apt: improve error messages
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-22 17:13:26 +02:00
Stefan Reiter 9e61c01ce4 apt: add /changelog API call similar to PVE
For proxmox packages it works the same way as PVE, by retrieving the
changelog URL and issuing a HTTP GET to it, forwarding the output to the
client. As this is only supposed to be a workaround removed in the
future, a simple block_on is used to avoid async.

For debian packages we can simply call 'apt-get changelog' and forward
it's output.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter 91c9b42da3 fix #2934: list to-be-installed packages in updates
As always, libapt is mocking us with complexity, but we can get the
approximate result we want by retrieving dependencies of all
to-be-updated packages and then seeing if they are missing.

If they are, we assume they will be installed.

For this, query_detailed_info is extended to allow reading details for
non-installed packages, and this is also exposed in
list_installed_apt_packages via 'all_versions_for'. This is necessary so
we can retrieve changelogs for such packages.

Note that we cannot retrieve all that information all the time, as
querying details for packages that aren't installed takes a rather long
time.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter 52d2ae48f0 apt: refactor package detail reading into function
No functional change intended.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter 1872050564 apt: use 'apt-get changelog --print-uris' in get_changelog_url
Avoids custom hardcoded logic, but can only be used for debian packages
as of now. Adds a FIXME to switch over to use --print-uris only once our
package repos support that changelog format.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:26 +02:00
Stefan Reiter efeb92efee apt: allow filter to select different package version
To get package details for a specific version instead of only the
candidate.

Also cleanup filter function with extra struct instead of unnamed &str
parameters.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:15 +02:00
Dietmar Maurer 4ebda996e5 upid: use systemd escape to decode/encode the worker_id
This way we can store values containing "/" and ":".
2020-10-22 12:24:58 +02:00
Stefan Reiter 5eb9dd0c8a add tools::http for generic HTTP GET and move HttpsConnector there
...to avoid having the tools:: module depend on api2.

The get_string function is based directly on hyper and thus relatively
simple, not supporting redirects for example.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-21 16:22:08 +02:00
Dominik Csapak dc2876f6bb tools/zip: fix doc tests
the doc code was not compiling and blocking cargo test

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-21 14:20:16 +02:00
Hannes Laimer 2ef1b6290f api proxy: remove old verification scheduling
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 133042b5d8 set a different worker_type based on what is going to be verified(snapshot, group, ds)
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 73df9c515b proxy: add scheduling for verification jobs
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 8d1beca7e8 api2: add verification admin endpoint and do_verification_job function
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 9b2bad7af0 api2: add verification job config endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 78efafc2d0 rename VERIFY_SCHEDULE_SCHEMA to VERIFICATION_SCHEDULE_SCHEMA
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Dietmar Maurer 2d3d91b1db add test for escape_unit 2020-10-21 11:31:24 +02:00
Dietmar Maurer 030c5c6d8a systemd::escape_unit - allow '.' and '_' 2020-10-21 11:31:24 +02:00
Wolfgang Bumiller 53a561a222 pass params by ref to recurse_files
gets rid of the return value and moving around of the zip
and decoder data
avoids cloning the path prefix on every recursion

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:50:25 +02:00
Wolfgang Bumiller e832860a3c whitespace fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:45:44 +02:00
Dominik Csapak 804f61432d api2/admin/datastore/pxar_file_download: download directory as zip
by using the new ZipEncoder and recursively add files to it
the zip only contains directories, normal files and hardlinks (by simply
copying the content), no symlinks, etc.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:24 +02:00
Dominik Csapak 943479f5f6 tools: add AsyncChannelWriter
similar to StdChannelWriter, but implements AsyncWrite and sends
to a tokio::sync::mpsc::Sender

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:22 +02:00
Dominik Csapak fdce52aa99 tools: add zip module
This modules contains the 'ZipEncoder' struct, which wraps an async writer,
to create a ZIP archive on the fly

To create a ZIP file, have a target that implements AsyncWrite,
give it to ZipEncoder::new, add entries via 'add_entry' and
at the end, call 'finish'

for now, this does not implement compression (uses ZIPs STORE mode), and
does not support empty directories or hardlinks (or any other special
files)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:18 +02:00
Dietmar Maurer 4e32d1c590 fix for prevoius patch: we want to copy all valid tickets 2020-10-21 08:40:04 +02:00
Dylan Whyte afef7f3bba fix #3038: check user before renewing ticket
Fixes a bug in which the userid of the ticket cache is updated,
when a user connects, but the ticket itself is not.
This means a newly connected user has a previously connected
user's ticket and thus, cannot do anything, as the client will
attempt to use the invalid ticket.

e.g. if john@pbs connected to the server first, followed by
mike@pbs, the following would be stored in the ticket cache.

{
  "localhost": {
    "mike@pbs": {
      "ticket": "PBS:john@pbs:AAAA",
      "timestamp": 1601039326,
      "token": "BBBB"
    }
  }
}

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-21 08:34:30 +02:00
Dietmar Maurer b428af9781 backup: avoid Transport endpoint is not connected error
We simply supress the error message if the finish flag is set.
2020-10-20 14:20:04 +02:00
Dietmar Maurer c8774067ee paperkey: use svg as image format to provide better scalability 2020-10-20 12:04:51 +02:00
Dietmar Maurer 23440482d4 proxmox-backup-client: use HumanByte to render snapshot size 2020-10-20 11:43:48 +02:00
Thomas Lamprecht 6f757b8458 logrotate: drop useless comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:11:36 +02:00
Thomas Lamprecht 95ade8fdb5 log rotate: move basic rotation logic into module for reuse
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht 9e870b5f39 log rotate: do NOT compress first rotation
The first rotation is normally the one still opened by one or more
processes for writing, so it must NOT be replaced, removed, ..., as
this then makes the remaining logging, until those processes are
noticed that they should reopen the logfile due to rotation, goes
into nirvana, which is far from ideal for a log.

Only rotating (renaming) is OK for this active file, as this does not
invalidates the file and keeps open FDs intact.

So start compressing with the second rotation, which should be clear
to use, as all writers must have been told to reopen the log during
the last rotation, reopen is a fast operation and normally triggered
at least day ago (at least if one did not dropped the state file
manually), so we are fine to archive that one for real.
If we plan to allow faster rotation the whole rotation+reopen should
be locked, so that we can guarantee that all writers switched over,
but this is unlikely to be needed.

Again, this is was logrotate sanely does by default since forever.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht 7827e3b93e log rotate: factor out compression in private function
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht e6ca9c3235 log rotate: do NOT overwrite file with possible writers
this is not the job of logrotate, and the real 20+ years battle
tested log rotate binary does not do so either as it's actually
pretty dangerous.

If we "replace" the file we break any logger which already opened a
new one here, e.g., a dameon starting up, and thus that writer would
log to nirvana.

It's the job of a logger to create a file if not existing, it makes
no sense to do it here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Stefan Reiter 0698f78df5 fix #2988: allow verification after finishing a snapshot
To cater to the paranoid, a new datastore-wide setting "verify-new" is
introduced. When set, a verify job will be spawned right after a new
backup is added to the store (only verifying the added snapshot).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 10:51:13 +02:00
Stefan Reiter bcc2880461 add verify_backup_dir_with_lock for callers already holding locks
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 10:49:19 +02:00
Fabian Grünbichler 115d927c15 unbreak build
and silence warning.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-20 09:07:32 +02:00
Stefan Reiter df729017b4 datastore: cleanup open and load config only once
Force consumers to use the lookup_datastore method instead of
potentially opening a datastore twice, and pass the config we have
already loaded into open_with_path, removing the need for open(1).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 07:51:05 +02:00
Stefan Reiter 455f2ad228 fix missing block_in_place for remove_backup
Commit 9070d11f4c introduced this change for other call sites,
assuming it is correct, this one was missed.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 07:48:51 +02:00
Thomas Lamprecht e4f5f59eea code/fmt cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 15:11:51 +02:00
Fabian Grünbichler 16cdb9563b completion: fix ACL path completion
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 15:06:13 +02:00
Fabian Grünbichler 02479720c0 REST: rename token to csrf_token
for easier differentiation with (future) api_token

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 14:02:19 +02:00
Dietmar Maurer 97168f920e set reasonable TCP keepalive timeout 2020-10-19 14:01:17 +02:00
Fabian Grünbichler 9809772b23 fix typos
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 14:00:38 +02:00
Fabian Grünbichler 4940012d0d fix indentation
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 14:00:26 +02:00
Thomas Lamprecht e5adbc3419 fixup worker task: add time prefix again
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 13:22:37 +02:00
Fabian Grünbichler 0c4c6a7b1c build: bump nix dependency
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 12:12:33 +02:00
Thomas Lamprecht c7e18ba08a file logger: add option to make the backup user the log file owner
and use that in ApiConfig to avoid that it is owned by root if the
proxmox-backup-api process creates it first.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 10:37:26 +02:00
Dietmar Maurer bb14d46796 http_client: set connect timeout to 10 seconds 2020-10-19 09:36:01 +02:00
Thomas Lamprecht d39d095fa4 api: access: log to separate file, reduce syslog to errors
for now log auth errors also to the syslog, on a protected (LAN
and/or firewalled) setup this should normally happen due to
missconfiguration, not tries to break in.

This reduces syslog noise *a lot*. A current full journal output from
the current boot here has 72066 lines, of which 71444 (>99% !!) are
"successful auth for user ..." messages

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht 86f3c2363c server/rest: also log user agent
allows easily to see if a request is from a browser or a proxmox-backup-client
CLI

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht 8e7e2223d8 server/rest: implement request access log
reuse the FileLogger module in append mode.
As it implements write, which is not thread safe (mutable self) and
we use it in a async context we need to serialize access using a
mutex.

Try to use the same format we do in pveproxy, namely the one which is
also used in apache or nginx by default.

Use the response extensions to pass up the userid, if we extract it
from a ticket.

The privileged and unprivileged dameons log both to the same file, to
have a unified view, and avoiding the need to handle more log files.
We avoid extra intra-process locking by reusing the fact that a write
smaller than PIPE_BUF (4k on linux) is atomic for files opened with
the 'O_APPEND' flag. For now the logged request path is not yet
guaranteed to be smaller than that, this will be improved in a future
patch.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht 081c37cccf tools file logger: fix example and comments
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:16:29 +02:00
Thomas Lamprecht c0df91f8bd tools: file logger: use option struct to control behavior
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:48:36 +02:00
Thomas Lamprecht 400c568f8e server: rest: also log the query part of URL
As it is part of the request and we do so in our other products

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:41:05 +02:00
Thomas Lamprecht 4703ba81ce server: rest: implement max URI path and query length request limits
Add a generous limit now and return the correct error (414 URI Too
Long). Otherwise we could to pretty larger GET requests, 64 KiB and
possible bigger (at 64 KiB my simple curl test failed due to
shell/curl limitations).

For now allow a 3072 characters as combined length of URI path and
query.

This is conform with the HTTP/1.1 RFCs (e.g., RFC 7231, 6.5.12 and
RFC 2616, 3.2.1) which do not specify any limits, upper or lower, but
require that all server accessible resources mus be reachable without
getting 414, which is normally fulfilled as we have various length
limits for stuff which could be in an URI, in place, e.g.:
 * user id: max. 64 chars
 * datastore: max. 32 chars

The only known problematic API endpoint is the catalog one, used in
the GUI's pxar file browser:
GET /api2/json/admin/datastore/<id>/catalog?..&filepath=<path>

The <path> is the encoded archive path, and can be arbitrary long.

But, this is a flawed design, as even without this new limit one can
easily generate archives which cannot be browsed anymore, as hyper
only accepts requests with max. 64 KiB in the URI.
So rather, we should move that to a GET-as-POST call, which has no
such limitations (and would not need to base32 encode the path).

Note: This change was inspired by adding a request access log, which
profits from such limits as we can then rely on certain atomicity
guarantees when writing requests to the log.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:40:39 +02:00
Thomas Lamprecht 29633e2fe9 server/rest: forward real client IP on proxied request
needs new proxmox dependency to get the RpcEnvironment changes,
adding client_ip getter and setter.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:36:54 +02:00