Commit Graph

4332 Commits

Author SHA1 Message Date
Wolfgang Bumiller 38774184a9 tree-wide: replace serde_json::from_value(a_value.clone())
`&Value` itself implements `Deserializer` and can therefore
be passed directly to `T::deserialize` without requiring an
intermediate `clone()`. (This also enables optionally
borrowing strings if the result has a short enough lifetime)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-12 16:12:15 +02:00
Hannes Laimer 5fd823c3f2 api: add get_active_operations endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 758c6ed588 api: make maintenance_type updatable
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 4bc84a6549 pbs-datastore: add active operations tracking
Saves the currently active read/write operation counts in a file. The
file is updated whenever a reference returned by lookup_datastore is
dropped and whenever a reference is returned by lookup_datastore. The
files are locked before every access, there is one file per datastore.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer e9d2fc9362 datastore: add check for maintenance in lookup
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Thomas Lamprecht 12558e0dde tree wide: some stylistic clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-11 08:14:28 +02:00
Thomas Lamprecht 4ad118c613 cli: backup manager: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:50:35 +02:00
Thomas Lamprecht 4de1c42c20 tape: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:49:03 +02:00
Dominik Csapak 416194d799 rest-server: add option to rotate task logs by 'max_days' instead of 'max_files'
and use it with the configurable: 'task_log_max_days' of the node config

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:12:49 +02:00
Dominik Csapak eb419c5267 config/node: add 'task_log_max_days' config
to be able to configure the maximum days to keep task logs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:10:02 +02:00
Dominik Csapak baefc29544 rest-server: cleanup_old_tasks: improve error handling
by not bubbling up most errors, and continuing on. this avoids that we
stop cleaning up because e.g. one directory was missing.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:10:02 +02:00
Wolfgang Bumiller b300e6fbc2 use BufReader/Writer for Files passed to serde_json::from_reader/writer
As serde_json will otherwise read files 1 byte at a time.
Writing is a bit better, but syntacitcal elements (quotes, braces,
commas) still often show up as single write syscalls, so use BufWriter
there as well.

Note that while we do store the file in the resulting objects, we do not
need to keep the buffered read/writers as we always `seek` to the
beginning on further file operations.

Reported-by: Mark Schouten <mark@tuxis.nl>
Link: https://lists.proxmox.com/pipermail/pbs-devel/2022-April/004909.html
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-06 16:40:35 +02:00
Thomas Lamprecht 085ae87380 api: tape: rust format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:31:49 +02:00
Thomas Lamprecht 938a1f137c cli: tape: rust format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:27:32 +02:00
Thomas Lamprecht 5525ec246f tape: key recovery: refcator and split string/file case for cli params
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:25:34 +02:00
Markus Frank b676dbce78 fix #3854 paperkey import to proxmox-tape
added a parameter to the cli for importing tape key via a json-parameter or
via reading a exported paperkey-file or json-file.
For this i also added a backupkey parameter to the api, but here it only
accepts json.

The cli interprets the parameter first as json-string, then json-file
and last as paperkey-file.

functionality:
proxmox-tape key paperkey [fingerprint of existing key] > paperkey.backup
proxmox-tape key restore --backupkey paperkey.backup # key from line above
proxmox-tape key restore --backupkey paperkey.json # only the json
proxmox-tape key restore --backupkey '{"kdf": {"Scrypt": ...' # json as string

for importing the key as paperkey-file it is irrelevant, if the paperkey got exported as html
or txt.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2022-04-06 13:39:56 +02:00
Stefan Sterz 2b422b82fb fix #3067: api: add support for multi-line comments in node.cfg
add support for multi-line comments to node.cfg and the api, similar to
how pve handles multi-line comments

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-23 10:43:43 +01:00
Dominik Csapak 39ffb75d91 api: datastore_status: restore api/gui compatibility
the latest changes to this api call changed/removed some things that
were actually necessary for the gui. Readd those and document them this
time.

The change from u64 to i64 limits us to 8EiB of Datastore sizes (instead if
16EiB) but if we reach that, we must adapt most other parts to use 128bit
sizes anyway

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-03-22 10:31:25 +01:00
Dietmar Maurer 762f7d15dc datastore status: factor out api type DataStoreStatusListItem
And use the rust type instead of json::Value.
2022-03-20 09:38:50 +01:00
Stefan Sterz 171a00ca97 tape, docs, api: fix miscellaneous typos
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-03-11 12:52:06 +01:00
Wolfgang Bumiller 787c6550d4 proxmox-backup-debug api: fewer cloning
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-09 10:10:54 +01:00
Dominik Csapak c6140c62ab proxmox-backup-debug api: rustfmt fixes
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-09 10:10:54 +01:00
Dominik Csapak 9735f5de84 proxmox-backup-debug api: parse parameters before sending to api
when we use http to make the api call, we have to parse the parameters
before, else we might send the string "true" instead of the boolean true
and the api rejects it with a 'Parameter verification error'.

We already have all api call schemas here, so parsing is possible.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-09 10:10:54 +01:00
Dominik Csapak 904ce33d9f tools: parse_objset_stat: drop the unecessary 'objset-' from the log
'objset_id' already contains that, so the error was
"could not parse 'objset-objset-0xFFFF' stat file"

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-08 09:13:05 +01:00
Dominik Csapak 6dd5944772 tools: zfs_dataset_stats: remove dataset <-> obset file mapping on error
this can only real fail for two reasons:
* the format is wrong:
    this should not happen unless the format changed, then it will
    happen every time
* the file can't be read:
    this can happen if a user deletes and recreates a dataset manually,
    since the mapped file does not exist anymore but the dataset does

for the second case, delete the mapping from the hashmap, so that the
next call will refresh the mapping with the correct file

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-08 09:12:52 +01:00
Dominik Csapak dcd1518e10 api/config: use http_bail for 'not found' errors
the api should return a 404 error for entries that do not exist

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-08 09:09:25 +01:00
Dominik Csapak 8d6425aa24 api/config: use param_bail for parameter errors
when using the 'extjs' formatter, it marks them in a way, so that
the gui can mark the form fields with the error

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-08 09:09:22 +01:00
Dietmar Maurer 1c8efc0062 cleanup: move BasicRealmInfo to pbs-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-03-07 08:06:55 +01:00
Dominik Csapak fef61684b4 datastore: add tuning option for chunk order
currently, we sort chunks by inode when verifying or backing up to tape.
we get the inode# by stat'ing each chunk, which may be more expensive
than the gains of reading the chunks in order

Since that is highly dependent on the underlying storage of the datastore,
introduce a tuning option  so that the admin can tune that behaviour
for each datastore.

The default stays the same (sorting by inode)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-02-23 09:06:03 +01:00
Thomas Lamprecht c2f84841b6 bin: daily-update: use syslog/log crates instead of printing to stderr
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-22 10:58:44 +01:00
Thomas Lamprecht b0728103b6 bin: daily-update: make single checks/updates fail gracefully
avoid that the acme renewal is skipped due to bailing out earlier
from a subscription or apt update error.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-22 10:27:00 +01:00
Thomas Lamprecht 00d41438b9 bin: daily-update: use from_millis instead of big nanosecond value
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-22 10:25:40 +01:00
Thomas Lamprecht 50654b22df bin: daily-update: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-21 15:52:28 +01:00
Dominik Csapak f26d7ca5c5 use io_format_err, io_bail, io_err_other from proxmox-lang
and move the comment from the local io_bail in pbs-client/src/pxar/fuse.rs
to the only use

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-02-21 14:24:13 +01:00
Dominik Csapak dcd9c17fff tape/pool_writer: skip already backed up chunks in iterator
currently, the iterator goes over *all* chunks of the index, even
those already backed up by a previous snapshots in the same tape
backup. this is bad since for each iterator, we stat each chunk to
sort by inode number. so to avoid stat'ing the same chunks over
and over for consecutive snapshots, add a 'skip_fn' to the iterator
and in the pool writer and check the catalog_set if we can skip it

this means we can drop the later check for the catalog_set
(since we don't modify that here)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-02-18 10:40:41 +01:00
Dietmar Maurer d6644e29fe move src/shared_rate_limiter.rs to src/tools/shared_rate_limiter.rs
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-02-14 14:57:56 +01:00
Dietmar Maurer 260147bd73 ParallelHandler: avoid re-export (cleanup)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-02-14 14:12:39 +01:00
Dietmar Maurer e705b3057f rename cached_traffic_control.rs to traffic_control_cache.rs, improve dev docs
Keep things inside crate::traffic_control_cache (do not pollute root namespace).

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-02-14 13:45:44 +01:00
Dietmar Maurer 192ece47fb rrd_cache: add developer docs
and make RRD_CACHE private (please use get_rrd_cache instead).

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2022-02-14 12:07:10 +01:00
Stefan Sterz 41adda1c64 fix #3853: tape cli: add force flag to key change-passphrase
Adds the '--force' flag to the proxmox-tape command allowing users
with root privileges to overwrite the passphrase of a given key.

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2022-02-14 09:52:20 +01:00
Stefan Sterz 77d6d7a22c fix #3853: api: add force option to tape key change-passphrase
When force is used, the current passphrase is not required. Instead
it will be read from the file pointed to by TAPE_KEYS_FILENAME and
the old key configuration will be overwritten using the new
passphrase. Requires super user privileges.

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2022-02-14 09:52:20 +01:00
Thomas Lamprecht af6fdb9d0d tools: disk: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-10 18:39:56 +01:00
Thomas Lamprecht a1c906cb02 api: node/disk: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-10 13:12:15 +01:00
Fabian Grünbichler dcf5a0f62d misc clippy fixes
the trivial ones ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-02-08 14:57:16 +01:00
Aaron Lauterer bb9e503964 report: add tape, traffic control and disk infos
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-02-07 15:37:06 +01:00
Aaron Lauterer b2fc573a62 report: move subscription info further up
This is something that is checked all the time. Having it further up
saves on scrolling and brings it into better alignment with PVE & PMG
regarding where in the report the info is located.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2022-02-07 15:37:06 +01:00
Matthias Heiserer 415da09826 node config: add english to translation enum for default-lang
Signed-off-by: Matthias Heiserer <m.heiserer@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-07 15:24:38 +01:00
Matthias Heiserer 5ffa68d2c4 api: node config: add default-lang integration
Signed-off-by: Matthias Heiserer <m.heiserer@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-07 15:24:38 +01:00
Dominik Csapak 7b944ff11a re-use PROXMOX_DEBUG env variable to control log level filter
So that we can make 'log::debug' messages actually appear in the
syslog.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-02-04 11:21:47 +01:00
Markus Frank fce49eab30 fix #3856 hint parameter is not optional
For the API the parameter --hint is not optional. This patch fixes
the man page and cli command doesn't send an API call, if the
parameter does not exist.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2022-02-03 14:49:25 +01:00