Commit Graph

6461 Commits

Author SHA1 Message Date
Dominik Csapak 2ef2c0fe0c proxmox-rest-server: OutputFormatter: add new format_data_streaming method
that takes the data in form of a `Box<dyn SerializableReturn + Send>`
instead of a Value.

Implement it in json and extjs formatter, by starting a thread and
stream the serialized data via a `BufWriter<SenderWriter>` and use
the Receiver side as a stream for the response body.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:13:36 +02:00
Thomas Lamprecht 9c3b29bd8f ui: datastore options: maintenance mode related refactorings
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:54:56 +02:00
Thomas Lamprecht 3c8f240712 ui: datastore options: fix active-ops-tracking store leak
without this the store stayed active in the background and kept
updating every 3s for every datastore the ui was opened.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:21:41 +02:00
Thomas Lamprecht 6353e22c00 ui: datastore options: factor out update stop/start to controller
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:18:43 +02:00
Wolfgang Bumiller 38774184a9 tree-wide: replace serde_json::from_value(a_value.clone())
`&Value` itself implements `Deserializer` and can therefore
be passed directly to `T::deserialize` without requiring an
intermediate `clone()`. (This also enables optionally
borrowing strings if the result has a short enough lifetime)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-12 16:12:15 +02:00
Thomas Lamprecht 845baef61b ui: maintenance mode: also render message
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:12:15 +02:00
Thomas Lamprecht 73ce2ae1c7 ui: maintenance mode: refactor renderer
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:12:12 +02:00
Hannes Laimer 556eda0537 ui: add option to change the maintenance type
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 5fd823c3f2 api: add get_active_operations endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 758c6ed588 api: make maintenance_type updatable
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 4bc84a6549 pbs-datastore: add active operations tracking
Saves the currently active read/write operation counts in a file. The
file is updated whenever a reference returned by lookup_datastore is
dropped and whenever a reference is returned by lookup_datastore. The
files are locked before every access, there is one file per datastore.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer e9d2fc9362 datastore: add check for maintenance in lookup
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 2a05c75ff1 api-types: add maintenance type
+ bump proxmox-schema dep to 1.2.1 (for quoted property string)

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Thomas Lamprecht 66b88dadba ui: node config: avoid split listeners
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 15:29:03 +02:00
Wolfgang Bumiller 9ee2ef2e55 client: drop unnecessary clone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-12 12:34:52 +02:00
Thomas Lamprecht 12558e0dde tree wide: some stylistic clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-11 08:14:28 +02:00
Thomas Lamprecht b22d785c18 api types: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 18:00:18 +02:00
Thomas Lamprecht 4ad118c613 cli: backup manager: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:50:35 +02:00
Thomas Lamprecht 6082d75966 tests: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:49:26 +02:00
Thomas Lamprecht 4de1c42c20 tape: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:49:03 +02:00
Thomas Lamprecht 429bc9d0a2 restore daemon: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:47:20 +02:00
Thomas Lamprecht a22d338831 examples: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:44:34 +02:00
Thomas Lamprecht 1e724828b4 rest server: log rotation: refactor and comment improvements
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-07 14:04:18 +02:00
Thomas Lamprecht 40853461d1 rest server: log rotation: fix off-by-one for max_days
The entries in a file go from oldest end-time in the first time to
newest end-time in the last line. So, just because the first line is
older than the cut-off time, the remaining one doesn't necessarily
have to be old enough too. What we can know for sure that older than
the current checked rotations of the task archive are definitively up
for deletion.

Another possibility would be to check the last line, but as scanning
backwards is more expensive/complex to do while only being an actual
improvement in a very specific edge case (it's more likely to have a
mixed time-cutoff vs. task-log-file boundary than that those are
aligned)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-07 12:58:32 +02:00
Dominik Csapak 416194d799 rest-server: add option to rotate task logs by 'max_days' instead of 'max_files'
and use it with the configurable: 'task_log_max_days' of the node config

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:12:49 +02:00
Dominik Csapak eb419c5267 config/node: add 'task_log_max_days' config
to be able to configure the maximum days to keep task logs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:10:02 +02:00
Dominik Csapak baefc29544 rest-server: cleanup_old_tasks: improve error handling
by not bubbling up most errors, and continuing on. this avoids that we
stop cleaning up because e.g. one directory was missing.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:10:02 +02:00
Thomas Lamprecht b23adfd4ee pbs tape: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 17:00:29 +02:00
Thomas Lamprecht a527b54f84 pbs fuse loop: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:59:54 +02:00
Fabian Ebner b2df21bb02 docs: client: file exclusion: add note about leading slash
It's not documented yet and not intuitive:
https://forum.proxmox.com/threads/98810
https://forum.proxmox.com/threads/107143

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-06 16:59:00 +02:00
Fabian Ebner 2b323a359d pxar: create: add entry: fix anchored path pattern matching
Similar to 874bd545 ("pxar: fix anchored exclusion at archive root"),
but this time for inclusion. Because of the inconsistency, it could
happen that a file included in generate_directory_file_list() got
excluded in add_entry(), e.g. with a .pxarexclude file like
> *
> !/supposed-to-be-included

Reported-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-06 16:58:08 +02:00
Thomas Lamprecht 48fcee6a50 pxar bin: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:58:04 +02:00
Thomas Lamprecht c650378a39 pbs build config: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:57:36 +02:00
Thomas Lamprecht 40ea990c05 file restore: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:57:07 +02:00
Thomas Lamprecht aaaa10894d rrd: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:56:33 +02:00
Thomas Lamprecht 41583796b1 rest server: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:55:39 +02:00
Wolfgang Bumiller b300e6fbc2 use BufReader/Writer for Files passed to serde_json::from_reader/writer
As serde_json will otherwise read files 1 byte at a time.
Writing is a bit better, but syntacitcal elements (quotes, braces,
commas) still often show up as single write syscalls, so use BufWriter
there as well.

Note that while we do store the file in the resulting objects, we do not
need to keep the buffered read/writers as we always `seek` to the
beginning on further file operations.

Reported-by: Mark Schouten <mark@tuxis.nl>
Link: https://lists.proxmox.com/pipermail/pbs-devel/2022-April/004909.html
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-06 16:40:35 +02:00
Thomas Lamprecht 085ae87380 api: tape: rust format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:31:49 +02:00
Thomas Lamprecht 938a1f137c cli: tape: rust format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:27:32 +02:00
Thomas Lamprecht 5525ec246f tape: key recovery: refcator and split string/file case for cli params
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:25:34 +02:00
Markus Frank b676dbce78 fix #3854 paperkey import to proxmox-tape
added a parameter to the cli for importing tape key via a json-parameter or
via reading a exported paperkey-file or json-file.
For this i also added a backupkey parameter to the api, but here it only
accepts json.

The cli interprets the parameter first as json-string, then json-file
and last as paperkey-file.

functionality:
proxmox-tape key paperkey [fingerprint of existing key] > paperkey.backup
proxmox-tape key restore --backupkey paperkey.backup # key from line above
proxmox-tape key restore --backupkey paperkey.json # only the json
proxmox-tape key restore --backupkey '{"kdf": {"Scrypt": ...' # json as string

for importing the key as paperkey-file it is irrelevant, if the paperkey got exported as html
or txt.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2022-04-06 13:39:56 +02:00
Dominik Csapak 7c22932c64 pbs-client: print error when we couldn't download previous fidx/didx
When we have a previous manifest, we try to download the fidx/didx files
to get the known chunks list. We continue if that fails (which is ok),
but we did not print any error, leading to a confusing backup output,
since the users would expect that chunks will be reused.

Printing the error should at least make it apparent that something did
not work correctly.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 10:12:29 +02:00
Stefan Sterz 2b422b82fb fix #3067: api: add support for multi-line comments in node.cfg
add support for multi-line comments to node.cfg and the api, similar to
how pve handles multi-line comments

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Acked-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-23 10:43:43 +01:00
Wolfgang Bumiller 9e2b423e27 tools: improve PhantomData usage
The ticket doesn't contain a `T`, it's stringified. We only
produce a new T when verifying.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-22 12:41:14 +01:00
Dominik Csapak 39ffb75d91 api: datastore_status: restore api/gui compatibility
the latest changes to this api call changed/removed some things that
were actually necessary for the gui. Readd those and document them this
time.

The change from u64 to i64 limits us to 8EiB of Datastore sizes (instead if
16EiB) but if we reach that, we must adapt most other parts to use 128bit
sizes anyway

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-03-22 10:31:25 +01:00
Dietmar Maurer 762f7d15dc datastore status: factor out api type DataStoreStatusListItem
And use the rust type instead of json::Value.
2022-03-20 09:38:50 +01:00
Markus Frank 80ab05e40c fix #3934 tape owner-selector to Authid
changed pmxUserSelector to pbsAuthidSelector, because it is currently
not possible to restore with a api token via gui.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2022-03-17 10:13:18 +01:00
Stefan Sterz e099bd0717 ui: fix panel height in the dashboard
this fixes an issue where the layout looks misaligned in three column
layouts

Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
2022-03-11 12:52:50 +01:00
Stefan Sterz 171a00ca97 tape, docs, api: fix miscellaneous typos
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-03-11 12:52:06 +01:00
Wolfgang Bumiller c8322f8a33 config: don't manually track padding size
make ConfigVersionCacheData a #[repr(C)] union to fix its
size and let it transparently `Deref{,Mut}` to its actual
contents

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-03-10 10:32:46 +01:00