Commit Graph

6380 Commits

Author SHA1 Message Date
Thomas Lamprecht 42c2b5bec9 datastore: rustfmt whole package
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 13:27:53 +02:00
Thomas Lamprecht fb3c007f8a d/changelog: fixup released
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-14 09:48:00 +02:00
Thomas Lamprecht ff7568f1d9 bump version to 2.1.6-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-13 17:31:21 +02:00
Thomas Lamprecht 1fd46218ea cli: tape key-restore: print more info for better ux
as getting the marker error if we passed valid json (but not valid
key) is confusing

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-13 16:59:47 +02:00
Thomas Lamprecht ede9dc0d1a api: tape key restore: fix optional param handling and code refactoring
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-13 16:59:47 +02:00
Markus Frank ae60eed310 proxmox-tape: api: restore_key-code moved to tape-encryption-keys
The restore_key api-endpoint is tape/drive/{drive}/restore-key.
Since I cannot set the url parameter for the drivename to null or
undefined, when restoring by exported-key, I moved the
added restore_key-api-code to
"create_key aka POST api2/json/config/tape-encryption-keys" and
added an ApiHandler call in the cli's "restore_key" to call
"create_key" in the api.

Signed-off-by: Markus Frank <m.frank@proxmox.com>
2022-04-13 16:31:17 +02:00
Wolfgang Bumiller e3746a329e pbs-client: pxar: avoid some more clones
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:30:40 +02:00
Wolfgang Bumiller 7546e9c997 pbs-client: pxar: avoid some vec extensions
The `Components` `Iterator` has an `as_path()` method to get
the remainder as a borrowed path. This is more efficient
iterating and joining the components into a new `PathBuf`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:28:01 +02:00
Wolfgang Bumiller 0bb4036f25 pbs-client: pxar: drop link_to_pathbuf
pxar's Hardlink and Symlink structs implement `AsRef<OsStr>`
and have an `.as_os_str()` method.

Simply use `Path::new(link)`.

Also, the function was not very well written, and we don't
always need an owned copy.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:18:31 +02:00
Wolfgang Bumiller 84d3af3a0e pbs-client: pxar: fmt
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:17:20 +02:00
Dominik Csapak 055eab54ff ui: datastore/Content: enable tar download in ui
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:08:34 +02:00
Dominik Csapak 984ddb2ff2 api: admin/datastore: add tar support for pxar_file_download
by using the newly added 'create_tar' and the 'ZstdEncoder'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:08:26 +02:00
Dominik Csapak 23af572d3f pbs-client: add 'create_tar' helper function
similar to create_zip, uses an accessor to write a tar into an output
that implements AsyncWrite, but we use a Decoder to iterate instead
of having a recursive function. This is done so that we get the
entries in the correct order, and it should be faster as well.

Includes files, directories, symlinks, hardlink, block/char devs, fifos
into the tar. If the hardlink points to outside the current dir to
archive, promote the first instance to a 'real' file, and use a
hardlink for the rest.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 10:05:08 +02:00
Wolfgang Bumiller 99f09fd3c1 bump proxmox-compression dependency to 0.1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 09:37:20 +02:00
Wolfgang Bumiller da7a71115c bump d/control
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:21:18 +02:00
Wolfgang Bumiller ebb85c1ca3 bump proxmox-schema dependency to 1.3.1 for streaming attribute
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:20:27 +02:00
Wolfgang Bumiller fb6e48f402 bump proxmox-router dependency to 1.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:17:08 +02:00
Dominik Csapak b7c3eaa981 api: admin/datastore: enable streaming for some api calls
namely /admin/datastore/{store}/snapshots
and /nodes/{node}/tasks

since those are api calls where the result can get quite large
with this change, the serialization is now streaming instead of making
a `Value` in memory.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:13:42 +02:00
Dominik Csapak 32e2b5abe6 adapt to the new ApiHandler variants
namely 'StreamingSync' and 'StreamingAsync'
in rest-server by using the new formatter function,
and in the debug binary by using 'to_value'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:13:40 +02:00
Dominik Csapak 2ef2c0fe0c proxmox-rest-server: OutputFormatter: add new format_data_streaming method
that takes the data in form of a `Box<dyn SerializableReturn + Send>`
instead of a Value.

Implement it in json and extjs formatter, by starting a thread and
stream the serialized data via a `BufWriter<SenderWriter>` and use
the Receiver side as a stream for the response body.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-13 08:13:36 +02:00
Thomas Lamprecht 9c3b29bd8f ui: datastore options: maintenance mode related refactorings
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:54:56 +02:00
Thomas Lamprecht 3c8f240712 ui: datastore options: fix active-ops-tracking store leak
without this the store stayed active in the background and kept
updating every 3s for every datastore the ui was opened.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:21:41 +02:00
Thomas Lamprecht 6353e22c00 ui: datastore options: factor out update stop/start to controller
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:18:43 +02:00
Wolfgang Bumiller 38774184a9 tree-wide: replace serde_json::from_value(a_value.clone())
`&Value` itself implements `Deserializer` and can therefore
be passed directly to `T::deserialize` without requiring an
intermediate `clone()`. (This also enables optionally
borrowing strings if the result has a short enough lifetime)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-12 16:12:15 +02:00
Thomas Lamprecht 845baef61b ui: maintenance mode: also render message
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:12:15 +02:00
Thomas Lamprecht 73ce2ae1c7 ui: maintenance mode: refactor renderer
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 16:12:12 +02:00
Hannes Laimer 556eda0537 ui: add option to change the maintenance type
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 5fd823c3f2 api: add get_active_operations endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 758c6ed588 api: make maintenance_type updatable
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 4bc84a6549 pbs-datastore: add active operations tracking
Saves the currently active read/write operation counts in a file. The
file is updated whenever a reference returned by lookup_datastore is
dropped and whenever a reference is returned by lookup_datastore. The
files are locked before every access, there is one file per datastore.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer e9d2fc9362 datastore: add check for maintenance in lookup
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Hannes Laimer 2a05c75ff1 api-types: add maintenance type
+ bump proxmox-schema dep to 1.2.1 (for quoted property string)

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-04-12 15:29:14 +02:00
Thomas Lamprecht 66b88dadba ui: node config: avoid split listeners
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-12 15:29:03 +02:00
Wolfgang Bumiller 9ee2ef2e55 client: drop unnecessary clone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-04-12 12:34:52 +02:00
Thomas Lamprecht 12558e0dde tree wide: some stylistic clippy fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-11 08:14:28 +02:00
Thomas Lamprecht b22d785c18 api types: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 18:00:18 +02:00
Thomas Lamprecht 4ad118c613 cli: backup manager: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:50:35 +02:00
Thomas Lamprecht 6082d75966 tests: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:49:26 +02:00
Thomas Lamprecht 4de1c42c20 tape: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:49:03 +02:00
Thomas Lamprecht 429bc9d0a2 restore daemon: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:47:20 +02:00
Thomas Lamprecht a22d338831 examples: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-10 17:44:34 +02:00
Thomas Lamprecht 1e724828b4 rest server: log rotation: refactor and comment improvements
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-07 14:04:18 +02:00
Thomas Lamprecht 40853461d1 rest server: log rotation: fix off-by-one for max_days
The entries in a file go from oldest end-time in the first time to
newest end-time in the last line. So, just because the first line is
older than the cut-off time, the remaining one doesn't necessarily
have to be old enough too. What we can know for sure that older than
the current checked rotations of the task archive are definitively up
for deletion.

Another possibility would be to check the last line, but as scanning
backwards is more expensive/complex to do while only being an actual
improvement in a very specific edge case (it's more likely to have a
mixed time-cutoff vs. task-log-file boundary than that those are
aligned)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-07 12:58:32 +02:00
Dominik Csapak 416194d799 rest-server: add option to rotate task logs by 'max_days' instead of 'max_files'
and use it with the configurable: 'task_log_max_days' of the node config

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:12:49 +02:00
Dominik Csapak eb419c5267 config/node: add 'task_log_max_days' config
to be able to configure the maximum days to keep task logs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:10:02 +02:00
Dominik Csapak baefc29544 rest-server: cleanup_old_tasks: improve error handling
by not bubbling up most errors, and continuing on. this avoids that we
stop cleaning up because e.g. one directory was missing.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-04-06 17:10:02 +02:00
Thomas Lamprecht b23adfd4ee pbs tape: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 17:00:29 +02:00
Thomas Lamprecht a527b54f84 pbs fuse loop: rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-04-06 16:59:54 +02:00
Fabian Ebner b2df21bb02 docs: client: file exclusion: add note about leading slash
It's not documented yet and not intuitive:
https://forum.proxmox.com/threads/98810
https://forum.proxmox.com/threads/107143

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-06 16:59:00 +02:00
Fabian Ebner 2b323a359d pxar: create: add entry: fix anchored path pattern matching
Similar to 874bd545 ("pxar: fix anchored exclusion at archive root"),
but this time for inclusion. Because of the inconsistency, it could
happen that a file included in generate_directory_file_list() got
excluded in add_entry(), e.g. with a .pxarexclude file like
> *
> !/supposed-to-be-included

Reported-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-04-06 16:58:08 +02:00