Commit Graph

6472 Commits

Author SHA1 Message Date
Dominik Csapak 6b61d319c5 api: tape/restore: add optional namespace map to DataStoreMap
and change the interface from 'get_datastore' to 'get_targets'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 14:08:32 +02:00
Dominik Csapak be97e0a55b tape: add namespaces mapping type
and the relevant parser for it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 14:08:32 +02:00
Dominik Csapak 999293bbca tape: add namespaces/recursion depth to tape backup jobs
and manual api via TapeBackupJobSetup

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 14:08:31 +02:00
Dominik Csapak 9c65e6ab4a tape: fix snapshot path in catalog and snapshot_archive
both used the 'Display' trait of pbs_datastore::BackupDir, which is not
intended to be serialized anywhere. Instead, manually format the path
using the print_ns_and_snapshot helper, and conversely, parse with
'parse_ns_and_snapshot'. to be a bit safer, change the register_snapshot
signature to take a BackupNamespace and BackupDir instead of a string.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 13:52:50 +02:00
Dominik Csapak 1e4e1514d3 pbs-api-types: add parse and print ns_and_snapshot
these are helpers for the few cases where we want to print and parse
from a format that has the namespace and snapshot combined, like for
the on-tape catalog and snapshot archive.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 13:52:50 +02:00
Dominik Csapak 05b7175a56 tape: notify when arriving at end of media
when continuing a media set, we first move to the end of the tape and
start with the next (chunk) archive. If that takes long, the task logs
last line is 'moving to end of media' even if we already startet
writing. To make this less confusing, log that we arrived at the
end of the media.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-13 13:52:50 +02:00
Wolfgang Bumiller bc21ade293 tree-wide: rename 'backup-ns' API parameters to 'ns'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-13 13:46:13 +02:00
Thomas Lamprecht f07e660153 ui: move max NS prefix length logic to reduced max-depth selector
for better re-usability in the future and it felt a bit odd to have
such specific logic in the sync job edit directly

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-13 13:15:02 +02:00
Thomas Lamprecht addcb7803e datastore: inline some format variables
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-13 12:42:41 +02:00
Thomas Lamprecht 1ddfae5499 api types: set NS_MAX_DEPTH schema default to MAX_NAMESPACE_DEPTH
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-13 12:32:25 +02:00
Thomas Lamprecht 54d315c951 ui: group filter: make also local filter NS aware
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-13 12:32:25 +02:00
Fabian Grünbichler 9dde8cd625 ui: sync: add reduced max-depth selector
that allows setting the limit based on sync namespace prefix lengths.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-13 12:20:29 +02:00
Fabian Grünbichler 87be232d1c pull/sync: clamp (local) max-depth if unset
to handle the unlikely case of `ns` being deeper than `remote-ns`,
`max-depth` being set to `None` and a too-deep sub-ns of `ns` existing.
such a sub-ns cannot have been created by a previous run of this sync
job, so avoid unexpectedly removing it.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-13 12:07:22 +02:00
Fabian Grünbichler e40c7fb906 api: split max-depth schema/types
into the regular one (with default == MAX) and the one used for
pull/sync, where the default is 'None' which actually means the remote
end reduces the scope of sync automatically (or, if needed,
backwards-compat mode without any remote namespaces at all).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-13 12:07:22 +02:00
Fabian Grünbichler 66abc4cb7d namespaces: move max-depth check to api type
and use it when creating a sync job, and simplify the check on updating
(only check the final, resulting config instead of each intermediate
version).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-13 12:07:22 +02:00
Fabian Grünbichler 11567dfbad pull/sync: correctly query with remote-ns as parent
else (grand)-parents and siblings/cousins of remote-ns are also
included, and mapping the remote-ns prefix fails.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-13 12:07:22 +02:00
Fabian Grünbichler 7a3e777ded pull/sync: detect remote lack of namespace support
and fall back to only syncing the root namespace, if possible. the sync
job will still be marked as failed to prompt the admin to resolve the
situation:
- explicitly mark the job as syncing *only* the root namespace
- or upgrade remote end to support namespaces

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-12 17:00:38 +02:00
Fabian Grünbichler b9310489cf pull/sync: treat unset max-depth as full recursion
to be consistent with tape backup and verification jobs.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-12 17:00:38 +02:00
Fabian Grünbichler d9aad37f2f pull: pass params as non-ref in pull_store
so that it's possible to modify them in-place without cloning.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2022-05-12 16:26:26 +02:00
Thomas Lamprecht 2a088b9975 datastore: drop bogus chunk size check, can cause trouble
other sizes can happen in legitimate and illegitimate ways:
 - illegitimate: encryped chunks and bad actor client
 - legitimate: same chunk but newer zstd version (or compression
   level) can compress it better (or worse) so the

Ideally we could take the actual smaller chunk so that improved zstd
tech gets leveraged, but we could only allow to do that for
un-encrypted ones.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 15:41:20 +02:00
Thomas Lamprecht 78e1ee5230 bump version to 2.1.9-2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 14:28:16 +02:00
Dominik Csapak c7d42dac97 ui: navigation tree: fix losing datastore selection on store load
instead of using 'replaceChild', simply set the appropriate
properties. When using the 'nodeUpdate' (protected function of extjs,
intended to be overwritten) instead of the private 'updateNode', it
will be called when the properties change

This way, the treenode stays the same and it can keep the selection

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2022-05-12 14:28:16 +02:00
Wolfgang Bumiller 8ca7cccf5f file-restore: add namespace support to qemu part
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-12 13:35:34 +02:00
Thomas Lamprecht e30a2e9058 ui: content: fix various tree-checks from action handlers
they all still used some odd side effects of the tree structure to
decided what record type they operated on, just move them over to the
new `ty` record.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 13:28:20 +02:00
Thomas Lamprecht 08982a3746 rest: example: fix comment width
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:57:51 +02:00
Thomas Lamprecht 42fb291c7c cargo fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:54:21 +02:00
Fabian Ebner e9b9f33aee rest server: daemon: update PID file before sending MAINPID notification
There is a race upon reload, where it can happen that:
1. systemd forks off /bin/kill -HUP $MAINPID
2. Current instance forks off new one and notifies systemd with the
   new MAINPID.
3. systemd sets new MAINPID.
4. systemd receives SIGCHLD for the kill process (which is the current
   control process for the service) and reads the PID of the old
   instance from the PID file, resetting MAINPID to the PID of the old
   instance.
5. Old instance exits.
6. systemd receives SIGCHLD for the old instance, reads the PID of the
   old instance from the PID file once more. systemd sees that the
   MAINPID matches the child PID and considers the service exited.
7. systemd receivese notification from the new PID and is confused.
   The service won't get active, because the notification wasn't
   handled.

To fix it, update the PID file before sending the MAINPID
notification, similar to what a comment in systemd's
src/core/service.c suggests:
> /* Forking services may occasionally move to a new PID.
>  * As long as they update the PID file before exiting the old
>  * PID, they're fine. */
but for our Type=notify "before sending the notification" rather than
"before exiting", because otherwise, the mix-up in 4. could still
happen (although it might not actually be problematic without the
mix-up in 6., it still seems better to avoid).

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2022-05-12 11:53:54 +02:00
Thomas Lamprecht f4d246072d ui: avoid ascending to upper NS on double click of current
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:48:27 +02:00
Thomas Lamprecht 15808a9023 ui: add namespace: preselect current NS as parent for new one
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:47:14 +02:00
Thomas Lamprecht e22ad28302 GC scheduling: avoid triggering operation tracking error for upfront checks
without that one gets a "failed to lookup datastore X" in the log for
every datastore that is in read-only or offline maintenance mode,
even if they aren't scheduled for GC anyway.

Avoid that by first opening the datastore through a Lookup operation,
and only re-open it as Write op once we know that GC needs to get
scheduled for it.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:36:56 +02:00
Thomas Lamprecht 0408f60b58 datastore: add new Lookup for operations tracking
We sometimes need to do some in-memory only stuff, e.g., to check if
GC is already running for a datastore, which is a try_lock on a mutex
that is in-memory.

Actually the whole thing would be nicer if we could guarantee to hold
the correct contract statically, e.g., like
https://docs.rust-embedded.org/book/static-guarantees/design-contracts.html

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 11:36:56 +02:00
Hannes Laimer d4d730e589 proxy: rrd: skip update disk stats for offline datastores
RDD update did not use lookup_datastore() and therefore bypassed
the maintenance mode checks. This adds the needed check directly.

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2022-05-12 11:36:56 +02:00
Wolfgang Bumiller 5b460ef525 client: add --ns parameters to snapshot commands
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-12 11:02:06 +02:00
Wolfgang Bumiller 03d4f43d5a client: rename --backup-ns to --ns in backup command
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2022-05-12 10:43:56 +02:00
Stoiko Ivanov 5225817de6 docs: zfs: update documentation about ZED
This closely follows commit aa425868069818167ff0a3cca5c64a2acc88173e
in pve-docs.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2022-05-12 10:08:30 +02:00
Stoiko Ivanov 0ae5f76277 docs: local-zfs: minor cleanup and adaptation
fixes a few small glitches in the markup.

rephrases a few PVEisms (PBS will not swap when starting a backup to
an external storage)

add zstd to available compression algorithms

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2022-05-12 10:08:30 +02:00
Stoiko Ivanov 09d903034f docs: system-booting: (re)add screenshots
add the grub+systemdboot screen from a PBS system (taken via
spice-viewer).

The alingment of left/right looked better to me than keeping both on the
right).

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2022-05-12 10:08:30 +02:00
Stoiko Ivanov 9eb804006c docs: add system-booting from pve-docs
and transform to reST.

semantic changes to the content are:
* s/{pve}/`Proxmox Backup`_/g
* changing footnotes to parenthesized notes (did not see footnote use in
  the current docs)
* removed the comment about systems setup before the introduction of
  p-b-t (which was introduced before pbs)

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2022-05-12 10:08:30 +02:00
Stoiko Ivanov 6e3391c85b docs: sysadmin: adapt kernel-specifics for PBS
while all statements here are technically true - adding all
virtualization improvements is not relevant for proxmox backup in most
cases.
The intel nic driver seems like a left-over from a time (pre PVE 5.1)
where the pve-kernel included the out-of-tree drivers.

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2022-05-12 10:08:30 +02:00
Thomas Lamprecht 71139be203 bump version to 2.1.9-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:48:51 +02:00
Thomas Lamprecht fbca018229 ui: content: code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht 7e8b24bd8c ui: content: show namespaces also inline and rework node type detection
this not only makes the action disable/hide checks simpler, but also
prepares the view a bit for the idea of adding a new API endpoint
that returns the whole datastore content tree as structured JSON so
that it can be directly loaded into a tree store.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht fe79687c59 pull group: add error context for cleanup_unreferenced_files
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht 9ccf933be5 datastore: move update_manifest into BackupDir impl
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht 87cdc327b9 sync: pull snapshot: use template variables for bloat reduction
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht 5566099849 datastore: move cleanup_unreferenced_files to BackupDir impl and fix NS awareness
sync failed on cleanup due to always trying to do so in the root NS

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht 92b9cc1554 ui: remote target selectors: code cleanups
just a small start...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht 0e3de42aa7 ui: sync job: use namespace selector for localNS
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Thomas Lamprecht 8c29bca57c ui: move remote target datastore/ns selectors to own file
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00
Fabian Grünbichler d895b26bb9 ui: add namespace fields to sync
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2022-05-12 09:40:43 +02:00