Commit Graph
100 Commits
Author SHA1 Message Date
Stefan ReiterandThomas Lamprecht 572cd0381b file-restore: add debug mode with serial access
Set PBS_QEMU_DEBUG=1 on a command that starts a VM and then connect to
the debug root shell via:
  minicom -D \unix#/run/proxmox-backup/file-restore-serial-10.sock
or similar.

Note that this requires 'proxmox-backup-restore-image-debug' to work,
the postinst script is updated to also generate the corresponding image.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 10:00:12 +02:00
Stefan ReiterandThomas Lamprecht 936eceda61 file-restore: support more drives
A PCI bus can only support up to 32 devices, so excluding built-in
devices that left us with a maximum of about 25 drives. By adding a new
PCI bridge every 32 devices (starting at bridge ID 2 to avoid conflicts
with automatic bridges), we can theoretically support up to 8096 drives.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 09:03:17 +02:00
Stefan ReiterandThomas Lamprecht 61c4087041 file-restore: add more RAM for VMs with many drives or debug
The guest kernel requires more memory depending on how many disks are
attached. 256 seems to be enough for basically any reasonable and
unreasonable amount of disks though.

For debug instance, make it 1G, as these are never started automatically
anyway, and need at least 512MB since the initramfs (especially when
including a debug build of the daemon) is substantially bigger.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 09:03:17 +02:00
Stefan ReiterandThomas Lamprecht 7d39e47182 file-restore: try to kill VM when stale
Helps to clean up a VM that has crashed, is not responding to vsock API
calls, but still has a running QEMU instance.

We always check the process commandline to ensure we don't kill a random
process that took over the PID.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-05-07 09:03:17 +02:00
Stefan ReiterandThomas Lamprecht 18210d8958 file-restore: use 'norecovery' for xfs filesystem
This allows mounting XFS partitons with 'dirty' states, like from a
running VM. Otherwise XFS tries to write recovery information, which
fails on a read-only mount.

Tested-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Tested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-29 15:09:09 +02:00
Stefan ReiterandThomas Lamprecht 606828cc65 file-restore: strip .img.fidx suffix from drive serials
Drive serials have a character limit of 20, longer names like
"drive-virtio0.img.fidx" or "drive-efidisk0.img.fidx" would get cut off.

Fix this by removing the suffix, it is not necessary to uniquely
identify an image.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-27 16:41:29 +02:00
Stefan ReiterandThomas Lamprecht 8369ade880 file-restore: fix package name for kernel/initramfs image
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-27 11:17:38 +02:00
Stefan ReiterandThomas Lamprecht a862835be2 file-restore: use less memory for VM and reboot on panic
With the vsock-pkt-buffer fix in proxmox-backup-restore-image, we can
use way less memory for the VM without risking any crashes. 128 MiB
seems to be the lowest it will go and still be fully reliable.

While at it, add the "panic=1" argument to the kernel command line, so
in case the kernel *does* run out of memory, it will at least restart
automatically.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-26 15:46:37 +02:00
Stefan ReiterandThomas Lamprecht ddbd63ed5f file-restore: exit with code 1 in case streaming fails
This way the task gets marked as "failed" in PVE.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-26 15:46:37 +02:00
Stefan ReiterandThomas Lamprecht 6a59fa0e18 file-restore: add size to image files and components
Read image sizes (.pxar.fidx/.img.didx) from manifest and partition
sizes from /sys/...

Requires a change to ArchiveEntry, as DirEntryAttribute::Directory
does not have a size associated with it (and that's probably good).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-26 15:46:37 +02:00
Stefan ReiterandThomas Lamprecht 15998ed12a file-restore: support encrypted VM backups
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-22 17:55:30 +02:00
Stefan ReiterandThomas Lamprecht 9d8ab62769 client-tools: add crypto_parameters_keep_fd
same functionality as crypto_parameters, except it keeps the file
descriptor passed as "keyfd" open (and seeks to the beginning after
reading), if one is given.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-22 17:55:30 +02:00
Stefan ReiterandThomas Lamprecht 3526a76ef3 file-restore: don't force PBS_FINGERPRINT env var
It is valid to not set it, in case the server has a valid certificate.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-22 17:55:30 +02:00
Stefan ReiterandThomas Lamprecht 4adf47b606 file-restore: allow extracting a full pxar archive
If the path for within the archive is empty, assume "/" to extract all
of it.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:20:54 +02:00
Stefan ReiterandThomas Lamprecht 4d0dc29951 file-restore: Add 'v' (Virtual) ArchiveEntry type
For the actual partitions and blockdevices in a backup, which the
user sees like folders in the file-restore ui

Encoded as "None", to avoid cluttering DirEntryAttribute, where it
wouldn't make any sense to have.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-21 17:19:40 +02:00
Stefan ReiterandThomas Lamprecht 1011fb552b file-restore: print warnings on stderr
as we print JSON on stdout to be parsed

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:18:12 +02:00
Stefan ReiterandThomas Lamprecht 2fd2d29281 file-restore: don't list non-pxar/-img *idx archives
These can't be entered or restored anyway, and cause issues with catalog
files for example.

Also a clippy fix.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:18:06 +02:00
Stefan ReiterandThomas Lamprecht 3b0321365b use FileBrowser from proxmox-widget-toolkit
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-13 08:44:48 +02:00
Stefan ReiterandThomas Lamprecht 0dfce17a43 api/datastore: allow pxar file download of entire archive
Treat filepaths like "/root.pxar.didx" without a trailing slash as
wanting to download the entire archive content instead of erroring. The
zip-creation code already works fine for this scenario.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-13 08:26:41 +02:00
Stefan ReiterandThomas Lamprecht b13089cdf5 file-restore: add 'extract' command for VM file restore
The data on the restore daemon is either encoded into a pxar archive, to
provide the most accurate data for local restore, or encoded directly
into a zip file (or written out unprocessed for files), depending on the
'pxar' argument to the 'extract' API call.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 14:43:41 +02:00
Stefan ReiterandThomas Lamprecht 1f03196c0b tools/zip: add zip_directory helper
Encodes an entire local directory into an AsyncWrite recursively.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-08 14:32:03 +02:00
Stefan ReiterandThomas Lamprecht edf0940649 pxar/extract: add sequential variant of extract_sub_dir
extract_sub_dir_seq, together with seq_files_extractor, allow extracting
files from a pxar Decoder, along with the existing option for an
Accessor. To facilitate code re-use, some helper functions are extracted
in the process.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 14:24:23 +02:00
Stefan ReiterandThomas Lamprecht 801ec1dbf9 file-restore(-daemon): implement list API
Allows listing files and directories on a block device snapshot.
Hierarchy displayed is:

/archive.img.fidx/bucket/component/<path>
e.g.
/drive-scsi0.img.fidx/part/2/etc/passwd
(corresponding to /etc/passwd on the second partition of drive-scsi0)

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-08 14:24:14 +02:00
Stefan ReiterandThomas Lamprecht 34ac5cd889 debian/client: add postinst hook to rebuild file-restore initramfs
This will be triggered on updating proxmox-file-restore (via configure,
necessary since the daemon binary might change) and
proxmox-backup-restore-image (via 'activate-noawait', necessary since
the base image might change).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 14:20:05 +02:00
Stefan ReiterandThomas Lamprecht 58421ec112 file-restore: add basic VM/block device support
Includes methods to start, stop and list QEMU file-restore VMs, as well
as CLI commands do the latter two (start is implicit).

The implementation is abstracted behind the concept of a
"BlockRestoreDriver", so other methods can be implemented later (e.g.
mapping directly to loop devices on the host, using other hypervisors
then QEMU, etc...).

Starting VMs is currently unused but will be needed for further changes.

The design for the QEMU driver uses a locked 'map' file
(/run/proxmox-backup/$UID/restore-vm-map.json) containing a JSON
encoding of currently running VMs. VMs are addressed by a 'name', which
is a systemd-unit encoded combination of repository and snapshot string,
thus uniquely identifying it.

Note that currently you need to run proxmox-file-restore as root to use
this method of restoring.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 14:11:02 +02:00
Stefan ReiterandThomas Lamprecht a5bdc987dc add tools/cpio encoding module
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 14:10:45 +02:00
Stefan ReiterandThomas Lamprecht d32a8652bd file-restore-daemon: add disk module
Includes functionality for scanning and referring to partitions on
attached disks (i.e. snapshot images).

Fairly modular structure, so adding ZFS/LVM/etc... support in the future
should be easy.

The path is encoded as "/disk/bucket/component/path/to/file", e.g.
"/drive-scsi0/part/0/etc/passwd". See the comments for further
explanations on the design.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 14:03:54 +02:00
Stefan ReiterandThomas Lamprecht a26ebad5f9 file-restore-daemon: add watchdog module
Add a watchdog that will automatically shut down the VM after 10
minutes, if no API call is received.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:58:29 +02:00
Stefan ReiterandThomas Lamprecht dd9cef56fc file-restore-daemon: add binary with virtio-vsock API server
Implements the base of a small daemon to run within a file-restore VM.

The binary spawns an API server on a virtio-vsock socket, listening for
connections from the host. This happens mostly manually via the standard
Unix socket API, since tokio/hyper do not have support for vsock built
in. Once we have the accept'ed file descriptor, we can create a
UnixStream and use our tower service implementation for that.

The binary is deliberately not installed in the usual $PATH location,
since it shouldn't be executed on the host by a user anyway.

For now, only the API calls 'status' and 'stop' are implemented, to
demonstrate and test proxmox::api functionality.

Authorization is provided via a custom ApiAuth only checking a header
value against a static /ticket file.

Since the REST server implementation uses the log!() macro, we can
redirect its output to stdout by registering env_logger as the logging
target. env_logger is already in our dependency tree via zstd/bindgen.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-08 13:57:57 +02:00
Stefan ReiterandThomas Lamprecht 26858dba84 server/rest: add ApiAuth trait to make user auth generic
This allows switching the base user identification/authentication method
in the rest server. Will initially be used for single file restore VMs,
where authentication is based on a ticket file, not the PBS user
backend (PAM/local).

To avoid putting generic types into the RestServer type for this, we
merge the two calls "extract_auth_data" and "check_auth" into a single
one, which can use whatever type it wants internally.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:57:57 +02:00
Stefan ReiterandThomas Lamprecht 9fe3358ce6 file-restore: allow specifying output-format
Makes CLI use more comfortable by not just printing JSON to the
terminal.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:57:57 +02:00
Stefan ReiterandThomas Lamprecht ba39ab20fb server/rest: extract auth to seperate module
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-01 11:26:28 +02:00
Stefan ReiterandThomas Lamprecht ff8945fd2f proxmox_client_tools: move common key related functions to key_source.rs
Add a new module containing key-related functions and schemata from all
over, code moved is not changed as much as possible.

Requires adapting some 'use' statements across proxmox-backup-client and
putting the XDG helpers quite cozily into proxmox_client_tools/mod.rs

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-01 11:09:28 +02:00
Stefan ReiterandThomas Lamprecht 4876393562 vsock_client: support authorization header
Pass in an optional auth tag, which will be passed as an Authorization
header on every subsequent call.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-01 11:09:28 +02:00
Stefan ReiterandThomas Lamprecht 971bc6f94b vsock_client: remove some &mut restrictions and rustfmt
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-01 11:09:28 +02:00
Stefan ReiterandThomas Lamprecht cab92acb3c vsock_client: remove wrong comment
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-01 11:09:28 +02:00
Stefan ReiterandDietmar Maurer f1a83e9759 client: extract common functions to proxmox_client_tools module
...including common schemata, connect(), extract_*() and completion
functions.

For later use with proxmox-file-restore binary.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 10:13:10 +01:00
Stefan ReiterandWolfgang Bumiller 6afb60abf5 asyncify pxar create_archive
...to take advantage of the aio::Encoder from the pxar create.

Rather straightforward conversion, but does require getting rid of
references in the Archiver struct, and thus has to be given the Mutex
for the catalog directly. The callback is boxed.

archive_dir_contents can call itself recursively, and thus needs to
return a boxed future.

Users are adjusted, namely PxarBackupStream is converted to use an
Abortable future instead of a thread so it supports async in its handler
function, and the pxar bin create_archive is converted to an async API
function. One test case is made to just use 'block_on'.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-02-17 09:24:20 +01:00
Stefan ReiterandDietmar Maurer 89d25b1931 client: add VsockClient to connect to virtio-vsock VMs
Currently useful only for single file restore, but kept generic enough
to use any compatible API endpoint over a virtio-vsock[0,1] interface.

VsockClient is adapted and slimmed down from HttpClient.

A tower-compatible VsockConnector is implemented, using a wrapped
UnixStream as transfer. The UnixStream has to be wrapped in a custom
struct to implement 'Connection', Async{Read,Write} are simply forwarded
directly to the underlying stream.

[0] https://www.man7.org/linux/man-pages/man7/vsock.7.html
[1] https://wiki.qemu.org/Features/VirtioVsock

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 08:15:43 +01:00
Stefan ReiterandDietmar Maurer b57c0dbe30 rest: implement tower service for UnixStream
This allows anything that can be represented as a UnixStream to be used
as transport for an API server (e.g. virtio sockets).

A tower service expects an IP address as it's peer, which we can't
reliably provide for unix socket based transports, so just fake one.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-02-17 07:50:35 +01:00
Stefan ReiterandDietmar Maurer 4799280ccd http_client: add timeouts for critical connects
Use timeout futures for sections that might hang in certain error
conditions. This is mostly intended to be used as a safeguard, not a
first line of defense - i.e. best-effort avoidance of total hangs.

Not every future used for the HttpClient/H2Client is changed, only those
where a quick response is to be expected. For example, the response
reading futures are left alone, so data transfer is never capped with
timeout, only the initial server connect.

It is also used for upgrading to H2 connections, as that can take a long
time on overloaded servers.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-12-22 13:31:10 +01:00
Stefan ReiterandWolfgang Bumiller fd19256470 gc: treat .bad files like regular chunks
Simplify the phase 2 code by treating .bad files just like regular
chunks, with the exception of stat logging.

To facilitate, we need to touch .bad files in phase 1. We only do this
under the condition that 1) the original chunk is missing (as before),
and 2) the original chunk is still referenced somewhere (since the code
lives in the error handler for a failed chunk touch, it only gets called
for chunks we expect to be there, i.e. ones that are referenced).

Untouched they will then be cleaned up after 24 hours (or after the last
longer-running task finishes).

Reason 2) is also a fix for .bad files not being cleaned up at all if
the original is no longer referenced anywhere (e.g. a user deleting all
snapshots after seeing some corrupt chunks appear).

cond_touch_path is introduced to touch arbitrary paths in the chunk
store with the same logic as touching chunks.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-18 14:04:49 +01:00
Stefan ReiterandThomas Lamprecht 5e293f1315 apt: use typed response for get_versions
...and cleanup get_versions for manager CLI.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-12 10:15:32 +01:00
Stefan ReiterandThomas Lamprecht 137a6ebcad apt: allow changelog retrieval from enterprise repo
If a package is or will be installed from the enterprise repo, retrieve
the changelog from there as well (securely via HTTPS and authenticated
with the subcription key).

Extends the get_string method to take additional headers, in this case
used for 'Authorization'. Hyper does not have built-in basic auth
support AFAICT but it's simple enough to just build the header manually.

Take the opportunity and also set the User-Agent sensibly for GET
requests, just like for POST.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-11-09 17:28:58 +01:00
Stefan ReiterandFabian Grünbichler 9e61c01ce4 apt: add /changelog API call similar to PVE
For proxmox packages it works the same way as PVE, by retrieving the
changelog URL and issuing a HTTP GET to it, forwarding the output to the
client. As this is only supposed to be a workaround removed in the
future, a simple block_on is used to avoid async.

For debian packages we can simply call 'apt-get changelog' and forward
it's output.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan ReiterandFabian Grünbichler 91c9b42da3 fix #2934: list to-be-installed packages in updates
As always, libapt is mocking us with complexity, but we can get the
approximate result we want by retrieving dependencies of all
to-be-updated packages and then seeing if they are missing.

If they are, we assume they will be installed.

For this, query_detailed_info is extended to allow reading details for
non-installed packages, and this is also exposed in
list_installed_apt_packages via 'all_versions_for'. This is necessary so
we can retrieve changelogs for such packages.

Note that we cannot retrieve all that information all the time, as
querying details for packages that aren't installed takes a rather long
time.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan ReiterandFabian Grünbichler 52d2ae48f0 apt: refactor package detail reading into function
No functional change intended.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan ReiterandFabian Grünbichler 1872050564 apt: use 'apt-get changelog --print-uris' in get_changelog_url
Avoids custom hardcoded logic, but can only be used for debian packages
as of now. Adds a FIXME to switch over to use --print-uris only once our
package repos support that changelog format.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:26 +02:00
Stefan ReiterandFabian Grünbichler efeb92efee apt: allow filter to select different package version
To get package details for a specific version instead of only the
candidate.

Also cleanup filter function with extra struct instead of unnamed &str
parameters.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:15 +02:00
Stefan ReiterandThomas Lamprecht 5eb9dd0c8a add tools::http for generic HTTP GET and move HttpsConnector there
...to avoid having the tools:: module depend on api2.

The get_string function is based directly on hyper and thus relatively
simple, not supporting redirects for example.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-21 16:22:08 +02:00
Stefan ReiterandDietmar Maurer 0698f78df5 fix #2988: allow verification after finishing a snapshot
To cater to the paranoid, a new datastore-wide setting "verify-new" is
introduced. When set, a verify job will be spawned right after a new
backup is added to the store (only verifying the added snapshot).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 10:51:13 +02:00
Stefan ReiterandDietmar Maurer bcc2880461 add verify_backup_dir_with_lock for callers already holding locks
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 10:49:19 +02:00
Stefan ReiterandDietmar Maurer df729017b4 datastore: cleanup open and load config only once
Force consumers to use the lookup_datastore method instead of
potentially opening a datastore twice, and pass the config we have
already loaded into open_with_path, removing the need for open(1).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 07:51:05 +02:00
Stefan ReiterandDietmar Maurer 455f2ad228 fix missing block_in_place for remove_backup
Commit 9070d11f4c introduced this change for other call sites,
assuming it is correct, this one was missed.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 07:48:51 +02:00
Stefan ReiterandDietmar Maurer b64e9a97f3 rustdoc: overhaul backup rustdoc and add locking table
Rewrite most of the documentation to be more readable and correct
(according to the current implementations).

Add a table visualizing all different locks used to synchronize
concurrent operations.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-16 09:38:01 +02:00
Stefan ReiterandDietmar Maurer 254b1f2213 rustdoc: add crate level doc
Contains a link to the 'backup' module's doc, as that explains a lot
about the inner workings of PBS and probably marks a good entry point
for new readers.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-16 09:37:50 +02:00
Stefan ReiterandDietmar Maurer 1a374fcfd6 datastore: add manifest locking
Avoid races when updating manifest data by flocking a lock file.
update_manifest is used to ensure updates always happen with the lock
held.

Snapshot deletion also acquires the lock, so it cannot interfere with an
outstanding manifest write.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-16 09:34:12 +02:00
Stefan ReiterandDietmar Maurer 4c0ae82e23 datastore: remove individual snapshots before group
Removing a snapshot has some more safety checks which we don't want to
ignore when removing an entire group (i.e. locking the manifest and
notifying GC).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:51:09 +02:00
Stefan ReiterandDietmar Maurer 883aa6d5a4 datastore: remove load_manifest_json
There's no point in having that as a seperate method, just parse the
thing into a struct and write it back out correctly.

Also makes further changes to the method simpler.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:19:32 +02:00
Stefan ReiterandDietmar Maurer bfa54f2e85 verify: acquire shared snapshot flock and skip on error
If we can't acquire a lock (either because the snapshot disappeared, it
is about to be forgotten/pruned, or it is currently still running) skip
the snapshot. Hold the lock during verification, so that it cannot be
deleted while we are still verifying.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan ReiterandDietmar Maurer 238a872d1f reader: acquire shared flock on open snapshot
...to avoid it being forgotten or pruned while in use.

Update lock error message for deletions to be consistent.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan ReiterandDietmar Maurer 7d6c4c39e9 backup: use shared flock for base snapshot
To allow other reading operations on the base snapshot as well. No
semantic changes with this patch alone, as all other locks on snapshots
are exclusive.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan ReiterandDietmar Maurer f153930066 prune: never fail, just warn about failed removals
A removal can fail if the snapshot is already gone (this is fine, our
job is done either way) or we couldn't get a lock (also fine, it can't
be removed then, just warn the user so he knows what happened and why it
wasn't removed) - keep going either way.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan ReiterandDietmar Maurer 836c4a278d prune: respect snapshot flock
A snapshot that's currently being read can still appear in the prune
list, but should not be removed.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan ReiterandDietmar Maurer 735ee5206a fuse_loop: handle unmap on crashed instance
If a fuse_loop instance dies suddenly (e.g. SIGKILL), the FUSE mount and
loop device assignment are left behind. We can determine this scenario
on specific unmap, when the PID file is either missing or contains a PID
of a non-running process, but the backing file and potentially loop
device are still there.

If that's the case, do an "emergency cleanup", by unassigning the
loopdev, calling 'fusermount -u' and then cleaning any leftover files
manually.

With this in place, pretty much any situation is now recoverable via
only the 'proxmox-backup-client' binary, by either calling 'unmap' with
or without parameters.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:38:25 +02:00
Stefan ReiterandDietmar Maurer a86bf52390 fuse_loop: wait for instance to close after killing
On unmap, only report success if the instance we are killing actually
terminates. This is especially important so that cleanup routines can be
assured that /run files are actually cleaned up after calling
cleanup_unused_run_files.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:37:47 +02:00
Stefan ReiterandDietmar Maurer 2deee0e01f fuse_loop: add automatic cleanup of run files and dangling instances
A 'map' call will only clean up what it needs, that is only leftover
files or dangling instances of it's own name.

For a full cleanup the user can call 'unmap' without any arguments.

The 'cleanup on error' behaviour of map_loop is removed. It is no longer
needed (since the next call will clean up anyway), and in fact fixes a
bug where trying to map an image twice would result in an error, but
also cleanup the .pid file of the running instance, causing 'unmap' to
fail afterwards.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:37:05 +02:00
Stefan ReiterandDietmar Maurer 2d7d6e61be mount/map: use names for map/unmap for easier use
So user doesn't need to remember which loop devices he has mapped to
what.

systemd unit encoding is used to transform a unique identifier for the
mapped image into a suitable name. The files created in /run/pbs-loopdev
will be named accordingly.

The encoding all happens outside fuse_loop.rs, so the fuse_loop module
does not need to care about encodings - it can always assume a name is a
valid filename.

'unmap' without parameter displays all current mappings. It's
autocompletion handler will list the names of all currently mapped
images for easy selection. Unmap by /dev/loopX or loopdev number is
maintained, as those can be distinguished from mapping names.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:35:52 +02:00
Stefan ReiterandDietmar Maurer 4ec17f7eb5 loopdev: add module doc
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:34:39 +02:00
Stefan ReiterandDietmar Maurer fcad02e1de fuse_loop: add documentation
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:33:34 +02:00
Stefan ReiterandDietmar Maurer 708fab3082 format: fix typo in function name
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:32:21 +02:00
Stefan ReiterandDietmar Maurer 45f9b32e0f client: implement map/unmap commands for .img backups
Allows mapping fixed-index .img files (usually from VM backups) to be
mapped to a local loopback device.

The architecture uses a FUSE-backed temp file mapped to a loopdev:

  /dev/loopX -> FUSE /run/pbs-loopdev/xxx -> backup client -> PBS

Since unmapping requires some cleanup (unmap the loopdev, stop FUSE,
remove the temp files) a special 'unmap' command is added, which uses a
PID file to send SIGINT to the backup-client instance started with
'map', which will handle the cleanup itself.

The polling with select! in mount.rs needs to be split in two, since we
have a chicken and egg problem between running FUSE and setting up the
loop device - so we need to do them concurrently, until the loopdev is
assigned, at which point we can report success and daemonize, and then
continue polling the FUSE loop future.

A loopdev module is added to tools containing all required functions for
mapping a loop device to the FUSE file, with the ioctls moved into an
inline module to avoid exposing them directly.

The client code is placed in the 'mount' module, which, while
admittedly a loose fit, allows reuse of the daemonizing code.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-06 09:08:14 +02:00
Stefan ReiterandDietmar Maurer 1d0b662b42 mount: handle SIGTERM as well
instead of only SIGINT

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-06 09:08:14 +02:00
Stefan ReiterandDietmar Maurer da6e67b321 rrd: fix integer underflow
Causes a panic if last_update is smaller than RRD_DATA_ENTRIES*reso,
which (I believe) can happen when inserting the first value for a DB.

Clamp the value to 0 in that case.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Stefan ReiterandDietmar Maurer dec00364b3 ParallelHandler: check for errors during thread join
Fix a potential bug where errors that happen after the SendHandle has
been dropped while doing the thread join might have been ignored.
Requires internal check_abort to be moved out of 'impl SendHandle' since
we only have the Mutex left, not the SendHandle.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Stefan ReiterandDietmar Maurer d10332a15d SnapshotVerifyState: use enum for state
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 13:06:04 +02:00
Stefan ReiterandDietmar Maurer 43772efc6e backup: check all referenced chunks actually exist
A client can omit uploading chunks in the "known_chunks" list, those
then also won't be written on the server side. Check all those chunks
mentioned in the index but not uploaded for existance and report an
error if they don't exist instead of marking a potentially broken backup
as "successful".

This is only important if the base snapshot references corrupted chunks,
but has not been negatively verified. Also, it is important to only
verify this at the end, *after* all index writers are closed, since only
then can it be guaranteed that no GC will sweep referenced chunks away.

If a chunk is found missing, also mark the previous backup with a
verification failure, since we know the missing chunk has to referenced
in it (only way it could have been inserted to known_chunks with
checked=false). This has the benefit of automatically doing a
full-upload backup if the user attempts to retry after seeing the new
error, instead of requiring a manual verify or forget.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 10:00:05 +02:00
Stefan ReiterandDietmar Maurer 0af2da0437 backup: check verify state of previous backup before allowing reuse
Do not allow clients to reuse chunks from the previous backup if it has
a failed validation result. This would result in a new "successful"
backup that potentially references broken chunks.

If the previous backup has not been verified, assume it is fine and
continue on.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 09:59:29 +02:00
Stefan ReiterandDietmar Maurer 597427afaf clean up .bad file handling in sweep_unused_chunks
Code cleanup, no functional change intended.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-10 06:31:22 +02:00
Stefan ReiterandDietmar Maurer 3cddfb29be backup: ensure no fixed index writers are left over either
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-10 06:29:38 +02:00
Stefan ReiterandDietmar Maurer 068e526862 backup: touch all chunks, even if they exist
We need to update the atime of chunk files if they already exist,
otherwise a concurrently running GC could sweep them away.

This is protected with ChunkStore.mutex, so the fstat/unlink does not
race with touching.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:51:03 +02:00
Stefan ReiterandDietmar Maurer a9767cf7de gc: remove .bad files on garbage collect
The iterator of get_chunk_iterator is extended with a third parameter
indicating whether the current file is a chunk (false) or a .bad file
(true).

Count their sizes to the total of removed bytes, since it also frees
disk space.

.bad files are only deleted if the corresponding chunk exists, i.e. has
been rewritten. Otherwise we might delete data only marked bad because
of transient errors.

While at it, also clean up and use nix::unistd::unlinkat instead of
unsafe libc calls.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:43:13 +02:00
Stefan ReiterandDietmar Maurer 0f3b7efa84 verify: rename corrupted chunks with .bad extension
This ensures that following backups will always upload the chunk,
thereby replacing it with a correct version again.

Format for renaming is <digest>.<counter>.bad where <counter> is used if
a chunk is found to be bad again before a GC cleans it up.

Care has been taken to deliberately only rename a chunk in conditions
where it is guaranteed to be an error in the chunk itself. Otherwise a
broken index file could lead to an unwanted mass-rename of chunks.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:20:57 +02:00
Stefan ReiterandDietmar Maurer 7c77e2f94a verify: fix log units
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:10:19 +02:00
Stefan ReiterandThomas Lamprecht ab6615134c d/postinst: always fixup termproxy user id and for all users
Anyone with a PAM account and Sys.Console access could have started a
termproxy session, adapt the regex.

Always test for broken entries and run the sed expression to make sure
eventually all occurences of the broken syntax are fixed.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-01 18:02:11 +02:00
Stefan ReiterandDietmar Maurer 81f293513e backup: lock base snapshot and ensure existance on finish
To prevent forgetting the base snapshot of a running backup, and catch
the case when it still happens (e.g. via manual rm) to at least error
out instead of storing a potentially invalid backup.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:04:47 +02:00
Stefan ReiterandDietmar Maurer 8b5f72b176 Revert "backup: ensure base snapshots are still available after backup"
This reverts commit d53fbe2474.

The HashSet and "register" function are unnecessary, as we already know
which backup is the one we need to check: the last one, stored as
'last_backup'.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:03:53 +02:00
Stefan ReiterandDietmar Maurer f23f75433f backup: flock snapshot on backup start
An flock on the snapshot dir itself is used in addition to the group dir
lock. The lock is used to avoid races with forget and prune, while
having more granularity than the group lock (i.e. the group lock is
necessary to prevent more than one backup per group, but the snapshot
lock still allows backups unrelated to the currently running to be
forgotten/pruned).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:02:21 +02:00
Stefan ReiterandDietmar Maurer 6d6b4e72d3 datastore: prevent in-use deletion with locks instead of heuristic
Attempt to lock the backup directory to be deleted, if it works keep the
lock until the deletion is complete. This way we ensure that no other
locking operation (e.g. using a snapshot as base for another backup) can
happen concurrently.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:00:29 +02:00
Stefan ReiterandDietmar Maurer 4dbe129284 backup: only allow finished backups as base snapshot
If the datastore holds broken backups for some reason, do not attempt to
base following snapshots on those. This would lead to an error on
/previous, leaving the client no choice but to upload all chunks, even
though there might be potential for incremental savings.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-07 07:32:56 +02:00
Stefan ReiterandDietmar Maurer d74edc3d89 finish_backup: mark backup as finished only after checks have passed
Commit 9fa55e09 "finish_backup: test/verify manifest at server side"
moved the finished-marking above some checks, which means if those fail
the backup would still be marked as successful on the server.

Revert that part and comment the line for the future.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-06 06:39:34 +02:00
Stefan ReiterandDietmar Maurer d53fbe2474 backup: ensure base snapshots are still available after backup
This should never trigger if everything else works correctly, but it is
still a very cheap check to avoid wrongly marking a backup as "OK" when
in fact some chunks might be missing.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:28:54 +02:00
Stefan ReiterandDietmar Maurer 95bda2f25d backup: use flock on backup group to forbid multiple backups at once
Multiple backups within one backup group don't really make sense, but
break all sorts of guarantees (e.g. a second backup started after a
first would use a "known-chunks" list from the previous unfinished one,
which would be empty - but using the list from the last finished one is
not a fix either, as that one could be deleted or pruned once the first
simultaneous backup is finished).

Fix it by only allowing one backup per backup group at one time. This is
done via a flock on the backup group directory, thus remaining intact
even after a reload.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:26:26 +02:00
Stefan ReiterandDietmar Maurer c9756b40d1 datastore: prevent deletion of snaps in use as "previous backup"
To prevent a race with a background GC operation, do not allow deletion
of backups who's index might currently be referenced as the "known chunk
list" for successive backups. Otherwise the GC could delete chunks it
thinks are no longer referenced, while at the same time telling the
client that it doesn't need to upload said chunks because they already
exist.

Additionally, prevent deletion of whole backup groups, if there are
snapshots contained that appear to be currently in-progress. This is
currently unlikely to trigger, as that function is only used for sync
jobs, but it's a useful safeguard either way.

Deleting a single snapshot has a 'force' parameter, which is necessary
to allow deleting incomplete snapshots on an aborted backup. Pruning
also sets force=true to avoid the check, since it calculates which
snapshots to keep on its own.

To avoid code duplication, the is_finished method is factored out.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:26:01 +02:00
Stefan ReiterandDietmar Maurer 8cd29fb24a tools: add nonblocking mode to lock_file
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:18:10 +02:00
Stefan ReiterandDietmar Maurer 505c5f0f76 fix typo: avgerage to average
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 07:08:08 +02:00
Stefan ReiterandDietmar Maurer 71282dd988 ui: fix in-progress snapshots always showing as "Encrypted"
We can't know if they are encrypted or not when they're not even
finished yet.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-29 07:13:25 +02:00
Stefan ReiterandDietmar Maurer 80db161e05 ui: fix error when reloading DataStoreContent
...when an entry is selected, that doesn't exist after the reload.

E.g. when one deletes selects a file within a snapshot and then clicks
the delete icon for said snapshot, focusRow would then fail and the
loading mask stay on until a reload.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-29 07:13:12 +02:00
Stefan ReiterandThomas Lamprecht a4e86972a4 add .../apt/update API call
Depends on patched apt-pkg-native-rs. Changelog-URL detection is
inspired by PVE perl code for now, though marked with fixme to use 'apt
changelog' later on, if/when our repos have APT-compatible changelogs
set up.

list_installed_apt_packages iterates all packages and creates an
APTUpdateInfo with detailed information for every package matched by the
given filter Fn.

Sadly, libapt-pkg has some questionable design choices regarding their
use of 'iterators', which means quite a bit of nesting...

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-23 10:41:14 +02:00
Stefan ReiterandThomas Lamprecht e713ee5c56 remove BufferedFixedReader interface
replaced by AsyncIndexReader

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-22 17:28:49 +02:00