Commit Graph

5809 Commits

Author SHA1 Message Date
Stefan Reiter a862835be2 file-restore: use less memory for VM and reboot on panic
With the vsock-pkt-buffer fix in proxmox-backup-restore-image, we can
use way less memory for the VM without risking any crashes. 128 MiB
seems to be the lowest it will go and still be fully reliable.

While at it, add the "panic=1" argument to the kernel command line, so
in case the kernel *does* run out of memory, it will at least restart
automatically.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-26 15:46:37 +02:00
Stefan Reiter ddbd63ed5f file-restore: exit with code 1 in case streaming fails
This way the task gets marked as "failed" in PVE.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-26 15:46:37 +02:00
Stefan Reiter 6a59fa0e18 file-restore: add size to image files and components
Read image sizes (.pxar.fidx/.img.didx) from manifest and partition
sizes from /sys/...

Requires a change to ArchiveEntry, as DirEntryAttribute::Directory
does not have a size associated with it (and that's probably good).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-26 15:46:37 +02:00
Dietmar Maurer 1ed9069ad3 http proxy: improve response parser
Avoid strange error message in case of connect error (only parse status + headers).
We are not interested in the response body, so simply ignore it.
2021-04-26 11:21:11 +02:00
Dominik Csapak a588b67906 api2/config/datastore: use update_job_last_run_time for schedules
this way, the api call does not error out when the file is locked
currently (which means that job is running and we do not need
to update the time)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-26 10:51:06 +02:00
Dominik Csapak 37a634f550 server/jobstate: improve name of 'try_update_state_file'
and improve comment

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-26 10:50:36 +02:00
Dominik Csapak 951fe0cb7d server/jobstate: add 'updatd' to Finish variant
when a user updates a job schedule, we want to save that point in time
to calculate future runs, otherwise when a user updates a schedule to
a time that would have been between the last run and 'now' the
schedule is triggered instantly

for example:
schedule 08:00
last run today 08:00
now it is 12:00

before this patch:
update schedule to 11:00
 -> triggered instantly since we calculate from 08:00

after this patch:
update schedule to 11:00
 -> triggered tomorrow 11:00 since we calculate from today 12:00

the change in the enum type is ok, since by default serde does not
error on unknown fields and the new field is optional

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-26 09:48:34 +02:00
Dominik Csapak 4ca3f0c6ae api2/tape/backup: list backed up snapshots on failed backup notification
if a backup task failed (e.g. it was aborted), show the snapshots
which were successfully backed up in the notification

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 16:25:17 +02:00
Dominik Csapak 69e5ba29c4 ui: tape: reload drive status on user actions
when the user start an action where we know that it locks the drive,
reload the tape store, so that the state is refreshed

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 16:25:17 +02:00
Wolfgang Bumiller e045d154e9 file-restore: avoid unnecessary clone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-23 13:22:30 +02:00
Fabian Grünbichler 6526709d48 file-restore: add context to b64-decode error
to make the following cryptic error:

 proxmox-file-restore failed: Error: Invalid byte 46, offset 5.

more understandable:

 proxmox-file-restore failed: Error: Failed base64-decoding path '/root.pxar.didx' - Invalid byte 46, offset 5.

when a user passes in a non-base64 path but sets `--base64`.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-04-23 13:19:40 +02:00
Thomas Lamprecht 603f80d813 bump version to 1.1.3-2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-23 10:52:17 +02:00
Dominik Csapak 398636b61c api2/node/status: extend node status
to be more on par with pve

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 10:30:30 +02:00
Dominik Csapak eb70464839 api2/nodes/status: use NodeStatus struct
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 10:30:30 +02:00
Dominik Csapak 75054859ff api2/types: add necessary types for node status
we want to use concrete types instead of value

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-23 10:30:30 +02:00
Dietmar Maurer 8e898895cc tape: do not query density_code in SgTape::new()
Because this can fail with NoSense/MediumChanged and other informational
Sense codes.
2021-04-23 09:56:44 +02:00
Dietmar Maurer 4be6beab6f tape: format_media - implement special case for WORM media 2021-04-23 08:33:13 +02:00
Dietmar Maurer a3b4b5b50e tape: define and use MediumType enum 2021-04-23 07:54:42 +02:00
Dietmar Maurer 33b8d7e5e8 tape: use loaded media_type in format_media (instead of drive_density)
Required to format LTO4 media loaded in LTO5 drive).

Also contains some SCSI code cleanups.
2021-04-23 07:27:30 +02:00
Thomas Lamprecht f2f43e1904 server/rest: fix new type ambiguity
basically the same as commit eeff085d9d
Will be required once we get to use a newer rustc, at least the
client build for archlinux was broken due to this.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-22 21:24:44 +02:00
Thomas Lamprecht c002d48b0c bump version to 1.1.3-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-22 20:15:18 +02:00
Stefan Reiter 15998ed12a file-restore: support encrypted VM backups
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-22 17:55:30 +02:00
Stefan Reiter 9d8ab62769 client-tools: add crypto_parameters_keep_fd
same functionality as crypto_parameters, except it keeps the file
descriptor passed as "keyfd" open (and seeks to the beginning after
reading), if one is given.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-22 17:55:30 +02:00
Stefan Reiter 3526a76ef3 file-restore: don't force PBS_FINGERPRINT env var
It is valid to not set it, in case the server has a valid certificate.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-22 17:55:30 +02:00
Dietmar Maurer b9e0fcbdcd tape: implement report_desnity 2021-04-22 13:54:31 +02:00
Dietmar Maurer a7188b3a75 tape: fix FORMAT for LTO-4 drives
FORMAT requires LTO-5 or newer, so we do a rewind/erase if FORMAT fails.
2021-04-22 11:44:49 +02:00
Dietmar Maurer b6c06dce9d http proxy: implement read_connect_response()
Limit memory usage in case we get strange data from proxy.
2021-04-22 10:06:14 +02:00
Stefan Reiter 4adf47b606 file-restore: allow extracting a full pxar archive
If the path for within the archive is empty, assume "/" to extract all
of it.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:20:54 +02:00
Stefan Reiter 4d0dc29951 file-restore: Add 'v' (Virtual) ArchiveEntry type
For the actual partitions and blockdevices in a backup, which the
user sees like folders in the file-restore ui

Encoded as "None", to avoid cluttering DirEntryAttribute, where it
wouldn't make any sense to have.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-21 17:19:40 +02:00
Stefan Reiter 1011fb552b file-restore: print warnings on stderr
as we print JSON on stdout to be parsed

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:18:12 +02:00
Stefan Reiter 2fd2d29281 file-restore: don't list non-pxar/-img *idx archives
These can't be entered or restored anyway, and cause issues with catalog
files for example.

Also a clippy fix.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:18:06 +02:00
Dietmar Maurer 9104152a83 HttpsConnector: add proxy support 2021-04-21 15:29:17 +02:00
Dietmar Maurer 02a58862dd HttpsConnector: code cleanup 2021-04-21 15:29:17 +02:00
Dietmar Maurer 26153589ba new http client implementation SimpleHttp (avoid static HTTP_CLIENT)
This one will have proxy support.
2021-04-21 15:29:17 +02:00
Dietmar Maurer 17b3e4451f MaybeTlsStream: implement poll_write_vectored()
This is just an performance optimization.
2021-04-21 15:29:17 +02:00
Dietmar Maurer a2072cc346 http: rename EitherStream to MaybeTlsStream
And rename the enum values. Added an additional enum called Proxied.

The enum in now more specialized, but we only use it for the http client anyways.
2021-04-21 15:29:17 +02:00
Dominik Csapak fea23d0323 fix #3393: tools/xattr: allow xattr 'security.NTACL'
in some configurations, samba stores NTFS-ACLs in this xattr[0], so
we should backup (if we can)

altough the 'security' namespace is special (e.g. in use by
selinux, etc.) this value is normally only used by samba and we
should be able to back it up.

to restore it, the user needs at least 'CAP_SYS_ADMIN' rights, otherwise
it cannot be set

0: https://www.samba.org/samba/docs/current/man-html/vfs_acl_xattr.8.html

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-21 14:49:46 +02:00
Dominik Csapak 71e83e1b1f tape/changer/sg_pt_changer: read whole descriptor size for each entry
Some changer seem to append more data than we expect, but correctly
annotates that size in the subheader.

For each descriptor entry, read as much as the size given in the
subheader (or until the end of the reader), else our position in
the reader is wrong for the next entry, and we will parse
incorrect data.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-21 14:07:41 +02:00
Dietmar Maurer 28570d19a6 tape restore: avoid multiple stat calls for same chunk 2021-04-16 13:17:17 +02:00
Dietmar Maurer 1369bcdbba tape restore: verify if all chunks exist 2021-04-16 12:20:44 +02:00
Dietmar Maurer 5e4d81e957 tape restore: simplify log (list datastores on single line) 2021-04-16 11:35:05 +02:00
Dietmar Maurer 0f4721f305 tape restore: fix datastore locking 2021-04-16 09:09:05 +02:00
Thomas Lamprecht 5547f90ba7 bump version to 1.1.2-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 13:26:59 +02:00
Thomas Lamprecht 2e1b63fb25 backup verify: do not check every loop iteration for abort/shutdown
only check every 1024'th, which is cheaper to do than a modulo, as we
can just mask the 10 least-significant-bits and check if the result
is zero.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 13:21:36 +02:00
Thomas Lamprecht 7b2d3a5fe9 backup verify: unify check if chunk can be skipped
This also re-checks the corrupt chunk list before actually loading a
chunk.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 13:21:07 +02:00
Thomas Lamprecht 0216f56241 config: tfa: drop now unused schema::Updatable
was used in a macro expansion, now handled otherwise

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 12:35:11 +02:00
Dietmar Maurer 80acdd71fa tape: do not try to backup unfinished backups 2021-04-15 10:24:14 +02:00
Thomas Lamprecht 26af61debc backup verify: re-check if we can skip a chunk in the actual verify loop
Fixes a non-negligible performance regression from commit
7f394c807b

While we skip known-verified chunks in the stat-and-inode-sort loop,
those are only the ones from previous indexes. If there's a repeated
chunk in one index they would get re-verified more often as required.

So, add the check again explicitly to the read+verify loop.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 10:00:06 +02:00
Thomas Lamprecht e7f94010d3 cargo toml: update proxmox version
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 09:56:45 +02:00
Dominik Csapak a4e871f52c api2/access/user: remove password for @pbs users on removal
so that their password entry is not left in the shadow.json

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-15 08:33:20 +02:00