Commit Graph

6583 Commits

Author SHA1 Message Date
b6c06dce9d http proxy: implement read_connect_response()
Limit memory usage in case we get strange data from proxy.
2021-04-22 10:06:14 +02:00
4adf47b606 file-restore: allow extracting a full pxar archive
If the path for within the archive is empty, assume "/" to extract all
of it.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:20:54 +02:00
4d0dc29951 file-restore: Add 'v' (Virtual) ArchiveEntry type
For the actual partitions and blockdevices in a backup, which the
user sees like folders in the file-restore ui

Encoded as "None", to avoid cluttering DirEntryAttribute, where it
wouldn't make any sense to have.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-21 17:19:40 +02:00
1011fb552b file-restore: print warnings on stderr
as we print JSON on stdout to be parsed

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:18:12 +02:00
2fd2d29281 file-restore: don't list non-pxar/-img *idx archives
These can't be entered or restored anyway, and cause issues with catalog
files for example.

Also a clippy fix.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-21 17:18:06 +02:00
9104152a83 HttpsConnector: add proxy support 2021-04-21 15:29:17 +02:00
02a58862dd HttpsConnector: code cleanup 2021-04-21 15:29:17 +02:00
26153589ba new http client implementation SimpleHttp (avoid static HTTP_CLIENT)
This one will have proxy support.
2021-04-21 15:29:17 +02:00
17b3e4451f MaybeTlsStream: implement poll_write_vectored()
This is just an performance optimization.
2021-04-21 15:29:17 +02:00
a2072cc346 http: rename EitherStream to MaybeTlsStream
And rename the enum values. Added an additional enum called Proxied.

The enum in now more specialized, but we only use it for the http client anyways.
2021-04-21 15:29:17 +02:00
fea23d0323 fix #3393: tools/xattr: allow xattr 'security.NTACL'
in some configurations, samba stores NTFS-ACLs in this xattr[0], so
we should backup (if we can)

altough the 'security' namespace is special (e.g. in use by
selinux, etc.) this value is normally only used by samba and we
should be able to back it up.

to restore it, the user needs at least 'CAP_SYS_ADMIN' rights, otherwise
it cannot be set

0: https://www.samba.org/samba/docs/current/man-html/vfs_acl_xattr.8.html

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-21 14:49:46 +02:00
71e83e1b1f tape/changer/sg_pt_changer: read whole descriptor size for each entry
Some changer seem to append more data than we expect, but correctly
annotates that size in the subheader.

For each descriptor entry, read as much as the size given in the
subheader (or until the end of the reader), else our position in
the reader is wrong for the next entry, and we will parse
incorrect data.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-21 14:07:41 +02:00
28570d19a6 tape restore: avoid multiple stat calls for same chunk 2021-04-16 13:17:17 +02:00
1369bcdbba tape restore: verify if all chunks exist 2021-04-16 12:20:44 +02:00
5e4d81e957 tape restore: simplify log (list datastores on single line) 2021-04-16 11:35:05 +02:00
0f4721f305 tape restore: fix datastore locking 2021-04-16 09:09:05 +02:00
5547f90ba7 bump version to 1.1.2-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
v1.1.2
2021-04-15 13:26:59 +02:00
2e1b63fb25 backup verify: do not check every loop iteration for abort/shutdown
only check every 1024'th, which is cheaper to do than a modulo, as we
can just mask the 10 least-significant-bits and check if the result
is zero.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 13:21:36 +02:00
7b2d3a5fe9 backup verify: unify check if chunk can be skipped
This also re-checks the corrupt chunk list before actually loading a
chunk.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 13:21:07 +02:00
0216f56241 config: tfa: drop now unused schema::Updatable
was used in a macro expansion, now handled otherwise

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 12:35:11 +02:00
80acdd71fa tape: do not try to backup unfinished backups 2021-04-15 10:24:14 +02:00
26af61debc backup verify: re-check if we can skip a chunk in the actual verify loop
Fixes a non-negligible performance regression from commit
7f394c807b

While we skip known-verified chunks in the stat-and-inode-sort loop,
those are only the ones from previous indexes. If there's a repeated
chunk in one index they would get re-verified more often as required.

So, add the check again explicitly to the read+verify loop.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 10:00:06 +02:00
e7f94010d3 cargo toml: update proxmox version
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-15 09:56:45 +02:00
a4e871f52c api2/access/user: remove password for @pbs users on removal
so that their password entry is not left in the shadow.json

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-15 08:33:20 +02:00
bc3072ef7a bump version to 1.1.1-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 14:50:41 +02:00
f4bb2510b9 docs: tape: replace changer overview screenshot
Replace previous screenshot with one that shows a more realistic amount
of drives.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2021-04-14 14:41:00 +02:00
2ab12cd0cb verify: add comment for inode sorting
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 14:39:24 +02:00
c894909e17 verify: partially rust fmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 14:39:24 +02:00
7f394c807b backup/verify: improve speed by sorting chunks by inode
before reading the chunks from disk in the order of the index file,
stat them first and sort them by inode number.

this can have a very positive impact on read speed on spinning disks,
even with the additional stat'ing of the chunks.

memory footprint should be tolerable, for 1_000_000 chunks
we need about ~16MiB of memory (Vec of 64bit position + 64bit inode)
(assuming 4MiB Chunks, such an index would reference 4TiB of data)

two small benchmarks (single spinner, ext4) here showed an improvement from
~430 seconds to ~330 seconds for a 32GiB fixed index
and from
~160 seconds to ~120 seconds for a 10GiB dynamic index

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-14 14:39:24 +02:00
7afb98a912 docs: fix tape.cfg format description 2021-04-14 14:30:20 +02:00
3847008e1b docs: pmt - remove old linux driver options 2021-04-14 14:26:39 +02:00
f6ed2eff47 docs: move tape config/syntax to appendix 2021-04-14 14:25:10 +02:00
23eed6755a ui: tape/ChangerStatus: hide drive-slots without assigned drives
if a user has not configured a drive for a specified driveslot of the
changer, simply hide that slot

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-14 14:24:07 +02:00
384a2b4d4f docs: faq: fix encryption link
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 14:18:05 +02:00
910177a388 docs: pve-integration: mention gui integration
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 14:18:05 +02:00
54311a38c6 docs: update proxmox-tape status output 2021-04-14 14:03:45 +02:00
983edbc54a ui: tape/ChangerStatus: add Format button to drivegrid
so that the user can also format an already inserted tape directly

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-14 13:53:16 +02:00
10439718e2 docs: use defininition list for tape Terminology
To avoid those strange line breaks in Field list labels.
2021-04-14 13:25:29 +02:00
ebddccef5f docs: gui: add short tape backup section
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 12:47:05 +02:00
9cfe0ff350 docs: include tape backup in TOC
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 12:47:05 +02:00
295bae14b7 docs: tape: add a bit of text to changers for better image flow
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 12:47:05 +02:00
53939bb438 docs: tape screenshots
Add tape screenshots throughout the tape docs with some references to the GUI

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2021-04-14 12:47:05 +02:00
329c2cbe66 docs: reorder maintenance & network after client usage/tools
The idea is that people first need to make actual backups before they
need to do maintenance tasks.
Network is already setup when installing with the ISO or on-top of
Debian, so that is not a priority either.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 12:47:05 +02:00
55334cf45a docs: use "Backup Storage" heading
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-14 12:47:05 +02:00
a2e30cd51d ui: tape: rename erase to format
erase is a different action, so correctly call it 'format'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-14 12:25:53 +02:00
4bf2ab1109 cleanup: remove debug println 2021-04-14 10:39:29 +02:00
1dd1c9eb5c api2/tape/restore: restore_chunk_archive: only ignore tape related errors
when we get an error from the tape, we possibly want to ignore it,
i.e. when the file was incomplete, but we still want to error
out if the error came from e.g, the datastore, so we have to move
the error checking code to the 'next_chunk' call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-14 10:38:26 +02:00
6dde015f8c bump version to 1.1.0-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-13 14:42:50 +02:00
5f3b2330c8 docs: typo fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-13 14:42:50 +02:00
4ba5d3b3dd docs: mention client repository and rework client installation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-13 14:42:50 +02:00