Commit Graph

5935 Commits

Author SHA1 Message Date
Stefan Reiter
9fe3358ce6 file-restore: allow specifying output-format
Makes CLI use more comfortable by not just printing JSON to the
terminal.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:57:57 +02:00
Dominik Csapak
76425d84b3 file-restore: add binary and basic commands
For now it only supports 'list' and 'extract' commands for 'pxar.didx'
files. This should be the foundation for a general file-restore
interface that is shared with block-level snapshots.

This is packaged as a seperate .deb file, since for block level restore
it will need to depend on pve-qemu-kvm, which we want to seperate from
proxmox-backup-client.

[original code for proxmox-file-restore.rs]
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>

[code cleanups/clippy, use helpers::list_dir_content/ArchiveEntry, no
/block subdir for .fidx files, seperate binary and package]
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 13:57:57 +02:00
Thomas Lamprecht
42355b11a4 update d/control
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-08 13:57:57 +02:00
Dominik Csapak
511e4f6987 ui: tape/DriveStatus: improve status grid a bit
by using format_boolean for compression/write protect,
combining file/block posiition into one (saves a line)

and adding the missing alert-flags

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:56:46 +02:00
Dominik Csapak
3f0e344bc1 ui: tape/ChangerStatus: hide selector for single drives in barcode-label
it is rather pointless to let the user select something were there
is no choice. We have to keep the window though, since the user may
want to choose a pool

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:56:46 +02:00
Dominik Csapak
a316178768 ui: tape/ChangerStatus: shortcut Inventory for single drives
like 'load-media'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:56:46 +02:00
Dominik Csapak
dff8ea92aa ui: tape/ChangerStatus: shortcut 'load-media' for single drive
if a changer only has a single drive, there is no point in showing
a window with a DriveSelector, just do want the user wanted.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:56:46 +02:00
Dominik Csapak
88e1f7997c ui: tape/ChangerStatus: rework EraseWindow
to make it more like a 'dangerous' remove window
also works in the singleDrive logic to hide/show the driveselector

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:56:46 +02:00
Dominik Csapak
4c3eabeaf3 ui: tape/ChangerStatus: save assigned drives
so that we can shortcut later if we only have one

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:56:46 +02:00
Dominik Csapak
4c7be5f59d ui: tape/ChangerStatus: add missing property
it will actually not fail, but we declare it nonetheless to indicate
that it exists

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:56:46 +02:00
Dominik Csapak
6d4fbbc3ea ui: dashobard/DataStoreStatistics: add 'Available' column
for some storages, it is valuable information, e.g. if one has datastores
on separate datasets of the same zpool

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 13:27:22 +02:00
Dietmar Maurer
1a23132262 tape: add TapeDensity::Unknown 2021-04-08 12:23:54 +02:00
Dietmar Maurer
48c4193f7c ui: update tape DriveStatus for new driver 2021-04-08 12:04:14 +02:00
Dietmar Maurer
8204d9b095 tape: avoid unneccessary SCSI request in Drop 2021-04-08 11:26:08 +02:00
Dietmar Maurer
fad95a334a tape: clear encryption key after backup (for security reasons) 2021-04-08 10:37:49 +02:00
Dietmar Maurer
973e985d73 cleanup: remove unused linux tape driver code 2021-04-08 10:15:52 +02:00
Dominik Csapak
e5a13382b2 ui: tape/TapeRestore: use correct value check for store & mapping
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 10:02:31 +02:00
Dominik Csapak
81c0b90447 ui: tape/TapeRestore: fix restoring without mapping
we have to delete the 'mapping' variable in any case since it's not
a valid api parameter

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-08 10:02:17 +02:00
Dietmar Maurer
ee9fa953de docs: Mention our new user space tape driver, adopt device path names 2021-04-08 09:50:09 +02:00
Dietmar Maurer
09acf0a70d do not depend on mt-st and mtx
We now use our own driver, so those tools are no longer required.
2021-04-08 09:48:47 +02:00
Dietmar Maurer
15d1435789 tape: add vendor, product and revision to LtoDriveAndMediaStatus 2021-04-08 08:34:46 +02:00
Dietmar Maurer
80ea23e1b9 tape: pmt - implement options command 2021-04-08 08:34:45 +02:00
Dietmar Maurer
5d6379f8db tape: implement locate_file without LOCATE(10) 2021-04-08 08:34:45 +02:00
Dietmar Maurer
566b946f9b tape: pmt - re-implement lock/unlock command 2021-04-08 07:28:30 +02:00
Dietmar Maurer
7f7459677d tape: pmt - re-implement fsr/bsr 2021-04-08 07:28:30 +02:00
Dietmar Maurer
0892a512bc tape: correctly set/display drive option 2021-04-08 07:28:30 +02:00
Dietmar Maurer
b717871d2a sgutils2: add scsi_mode_sense helper 2021-04-08 07:28:30 +02:00
Dietmar Maurer
7b11a8098d tape: make sure there is a filemark at the end of the tape 2021-04-08 07:28:30 +02:00
Dietmar Maurer
8b2c6f5dbc tape: make fsf/bsf driver specific
Because the virtual tape driver behaves different than LTO drives.
2021-04-08 07:28:30 +02:00
Dietmar Maurer
d26985a600 tape: fix LEOM handling 2021-04-08 07:28:30 +02:00
Dietmar Maurer
e29f456efc tape: implement format/erase 2021-04-08 07:28:30 +02:00
Dietmar Maurer
a79082a0dd tape: implement LTO userspace driver 2021-04-08 07:28:30 +02:00
Dietmar Maurer
1336ae8249 tape: introduce trait BlockWrite 2021-04-08 07:28:30 +02:00
Dietmar Maurer
0db5712493 tape: introduce trait BlockRead 2021-04-08 07:28:30 +02:00
Thomas Lamprecht
c47609fedb server: rest: collapse nested if for less indentation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-07 17:57:46 +02:00
Thomas Lamprecht
b84e8aaee9 server: rest: switch from fastest to default deflate compression level
I made some comparision with bombardier[0], the one listed here are
30s looped requests with two concurrent clients:

[ static download of ext-all.js ]:
  lvl                              avg /   stdev  / max
 none        1.98 MiB  100 %    5.17ms /  1.30ms / 32.38ms
 fastest   813.14 KiB   42 %   20.53ms /  2.85ms / 58.71ms
 default   626.35 KiB   30 %   39.70ms /  3.98ms / 85.47ms

[ deterministic (pre-defined data), but real API call ]:
  lvl                              avg /   stdev  / max
 none      129.09 KiB  100 %    2.70ms / 471.58us / 26.93ms
 fastest    42.12 KiB   33 %    3.47ms / 606.46us / 32.42ms
 default    34.82 KiB   27 %    4.28ms / 737.99us / 33.75ms

The reduction is quite better with default, but it's also slower, but
only when testing over unconstrained network. For real world
scenarios where compression actually matters, e.g., when using a
spotty train connection, we will be faster again with better
compression.

A GPRS limited connection (Firefox developer console) requires the
following load (until the DOMContentLoaded event triggered) times:
  lvl        t      x faster
 none      9m 18.6s   x 1.0
 fastest   3m 20.0s   x 2.8
 default   2m 30.0s   x 3.7

So for worst case using sligthly more CPU time on the server has a
tremendous effect on the client load time.

Using a more realistical example and limiting for "Good 2G" gives:

 none      1m  1.8s   x 1.0
 fastest      22.6s   x 2.7
 default      16.6s   x 3.7

16s is somewhat OK, >1m just isn't...

So, use default level to ensure we get bearable load times on
clients, and if we want to improve transmission size AND speed then
we could always use a in-memory cache, only a few MiB would be
required for the compressable static files we server.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-07 17:57:42 +02:00
Dominik Csapak
d84e4073af tools/zip: compress zips with deflate
by using our DeflateEncoder

for this to work, we have to create wrapper reader that generates the crc32
checksum while reading.

also we need to put the target writer in an Option, so that we can take
it out of self and move it into the DeflateEncoder while writing
compressed

we can drop the internal buffer then, since that is managed by the
deflate encoder now

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
e8656da70d tools/zip: run rustfmt
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
59477ad252 server/rest: compress static files
compress them on the fly
and refactor the size limit for chunking files

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
2f29f1c765 server/rest: compress api calls
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
4d84e869bf server/rest: add helper to extract compression headers
for now we only extract 'deflate'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
79d841014e tools/compression: add DeflateEncoder and helpers
implements a deflate encoder that can compress anything that implements
AsyncRead + Unpin into a file with the helper 'compress'

if the inner type is a Stream, it implements Stream itself, this way
some streaming data can be streamed compressed

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
ea62611d8e tools: add compression module
only contains a basic enum for the different compresssion methods

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Thomas Lamprecht
f3c867a034 docs: fix MathJax inclusion in build
Else it does not gets picked up on release builds...

Also the mathjax path option affects HTML not EPUB so move it to the
correct section in conf.py

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-05 11:59:45 +02:00
Thomas Lamprecht
aae5db916e docs: fix heading
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-05 11:59:18 +02:00
Thomas Lamprecht
a417c8a93e bump version to 1.0.13-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-02 15:32:27 +02:00
Wolfgang Bumiller
79e58a903e pxar: handle missing GROUP_OBJ ACL entries
Previously we did not store GROUP_OBJ ACL entries for
directories, this means that these were lost which may
potentially elevate group permissions if they were masked
before via ACLs, so we also show a warning.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-02 11:10:20 +02:00
Wolfgang Bumiller
9f40e09d0a pxar: fix directory ACL entry creation
Don't override `group_obj` with `None` when handling
`ACL_TYPE_DEFAULT` entries for directories.

Reproducer: /var/log/journal ends up without a `MASK` type
entry making it invalid as it has `USER` and `GROUP`
entries.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-02 10:22:04 +02:00
Thomas Lamprecht
553e57f914 server/rest: drop now unused imports
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-01 11:53:13 +02:00
Thomas Lamprecht
2200a38671 code cleanup: drop extra newlines at EOF
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-01 11:27:07 +02:00