Commit Graph

473 Commits

Author SHA1 Message Date
Thomas Lamprecht
888d89e2dd temporarily disable broken test
this test was added before the used NodeConfig schema was committed,
cannot work...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-29 16:18:20 +02:00
Wolfgang Bumiller
bc5c1a9aa6 add 'config file format' to tools::config
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-29 12:11:54 +02:00
Wolfgang Bumiller
134ed9e14f CertInfo: add is_expired_after_epoch
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-29 11:07:04 +02:00
Wolfgang Bumiller
0796b642de CertInfo: add not_{after, before}_unix
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-29 10:59:02 +02:00
Wolfgang Bumiller
a576e6685b tools::fs::scan_subdir: use nix::Error instead of anyhow
allows using SysError trait on it

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-28 12:32:57 +02:00
Wolfgang Bumiller
b1c793cfa5 systemd: add reload_unit
via try-reload-or-restart

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-04-28 12:15:26 +02:00
Dietmar Maurer
c0147e49c4 tools/http: make user agent configurable 2021-04-28 12:15:26 +02:00
Dietmar Maurer
d52b120905 tools/http: set USER_AGENT inside request 2021-04-28 12:15:26 +02:00
Dietmar Maurer
7a7fcb4715 http: add helper to parse proxy configuration 2021-04-28 11:23:06 +02:00
Dietmar Maurer
cf8e44bc30 HttpsConnector: add proxy authorization support 2021-04-28 11:23:06 +02:00
Dominik Csapak
8fd1e10830 tools/sgutils2: add size workaround for mode_sense
Some drives will always return the number of bytes given in the
allocation_length field, but correctly report the data len in the mode
sense header. Simply ignore the excess data.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-27 11:37:03 +02:00
Dietmar Maurer
12509a6d9e tape: improve inline docs 2021-04-27 11:37:03 +02:00
Dietmar Maurer
73cef112eb tape: remove MediumType struct, which is only valid on IBM drives
HP drives do not return this information.

Note: This breaks format on WORM media, because we have not way
to detect WOREM media (how?).
2021-04-27 09:58:27 +02:00
Dominik Csapak
6ee69fccd3 tools/sgutils2: improve error messages
include the expected and unexpected sizes in the error message,
so that it's easier to debug in case of an error

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-27 08:24:50 +02:00
Dietmar Maurer
1ed9069ad3 http proxy: improve response parser
Avoid strange error message in case of connect error (only parse status + headers).
We are not interested in the response body, so simply ignore it.
2021-04-26 11:21:11 +02:00
Dietmar Maurer
a3b4b5b50e tape: define and use MediumType enum 2021-04-23 07:54:42 +02:00
Dietmar Maurer
33b8d7e5e8 tape: use loaded media_type in format_media (instead of drive_density)
Required to format LTO4 media loaded in LTO5 drive).

Also contains some SCSI code cleanups.
2021-04-23 07:27:30 +02:00
Dietmar Maurer
b6c06dce9d http proxy: implement read_connect_response()
Limit memory usage in case we get strange data from proxy.
2021-04-22 10:06:14 +02:00
Dietmar Maurer
9104152a83 HttpsConnector: add proxy support 2021-04-21 15:29:17 +02:00
Dietmar Maurer
02a58862dd HttpsConnector: code cleanup 2021-04-21 15:29:17 +02:00
Dietmar Maurer
26153589ba new http client implementation SimpleHttp (avoid static HTTP_CLIENT)
This one will have proxy support.
2021-04-21 15:29:17 +02:00
Dietmar Maurer
17b3e4451f MaybeTlsStream: implement poll_write_vectored()
This is just an performance optimization.
2021-04-21 15:29:17 +02:00
Dietmar Maurer
a2072cc346 http: rename EitherStream to MaybeTlsStream
And rename the enum values. Added an additional enum called Proxied.

The enum in now more specialized, but we only use it for the http client anyways.
2021-04-21 15:29:17 +02:00
Dominik Csapak
fea23d0323 fix #3393: tools/xattr: allow xattr 'security.NTACL'
in some configurations, samba stores NTFS-ACLs in this xattr[0], so
we should backup (if we can)

altough the 'security' namespace is special (e.g. in use by
selinux, etc.) this value is normally only used by samba and we
should be able to back it up.

to restore it, the user needs at least 'CAP_SYS_ADMIN' rights, otherwise
it cannot be set

0: https://www.samba.org/samba/docs/current/man-html/vfs_acl_xattr.8.html

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-21 14:49:46 +02:00
Dominik Csapak
0417e9af1b tools/async_io: do not error on Accept for StaticIncoming
in proxmox-backup-proxy, we log and discard any errors on 'accept',
so that we can continue to server requests

in proxmox-backup-api, we just have the StaticIncoming that accepts,
which will forward any errors from the underlying TcpListener

this patch also logs and discards the errors, like in the proxy.
Otherwise it could happen that if the api-daemon has more files open
than the proxy, it will shut itself down because of a
'too many open files' error if there are many open connections

(the service should also restart on exit i think, but this is
a separate issue)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-12 15:43:13 +02:00
Dominik Csapak
368f4c5416 fix gathering io stats for zpools
if a datastore or root is not used directly on the pool dir
(e.g. the installer creates 2 sub datasets ROOT/pbs-1), info in
/proc/self/mountinfo returns not the pool, but the path to the
dataset, which has no iostats itself in /proc/spl/kstat/zfs/
but only the pool itself

so instead of not gathering data at all, gather the info from the
underlying pool instead. if one has multiple datastores on the same
pool those rrd stats will be the same for all those datastores now
(instead of empty) similar to 'normal' directories

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-12 13:35:38 +02:00
Dietmar Maurer
164ad7b706 sgutils2: use thiserror to derive Error 2021-04-12 13:27:34 +02:00
Dietmar Maurer
44f6bb019c sgutils2: implement scsi_request_sense() 2021-04-09 09:46:19 +02:00
Stefan Reiter
1f03196c0b tools/zip: add zip_directory helper
Encodes an entire local directory into an AsyncWrite recursively.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-04-08 14:32:03 +02:00
Stefan Reiter
a5bdc987dc add tools/cpio encoding module
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-04-08 14:10:45 +02:00
Dietmar Maurer
b717871d2a sgutils2: add scsi_mode_sense helper 2021-04-08 07:28:30 +02:00
Dietmar Maurer
d26985a600 tape: fix LEOM handling 2021-04-08 07:28:30 +02:00
Dominik Csapak
d84e4073af tools/zip: compress zips with deflate
by using our DeflateEncoder

for this to work, we have to create wrapper reader that generates the crc32
checksum while reading.

also we need to put the target writer in an Option, so that we can take
it out of self and move it into the DeflateEncoder while writing
compressed

we can drop the internal buffer then, since that is managed by the
deflate encoder now

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
e8656da70d tools/zip: run rustfmt
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
79d841014e tools/compression: add DeflateEncoder and helpers
implements a deflate encoder that can compress anything that implements
AsyncRead + Unpin into a file with the helper 'compress'

if the inner type is a Stream, it implements Stream itself, this way
some streaming data can be streamed compressed

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dominik Csapak
ea62611d8e tools: add compression module
only contains a basic enum for the different compresssion methods

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-04-07 12:34:31 +02:00
Dietmar Maurer
cf1e117fc7 sgutils2: use enum for ScsiError
This avoids string allocation when we return SenseInfo.
2021-03-27 15:57:48 +01:00
Dietmar Maurer
03eac20b87 SgRaw: add do_in_command() 2021-03-27 15:38:08 +01:00
Dominik Csapak
f1d76ecf6c fix #3359: fix blocking writes in async code during pxar create
in commit `asyncify pxar create_archive`, we changed from a
separate thread for creating a pxar to using async code, but the
StdChannelWriter used for both pxar and catalog can block, which
may block the tokio runtime for single (and probably dual) core
environments

this patch adds a wrapper struct for any writer that implements
'std::io::Write' and wraps the write calls with 'block_in_place'
so that if called in a tokio runtime, it knows that this code
potentially blocks

Fixes: 6afb60abf5 ("asyncify pxar create_archive")

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-24 09:00:07 +01:00
Dominik Csapak
2729d134bd tools/systemd/time: implement some Traits for TimeSpan
namely
* From<Duration> (to convert easily from duration to timespan)
* Display (for better formatting)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-19 07:00:55 +01:00
Dominik Csapak
caf76ec592 tools/subscription: ignore ENOENT for apt auth config removal
deleting a nonexistant file is hardly an error worth mentioning

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-17 20:12:58 +01:00
Dominik Csapak
7914e62b10 tools/zip: only add zip64 field when necessary
if neither offset nor size exceeds 32bit, do not add the
zip64 extension field

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-16 09:13:39 +01:00
Dominik Csapak
7c1666289d tools/zip: add missing start_disk field for zip64 extension
it is not optional, even though we give the size explicitely

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-03-15 12:36:40 +01:00
Thomas Lamprecht
d1d74c4367 typo fixes all over the place
found and semi-manually replaced by using:
 codespell -L mut -L crate -i 3 -w

Mostly in comments, but also email notification and two occurrences
of misspelled  'reserved' struct member, which where not used and
cargo build did not complain about the change, soo ...

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-10 16:39:57 +01:00
Thomas Lamprecht
3bb7e62e88 zpool parsers: small code cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-03 14:05:05 +01:00
Thomas Lamprecht
3b060167f6 zpool parsers: fixup test value
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-03 14:05:05 +01:00
Thomas Lamprecht
d64226efee disks/zfs: fix test input and enforce check for config key
we test for the config key in the API so it makes sense to have as
test here too. Actually it would be better if we'd have a expect
Value defined here and enforce that it matches, but better than
nothing.

Fix the input for test 1, where tabs got replaced by spaces, as else
it fails

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-01 17:36:16 +01:00
Thomas Lamprecht
2440eaa2df disks/zfs: pull common test code out in helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-03-01 14:48:35 +01:00
Wolfgang Bumiller
fa016c1697 HttpsConnector: use hostname instead of URL again
fixes connecting to hosts with valid certificates without a
pinned fingerprint
this was accidentally changed in the tokio-1.0 updates
apparently

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Fixes: 0f860f712f ("tokio 1.0: update to new tokio-openssl interface")
2021-02-03 15:18:18 +01:00
Fabian Grünbichler
01c023d50f paperkey: rustfmt
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-02-01 17:05:40 +01:00