4280 Commits

Author SHA1 Message Date
Dominik Csapak
6b8329ee34 tape: simplify export_media_set for pool writer
our export code can handle if the tape is inside the drive, so unloading
it first does not have an benefit, it even makes the exporting slower,
since we first unload it into its original slot, and then moving it
to an import/export slot

so drop the code that unloads the tape from the drive, and let the
export code itself handle that

change the 'eject' into a 'rewind' and comment why we do that first

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-21 06:37:48 +02:00
Dietmar Maurer
934c8724e2 proxmox-rrd: add option to avoid page cache for load/save
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dietmar Maurer
98eb435d90 proxmox-rrd: use syncfs after writing rrd files
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dominik Csapak
ac2ca6c341 tape: improve export_media error message for not found tape
'export_media' can handle if the tape is in either a normal slot of the
library, or in the drive assigned to the current pool writer.
(because we need to lock the drive)

if it is, for some reason, in a different drive, the error message
 'media is not online'
could be slightly confusing for a user, since it would appear in the drive list

add the 'or a differen drive' to make it clearer

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-18 10:40:56 +02:00
Dietmar Maurer
fce7cd0d36 proxmox-rrd: avoild blocking readers while applying the journal
By using and extra RwLock<RRDMap> on the rrd data.
2021-10-15 09:22:07 +02:00
Dietmar Maurer
2e3f94e12f proxmox-rrd: make rrd load callback configurable 2021-10-14 11:41:26 +02:00
Dietmar Maurer
58f70bccbb proxmox-rrd: pass time and value to update function 2021-10-14 08:12:56 +02:00
Dietmar Maurer
fae4f6c509 cleanup: move rrd cache related code into extra file 2021-10-14 07:57:27 +02:00
Dietmar Maurer
c68fa58a59 remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
eb37d4ece2 proxmox-rrd: remove dependency to proxmox-rrd-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
1198f8d4e6 proxmox-rrd: implement new CBOR based format
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
4b709ade68 proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place
allow the current thread to do some other work in-between

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
fa49d0fde9 RRD_CACHE: use a OnceCell instead of lazy_static
And initialize only with proxmox-backup-proxy. Other binaries dont need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
1d44f175c6 proxmox-rrd: use a journal to reduce amount of bytes written
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Wolfgang Bumiller
890b88cbef remove pbs-tools::ops::ControlFlow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-12 14:36:40 +02:00
Dominik Csapak
75442e813e api daemons: fix sending log-reopen command
send_command serializes everything so it cannot be used to send a
raw, optimized command. Normally that means we get an error like
> 'unable to parse parameters (expected json object)'
when used that way.

Switch over to send_raw_command which does not re-serializes the
command.

Fixes: 45b8a032 ("refactor send_command")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-11 14:35:50 +02:00
Wolfgang Bumiller
6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Dominik Csapak
0e1edf19b1 proxmox-backup-proxy: clean up old tasks when the task log was rotated
we maybe have old tasks when the task list was rotated, so clean them up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-08 06:47:38 +02:00
Dietmar Maurer
0191759316 proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
Dietmar Maurer
d1c3bc5350 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
Dietmar Maurer
09340f28f5 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Dietmar Maurer
608806e884 proxmox-rest-server: use new ServerAdapter trait instead of callbacks
Async callbacks are a PITA, so we now pass a single trait object which
implements check_auth and get_index.
2021-10-05 11:13:10 +02:00
Dietmar Maurer
48176b0a77 proxmox-rest-server: pass owned RestEnvironment to get_index
This way we avoid pointers with lifetimes.
2021-10-05 11:12:53 +02:00
Dominik Csapak
0a6df20986 rest-server/daemon: use sd_notify_barrier for service reloading
until now, we manually polled the systemd service state during a reload
so that the sd_notify messages get processed in the correct order
(RELOAD(old) -> MAINPID(old) -> READY(new))

with systemd >= 246 there is now 'sd_notify_barrier' which
blocks until systemd processed all prior messages

with that change, the daemon does not need to know the service name anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-02 11:44:20 +02:00
Dietmar Maurer
6680878b5c proxmox-rest-server: make get_index async 2021-10-01 09:38:10 +02:00
Dietmar Maurer
038f385089 proxmox-rest-server: make check_auth async 2021-10-01 07:53:59 +02:00
Dietmar Maurer
49e25688f1 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
Wolfgang Bumiller
8735247f29 drop fd_change_cloexec from proxmox-rest-server
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-30 12:43:22 +02:00
Dietmar Maurer
0d5d15c9d1 proxmox-rest-server: improve docs
And rename enable_file_log to enable_access_log.
2021-09-30 12:29:15 +02:00
Dietmar Maurer
2e44983a37 proxmox-rest-server: improve docs
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
2021-09-30 10:51:41 +02:00
Thomas Lamprecht
aaf4f40285 subscription: switch verification domain over to shop.proxmox.com
With the merger the shop got moved from shop.maurer-it to
shop.proxmox.com, while we transparently redirect we also want to
stop doing that in a few years, so use new domain.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-30 10:28:53 +02:00
Dietmar Maurer
fd1b65cc3c proxmox-rest-server: allow to catch SIGINT and SIGHUP separately
And make ServerState private.
2021-09-30 08:41:30 +02:00
Dietmar Maurer
38da8ca1bc proxmox-rest-server: improve logging
And rename server_state_init() into catch_shutdown_and_reload_signals().
2021-09-29 14:48:46 +02:00
Wolfgang Bumiller
450105b0c3 make pbs_tools::cert not depend on pbs-buildcfg
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
Wolfgang Bumiller
b62edce929 remove pbs_client::connect_to_localhost
It also used `CertInfo` from pbs-tools which is also server
specific.

The original helper is now in the main crate's
client_helpers instead.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
Dietmar Maurer
bf95fba72e remove wrong calls to systemd_notify
We alrteady call systemd_notify inside the create_service future.
2021-09-29 12:04:48 +02:00
Dietmar Maurer
d265420025 daemon: simlify code (make it easier to use) 2021-09-29 12:04:48 +02:00
Wolfgang Bumiller
01a080215d drop pbs_tools::auth
`pbs_client::connect_to_localhost` now requires the key as
optional parameter

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 11:08:52 +02:00
Dominik Csapak
be5b43cb87 remove tools/async_io.rs
nothing from here is used anymore, so remove it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
Dominik Csapak
6f0565fa60 rest-server: use hypers AddrIncoming for proxmox-backup-api
this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).

this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.

The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
Dietmar Maurer
53daae8e89 proxmox-rest-server: cleanup formatter, improve docs
Use trait for OutputFormatter. This is functionally equivalent,
but more rust-like...
2021-09-28 07:45:50 +02:00
Dietmar Maurer
8a23ea4656 move src/backup/read_chunk.rs to pbs-datastore/src/local_chunk_reader.rs 2021-09-27 11:10:14 +02:00
Dietmar Maurer
c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
Dietmar Maurer
b446fa14c5 WorkerTaskContext: make it Send + Sync 2021-09-27 09:11:38 +02:00
Dietmar Maurer
6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dominik Csapak
af2eb422d5 tools: smart: only throw error for smartctl fatal errors
only bit 0-2 are fatal errors, bit 3-7 are used to indicate
some drive conditions. for details see the manpage of smartctl(8)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ Thomas: resolved merge-conflict due to moved run_command ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-27 08:59:13 +02:00
Dietmar Maurer
bbd57396d7 proxmox-backup-manager: avoid proxmox_rest_server::init_worker_tasks() for "bashcomplete" and "printdoc" 2021-09-24 12:31:42 +02:00
Dietmar Maurer
0fd55b08d9 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer
619cd5cbcb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer
1ec0d70d09 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00