Commit Graph

4320 Commits

Author SHA1 Message Date
1198f8d4e6 proxmox-rrd: implement new CBOR based format
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
4b709ade68 proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place
allow the current thread to do some other work in-between

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
fa49d0fde9 RRD_CACHE: use a OnceCell instead of lazy_static
And initialize only with proxmox-backup-proxy. Other binaries dont need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
1d44f175c6 proxmox-rrd: use a journal to reduce amount of bytes written
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
890b88cbef remove pbs-tools::ops::ControlFlow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-12 14:36:40 +02:00
75442e813e api daemons: fix sending log-reopen command
send_command serializes everything so it cannot be used to send a
raw, optimized command. Normally that means we get an error like
> 'unable to parse parameters (expected json object)'
when used that way.

Switch over to send_raw_command which does not re-serializes the
command.

Fixes: 45b8a032 ("refactor send_command")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-11 14:35:50 +02:00
6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
0e1edf19b1 proxmox-backup-proxy: clean up old tasks when the task log was rotated
we maybe have old tasks when the task list was rotated, so clean them up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-08 06:47:38 +02:00
0191759316 proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
d1c3bc5350 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
09340f28f5 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
608806e884 proxmox-rest-server: use new ServerAdapter trait instead of callbacks
Async callbacks are a PITA, so we now pass a single trait object which
implements check_auth and get_index.
2021-10-05 11:13:10 +02:00
48176b0a77 proxmox-rest-server: pass owned RestEnvironment to get_index
This way we avoid pointers with lifetimes.
2021-10-05 11:12:53 +02:00
0a6df20986 rest-server/daemon: use sd_notify_barrier for service reloading
until now, we manually polled the systemd service state during a reload
so that the sd_notify messages get processed in the correct order
(RELOAD(old) -> MAINPID(old) -> READY(new))

with systemd >= 246 there is now 'sd_notify_barrier' which
blocks until systemd processed all prior messages

with that change, the daemon does not need to know the service name anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-02 11:44:20 +02:00
6680878b5c proxmox-rest-server: make get_index async 2021-10-01 09:38:10 +02:00
038f385089 proxmox-rest-server: make check_auth async 2021-10-01 07:53:59 +02:00
49e25688f1 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
8735247f29 drop fd_change_cloexec from proxmox-rest-server
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-30 12:43:22 +02:00
0d5d15c9d1 proxmox-rest-server: improve docs
And rename enable_file_log to enable_access_log.
2021-09-30 12:29:15 +02:00
2e44983a37 proxmox-rest-server: improve docs
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
2021-09-30 10:51:41 +02:00
aaf4f40285 subscription: switch verification domain over to shop.proxmox.com
With the merger the shop got moved from shop.maurer-it to
shop.proxmox.com, while we transparently redirect we also want to
stop doing that in a few years, so use new domain.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-30 10:28:53 +02:00
fd1b65cc3c proxmox-rest-server: allow to catch SIGINT and SIGHUP separately
And make ServerState private.
2021-09-30 08:41:30 +02:00
38da8ca1bc proxmox-rest-server: improve logging
And rename server_state_init() into catch_shutdown_and_reload_signals().
2021-09-29 14:48:46 +02:00
450105b0c3 make pbs_tools::cert not depend on pbs-buildcfg
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
b62edce929 remove pbs_client::connect_to_localhost
It also used `CertInfo` from pbs-tools which is also server
specific.

The original helper is now in the main crate's
client_helpers instead.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
bf95fba72e remove wrong calls to systemd_notify
We alrteady call systemd_notify inside the create_service future.
2021-09-29 12:04:48 +02:00
d265420025 daemon: simlify code (make it easier to use) 2021-09-29 12:04:48 +02:00
01a080215d drop pbs_tools::auth
`pbs_client::connect_to_localhost` now requires the key as
optional parameter

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 11:08:52 +02:00
be5b43cb87 remove tools/async_io.rs
nothing from here is used anymore, so remove it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
6f0565fa60 rest-server: use hypers AddrIncoming for proxmox-backup-api
this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).

this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.

The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
53daae8e89 proxmox-rest-server: cleanup formatter, improve docs
Use trait for OutputFormatter. This is functionally equivalent,
but more rust-like...
2021-09-28 07:45:50 +02:00
8a23ea4656 move src/backup/read_chunk.rs to pbs-datastore/src/local_chunk_reader.rs 2021-09-27 11:10:14 +02:00
c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
b446fa14c5 WorkerTaskContext: make it Send + Sync 2021-09-27 09:11:38 +02:00
6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
af2eb422d5 tools: smart: only throw error for smartctl fatal errors
only bit 0-2 are fatal errors, bit 3-7 are used to indicate
some drive conditions. for details see the manpage of smartctl(8)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ Thomas: resolved merge-conflict due to moved run_command ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-27 08:59:13 +02:00
bbd57396d7 proxmox-backup-manager: avoid proxmox_rest_server::init_worker_tasks() for "bashcomplete" and "printdoc" 2021-09-24 12:31:42 +02:00
0fd55b08d9 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
619cd5cbcb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
1ec0d70d09 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
f7348a23cd move src/server/h2service.rs into proxmox-rest-server crate 2021-09-24 10:28:17 +02:00
ae18c436dd proxmox-backup-manager: setup worker and command socket 2021-09-24 10:28:17 +02:00
b0e20a71e2 proxmox-daily-update: setup worker and command socket 2021-09-24 10:28:17 +02:00
b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
81867f0539 use UPID and systemd helpers from proxmox 0.13.4 2021-09-23 12:01:43 +02:00
0a33fba49c worker task: allow to configure path and owner/group
And application now needs to call init_worker_tasks() before using
worker tasks.

Notable changes:
- need to call  init_worker_tasks() before using worker tasks.
- create_task_log_dirs() ís called inside init_worker_tasks()
- removed UpidExt trait
- use atomic_open_or_create_file()
- remove pbs_config and pbs_buildcfg dependency
2021-09-23 11:59:45 +02:00
049a22a3a3 src/server/worker_task.rs: Avoid using pbs-api-type::Authid
Because we want to move worker_task.rs into proxmox-rest-server crate.
2021-09-23 11:59:25 +02:00
a844fa0ba0 move dump-catalog-shell-cli doc-helper to proxmox-backup-client crate
it's only used for generating the docs for the interactive-shell
parts of the client.

Ideally we'd avoid that whole separate binary in the first place and
let the client dump it, but we'd need to have some more elaborate
"hide this command from the help/usage" mechanisms in the CLI
helper/formatter code to make that play out more nicely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-22 16:25:09 +02:00
c25ea25f0a debug: api ls: make path optional and default to "/"
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:11:36 +02:00