Commit Graph

1118 Commits

Author SHA1 Message Date
Dietmar Maurer
26e949d5fe traffic-control api: return current traffic with config
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-18 12:24:33 +01:00
Thomas Lamprecht
062edce27f group filter: rename CLI/API/Config "groups" option to "group-filter"
we even use that for basically all the related schema names, "groups"
allone is just rather not so telling, i.e., "groups" what?

While due to the additive nature of `group-filter` is not the best
possible name for passing multiple arguments on the CLI (the web-ui
can present this more UX-friendly anyway) due to possible confusion
about if the filter act like AND vs OR it can be documented and even
if a user is confused they still are safe on more being synced than
less. Also, the original param name wasn't really _that_ better in
that regards

Dietmar also suggested to use singular for the CLI option, while
there can be more they're passed over repeating the option, each with
a single filter.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Dominik Csapak
efd2713aa8 proxmox-tape: add groups filter to backup command
and add a completion handler to complete the backup groups

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Dominik Csapak
097ccfe1d5 proxmox-tape: add missing 'notify-user' option to backup command
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler
1b52122a1f manager: render group filter properly
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler
1d9bc184f5 remote: add backup group scanning
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler
71e534631f pull: allow pulling groups selectively
without requiring workarounds based on ownership and limited
visibility/access.

if a group filter is set, remove_vanished will only consider filtered
groups for removal to prevent concurrent disjunct filters from trashing
eachother's synced groups.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Dietmar Maurer
15cc41b6cb proxmox-systemd: remove crate, use new proxmox-time 1.1.0 instead
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-17 13:07:51 +01:00
Dietmar Maurer
a0172d766b traffic-controls: add API/CLI to show current traffic
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-14 17:21:45 +01:00
Dietmar Maurer
d5f58006d3 cached_traffic_control: use ShareableRateLimit trait object 2021-11-13 17:49:38 +01:00
Dietmar Maurer
bf013be1c4 create /var/lib/proxmox-bnackup at server startup
This was missing in previous patch...

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-12 08:12:18 +01:00
Dietmar Maurer
e511e0e553 proxmox-backup-proxy: implement traffic control
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-10 10:15:40 +01:00
Dietmar Maurer
bfd12e871f Add traffic control configuration config with API
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-10 10:15:40 +01:00
Dominik Csapak
9d42e0475b acme: interpret no TOS as accepted
some custom ACME endpoints do not have TOS, interpret this as
'the user has accepted the TOS', like we do for PVE.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-22 10:55:14 +02:00
Dietmar Maurer
98eb435d90 proxmox-rrd: use syncfs after writing rrd files
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dietmar Maurer
fae4f6c509 cleanup: move rrd cache related code into extra file 2021-10-14 07:57:27 +02:00
Dietmar Maurer
1198f8d4e6 proxmox-rrd: implement new CBOR based format
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
4b709ade68 proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place
allow the current thread to do some other work in-between

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
fa49d0fde9 RRD_CACHE: use a OnceCell instead of lazy_static
And initialize only with proxmox-backup-proxy. Other binaries dont need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
1d44f175c6 proxmox-rrd: use a journal to reduce amount of bytes written
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dominik Csapak
75442e813e api daemons: fix sending log-reopen command
send_command serializes everything so it cannot be used to send a
raw, optimized command. Normally that means we get an error like
> 'unable to parse parameters (expected json object)'
when used that way.

Switch over to send_raw_command which does not re-serializes the
command.

Fixes: 45b8a032 ("refactor send_command")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-11 14:35:50 +02:00
Wolfgang Bumiller
6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Dominik Csapak
0e1edf19b1 proxmox-backup-proxy: clean up old tasks when the task log was rotated
we maybe have old tasks when the task list was rotated, so clean them up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-08 06:47:38 +02:00
Dietmar Maurer
09340f28f5 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Dietmar Maurer
608806e884 proxmox-rest-server: use new ServerAdapter trait instead of callbacks
Async callbacks are a PITA, so we now pass a single trait object which
implements check_auth and get_index.
2021-10-05 11:13:10 +02:00
Dietmar Maurer
48176b0a77 proxmox-rest-server: pass owned RestEnvironment to get_index
This way we avoid pointers with lifetimes.
2021-10-05 11:12:53 +02:00
Dominik Csapak
0a6df20986 rest-server/daemon: use sd_notify_barrier for service reloading
until now, we manually polled the systemd service state during a reload
so that the sd_notify messages get processed in the correct order
(RELOAD(old) -> MAINPID(old) -> READY(new))

with systemd >= 246 there is now 'sd_notify_barrier' which
blocks until systemd processed all prior messages

with that change, the daemon does not need to know the service name anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-02 11:44:20 +02:00
Dietmar Maurer
6680878b5c proxmox-rest-server: make get_index async 2021-10-01 09:38:10 +02:00
Dietmar Maurer
49e25688f1 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
Dietmar Maurer
0d5d15c9d1 proxmox-rest-server: improve docs
And rename enable_file_log to enable_access_log.
2021-09-30 12:29:15 +02:00
Dietmar Maurer
fd1b65cc3c proxmox-rest-server: allow to catch SIGINT and SIGHUP separately
And make ServerState private.
2021-09-30 08:41:30 +02:00
Dietmar Maurer
38da8ca1bc proxmox-rest-server: improve logging
And rename server_state_init() into catch_shutdown_and_reload_signals().
2021-09-29 14:48:46 +02:00
Wolfgang Bumiller
450105b0c3 make pbs_tools::cert not depend on pbs-buildcfg
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
Dietmar Maurer
bf95fba72e remove wrong calls to systemd_notify
We alrteady call systemd_notify inside the create_service future.
2021-09-29 12:04:48 +02:00
Dietmar Maurer
d265420025 daemon: simlify code (make it easier to use) 2021-09-29 12:04:48 +02:00
Wolfgang Bumiller
01a080215d drop pbs_tools::auth
`pbs_client::connect_to_localhost` now requires the key as
optional parameter

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 11:08:52 +02:00
Dominik Csapak
6f0565fa60 rest-server: use hypers AddrIncoming for proxmox-backup-api
this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).

this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.

The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
Dietmar Maurer
6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dietmar Maurer
bbd57396d7 proxmox-backup-manager: avoid proxmox_rest_server::init_worker_tasks() for "bashcomplete" and "printdoc" 2021-09-24 12:31:42 +02:00
Dietmar Maurer
1ec0d70d09 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
Dietmar Maurer
ae18c436dd proxmox-backup-manager: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer
b0e20a71e2 proxmox-daily-update: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer
b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer
0a33fba49c worker task: allow to configure path and owner/group
And application now needs to call init_worker_tasks() before using
worker tasks.

Notable changes:
- need to call  init_worker_tasks() before using worker tasks.
- create_task_log_dirs() ís called inside init_worker_tasks()
- removed UpidExt trait
- use atomic_open_or_create_file()
- remove pbs_config and pbs_buildcfg dependency
2021-09-23 11:59:45 +02:00
Dietmar Maurer
049a22a3a3 src/server/worker_task.rs: Avoid using pbs-api-type::Authid
Because we want to move worker_task.rs into proxmox-rest-server crate.
2021-09-23 11:59:25 +02:00
Thomas Lamprecht
a844fa0ba0 move dump-catalog-shell-cli doc-helper to proxmox-backup-client crate
it's only used for generating the docs for the interactive-shell
parts of the client.

Ideally we'd avoid that whole separate binary in the first place and
let the client dump it, but we'd need to have some more elaborate
"hide this command from the help/usage" mechanisms in the CLI
helper/formatter code to make that play out more nicely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-22 16:25:09 +02:00
Thomas Lamprecht
c25ea25f0a debug: api ls: make path optional and default to "/"
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:11:36 +02:00
Dominik Csapak
bfa942c0cf api: make some workers log on CLI
some workers did not log when called via cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:37 +02:00
Dominik Csapak
f54634a890 api: add missing token list match_all property
to have the proper link between the token list and the sub routes
in the api, include the 'tokenname' property in the token listing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:30 +02:00
Dominik Csapak
efb7c5348c proxmox-backup-debug: add 'api' subcommands
this provides some generic api call mechanisms like pvesh/pmgsh.
by default it uses the https api on localhost (creating a token
if called as root, else requesting the root@pam password interactively)

this is mainly intended for debugging, but it is also useful for
situations where some api calls do not have an equivalent in a binary
and a user does not want to go through the api

not implemented are the http2 api calls (since it is a separate api an
it wouldn't be that easy to do)

there are a few quirks though, related to the 'ls' command:
i extract the 'child-link' from the property name of the
'match_all' statement of the router, but this does not
always match with the property from the relevant 'get' api call
so it fails there (e.g. /tape/drive )

this can be fixed in the respective api calls (e.g. by renaming
the parameter that comes from the path)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:30 +02:00