Commit Graph

1298 Commits

Author SHA1 Message Date
Fabian Grünbichler
1d9bc184f5 remote: add backup group scanning
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler
5f83d3f636 sync: add group filtering
like for manual pulls, but persisted in the sync job config and visible
in the relevant GUI parts.

GUI is read-only for now (and defaults to no filtering on creation), as
this is a rather advanced feature that requires a complex GUI to be
user-friendly (regex-freeform, type-combobox, remote group scanning +
selector with additional freeform input).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler
71e534631f pull: allow pulling groups selectively
without requiring workarounds based on ownership and limited
visibility/access.

if a group filter is set, remove_vanished will only consider filtered
groups for removal to prevent concurrent disjunct filters from trashing
eachother's synced groups.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler
6e9e6c7a54 pull/sync: extract passed along vars into struct
this is basically the sync job config without ID and some stuff
converted already, and a convenient helper to generate the http client
from it.

Suggested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Dietmar Maurer
15cc41b6cb proxmox-systemd: remove crate, use new proxmox-time 1.1.0 instead
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-17 13:07:51 +01:00
Wolfgang Bumiller
9407810fe1 switch tfa api to use proxmox-tfa::api
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-11-17 09:33:04 +01:00
Dietmar Maurer
c42a54795d move fingerprint helpers from pbs-tools to pbs-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-17 07:07:40 +01:00
Dietmar Maurer
24f9af9e0f add missing file from previous commit
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-14 18:49:29 +01:00
Dietmar Maurer
a0172d766b traffic-controls: add API/CLI to show current traffic
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-14 17:21:45 +01:00
Dietmar Maurer
bfd12e871f Add traffic control configuration config with API
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-10 10:15:40 +01:00
Dietmar Maurer
68fd9ca6d6 openid_login: vertify that firstname, lastname and email fits our schema definitions
If not, we do not copy the values to our user.cfg.
2021-11-10 06:48:40 +01:00
Dietmar Maurer
4beb7d2dbe correctly lock remote config
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-06 17:35:10 +01:00
Dominik Csapak
9b1e2ae83c api: admin/datastore: reuse 'is_protected' implementation
we already have that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 12:54:54 +02:00
Dominik Csapak
8292d3d20e api2/admin/datastore: add get/set_protection
for gettin/setting the protected flag for snapshots (akin to notes)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:31:11 +02:00
Dominik Csapak
5cc7d89139 api2: datastore/delete_group: throw error for partially removed group
when a group could not be completely removed due to protected snapshot,
throw an error

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:31:07 +02:00
Dominik Csapak
02db72678f add protected info of snapshots to api and task logs
adds the info that a snapshot is protected to:
* snapshot list
* manual pruning (also dry-run)
* prune jobs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:51 +02:00
Dietmar Maurer
e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
Dietmar Maurer
fae4f6c509 cleanup: move rrd cache related code into extra file 2021-10-14 07:57:27 +02:00
Dietmar Maurer
c68fa58a59 remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
eb37d4ece2 proxmox-rrd: remove dependency to proxmox-rrd-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
1198f8d4e6 proxmox-rrd: implement new CBOR based format
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer
fa49d0fde9 RRD_CACHE: use a OnceCell instead of lazy_static
And initialize only with proxmox-backup-proxy. Other binaries dont need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Wolfgang Bumiller
890b88cbef remove pbs-tools::ops::ControlFlow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-12 14:36:40 +02:00
Wolfgang Bumiller
6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Dietmar Maurer
0191759316 proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
Dietmar Maurer
d1c3bc5350 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
Dietmar Maurer
09340f28f5 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Wolfgang Bumiller
8735247f29 drop fd_change_cloexec from proxmox-rest-server
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-30 12:43:22 +02:00
Dietmar Maurer
2e44983a37 proxmox-rest-server: improve docs
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
2021-09-30 10:51:41 +02:00
Wolfgang Bumiller
450105b0c3 make pbs_tools::cert not depend on pbs-buildcfg
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
Wolfgang Bumiller
01a080215d drop pbs_tools::auth
`pbs_client::connect_to_localhost` now requires the key as
optional parameter

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 11:08:52 +02:00
Dietmar Maurer
53daae8e89 proxmox-rest-server: cleanup formatter, improve docs
Use trait for OutputFormatter. This is functionally equivalent,
but more rust-like...
2021-09-28 07:45:50 +02:00
Dietmar Maurer
8a23ea4656 move src/backup/read_chunk.rs to pbs-datastore/src/local_chunk_reader.rs 2021-09-27 11:10:14 +02:00
Dietmar Maurer
c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
Dietmar Maurer
6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dietmar Maurer
619cd5cbcb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer
1ec0d70d09 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
Dietmar Maurer
c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer
f7348a23cd move src/server/h2service.rs into proxmox-rest-server crate 2021-09-24 10:28:17 +02:00
Dietmar Maurer
b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer
81867f0539 use UPID and systemd helpers from proxmox 0.13.4 2021-09-23 12:01:43 +02:00
Dietmar Maurer
0a33fba49c worker task: allow to configure path and owner/group
And application now needs to call init_worker_tasks() before using
worker tasks.

Notable changes:
- need to call  init_worker_tasks() before using worker tasks.
- create_task_log_dirs() ís called inside init_worker_tasks()
- removed UpidExt trait
- use atomic_open_or_create_file()
- remove pbs_config and pbs_buildcfg dependency
2021-09-23 11:59:45 +02:00
Dietmar Maurer
049a22a3a3 src/server/worker_task.rs: Avoid using pbs-api-type::Authid
Because we want to move worker_task.rs into proxmox-rest-server crate.
2021-09-23 11:59:25 +02:00
Dominik Csapak
bfa942c0cf api: make some workers log on CLI
some workers did not log when called via cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:37 +02:00
Dominik Csapak
f54634a890 api: add missing token list match_all property
to have the proper link between the token list and the sub routes
in the api, include the 'tokenname' property in the token listing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:30 +02:00
Dietmar Maurer
84af82e8cf rename pbs-systemd to proxmox-systemd 2021-09-21 10:06:27 +02:00
Dietmar Maurer
fd18775ac1 worker_state: move tasktype() code to src/api2/node/tasks.rs
Because this is API related code, and only used there.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:47:48 +02:00
Dietmar Maurer
e678a50ea1 buildsys: drop double-build hack to avoid linkage issues
basically a (semantic) revert of commit
991be99c37 "buildsys: workaround
linkage issues from openid/curl build server stuff separate"

This is no longer required because we moved proxmox_restore_daemon
code into extra crate (previous commit)

Originally-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
36b7085ec2 rest server: cleanup auth-log handling
Handle auth logs the same way as access log.
- Configure with ApiConfig
- CommandoSocket command to reload auth-logs "api-auth-log-reopen"

Inside API calls, we now access the ApiConfig using the RestEnvironment.

The openid_login api now also logs failed logins and return http_err!(UNAUTHORIZED, ..)
on failed logins.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
1b552c109d move src/server/formatter.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
8bca935f08 move src/tools/daemon.rs to proxmox-rest-server workspace
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer
fd6d243843 move ApiConfig, FileLogger and CommandoSocket to proxmox-rest-server workspace
ApiConfig: avoid using  pbs_config::backup_user()
CommandoSocket: avoid using  pbs_config::backup_user()
FileLogger: avoid using  pbs_config::backup_user()
- use atomic_open_or_create_file()

Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dominik Csapak
8eef31724f api: disks/directory: add 'name' property to list of mounts
so that we actually have the property that 'match_all' refers to for
the templated API path.

This is mostly for improving usage of the WIP pbs-shell, i.e., its
`ls` command, it has no other functional/semantic impact.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-15 11:59:02 +02:00
Dominik Csapak
2de1b06a06 api: disks/directory: factor out BASE_MOUNT_DIR path
will be reused in the next commit

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-15 11:54:18 +02:00
Thomas Lamprecht
a332040a7f api: nodes: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-15 11:42:28 +02:00
Dominik Csapak
957133077f api2: nodes: add missing node list api call
to have an api call for api path traversal

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-09-15 11:32:58 +02:00
Dietmar Maurer
ccc3896ff3 avoid type re-exports 2021-09-14 08:35:43 +02:00
Dietmar Maurer
cef5c72682 move src/tape/helpers/snapshot_reader.rs to src/backup/snapshot_reader.rs 2021-09-14 07:42:06 +02:00
Dietmar Maurer
048b43af24 split tape code into new pbs_tape workspace 2021-09-13 12:54:59 +02:00
Dietmar Maurer
67a5cf4714 fix regression tests 2021-09-10 12:45:06 +02:00
Dietmar Maurer
6227654ad8 more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dietmar Maurer
89725197c0 move PruneOptions to pbs_api_types workspace 2021-09-10 09:21:27 +02:00
Dietmar Maurer
e7d4be9d85 move datastore config to pbs_config workspace 2021-09-10 08:40:58 +02:00
Dietmar Maurer
ba3d7e19fb move user configuration to pbs_config workspace
Also moved memcom.rs and cached_user_info.rs
2021-09-10 07:09:04 +02:00
Dietmar Maurer
b65dfff574 cleanup User configuration: use Updater 2021-09-09 13:14:28 +02:00
Dietmar Maurer
8cc3760e74 move acl to pbs_config workspaces, pbs_api_types cleanups 2021-09-09 10:50:08 +02:00
Dietmar Maurer
1cb08a0a05 move token_shadow to pbs_config workspace
Also moved out crypt.rs (libcrypt bindings) to pbs_tools workspace.
2021-09-08 14:00:14 +02:00
Dietmar Maurer
6f4228809e move network config to pbs_config workspace 2021-09-08 12:22:48 +02:00
Dietmar Maurer
5af3bcf062 changer config cleanup: use Updater 2021-09-08 09:29:01 +02:00
Dietmar Maurer
cdc83c4eb2 tape job cleanup: user Updater 2021-09-08 08:55:55 +02:00
Dietmar Maurer
ffa403b5fd verify job cleanup: use Updater/flatten 2021-09-08 08:40:32 +02:00
Dietmar Maurer
5bd77f00e2 sync job cleanup: use Updater/flatten 2021-09-08 08:28:09 +02:00
Dietmar Maurer
802189f7f5 move verify.rs to pbs_config workspace 2021-09-08 08:01:07 +02:00
Dietmar Maurer
a4e5a0fc9f move sync.rs to pbs_config workspace 2021-09-08 06:57:23 +02:00
Dietmar Maurer
e3619d4101 moved tape_job.rs to pbs_config workspace 2021-09-07 12:40:15 +02:00
Dietmar Maurer
5839c469c1 move tape_encryption_keys.rs to pbs_config workspace 2021-09-07 10:37:08 +02:00
Dietmar Maurer
bbdda58b35 moved key_derivation.rs from pbs_datastore to pbs-config/src/key_config.rs
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
2021-09-07 10:12:17 +02:00
Dietmar Maurer
45d5d873ce move Kdf and KeyInfo to pbs_api_types workspace 2021-09-07 09:59:59 +02:00
Dietmar Maurer
aad2d162ab move media_pool config to pbs_config workspace 2021-09-06 08:56:04 +02:00
Dietmar Maurer
1ce8e905ea move drive config to pbs_config workspace
Also moved the tape type definitions to pbs_api_types.
2021-09-03 09:10:18 +02:00
Dietmar Maurer
6afdda8832 move remote config into pbs-config workspace 2021-09-02 14:25:15 +02:00
Dietmar Maurer
2121174827 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Wolfgang Bumiller
4c1b776168 another import cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 14:46:01 +02:00
Wolfgang Bumiller
dd2162f6bd more import cleanups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 14:01:03 +02:00
Wolfgang Bumiller
86582454e8 make api2::helpers::list_dir_content a CatalogReader method
this is its natural place and everything required is already
part of the catalog module

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller
013b1e8bca move some more API types
ArchiveEntry -> pbs-datastore
RestoreDaemonStatus -> pbs-api-types

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller
b2065dc7d2 cleanup proxmox_backup::backup module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 14:14:04 +02:00
Dietmar Maurer
97dfc62f0d remote config: derive and use Updater
Defined a new struct RemoteConfig (without name and password). This makes it
possible to bas64-encode the pasword in the config, but still allow plain
passwords with the API.
2021-08-30 12:48:45 +02:00
Wolfgang Bumiller
7b570c177d move some API return types to pbs-api-types
they'll be required by the api client

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 11:39:01 +02:00
Dietmar Maurer
dbda1513c5 tape: media_pool: derive and use Updater 2021-08-30 11:17:14 +02:00
Dietmar Maurer
c62a6acb2e drive config cleanup: derive and use Updater 2021-08-30 10:50:20 +02:00
Dietmar Maurer
e4a5c072b4 openid cleanup: derive and use Updater 2021-08-30 09:48:53 +02:00
Wolfgang Bumiller
a37c8d2431 use ApiType trait
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-25 10:43:58 +02:00
Dietmar Maurer
a8a20e9210 use new api updater features
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-25 10:43:58 +02:00
Dominik Csapak
cffe0b81e3 tape backup: mention groups that were empty
otherwise a user might get a task log like this:

-----
...
found 7 groups
TASK OK
-----

which could confuse the users as why there were no snapshots backed up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-09 09:28:01 +02:00
Dietmar Maurer
efe96ec039 tape: compute next-media-label for each tape backup job 2021-08-04 09:59:12 +02:00
Dominik Csapak
8be69a8453 api/ui: allow zstd compression for new zpools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-30 17:51:13 +02:00
Fabian Ebner
0b12a5a698 api: apt: adapt to further proxmox-apt back-end changes
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-30 10:37:27 +02:00
Fabian Ebner
2eac359430 api: apt: adapt to proxmox-apt back-end changes
It's up to the caller to provide the current release for standard
repository detection/addition.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-30 10:37:27 +02:00
Dominik Csapak
855b55dc14 api2: tape: media: use MediaCatalog::snapshot_list for content listing
this should make the api call much faster, since it is not reading
the whole catalog anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-07-29 13:34:36 +02:00