Commit Graph

4413 Commits

Author SHA1 Message Date
Fabian Grünbichler 61ef4ae8cb fix #sync.cfg/pull: don't remove by default
and convert existing (manually created/edited) jobs to the previous
default value of 'true'. the GUI has always set this value and defaults
to 'false'.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler 1b52122a1f manager: render group filter properly
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler 1d9bc184f5 remote: add backup group scanning
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler 5f83d3f636 sync: add group filtering
like for manual pulls, but persisted in the sync job config and visible
in the relevant GUI parts.

GUI is read-only for now (and defaults to no filtering on creation), as
this is a rather advanced feature that requires a complex GUI to be
user-friendly (regex-freeform, type-combobox, remote group scanning +
selector with additional freeform input).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler 71e534631f pull: allow pulling groups selectively
without requiring workarounds based on ownership and limited
visibility/access.

if a group filter is set, remove_vanished will only consider filtered
groups for removal to prevent concurrent disjunct filters from trashing
eachother's synced groups.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler 6e9e6c7a54 pull/sync: extract passed along vars into struct
this is basically the sync job config without ID and some stuff
converted already, and a convenient helper to generate the http client
from it.

Suggested-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Fabian Grünbichler e2e7560d5e pull: use BackupGroup consistently
instead of `GroupListItem`s. we convert it anyway, so might as well do
that at the start.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Reviewed-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-11-18 10:36:57 +01:00
Dietmar Maurer 15cc41b6cb proxmox-systemd: remove crate, use new proxmox-time 1.1.0 instead
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-17 13:07:51 +01:00
Wolfgang Bumiller 729bd1fd16 remove now unused serde_filter module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-11-17 09:50:08 +01:00
Wolfgang Bumiller 9407810fe1 switch tfa api to use proxmox-tfa::api
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-11-17 09:33:04 +01:00
Dietmar Maurer c42a54795d move fingerprint helpers from pbs-tools to pbs-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-17 07:07:40 +01:00
Dietmar Maurer 24f9af9e0f add missing file from previous commit
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-14 18:49:29 +01:00
Dietmar Maurer a0172d766b traffic-controls: add API/CLI to show current traffic
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-14 17:21:45 +01:00
Dietmar Maurer 09f999337a update to proxmox-http 0.5.4
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-14 08:27:45 +01:00
Dietmar Maurer e3eb062c09 cached_traffic_control: fix regression tests
Avoid using shared memory in tests because of permission problems.
2021-11-14 08:05:40 +01:00
Dietmar Maurer de21d4efdc implement rate limiter in shared memory
This kind of rate limiter can be used among several processes (as long
as all set the same rate/burst).
2021-11-13 17:49:38 +01:00
Dietmar Maurer d5f58006d3 cached_traffic_control: use ShareableRateLimit trait object 2021-11-13 17:49:38 +01:00
Dietmar Maurer cb80ffc1de pbs-config: use new SharedMemory helpers from proxmox-shared-memory crate
depend on proxmox-shared-memory crate.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-13 17:49:38 +01:00
Dietmar Maurer bf013be1c4 create /var/lib/proxmox-bnackup at server startup
This was missing in previous patch...

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-12 08:12:18 +01:00
Dietmar Maurer b935209584 fix directory permission problems
By carefully setting options on all create_path() calls,
and by creating "/var/lib/proxmox-backup" at api server startup.
2021-11-12 07:29:18 +01:00
Dietmar Maurer e511e0e553 proxmox-backup-proxy: implement traffic control
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-10 10:15:40 +01:00
Dietmar Maurer 610150a4b4 implement a traffic control cache for fast rate control limiter lockups
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-10 10:15:40 +01:00
Dietmar Maurer bfd12e871f Add traffic control configuration config with API
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-10 10:15:40 +01:00
Dietmar Maurer 68fd9ca6d6 openid_login: vertify that firstname, lastname and email fits our schema definitions
If not, we do not copy the values to our user.cfg.
2021-11-10 06:48:40 +01:00
Dietmar Maurer 4beb7d2dbe correctly lock remote config
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-06 17:35:10 +01:00
Dominik Csapak 9b1e2ae83c api: admin/datastore: reuse 'is_protected' implementation
we already have that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 12:54:54 +02:00
Dominik Csapak 8292d3d20e api2/admin/datastore: add get/set_protection
for gettin/setting the protected flag for snapshots (akin to notes)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:31:11 +02:00
Dominik Csapak 5cc7d89139 api2: datastore/delete_group: throw error for partially removed group
when a group could not be completely removed due to protected snapshot,
throw an error

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:31:07 +02:00
Dominik Csapak 343392613d pull_store/group: dont try remove locally protected snapshots
and log if a vanished groups could not be completely deleted if it
contains protected snapshots

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:31:04 +02:00
Dominik Csapak 02db72678f add protected info of snapshots to api and task logs
adds the info that a snapshot is protected to:
* snapshot list
* manual pruning (also dry-run)
* prune jobs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 11:30:51 +02:00
Dominik Csapak 9d42e0475b acme: interpret no TOS as accepted
some custom ACME endpoints do not have TOS, interpret this as
'the user has accepted the TOS', like we do for PVE.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-22 10:55:14 +02:00
Wolfgang Bumiller 0a33951e9e acme: new_account: prevent replacing existing accounts
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-22 08:35:24 +02:00
Dietmar Maurer e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
Dominik Csapak 6b8329ee34 tape: simplify export_media_set for pool writer
our export code can handle if the tape is inside the drive, so unloading
it first does not have an benefit, it even makes the exporting slower,
since we first unload it into its original slot, and then moving it
to an import/export slot

so drop the code that unloads the tape from the drive, and let the
export code itself handle that

change the 'eject' into a 'rewind' and comment why we do that first

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-21 06:37:48 +02:00
Dietmar Maurer 934c8724e2 proxmox-rrd: add option to avoid page cache for load/save
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dietmar Maurer 98eb435d90 proxmox-rrd: use syncfs after writing rrd files
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dominik Csapak ac2ca6c341 tape: improve export_media error message for not found tape
'export_media' can handle if the tape is in either a normal slot of the
library, or in the drive assigned to the current pool writer.
(because we need to lock the drive)

if it is, for some reason, in a different drive, the error message
 'media is not online'
could be slightly confusing for a user, since it would appear in the drive list

add the 'or a differen drive' to make it clearer

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-18 10:40:56 +02:00
Dietmar Maurer fce7cd0d36 proxmox-rrd: avoild blocking readers while applying the journal
By using and extra RwLock<RRDMap> on the rrd data.
2021-10-15 09:22:07 +02:00
Dietmar Maurer 2e3f94e12f proxmox-rrd: make rrd load callback configurable 2021-10-14 11:41:26 +02:00
Dietmar Maurer 58f70bccbb proxmox-rrd: pass time and value to update function 2021-10-14 08:12:56 +02:00
Dietmar Maurer fae4f6c509 cleanup: move rrd cache related code into extra file 2021-10-14 07:57:27 +02:00
Dietmar Maurer c68fa58a59 remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer eb37d4ece2 proxmox-rrd: remove dependency to proxmox-rrd-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 1198f8d4e6 proxmox-rrd: implement new CBOR based format
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 4b709ade68 proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place
allow the current thread to do some other work in-between

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer fa49d0fde9 RRD_CACHE: use a OnceCell instead of lazy_static
And initialize only with proxmox-backup-proxy. Other binaries dont need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 1d44f175c6 proxmox-rrd: use a journal to reduce amount of bytes written
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Wolfgang Bumiller 890b88cbef remove pbs-tools::ops::ControlFlow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-12 14:36:40 +02:00
Dominik Csapak 75442e813e api daemons: fix sending log-reopen command
send_command serializes everything so it cannot be used to send a
raw, optimized command. Normally that means we get an error like
> 'unable to parse parameters (expected json object)'
when used that way.

Switch over to send_raw_command which does not re-serializes the
command.

Fixes: 45b8a032 ("refactor send_command")
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-11 14:35:50 +02:00
Wolfgang Bumiller 6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Dominik Csapak 0e1edf19b1 proxmox-backup-proxy: clean up old tasks when the task log was rotated
we maybe have old tasks when the task list was rotated, so clean them up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-10-08 06:47:38 +02:00
Dietmar Maurer 0191759316 proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
Dietmar Maurer d1c3bc5350 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
Dietmar Maurer 09340f28f5 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Dietmar Maurer 608806e884 proxmox-rest-server: use new ServerAdapter trait instead of callbacks
Async callbacks are a PITA, so we now pass a single trait object which
implements check_auth and get_index.
2021-10-05 11:13:10 +02:00
Dietmar Maurer 48176b0a77 proxmox-rest-server: pass owned RestEnvironment to get_index
This way we avoid pointers with lifetimes.
2021-10-05 11:12:53 +02:00
Dominik Csapak 0a6df20986 rest-server/daemon: use sd_notify_barrier for service reloading
until now, we manually polled the systemd service state during a reload
so that the sd_notify messages get processed in the correct order
(RELOAD(old) -> MAINPID(old) -> READY(new))

with systemd >= 246 there is now 'sd_notify_barrier' which
blocks until systemd processed all prior messages

with that change, the daemon does not need to know the service name anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-02 11:44:20 +02:00
Dietmar Maurer 6680878b5c proxmox-rest-server: make get_index async 2021-10-01 09:38:10 +02:00
Dietmar Maurer 038f385089 proxmox-rest-server: make check_auth async 2021-10-01 07:53:59 +02:00
Dietmar Maurer 49e25688f1 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
Wolfgang Bumiller 8735247f29 drop fd_change_cloexec from proxmox-rest-server
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-30 12:43:22 +02:00
Dietmar Maurer 0d5d15c9d1 proxmox-rest-server: improve docs
And rename enable_file_log to enable_access_log.
2021-09-30 12:29:15 +02:00
Dietmar Maurer 2e44983a37 proxmox-rest-server: improve docs
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
2021-09-30 10:51:41 +02:00
Thomas Lamprecht aaf4f40285 subscription: switch verification domain over to shop.proxmox.com
With the merger the shop got moved from shop.maurer-it to
shop.proxmox.com, while we transparently redirect we also want to
stop doing that in a few years, so use new domain.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-30 10:28:53 +02:00
Dietmar Maurer fd1b65cc3c proxmox-rest-server: allow to catch SIGINT and SIGHUP separately
And make ServerState private.
2021-09-30 08:41:30 +02:00
Dietmar Maurer 38da8ca1bc proxmox-rest-server: improve logging
And rename server_state_init() into catch_shutdown_and_reload_signals().
2021-09-29 14:48:46 +02:00
Wolfgang Bumiller 450105b0c3 make pbs_tools::cert not depend on pbs-buildcfg
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
Wolfgang Bumiller b62edce929 remove pbs_client::connect_to_localhost
It also used `CertInfo` from pbs-tools which is also server
specific.

The original helper is now in the main crate's
client_helpers instead.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 14:11:26 +02:00
Dietmar Maurer bf95fba72e remove wrong calls to systemd_notify
We alrteady call systemd_notify inside the create_service future.
2021-09-29 12:04:48 +02:00
Dietmar Maurer d265420025 daemon: simlify code (make it easier to use) 2021-09-29 12:04:48 +02:00
Wolfgang Bumiller 01a080215d drop pbs_tools::auth
`pbs_client::connect_to_localhost` now requires the key as
optional parameter

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-29 11:08:52 +02:00
Dominik Csapak be5b43cb87 remove tools/async_io.rs
nothing from here is used anymore, so remove it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
Dominik Csapak 6f0565fa60 rest-server: use hypers AddrIncoming for proxmox-backup-api
this has a 'from_listener' (tokio::net::TcpListener) since hyper 0.14.5 in
the 'tcp' feature (we use 'full', which includes that; since 0.14.13
it is not behind a feature flag anymore).

this makes it possible to create a hyper server without our
'StaticIncoming' wrapper and thus makes it unnecessary.

The only other thing we have to do is to change the Service impl from
tokio::net::TcpStream to hyper::server::conn::AddStream to fulfill the trait
requirements.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-29 09:38:40 +02:00
Dietmar Maurer 53daae8e89 proxmox-rest-server: cleanup formatter, improve docs
Use trait for OutputFormatter. This is functionally equivalent,
but more rust-like...
2021-09-28 07:45:50 +02:00
Dietmar Maurer 8a23ea4656 move src/backup/read_chunk.rs to pbs-datastore/src/local_chunk_reader.rs 2021-09-27 11:10:14 +02:00
Dietmar Maurer c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
Dietmar Maurer b446fa14c5 WorkerTaskContext: make it Send + Sync 2021-09-27 09:11:38 +02:00
Dietmar Maurer 6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dominik Csapak af2eb422d5 tools: smart: only throw error for smartctl fatal errors
only bit 0-2 are fatal errors, bit 3-7 are used to indicate
some drive conditions. for details see the manpage of smartctl(8)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 [ Thomas: resolved merge-conflict due to moved run_command ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-27 08:59:13 +02:00
Dietmar Maurer bbd57396d7 proxmox-backup-manager: avoid proxmox_rest_server::init_worker_tasks() for "bashcomplete" and "printdoc" 2021-09-24 12:31:42 +02:00
Dietmar Maurer 0fd55b08d9 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer 619cd5cbcb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer 1ec0d70d09 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
Dietmar Maurer c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer f7348a23cd move src/server/h2service.rs into proxmox-rest-server crate 2021-09-24 10:28:17 +02:00
Dietmar Maurer ae18c436dd proxmox-backup-manager: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer b0e20a71e2 proxmox-daily-update: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer 81867f0539 use UPID and systemd helpers from proxmox 0.13.4 2021-09-23 12:01:43 +02:00
Dietmar Maurer 0a33fba49c worker task: allow to configure path and owner/group
And application now needs to call init_worker_tasks() before using
worker tasks.

Notable changes:
- need to call  init_worker_tasks() before using worker tasks.
- create_task_log_dirs() ís called inside init_worker_tasks()
- removed UpidExt trait
- use atomic_open_or_create_file()
- remove pbs_config and pbs_buildcfg dependency
2021-09-23 11:59:45 +02:00
Dietmar Maurer 049a22a3a3 src/server/worker_task.rs: Avoid using pbs-api-type::Authid
Because we want to move worker_task.rs into proxmox-rest-server crate.
2021-09-23 11:59:25 +02:00
Thomas Lamprecht a844fa0ba0 move dump-catalog-shell-cli doc-helper to proxmox-backup-client crate
it's only used for generating the docs for the interactive-shell
parts of the client.

Ideally we'd avoid that whole separate binary in the first place and
let the client dump it, but we'd need to have some more elaborate
"hide this command from the help/usage" mechanisms in the CLI
helper/formatter code to make that play out more nicely.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-22 16:25:09 +02:00
Thomas Lamprecht c25ea25f0a debug: api ls: make path optional and default to "/"
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:11:36 +02:00
Dominik Csapak bfa942c0cf api: make some workers log on CLI
some workers did not log when called via cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:37 +02:00
Dominik Csapak f54634a890 api: add missing token list match_all property
to have the proper link between the token list and the sub routes
in the api, include the 'tokenname' property in the token listing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:30 +02:00
Dominik Csapak efb7c5348c proxmox-backup-debug: add 'api' subcommands
this provides some generic api call mechanisms like pvesh/pmgsh.
by default it uses the https api on localhost (creating a token
if called as root, else requesting the root@pam password interactively)

this is mainly intended for debugging, but it is also useful for
situations where some api calls do not have an equivalent in a binary
and a user does not want to go through the api

not implemented are the http2 api calls (since it is a separate api an
it wouldn't be that easy to do)

there are a few quirks though, related to the 'ls' command:
i extract the 'child-link' from the property name of the
'match_all' statement of the router, but this does not
always match with the property from the relevant 'get' api call
so it fails there (e.g. /tape/drive )

this can be fixed in the respective api calls (e.g. by renaming
the parameter that comes from the path)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:30 +02:00
Dominik Csapak d6fcc1170a move proxmox-backup-debug back to main crate
we want to add something to it that needs access to the
proxmox_backup::api2 stuff, so it cannot live in a sub crate

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:30 +02:00
Dominik Csapak 3f742f952a server: refactor abort_local_worker
we'll need this outside the module

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 15:10:30 +02:00
Dietmar Maurer 84af82e8cf rename pbs-systemd to proxmox-systemd 2021-09-21 10:06:27 +02:00
Dietmar Maurer fd18775ac1 worker_state: move tasktype() code to src/api2/node/tasks.rs
Because this is API related code, and only used there.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:47:48 +02:00
Dietmar Maurer e678a50ea1 buildsys: drop double-build hack to avoid linkage issues
basically a (semantic) revert of commit
991be99c37 "buildsys: workaround
linkage issues from openid/curl build server stuff separate"

This is no longer required because we moved proxmox_restore_daemon
code into extra crate (previous commit)

Originally-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 6523588c8d move proxmox_restore_daemon code into extra crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 6fbf0acc76 move src/server/rest.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 36b7085ec2 rest server: cleanup auth-log handling
Handle auth logs the same way as access log.
- Configure with ApiConfig
- CommandoSocket command to reload auth-logs "api-auth-log-reopen"

Inside API calls, we now access the ApiConfig using the RestEnvironment.

The openid_login api now also logs failed logins and return http_err!(UNAUTHORIZED, ..)
on failed logins.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 1b1a553741 rest server: do not use pbs_api_types::Authid
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 98b7d58b94 rest server: return UserInformation from ApiAuth::check_auth
This need impl UserInformation for Arc<CachedUserInfo> which is implemented
with proxmox 0.13.2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 7fa9a37c7c make get_index and ApiConfig property (callback)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer f533d16ef6 rest server: simplify get_index() method signature
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 778c7d954b move normalize_uri_path and extract_cookie to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 605fe2e7e7 move src/tools/compression.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 1b552c109d move src/server/formatter.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer d4d49f7325 move src/server/environment.rs to proxmox-rest-server crate
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer 8bca935f08 move src/tools/daemon.rs to proxmox-rest-server workspace
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dietmar Maurer fd6d243843 move ApiConfig, FileLogger and CommandoSocket to proxmox-rest-server workspace
ApiConfig: avoid using  pbs_config::backup_user()
CommandoSocket: avoid using  pbs_config::backup_user()
FileLogger: avoid using  pbs_config::backup_user()
- use atomic_open_or_create_file()

Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-21 08:46:41 +02:00
Dominik Csapak 8eef31724f api: disks/directory: add 'name' property to list of mounts
so that we actually have the property that 'match_all' refers to for
the templated API path.

This is mostly for improving usage of the WIP pbs-shell, i.e., its
`ls` command, it has no other functional/semantic impact.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-15 11:59:02 +02:00
Dominik Csapak 2de1b06a06 api: disks/directory: factor out BASE_MOUNT_DIR path
will be reused in the next commit

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-15 11:54:18 +02:00
Thomas Lamprecht a332040a7f api: nodes: rustfmt
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-09-15 11:42:28 +02:00
Dominik Csapak 957133077f api2: nodes: add missing node list api call
to have an api call for api path traversal

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-09-15 11:32:58 +02:00
Dietmar Maurer ccc3896ff3 avoid type re-exports 2021-09-14 08:35:43 +02:00
Dietmar Maurer cef5c72682 move src/tape/helpers/snapshot_reader.rs to src/backup/snapshot_reader.rs 2021-09-14 07:42:06 +02:00
Dietmar Maurer 048b43af24 split tape code into new pbs_tape workspace 2021-09-13 12:54:59 +02:00
Dietmar Maurer 67a5cf4714 fix regression tests 2021-09-10 12:45:06 +02:00
Dietmar Maurer 6227654ad8 more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dominik Csapak e384f16a19 proxmox-tape: add 'force-media-set' also to cli
we have it in the api and gui, but the cli was missing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-09-10 10:06:21 +02:00
Dietmar Maurer 89725197c0 move PruneOptions to pbs_api_types workspace 2021-09-10 09:21:27 +02:00
Dietmar Maurer e7d4be9d85 move datastore config to pbs_config workspace 2021-09-10 08:40:58 +02:00
Dietmar Maurer ba3d7e19fb move user configuration to pbs_config workspace
Also moved memcom.rs and cached_user_info.rs
2021-09-10 07:09:04 +02:00
Dietmar Maurer b65dfff574 cleanup User configuration: use Updater 2021-09-09 13:14:28 +02:00
Dietmar Maurer 8cc3760e74 move acl to pbs_config workspaces, pbs_api_types cleanups 2021-09-09 10:50:08 +02:00
Dietmar Maurer 1cb08a0a05 move token_shadow to pbs_config workspace
Also moved out crypt.rs (libcrypt bindings) to pbs_tools workspace.
2021-09-08 14:00:14 +02:00
Dietmar Maurer 6f4228809e move network config to pbs_config workspace 2021-09-08 12:22:48 +02:00
Dietmar Maurer 5af3bcf062 changer config cleanup: use Updater 2021-09-08 09:29:01 +02:00
Dietmar Maurer cdc83c4eb2 tape job cleanup: user Updater 2021-09-08 08:55:55 +02:00
Dietmar Maurer ffa403b5fd verify job cleanup: use Updater/flatten 2021-09-08 08:40:32 +02:00
Dietmar Maurer 5bd77f00e2 sync job cleanup: use Updater/flatten 2021-09-08 08:28:09 +02:00
Dietmar Maurer 802189f7f5 move verify.rs to pbs_config workspace 2021-09-08 08:01:07 +02:00
Dietmar Maurer a4e5a0fc9f move sync.rs to pbs_config workspace 2021-09-08 06:57:23 +02:00
Dietmar Maurer e3619d4101 moved tape_job.rs to pbs_config workspace 2021-09-07 12:40:15 +02:00
Dietmar Maurer 5839c469c1 move tape_encryption_keys.rs to pbs_config workspace 2021-09-07 10:37:08 +02:00
Dietmar Maurer bbdda58b35 moved key_derivation.rs from pbs_datastore to pbs-config/src/key_config.rs
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
2021-09-07 10:12:17 +02:00
Dietmar Maurer 45d5d873ce move Kdf and KeyInfo to pbs_api_types workspace 2021-09-07 09:59:59 +02:00
Dominik Csapak f46806414a tape/inventory: fix the tape tests as user, by mocking the lock
locking during the tests as regular user failed because we try to
chown to the backup user (which is not always possible).

Instead, do not lock at all, by implementing 'open_backup_lockfile' with
'create_mocked_lock'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-07 08:42:04 +02:00
Dietmar Maurer aad2d162ab move media_pool config to pbs_config workspace 2021-09-06 08:56:04 +02:00
Dietmar Maurer 1ce8e905ea move drive config to pbs_config workspace
Also moved the tape type definitions to pbs_api_types.
2021-09-03 09:10:18 +02:00
Dietmar Maurer 6afdda8832 move remote config into pbs-config workspace 2021-09-02 14:25:15 +02:00
Dietmar Maurer 2121174827 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Wolfgang Bumiller 4c1b776168 another import cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 14:46:01 +02:00
Wolfgang Bumiller 42dad3abd3 fixup imports in tests and examples
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:32:21 +02:00
Wolfgang Bumiller 6c76aa434d split proxmox-file-restore into its own crate
This also moves a couple of required utilities such as
logrotate and some file descriptor methods to pbs-tools.

Note that the logrotate usage and run-dir handling should be
improved to work as a regular user as this *should* (IMHO)
be a regular unprivileged command (including running
qemu given the kvm privileges...)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:23:29 +02:00
Wolfgang Bumiller e5f9b7f79e split out proxmox-backup-debug binary
and introduce pbs_tools::cli module

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 14:45:48 +02:00
Wolfgang Bumiller dd2162f6bd more import cleanups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 14:01:03 +02:00
Wolfgang Bumiller cabdabba3d fixup imports in debug binary
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:29:06 +02:00
Hannes Laimer 3e593a2459 add index recovery to pb-debug
Adds possibility to recover data from an index file. Options:
 - chunks: path to the directory where the chunks are saved
 - file: the index file that should be recovered(must be either .fidx or
   didx)
 - [opt] keyfile: path to a keyfile, if the data was encrypted, a keyfile is
   needed
 - [opt] skip-crc: boolean, if true, read chunks wont be verified with their
   crc-sum, increases the restore speed by a lot

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:19:56 +02:00
Hannes Laimer 7c5287bb95 add file inspection to pb-debug
Adds possibility to inspect .blob, .fidx and .didx files. For index
files a list of the chunks referenced will be printed in addition to
some other information. .blob files can be decoded into file or directly
into stdout. Without decode the tool just prints the size and encryption
mode of the blob file. Options:
 - file: path to the file
 - [opt] decode: path to a file or stdout(-), if specidied, the file will be
   decoded into the specified location [only for blob files, no effect
   with index files]
 - [opt] keyfile: path to a keyfile, needed if decode is specified and the
   data was encrypted

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:19:54 +02:00
Hannes Laimer 7c72ae04f1 add chunk inspection to pb-debug
Adds possibility to inspect chunks and find indexes that reference the
chunk. Options:
 - chunk: path to the chunk file
 - [opt] decode: path to a file or to stdout(-), if specified, the
   chunk will be decoded into the specified location
 - [opt] digest: needed when searching for references, if set, it will
   be used for verification when decoding
 - [opt] keyfile: path to a keyfile, needed if decode is specified and
   the data was encrypted
 - [opt] reference-filter: path in which indexes that reference the
   chunk should be searched, can be a group, snapshot or the whole
   datastore, if not specified no references will be searched
 - [default=true] use-filename-as-digest: use chunk-filename as digest,
   if no digest is specified

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 13:19:51 +02:00
Wolfgang Bumiller 86582454e8 make api2::helpers::list_dir_content a CatalogReader method
this is its natural place and everything required is already
part of the catalog module

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller 013b1e8bca move some more API types
ArchiveEntry -> pbs-datastore
RestoreDaemonStatus -> pbs-api-types

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 11:29:17 +02:00
Wolfgang Bumiller b2065dc7d2 cleanup proxmox_backup::backup module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 14:14:04 +02:00
Dietmar Maurer 97dfc62f0d remote config: derive and use Updater
Defined a new struct RemoteConfig (without name and password). This makes it
possible to bas64-encode the pasword in the config, but still allow plain
passwords with the API.
2021-08-30 12:48:45 +02:00
Wolfgang Bumiller e351ac786d split out proxmox-backup-client binary
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 11:39:01 +02:00
Wolfgang Bumiller 7b570c177d move some API return types to pbs-api-types
they'll be required by the api client

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 11:39:01 +02:00
Dietmar Maurer dbda1513c5 tape: media_pool: derive and use Updater 2021-08-30 11:17:14 +02:00
Dietmar Maurer c62a6acb2e drive config cleanup: derive and use Updater 2021-08-30 10:50:20 +02:00
Dietmar Maurer e4a5c072b4 openid cleanup: derive and use Updater 2021-08-30 09:48:53 +02:00
Wolfgang Bumiller a37c8d2431 use ApiType trait
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-25 10:43:58 +02:00
Dietmar Maurer a8a20e9210 use new api updater features
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-25 10:43:58 +02:00
Dominik Csapak 9f58e312d7 tape/pool_writer: fix typo
s/wrinting/writing/

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-09 09:36:38 +02:00
Dominik Csapak cffe0b81e3 tape backup: mention groups that were empty
otherwise a user might get a task log like this:

-----
...
found 7 groups
TASK OK
-----

which could confuse the users as why there were no snapshots backed up

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-09 09:28:01 +02:00
Dietmar Maurer bb14ed8cab cleanup: simplify next_expired_media() 2021-08-04 11:01:18 +02:00
Dietmar Maurer e5545c9804 cli: proxmox-tape backup-job list: use status api and display next-run an d next-media-label 2021-08-04 09:59:12 +02:00
Dietmar Maurer efe96ec039 tape: compute next-media-label for each tape backup job 2021-08-04 09:59:12 +02:00
Dietmar Maurer 1d3ae83359 tape: media_pool: implement guess_next_writable_media() 2021-08-04 09:59:12 +02:00
Dominik Csapak 4bb3876352 tape: lto: increase default timeout to 10 minutes
it seems that for some actions or in some circumstances, two minutes is
simply too short and the command aborts. Increase the default timeout to
10 minutes.

While it should give most commands enough time to finish, in case of a real
failure the procedure now takes up to 5 times longer, but IMHO thats an
OK tradeoff.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-03 09:19:13 +02:00
Dominik Csapak 8be69a8453 api/ui: allow zstd compression for new zpools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-30 17:51:13 +02:00
Fabian Ebner 0b12a5a698 api: apt: adapt to further proxmox-apt back-end changes
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-30 10:37:27 +02:00
Fabian Ebner 2eac359430 api: apt: adapt to proxmox-apt back-end changes
It's up to the caller to provide the current release for standard
repository detection/addition.

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2021-07-30 10:37:27 +02:00
Dominik Csapak 855b55dc14 api2: tape: media: use MediaCatalog::snapshot_list for content listing
this should make the api call much faster, since it is not reading
the whole catalog anymore

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-07-29 13:34:36 +02:00
Dietmar Maurer 5ad40a3dd1 tape: media_catalog: add snapshot list cache for catalog
For some parts of the ui, we only need the snapshot list from the catalog,
and reading the whole catalog (can be multiple hundred MiB) is not
really necessary.

Instead, we write the list of snapshots into a seperate .index file. This file
is generated on demand and is much smaller and thus faster to read.
2021-07-29 13:34:31 +02:00
Dietmar Maurer 7116a2d9da tape: lock media_catalog file to to get a consistent view with load_catalog 2021-07-29 13:34:25 +02:00
Dietmar Maurer 0d5e990a62 cleanup: factor out tape catalog path helpers 2021-07-29 13:34:18 +02:00
Dominik Csapak 4f57f4ad84 tape: changer: add tests for decode_element_status_page
a test for a valid status_page, one with excess data
(in the descriptor as well in the page as a whole)
and a test with too little data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:23:21 +02:00
Dominik Csapak 13e13d836f tape: changer: handle libraries that sends wrong amount of data
if the library sends more data than advertised, simply cut it off,
but if it sends less data, bail out (depending on how much data is
missing, trying to parse it could lead to a panic, so bail out early)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:22:48 +02:00
Dominik Csapak 3ab2432ab6 tape: changer: remove unnecesary inquiry parameter
this is never used, so remove it.
Ok, since they are only non public functions.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:17:07 +02:00
Dominik Csapak 76e8565076 api2: tape/restore: commit temporary catalog at the end
in 'restore_archive', we reach that 'catalog.commit()' for
* every skipped snapshot (we already call 'commit_if_large' then before)
* every skipped chunk archive (no change in catalog since we do not read
  the chunk archive in that case)
* after reading a catalog (no change in catalog)

in all other cases, we call 'commit_if_large' and return early,
meaning that the 'commit' there was executed too often and
unnecessary, so move it after the loop over the files, before
finishing the temporary database.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 11:28:03 +02:00
Dominik Csapak a2ef36d445 tape: media_catalog: improve chunk_archive interface
instead of having a public start/end_chunk_archive and register_chunks,
simply expose a 'register_chunk_archive' method since we always have
a list of chunks anywhere we want to add them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:18:13 +02:00
Dominik Csapak 42b010174e tape: changer: handle invalid descriptor data from library in status page
We get the descriptor length from the library and use that in
'chunks_exact', which panics on length 0. Catch that case
and bail out, since that makes no sense here anyway.

This could prevent a panic, in case a library sends wrong data.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:05:37 +02:00
Dietmar Maurer 68e77657e6 datastore config: cleanup code (use flatten attribute) 2021-07-23 12:43:33 +02:00
Thomas Lamprecht cc99866ea3 restore daemon: rust format
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:26:10 +02:00
Thomas Lamprecht 1ea3f23f7e file restore: improve some comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:25:34 +02:00
Thomas Lamprecht 3f780ddf73 restore daemon: log about doing basic system env setup
debugging history showed that its surely nice to have more logs at
when stuff happens (and thus fails)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:24:30 +02:00
Thomas Lamprecht 9edf96e6b6 restore daemon: setup backup system user and group
now required as we always enforce lock files to be owned by the
backup user, and the restore code uses such code indirectly as the
REST server module is reused from proxmox-backup-server. Once that is
refactored out we may do away such things, but until then we need to
have a somewhat complete system env.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:19:38 +02:00
Thomas Lamprecht 73e1ba65ca restore daemon: add setup_system_env helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-23 08:10:55 +02:00
Dominik Csapak 02631056b8 tape: changer: handle missing dvcid information
the dvcid information is not always available, so skip it if is missing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 12:00:30 +02:00
Dominik Csapak 131d0f10c2 tape: changer: improve error message on wrong counts
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 11:37:14 +02:00
Dominik Csapak f9aa980c7d tape: changer: correctly consume data in decode_element_status_page
instead of 'blindly' trusting the changer to deliver the fields written
in the specification, trust the length data it returns in the header.

we slice the descriptor data into equal sized chunks of the correct
size, then we do not have care bout the len and empty checks anymore

this also makes the code to read the rest of the page obsolete,
since the next descriptor is on the correct offset anyway

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 11:37:14 +02:00
Dominik Csapak c1feb447e8 tape: changer: sg_pt: fix typo
ok, since its a private struct

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 17:02:16 +02:00
Dominik Csapak 62a0e190cb tape: changer: sg_pt: add SCSI_VOLUME_TAG_LEN const
so that we do have less 'magic' constants without description

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 17:01:37 +02:00
Dominik Csapak 5890143920 api: types: CHANGER_DRIVENUM_SCHEMA: increase maximum drives per changer
to 255. 8 drives per changer was a rather arbitrary limitation and could
well be reached in practice with big libraries.

Altough 255 is still a arbirtrary limitation, this is much less likely
to be reached in practice.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 16:59:02 +02:00
Wolfgang Bumiller ef4df211ab move CachedChunkReader to pbs-datastore
this was actually still missing from the previous commit

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:20:03 +02:00
Wolfgang Bumiller eb5e0ae65a move remaining client tools to pbs-tools/datastore
pbs-datastore now ended up depending on tokio after all, but
that's fine for now

for the fuse code I added pbs-fuse-loop (has the old
fuse_loop and its 'loopdev' module)
ultimately only binaries should depend on this to avoid the
library link

the only thins remaining to move out the client binary are
the api method return types, those will need to be moved to
pbs-api-types...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 14:12:24 +02:00
Wolfgang Bumiller bbc71e3b02 client: fix panic message
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-21 13:28:55 +02:00
Dietmar Maurer ac81ed17b9 fix regression test file permission problems
By simply using the current user/group instead of backup:backup
2021-07-21 09:30:22 +02:00
Dietmar Maurer a301c362e3 add helpers to write configuration files 2021-07-20 18:54:23 +02:00
Dietmar Maurer 7526d86419 use new atomic_open_or_create_file
Factor out open_backup_lockfile() method to acquire locks owned by
user backup with permission 0660.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-20 18:54:23 +02:00
Wolfgang Bumiller fc5870be53 move channel/stream helpers to pbs-tools
pbs_tools
  ::blocking: std/async wrapping with block_in_place
  ::stream: stream <-> AsyncRead/AsyncWrite wrapping

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 11:27:40 +02:00
Wolfgang Bumiller 3c8c2827cb move required_X_param to pbs_tools::json
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 11:09:52 +02:00
Wolfgang Bumiller 6c221244df move lru cachers to pbs-tools
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 10:57:22 +02:00
Wolfgang Bumiller 38629c3961 move ChunkStream to pbs-client
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-20 10:52:21 +02:00
Wolfgang Bumiller 3fa1b4b48c cleanup unused imports
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:55:19 +02:00
Wolfgang Bumiller 58a3fae773 move pxar binary to separate crate
and move its few remaining proxmox_backup deps out to
pbs-tools

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:53:43 +02:00
Wolfgang Bumiller 0889806a3c resolve some more client imports
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:03:24 +02:00
Wolfgang Bumiller 51ec8a3c62 move some api types to pbs-api-types
and resolve some imports in the client binary

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 15:01:03 +02:00
Wolfgang Bumiller a12b1be728 move build.rs and friends to pbs-buildcfg
with this the main crate won't be re-compiled every time a
*binary* is modified

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 14:59:18 +02:00
Wolfgang Bumiller a3399f4337 doc and tests fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 14:16:28 +02:00
Wolfgang Bumiller 2b7f8dd5ea move client to pbs-client subcrate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 12:58:43 +02:00
Wolfgang Bumiller 72fbe9ffa5 move 'wait_for_local_worker' from client to server
this just made no sense in the client

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:44:44 +02:00
Wolfgang Bumiller 4805edc4ec move more tools for the client into subcrates
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:07:12 +02:00
Wolfgang Bumiller 9eb784076c move more helpers to pbs-tools
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:07:12 +02:00
Wolfgang Bumiller b9c5cd8291 add proxmox-backup-banner binary crate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-19 10:07:12 +02:00
Dominik Csapak 9805207aa5 api: admin/datastore: add new 'prune-datastore' api call
to prune the whole datastore at once, with the given parameters.
We need a new api call since this can take a while and we need to start
a worker for this. The exisiting api call returns a list of removed/kept
snapshots and is synchronous.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:40:05 +02:00
Dominik Csapak 8e0b852f24 server/prune_job: add proper permission checks to 'prune_datastore'
checks for PRIV_DATASTORE_MODIFY, or else if the auth_id is the backup
owner, and skips the group if not.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:39:01 +02:00
Dominik Csapak 0052dc6d28 server/prune_job: add 'keep_all' logic to 'prune_datastore'
it is the same as when pruning single groups.
for prune_jobs, we never start the worker if there is no prune option set.
but if we want to call 'prune_datastore' from somewhere else, we
have to check it here again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:38:28 +02:00
Dominik Csapak 61f05679d2 server/prune_job: factor out 'prune_datastore'
we want to use that outside of a prune job

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:36:45 +02:00
Dominik Csapak 9751ef4b36 backup/datastore: refactor check_backup_owner there
and add a 'owns_backup' convenience function

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:36:02 +02:00
Dominik Csapak 0a240aaa9a api: admin/datastore: simplify prune api call
by using the api macro and reusing the PruneOptions from pbs-datastore

this means we can now drop the 'add_common_prune_prameters' macro

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:34:36 +02:00
Dominik Csapak e0665a64bd client: simplify prune api method
by using the api macro on the async method and reusing the PruneOptions
from pbs-datastore with 'flatten: true'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:34:28 +02:00
Dominik Csapak ced694589d api-types: move PRUNE_SCHEMA_KEEP_* to pbs-api-types
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 11:26:09 +02:00
Dominik Csapak 6c053ffc89 tape: changer: sg_pt: make extra scsi request for dvcid
some libraries cannot handle a request with volume tags and DVCID set at
the same time.

So we make 2 separate requests and merge them, since we want to keep
the vendor/model/serial data.

to not overcomplicate the code, add another special type to ElementType

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 08:46:06 +02:00
Dietmar Maurer bfd357c5a1 depend on proxmox 0.11.6 (changed make_tmp_file() return type) 2021-07-14 13:37:26 +02:00
Dominik Csapak 9517a5759a fix #3526: correctly filter tasks with 'since' and 'until'
The previous assumption was that the Tasks returned by the Iterator are
sorted by the starttime, but that is not actually the case, and
could never have been, since we append the tasks into the log when
they are finished (not started) and running tasks are always iterated
first.

To correctly filter (and simplify the the api call) we forgo the
combinators, and use a for loop instead. This way we only have to do
the since/until checks only once per Task, but have to do the
start/limit counting ourselves.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-14 09:39:14 +02:00
Stefan Reiter d9822cd3cb fix #3515: file-restore-daemon: allow LVs/PVs with dash in name
LVM replaces any dashes '-' in an LV or PV name with two '--' for the
created device node in /dev/mapper/ to distinguish the seperating
character between the PV and LV name.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-07-13 12:07:51 +02:00
Stefan Reiter 66501529a2 file-restore: increase lock timeout on QEMU map
This lock is held during VM startup, so that multiple calls will not
start VMs twice. But this means that the timeout needs to incorporate
the time it might take a VM to boot, so increase it quite a bit.

This could previously lead to "interrupted system call" errors when
accessing backups with many disks.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-07-13 12:07:23 +02:00
Dominik Csapak 2072dede4a api2: tape: restore: add warning for list restore
if an error occurs, the snapshot dirs will already be created, and we
do not clean them up (some might already be finished).

Warn the user that they are not cleaned up.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-13 12:02:01 +02:00
Dietmar Maurer 9ee4c23833 tape: changer: sg_pt: always retry until timeout 2021-07-13 10:39:28 +02:00
Stefan Reiter c4c4b5a3ef auth: 'crypt' is not thread safe
According to crypt(3):
"crypt places its result in a static storage area, which will be
overwritten by subsequent calls to crypt. It is not safe to call crypt
from multiple threads simultaneously."

This means that multiple login calls as a PBS-realm user can collide and
produce intermittent authentication failures. A visible case is for
file-restore, where VMs with many disks lead to just as many auth-calls
at the same time, as the GUI tries to expand each tree element on load.

Instead, use the thread-safe variant 'crypt_r', which places the result
into a pre-allocated buffer of type 'crypt_data'. The C struct is laid
out according to 'lib/crypt.h.in' and the man page mentioned above.

Use the opportunity and make both arguments to the rust 'crypt' function
take a &[u8].

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-07-12 18:38:48 +02:00
Dominik Csapak 0ed40b19c7 tape: changer: sg_pt: query element types separately
Some changers do not like the DVCID bit when querying non-drives,
this includes when querying 'all' elements.

To circumvent this, we query each type by itself (like mtx does it),
and only add the DVCID bit for drives (Data Transfer Elements).

Reported by a user in the forum:
https://forum.proxmox.com/threads/ibm-3584-ts3500-support.92291/

and limit to 1000 elements per request.
(Because some changers limit that request with the options we set)

instead of checking if the data len was equal to the allocation_len
for getting more data, we count the returned elements and compare
that with the number we requested

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-07-12 18:19:26 +02:00
Dietmar Maurer a0cd0f9cec change tape drive lock path
New kernel has stricter checks on tmpfs with stick-bit on directories, so some
commands (i.e. proxmox-tape changer status) fails when executed as root, because
permission checks fails when locking the drive.

This patch move the drive locks to /run/proxmox-backup/drive-lock.

Note: This is incompatible to old locking mechmanism, so users may not
run tape backups during update (or running backup can fail).
2021-07-12 17:26:49 +02:00
Dominik Csapak 2c0abe9234 Revert "api: access: domains: add ExtraRealmInfo and RealmInfo structs"
This reverts commit da7ec1d2af.

not necessary, since we have the api in config/access/openid

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-12 09:53:07 +02:00
Dominik Csapak 2649c89358 Revert "api: access: domains: add get/create/update/delete domain call"
This reverts commit 5117cf4f17.

we already have that in api2/config/access

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-12 09:53:07 +02:00
Dominik Csapak bbd34d70d5 api: config: access: openid: use better Privilige Realm.Allocate
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-12 09:53:07 +02:00
Dominik Csapak 9779ad0b00 api: config: access: openid: use correct parameter for matching
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-12 09:53:07 +02:00
Wolfgang Bumiller 82bdf6b5e7 api: tfa: module path cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-12 08:43:14 +02:00
Stefan Reiter d6688884f6 api: add support for notes on backup groups
Stored in atomically-updated 'notes' file in backup group directory.
Available via dedicated GET/PUT API calls, as well as the first line
being included in list_groups (similar to list_snapshots).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-12 07:13:28 +02:00
Thomas Lamprecht 6b5013edb3 rest: log response: avoid unnecessary mut on variable
a match expresses the fallback slightly nicer and needs no mut,
which is always nice to avoid.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-11 13:05:19 +02:00
Dominik Csapak 5117cf4f17 api: access: domains: add get/create/update/delete domain call
modeled like our other section config api calls
two drawbacks of doing it this way:
* we have to copy some api properties again for the update call,
  since not all of them are updateable (username-claim)
* we only handle openid for now, which we would have to change
  when we add ldap/ad

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-09 15:36:54 +02:00
Dominik Csapak da7ec1d2af api: access: domains: add ExtraRealmInfo and RealmInfo structs
these will be used as parameters/return types for the read/create/etc.
calls for realms

for now we copy the necessary attributes (only from openid) since
our api macros/tools are not good enought to generate the necessary
api definitions for section configs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-09 15:36:54 +02:00
Dominik Csapak 934de1d691 config: acl: add PRIV_REALM_ALLOCATE
will be used for realm creation/update/deletion

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-09 15:36:54 +02:00
Dominik Csapak 0c27d880b0 api: access: domains: add BasicRealmInfo struct and use it
to have better type safety and as preparation for adding more types

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-09 15:36:54 +02:00
Wolfgang Bumiller be3a0295b6 client: import updates
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 15:32:12 +02:00
Wolfgang Bumiller aa2838c27a move client::pull to server::pull
it's not used by the client and not part of the client, it
just makes use *of* the client, but is used on the
datastore/server...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-09 15:17:53 +02:00