tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
5,773 Commits 2 Branches 54 Tags
b3f279e2d91d83f0b28fe0ce87f8f23aaf40bff0
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Dietmar Maurer b3f279e2d9 use complete_file_name from proxmox-router 1.1 2021-10-13 14:10:02 +02:00
Dietmar MaurerandThomas Lamprecht bacc99c7f8 proxmox-rrd: add more commands to the rrd cli tool 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 6728d0977b proxmox-rrd: rename last_counter to last_value 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht bff7c027c9 proxmox-rrd: protect against negative update time 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 79b3113361 proxmox-rrd: new helpers: slot, slot_start_time & slot_end_time 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 5885767b91 proxmox-rrd: avoid expensive modulo (%) inside loop 
Modulo is very slow, so we try to avoid it inside loops.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht ec08247e5c proxmox-rrd: add binary to create/manage rrd files 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 400f081487 proxmox-rrd: split out load_rrd (cleanup) 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 03664514ab proxmox-rrd: support CF::Last 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht c68fa58a59 remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ 
Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 426dda0730 proxmox-rrd: extract_data: include values from current slot 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht eb37d4ece2 proxmox-rrd: remove dependency to proxmox-rrd-api-types 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 1198f8d4e6 proxmox-rrd: implement new CBOR based format 
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 4b709ade68 proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place 
allow the current thread to do some other work in-between

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht fa49d0fde9 RRD_CACHE: use a OnceCell instead of lazy_static 
And initialize only with proxmox-backup-proxy. Other binaries dont need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 1d44f175c6 proxmox-rrd: use a journal to reduce amount of bytes written 
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar Maurer b3a67f1f14 proxmox-rrd: use correct directory options in create_rrdb_dir 2021-10-07 08:50:50 +02:00
Dietmar Maurer 3cc23ca6cc proxmox-rrd: cleanup error handling 2021-10-07 08:01:12 +02:00
Dietmar Maurer 3def6bfc64 proxmox-rrd: use log crate instead of eprintln, avoid duplicate logs 2021-10-06 18:19:22 +02:00
Dietmar Maurer 18e8bc17e4 proxmox-rrd: fix update (do not update) when time is in the past 2021-10-06 18:01:48 +02:00
Dietmar Maurer 0191759316 proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
Dietmar Maurer dbc42e6f75 proxmox-rrd: remove serde dependency 2021-10-06 10:55:46 +02:00
Dietmar Maurer d1c3bc5350 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
Dietmar Maurer a97301350f proxmox-rrd: use create_path instead of std::fs::create_dir_all 
To ensure correct file ownership.
 2021-10-06 08:37:14 +02:00
Dietmar Maurer 09340f28f5 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Dietmar Maurer 608806e884 proxmox-rest-server: use new ServerAdapter trait instead of callbacks 
Async callbacks are a PITA, so we now pass a single trait object which
implements check_auth and get_index.
 2021-10-05 11:13:10 +02:00
Dietmar Maurer 48176b0a77 proxmox-rest-server: pass owned RestEnvironment to get_index 
This way we avoid pointers with lifetimes.
 2021-10-05 11:12:53 +02:00
Dietmar Maurer 3483a3b3a1 proxmox-rest-server: cleanup, access api_auth using a method 2021-10-05 11:12:53 +02:00
Dietmar Maurer 6680878b5c proxmox-rest-server: make get_index async 2021-10-01 09:38:10 +02:00
Dietmar Maurer 593043ed53 proxmox-rest-server: add comment why ApiService needs to be 'pub' 2021-10-01 08:35:51 +02:00
Dietmar Maurer 038f385089 proxmox-rest-server: make check_auth async 2021-10-01 07:53:59 +02:00
Dietmar Maurer b914b94773 proxmox-rest-server: fix spelling errors 2021-10-01 06:43:30 +02:00
Dietmar Maurer 2194bc59c8 proxmox-rest-server: improve ApiService docs 2021-09-30 17:18:47 +02:00
Dietmar Maurer a98a288e2d proxmox-rest-server: start module docs 2021-09-30 13:49:29 +02:00
Dietmar Maurer 49e25688f1 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
Dietmar Maurer 0d5d15c9d1 proxmox-rest-server: improve docs 
And rename enable_file_log to enable_access_log.
 2021-09-30 12:29:15 +02:00
Dietmar Maurer 2e44983a37 proxmox-rest-server: improve docs 
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
 2021-09-30 10:51:41 +02:00
Dietmar Maurer c76ff4b472 proxmox-rest-server: cleanup FileLogger docs 2021-09-30 10:51:31 +02:00
Dietmar Maurer e64f77b716 cleanup: move use clause to top 2021-09-30 08:42:37 +02:00
Dietmar Maurer fd1b65cc3c proxmox-rest-server: allow to catch SIGINT and SIGHUP separately 
And make ServerState private.
 2021-09-30 08:41:30 +02:00
Dietmar Maurer 11148dce43 proxmox-rtest-server: make Reloader and Reloadable private 2021-09-30 07:44:19 +02:00
Dietmar Maurer 38da8ca1bc proxmox-rest-server: improve logging 
And rename server_state_init() into catch_shutdown_and_reload_signals().
 2021-09-29 14:48:46 +02:00
Dietmar Maurer a0ffd4a413 proxmox-rest-server: avoid useless call to request_shutdown 
Also avoid unsafe code.
 2021-09-29 14:37:07 +02:00
Dietmar Maurer bf95fba72e remove wrong calls to systemd_notify 
We alrteady call systemd_notify inside the create_service future.
 2021-09-29 12:04:48 +02:00
Dietmar Maurer d265420025 daemon: simlify code (make it easier to use) 2021-09-29 12:04:48 +02:00
Dietmar Maurer 8cf445ecc4 cleanup: make BoxedStoreFunc private 
There is no need to export this type.
 2021-09-29 09:55:43 +02:00
Dietmar Maurer 99940358e3 ExtJsFormatter: use ParameterError to correctly compute 'errors' 
By default, 'errors' is now empty.

Depend on proxmox 0.13.5.
 2021-09-28 10:19:55 +02:00
Dietmar Maurer 53daae8e89 proxmox-rest-server: cleanup formatter, improve docs 
Use trait for OutputFormatter. This is functionally equivalent,
but more rust-like...
 2021-09-28 07:45:50 +02:00
Dietmar Maurer 8a23ea4656 move src/backup/read_chunk.rs to pbs-datastore/src/local_chunk_reader.rs 2021-09-27 11:10:14 +02:00
Dietmar Maurer c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
Dietmar Maurer b446fa14c5 WorkerTaskContext: make it Send + Sync 2021-09-27 09:11:38 +02:00
Dietmar Maurer 6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dietmar Maurer bbd57396d7 proxmox-backup-manager: avoid proxmox_rest_server::init_worker_tasks() for "bashcomplete" and "printdoc" 2021-09-24 12:31:42 +02:00
Dietmar Maurer 0fd55b08d9 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer 619cd5cbcb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer 1ec0d70d09 cleanup worker task logging 
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
 2021-09-24 10:34:11 +02:00
Dietmar Maurer c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer f7348a23cd move src/server/h2service.rs into proxmox-rest-server crate 2021-09-24 10:28:17 +02:00
Dietmar Maurer ae18c436dd proxmox-backup-manager: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer b0e20a71e2 proxmox-daily-update: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer b9700a9fe5 move worker_task.rs into proxmox-rest-server crate 
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
 2021-09-24 10:28:17 +02:00
Dietmar Maurer 81867f0539 use UPID and systemd helpers from proxmox 0.13.4 2021-09-23 12:01:43 +02:00
Dietmar Maurer 0a33fba49c worker task: allow to configure path and owner/group 
And application now needs to call init_worker_tasks() before using
worker tasks.

Notable changes:
- need to call  init_worker_tasks() before using worker tasks.
- create_task_log_dirs() ís called inside init_worker_tasks()
- removed UpidExt trait
- use atomic_open_or_create_file()
- remove pbs_config and pbs_buildcfg dependency
 2021-09-23 11:59:45 +02:00
Dietmar Maurer 049a22a3a3 src/server/worker_task.rs: Avoid using pbs-api-type::Authid 
Because we want to move worker_task.rs into proxmox-rest-server crate.
 2021-09-23 11:59:25 +02:00
Dietmar MaurerandThomas Lamprecht 84af82e8cf rename pbs-systemd to proxmox-systemd 2021-09-21 10:06:27 +02:00
Dietmar MaurerandThomas Lamprecht 48109c5354 pbs-systemd: do not depend on pbs-tools 
Instead, copy a few line of nom helper code, and implement
a simple run_command helper.
 2021-09-21 10:06:27 +02:00
Dietmar MaurerandThomas Lamprecht fd18775ac1 worker_state: move tasktype() code to src/api2/node/tasks.rs 
Because this is API related code, and only used there.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:47:48 +02:00
Dietmar MaurerandThomas Lamprecht e678a50ea1 buildsys: drop double-build hack to avoid linkage issues 
basically a (semantic) revert of commit
991be99c37 "buildsys: workaround
linkage issues from openid/curl build server stuff separate"

This is no longer required because we moved proxmox_restore_daemon
code into extra crate (previous commit)

Originally-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 6523588c8d move proxmox_restore_daemon code into extra crate 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 6fbf0acc76 move src/server/rest.rs to proxmox-rest-server crate 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 36b7085ec2 rest server: cleanup auth-log handling 
Handle auth logs the same way as access log.
- Configure with ApiConfig
- CommandoSocket command to reload auth-logs "api-auth-log-reopen"

Inside API calls, we now access the ApiConfig using the RestEnvironment.

The openid_login api now also logs failed logins and return http_err!(UNAUTHORIZED, ..)
on failed logins.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 1b1a553741 rest server: do not use pbs_api_types::Authid 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 98b7d58b94 rest server: return UserInformation from ApiAuth::check_auth 
This need impl UserInformation for Arc<CachedUserInfo> which is implemented
with proxmox 0.13.2

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 7fa9a37c7c make get_index and ApiConfig property (callback) 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht f533d16ef6 rest server: simplify get_index() method signature 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 778c7d954b move normalize_uri_path and extract_cookie to proxmox-rest-server crate 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 605fe2e7e7 move src/tools/compression.rs to proxmox-rest-server crate 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 1b552c109d move src/server/formatter.rs to proxmox-rest-server crate 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht d4d49f7325 move src/server/environment.rs to proxmox-rest-server crate 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 8bca935f08 move src/tools/daemon.rs to proxmox-rest-server workspace 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht fd6d243843 move ApiConfig, FileLogger and CommandoSocket to proxmox-rest-server workspace 
ApiConfig: avoid using  pbs_config::backup_user()
CommandoSocket: avoid using  pbs_config::backup_user()
FileLogger: avoid using  pbs_config::backup_user()
- use atomic_open_or_create_file()

Auth Trait: moved definitions to proxmox-rest-server/src/lib.rs
- removed CachedUserInfo patrameter
- return user as String (not Authid)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar MaurerandThomas Lamprecht 037f6b6d5e start new proxmox-rest-server workspace 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-09-21 08:46:41 +02:00
Dietmar Maurer 36c6e7bb82 fix tests/worker-task-abort.rs - correctly spawn command socket 
And wait for the task.

Note: The test is still ignored (but works now when run a root)
 2021-09-14 10:42:44 +02:00
Dietmar Maurer ccc3896ff3 avoid type re-exports 2021-09-14 08:35:43 +02:00
Dietmar Maurer cef5c72682 move src/tape/helpers/snapshot_reader.rs to src/backup/snapshot_reader.rs 2021-09-14 07:42:06 +02:00
Dietmar Maurer 51a2d9e375 fix refs in generated docs 2021-09-13 13:40:20 +02:00
Dietmar Maurer 048b43af24 split tape code into new pbs_tape workspace 2021-09-13 12:54:59 +02:00
Dietmar Maurer 67a5cf4714 fix regression tests 2021-09-10 12:45:06 +02:00
Dietmar Maurer 6227654ad8 more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dietmar Maurer 89725197c0 move PruneOptions to pbs_api_types workspace 2021-09-10 09:21:27 +02:00
Dietmar Maurer e7d4be9d85 move datastore config to pbs_config workspace 2021-09-10 08:40:58 +02:00
Dietmar Maurer ba3d7e19fb move user configuration to pbs_config workspace 
Also moved memcom.rs and cached_user_info.rs
 2021-09-10 07:09:04 +02:00
Dietmar Maurer b65dfff574 cleanup User configuration: use Updater 2021-09-09 13:14:28 +02:00
Dietmar Maurer 8cc3760e74 move acl to pbs_config workspaces, pbs_api_types cleanups 2021-09-09 10:50:08 +02:00
Dietmar Maurer 1cb08a0a05 move token_shadow to pbs_config workspace 
Also moved out crypt.rs (libcrypt bindings) to pbs_tools workspace.
 2021-09-08 14:00:14 +02:00
Dietmar Maurer 6f4228809e move network config to pbs_config workspace 2021-09-08 12:22:48 +02:00
Dietmar Maurer 5af3bcf062 changer config cleanup: use Updater 2021-09-08 09:29:01 +02:00
Dietmar Maurer cdc83c4eb2 tape job cleanup: user Updater 2021-09-08 08:55:55 +02:00
Dietmar Maurer ffa403b5fd verify job cleanup: use Updater/flatten 2021-09-08 08:40:32 +02:00
Dietmar Maurer 5bd77f00e2 sync job cleanup: use Updater/flatten 2021-09-08 08:28:09 +02:00