tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
5,842 Commits 2 Branches 54 Tags
485b2438ac024a7fa316bbe97eaa3ed7598755b3
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Dietmar Maurer 485b2438ac traffic_control: use Memcom to track. config versions 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer bfd12e871f Add traffic control configuration config with API 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer 0c136bfab1 DailyDuration: implement time_match() 2021-11-10 10:15:40 +01:00
Dietmar Maurer 245e2aea23 New DailyDuration type with nom parser 
We will use this to specify timesframes for network rate limits (only
apply limite when inside the time frame).

Note: This is not systemd related, but we can reuse some of the parser
method.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer b9d588ffde implement Servive for RateLimitedStream 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer e4bc3e0e8d proxmox-backup-client: add rate/burst parameter to backup CLI 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer 2419dc0de9 pbs-client: add option to use the new RateLimiter 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:32 +01:00
Dietmar Maurer 68fd9ca6d6 openid_login: vertify that firstname, lastname and email fits our schema definitions 
If not, we do not copy the values to our user.cfg.
 2021-11-10 06:48:40 +01:00
Dietmar Maurer 4beb7d2dbe correctly lock remote config 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-06 17:35:10 +01:00
Dietmar Maurer 7a356a748a bump version to 2.0.13-1 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-21 08:36:41 +02:00
Dietmar Maurer 1c402740a2 update dedian/control 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-21 07:59:45 +02:00
Dietmar Maurer e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create 
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-21 07:28:32 +02:00
Dietmar Maurer d6473f5359 proxmox-rrd: use fsync instead of syncfs 
syncfs can sync unrelated data, and we do not want that.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-20 11:46:59 +02:00
Dietmar Maurer f5f9ec81d2 proxmox-rrd: fix regression tests 2021-10-19 18:41:03 +02:00
Dietmar Maurer fea950155f proxmox-rrd: improve dev docs 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer ef2944bc24 proxmox-rrd: cleanup - impl FromStr for JournalEntry 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer 934c8724e2 proxmox-rrd: add option to avoid page cache for load/save 
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer 98eb435d90 proxmox-rrd: use syncfs after writing rrd files 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer 7f381a6246 proxmox-rrd: use fine grained locking in commit_journal_impl 
Aquire the rrd_map lock for each file (else we block access for a long time)
 2021-10-18 14:55:47 +02:00
Dietmar Maurer c17fbbbc07 proxmox-rrd: log all errors from apply_and_commit_journal_thread (but only once) 2021-10-18 11:57:19 +02:00
Dietmar Maurer d26865c52c proxmox-rrd: cleanup list_old_journals 2021-10-18 10:00:58 +02:00
Dietmar Maurer 2b05008a11 proxmox-rrd: cleanup - use struct instead of tuple 2021-10-16 12:45:03 +02:00
Dietmar Maurer 45700e2ecf proxmox-rrd: move RRDMap into extra file 2021-10-16 12:45:03 +02:00
Dietmar Maurer f84304235b proxmox-rrd: move JournalState into extra file 2021-10-16 12:45:03 +02:00
Dietmar Maurer 0ca41155b2 proxmox-rrd: implement non blocking journal 
Do not block while applying the journal.
 2021-10-16 12:45:03 +02:00
Dietmar Maurer a291ab59ba proxmox-rrd: rename RRDCacheState to JournalState 2021-10-15 09:35:44 +02:00
Dietmar Maurer fce7cd0d36 proxmox-rrd: avoild blocking readers while applying the journal 
By using and extra RwLock<RRDMap> on the rrd data.
 2021-10-15 09:22:07 +02:00
Dietmar Maurer 658357c5a8 proxmox-rrd: log journal apply/flush times, split apply and flush 
We need to apply the journal only once.
 2021-10-15 07:16:41 +02:00
Dietmar Maurer 7484fce24d proxmox-rrd: cleanup - use slot_end_time() 2021-10-14 16:29:00 +02:00
Dietmar Maurer f28a713e2b proxmox-rrd: cleanup - use staturating_add instead of if/else 2021-10-14 16:10:55 +02:00
Dietmar Maurer a9017805b7 proxmox-rrd: improve dev docs 2021-10-14 11:53:54 +02:00
Dietmar Maurer 2e3f94e12f proxmox-rrd: make rrd load callback configurable 2021-10-14 11:41:26 +02:00
Dietmar Maurer d531c7ae61 proxmox-rrd: add more regression tests 2021-10-14 10:55:12 +02:00
Dietmar Maurer 7df1580fa6 proxmox-rrd: add regression tests and two minor fixes 2021-10-14 10:17:07 +02:00
Dietmar Maurer 58f70bccbb proxmox-rrd: pass time and value to update function 2021-10-14 08:12:56 +02:00
Dietmar Maurer fae4f6c509 cleanup: move rrd cache related code into extra file 2021-10-14 07:57:27 +02:00
Dietmar Maurer ddafb28572 proxmox-rrd: add some integration tests (file format tests) 2021-10-13 18:21:23 +02:00
Dietmar Maurer b3f279e2d9 use complete_file_name from proxmox-router 1.1 2021-10-13 14:10:02 +02:00
Dietmar MaurerandThomas Lamprecht bacc99c7f8 proxmox-rrd: add more commands to the rrd cli tool 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 6728d0977b proxmox-rrd: rename last_counter to last_value 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht bff7c027c9 proxmox-rrd: protect against negative update time 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 79b3113361 proxmox-rrd: new helpers: slot, slot_start_time & slot_end_time 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 5885767b91 proxmox-rrd: avoid expensive modulo (%) inside loop 
Modulo is very slow, so we try to avoid it inside loops.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht ec08247e5c proxmox-rrd: add binary to create/manage rrd files 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 400f081487 proxmox-rrd: split out load_rrd (cleanup) 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 03664514ab proxmox-rrd: support CF::Last 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht c68fa58a59 remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/ 
Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 426dda0730 proxmox-rrd: extract_data: include values from current slot 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht eb37d4ece2 proxmox-rrd: remove dependency to proxmox-rrd-api-types 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 1198f8d4e6 proxmox-rrd: implement new CBOR based format 
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 4b709ade68 proxmox-backup-proxy: use tokio::task::spawn_blocking instead of block_in_place 
allow the current thread to do some other work in-between

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht fa49d0fde9 RRD_CACHE: use a OnceCell instead of lazy_static 
And initialize only with proxmox-backup-proxy. Other binaries dont need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar MaurerandThomas Lamprecht 1d44f175c6 proxmox-rrd: use a journal to reduce amount of bytes written 
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-10-13 13:36:02 +02:00
Dietmar Maurer b3a67f1f14 proxmox-rrd: use correct directory options in create_rrdb_dir 2021-10-07 08:50:50 +02:00
Dietmar Maurer 3cc23ca6cc proxmox-rrd: cleanup error handling 2021-10-07 08:01:12 +02:00
Dietmar Maurer 3def6bfc64 proxmox-rrd: use log crate instead of eprintln, avoid duplicate logs 2021-10-06 18:19:22 +02:00
Dietmar Maurer 18e8bc17e4 proxmox-rrd: fix update (do not update) when time is in the past 2021-10-06 18:01:48 +02:00
Dietmar Maurer 0191759316 proxmox-rrd: improve developer docs 2021-10-06 12:19:54 +02:00
Dietmar Maurer dbc42e6f75 proxmox-rrd: remove serde dependency 2021-10-06 10:55:46 +02:00
Dietmar Maurer d1c3bc5350 split out RRD api types into proxmox-rrd-api-types crate 2021-10-06 09:49:51 +02:00
Dietmar Maurer a97301350f proxmox-rrd: use create_path instead of std::fs::create_dir_all 
To ensure correct file ownership.
 2021-10-06 08:37:14 +02:00
Dietmar Maurer 09340f28f5 move RRD code into proxmox-rrd crate 2021-10-06 08:13:28 +02:00
Dietmar Maurer 608806e884 proxmox-rest-server: use new ServerAdapter trait instead of callbacks 
Async callbacks are a PITA, so we now pass a single trait object which
implements check_auth and get_index.
 2021-10-05 11:13:10 +02:00
Dietmar Maurer 48176b0a77 proxmox-rest-server: pass owned RestEnvironment to get_index 
This way we avoid pointers with lifetimes.
 2021-10-05 11:12:53 +02:00
Dietmar Maurer 3483a3b3a1 proxmox-rest-server: cleanup, access api_auth using a method 2021-10-05 11:12:53 +02:00
Dietmar Maurer 6680878b5c proxmox-rest-server: make get_index async 2021-10-01 09:38:10 +02:00
Dietmar Maurer 593043ed53 proxmox-rest-server: add comment why ApiService needs to be 'pub' 2021-10-01 08:35:51 +02:00
Dietmar Maurer 038f385089 proxmox-rest-server: make check_auth async 2021-10-01 07:53:59 +02:00
Dietmar Maurer b914b94773 proxmox-rest-server: fix spelling errors 2021-10-01 06:43:30 +02:00
Dietmar Maurer 2194bc59c8 proxmox-rest-server: improve ApiService docs 2021-09-30 17:18:47 +02:00
Dietmar Maurer a98a288e2d proxmox-rest-server: start module docs 2021-09-30 13:49:29 +02:00
Dietmar Maurer 49e25688f1 rename CommandoSocket to CommandSocket 2021-09-30 12:52:35 +02:00
Dietmar Maurer 0d5d15c9d1 proxmox-rest-server: improve docs 
And rename enable_file_log to enable_access_log.
 2021-09-30 12:29:15 +02:00
Dietmar Maurer 2e44983a37 proxmox-rest-server: improve docs 
And renames abort_worker_async to abort_worker_nowait (avoid confusion,
because the function itself is not async).
 2021-09-30 10:51:41 +02:00
Dietmar Maurer c76ff4b472 proxmox-rest-server: cleanup FileLogger docs 2021-09-30 10:51:31 +02:00
Dietmar Maurer e64f77b716 cleanup: move use clause to top 2021-09-30 08:42:37 +02:00
Dietmar Maurer fd1b65cc3c proxmox-rest-server: allow to catch SIGINT and SIGHUP separately 
And make ServerState private.
 2021-09-30 08:41:30 +02:00
Dietmar Maurer 11148dce43 proxmox-rtest-server: make Reloader and Reloadable private 2021-09-30 07:44:19 +02:00
Dietmar Maurer 38da8ca1bc proxmox-rest-server: improve logging 
And rename server_state_init() into catch_shutdown_and_reload_signals().
 2021-09-29 14:48:46 +02:00
Dietmar Maurer a0ffd4a413 proxmox-rest-server: avoid useless call to request_shutdown 
Also avoid unsafe code.
 2021-09-29 14:37:07 +02:00
Dietmar Maurer bf95fba72e remove wrong calls to systemd_notify 
We alrteady call systemd_notify inside the create_service future.
 2021-09-29 12:04:48 +02:00
Dietmar Maurer d265420025 daemon: simlify code (make it easier to use) 2021-09-29 12:04:48 +02:00
Dietmar Maurer 8cf445ecc4 cleanup: make BoxedStoreFunc private 
There is no need to export this type.
 2021-09-29 09:55:43 +02:00
Dietmar Maurer 99940358e3 ExtJsFormatter: use ParameterError to correctly compute 'errors' 
By default, 'errors' is now empty.

Depend on proxmox 0.13.5.
 2021-09-28 10:19:55 +02:00
Dietmar Maurer 53daae8e89 proxmox-rest-server: cleanup formatter, improve docs 
Use trait for OutputFormatter. This is functionally equivalent,
but more rust-like...
 2021-09-28 07:45:50 +02:00
Dietmar Maurer 8a23ea4656 move src/backup/read_chunk.rs to pbs-datastore/src/local_chunk_reader.rs 2021-09-27 11:10:14 +02:00
Dietmar Maurer c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
Dietmar Maurer b446fa14c5 WorkerTaskContext: make it Send + Sync 2021-09-27 09:11:38 +02:00
Dietmar Maurer 6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dietmar Maurer bbd57396d7 proxmox-backup-manager: avoid proxmox_rest_server::init_worker_tasks() for "bashcomplete" and "printdoc" 2021-09-24 12:31:42 +02:00
Dietmar Maurer 0fd55b08d9 WorkerTaskContext: add shutdown_requested() and fail_on_shutdown() 2021-09-24 12:04:31 +02:00
Dietmar Maurer 619cd5cbcb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer 1ec0d70d09 cleanup worker task logging 
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
 2021-09-24 10:34:11 +02:00
Dietmar Maurer c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer f7348a23cd move src/server/h2service.rs into proxmox-rest-server crate 2021-09-24 10:28:17 +02:00
Dietmar Maurer ae18c436dd proxmox-backup-manager: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer b0e20a71e2 proxmox-daily-update: setup worker and command socket 2021-09-24 10:28:17 +02:00
Dietmar Maurer b9700a9fe5 move worker_task.rs into proxmox-rest-server crate 
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
 2021-09-24 10:28:17 +02:00
Dietmar Maurer 81867f0539 use UPID and systemd helpers from proxmox 0.13.4 2021-09-23 12:01:43 +02:00
Dietmar Maurer 0a33fba49c worker task: allow to configure path and owner/group 
And application now needs to call init_worker_tasks() before using
worker tasks.

Notable changes:
- need to call  init_worker_tasks() before using worker tasks.
- create_task_log_dirs() ís called inside init_worker_tasks()
- removed UpidExt trait
- use atomic_open_or_create_file()
- remove pbs_config and pbs_buildcfg dependency
 2021-09-23 11:59:45 +02:00