Commit Graph

57 Commits

Author SHA1 Message Date
Fabian Grünbichler 9a37bd6c84 tree-wide: fix needless borrows
found and fixed via clippy

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-12-30 13:55:33 +01:00
Wolfgang Bumiller 9fa3026a08 cleanup schema function calls
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-12-16 11:25:02 +01:00
Wolfgang Bumiller 821aa8eae6 bump proxmox-schema to 1.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-12-16 11:25:02 +01:00
Dietmar Maurer 25877d05ac update to proxmox-sys 0.2 crate
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
  proxmox-serde instead)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-24 10:32:27 +01:00
Dietmar Maurer 860eaec58f use proxmox::tools::fd::fd_change_cloexec from proxmox 0.15.3
Depend on proxmox 0.15.3

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-18 13:43:41 +01:00
Dominik Csapak 5c1cabdea1 rrd: use saturating_sub to avoid underflow
Without this, the tests fail in debug mode.
Also having start (u64) underflow to a value greater than end does
not really make sense

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 12:54:54 +02:00
Dietmar Maurer e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
Dietmar Maurer d6473f5359 proxmox-rrd: use fsync instead of syncfs
syncfs can sync unrelated data, and we do not want that.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-20 11:46:59 +02:00
Dietmar Maurer f5f9ec81d2 proxmox-rrd: fix regression tests 2021-10-19 18:41:03 +02:00
Dietmar Maurer fea950155f proxmox-rrd: improve dev docs
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dietmar Maurer ef2944bc24 proxmox-rrd: cleanup - impl FromStr for JournalEntry
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dietmar Maurer 934c8724e2 proxmox-rrd: add option to avoid page cache for load/save
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dietmar Maurer 98eb435d90 proxmox-rrd: use syncfs after writing rrd files
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
Dietmar Maurer 7f381a6246 proxmox-rrd: use fine grained locking in commit_journal_impl
Aquire the rrd_map lock for each file (else we block access for a long time)
2021-10-18 14:55:47 +02:00
Dietmar Maurer c17fbbbc07 proxmox-rrd: log all errors from apply_and_commit_journal_thread (but only once) 2021-10-18 11:57:19 +02:00
Dietmar Maurer d26865c52c proxmox-rrd: cleanup list_old_journals 2021-10-18 10:00:58 +02:00
Dietmar Maurer 2b05008a11 proxmox-rrd: cleanup - use struct instead of tuple 2021-10-16 12:45:03 +02:00
Dietmar Maurer 45700e2ecf proxmox-rrd: move RRDMap into extra file 2021-10-16 12:45:03 +02:00
Dietmar Maurer f84304235b proxmox-rrd: move JournalState into extra file 2021-10-16 12:45:03 +02:00
Dietmar Maurer 0ca41155b2 proxmox-rrd: implement non blocking journal
Do not block while applying the journal.
2021-10-16 12:45:03 +02:00
Dietmar Maurer a291ab59ba proxmox-rrd: rename RRDCacheState to JournalState 2021-10-15 09:35:44 +02:00
Dietmar Maurer fce7cd0d36 proxmox-rrd: avoild blocking readers while applying the journal
By using and extra RwLock<RRDMap> on the rrd data.
2021-10-15 09:22:07 +02:00
Dietmar Maurer 658357c5a8 proxmox-rrd: log journal apply/flush times, split apply and flush
We need to apply the journal only once.
2021-10-15 07:16:41 +02:00
Dietmar Maurer 7484fce24d proxmox-rrd: cleanup - use slot_end_time() 2021-10-14 16:29:00 +02:00
Dietmar Maurer f28a713e2b proxmox-rrd: cleanup - use staturating_add instead of if/else 2021-10-14 16:10:55 +02:00
Dietmar Maurer a9017805b7 proxmox-rrd: improve dev docs 2021-10-14 11:53:54 +02:00
Dietmar Maurer 2e3f94e12f proxmox-rrd: make rrd load callback configurable 2021-10-14 11:41:26 +02:00
Dietmar Maurer d531c7ae61 proxmox-rrd: add more regression tests 2021-10-14 10:55:12 +02:00
Dietmar Maurer 7df1580fa6 proxmox-rrd: add regression tests and two minor fixes 2021-10-14 10:17:07 +02:00
Dietmar Maurer 58f70bccbb proxmox-rrd: pass time and value to update function 2021-10-14 08:12:56 +02:00
Dietmar Maurer ddafb28572 proxmox-rrd: add some integration tests (file format tests) 2021-10-13 18:21:23 +02:00
Dietmar Maurer b3f279e2d9 use complete_file_name from proxmox-router 1.1 2021-10-13 14:10:02 +02:00
Thomas Lamprecht 82f5ad18f0 proxmox-rrd: move unshipped cli tool to examples
it's a rather low-level tool mostly useful for debugging and some of
it is rather "dumb" (by design) anyway, e.g., it does not
transparently applies journal but really only operates on the DB
files as is (which can conflict with daemon operations).

In summary, not (yet) a tool meant for end user consumption.
Move it to examples folder to avoid compilation on packaging (we do
not ship it anyway) which allows us to move the rather expensive
proxmox-router (pulls in hyper) to the dev-dependencies section.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer bacc99c7f8 proxmox-rrd: add more commands to the rrd cli tool
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 6728d0977b proxmox-rrd: rename last_counter to last_value
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer bff7c027c9 proxmox-rrd: protect against negative update time
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 79b3113361 proxmox-rrd: new helpers: slot, slot_start_time & slot_end_time
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 5885767b91 proxmox-rrd: avoid expensive modulo (%) inside loop
Modulo is very slow, so we try to avoid it inside loops.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer ec08247e5c proxmox-rrd: add binary to create/manage rrd files
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 400f081487 proxmox-rrd: split out load_rrd (cleanup)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 03664514ab proxmox-rrd: support CF::Last
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer c68fa58a59 remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
Because the types used inside the RRD have other requirements
than the API types:

- other serialization format
- the API may not support all RRD features

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 426dda0730 proxmox-rrd: extract_data: include values from current slot
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer eb37d4ece2 proxmox-rrd: remove dependency to proxmox-rrd-api-types
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 1198f8d4e6 proxmox-rrd: implement new CBOR based format
Storing much more data points now got get better graphs.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Dietmar Maurer 1d44f175c6 proxmox-rrd: use a journal to reduce amount of bytes written
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.

Apply the journal every 30 minutes and on daemon startup.

Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
Wolfgang Bumiller 6ef1b649d9 update to first proxmox crate split
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
Wolfgang Bumiller e3f3359c86 bump proxmox dependency to 0.14.0 and proxmox-http to 0.5.0
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-08 11:18:22 +02:00
Dietmar Maurer b3a67f1f14 proxmox-rrd: use correct directory options in create_rrdb_dir 2021-10-07 08:50:50 +02:00
Dietmar Maurer 3cc23ca6cc proxmox-rrd: cleanup error handling 2021-10-07 08:01:12 +02:00