860eaec58f
use proxmox::tools::fd::fd_change_cloexec from proxmox 0.15.3
...
Depend on proxmox 0.15.3
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-11-18 13:43:41 +01:00
5c1cabdea1
rrd: use saturating_sub to avoid underflow
...
Without this, the tests fail in debug mode.
Also having start (u64) underflow to a value greater than end does
not really make sense
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-28 12:54:54 +02:00
e0a19d3313
use new fsync parameter to replace_file and atomic_open_or_create
...
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-21 07:28:32 +02:00
d6473f5359
proxmox-rrd: use fsync instead of syncfs
...
syncfs can sync unrelated data, and we do not want that.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-20 11:46:59 +02:00
f5f9ec81d2
proxmox-rrd: fix regression tests
2021-10-19 18:41:03 +02:00
fea950155f
proxmox-rrd: improve dev docs
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
ef2944bc24
proxmox-rrd: cleanup - impl FromStr for JournalEntry
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
934c8724e2
proxmox-rrd: add option to avoid page cache for load/save
...
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
98eb435d90
proxmox-rrd: use syncfs after writing rrd files
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-10-19 11:17:09 +02:00
7f381a6246
proxmox-rrd: use fine grained locking in commit_journal_impl
...
Aquire the rrd_map lock for each file (else we block access for a long time)
2021-10-18 14:55:47 +02:00
c17fbbbc07
proxmox-rrd: log all errors from apply_and_commit_journal_thread (but only once)
2021-10-18 11:57:19 +02:00
d26865c52c
proxmox-rrd: cleanup list_old_journals
2021-10-18 10:00:58 +02:00
2b05008a11
proxmox-rrd: cleanup - use struct instead of tuple
2021-10-16 12:45:03 +02:00
45700e2ecf
proxmox-rrd: move RRDMap into extra file
2021-10-16 12:45:03 +02:00
f84304235b
proxmox-rrd: move JournalState into extra file
2021-10-16 12:45:03 +02:00
0ca41155b2
proxmox-rrd: implement non blocking journal
...
Do not block while applying the journal.
2021-10-16 12:45:03 +02:00
a291ab59ba
proxmox-rrd: rename RRDCacheState to JournalState
2021-10-15 09:35:44 +02:00
fce7cd0d36
proxmox-rrd: avoild blocking readers while applying the journal
...
By using and extra RwLock<RRDMap> on the rrd data.
2021-10-15 09:22:07 +02:00
658357c5a8
proxmox-rrd: log journal apply/flush times, split apply and flush
...
We need to apply the journal only once.
2021-10-15 07:16:41 +02:00
7484fce24d
proxmox-rrd: cleanup - use slot_end_time()
2021-10-14 16:29:00 +02:00
f28a713e2b
proxmox-rrd: cleanup - use staturating_add instead of if/else
2021-10-14 16:10:55 +02:00
a9017805b7
proxmox-rrd: improve dev docs
2021-10-14 11:53:54 +02:00
2e3f94e12f
proxmox-rrd: make rrd load callback configurable
2021-10-14 11:41:26 +02:00
d531c7ae61
proxmox-rrd: add more regression tests
2021-10-14 10:55:12 +02:00
7df1580fa6
proxmox-rrd: add regression tests and two minor fixes
2021-10-14 10:17:07 +02:00
58f70bccbb
proxmox-rrd: pass time and value to update function
2021-10-14 08:12:56 +02:00
ddafb28572
proxmox-rrd: add some integration tests (file format tests)
2021-10-13 18:21:23 +02:00
b3f279e2d9
use complete_file_name from proxmox-router 1.1
2021-10-13 14:10:02 +02:00
82f5ad18f0
proxmox-rrd: move unshipped cli tool to examples
...
it's a rather low-level tool mostly useful for debugging and some of
it is rather "dumb" (by design) anyway, e.g., it does not
transparently applies journal but really only operates on the DB
files as is (which can conflict with daemon operations).
In summary, not (yet) a tool meant for end user consumption.
Move it to examples folder to avoid compilation on packaging (we do
not ship it anyway) which allows us to move the rather expensive
proxmox-router (pulls in hyper) to the dev-dependencies section.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
bacc99c7f8
proxmox-rrd: add more commands to the rrd cli tool
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
6728d0977b
proxmox-rrd: rename last_counter to last_value
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
bff7c027c9
proxmox-rrd: protect against negative update time
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
79b3113361
proxmox-rrd: new helpers: slot, slot_start_time & slot_end_time
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
5885767b91
proxmox-rrd: avoid expensive modulo (%) inside loop
...
Modulo is very slow, so we try to avoid it inside loops.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
ec08247e5c
proxmox-rrd: add binary to create/manage rrd files
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
400f081487
proxmox-rrd: split out load_rrd (cleanup)
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
03664514ab
proxmox-rrd: support CF::Last
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
c68fa58a59
remove proxmox-rrd-api-types crate, s/RRDTimeFrameResolution/RRDTimeFrame/
...
Because the types used inside the RRD have other requirements
than the API types:
- other serialization format
- the API may not support all RRD features
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
426dda0730
proxmox-rrd: extract_data: include values from current slot
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
eb37d4ece2
proxmox-rrd: remove dependency to proxmox-rrd-api-types
...
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
1198f8d4e6
proxmox-rrd: implement new CBOR based format
...
Storing much more data points now got get better graphs.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
1d44f175c6
proxmox-rrd: use a journal to reduce amount of bytes written
...
Append pending changes in a simple text based format that allows for
lockless appends as long as we stay below 4 KiB data per write.
Apply the journal every 30 minutes and on daemon startup.
Note that we do not ensure that the journal is synced, this is a
perfomance optimization we can make as the kernel defaults to
writeback in-flight data every 30s (sysctl vm/dirty_expire_centisecs)
anyway, so we lose at max half a minute of data on a crash, here one
should have in mind that we normally expose 1 minute as finest
granularity anyway, so not really much lost.
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-10-13 13:36:02 +02:00
6ef1b649d9
update to first proxmox crate split
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-11 11:58:49 +02:00
e3f3359c86
bump proxmox dependency to 0.14.0 and proxmox-http to 0.5.0
...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-10-08 11:18:22 +02:00
b3a67f1f14
proxmox-rrd: use correct directory options in create_rrdb_dir
2021-10-07 08:50:50 +02:00
3cc23ca6cc
proxmox-rrd: cleanup error handling
2021-10-07 08:01:12 +02:00
3def6bfc64
proxmox-rrd: use log crate instead of eprintln, avoid duplicate logs
2021-10-06 18:19:22 +02:00
18e8bc17e4
proxmox-rrd: fix update (do not update) when time is in the past
2021-10-06 18:01:48 +02:00
0191759316
proxmox-rrd: improve developer docs
2021-10-06 12:19:54 +02:00
dbc42e6f75
proxmox-rrd: remove serde dependency
2021-10-06 10:55:46 +02:00
Dietmar Maurer
Dominik Csapak
Thomas Lamprecht
Wolfgang Bumiller