Commit Graph

3242 Commits

Author SHA1 Message Date
Dominik Csapak 7eebe1483e server/worker_task: fix panic on slice range when index is empty
since len() and MAX_INDEX_TASKS are both usize, they underflow
instead of getting negative values

instead check the sizes and set them accordingly

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:11:06 +02:00
Dominik Csapak 9a76091785 proxmox-backup-proxy: add task archive rotation
this starts a task once a day at "00:00" that rotates the task log
archive if it is bigger than 500k

if we want, we can make the schedule/size limit/etc. configurable,
but for now it's ok to set fixed values for that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:41:18 +02:00
Dominik Csapak c386b06fc6 server/worker_task: remove unecessary read_task_list
since there are no users of this anymore and we now have a nicer
TaskListInfoIterator to use, we can drop this function

this also means that 'update_active_workers' does not need to return
a list anymore since we never used that result besides in
read_task_list

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:50 +02:00
Dominik Csapak 6bcfc5c1a4 api2/status: use the TaskListInfoIterator here
this means that limiting with epoch now works correctly
also change the api type to i64, since that is what the starttime is
saved as

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:24 +02:00
Dominik Csapak 768e10d0b3 api2/node/tasks: use TaskListInfoIterator instead of read_task_list
this makes the filtering/limiting much nicer and readable

since we now have potentially an 'infinite' amount of tasks we iterate over,
and cannot now beforehand how many there are, we return the total count
as always 1 higher then requested iff we are not at the end (this is
the case when the amount of entries is smaller than the requested limit)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:02 +02:00
Dominik Csapak e7244387c7 server/worker_task: add TaskListInfoIterator
this is an iterator that reads/parses/updates the task list as
necessary and returns the tasks in descending order (newest first)

it does this by using our logrotate iterator and using a vecdeque

we can use this to iterate over all tasks, even if they are in the
archive and even if the archive is logrotated but only read
as much as we need

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:39:16 +02:00
Dominik Csapak 5ade6c25f3 server/worker_task: write older tasks into archive file
instead of removing tasks beyond the 1000 that are in the index
write them into an archive file by appending them at the end
this way we can later still read them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:38:44 +02:00
Dominik Csapak 784fa1c2e3 server/worker_task: split task list file into two
one for only the active tasks and one for up to 1000 finished tasks

factor out the parsing of a task file (we will later need this again)
and use iterator combinators for easier code

we now sort the tasks ascending (this will become important in a later patch)
but reverse (for now) it to keep compatibility

this code also omits the converting into an intermittent hash
since it cannot really happen that we have duplicate tasks in this list
(since the call is locked by an flock, and it is the only place where we
write into the lists)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:38:28 +02:00
Dominik Csapak 66f4e6a809 server/worker_task: refactor locking of the task list
also add the functionality of having a 'shared' (read) lock for the list
we will need this later

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:37:54 +02:00
Dominik Csapak 8074d2b0c3 tools: add logrotate module
this is a helper to rotate and iterate over log files
there is an iterator for open filehandles as well as
only the filename

also it has the possibilty to rotate them
for compression, zstd is used

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:33:21 +02:00
Dietmar Maurer b02d49ab26 proxmox_backup_client key: allow to generate paperkey for master key 2020-09-29 08:29:42 +02:00
Dietmar Maurer 82a0cd2ad4 proxmox_backup_client key: add new paper-key command 2020-09-29 08:29:42 +02:00
Wolfgang Bumiller ee1a9c3230 parallel_handler: clippy: 'while_let_loop'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-29 08:13:51 +02:00
Wolfgang Bumiller db24c01106 parallel_handler: explicit Arc::clone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-28 13:40:03 +02:00
Wolfgang Bumiller ae3cfa8f0d parallel_handler: formatting cleanup, doc comment typo fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-28 13:40:03 +02:00
Dietmar Maurer b56c111e93 depend on proxmox 0.4.2 2020-09-28 10:50:44 +02:00
Dominik Csapak bbeb0256f1 server/worker_task: factor out task list rendering
we will need this later again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-28 07:31:27 +02:00
Dominik Csapak 005a5b9677 api2/node/tasks: move userfilter to function signature
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-28 07:18:13 +02:00
Dietmar Maurer 55bee04856 src/tools/parallel_handler.rs: remove unnecessary Sync bound 2020-09-26 16:16:11 +02:00
Dietmar Maurer 42fd40a124 src/bin/proxmox_backup_client/benchmark.rs: avoid compiler warning 2020-09-26 16:13:19 +02:00
Dietmar Maurer f21508b9e1 src/backup/verify.rs: use ParallelHandler to verify chunks 2020-09-26 11:14:37 +02:00
Dietmar Maurer ee7a308de4 src/backup/verify.rs: cleanup use clause 2020-09-26 10:23:44 +02:00
Dietmar Maurer 636e674ee7 src/client/pull.rs: simplify code 2020-09-26 10:09:51 +02:00
Dietmar Maurer b02b374b46 src/tools/parallel_handler.rs: remove static lifetime bound from handler_fn 2020-09-26 09:26:06 +02:00
Dietmar Maurer 1c13afa8f9 src/tools/parallel_handler.rs: join all threads in drop handler 2020-09-26 08:47:56 +02:00
Dietmar Maurer 69b92fab7e src/tools/parallel_handler.rs: remove unnecessary Sync trait bound 2020-09-26 07:38:44 +02:00
Thomas Lamprecht 6ab77df3f5 ui: some more eslint auto-fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-25 18:47:25 +02:00
Thomas Lamprecht 264c19582b ui: some more eslint auto-fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-25 18:36:58 +02:00
Thomas Lamprecht 8acd4d9afc ui: some more eslint auto-fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-25 18:34:54 +02:00
Thomas Lamprecht 65b0cea6bd ui: some eslint auto-fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-25 18:29:42 +02:00
Dietmar Maurer cfe01b2e6a bump version to 0.8.21-1 2020-09-25 13:20:35 +02:00
Dietmar Maurer b19b032be3 debian/control: update 2020-09-25 13:17:49 +02:00
Dietmar Maurer 5441708634 src/client/pull.rs: use new ParallelHandler 2020-09-25 12:58:20 +02:00
Dietmar Maurer 3c9b370255 src/tools/parallel_handler.rs: execute closure inside a thread pool 2020-09-25 12:58:20 +02:00
Dietmar Maurer 510544770b depend on crossbeam-channel 2020-09-25 12:58:20 +02:00
Thomas Lamprecht e8293841c2 docs: html: show "Proxmox Backup" in navi for small devices
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-24 20:03:17 +02:00
Thomas Lamprecht 46114bf28e docs: html: improve css for small displays
fixed-width navi/toc links were not switched in color for small width
displays, and thus they were barely readable as the background
switches to dark for small widths.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-24 20:03:17 +02:00
Thomas Lamprecht 0d7e61f06f docs: buildsys: add more dependencies to html target
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-24 19:45:23 +02:00
Thomas Lamprecht fd6a54dfbc docs: conf: fix conf for new alabaster theme version
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-24 19:44:50 +02:00
Thomas Lamprecht 1ea5722b8f docs: html: adapt custom css
highlighting the current chapter and some other small formatting
improvements

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-24 19:44:00 +02:00
Thomas Lamprecht bc8fadf494 docs: index: hide todo list toctree and genindex
I do not found another way to disable inclusion in the sidebar...

The genindex information is alredy provided through glossary

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-24 19:43:18 +02:00
Thomas Lamprecht a76934ad33 docs: html: adapt sidebar in index page
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-24 19:41:19 +02:00
Dominik Csapak d7a122a026 use jobstate mechanism for verify/garbage_collection schedules
also changes:
* correct comment about reset (replace 'sync' with 'action')
* check schedule change correctly (only when it is actually changed)

with this changes, we can drop the 'lookup_last_worker' method

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:12 +02:00
Dominik Csapak 6c25588e63 proxy: fix error handling in prune scheduling
we rely on the jobstate handling to write the error of the worker
into its state file, but we used '?' here in a block which does not
return the error to the block, but to the function/closure instead

so if a prune job failed because of such an '?', we did not write
into the statefile and got a wrong state there

instead use our try_block! macro that wraps the code in a closure

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:09 +02:00
Dietmar Maurer 17a1f579d0 bump version to 0.8.20-1 2020-09-24 13:17:06 +02:00
Dietmar Maurer 998db63933 src/client/pull.rs: decode, verify and write in a separate threads
To maximize throughput.
2020-09-24 13:12:04 +02:00
Dietmar Maurer c0fa14d94a src/backup/data_blob.rs: add is_encrypted helper 2020-09-24 13:00:16 +02:00
Wolfgang Bumiller 6fd129844d remove DummyCatalogWriter
we're using an `Option` instead now

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-24 09:13:54 +02:00
Dietmar Maurer baae780c99 benchmark: use compressable data to get more realistic result
And add a benchmatrk to test chunk verify speed (decompress+sha256).
2020-09-24 08:58:13 +02:00
Dietmar Maurer 09a1da25ed src/backup/data_blob.rs: improve decompress speed 2020-09-24 08:52:35 +02:00