Commit Graph

2923 Commits

Author SHA1 Message Date
Dietmar Maurer
a20fcab060 fix compile warning 2020-10-28 10:47:30 +01:00
Dietmar Maurer
b9e7bcc272 send notification mails for GC and verify jobs 2020-10-28 10:44:23 +01:00
Dietmar Maurer
acc3d9df5a src/server/verify_job.rs: add missing file 2020-10-28 07:58:07 +01:00
Dietmar Maurer
1298618a83 move jobstate to server 2020-10-28 07:37:01 +01:00
Dominik Csapak
d58e6313e1 api/{verify, syncjobs}: add optional datastore parameter
to limit the lists to the given datastores

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Dominik Csapak
16f9f244cf admin/datastore: add more info to status call
add also the snapshot counts as well as the status of the last garbage
collection

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Dominik Csapak
b683fd589c backup/datastore: save garbage collection status to disk
and load it again when opening it

this way we can persist the status of the last garbage collect across
daemon reloads and reboots

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Dominik Csapak
a2285525be backup/datastore: count still bad chunks for the status
we want to show the user that there are still bad chunks after a garbage
collection

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht
f23497b088 apt auth: add newline to the end
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht
b57b3c9bfc hack: workaround unused code warning until proxmox-api-macro bump
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht
72c0e102ff tools: get_hardware_address: better error handling
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht
7b22fb257f implement subscription handling and api
mostly modelled after PVE

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht
2e201e7da6 tools: http: add simple general post method
This is intended for when the server needs to do requests on
arbitrary, non PBS, external HTTP resources.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht
ee89416319 api: disks: cleanup use statement
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht
52fe9e8ece get_hardware_address: must be uppercased
we're a bit strict here what we accept, rather than changing that
lets do it like PVE/PMG and uppercase.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-26 20:18:28 +01:00
Thomas Lamprecht
eed1bae554 api: add world accessible ping dummy endpoint
This is indented to be used for the PVE storage library, replacing
the missuse of the much more expensive status API call.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-24 19:12:14 +02:00
Fabian Grünbichler
6eb41487ce apt: improve error messages
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-22 17:13:26 +02:00
Stefan Reiter
9e61c01ce4 apt: add /changelog API call similar to PVE
For proxmox packages it works the same way as PVE, by retrieving the
changelog URL and issuing a HTTP GET to it, forwarding the output to the
client. As this is only supposed to be a workaround removed in the
future, a simple block_on is used to avoid async.

For debian packages we can simply call 'apt-get changelog' and forward
it's output.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter
91c9b42da3 fix #2934: list to-be-installed packages in updates
As always, libapt is mocking us with complexity, but we can get the
approximate result we want by retrieving dependencies of all
to-be-updated packages and then seeing if they are missing.

If they are, we assume they will be installed.

For this, query_detailed_info is extended to allow reading details for
non-installed packages, and this is also exposed in
list_installed_apt_packages via 'all_versions_for'. This is necessary so
we can retrieve changelogs for such packages.

Note that we cannot retrieve all that information all the time, as
querying details for packages that aren't installed takes a rather long
time.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter
52d2ae48f0 apt: refactor package detail reading into function
No functional change intended.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter
1872050564 apt: use 'apt-get changelog --print-uris' in get_changelog_url
Avoids custom hardcoded logic, but can only be used for debian packages
as of now. Adds a FIXME to switch over to use --print-uris only once our
package repos support that changelog format.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:26 +02:00
Stefan Reiter
efeb92efee apt: allow filter to select different package version
To get package details for a specific version instead of only the
candidate.

Also cleanup filter function with extra struct instead of unnamed &str
parameters.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:15 +02:00
Dietmar Maurer
4ebda996e5 upid: use systemd escape to decode/encode the worker_id
This way we can store values containing "/" and ":".
2020-10-22 12:24:58 +02:00
Stefan Reiter
5eb9dd0c8a add tools::http for generic HTTP GET and move HttpsConnector there
...to avoid having the tools:: module depend on api2.

The get_string function is based directly on hyper and thus relatively
simple, not supporting redirects for example.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-21 16:22:08 +02:00
Dominik Csapak
dc2876f6bb tools/zip: fix doc tests
the doc code was not compiling and blocking cargo test

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-21 14:20:16 +02:00
Hannes Laimer
2ef1b6290f api proxy: remove old verification scheduling
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer
133042b5d8 set a different worker_type based on what is going to be verified(snapshot, group, ds)
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer
73df9c515b proxy: add scheduling for verification jobs
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer
8d1beca7e8 api2: add verification admin endpoint and do_verification_job function
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer
9b2bad7af0 api2: add verification job config endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer
78efafc2d0 rename VERIFY_SCHEDULE_SCHEMA to VERIFICATION_SCHEDULE_SCHEMA
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Dietmar Maurer
2d3d91b1db add test for escape_unit 2020-10-21 11:31:24 +02:00
Dietmar Maurer
030c5c6d8a systemd::escape_unit - allow '.' and '_' 2020-10-21 11:31:24 +02:00
Wolfgang Bumiller
53a561a222 pass params by ref to recurse_files
gets rid of the return value and moving around of the zip
and decoder data
avoids cloning the path prefix on every recursion

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:50:25 +02:00
Wolfgang Bumiller
e832860a3c whitespace fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:45:44 +02:00
Dominik Csapak
804f61432d api2/admin/datastore/pxar_file_download: download directory as zip
by using the new ZipEncoder and recursively add files to it
the zip only contains directories, normal files and hardlinks (by simply
copying the content), no symlinks, etc.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:24 +02:00
Dominik Csapak
943479f5f6 tools: add AsyncChannelWriter
similar to StdChannelWriter, but implements AsyncWrite and sends
to a tokio::sync::mpsc::Sender

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:22 +02:00
Dominik Csapak
fdce52aa99 tools: add zip module
This modules contains the 'ZipEncoder' struct, which wraps an async writer,
to create a ZIP archive on the fly

To create a ZIP file, have a target that implements AsyncWrite,
give it to ZipEncoder::new, add entries via 'add_entry' and
at the end, call 'finish'

for now, this does not implement compression (uses ZIPs STORE mode), and
does not support empty directories or hardlinks (or any other special
files)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:18 +02:00
Dietmar Maurer
4e32d1c590 fix for prevoius patch: we want to copy all valid tickets 2020-10-21 08:40:04 +02:00
Dylan Whyte
afef7f3bba fix #3038: check user before renewing ticket
Fixes a bug in which the userid of the ticket cache is updated,
when a user connects, but the ticket itself is not.
This means a newly connected user has a previously connected
user's ticket and thus, cannot do anything, as the client will
attempt to use the invalid ticket.

e.g. if john@pbs connected to the server first, followed by
mike@pbs, the following would be stored in the ticket cache.

{
  "localhost": {
    "mike@pbs": {
      "ticket": "PBS:john@pbs:AAAA",
      "timestamp": 1601039326,
      "token": "BBBB"
    }
  }
}

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-21 08:34:30 +02:00
Dietmar Maurer
b428af9781 backup: avoid Transport endpoint is not connected error
We simply supress the error message if the finish flag is set.
2020-10-20 14:20:04 +02:00
Dietmar Maurer
c8774067ee paperkey: use svg as image format to provide better scalability 2020-10-20 12:04:51 +02:00
Dietmar Maurer
23440482d4 proxmox-backup-client: use HumanByte to render snapshot size 2020-10-20 11:43:48 +02:00
Thomas Lamprecht
6f757b8458 logrotate: drop useless comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:11:36 +02:00
Thomas Lamprecht
95ade8fdb5 log rotate: move basic rotation logic into module for reuse
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht
9e870b5f39 log rotate: do NOT compress first rotation
The first rotation is normally the one still opened by one or more
processes for writing, so it must NOT be replaced, removed, ..., as
this then makes the remaining logging, until those processes are
noticed that they should reopen the logfile due to rotation, goes
into nirvana, which is far from ideal for a log.

Only rotating (renaming) is OK for this active file, as this does not
invalidates the file and keeps open FDs intact.

So start compressing with the second rotation, which should be clear
to use, as all writers must have been told to reopen the log during
the last rotation, reopen is a fast operation and normally triggered
at least day ago (at least if one did not dropped the state file
manually), so we are fine to archive that one for real.
If we plan to allow faster rotation the whole rotation+reopen should
be locked, so that we can guarantee that all writers switched over,
but this is unlikely to be needed.

Again, this is was logrotate sanely does by default since forever.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht
7827e3b93e log rotate: factor out compression in private function
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht
e6ca9c3235 log rotate: do NOT overwrite file with possible writers
this is not the job of logrotate, and the real 20+ years battle
tested log rotate binary does not do so either as it's actually
pretty dangerous.

If we "replace" the file we break any logger which already opened a
new one here, e.g., a dameon starting up, and thus that writer would
log to nirvana.

It's the job of a logger to create a file if not existing, it makes
no sense to do it here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Stefan Reiter
0698f78df5 fix #2988: allow verification after finishing a snapshot
To cater to the paranoid, a new datastore-wide setting "verify-new" is
introduced. When set, a verify job will be spawned right after a new
backup is added to the store (only verifying the added snapshot).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 10:51:13 +02:00
Stefan Reiter
bcc2880461 add verify_backup_dir_with_lock for callers already holding locks
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 10:49:19 +02:00
Fabian Grünbichler
115d927c15 unbreak build
and silence warning.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-20 09:07:32 +02:00
Stefan Reiter
df729017b4 datastore: cleanup open and load config only once
Force consumers to use the lookup_datastore method instead of
potentially opening a datastore twice, and pass the config we have
already loaded into open_with_path, removing the need for open(1).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 07:51:05 +02:00
Stefan Reiter
455f2ad228 fix missing block_in_place for remove_backup
Commit 9070d11f4c introduced this change for other call sites,
assuming it is correct, this one was missed.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-20 07:48:51 +02:00
Thomas Lamprecht
e4f5f59eea code/fmt cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 15:11:51 +02:00
Fabian Grünbichler
16cdb9563b completion: fix ACL path completion
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 15:06:13 +02:00
Fabian Grünbichler
02479720c0 REST: rename token to csrf_token
for easier differentiation with (future) api_token

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 14:02:19 +02:00
Dietmar Maurer
97168f920e set reasonable TCP keepalive timeout 2020-10-19 14:01:17 +02:00
Fabian Grünbichler
9809772b23 fix typos
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 14:00:38 +02:00
Fabian Grünbichler
4940012d0d fix indentation
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 14:00:26 +02:00
Thomas Lamprecht
e5adbc3419 fixup worker task: add time prefix again
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 13:22:37 +02:00
Fabian Grünbichler
0c4c6a7b1c build: bump nix dependency
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 12:12:33 +02:00
Thomas Lamprecht
c7e18ba08a file logger: add option to make the backup user the log file owner
and use that in ApiConfig to avoid that it is owned by root if the
proxmox-backup-api process creates it first.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 10:37:26 +02:00
Dietmar Maurer
bb14d46796 http_client: set connect timeout to 10 seconds 2020-10-19 09:36:01 +02:00
Thomas Lamprecht
d39d095fa4 api: access: log to separate file, reduce syslog to errors
for now log auth errors also to the syslog, on a protected (LAN
and/or firewalled) setup this should normally happen due to
missconfiguration, not tries to break in.

This reduces syslog noise *a lot*. A current full journal output from
the current boot here has 72066 lines, of which 71444 (>99% !!) are
"successful auth for user ..." messages

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht
86f3c2363c server/rest: also log user agent
allows easily to see if a request is from a browser or a proxmox-backup-client
CLI

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht
8e7e2223d8 server/rest: implement request access log
reuse the FileLogger module in append mode.
As it implements write, which is not thread safe (mutable self) and
we use it in a async context we need to serialize access using a
mutex.

Try to use the same format we do in pveproxy, namely the one which is
also used in apache or nginx by default.

Use the response extensions to pass up the userid, if we extract it
from a ticket.

The privileged and unprivileged dameons log both to the same file, to
have a unified view, and avoiding the need to handle more log files.
We avoid extra intra-process locking by reusing the fact that a write
smaller than PIPE_BUF (4k on linux) is atomic for files opened with
the 'O_APPEND' flag. For now the logged request path is not yet
guaranteed to be smaller than that, this will be improved in a future
patch.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht
081c37cccf tools file logger: fix example and comments
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:16:29 +02:00
Thomas Lamprecht
c0df91f8bd tools: file logger: use option struct to control behavior
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:48:36 +02:00
Thomas Lamprecht
400c568f8e server: rest: also log the query part of URL
As it is part of the request and we do so in our other products

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:41:05 +02:00
Thomas Lamprecht
4703ba81ce server: rest: implement max URI path and query length request limits
Add a generous limit now and return the correct error (414 URI Too
Long). Otherwise we could to pretty larger GET requests, 64 KiB and
possible bigger (at 64 KiB my simple curl test failed due to
shell/curl limitations).

For now allow a 3072 characters as combined length of URI path and
query.

This is conform with the HTTP/1.1 RFCs (e.g., RFC 7231, 6.5.12 and
RFC 2616, 3.2.1) which do not specify any limits, upper or lower, but
require that all server accessible resources mus be reachable without
getting 414, which is normally fulfilled as we have various length
limits for stuff which could be in an URI, in place, e.g.:
 * user id: max. 64 chars
 * datastore: max. 32 chars

The only known problematic API endpoint is the catalog one, used in
the GUI's pxar file browser:
GET /api2/json/admin/datastore/<id>/catalog?..&filepath=<path>

The <path> is the encoded archive path, and can be arbitrary long.

But, this is a flawed design, as even without this new limit one can
easily generate archives which cannot be browsed anymore, as hyper
only accepts requests with max. 64 KiB in the URI.
So rather, we should move that to a GET-as-POST call, which has no
such limitations (and would not need to base32 encode the path).

Note: This change was inspired by adding a request access log, which
profits from such limits as we can then rely on certain atomicity
guarantees when writing requests to the log.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:40:39 +02:00
Thomas Lamprecht
29633e2fe9 server/rest: forward real client IP on proxied request
needs new proxmox dependency to get the RpcEnvironment changes,
adding client_ip getter and setter.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:36:54 +02:00
Stefan Reiter
b64e9a97f3 rustdoc: overhaul backup rustdoc and add locking table
Rewrite most of the documentation to be more readable and correct
(according to the current implementations).

Add a table visualizing all different locks used to synchronize
concurrent operations.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-16 09:38:01 +02:00
Stefan Reiter
254b1f2213 rustdoc: add crate level doc
Contains a link to the 'backup' module's doc, as that explains a lot
about the inner workings of PBS and probably marks a good entry point
for new readers.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-16 09:37:50 +02:00
Stefan Reiter
1a374fcfd6 datastore: add manifest locking
Avoid races when updating manifest data by flocking a lock file.
update_manifest is used to ensure updates always happen with the lock
held.

Snapshot deletion also acquires the lock, so it cannot interfere with an
outstanding manifest write.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-16 09:34:12 +02:00
Dietmar Maurer
e07620028d mark_used_chunks: simply ignore vanished files
In case a prune operation removed a file in the meantime.
2020-10-16 08:10:46 +02:00
Thomas Lamprecht
b947b1e7ee server: rest: refactor code to avoid multiple log_response calls
The 'Ok::<_, Self::Error>(res)' type annotation was from a time where
we could not use async, and had a combinator here which needed
explicity type information. We switched over to async in commit
91e4587343 and, as the type annotation
is already included in the Future type, we can safely drop it.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-15 13:58:47 +02:00
Thomas Lamprecht
1e80fb8e92 code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-15 13:58:47 +02:00
Wolfgang Bumiller
8d841f81ee pxar: anchor pxarexcludes starting with a slash
Given the .pxarexclude file

    foo
    /bar

The following happens:

    exclude: /foo
    exclude: /bar
    exclude: /subdir/foo
    include: /subdir/bar

since the `/bar` line is an absolute path

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-15 12:28:31 +02:00
Wolfgang Bumiller
32a4695c46 pxar: fix relative '!' rules in .pxarexclude
and reduce indentation

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-15 12:18:34 +02:00
Wolfgang Bumiller
2081327428 more clippy lints
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-15 12:18:34 +02:00
Stefan Reiter
4c0ae82e23 datastore: remove individual snapshots before group
Removing a snapshot has some more safety checks which we don't want to
ignore when removing an entire group (i.e. locking the manifest and
notifying GC).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:51:09 +02:00
Stefan Reiter
883aa6d5a4 datastore: remove load_manifest_json
There's no point in having that as a seperate method, just parse the
thing into a struct and write it back out correctly.

Also makes further changes to the method simpler.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:19:32 +02:00
Stefan Reiter
bfa54f2e85 verify: acquire shared snapshot flock and skip on error
If we can't acquire a lock (either because the snapshot disappeared, it
is about to be forgotten/pruned, or it is currently still running) skip
the snapshot. Hold the lock during verification, so that it cannot be
deleted while we are still verifying.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
238a872d1f reader: acquire shared flock on open snapshot
...to avoid it being forgotten or pruned while in use.

Update lock error message for deletions to be consistent.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
7d6c4c39e9 backup: use shared flock for base snapshot
To allow other reading operations on the base snapshot as well. No
semantic changes with this patch alone, as all other locks on snapshots
are exclusive.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
f153930066 prune: never fail, just warn about failed removals
A removal can fail if the snapshot is already gone (this is fine, our
job is done either way) or we couldn't get a lock (also fine, it can't
be removed then, just warn the user so he knows what happened and why it
wasn't removed) - keep going either way.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
836c4a278d prune: respect snapshot flock
A snapshot that's currently being read can still appear in the prune
list, but should not be removed.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Wolfgang Bumiller
61c6eafc08 AsyncIndexReader: avoid memcpy, add clippy lint fixup comment
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 14:10:28 +02:00
Wolfgang Bumiller
8db1468952 more clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 13:58:35 +02:00
Dietmar Maurer
62c74d7749 use SslAcceptor::mozilla_intermediate_v5
This allows TLSv1.3, and let the client select ciphers. After this
change AES is prefered over chacha20, so TLS speed is now much faster.
2020-10-14 12:37:42 +02:00
Wolfgang Bumiller
254ec19412 pxar: remove unused parameter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:32:22 +02:00
Wolfgang Bumiller
97bbd1bf9e pxar: pass full path to callback, let verbose flag list files
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:31:42 +02:00
Wolfgang Bumiller
54aec2fa8b clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:20:07 +02:00
Dylan Whyte
344add3885 fix #2847: proxmox-backup-client: add change-owner cmd
This adds a change-owner command to proxmox-backup-client,
that allows a caller with datastore modify privileges
to change the owner of a backup-group.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-14 08:56:54 +02:00
Dietmar Maurer
752dfc4bda avoid compiler warning 2020-10-14 08:36:39 +02:00
Dylan Whyte
72be0eb189 fix #2847: api: datastore: change backup owner
This adds an api method to change the owner of
a backup-group.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-14 08:31:17 +02:00
Dietmar Maurer
fdc00811ce proxmox-backup-client key: rename 'paper-key' command to 'paperkey' (remove dash) 2020-10-13 09:44:40 +02:00
Thomas Lamprecht
6c5bdef567 server/REST: check auth: code cleanup, better variable names
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 18:39:45 +02:00
Thomas Lamprecht
ea545b395b server/REST: make handle_request private
it's not used anywhere else, so do not suggest so

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 18:39:00 +02:00
Wolfgang Bumiller
f6b1d1cc66 don't require WorkerTask in backup/
To untangle the server code from the actual backup
implementation.
It would be ideal if the whole backup/ dir could become its
own crate with minimal dependencies, certainly without
depending on the actual api server. That would then also be
used more easily to create forensic tools for all the data
file types we have in the backup repositories.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-12 14:11:57 +02:00
Wolfgang Bumiller
d1993187b6 introduce TaskState trait
Used to not require access to the WorkerTask struct outside
the `server` and `api2` module, so it'll be easier to
separate those backup/server/client parts into separate
crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-12 14:11:57 +02:00
Thomas Lamprecht
adfcfb6788 server: get index: make content-type non mutable
feels more idiomatic

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 13:36:45 +02:00
Thomas Lamprecht
07995a3ca3 server/rest: code cleanup: use async
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 13:36:45 +02:00
Thomas Lamprecht
fe0efb25e8 backup: index readers: drop useless shared lock
This is only acquired in those two methods, both as shared. So it has
no use.

It seems, that it was planned in the past that the index deletion
should take the exclusive, while read and write takes the shared
flock on the index, as one can guess from the lock comments in commit
0465218953

But then later, in commit c8ec450e37)
the documented semantics where changed to use a temp file and do an
atomic rename instead for atomicity.

The reader shared flock on the index file was done inbetween,
probably as preparatory step, but was not removed again when strategy
was changed to using the file rename instead.

Do so now, to avoid confusion of readers and a useless flock.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-09 12:59:31 +02:00
Fabian Grünbichler
19ca962b15 reader: actually allow users to downlod their own backups
via HTTP2/backup reader protocol. they already could do so via the plain
HTTP download-file/.. API calls that the GUI uses, but the reader
environment required READ permission on the whole datastore instead of
just BACKUP on the backup group itself.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-09 12:54:30 +02:00
Fabian Grünbichler
d479f0c810 reader: track index chunks and limit access
a reader connection should not be allowed to read arbitrary chunks in
the datastore, but only those that were previously registered by opening
the corresponding index files.

this mechanism is needed to allow unprivileged users (that don't have
full READ permissions on the whole datastore) access to their own
backups via a reader environment.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-09 12:54:09 +02:00
Fabian Grünbichler
1d5dac1b1d REST: don't print CSRF token
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 15:57:22 +02:00
Fabian Grünbichler
96c3d98256 Userid: fix borrow/deref recursion
not triggered by any current code, but this would lead to a stack
exhaustion since borrow would call deref which would call borrow again..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 15:57:10 +02:00
Fabian Grünbichler
0b3dc8ed8c Userid: simplify comparison with str
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 15:56:57 +02:00
Thomas Lamprecht
92dd02aaf6 api: datastore: require allocate privilege for deletion
makes only sense if we allow addition of a datastore also just with
that privilege

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-08 09:44:46 +02:00
Thomas Lamprecht
41bfd24919 server: add Datastore.Allocate privilege
Previously only Datastore.Modify was required for creating a new
datastore.

But, that endpoint allows one to pass an arbitrary path, of which all
parent directories will be created, this can allow any user with the
"Datastore Admin" role on "/datastores" to do some damage to the
system. Further, it is effectively a side channel for revealing the
systems directory structure through educated guessing and error
handling.

Add a new privilege "Datastore.Allocate" which, for now, is used
specifically for the create datastore API endpoint.

Add it only to the "Admin" role.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-08 09:12:08 +02:00
Thomas Lamprecht
fddc8aa410 acl: use modified constnamedbitmap macro
avoiding the need for reshuffling all bits when a new privilege is
added at the start or in the middle of this definition.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-08 09:09:39 +02:00
Stefan Reiter
735ee5206a fuse_loop: handle unmap on crashed instance
If a fuse_loop instance dies suddenly (e.g. SIGKILL), the FUSE mount and
loop device assignment are left behind. We can determine this scenario
on specific unmap, when the PID file is either missing or contains a PID
of a non-running process, but the backing file and potentially loop
device are still there.

If that's the case, do an "emergency cleanup", by unassigning the
loopdev, calling 'fusermount -u' and then cleaning any leftover files
manually.

With this in place, pretty much any situation is now recoverable via
only the 'proxmox-backup-client' binary, by either calling 'unmap' with
or without parameters.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:38:25 +02:00
Stefan Reiter
a86bf52390 fuse_loop: wait for instance to close after killing
On unmap, only report success if the instance we are killing actually
terminates. This is especially important so that cleanup routines can be
assured that /run files are actually cleaned up after calling
cleanup_unused_run_files.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:37:47 +02:00
Stefan Reiter
2deee0e01f fuse_loop: add automatic cleanup of run files and dangling instances
A 'map' call will only clean up what it needs, that is only leftover
files or dangling instances of it's own name.

For a full cleanup the user can call 'unmap' without any arguments.

The 'cleanup on error' behaviour of map_loop is removed. It is no longer
needed (since the next call will clean up anyway), and in fact fixes a
bug where trying to map an image twice would result in an error, but
also cleanup the .pid file of the running instance, causing 'unmap' to
fail afterwards.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:37:05 +02:00
Stefan Reiter
2d7d6e61be mount/map: use names for map/unmap for easier use
So user doesn't need to remember which loop devices he has mapped to
what.

systemd unit encoding is used to transform a unique identifier for the
mapped image into a suitable name. The files created in /run/pbs-loopdev
will be named accordingly.

The encoding all happens outside fuse_loop.rs, so the fuse_loop module
does not need to care about encodings - it can always assume a name is a
valid filename.

'unmap' without parameter displays all current mappings. It's
autocompletion handler will list the names of all currently mapped
images for easy selection. Unmap by /dev/loopX or loopdev number is
maintained, as those can be distinguished from mapping names.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:35:52 +02:00
Stefan Reiter
4ec17f7eb5 loopdev: add module doc
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:34:39 +02:00
Stefan Reiter
fcad02e1de fuse_loop: add documentation
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:33:34 +02:00
Stefan Reiter
708fab3082 format: fix typo in function name
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:32:21 +02:00
Dominik Csapak
1a558edd0b api2/status: add type- and statusfilter to tasks api call
we will use this for the pbs dashboard

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-06 12:58:52 +02:00
Dominik Csapak
5976c392ad api2/types: add TaskStateType struct
the same as the regular TaskState, but without its fields, so that
we can use the api macro and use it as api call parameter

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-06 12:58:52 +02:00
Thomas Lamprecht
05be0984b4 acl: document Admin and NoAccess a bit
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-06 12:23:22 +02:00
Dietmar Maurer
2995aedf1d src/bin/proxmox_backup_client/mount.rs: fix img name completion 2020-10-06 09:40:00 +02:00
Stefan Reiter
45f9b32e0f client: implement map/unmap commands for .img backups
Allows mapping fixed-index .img files (usually from VM backups) to be
mapped to a local loopback device.

The architecture uses a FUSE-backed temp file mapped to a loopdev:

  /dev/loopX -> FUSE /run/pbs-loopdev/xxx -> backup client -> PBS

Since unmapping requires some cleanup (unmap the loopdev, stop FUSE,
remove the temp files) a special 'unmap' command is added, which uses a
PID file to send SIGINT to the backup-client instance started with
'map', which will handle the cleanup itself.

The polling with select! in mount.rs needs to be split in two, since we
have a chicken and egg problem between running FUSE and setting up the
loop device - so we need to do them concurrently, until the loopdev is
assigned, at which point we can report success and daemonize, and then
continue polling the FUSE loop future.

A loopdev module is added to tools containing all required functions for
mapping a loop device to the FUSE file, with the ioctls moved into an
inline module to avoid exposing them directly.

The client code is placed in the 'mount' module, which, while
admittedly a loose fit, allows reuse of the daemonizing code.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-06 09:08:14 +02:00
Stefan Reiter
1d0b662b42 mount: handle SIGTERM as well
instead of only SIGINT

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-06 09:08:14 +02:00
Dominik Csapak
2d81f7b0c0 task archive rotation: better handle non-existing archive
if the archive file does not exist yet, we cannot rotate it, but it's not
actually an error, so just return Ok(false) to indicate no rotation took
place

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-05 10:04:16 +02:00
Thomas Lamprecht
0ac612476a REST server: avoid hard coding world readable API endpoints
while we probably do not add much more to them, it still looks ugly.

If this was made so that adding a World readable API call is "hard"
and not done by accident, it rather should be done as a test on build
time. But, IMO, the API permission schema definitions are easy to
review, and not often changed/added - so any wrong World readable API
call will normally still caught.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-05 08:29:43 +02:00
Thomas Lamprecht
4e6dc58727 rest server: cleanup use statements
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-02 13:04:08 +02:00
Dietmar Maurer
a71bc08ff4 src/tools/parallel_handler.rs: remove lifetime hacks, require 'static
In theory, one can do std::mem::forget, and ignore the drop handler. With
the lifetime hack, this could result in a crash.

So we simply require 'static lifetime now (futures also needs that).
2020-10-01 14:52:48 +02:00
Dietmar Maurer
0a8f3ae0b3 src/tools/parallel_handler.rs: cleanup check_abort code 2020-10-01 14:37:29 +02:00
Stefan Reiter
da6e67b321 rrd: fix integer underflow
Causes a panic if last_update is smaller than RRD_DATA_ENTRIES*reso,
which (I believe) can happen when inserting the first value for a DB.

Clamp the value to 0 in that case.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Stefan Reiter
dec00364b3 ParallelHandler: check for errors during thread join
Fix a potential bug where errors that happen after the SendHandle has
been dropped while doing the thread join might have been ignored.
Requires internal check_abort to be moved out of 'impl SendHandle' since
we only have the Mutex left, not the SendHandle.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Thomas Lamprecht
823867f5b7 datastore: gc: avoid unsafe call into libc, use epoch_i64 helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 12:38:38 +02:00
Thomas Lamprecht
c6772c92b8 datastore: gc: comment exclusive process lock
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 12:38:04 +02:00
Thomas Lamprecht
79f6a79cfc assume correct backup, avoid verifying chunk existance
This can slow things down by a lot on setups with (relatively) high
seek time, in the order of doubling the backup times if cache isn't
populated with the last backups chunk inode info.

Effectively there's nothing known this protects us from in the
codebase. The only thing which was theorized about was the case
where a really long running backup job (over 24 hours) is still
running and writing new chunks, not indexed yet anywhere, then an
update (or manual action) triggers a reload of the proxy. There was
some theory that then a GC in the new daemon would not know about the
oldest writer in the old one, and thus use a less strict atime limit
for chunk sweeping - opening up a window for deleting chunks from the
long running backup.
But, this simply cannot happen as we have a per datastore process
wide flock, which is acquired shared by backup jobs and exclusive by
GC. In the same process GC and backup can both get it, as it has a
process locking granularity. If there's an old daemon with a writer,
that also has the lock open shared, and so no GC in the new process
can get exclusive access to it.

So, with that confirmed we have no need for a "half-assed"
verification in the backup finish step. Rather, we plan to add an
opt-in "full verify each backup on finish" option (see #2988)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 12:06:59 +02:00
Dietmar Maurer
4c7f100d22 src/api2/reader.rs: fix speedtest description 2020-10-01 11:16:15 +02:00
Dietmar Maurer
9070d11f4c src/api2/backup.rs: use block_in_place for remove_backup 2020-10-01 11:11:14 +02:00
Dietmar Maurer
124b93f31c upload_chunk: use block_in_place 2020-10-01 11:00:23 +02:00
Dominik Csapak
4c95d58c41 api2/types: fix DNS_NAME Regexes
We forgot to put braces around the DNS_NAME regex, and in
DNS_NAME_OR_IP_REGEX

this is wrong because the regex

 ^foo|bar$

matches 'foo' at the beginning and 'bar' at the end, so either

 foobaz
 bazbar

would match. only

 ^(foo|bar)$

 matches only 'foo' and 'bar'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-01 06:09:34 +02:00
Dominik Csapak
38d4675921 fix ipv6 handling for remotes/sync jobs
* add square brackets to ipv6 adresses in BackupRepository if they not
already have some (we save them without in the remote config)

* in get_pull_parameters, we now create a BackupRepository first and use
  those values (which does the [] mapping), this also has the advantage
  that we have one place less were we hardcode 8007 as port

* in the ui, add square brackets for ipv6 adresses for remotes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 13:40:03 +02:00
Dietmar Maurer
7b8aa893fa src/client/pull.rs: log progress 2020-09-30 13:35:09 +02:00
Dominik Csapak
df4827f2c0 tasks: improve behaviour on upgrade
when upgrading from a version where we stored all tasks in the 'active' file,
we did not completly account for finished tasks still there

we should update the file when encountering any finished task in
'active' as well as filter them out on the api call (if they get through)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 11:05:50 +02:00
Dietmar Maurer
ef1b436350 paperkey: add html output 2020-09-30 10:49:20 +02:00
Dietmar Maurer
e64b9f9204 src/tools.rs: make command_output return Vec<u8>
And add a new helper to return output as string.
2020-09-30 10:49:20 +02:00
Dominik Csapak
ba20987ae7 client/remote: add support to specify port number
this adds the ability to add port numbers in the backup repo spec
as well as remotes, so that user that are behind a
NAT/Firewall/Reverse proxy can still use it

also adds some explanation and examples to the docs to make it clearer
for h2 client i left the localhost:8007 part, since it is not
configurable where we bind to

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 10:49:20 +02:00
Thomas Lamprecht
729d41fe6a api: disks/zfs: check template exsits before enabling zfs-import service
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-30 09:34:21 +02:00
Stoiko Ivanov
905147a5ee api2/node/disks/zfs: instantiate import service
When creating a new zpool for a datastore, also instantiate an
import-unit for it. This helps in cases where '/etc/zfs/zool.cache'
get corrupted and thus the pool is not imported upon boot.

This patch needs the corresponding addition of 'zfs-import@.service' in
the zfsonlinux repository.

Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-09-30 08:43:38 +02:00
Dominik Csapak
60b9b48e71 require square brackets for ipv6 addresses
we need this, because we append the port to this to get a target url
e.g. we print

format!("https://{}:8007/", address)

if address is now an ipv6 (e.g. fe80::1) it would become

https://fe80::1:8007/ which is a valid ipv6 on its own

by using square brackets we get:

https://[fe80::1]:8007/ which now connects to the correct ip/port

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:16:27 +02:00
Dominik Csapak
7eebe1483e server/worker_task: fix panic on slice range when index is empty
since len() and MAX_INDEX_TASKS are both usize, they underflow
instead of getting negative values

instead check the sizes and set them accordingly

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:11:06 +02:00
Dominik Csapak
9a76091785 proxmox-backup-proxy: add task archive rotation
this starts a task once a day at "00:00" that rotates the task log
archive if it is bigger than 500k

if we want, we can make the schedule/size limit/etc. configurable,
but for now it's ok to set fixed values for that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:41:18 +02:00
Dominik Csapak
c386b06fc6 server/worker_task: remove unecessary read_task_list
since there are no users of this anymore and we now have a nicer
TaskListInfoIterator to use, we can drop this function

this also means that 'update_active_workers' does not need to return
a list anymore since we never used that result besides in
read_task_list

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:50 +02:00
Dominik Csapak
6bcfc5c1a4 api2/status: use the TaskListInfoIterator here
this means that limiting with epoch now works correctly
also change the api type to i64, since that is what the starttime is
saved as

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:24 +02:00
Dominik Csapak
768e10d0b3 api2/node/tasks: use TaskListInfoIterator instead of read_task_list
this makes the filtering/limiting much nicer and readable

since we now have potentially an 'infinite' amount of tasks we iterate over,
and cannot now beforehand how many there are, we return the total count
as always 1 higher then requested iff we are not at the end (this is
the case when the amount of entries is smaller than the requested limit)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:02 +02:00
Dominik Csapak
e7244387c7 server/worker_task: add TaskListInfoIterator
this is an iterator that reads/parses/updates the task list as
necessary and returns the tasks in descending order (newest first)

it does this by using our logrotate iterator and using a vecdeque

we can use this to iterate over all tasks, even if they are in the
archive and even if the archive is logrotated but only read
as much as we need

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:39:16 +02:00
Dominik Csapak
5ade6c25f3 server/worker_task: write older tasks into archive file
instead of removing tasks beyond the 1000 that are in the index
write them into an archive file by appending them at the end
this way we can later still read them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:38:44 +02:00
Dominik Csapak
784fa1c2e3 server/worker_task: split task list file into two
one for only the active tasks and one for up to 1000 finished tasks

factor out the parsing of a task file (we will later need this again)
and use iterator combinators for easier code

we now sort the tasks ascending (this will become important in a later patch)
but reverse (for now) it to keep compatibility

this code also omits the converting into an intermittent hash
since it cannot really happen that we have duplicate tasks in this list
(since the call is locked by an flock, and it is the only place where we
write into the lists)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:38:28 +02:00
Dominik Csapak
66f4e6a809 server/worker_task: refactor locking of the task list
also add the functionality of having a 'shared' (read) lock for the list
we will need this later

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:37:54 +02:00
Dominik Csapak
8074d2b0c3 tools: add logrotate module
this is a helper to rotate and iterate over log files
there is an iterator for open filehandles as well as
only the filename

also it has the possibilty to rotate them
for compression, zstd is used

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:33:21 +02:00
Dietmar Maurer
b02d49ab26 proxmox_backup_client key: allow to generate paperkey for master key 2020-09-29 08:29:42 +02:00
Dietmar Maurer
82a0cd2ad4 proxmox_backup_client key: add new paper-key command 2020-09-29 08:29:42 +02:00
Wolfgang Bumiller
ee1a9c3230 parallel_handler: clippy: 'while_let_loop'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-29 08:13:51 +02:00
Wolfgang Bumiller
db24c01106 parallel_handler: explicit Arc::clone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-28 13:40:03 +02:00
Wolfgang Bumiller
ae3cfa8f0d parallel_handler: formatting cleanup, doc comment typo fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-28 13:40:03 +02:00
Dietmar Maurer
b56c111e93 depend on proxmox 0.4.2 2020-09-28 10:50:44 +02:00
Dominik Csapak
bbeb0256f1 server/worker_task: factor out task list rendering
we will need this later again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-28 07:31:27 +02:00
Dominik Csapak
005a5b9677 api2/node/tasks: move userfilter to function signature
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-28 07:18:13 +02:00
Dietmar Maurer
55bee04856 src/tools/parallel_handler.rs: remove unnecessary Sync bound 2020-09-26 16:16:11 +02:00
Dietmar Maurer
42fd40a124 src/bin/proxmox_backup_client/benchmark.rs: avoid compiler warning 2020-09-26 16:13:19 +02:00
Dietmar Maurer
f21508b9e1 src/backup/verify.rs: use ParallelHandler to verify chunks 2020-09-26 11:14:37 +02:00
Dietmar Maurer
ee7a308de4 src/backup/verify.rs: cleanup use clause 2020-09-26 10:23:44 +02:00
Dietmar Maurer
636e674ee7 src/client/pull.rs: simplify code 2020-09-26 10:09:51 +02:00
Dietmar Maurer
b02b374b46 src/tools/parallel_handler.rs: remove static lifetime bound from handler_fn 2020-09-26 09:26:06 +02:00
Dietmar Maurer
1c13afa8f9 src/tools/parallel_handler.rs: join all threads in drop handler 2020-09-26 08:47:56 +02:00
Dietmar Maurer
69b92fab7e src/tools/parallel_handler.rs: remove unnecessary Sync trait bound 2020-09-26 07:38:44 +02:00
Dietmar Maurer
5441708634 src/client/pull.rs: use new ParallelHandler 2020-09-25 12:58:20 +02:00
Dietmar Maurer
3c9b370255 src/tools/parallel_handler.rs: execute closure inside a thread pool 2020-09-25 12:58:20 +02:00
Dominik Csapak
d7a122a026 use jobstate mechanism for verify/garbage_collection schedules
also changes:
* correct comment about reset (replace 'sync' with 'action')
* check schedule change correctly (only when it is actually changed)

with this changes, we can drop the 'lookup_last_worker' method

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:12 +02:00
Dominik Csapak
6c25588e63 proxy: fix error handling in prune scheduling
we rely on the jobstate handling to write the error of the worker
into its state file, but we used '?' here in a block which does not
return the error to the block, but to the function/closure instead

so if a prune job failed because of such an '?', we did not write
into the statefile and got a wrong state there

instead use our try_block! macro that wraps the code in a closure

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:09 +02:00
Dietmar Maurer
998db63933 src/client/pull.rs: decode, verify and write in a separate threads
To maximize throughput.
2020-09-24 13:12:04 +02:00
Dietmar Maurer
c0fa14d94a src/backup/data_blob.rs: add is_encrypted helper 2020-09-24 13:00:16 +02:00
Wolfgang Bumiller
6fd129844d remove DummyCatalogWriter
we're using an `Option` instead now

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-24 09:13:54 +02:00
Dietmar Maurer
baae780c99 benchmark: use compressable data to get more realistic result
And add a benchmatrk to test chunk verify speed (decompress+sha256).
2020-09-24 08:58:13 +02:00
Dietmar Maurer
09a1da25ed src/backup/data_blob.rs: improve decompress speed 2020-09-24 08:52:35 +02:00
Dietmar Maurer
a83e2ffeab src/api2/reader.rs: use std::fs::read instead of tokio::fs::read
Because it is about 10%& faster this way.
2020-09-22 13:27:23 +02:00
Dietmar Maurer
ebbe4958c6 src/client/pull.rs: avoid duplicate downloads using in memory HashSet 2020-09-22 12:34:06 +02:00
Dietmar Maurer
73b2cc4977 src/client/pull.rs: allow up to 20 concurrent download streams 2020-09-22 11:39:31 +02:00
Dietmar Maurer
7ecfde8150 remote_chunk_reader.rs: use Arc for cache_hint to make clone faster 2020-09-22 11:39:31 +02:00
Dietmar Maurer
e0085e6612 src/client/pull.rs: remove temporary manifest 2020-09-21 14:03:01 +02:00
Dietmar Maurer
194da6f867 src/client/pull.rs: open temporary manifest with truncate(true)
To delete any data if the file already exists.
2020-09-21 13:53:35 +02:00
Dominik Csapak
5e39918fe1 fix #3017: check array boundaries before using
else we panic here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-21 09:22:06 +02:00
Dietmar Maurer
12c65bacf1 src/backup/chunk_store.rs: disable debug output 2020-09-19 15:26:21 +02:00
Dietmar Maurer
ba37f3562d src/backup/datastore.rs - open_with_path: use Path instead of str 2020-09-19 10:01:57 +02:00
Dietmar Maurer
fce4659388 src/backup/datastore.rs: new method open_with_path
To make testing easier.
2020-09-19 09:55:21 +02:00
Dominik Csapak
9866de5e3d datastore/prune schedules: use JobState for tracking of schedules
like the sync jobs, so that if an admin configures a schedule it
really starts the next time that time is reached not immediately

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-19 06:24:37 +02:00
Fabian Grünbichler
be3bd0f90b fix #3015: allow user self-service
listing, updating or deleting a user is now possible for the user
itself, in addition to higher-privileged users that have appropriate
privileges on '/access/users'.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 15:45:11 +02:00
Fabian Grünbichler
3c053adbb5 role api: fix description
wrongly copy-pasted at some point

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 14:55:00 +02:00
Hannes Laimer
c040ec22f7 add verification scheduling to proxmox-backup-proxy
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:14:05 +02:00
Hannes Laimer
2b67de2e3f api2: make verify_schedule deletable
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:12:29 +02:00
Hannes Laimer
477859662a api2: add optional verify-schdule field to create/update datastore endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:12:16 +02:00
Hannes Laimer
ccd7241e2f add verify_schedule field to DataStoreConfig
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:11:55 +02:00