Commit Graph

4060 Commits

Author SHA1 Message Date
Dietmar Maurer
e07620028d mark_used_chunks: simply ignore vanished files
In case a prune operation removed a file in the meantime.
2020-10-16 08:10:46 +02:00
Thomas Lamprecht
b947b1e7ee server: rest: refactor code to avoid multiple log_response calls
The 'Ok::<_, Self::Error>(res)' type annotation was from a time where
we could not use async, and had a combinator here which needed
explicity type information. We switched over to async in commit
91e4587343 and, as the type annotation
is already included in the Future type, we can safely drop it.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-15 13:58:47 +02:00
Thomas Lamprecht
1e80fb8e92 code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-15 13:58:47 +02:00
Wolfgang Bumiller
8d841f81ee pxar: anchor pxarexcludes starting with a slash
Given the .pxarexclude file

    foo
    /bar

The following happens:

    exclude: /foo
    exclude: /bar
    exclude: /subdir/foo
    include: /subdir/bar

since the `/bar` line is an absolute path

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-15 12:28:31 +02:00
Dylan Whyte
d9f365d79f Introduction: reword & link to encryption section
Add link from encryption sentence in  "What is Proxmox
Backup Server?" to the Encryption section of the docs.
Also, reword the sentence.

V2:
Clarify that encryption takes place on the client side

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-15 12:20:33 +02:00
Wolfgang Bumiller
32a4695c46 pxar: fix relative '!' rules in .pxarexclude
and reduce indentation

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-15 12:18:34 +02:00
Wolfgang Bumiller
2081327428 more clippy lints
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-15 12:18:34 +02:00
Stefan Reiter
4c0ae82e23 datastore: remove individual snapshots before group
Removing a snapshot has some more safety checks which we don't want to
ignore when removing an entire group (i.e. locking the manifest and
notifying GC).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:51:09 +02:00
Stefan Reiter
883aa6d5a4 datastore: remove load_manifest_json
There's no point in having that as a seperate method, just parse the
thing into a struct and write it back out correctly.

Also makes further changes to the method simpler.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:19:32 +02:00
Stefan Reiter
bfa54f2e85 verify: acquire shared snapshot flock and skip on error
If we can't acquire a lock (either because the snapshot disappeared, it
is about to be forgotten/pruned, or it is currently still running) skip
the snapshot. Hold the lock during verification, so that it cannot be
deleted while we are still verifying.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
238a872d1f reader: acquire shared flock on open snapshot
...to avoid it being forgotten or pruned while in use.

Update lock error message for deletions to be consistent.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
7d6c4c39e9 backup: use shared flock for base snapshot
To allow other reading operations on the base snapshot as well. No
semantic changes with this patch alone, as all other locks on snapshots
are exclusive.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
f153930066 prune: never fail, just warn about failed removals
A removal can fail if the snapshot is already gone (this is fine, our
job is done either way) or we couldn't get a lock (also fine, it can't
be removed then, just warn the user so he knows what happened and why it
wasn't removed) - keep going either way.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Stefan Reiter
836c4a278d prune: respect snapshot flock
A snapshot that's currently being read can still appear in the prune
list, but should not be removed.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-15 07:09:34 +02:00
Dylan Whyte
6cd8496008 introduction: history: minor rewording and fixup
Some minor spelling and grammar fixes.
Rewording of some sentences.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-15 07:09:34 +02:00
Wolfgang Bumiller
61c6eafc08 AsyncIndexReader: avoid memcpy, add clippy lint fixup comment
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 14:10:28 +02:00
Wolfgang Bumiller
8db1468952 more clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 13:58:35 +02:00
Dietmar Maurer
39cd81de92 bump version to 0.9.1-1 2020-10-14 13:42:30 +02:00
Dietmar Maurer
62c74d7749 use SslAcceptor::mozilla_intermediate_v5
This allows TLSv1.3, and let the client select ciphers. After this
change AES is prefered over chacha20, so TLS speed is now much faster.
2020-10-14 12:37:42 +02:00
Wolfgang Bumiller
254ec19412 pxar: remove unused parameter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:32:22 +02:00
Wolfgang Bumiller
97bbd1bf9e pxar: pass full path to callback, let verbose flag list files
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:31:42 +02:00
Wolfgang Bumiller
54aec2fa8b clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:20:07 +02:00
Dietmar Maurer
e1dfcddc79 introduction.rst: add History 2020-10-14 09:46:52 +02:00
Dylan Whyte
344add3885 fix #2847: proxmox-backup-client: add change-owner cmd
This adds a change-owner command to proxmox-backup-client,
that allows a caller with datastore modify privileges
to change the owner of a backup-group.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-14 08:56:54 +02:00
Dietmar Maurer
752dfc4bda avoid compiler warning 2020-10-14 08:36:39 +02:00
Dylan Whyte
72be0eb189 fix #2847: api: datastore: change backup owner
This adds an api method to change the owner of
a backup-group.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-14 08:31:17 +02:00
Dietmar Maurer
fdc00811ce proxmox-backup-client key: rename 'paper-key' command to 'paperkey' (remove dash) 2020-10-13 09:44:40 +02:00
Thomas Lamprecht
6c5bdef567 server/REST: check auth: code cleanup, better variable names
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 18:39:45 +02:00
Thomas Lamprecht
ea545b395b server/REST: make handle_request private
it's not used anywhere else, so do not suggest so

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 18:39:00 +02:00
Wolfgang Bumiller
f6b1d1cc66 don't require WorkerTask in backup/
To untangle the server code from the actual backup
implementation.
It would be ideal if the whole backup/ dir could become its
own crate with minimal dependencies, certainly without
depending on the actual api server. That would then also be
used more easily to create forensic tools for all the data
file types we have in the backup repositories.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-12 14:11:57 +02:00
Wolfgang Bumiller
d1993187b6 introduce TaskState trait
Used to not require access to the WorkerTask struct outside
the `server` and `api2` module, so it'll be easier to
separate those backup/server/client parts into separate
crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-12 14:11:57 +02:00
Thomas Lamprecht
adfcfb6788 server: get index: make content-type non mutable
feels more idiomatic

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 13:36:45 +02:00
Thomas Lamprecht
07995a3ca3 server/rest: code cleanup: use async
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 13:36:45 +02:00
Thomas Lamprecht
dd76eba73e readme: allow to directly copy+paste+execute commands
plus fix s/ssh:/git:/ for protocol

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 13:36:34 +02:00
Fabian Grünbichler
b13da548b9 fix #3070: replace internal with public URLs
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-12 13:33:16 +02:00
Thomas Lamprecht
fe0efb25e8 backup: index readers: drop useless shared lock
This is only acquired in those two methods, both as shared. So it has
no use.

It seems, that it was planned in the past that the index deletion
should take the exclusive, while read and write takes the shared
flock on the index, as one can guess from the lock comments in commit
0465218953

But then later, in commit c8ec450e37)
the documented semantics where changed to use a temp file and do an
atomic rename instead for atomicity.

The reader shared flock on the index file was done inbetween,
probably as preparatory step, but was not removed again when strategy
was changed to using the file rename instead.

Do so now, to avoid confusion of readers and a useless flock.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-09 12:59:31 +02:00
Hannes Laimer
b0b00c4a47 add "Build" section to README.rst
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-09 12:56:11 +02:00
Fabian Grünbichler
19ca962b15 reader: actually allow users to downlod their own backups
via HTTP2/backup reader protocol. they already could do so via the plain
HTTP download-file/.. API calls that the GUI uses, but the reader
environment required READ permission on the whole datastore instead of
just BACKUP on the backup group itself.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-09 12:54:30 +02:00
Fabian Grünbichler
d479f0c810 reader: track index chunks and limit access
a reader connection should not be allowed to read arbitrary chunks in
the datastore, but only those that were previously registered by opening
the corresponding index files.

this mechanism is needed to allow unprivileged users (that don't have
full READ permissions on the whole datastore) access to their own
backups via a reader environment.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-09 12:54:09 +02:00
Fabian Grünbichler
1d5dac1b1d REST: don't print CSRF token
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 15:57:22 +02:00
Fabian Grünbichler
96c3d98256 Userid: fix borrow/deref recursion
not triggered by any current code, but this would lead to a stack
exhaustion since borrow would call deref which would call borrow again..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 15:57:10 +02:00
Fabian Grünbichler
0b3dc8ed8c Userid: simplify comparison with str
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 15:56:57 +02:00
Fabian Grünbichler
9a75eb11cb depend on proxmox 0.4.3
needed for constnamedbitmap macro

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 13:09:52 +02:00
Thomas Lamprecht
92dd02aaf6 api: datastore: require allocate privilege for deletion
makes only sense if we allow addition of a datastore also just with
that privilege

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-08 09:44:46 +02:00
Thomas Lamprecht
41bfd24919 server: add Datastore.Allocate privilege
Previously only Datastore.Modify was required for creating a new
datastore.

But, that endpoint allows one to pass an arbitrary path, of which all
parent directories will be created, this can allow any user with the
"Datastore Admin" role on "/datastores" to do some damage to the
system. Further, it is effectively a side channel for revealing the
systems directory structure through educated guessing and error
handling.

Add a new privilege "Datastore.Allocate" which, for now, is used
specifically for the create datastore API endpoint.

Add it only to the "Admin" role.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-08 09:12:08 +02:00
Thomas Lamprecht
fddc8aa410 acl: use modified constnamedbitmap macro
avoiding the need for reshuffling all bits when a new privilege is
added at the start or in the middle of this definition.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-08 09:09:39 +02:00
Stefan Reiter
735ee5206a fuse_loop: handle unmap on crashed instance
If a fuse_loop instance dies suddenly (e.g. SIGKILL), the FUSE mount and
loop device assignment are left behind. We can determine this scenario
on specific unmap, when the PID file is either missing or contains a PID
of a non-running process, but the backing file and potentially loop
device are still there.

If that's the case, do an "emergency cleanup", by unassigning the
loopdev, calling 'fusermount -u' and then cleaning any leftover files
manually.

With this in place, pretty much any situation is now recoverable via
only the 'proxmox-backup-client' binary, by either calling 'unmap' with
or without parameters.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:38:25 +02:00
Stefan Reiter
a86bf52390 fuse_loop: wait for instance to close after killing
On unmap, only report success if the instance we are killing actually
terminates. This is especially important so that cleanup routines can be
assured that /run files are actually cleaned up after calling
cleanup_unused_run_files.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:37:47 +02:00
Stefan Reiter
2deee0e01f fuse_loop: add automatic cleanup of run files and dangling instances
A 'map' call will only clean up what it needs, that is only leftover
files or dangling instances of it's own name.

For a full cleanup the user can call 'unmap' without any arguments.

The 'cleanup on error' behaviour of map_loop is removed. It is no longer
needed (since the next call will clean up anyway), and in fact fixes a
bug where trying to map an image twice would result in an error, but
also cleanup the .pid file of the running instance, causing 'unmap' to
fail afterwards.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:37:05 +02:00
Stefan Reiter
2d7d6e61be mount/map: use names for map/unmap for easier use
So user doesn't need to remember which loop devices he has mapped to
what.

systemd unit encoding is used to transform a unique identifier for the
mapped image into a suitable name. The files created in /run/pbs-loopdev
will be named accordingly.

The encoding all happens outside fuse_loop.rs, so the fuse_loop module
does not need to care about encodings - it can always assume a name is a
valid filename.

'unmap' without parameter displays all current mappings. It's
autocompletion handler will list the names of all currently mapped
images for easy selection. Unmap by /dev/loopX or loopdev number is
maintained, as those can be distinguished from mapping names.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:35:52 +02:00