Commit Graph

240 Commits

Author SHA1 Message Date
Thomas Lamprecht
a68768cf31 server: use generalized commando socket for worker tasks commands
Allows to extend the use of that socket in the future, e.g., for log
rotate re-open signaling.

To reflect this we use a more general name, and change the commandos
to a more clear namespace.

Both are actually somewhat a breaking change, but the single real
world issue it should be able to cause is, that one won't be able to
stop task from older daemons, which still use the older abstract
socket name format.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:48:04 +01:00
Thomas Lamprecht
f3df613cb7 server: add CommandoSocket where multiple users can register commands
This is a preparatory step to replace the task control socket with it
and provide a "reopen log file" command for the rest server.

Kept it simple by disallowing to register new commands after the
socket gets spawned, this avoids the need for locking.

If we really need that we can always wrap it in a Arc<RWLock<..>> or
something like that, or even nicer, register at compile time.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 19:32:22 +01:00
Thomas Lamprecht
346a488e35 pull out /run and /var/log directory constants to buildcfg
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-11-02 10:03:10 +01:00
Dietmar Maurer
3066f56481 notify: add link to server GUI 2020-11-02 09:12:14 +01:00
Dietmar Maurer
07ca4e3609 gc: remove extra empty lines in email notification template 2020-11-02 09:12:14 +01:00
Thomas Lamprecht
86d602457a api: apt: implement support to send notification email on new updates
again, base idea copied off PVE, but, we safe the information about
which pending version we send a mail out already in a separate
object, to keep the api return type APTUpdateInfo clean.

This also makes a few things a bit easier, as we can update the
package status without saving/restoring the notify information.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 22:51:26 +01:00
Thomas Lamprecht
0bb74e54b1 worker task: drop debug prints
they are not useful anymore, rather noisy

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:36 +01:00
Thomas Lamprecht
d0abba3397 trivial: fix typo in comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-31 21:31:36 +01:00
Dietmar Maurer
1143f6ca93 cleanup: fix wording in GC status emails 2020-10-31 07:56:42 +01:00
Fabian Grünbichler
09f6a24078 verify: introduce & use new Datastore.Verify privilege
for verifying a whole datastore. Datastore.Backup now allows verifying
only backups owned by the triggering user.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 16:36:52 +01:00
Thomas Lamprecht
625a56b75e server/rest: accept also = as token separator
Like we do in Proxmox VE

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-30 13:34:26 +01:00
Thomas Lamprecht
6d8a1ac9e4 server/rest: user constants for HTTP headers
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-30 13:33:36 +01:00
Fabian Grünbichler
362739054e api tokens: add authorization method
and properly decode secret (which is a no-op with the current scheme).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 13:15:14 +01:00
Dietmar Maurer
3b707fbb8f proxy: split out code to run garbage collection job 2020-10-30 11:01:45 +01:00
Hannes Laimer
b8d9079835 proxy: move prune logic into new file
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-30 09:49:50 +01:00
Fabian Grünbichler
c30816c1f8 REST: extract and handle API tokens
and refactor handling of headers in the REST server while we're at it.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler
e6dc35acb8 replace Userid with Authid
in most generic places. this is accompanied by a change in
RpcEnvironment to purposefully break existing call sites.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:11:39 +01:00
Dominik Csapak
264779e704 server/worker_task: simplify task log writing
instead of prerotating 1000 tasks
(which resulted in 2 writes each time an active worker was finished)
simply append finished tasks to the archive (which will be rotated)

page cache should be good enough so that we can get the task logs fast

since existing installations might have an 'index' file, we
still have to read tasks from there, but only if it exists

this simplifies the TaskListInfoIterator a good amount

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-29 12:41:20 +01:00
Thomas Lamprecht
7f3d91003c worker task: remove debug print, faster modulo
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-29 12:35:33 +01:00
Dietmar Maurer
9e733dae48 send sync job status emails 2020-10-29 12:22:50 +01:00
Dietmar Maurer
385cf2bd9d send_job_status_mail: corectly escape html characters 2020-10-29 11:22:08 +01:00
Dietmar Maurer
d6373f3525 garbage_collection: log deduplication factor 2020-10-29 11:13:01 +01:00
Dietmar Maurer
d771a608f5 verify: directly pass manifest to filter function
In order to avoid loading the manifest twice during verify.
2020-10-29 07:59:19 +01:00
Thomas Lamprecht
b4b14dc16e do_verification_job: fix "never-reverify" and refactor/comment
commit a4915dfc2b made a wrong fix, as
it did not observed that the last expressions was done under the
invariant that we had a last verification result, because if none
could be loaded we already returned true (include).

It thus broke the case for "never re-verify", which is important when
using multiple schedules, a more high frequent one for new,
unverified snapshots, and a low frequency to re-verify older snapshots,
e.g., monthly.

Fix this case again, rework the code to avoid this easy to oversee
invariant. Use a nested match to better express the implication of
each setting, and add some comments.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-28 16:12:09 +01:00
Thomas Lamprecht
5428f5ca29 do verification: always verify if manifest load fails
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-28 14:11:44 +01:00
Dietmar Maurer
a4915dfc2b verify: improve code reuse, fix filter function
Try to reuse verify_all_backups(), because this function has better
logging and well defined snaphot order.
2020-10-28 12:58:15 +01:00
Dietmar Maurer
d642802d8c jobstate: fix doctest 2020-10-28 10:52:16 +01:00
Dietmar Maurer
b9e7bcc272 send notification mails for GC and verify jobs 2020-10-28 10:44:23 +01:00
Dietmar Maurer
acc3d9df5a src/server/verify_job.rs: add missing file 2020-10-28 07:58:07 +01:00
Dietmar Maurer
1298618a83 move jobstate to server 2020-10-28 07:37:01 +01:00
Dietmar Maurer
4ebda996e5 upid: use systemd escape to decode/encode the worker_id
This way we can store values containing "/" and ":".
2020-10-22 12:24:58 +02:00
Thomas Lamprecht
95ade8fdb5 log rotate: move basic rotation logic into module for reuse
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht
e4f5f59eea code/fmt cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 15:11:51 +02:00
Fabian Grünbichler
02479720c0 REST: rename token to csrf_token
for easier differentiation with (future) api_token

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-19 14:02:19 +02:00
Thomas Lamprecht
e5adbc3419 fixup worker task: add time prefix again
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 13:22:37 +02:00
Thomas Lamprecht
c7e18ba08a file logger: add option to make the backup user the log file owner
and use that in ApiConfig to avoid that it is owned by root if the
proxmox-backup-api process creates it first.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-19 10:37:26 +02:00
Thomas Lamprecht
86f3c2363c server/rest: also log user agent
allows easily to see if a request is from a browser or a proxmox-backup-client
CLI

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht
8e7e2223d8 server/rest: implement request access log
reuse the FileLogger module in append mode.
As it implements write, which is not thread safe (mutable self) and
we use it in a async context we need to serialize access using a
mutex.

Try to use the same format we do in pveproxy, namely the one which is
also used in apache or nginx by default.

Use the response extensions to pass up the userid, if we extract it
from a ticket.

The privileged and unprivileged dameons log both to the same file, to
have a unified view, and avoiding the need to handle more log files.
We avoid extra intra-process locking by reusing the fact that a write
smaller than PIPE_BUF (4k on linux) is atomic for files opened with
the 'O_APPEND' flag. For now the logged request path is not yet
guaranteed to be smaller than that, this will be improved in a future
patch.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 11:23:49 +02:00
Thomas Lamprecht
c0df91f8bd tools: file logger: use option struct to control behavior
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:48:36 +02:00
Thomas Lamprecht
400c568f8e server: rest: also log the query part of URL
As it is part of the request and we do so in our other products

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:41:05 +02:00
Thomas Lamprecht
4703ba81ce server: rest: implement max URI path and query length request limits
Add a generous limit now and return the correct error (414 URI Too
Long). Otherwise we could to pretty larger GET requests, 64 KiB and
possible bigger (at 64 KiB my simple curl test failed due to
shell/curl limitations).

For now allow a 3072 characters as combined length of URI path and
query.

This is conform with the HTTP/1.1 RFCs (e.g., RFC 7231, 6.5.12 and
RFC 2616, 3.2.1) which do not specify any limits, upper or lower, but
require that all server accessible resources mus be reachable without
getting 414, which is normally fulfilled as we have various length
limits for stuff which could be in an URI, in place, e.g.:
 * user id: max. 64 chars
 * datastore: max. 32 chars

The only known problematic API endpoint is the catalog one, used in
the GUI's pxar file browser:
GET /api2/json/admin/datastore/<id>/catalog?..&filepath=<path>

The <path> is the encoded archive path, and can be arbitrary long.

But, this is a flawed design, as even without this new limit one can
easily generate archives which cannot be browsed anymore, as hyper
only accepts requests with max. 64 KiB in the URI.
So rather, we should move that to a GET-as-POST call, which has no
such limitations (and would not need to base32 encode the path).

Note: This change was inspired by adding a request access log, which
profits from such limits as we can then rely on certain atomicity
guarantees when writing requests to the log.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:40:39 +02:00
Thomas Lamprecht
29633e2fe9 server/rest: forward real client IP on proxied request
needs new proxmox dependency to get the RpcEnvironment changes,
adding client_ip getter and setter.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-16 10:36:54 +02:00
Thomas Lamprecht
b947b1e7ee server: rest: refactor code to avoid multiple log_response calls
The 'Ok::<_, Self::Error>(res)' type annotation was from a time where
we could not use async, and had a combinator here which needed
explicity type information. We switched over to async in commit
91e4587343 and, as the type annotation
is already included in the Future type, we can safely drop it.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-15 13:58:47 +02:00
Thomas Lamprecht
1e80fb8e92 code cleanups
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-15 13:58:47 +02:00
Thomas Lamprecht
6c5bdef567 server/REST: check auth: code cleanup, better variable names
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 18:39:45 +02:00
Thomas Lamprecht
ea545b395b server/REST: make handle_request private
it's not used anywhere else, so do not suggest so

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 18:39:00 +02:00
Wolfgang Bumiller
d1993187b6 introduce TaskState trait
Used to not require access to the WorkerTask struct outside
the `server` and `api2` module, so it'll be easier to
separate those backup/server/client parts into separate
crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-12 14:11:57 +02:00
Thomas Lamprecht
adfcfb6788 server: get index: make content-type non mutable
feels more idiomatic

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 13:36:45 +02:00
Thomas Lamprecht
07995a3ca3 server/rest: code cleanup: use async
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-12 13:36:45 +02:00
Fabian Grünbichler
1d5dac1b1d REST: don't print CSRF token
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-08 15:57:22 +02:00