Commit Graph

330 Commits

Author SHA1 Message Date
Dietmar Maurer
e68269fcaf tape: proxmox-tape inventory: call API 2021-01-29 11:21:57 +01:00
Dietmar Maurer
4470eba551 cleanup: factor out common client code to view task log/result 2021-01-29 10:10:04 +01:00
Dietmar Maurer
52f7a73009 display_task_log: make it possible to abort tasks with CTRL-C 2021-01-29 09:06:15 +01:00
Fabian Grünbichler
367c0ff7c6 clippy: allow api functions with many arguments
some of those can be reduced/cleaned up when we have updater support in
the api macro.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:52 +01:00
Fabian Grünbichler
93e3581ce7 derive/impl and use Default for some structs
and revamp HttpClientOptions with two constructors for the common use
cases

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:45 +01:00
Fabian Grünbichler
77486a608e pxar: factor out PxarCreateOptions
containing the CLI parameters that are mostly passed-through from the
client to our pxar archive creation wrapper in pxar::create

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:32 +01:00
Fabian Grünbichler
e43b9175c0 client: factor out UploadOptions
to reduce function signature complexity.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-26 09:54:18 +01:00
Fabian Grünbichler
29077d95db http-client: further clippy cleanups
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:54 +01:00
Fabian Grünbichler
dbd00a57b0 http-client: fix typoed ticket cache condition
which was even copy-pasted once without noticing.

found with clippy.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-01-25 11:41:51 +01:00
Fabian Grünbichler
44de5bcc00 pull: add error context for initial group list call
otherwise the user is confronted with a generic error like "permission
check failed" with no indication that it refers to a request made to the
remote PBS instance..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-18 06:51:05 +01:00
Fabian Grünbichler
e2956c605d pull: rustfmt
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-18 06:50:23 +01:00
Fabian Grünbichler
0bfcea6a11 cleanup: remove unnecessary 'mut' and '.clone()'
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler
89e9134a3f hyper: use new hyper::upgrade
the old Body::on_upgrade method is no more

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler
7c66701366 tokio 1.0: use ReceiverStream from tokio-stream
to wrap a Receiver in a Stream. this will likely move back into tokio
proper once we have a std Stream..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler
d148958b67 proxmox 0.10: use tokio::time::timeout directly
TimeoutFutureExt is no more

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Fabian Grünbichler
0a8d773ad0 tokio 1.0: delay -> sleep
almost the same thing, new name(s), no longer Unpin

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-01-14 16:01:33 +01:00
Stefan Reiter
4799280ccd http_client: add timeouts for critical connects
Use timeout futures for sections that might hang in certain error
conditions. This is mostly intended to be used as a safeguard, not a
first line of defense - i.e. best-effort avoidance of total hangs.

Not every future used for the HttpClient/H2Client is changed, only those
where a quick response is to be expected. For example, the response
reading futures are left alone, so data transfer is never capped with
timeout, only the initial server connect.

It is also used for upgrading to H2 connections, as that can take a long
time on overloaded servers.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-12-22 13:31:10 +01:00
Dominik Csapak
f6e28f4e62 client/pull: log how many groups to pull were found
if no groups were found, the task log was very confusing as it
contained no real information why nothing was synced, e.g.:

 Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
 Sync datastore 'local-datastore' from 'remote/datastore'
 sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
 TASK OK

this patch simply logs how many groups were found and are about to be synced:

 Starting datastore sync job 'remote:datastore:local-datastore:s-79412799-e6ee'
 Sync datastore 'local-datastore' from 'remote/datastore'
 found 0 groups to sync
 sync job 'remote:datastore:local-datastore:s-79412799-e6ee' end
 TASK OK

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-12-02 07:22:50 +01:00
Dietmar Maurer
6eff8dec4f cleanup: remove unnecessary StoreProgress clone() 2020-12-01 06:29:11 +01:00
Fabian Grünbichler
fc8920e35d pull: factor out interpolated progress
and add group/snapshot count info.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:13:11 +01:00
Fabian Grünbichler
7f3b0f67e7 remove BackupGroup::list_groups
BackupInfo::list_backup_groups is identical code-wise, and makes more
sense as entry point for listing groups.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-12-01 06:09:44 +01:00
Fabian Grünbichler
8b7f8d3f3d expose previous backup time in backup env
and use this information to add more information to client backup log
and guide the download manifest decision.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-11-24 08:44:55 +01:00
Dominik Csapak
968a0ab261 fix systemd-encoded upid strings in http client
since we systemd-encode parts of the upid string, and those can contain
characters that are invalid in urls (e.g. '\'), we have to percent encode
those

add a 'percent_encode_component' helper, so that we can maybe change
the AsciiSet for all uses at the same time

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-19 11:01:19 +01:00
Dominik Csapak
3c945d73c2 client/http_client: add put method
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-11-16 16:59:14 +01:00
Wolfgang Bumiller
5d1d0f5d6c use pin-project to remove more unsafe blocks
we already have it in our dependency tree, so use it

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-11-12 09:43:38 +01:00
Dietmar Maurer
30f73fa2e0 fix bug #3060: continue sync if we cannot aquire the group lock 2020-11-10 11:29:36 +01:00
Dietmar Maurer
42ca9e918a sync: improve log format 2020-11-04 09:10:56 +01:00
Dietmar Maurer
0081903f7c fix bug #2870: use updated tickets 2020-11-04 08:20:36 +01:00
Fabian Grünbichler
362739054e api tokens: add authorization method
and properly decode secret (which is a no-op with the current scheme).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-30 13:15:14 +01:00
Fabian Grünbichler
34aa8e13b6 client/remote: allow using ApiToken + secret
in place of user + password.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:14:27 +01:00
Fabian Grünbichler
e6dc35acb8 replace Userid with Authid
in most generic places. this is accompanied by a change in
RpcEnvironment to purposefully break existing call sites.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-29 15:11:39 +01:00
Stefan Reiter
5eb9dd0c8a add tools::http for generic HTTP GET and move HttpsConnector there
...to avoid having the tools:: module depend on api2.

The get_string function is based directly on hyper and thus relatively
simple, not supporting redirects for example.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-21 16:22:08 +02:00
Dietmar Maurer
4e32d1c590 fix for prevoius patch: we want to copy all valid tickets 2020-10-21 08:40:04 +02:00
Dylan Whyte
afef7f3bba fix #3038: check user before renewing ticket
Fixes a bug in which the userid of the ticket cache is updated,
when a user connects, but the ticket itself is not.
This means a newly connected user has a previously connected
user's ticket and thus, cannot do anything, as the client will
attempt to use the invalid ticket.

e.g. if john@pbs connected to the server first, followed by
mike@pbs, the following would be stored in the ticket cache.

{
  "localhost": {
    "mike@pbs": {
      "ticket": "PBS:john@pbs:AAAA",
      "timestamp": 1601039326,
      "token": "BBBB"
    }
  }
}

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-21 08:34:30 +02:00
Dietmar Maurer
97168f920e set reasonable TCP keepalive timeout 2020-10-19 14:01:17 +02:00
Dietmar Maurer
bb14d46796 http_client: set connect timeout to 10 seconds 2020-10-19 09:36:01 +02:00
Wolfgang Bumiller
2081327428 more clippy lints
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-15 12:18:34 +02:00
Wolfgang Bumiller
8db1468952 more clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 13:58:35 +02:00
Wolfgang Bumiller
254ec19412 pxar: remove unused parameter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:32:22 +02:00
Wolfgang Bumiller
97bbd1bf9e pxar: pass full path to callback, let verbose flag list files
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:31:42 +02:00
Wolfgang Bumiller
54aec2fa8b clippy fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-14 11:20:07 +02:00
Stefan Reiter
708fab3082 format: fix typo in function name
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-08 08:32:21 +02:00
Dietmar Maurer
a71bc08ff4 src/tools/parallel_handler.rs: remove lifetime hacks, require 'static
In theory, one can do std::mem::forget, and ignore the drop handler. With
the lifetime hack, this could result in a crash.

So we simply require 'static lifetime now (futures also needs that).
2020-10-01 14:52:48 +02:00
Dominik Csapak
38d4675921 fix ipv6 handling for remotes/sync jobs
* add square brackets to ipv6 adresses in BackupRepository if they not
already have some (we save them without in the remote config)

* in get_pull_parameters, we now create a BackupRepository first and use
  those values (which does the [] mapping), this also has the advantage
  that we have one place less were we hardcode 8007 as port

* in the ui, add square brackets for ipv6 adresses for remotes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 13:40:03 +02:00
Dietmar Maurer
7b8aa893fa src/client/pull.rs: log progress 2020-09-30 13:35:09 +02:00
Dominik Csapak
ba20987ae7 client/remote: add support to specify port number
this adds the ability to add port numbers in the backup repo spec
as well as remotes, so that user that are behind a
NAT/Firewall/Reverse proxy can still use it

also adds some explanation and examples to the docs to make it clearer
for h2 client i left the localhost:8007 part, since it is not
configurable where we bind to

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 10:49:20 +02:00
Dietmar Maurer
636e674ee7 src/client/pull.rs: simplify code 2020-09-26 10:09:51 +02:00
Dietmar Maurer
5441708634 src/client/pull.rs: use new ParallelHandler 2020-09-25 12:58:20 +02:00
Dietmar Maurer
998db63933 src/client/pull.rs: decode, verify and write in a separate threads
To maximize throughput.
2020-09-24 13:12:04 +02:00
Dietmar Maurer
ebbe4958c6 src/client/pull.rs: avoid duplicate downloads using in memory HashSet 2020-09-22 12:34:06 +02:00