Commit Graph

4490 Commits

Author SHA1 Message Date
Dominik Csapak b683fd589c backup/datastore: save garbage collection status to disk
and load it again when opening it

this way we can persist the status of the last garbage collect across
daemon reloads and reboots

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Dominik Csapak a2285525be backup/datastore: count still bad chunks for the status
we want to show the user that there are still bad chunks after a garbage
collection

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht f23497b088 apt auth: add newline to the end
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht b57b3c9bfc hack: workaround unused code warning until proxmox-api-macro bump
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 17:41:30 +01:00
Thomas Lamprecht d3444c0891 ui: allow one to delete the description
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht d28e688666 ui: code cleanup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht 72c0e102ff tools: get_hardware_address: better error handling
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht 7b22fb257f implement subscription handling and api
mostly modelled after PVE

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht 2e201e7da6 tools: http: add simple general post method
This is intended for when the server needs to do requests on
arbitrary, non PBS, external HTTP resources.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Thomas Lamprecht ee89416319 api: disks: cleanup use statement
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-27 13:13:00 +01:00
Dylan Whyte 2357744bd0 introduction: fix title formatting
fix title formatting to remove warning from build

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-27 12:33:17 +01:00
Thomas Lamprecht 52fe9e8ece get_hardware_address: must be uppercased
we're a bit strict here what we accept, rather than changing that
lets do it like PVE/PMG and uppercase.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-26 20:18:28 +01:00
Thomas Lamprecht eed1bae554 api: add world accessible ping dummy endpoint
This is indented to be used for the PVE storage library, replacing
the missuse of the much more expensive status API call.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-24 19:12:14 +02:00
Fabian Grünbichler 6eb41487ce apt: improve error messages
Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-10-22 17:13:26 +02:00
Stefan Reiter 9e61c01ce4 apt: add /changelog API call similar to PVE
For proxmox packages it works the same way as PVE, by retrieving the
changelog URL and issuing a HTTP GET to it, forwarding the output to the
client. As this is only supposed to be a workaround removed in the
future, a simple block_on is used to avoid async.

For debian packages we can simply call 'apt-get changelog' and forward
it's output.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter 91c9b42da3 fix #2934: list to-be-installed packages in updates
As always, libapt is mocking us with complexity, but we can get the
approximate result we want by retrieving dependencies of all
to-be-updated packages and then seeing if they are missing.

If they are, we assume they will be installed.

For this, query_detailed_info is extended to allow reading details for
non-installed packages, and this is also exposed in
list_installed_apt_packages via 'all_versions_for'. This is necessary so
we can retrieve changelogs for such packages.

Note that we cannot retrieve all that information all the time, as
querying details for packages that aren't installed takes a rather long
time.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter 52d2ae48f0 apt: refactor package detail reading into function
No functional change intended.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:25:08 +02:00
Stefan Reiter 1872050564 apt: use 'apt-get changelog --print-uris' in get_changelog_url
Avoids custom hardcoded logic, but can only be used for debian packages
as of now. Adds a FIXME to switch over to use --print-uris only once our
package repos support that changelog format.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:26 +02:00
Stefan Reiter efeb92efee apt: allow filter to select different package version
To get package details for a specific version instead of only the
candidate.

Also cleanup filter function with extra struct instead of unnamed &str
parameters.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-22 16:24:15 +02:00
Dietmar Maurer 4ebda996e5 upid: use systemd escape to decode/encode the worker_id
This way we can store values containing "/" and ":".
2020-10-22 12:24:58 +02:00
Stefan Reiter 5eb9dd0c8a add tools::http for generic HTTP GET and move HttpsConnector there
...to avoid having the tools:: module depend on api2.

The get_string function is based directly on hyper and thus relatively
simple, not supporting redirects for example.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-21 16:22:08 +02:00
Thomas Lamprecht 12bcbf0734 ui: verify config: eslint fix
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-21 15:53:54 +02:00
Dominik Csapak dc2876f6bb tools/zip: fix doc tests
the doc code was not compiling and blocking cargo test

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-21 14:20:16 +02:00
Hannes Laimer bdc208af48 postinst: correct invalid old datastore configs
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 2ef1b6290f api proxy: remove old verification scheduling
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer df0bdf6be7 ui: add task descriptions for the different types of verification(job, snapshot, group, ds)
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 8b47a23002 ui: add verification job edit window
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 29615fe838 ui: add verification job view
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 133042b5d8 set a different worker_type based on what is going to be verified(snapshot, group, ds)
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 73df9c515b proxy: add scheduling for verification jobs
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 8d1beca7e8 api2: add verification admin endpoint and do_verification_job function
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 9b2bad7af0 api2: add verification job config endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Hannes Laimer 78efafc2d0 rename VERIFY_SCHEDULE_SCHEMA to VERIFICATION_SCHEDULE_SCHEMA
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-10-21 12:51:35 +02:00
Dietmar Maurer 2d3d91b1db add test for escape_unit 2020-10-21 11:31:24 +02:00
Dietmar Maurer 030c5c6d8a systemd::escape_unit - allow '.' and '_' 2020-10-21 11:31:24 +02:00
Wolfgang Bumiller 53a561a222 pass params by ref to recurse_files
gets rid of the return value and moving around of the zip
and decoder data
avoids cloning the path prefix on every recursion

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:50:25 +02:00
Wolfgang Bumiller e832860a3c whitespace fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:45:44 +02:00
Dominik Csapak 804f61432d api2/admin/datastore/pxar_file_download: download directory as zip
by using the new ZipEncoder and recursively add files to it
the zip only contains directories, normal files and hardlinks (by simply
copying the content), no symlinks, etc.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:24 +02:00
Dominik Csapak 943479f5f6 tools: add AsyncChannelWriter
similar to StdChannelWriter, but implements AsyncWrite and sends
to a tokio::sync::mpsc::Sender

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:22 +02:00
Dominik Csapak fdce52aa99 tools: add zip module
This modules contains the 'ZipEncoder' struct, which wraps an async writer,
to create a ZIP archive on the fly

To create a ZIP file, have a target that implements AsyncWrite,
give it to ZipEncoder::new, add entries via 'add_entry' and
at the end, call 'finish'

for now, this does not implement compression (uses ZIPs STORE mode), and
does not support empty directories or hardlinks (or any other special
files)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:18 +02:00
Dietmar Maurer 4e32d1c590 fix for prevoius patch: we want to copy all valid tickets 2020-10-21 08:40:04 +02:00
Dylan Whyte afef7f3bba fix #3038: check user before renewing ticket
Fixes a bug in which the userid of the ticket cache is updated,
when a user connects, but the ticket itself is not.
This means a newly connected user has a previously connected
user's ticket and thus, cannot do anything, as the client will
attempt to use the invalid ticket.

e.g. if john@pbs connected to the server first, followed by
mike@pbs, the following would be stored in the ticket cache.

{
  "localhost": {
    "mike@pbs": {
      "ticket": "PBS:john@pbs:AAAA",
      "timestamp": 1601039326,
      "token": "BBBB"
    }
  }
}

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-21 08:34:30 +02:00
Dietmar Maurer b428af9781 backup: avoid Transport endpoint is not connected error
We simply supress the error message if the finish flag is set.
2020-10-20 14:20:04 +02:00
Dietmar Maurer c8774067ee paperkey: use svg as image format to provide better scalability 2020-10-20 12:04:51 +02:00
Dietmar Maurer 23440482d4 proxmox-backup-client: use HumanByte to render snapshot size 2020-10-20 11:43:48 +02:00
Thomas Lamprecht 6f757b8458 logrotate: drop useless comment
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:11:36 +02:00
Thomas Lamprecht 95ade8fdb5 log rotate: move basic rotation logic into module for reuse
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht 9e870b5f39 log rotate: do NOT compress first rotation
The first rotation is normally the one still opened by one or more
processes for writing, so it must NOT be replaced, removed, ..., as
this then makes the remaining logging, until those processes are
noticed that they should reopen the logfile due to rotation, goes
into nirvana, which is far from ideal for a log.

Only rotating (renaming) is OK for this active file, as this does not
invalidates the file and keeps open FDs intact.

So start compressing with the second rotation, which should be clear
to use, as all writers must have been told to reopen the log during
the last rotation, reopen is a fast operation and normally triggered
at least day ago (at least if one did not dropped the state file
manually), so we are fine to archive that one for real.
If we plan to allow faster rotation the whole rotation+reopen should
be locked, so that we can guarantee that all writers switched over,
but this is unlikely to be needed.

Again, this is was logrotate sanely does by default since forever.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht 7827e3b93e log rotate: factor out compression in private function
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00
Thomas Lamprecht e6ca9c3235 log rotate: do NOT overwrite file with possible writers
this is not the job of logrotate, and the real 20+ years battle
tested log rotate binary does not do so either as it's actually
pretty dangerous.

If we "replace" the file we break any logger which already opened a
new one here, e.g., a dameon starting up, and thus that writer would
log to nirvana.

It's the job of a logger to create a file if not existing, it makes
no sense to do it here.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-20 11:09:17 +02:00