Commit Graph

5893 Commits

Author SHA1 Message Date
Thomas Lamprecht bd4c5607ca d/control: commit update
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-03 01:49:50 +02:00
Thomas Lamprecht e1d85f1840 ui: login: cleanups, mostly openID related
similar to what was done in PVE.

 - factor out openid_login_param to widget-toolkit as
   getOpenIDRedirectionAuthorization and use it
 - use camel case to match our JS style guide and our framework (and
   basically the rest of the JS world)
 - minor cleanups like moving variable definition into the single if
   branch their used

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-03 01:46:24 +02:00
Thomas Lamprecht 1ce1a5e5cc docs: installation: drop debian-release specific note
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-03 01:44:53 +02:00
Thomas Lamprecht 6f66a0ca71 docs: faq: update support table
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-03 01:44:09 +02:00
Thomas Lamprecht 62a5b3907b docs: initial update to repositories for bullseye
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-02 19:02:13 +02:00
Thomas Lamprecht 85b6c4ead4 ui: login: fix another bogus gettext usage
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-02 09:31:50 +02:00
Thomas Lamprecht a190979c04 ui: login: fix bogus gettext usage
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-02 08:40:14 +02:00
Dominik Csapak 4a489ae3de ui: dashboard/DataStoreStatistics: fix closing <i> tag
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-01 07:51:00 +02:00
Dominik Csapak 9ac8b73e07 tape/drive: fix logging when requesting media
we try to load the correct media in a loop until we find the correct tape.
when encountering an error or wrong tape, we want to log that (and send
an email if one is set) that requests the correct tape.

while trying to avoid printing the same errors more than once in a row,
we had at least one case (starting with an empty tape in the drive)
which would not print/send any tape request.

reworking that code to use a custom 'TapeRequest' enum, which contains
the state + error message, and a helper that prints and sends an email
when the state changes

this reduces the change check/log to a single variable, instead of 4
(tried, last_media_uuid, last_error, failure_reason)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-06-30 10:25:48 +02:00
Dietmar Maurer 414be8b675 tape: fix LTO locate_file for HP drives
Add test code to the first locate_file command, compute locate_offset.
Subsequent locate_file commands use that offset.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-06-30 09:08:58 +02:00
Dietmar Maurer fda19dcc6f fix CachedUserInfo by using a shared memory version counter 2021-06-30 08:54:30 +02:00
Dietmar Maurer cd975e5787 ui: implement OpenId login 2021-06-30 08:54:30 +02:00
Dietmar Maurer 3b7b1dfb8e api: add openid redirect/login API 2021-06-30 08:54:30 +02:00
Dietmar Maurer d8a47ec649 cleanup user/token is_active() check 2021-06-30 08:54:30 +02:00
Dietmar Maurer 252cd3b781 implement new helper is_active_user_id() 2021-06-30 08:54:30 +02:00
Dietmar Maurer 0decd11efb cli: add CLI to manage openid realms. 2021-06-30 08:54:30 +02:00
Dietmar Maurer b84d2592fb add API to manage openid realms 2021-06-30 08:54:30 +02:00
Dietmar Maurer 0219ba2cc5 check_acl_path: add /access/domains and /access/openid 2021-06-30 08:54:30 +02:00
Dietmar Maurer bbff6c4968 config: new domains.cfg to configure openid realm
Or other realmy types...
2021-06-30 08:54:30 +02:00
Dietmar Maurer bb88c6a29d depend on proxmox-openid-rs 2021-06-30 08:54:30 +02:00
Thomas Lamprecht a02466966d update enterprise repository to bullseye
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-28 19:57:50 +02:00
Thomas Lamprecht b0fc11804e d/changelog: add actual changelog for initial 2.0 build
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-28 19:40:44 +02:00
Thomas Lamprecht d9d81741e3 buildsys: switch to bullseye as upload dist target
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-28 19:10:26 +02:00
Thomas Lamprecht 9678366102 bump version to 2.0.0-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-28 19:07:46 +02:00
Thomas Lamprecht a2c73c78dd buildsys: call dpkg-buildpackage directly in deb-all
else we may double-build

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-28 19:07:15 +02:00
Thomas Lamprecht c6a0e7d98e d/control: bump versioned dependency for ExtJS 7.0
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-06-28 19:05:58 +02:00
Dominik Csapak 85417b2a88 docs: build api-viewer from widget-toolkit-dev
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 14:26:40 +02:00
Dominik Csapak d738669066 docs: add Toolkit.js to lto-barcode
and generate a single js file for it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 14:26:40 +02:00
Dominik Csapak 442d6da8fb docs: add Toolkit.js to prune simulator
from proxmox-widget-toolkit-dev and not as normal dependency,
else we would have to ship widget-toolkit on the wiki

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 14:26:40 +02:00
Dominik Csapak 62f10a01db docs/prune-simulator: remove displayField for Calendar Field
in extjs 7.0, specifying displayField overwrites the displayTpl,
which we want to use here, so remove it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 14:26:40 +02:00
Dominik Csapak 5667b76381 fix #3393 (again): pxar/create: try to read xattrs/fcaps/acls by default
we have a static list of filesystems and their capabilities regarding
file attributes and fs features (e.g. sockets/fifos/etc) which also
includes xattrs,acls and fcaps

if we did not know a filesystem by its magic number (for example cephfs),
we did not even attempt to read xattrs, etc.

this patch adds those flags by default to unknown filesystems, and
removes them when we encounter EOPNOTSUPP (to remove the number
of syscalls)

with this, we should be able to catch xattrs/acls/fcaps on all
(unknown) fs types that support them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-28 14:04:22 +02:00
Stefan Reiter d9b318a444 file-restore/disk: support ZFS subvols with mountpoint=legacy
These require mounting using the regular 'mount' syscall.
Auto-generates an appropriate mount path.

Note that subvols with mountpoint=none cannot be mounted this way, and
would require setting the mountpoint property, which is not possible as
the zpools have to be imported with readonly=on.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-28 13:58:41 +02:00
Stefan Reiter 86ce56f193 file-restore/disk: support ZFS pools
Uses the ZFS utils to detect, import and mount zpools. These are
available as a new Bucket type 'zpool'.

Requires some minor changes to the existing disk and partiton detection
code, so the ZFS-specific part can use the information gathered in the
previous pass to associate drive names with their 'drive-xxxN.img.fidx'
node.

For detecting size, the zpool has to be imported. This is only done with
pools containing 5 or less disks, as anything else might take too long
(and should be seldomly found within VMs).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-28 13:58:41 +02:00
Stefan Reiter 8d72c2c32e file-restore: increase RAM for ZFS and disable ARC
Even through best efforts at keeping it small, including the ZFS tools
in the initramfs seems to have exhausted the small overhead we had left
- give it a bit more RAM to compensate.

Also disable the ZFS ARC, as it's no use in such a memory constrained
environment, and we cache on the QEMU/rust layer anyway.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-28 13:58:41 +02:00
Stefan Reiter c48c38ab8c async_lru_cache: fix handling of errors in fetch
The future needs to be removed from the pending map in any case, even if
it returned an error, else all upcoming calls to access this key will
always return the same error.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-28 13:48:26 +02:00
Dominik Csapak 3d3769830b tape/helpers/snapshot_reader: sort chunks by inode (per index)
sort the chunks we want to backup to tape by inode, to gain some
speed on spinning disks. this is done per index, not globally.

costs a bit memory, but not too much, about 16 bytes per chunk which
would mean ~4MiB for a 1TiB index with 4MiB chunks.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 12:16:14 +02:00
Dominik Csapak 4921a411ad backup/datastore: refactor chunk inode sorting to the datastore
so that we can reuse that information

the removal of the adding to the corrupted list is ok, since
'get_chunks_in_order' returns them at the end of the list
and we do the same if the loading fails later in 'verify_index_chunks'
so we still mark them corrupt
(assuming that the load will fail if the stat does)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 12:14:52 +02:00
Dominik Csapak 81c767efce proxmox-backup-manager: show task log on datastore create
since the output:
Result: "<UPID>"
is not really interesting, show instead the task log while
the datastore is creating, since it is now run in a worker

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 12:09:55 +02:00
Hannes Laimer 60abf03f05 close #3459: manager: add --ignore-verified and --outdated-after parameters
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 11:03:51 +02:00
Hannes Laimer dcbf29e71b api: add ignore-verified and outdated-after to datastore verify endpoint
preparatory change for fixing #3459

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 11:03:51 +02:00
Hannes Laimer 037e6c0ca8 verify-job: move snapshot filter into function
preparatory steps for fixing #3459

Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 11:03:44 +02:00
Fabian Grünbichler c7024b282a d/control: set R-R-R to run binary d/rules targets as root
the build still requires root to make helper binaries setuid

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-06-14 13:02:20 +02:00
Fabian Grünbichler 90ff75f85c update to zstd 0.6
compatible with libzstd from bullseye.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2021-06-14 13:01:43 +02:00
Dietmar Maurer 2165f0d450 api: define and use REALM_ID_SCHEMA 2021-06-10 11:10:00 +02:00
Wolfgang Bumiller 1e7639bfc4 fixup minimum lru capacity
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-06-08 10:13:46 +02:00
Stefan Reiter 4121628d99 tools/lru_cache: make minimum capacity 1
Setting this to 0 is not just useless, but breaks the logic horribly
enough to cause random segfaults - better forbid this, to avoid someone
else having to debug it again ;)

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:55 +02:00
Stefan Reiter da78b90f9c backup: remove AsyncIndexReader
superseded by CachedChunkReader, with less code and more speed

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:46 +02:00
Stefan Reiter 1ef6e8b6a7 replace AsyncIndexReader with SeekableCachedChunkReader
admin/datastore reads linearly only, so no need for cache (capacity of 1
basically means no cache except for the currently active chunk).
mount can do random access too, so cache last 8 chunks for possibly a
mild performance improvement.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:44 +02:00
Stefan Reiter 10351f7075 backup: add AsyncRead/Seek to CachedChunkReader
Implemented as a seperate struct SeekableCachedChunkReader that contains
the original as an Arc, since the read_at future captures the
CachedChunkReader, which would otherwise not work with the lifetimes
required by AsyncRead. This is also the reason we cannot use a shared
read buffer and have to allocate a new one for every read. It also means
that the struct items required for AsyncRead/Seek do not need to be
included in a regular CachedChunkReader.

This is intended as a replacement for AsyncIndexReader, so we have less
code duplication and can utilize the LRU cache there too (even though
actual request concurrency is not supported in these traits).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:40 +02:00
Stefan Reiter 70a152deb7 backup: add CachedChunkReader utilizing AsyncLruCache
Provides a fast arbitrary read implementation with full async and
concurrency support.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2021-06-08 09:42:37 +02:00