Backport of commit 0bd9c87010e25634ea6a91c65c2ff8088372340d
When re-opening a datastore due to the cached entry being stale
(only on verify-new config change). On datastore open the chunk store
was also re-opened, which in turn creates a new ProcessLocker,
loosing any existing shared lock which can cause conflicts between
long running (24h+) backups and GC.
To fix this, reuse the existing ChunkStore, and thus its
ProcessLocker, when creating a up-to-date datastore instance on
lookup, since only the datastore config should be reloaded. This is
fine as the ChunkStore path is not updatable over our API.
Note that this is a precaution backport, the underlying issue this
fixes is relatively unlikely to cause any trouble in the 1.x branch
due to not often re-opening the datastore.
Originally-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
this fixes the leaked memory for the cache, as we had only pointers
in the map/list which were freed, not the underlying chunks
moves the 'clear' implementation out of the trait bounds so that
Drop can reuse it
this is used e.g. for file download from a pxar
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
(cherry picked from commit 98983a9dabacf8e6ea44e44c3b02f4d4e0e2c935)
this was also leaking a refcount before, this is fixed now
See-also: proxmox/proxmox-async:
* d0a3e38006fe ("drop RawWaker usage")
* ff132e93c6fd ("rustfmt")
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
send_command serializes everything so it cannot be used to send a
raw, optimized command. Normally that means we get an error like
> 'unable to parse parameters (expected json object)'
when used that way.
Switch over to send_raw_command which does not re-serializes the
command.
Fixes: 45b8a032 ("refactor send_command")
Originally-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
This lock is held during VM startup, so that multiple calls will not
start VMs twice. But this means that the timeout needs to incorporate
the time it might take a VM to boot, so increase it quite a bit.
This could previously lead to "interrupted system call" errors when
accessing backups with many disks.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
(cherry picked from commit 66501529a2efbd5a426eba56ff8604db157b7814)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
to make the log more useful..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 9a06eb16186f81c3a384b49cd3ee274ee767b40f)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 309e14ebb714e4b799b6b2597fcc21c31e49dc01)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
During startup most of the stuff is happening in milliseconds (or
less), so the timestamp granularity of seconds made it hard to tell
if the previous command required 990ms or 1ms, which is quite the
difference in the restore daemon context.
Using micros seems not to bring too much additional information, a
millisecond is already an ok lower time resolution for logging, so
switch only to millis for now.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit ecd66ecaf6fbf6f929f81a837df25757b6e7a290)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
fixes file restore again.
The new Memcom tracking file lives in `/run/proxmox-backup` and is
always created on REST interaction, as CachedUserInfo uses it to
efficiently track config changes, and such a cache is used in each
REST handle_request.
Further, the Memcom infra expects the base run PBS dir to exists
already, which is an OK assumption to have, but in the file-restore
daemon we have a significantly more minimal environment, and the run
dir was simply not required there, even /run isn't a tmpfs yet.
Fixes fda19dcc6f4 ("fix CachedUserInfo by using a shared memory version counter")
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 33d7292f29bedfd7570bebcef2a5d6943ebe8cbb)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
We send it already to the user via the response body, but the
log_response does not has, nor wants to have FWIW, access to the
async body stream, so pass it through the ErrorMessageExtension
mechanism like we do else where.
Note that this is not only useful for PBS API proxy/daemon but also
the REST server of the file-restore daemon running inside the restore
VM, and it really is *very* helpful to debug things there..
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit f4d371d2d2c5d79352e9681b5d942627e26c6fd4)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 2d48533378b33a679e24dfe3b9551cc01e6fc748)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
the dns plugin config allow for a specified amount of time to wait for
the TXT record to be set and propagated through DNS.
This patch adds a sleep for this amount of time.
The log message was taken from the perl implementation in proxmox-acme
for consistency.
Tested with the powerdns plugin in my test setup.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
(cherry picked from commit 3f845414126f6d40ca725fae155a41efd7b2161a)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we need the setting in acme::plugin.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
(cherry picked from commit 4d8bd036680a4fe2d73bef24f9af85cc7097437a)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
extract_challenge is used by both dns-01 and http-01 challenges.
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
(cherry picked from commit f9bd5e1691821da32919a97ce211b972fe623726)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 13a24457449ef8b23c4b841bcdbcc9024f0c0e60)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
So callers get more stable results. Most noticeable, the disk list in
the web UI doesn't jump around upon reloading, and while sorting could
be done directly there, like this other callers get the benefit too.
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
(cherry picked from commit bbff317aa7b183ce58a52df35d58997f2203c8d7)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
in preparation to also get the file system type from lsblk.
Co-developed-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
(cherry picked from commit 364299740f9b471b60c5b53be059f2d57f9461ef)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
a match expresses the fallback slightly nicer and needs no mut,
which is always nice to avoid.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit 6b5013edb3d7a59c176258d682cf264b479de69a)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
According to crypt(3):
"crypt places its result in a static storage area, which will be
overwritten by subsequent calls to crypt. It is not safe to call crypt
from multiple threads simultaneously."
This means that multiple login calls as a PBS-realm user can collide and
produce intermittent authentication failures. A visible case is for
file-restore, where VMs with many disks lead to just as many auth-calls
at the same time, as the GUI tries to expand each tree element on load.
Instead, use the thread-safe variant 'crypt_r', which places the result
into a pre-allocated buffer of type 'crypt_data'. The C struct is laid
out according to 'lib/crypt.h.in' and the man page mentioned above.
Use the opportunity and make both arguments to the rust 'crypt' function
take a &[u8].
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
(cherry picked from commit c4c4b5a3ef0e60e806a28c6180433bc8b5839c61)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
(cherry picked from commit 9f5b57a348add161938d8c204c3e1c6562feaf06)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
(cherry picked from commit aceae32baadd6e4077eb053706432246c287b83d)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
else we may double-build
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit a2c73c78dd3875c0cf2340bddbb1874c56e70d90)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
and not on window close
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
(cherry picked from commit 82cae19d192c97a16b7aff45f6c97be878174ca4)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
While the PVE one "bails" too, it has an eval around those and moves
the error to the message property, so lets do so too to ensure a user
can force an update on a too old subscription
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
(cherry picked from commit b81818b6ad2a2c4935c4a6679d57292069c5d48e)
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
we try to load the correct media in a loop until we find the correct tape.
when encountering an error or wrong tape, we want to log that (and send
an email if one is set) that requests the correct tape.
while trying to avoid printing the same errors more than once in a row,
we had at least one case (starting with an empty tape in the drive)
which would not print/send any tape request.
reworking that code to use a custom 'TapeRequest' enum, which contains
the state + error message, and a helper that prints and sends an email
when the state changes
this reduces the change check/log to a single variable, instead of 4
(tried, last_media_uuid, last_error, failure_reason)
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Add test code to the first locate_file command, compute locate_offset.
Subsequent locate_file commands use that offset.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
we have a static list of filesystems and their capabilities regarding
file attributes and fs features (e.g. sockets/fifos/etc) which also
includes xattrs,acls and fcaps
if we did not know a filesystem by its magic number (for example cephfs),
we did not even attempt to read xattrs, etc.
this patch adds those flags by default to unknown filesystems, and
removes them when we encounter EOPNOTSUPP (to remove the number
of syscalls)
with this, we should be able to catch xattrs/acls/fcaps on all
(unknown) fs types that support them
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
since the output:
Result: "<UPID>"
is not really interesting, show instead the task log while
the datastore is creating, since it is now run in a worker
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Setting this to 0 is not just useless, but breaks the logic horribly
enough to cause random segfaults - better forbid this, to avoid someone
else having to debug it again ;)
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
admin/datastore reads linearly only, so no need for cache (capacity of 1
basically means no cache except for the currently active chunk).
mount can do random access too, so cache last 8 chunks for possibly a
mild performance improvement.
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
