Saves the currently active read/write operation counts in a file. The
file is updated whenever a reference returned by lookup_datastore is
dropped and whenever a reference is returned by lookup_datastore. The
files are locked before every access, there is one file per datastore.
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
by not bubbling up most errors, and continuing on. this avoids that we
stop cleaning up because e.g. one directory was missing.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
added a parameter to the cli for importing tape key via a json-parameter or
via reading a exported paperkey-file or json-file.
For this i also added a backupkey parameter to the api, but here it only
accepts json.
The cli interprets the parameter first as json-string, then json-file
and last as paperkey-file.
functionality:
proxmox-tape key paperkey [fingerprint of existing key] > paperkey.backup
proxmox-tape key restore --backupkey paperkey.backup # key from line above
proxmox-tape key restore --backupkey paperkey.json # only the json
proxmox-tape key restore --backupkey '{"kdf": {"Scrypt": ...' # json as string
for importing the key as paperkey-file it is irrelevant, if the paperkey got exported as html
or txt.
Signed-off-by: Markus Frank <m.frank@proxmox.com>
when we use http to make the api call, we have to parse the parameters
before, else we might send the string "true" instead of the boolean true
and the api rejects it with a 'Parameter verification error'.
We already have all api call schemas here, so parsing is possible.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
avoid that the acme renewal is skipped due to bailing out earlier
from a subscription or apt update error.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Adds the '--force' flag to the proxmox-tape command allowing users
with root privileges to overwrite the passphrase of a given key.
Signed-off-by: Stefan Sterz <s.sterz@proxmox.com>
So that we can make 'log::debug' messages actually appear in the
syslog.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
For the API the parameter --hint is not optional. This patch fixes
the man page and cli command doesn't send an API call, if the
parameter does not exist.
Signed-off-by: Markus Frank <m.frank@proxmox.com>
instead of getting the 'peer_addr()' from the socket.
The advantage is that we must get this and thus can drop the mapping
from result -> option, and can drop the testing for None and a test case
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
glibc's malloc has a misguided heuristic to detect transient allocations that
will just result in allocation sizes below 32 MiB never using mmap.
That it turn means that those relatively big allocations are on the heap where
cleanup and returning memory to the OS is harder to do and easier to be blocked
by long living, small allocations at the top (end) of the heap.
Observing the malloc size distribution in a file-level backup run:
@size:
[0] 14 | |
[1] 25214 |@@@@@ |
[2, 4) 9090 |@ |
[4, 8) 12987 |@@ |
[8, 16) 93453 |@@@@@@@@@@@@@@@@@@@@ |
[16, 32) 30255 |@@@@@@ |
[32, 64) 237445 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[64, 128) 32692 |@@@@@@@ |
[128, 256) 22296 |@@@@ |
[256, 512) 16177 |@@@ |
[512, 1K) 5139 |@ |
[1K, 2K) 3352 | |
[2K, 4K) 214 | |
[4K, 8K) 1568 | |
[8K, 16K) 95 | |
[16K, 32K) 3457 | |
[32K, 64K) 3175 | |
[64K, 128K) 161 | |
[128K, 256K) 453 | |
[256K, 512K) 93 | |
[512K, 1M) 74 | |
[1M, 2M) 774 | |
[2M, 4M) 319 | |
[4M, 8M) 700 | |
[8M, 16M) 93 | |
[16M, 32M) 18 | |
We see that all allocations will be on the heap, and that while most
allocations are small, the relatively few big ones will still make up most of
the RSS and if blocked from being released back to the OS result in much higher
peak and average usage for the program than actually required.
Avoiding the "dynamic" mmap-threshold increasement algorithm and fixing it at
the original default of 128 KiB reduces RSS size by factor 10-20 when running
backups. As with memory mappings other mappings or the heap can never block
freeing the memory fully back to the OS.
But, the drawback of using mmap is more wasted space for unaligned or small
allocation sizes, and the fact that the kernel allegedly zeros out the data
before giving it to user space. The former doesn't really matter for us when
using it only for allocations bigger than 128 KiB, and the latter is a
trade-off, using 10 to 20 times less memory brings its own performance
improvement possibilities for the whole system after all ;-)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
[ Thomas: added to comment & commit message + extra-empty-line fixes ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Recently, ZFS removed the pool global io stats from
/proc/spl/kstat/zfs/POOL/io with no replacement.
To gather stats about the datastores, access now the objset specific
entries there. To be able to make that efficient, cache a map of
dataset <-> obset ids, so that we do not have to parse all files each time.
We update the cache each time we try to get the info for a dataset
where we do not have a mapping.
We cannot update it on datastore add/remove since that happens in the
proxmox-backup daemon, while we need the info here in proxmox-backup-proxy.
Sadly with this we lose the io wait metric, but it seems that this is no
longer tracked in zfs at all, so nothing we can do for that.
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
It is not necessary, so avoid it. The client can now be used
with multiple threads (without using a Mutex).
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
the 'utc' flag is now contained in the event itself and not given
as a flag to 'compute_next_event' anymore
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
proxmox-serde instead)
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
we even use that for basically all the related schema names, "groups"
allone is just rather not so telling, i.e., "groups" what?
While due to the additive nature of `group-filter` is not the best
possible name for passing multiple arguments on the CLI (the web-ui
can present this more UX-friendly anyway) due to possible confusion
about if the filter act like AND vs OR it can be documented and even
if a user is confused they still are safe on more being synced than
less. Also, the original param name wasn't really _that_ better in
that regards
Dietmar also suggested to use singular for the CLI option, while
there can be more they're passed over repeating the option, each with
a single filter.
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
and add a completion handler to complete the backup groups
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
Hannes Laimer
Thomas Lamprecht
Dominik Csapak
Markus Frank
Wolfgang Bumiller
Dietmar Maurer
Stefan Sterz
Fabian Grünbichler
Matthias Heiserer