tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
3,797 Commits 2 Branches 54 Tags 14 MiB
0a274ab0a0
Commit Graph

2931 Commits

Author SHA1 Message Date
Thomas Lamprecht
7e42ccdaf2 fixed index: chunk_from_offset: avoid slow modulo operation 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 17:46:07 +02:00
Stefan Reiter
e713ee5c56 remove BufferedFixedReader interface 
replaced by AsyncIndexReader

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-07-22 17:28:49 +02:00
Stefan Reiter
ec5f9d3525 implement AsyncSeek for AsyncIndexReader 
Requires updating the AsyncRead implementation to cope with byte-wise
seeks to intra-chunk positions.

Uses chunk_from_offset to get locations within chunks, but tries to
avoid it for sequential read to not reduce performance from before.

AsyncSeek needs to use the temporary seek_to_pos to avoid changing the
position in case an invalid seek is given and it needs to error in
poll_complete.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-07-22 17:28:49 +02:00
Stefan Reiter
d0463b67ca add and implement chunk_from_offset for IndexFile 
Necessary for byte-wise seeking through chunks in an index.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-07-22 17:28:49 +02:00
Thomas Lamprecht
2ff4c2cd5f datastore/chunker: fix comment typos 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 16:12:49 +02:00
Thomas Lamprecht
c3b090ac8a backup: list images: handle walkdir error, catch "lost+found" 
We support using an ext4 mountpoint directly as datastore and even do
so ourself when creating one through the disk manage code.

Such ext4 ountpoints have a lost+found directory which only root can
traverse into. As the GC list images is done as backup:backup user
walkdir gets an error.

We cannot ignore just all permission errors, as they could lead to
missing some backup indexes and thus possibly sweeping more chunks
than desired. While *normally* that should not happen through our
stack, we had already user report that they do rsyncs to move a
datastore from old to new server and got the permission wrong.

So for now be still very strict, only allow a "lost+found" directory
as immediate child of the datastore base directory, nothing else.

If deemed safe, this can always be made less strict. Possibly by
filtering the known backup-types on the highest level first.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 16:01:55 +02:00
Thomas Lamprecht
c47e294ea7 datastore: fix typo 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-22 15:04:14 +02:00
Fabian Grünbichler
25455bd06d fix #2871: close FDs when scanning backup group 
otherwise we leak those descriptors and run into EMFILE when a backup
group contains many snapshots.

fcntl::openat and Dir::openat are not the same ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 2020-07-22 09:19:29 +02:00
Fabian Grünbichler
c1c4a18f48 fix #2865: detect and skip vanished snapshots 
also when they have been removed/forgotten since we retrieved the
snapshot list for the currently syncing backup group.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 2020-07-22 09:19:29 +02:00
Fabian Grünbichler
91f5594c08 api: translate ENOTFOUND to 404 for downloads 
and percolate the HttpError back up on the client side

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 2020-07-22 09:19:29 +02:00
Fabian Grünbichler
86f6f74114 fix #2860: skip in-progress snapshots when syncing 
they don't have a final manifest yet and are not done, so they can't be
synced either.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 2020-07-22 09:19:29 +02:00
Stefan Reiter
113c9b5981 move subscription API path to /nodes 
This aligns it with PVE and allows the widget toolkit's update window
"refresh" to work without modifications once POST /apt/update is
implemented.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-07-21 19:33:52 +02:00
Thomas Lamprecht
956295cefe parse_calendar_event: support the weekly special expression 
While we do not yet support the date specs for CalendarEvent the left
out "weekly" special expression[0] dies not requires that support.
It is specified to be equivalent with `Mon *-*-* 00:00:00` [0] and
this can be implemented with the weekday and time support we already
have.

[0]: https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-21 13:24:51 +02:00
Dominik Csapak
a26c27c8e6 api2/status: fix estimation bug 
when a datastore has enough data to calculate the estimated full date,
but always has exactly the same usage, the factor b of the regression
is '0'

return 0 for that case so that the gui can show 'never' instead of
'not enough data'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-21 13:02:08 +02:00
Aaron Lauterer
b96b11cdb7 chunk_store: Fix typo in bail message 
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
 2020-07-21 12:51:41 +02:00
Aaron Lauterer
faa8e6948a backup: Fix typos and grammar 
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
 2020-07-21 12:51:41 +02:00
Fabian Grünbichler
1b1110581a manifest: revert canonicalization to old behaviour 
JSON keys MUST be quoted. this is a one-time break in signature
validation for backups created with the broken canonicalization code.
QEMU backups are not affected, as libproxmox-backup-qemu never linked
the broken versions.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
 2020-07-20 11:37:53 +02:00
Thomas Lamprecht
449e4a66fe tools/xattr: a char from C is not universally a rust i8 
Make it actually do the correct cast by using `libc::c_char`.

Fixes issues when building on other platforms, e.g., the aarch64
client only build on Arch Linux ARM I tested in my free time.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-19 19:46:27 +02:00
Thomas Lamprecht
217c22c754 server: add path value to NOT_FOUND http error 
Especially helpful for requests not coming from browsers (where the
URL is normally easy to find out).

Makes it easier to detect if one triggered a request with an old
client, or so..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-16 12:46:51 +02:00
Dominik Csapak
ac5e9e770b catalog_shell: add exit command 
it is nice to have a command to exit from the shell instead of
only allowing ctrl+d or ctrl+c

the api method is just for documentation/help purposes and does nothing
by itself, the real logic is directly in the read loop

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-15 12:19:57 +02:00
Wolfgang Bumiller
b25deec0be pxar: .pxarexclude: absolute paths and byte based paths 
Change the .pxarexclude parser to byte based parsing with
`.split(b'\n')` instead of `.lines()`, to not panic on
non-utf8 paths.

Specially deal with absolute paths by prefixing them with
the current directory.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-15 11:55:48 +02:00
Wolfgang Bumiller
cdf1da2872 tools: add strip_ascii_whitespace for byte slices 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-15 11:55:48 +02:00
Thomas Lamprecht
3cfc56f5c2 cached user info: check_privs: print privilege path in error message 
As else this is really user unfriendly, and it not printing it has no
advantage. If one doesn't wants to leak resource existence they just
need to *always* check permissions before checking if the requested
resource exists, if that's not done one can leak information also
without getting the path returned (as the system will either print
"resource doesn't exists" or "no permissions" respectively)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-15 08:55:58 +02:00
Wolfgang Bumiller
757d0ccc76 warning fixup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-14 10:37:14 +02:00
Wolfgang Bumiller
032cd1b862 pxar: restore file attributes, improve errors 
and use the correct integer types for these operations

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-14 10:25:45 +02:00
Dietmar Maurer
f58233a73a src/backup/data_blob_reader.rs: avoid unwrap() - return error instead 2020-07-10 11:28:19 +02:00
Dominik Csapak
bfcef26a99 api2/node/status: add fingerprint 
and rename get_usage to get_status (since its not usage only anymore)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-10 11:08:53 +02:00
Dominik Csapak
ec01eeadc6 refactor CertInfo to tools 
we want to reuse some of the functionality elsewhere

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-10 11:08:53 +02:00
Wolfgang Bumiller
660a34892d update proxmox crate to 0.2.0 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-10 11:08:27 +02:00
Dietmar Maurer
d86034afec src/bin/proxmox_backup_client/catalog.rs: fix keyfile handling 2020-07-10 10:36:45 +02:00
Dietmar Maurer
62593aba1e src/backup/manifest.rs: fix signature (exclude 'signature' property) 2020-07-10 10:36:45 +02:00
Wolfgang Bumiller
0eaef8eb84 client: show key path when creating/changing default key 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-10 09:58:24 +02:00
Thomas Lamprecht
e39974afbf client: add simple version command 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-10 09:34:07 +02:00
Dietmar Maurer
dde18bbb85 proxmox-backup-client benchmark: improve output format 2020-07-10 09:13:52 +02:00
Dietmar Maurer
a40e1b0e8b src/server/rest.rs: avoid compiler warning 2020-07-10 09:13:52 +02:00
Stoiko Ivanov
c687da9e8e datastore: chown base dir on creation 
When creating a new datastore the basedir is only owned by the backup
user if it did not exist beforehand (create_path chowns only if it
creates the directory), and returns false if it did not create the
directory).

This improves the experience when adding a new datastore on a fresh
disk or existing directory (not owned by backup) - backups/pulls can
be run instead of terminating with EPERM.

Tested on my local testinstall with a new disk, and a existing directory:

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
 2020-07-09 18:20:16 +02:00
Dietmar Maurer
4327a8462a proxmox-backup-client benchamrk: add more speed tests 2020-07-09 17:07:22 +02:00
Dietmar Maurer
e193544b8e src/server/rest.rs: disable debug logs 2020-07-09 16:18:14 +02:00
Dietmar Maurer
323b2f3dd6 proxmox-backup-client benchmark: add --verbose flag 2020-07-09 16:16:39 +02:00
Dominik Csapak
fae11693f0 fix cross process task listing 
it does not make sense to check if the worker is running if we already
have an endtime and state

our 'worker_is_active_local' heuristic returns true for non
process-local tasks, so we got 'running' for all tasks that were not
started by 'our' pid and were still running

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-09 15:30:52 +02:00
Wolfgang Bumiller
bf7e2a4648 simpler lost+found pattern 
the **/ is not required and currently also mistakenly
doesn't match /lost+found which is probably buggy on the
pathpatterns crate side and needs fixing there

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-09 14:06:42 +02:00
Dominik Csapak
3ec99affc8 get_disks: don't fail on zfs_devices 
zfs does not have to be installed, so simply log an error and
continue, users still get an error when clicking directly on
ZFS

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-09 13:47:31 +02:00
Dominik Csapak
a9649ddc44 disks/zpool_status: add test for pool with special character 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-09 13:37:31 +02:00
Dominik Csapak
4f9096a211 disks/zpool_list: allow some more characters for pool list 
not exhaustive of what zfs allows (space is missing), but this
can be done easily without problems

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-09 13:37:31 +02:00
Dominik Csapak
c3a4b5e2e1 zpool_list: add tests for special pool names 
those names are allowed for zpools

these will fail for now, but it will be fixed in the next commit

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-09 13:37:31 +02:00
Dominik Csapak
7957fabff2 api: add ZPOOL_NAME_SCHEMA and regex 
poolnames can containe spaces and some other special characters

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-07-09 13:37:31 +02:00
Wolfgang Bumiller
20a4e4e252 minor optimization to 'to_canonical_json' 
* don't clone hash keys, just use references
* we don't need a String, stick to Vec<u8> and use
  serde_json::to_writer to avoid a temporary strings
  altogether

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-09 13:32:11 +02:00
Dietmar Maurer
4459ffe30e src/backup/manifest.rs: add default toömake it compatible with older backus 2020-07-09 13:25:38 +02:00
Dietmar Maurer
dfa517ad6c src/backup/manifest.rs: rename into_string -> to_string 
And do not consume self.
 2020-07-09 11:28:05 +02:00
Dietmar Maurer
3dacedce71 src/backup/manifest.rs: use serde_json::from_value() to deserialize data 
Also modified from_data compute signature ditectly from json.
 2020-07-09 09:50:28 +02:00
Dietmar Maurer
b53f637914 src/backup/manifest.rs: cleanup signature generation 2020-07-09 09:20:49 +02:00
Dietmar Maurer
2107a5aebc src/backup/manifest.rs: include signature inside the manifest 
This is more flexible, because we can choose what fileds we want to sign.
 2020-07-08 16:23:26 +02:00
Dietmar Maurer
3638341aa4 src/backup/file_formats.rs: remove signed chunks 
We can include signature in the manifest instead (patch will follow).
 2020-07-08 16:23:26 +02:00
Wolfgang Bumiller
0351f23ba4 client: introduce --keyfd parameter 
This is a more convenient way to pass along the key when
creating encrypted backups of unprivileged containers in PVE
where the unprivileged user namespace cannot access
`/etc/pve/priv`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 13:56:38 +02:00
Dietmar Maurer
c1ff544eff src/backup/crypt_config.rs - compute_digest: make it more secure 2020-07-08 12:53:04 +02:00
Wolfgang Bumiller
05389a0109 more xdg cleanup and encryption parameter improvements 
Have a single common function to get the BaseDirectories
instance and a wrapper for `find()` and `place()` which
wrap the error with some context.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 10:57:28 +02:00
Wolfgang Bumiller
b65390ebc9 client: xdg usage: place() vs find() 
place() is used when creating a file, as it will create
intermediate directories, only use it when actually placing
a new file.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 10:57:28 +02:00
Dietmar Maurer
3bad3e6e52 src/client/backup_writer.rs - upload_stream: add crypt_mode 2020-07-08 10:43:28 +02:00
Wolfgang Bumiller
24be37e3f6 client: fix schema to include --crypt-mode parameter 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 10:09:15 +02:00
Wolfgang Bumiller
1008a69a13 pxar: less confusing logic 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 09:58:29 +02:00
Wolfgang Bumiller
521a0acb2e DataStore::load_manifest: also return CryptMode 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 09:19:53 +02:00
Wolfgang Bumiller
3b66040de6 add DataBlob::crypt_mode 
and move use statements up

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 09:19:53 +02:00
Wolfgang Bumiller
af3a0ae7b1 remove CryptMode::sign_only special method 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-08 09:19:53 +02:00
Dietmar Maurer
4e36f78438 src/backup/manifest.rs: support old encrypted property 
Just to avoid confusion.
 2020-07-08 08:52:27 +02:00
Wolfgang Bumiller
f28d9088ed introduce a CryptMode enum 
This also replaces the recently introduced --encryption
parameter on the client with a --crypt-mode parameter.

This can be "none", "encrypt" or "sign-only".

Note that this introduces various changes in the API types
which previously did not take the above distinction into
account properly:

Both `BackupContent` and the manifest's `FileInfo`:
    lose `encryption: Option<bool>`
    gain `crypt_mode: Option<CryptMode>`

Within the backup manifest itself, the "crypt-mode" property
will always be set.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-07 15:24:19 +02:00
Dietmar Maurer
f91d58e157 src/tools/runtime.rs: implement get_runtime_with_builder 2020-07-07 10:11:04 +02:00
Dietmar Maurer
7443a6e092 src/client/remote_chunk_reader.rs: implement clone for RemoteChunkReader 2020-07-07 07:34:58 +02:00
Wolfgang Bumiller
96ee857752 client: add --encryption boolen parameter 
This can be used to explicitly disable encryption even if a
default key file exists in ~/.config.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-06 14:36:04 +02:00
Wolfgang Bumiller
887018bb79 client: use default encryption key if it is available 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-06 14:36:04 +02:00
Wolfgang Bumiller
9696f5193b client: move key management into separate module 
and use api macro for methods and Kdf type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-06 14:36:04 +02:00
Wolfgang Bumiller
e13c4f66bb minor style & whitespace fixups 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-07-06 10:55:25 +02:00
Thomas Lamprecht
08cb2038bd api: disks: indentation fixup 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-07-03 17:59:30 +02:00
Dietmar Maurer
9de69cdb1a src/bin/proxmox_backup_client/catalog.rs: split out catalog code 2020-07-03 16:45:47 +02:00
Dietmar Maurer
e9764238df make ReadChunk not require mutable self. 
That way we can reduce lock contentions because we lock for much shorter
times.
 2020-07-03 07:37:29 +02:00
Dietmar Maurer
cc7995ac40 src/bin/proxmox_backup_client/task.rs: split out task command 2020-07-02 18:04:29 +02:00
Dietmar Maurer
43abba4b4f src/bin/proxmox_backup_client/mount.rs: split out mount code 2020-07-02 17:49:59 +02:00
Dietmar Maurer
caea8d611f proxmox-backup-client: add benchmark command 
This is just a start, We need to add more useful things here...
 2020-07-02 14:01:57 +02:00
Wolfgang Bumiller
7d0754a6d2 pxar: fixup 'vanished-file' logic a bit 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-30 14:41:42 +02:00
Wolfgang Bumiller
5afa0755ea pxar: fix missing newlines in warnings 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-30 14:37:20 +02:00
Dietmar Maurer
2162e2c15d src/api2/admin/datastore.rs: avoid slash in UPID strings 2020-06-30 13:11:22 +02:00
Dietmar Maurer
4059285649 fix typo 2020-06-29 12:59:25 +02:00
Dietmar Maurer
2e079b8bf2 partially revert commit 1f82f9b7b5 
do it backwards compatible. Also, code was wrong because FixedIndexWriter
still computed old style csums...
 2020-06-29 12:44:45 +02:00
Wolfgang Bumiller
a8e2940ff3 pxar: deal with files changing size during archiving 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-26 11:49:51 +02:00
Dominik Csapak
4eb4e94918 fix test output 
field separator for pools is always a tab when using -H

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-26 10:31:11 +02:00
Dietmar Maurer
817bcda848 src/backup/verify.rs: do not stop on server shutdown 
This is a read-only task, so there is no need to stop.
 2020-06-26 09:45:59 +02:00
Dominik Csapak
f6de2c7359 WorkerTask: add warnings and count them 
so that we have one level more between errors and OK

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-26 09:42:11 +02:00
Dominik Csapak
402c8861d8 fix typo 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-26 09:12:29 +02:00
Dominik Csapak
3f683799a8 improve 'debug' parameter 
instead of checking on '1' or 'true', check that it is there and not
'0' and 'false'. this allows using simply

https://foo:8007/?debug

instead of

https://foo:8007/?debug=1

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-26 09:12:14 +02:00
Dietmar Maurer
1f82f9b7b5 src/backup/index.rs: add compute_csum 
And use it for fixed and dynamic index. Please note that this
changes checksums for fixed indexes, so restore older backups
will fails now (not backward compatible).
 2020-06-26 09:00:34 +02:00
Dietmar Maurer
fe3e65c3ea src/api2/backup.rs: call register_chunk in previous download api 2020-06-26 08:22:46 +02:00
Dietmar Maurer
fdaab0df4e src/backup/index.rs: add chunk_info method 2020-06-26 08:14:45 +02:00
Dietmar Maurer
b957aa81bd update backup api for incremental backup 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-06-26 07:17:08 +02:00
Dietmar Maurer
8ea00f6e49 allow to abort verify jobs 
And improve job description rendering on gui.
 2020-06-25 12:56:36 +02:00
Stefan Reiter
e22f4882e7 extract create_download_response API helper 
and put it into a new "api2::helpers" module.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-06-25 11:57:37 +02:00
Dietmar Maurer
355c055e81 src/bin/proxmox-backup-manager.rs: implement verify 2020-06-24 13:35:21 +02:00
Dietmar Maurer
c2009e5309 src/api2/admin/datastore.rs: add verify api 2020-06-24 13:35:21 +02:00
Dietmar Maurer
23f74c190e src/backup/backup_info.rs: impl Display for BackupGroup 2020-06-24 13:35:21 +02:00
Wolfgang Bumiller
a6f8728339 update to pxar 0.1.9, update ReadAt implementations 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-24 11:57:12 +02:00
Stefan Reiter
facd9801cf add incremental backup support 
To support incremental backups (where not all chunks are sent to the
server), a new parameter "reuse-csum" is introduced on the
"create_fixed_index" API call. When set and equal to last backups'
checksum, the backup writer clones the data from the last index of this
archive file, and only updates chunks it actually receives.

In incremental mode some checks usually done on closing an index cannot
be made, since they would be inaccurate.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-06-24 10:01:25 +02:00
Dietmar Maurer
21302088de remove debug println 2020-06-24 09:15:13 +02:00
Stefan Reiter
8268c9d161 fix overflow panic during upload 
if *only* data chunks are registered (high chance during incremental
backup), then chunk_count might be one lower then upload_stat.count
because of the zero chunk being unconditionally uploaded but not used.
Thus when subtracting the two, an overflow would occur.

In general, don't let the client make the server panic, instead just set
duplicates to 0.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
 2020-06-24 09:07:22 +02:00
Dominik Csapak
b91b7d9ffd api2/node/disks/zfs: check if default zfs mount path exists 
and if it does bail, because otherwise we would get an
error on mounting and have a zpool that is not imported
and disks that are used

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:22:39 +02:00
Dominik Csapak
d33d8f4e6a api2/admin/datastore: add pxar-file-download api call 
streams a file from a pxar file of an unencrypted backup

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:21:15 +02:00
Dominik Csapak
5b1cfa01f1 api2/admin/datastore: add 'catalog' api call 
returns the dir listing of the given filepath of the backup snapshot
the filepath has to be base64 encoded or 'root'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:16:12 +02:00
Dominik Csapak
05d18b907a add From<&DirEntryAttribute to CatalogEntryType and make it pub(crate) 
we want to get a string representation of the DirEntryAttribute
like 'f' for file, etc. and since we have such a mapping already
in the CatalogEntryType, use that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:08:50 +02:00
Dominik Csapak
e44fe0c9f5 derive Clone for the LocalChunkReader 
this will be necessary for accessing local pxar behind didx files

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:07:28 +02:00
Dominik Csapak
4cf0ced950 add LocalDynamicReadAt 
mostly copied from BufferedDynamicReadAt from proxmox-backup-client
but the reader is wrapped in an Arc in addition to the Mutex

we will use this for local access to a pxar behind a didx file

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-24 07:05:31 +02:00
Dietmar Maurer
60f9a6ea8f src/backup/datastore.rs: add new helpers to load blobs and verify chunks 2020-06-24 06:58:14 +02:00
Dietmar Maurer
1090fd4424 src/backup/data_blob.rs: cleanup - improve code reuse 2020-06-24 06:56:48 +02:00
Dietmar Maurer
92c3fd2e22 src/backup/chunk_store.rs: allow to read name() 
This is helpful for logging ...
 2020-06-24 06:54:21 +02:00
Dietmar Maurer
d6d3b353be cleanup: implement FromStr for BackupGroup 2020-06-23 08:16:56 +02:00
Dietmar Maurer
a67f7d0a07 cleanup: implement FromStr for BackupDir 2020-06-23 08:09:52 +02:00
Dietmar Maurer
c8137518fe src/bin/proxmox_backup_manager/disk.rs: add renderer for wearout 
So that we display the same value as the GUI.
 2020-06-23 07:44:09 +02:00
Dominik Csapak
cbef49bf4f remove absolute paths when executing binaries 
we set the paths manually, so this is ok

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-23 07:09:06 +02:00
Dominik Csapak
0b99e5aebc remove debug prints 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-23 06:33:58 +02:00
Dominik Csapak
f386f512d0 add AsyncReaderStream 
and replace AsyncIndexReader's stream implementation with that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-23 06:33:31 +02:00
Dietmar Maurer
3ddb14889a src/tools/daemon.rs: reopen STDOUT/STDERR journald streams to get correct PID in logs 2020-06-22 13:06:53 +02:00
Wolfgang Bumiller
c08fac4d69 tools::daemon: sync with child after MainPid message 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-22 10:58:04 +02:00
Wolfgang Bumiller
c40440092d tools: add socketpair helper 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-22 10:49:56 +02:00
Wolfgang Bumiller
dc2ef2b54f tools::daemon: fetch exe name in the beginning 
We get the path to our executable via a readlink() on
"/proc/self/exe", which appends a " (deleted)" during
package reloads.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-22 10:31:54 +02:00
Dietmar Maurer
f28cfb322a avoid compiler warnings 2020-06-20 07:24:02 +02:00
Dietmar Maurer
3bbe291c51 zpool_status.rs - indented_list_to_tree: do not set name property 
This is no necessary. We only touch/set 'children' and 'leaf' properties.
 2020-06-20 07:19:25 +02:00
Dietmar Maurer
42d19fdf69 src/api2/node/disks/zfs.rs: always set pool name 2020-06-20 07:15:32 +02:00
Dietmar Maurer
215968e033 src/tools/disks/zpool_status.rs: add 'leaf' attribute to root node, rename 'prev' into 'parent' 2020-06-20 06:49:06 +02:00
Dietmar Maurer
eddd1a1b9c src/tools/disks/zpool_status.rs: move use clause top of file 2020-06-20 06:17:22 +02:00
Dietmar Maurer
d2ce211899 fixup for previous commit 2020-06-20 06:15:26 +02:00
Dietmar Maurer
1cb46c6f65 src/tools/disks/zpool_status.rs - cleanup: use struct StackItem instead of tuple 2020-06-19 18:58:57 +02:00
Dietmar Maurer
5d88c3a1c8 src/tools/disks/zpool_status.rs: remove unnecessary checks 
Thos things can never happen, so simply use unwrap().
 2020-06-19 18:27:39 +02:00
Dietmar Maurer
07fb504943 src/tools/disks/zpool_status.rs: simplify code by using serde_json::to_value 2020-06-19 17:51:13 +02:00
Dietmar Maurer
f675c5e978 src/tools/disks/zpool_status.rs - add all attributes to the tree 2020-06-19 16:55:28 +02:00
Wolfgang Bumiller
4e37d9ce67 add general indented_list_to_tree implementation 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-19 14:37:40 +02:00
Wolfgang Bumiller
e303077132 lru_cache: restrict and annotate Send impl 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-19 09:37:34 +02:00
Dominik Csapak
6ef9bb59eb api2/admin/datastore: add download-decoded endpoint 
similar to 'download', but streams the decoded file
when it is not encrypted

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 08:39:15 +02:00
Dominik Csapak
eeaa2c212b impl Sync for DataBlobReader 
this is safe for the reason explained in the comment

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 08:37:47 +02:00
Dominik Csapak
4a3adc3de8 add AsyncIndexReader 
implements AsyncRead as well as Stream for an IndexFile and a store
that implements AsyncReadChunk

we can use this to asyncread or stream the content of a FixedIndex or
DynamicIndex

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 08:32:33 +02:00
Dominik Csapak
abdb976340 add Display trait to BackupDir 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 08:28:35 +02:00
Dominik Csapak
3b62116ce6 implement AsyncReadChunk for LocalChunkReader 
same as the sync ReadChunk but uses tokio::fs::read instead
of file_get_contents

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 07:54:23 +02:00
Dominik Csapak
1c090810f5 api2/admin/datastore/snapshos: show encrypted and size info per file 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 07:39:56 +02:00
Dominik Csapak
e181d2f6da add encrypted info to Manifest 
we want to save if a file of a backup is encrypted, so that we can
* show that info on the gui
* can later decide if we need to decrypt the backup

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 07:35:39 +02:00
Dominik Csapak
16021f6ab7 use the existing async method for read_raw_chunk 
does the same, except the manual drop, but thats handled there by
letting the value go out of scope

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 07:23:17 +02:00
Dominik Csapak
ba694720fc api2/admin/datastore: log stream error during file download 
the client cannot get an error during an chunked http transfer, so at
least log it server side

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 06:58:55 +02:00
Dominik Csapak
bde8e243cf remove unsafe copy code 
copy_nonoverlapping is basically a memcpy which can also be done
via copy_from_slice which is not unsafe
(copy_from_slice uses copy_nonoverlapping internally)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-19 06:56:15 +02:00
Dietmar Maurer
3352ee5656 parse_zpool_status_field: handle tabs without copying input 2020-06-18 19:40:01 +02:00
Dietmar Maurer
b29cbc414d parse_zpool_status_vdev: consider tabs as 8 spaces 2020-06-18 18:38:56 +02:00
Dietmar Maurer
026dc1d11f src/api2/node/disks/zfs.rs: add zpool_details api 2020-06-18 15:04:46 +02:00
Dietmar Maurer
9438aca6c9 src/tools/disks/zpool_status.rs: improve parser 2020-06-18 14:55:22 +02:00
Dietmar Maurer
547f0c97e4 src/tools/nom.rs: new helper parse_complete_line() for single line parsers 
Like parse_complete(), but generates simpler error messages.
 2020-06-18 12:57:55 +02:00
Dietmar Maurer
177a2de992 src/tools/nom.rs: move nom helpers into separate file 2020-06-18 12:41:13 +02:00
Dietmar Maurer
0686b1f4db src/tools/disks/zpool_list.rs: split code into separate file 2020-06-18 10:31:07 +02:00
Dietmar Maurer
0727e56a06 src/tools/disks/zpool_status.rs: parse zpool status output 2020-06-18 10:23:15 +02:00
Dietmar Maurer
2fd3d57490 src/tools/disks/zfs.rs: rename ZFSPoolStatus into ZFSPoolInfo, fix error message 2020-06-17 09:08:26 +02:00
Dietmar Maurer
3f851d1321 src/api2/node/disks/directory.rs: add early check if disk is unused 2020-06-17 08:31:11 +02:00
Dietmar Maurer
1aef491e24 src/bin/proxmox_backup_manager/disk.rs: add cli to create mounted disks 2020-06-17 08:07:54 +02:00
Dietmar Maurer
d0eccae37d avoid compiler warning 2020-06-17 08:07:42 +02:00
Dietmar Maurer
a34154d900 src/tools/disks/zfs.rs: cleanup parse_pool_header 2020-06-17 07:47:11 +02:00
Dietmar Maurer
c2cc32b4dd src/tools/disks/zfs.rs: add more parser tests 2020-06-17 07:38:19 +02:00
Dietmar Maurer
46405fa35d src/tools/disks/zfs.rs: add comment 2020-06-17 07:14:26 +02:00
Dietmar Maurer
66af7f51bc src/tools/disks/zfs.rs: make zfs list parser private 2020-06-17 07:00:54 +02:00
Dietmar Maurer
c72ccd4e33 src/tools/disks/zfs.rs: add regression tests for parse_zfs_list 2020-06-16 18:14:35 +02:00
Dietmar Maurer
902b2cc278 src/tools/disks/zfs.rs: simplify code 2020-06-16 17:51:17 +02:00
Wolfgang Bumiller
8ecd7c9c21 move api dump binaries back to src/bin for package building 
they're required for docs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 14:48:01 +02:00
Dietmar Maurer
fb5a066500 src/api2/node/disks.rs: expose directory api 2020-06-16 13:36:32 +02:00
Wolfgang Bumiller
d19c96d507 move test binaries to examples/ 
These aren't installed and are only used for manual testing,
so there's no reason to force them to be built all the time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 13:32:24 +02:00
Dietmar Maurer
929a13b357 src/api2/node/disks/zfs.rs: add zpool api 2020-06-16 13:25:53 +02:00
Dietmar Maurer
36c65ee0b0 src/tools/disks/zfs.rs: cleanup (rename usage properties) 
And allow to parse zpool list output without -v flag.
 2020-06-16 13:25:53 +02:00
Dietmar Maurer
3378fd9fe5 src/tools/disks/zfs.rs: parse more infos (dedup, fragmentation, health) 2020-06-16 13:25:53 +02:00
Dietmar Maurer
58c51cf3d9 avoid compiler warnings 2020-06-16 13:25:53 +02:00
Dietmar Maurer
5509b199fb use new run_command helper 2020-06-16 13:25:53 +02:00
Wolfgang Bumiller
bb59df9134 catalog: don't panic on invalid file mtimes 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 11:25:54 +02:00
Wolfgang Bumiller
2564b0834f fix file timestamps in catalog 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 11:25:31 +02:00
Wolfgang Bumiller
9321bbd1f5 pxar: fix missing subdirectories in catalogs 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 11:04:56 +02:00
Wolfgang Bumiller
4264e52220 reuse some extractor code in catalog shell 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 10:54:54 +02:00
Wolfgang Bumiller
6988b29bdc use O_EXCL when creating files during extraction 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 10:33:27 +02:00
Wolfgang Bumiller
98c54240e6 pxar: make extractor state more reusable 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 10:32:13 +02:00
Wolfgang Bumiller
d30c192589 AsyncReadChunk: require Send 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-16 09:50:29 +02:00
Dietmar Maurer
ac7513e368 src/tools.rs: add setup_safe_path_env() 2020-06-15 10:38:30 +02:00
Dietmar Maurer
fbbcd85839 src/api2/node/disks/directory.rs: implement add-datastore feature 2020-06-15 10:01:50 +02:00
Wolfgang Bumiller
7a6b549270 dynamic index: make it hard to mess up endianess 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-15 09:19:35 +02:00
Wolfgang Bumiller
0196b9bf5b remove unnecessary .into 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 14:57:58 +02:00
Dominik Csapak
20b3094bcb api2/status: add task list endpoint 
for now mostly copy/paste from nodes/nodename/tasks
(without the parameters)
but we should replace the 'read_task_list' with a method
that gives us the tasks since some timestamp

so that we can get a longer list of tasks than for the node
(we could of course embed this then in the nodes/node/task api call and
remove this again as long as the api is not stable)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 14:50:17 +02:00
Dominik Csapak
df528ee6fa implement From<TaskListInfo> for TaskListItem 
and use it where its convenient

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 14:50:12 +02:00
Wolfgang Bumiller
57e50fb906 use new Mmap helper for dynamic index 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 13:57:56 +02:00
Wolfgang Bumiller
3d571d5509 some internal combinator-influenced api cleanup 
The download methods used to take the destination by value
and return them again, since this was required when using
combinators before we had `async fn`.
But this is just an ugly left-over now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 11:46:42 +02:00
Wolfgang Bumiller
8e6e18b77c client: make dump_image async, use async chunk reader 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 11:40:18 +02:00
Wolfgang Bumiller
4d16badf6f add an AsyncReadChunk trait 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 11:38:21 +02:00
Wolfgang Bumiller
a609cf210e more cleanup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 11:01:04 +02:00
Wolfgang Bumiller
1498659b4e cleanup 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 10:59:34 +02:00
Wolfgang Bumiller
4482f3fe11 pxar, acl: cleanup acl helper usage 
use NixPath for Acl::set_file to avoid memduping the c
string

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 10:52:18 +02:00
Dominik Csapak
5d85847f91 client: only start catalog upload if we have one 
else we start a dynamic writer and never close it, leading to a backup error

this fixes an issue with backing up vm templates
(and possibly vms without disks)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 10:38:25 +02:00
Dominik Csapak
476b4acadc BackupEnvironment: do not set finished flag prematurely 
we check if all dynamic_writers are closed and if the backup contains
any valid files, we can only mark the backup finished after those
checks, else the backup task gets marked as OK, even though it
is not finished and no cleanups run

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 10:37:52 +02:00
Wolfgang Bumiller
cf1bd08131 pxar: fcaps in fuse 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-12 10:37:48 +02:00
Dominik Csapak
ec8f042459 api2/status: use new rrd::extract_cached_data 
and drop the now unused extract_lists function

this also fixes a bug, where we did not add the datastore to the list at
all when there was no rrd data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 13:31:16 +02:00
Dominik Csapak
431cc7b185 rrd: move creation of serde value into api 
there is now a 'extract_cached_data' which just returns
the data of the specified field, and an api function that converts
a list of fields to the correct serde value

this way we do not have to create a serde value in rrd/cache.rs
(makes for a better interface)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 13:31:14 +02:00
Dominik Csapak
e693818afc refactor time functions to tools 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 13:31:10 +02:00
Wolfgang Bumiller
3d68536fc2 pxar: support .pxareclude files, error report updates 
Report vanished files (instead of erroring out on them),
also only warn about files inaccessible due to permissions
instead of bailing out.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 12:22:18 +02:00
Wolfgang Bumiller
26e78a2efb downgrade some FIXMEs to TODOs 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 11:09:23 +02:00
Wolfgang Bumiller
5444fa940b turn pxar::flags into bitflags, pxar::Flags 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 11:05:53 +02:00
Dietmar Maurer
d4f2397d4c add api to format disks and create datastores 2020-06-10 11:03:36 +02:00
Wolfgang Bumiller
fab2413741 catalog: remove unused SenderWriter 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 10:42:42 +02:00
Dietmar Maurer
669c137fec src/tools/systemd.rs: implement daemon_reload, start_unit, stop_unit and enable_unit 2020-06-10 08:56:04 +02:00
Wolfgang Bumiller
fc6047fcb1 pxar: don't skip list+found by default 
This used to be default-off and was accidentally set to
on-by-default with the pxar crate update.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 08:53:10 +02:00
Wolfgang Bumiller
3014088684 pxar: sort .pxareclude-cli file and fix its mode 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-10 08:49:14 +02:00
Dietmar Maurer
144006fade src/tools.rs: add new run_command helper 2020-06-10 07:16:47 +02:00
Dietmar Maurer
b9cf6ee797 src/tools/systemd/types.rs: add Mount config 2020-06-09 18:47:10 +02:00
Wolfgang Bumiller
cdde66d277 statistics: covariance(): avoid allocation 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-09 13:57:27 +02:00
Wolfgang Bumiller
239e49f927 pxar: create .pxarexclude-cli file 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-09 13:17:59 +02:00
Dominik Csapak
bda48e04da api2: add status/datastore-usages api call 
returns a list of the datastores and their usages, a list of usages of
the past month (for the gui) and an estimation of when its full
(using the linear regression)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-09 12:20:25 +02:00
Dominik Csapak
ba97479848 add statistics module 
provides some basic statistics functions (sum, mean, etc.)
and a function to return the parameters of the linear regression of
two variables

implemented using num_traits to be more flexible for the types

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-09 12:19:51 +02:00
Dominik Csapak
6cad8ce4ce rrd: add 'extract_lists' 
this is an interface to simply get the Vec<Option<f64>> out of rrd
without going through serde values

we return a list of timestamps and a HashMap with the lists we could find
(otherwise it is not in the map)

if no lists could be extracted, the time list is also empty

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-09 12:19:06 +02:00
Dominik Csapak
33070956af let disk_usage return StorageStatus and use it for datastores/nodes 
disk_usage returned the same values as defined in StorageStatus,
so simply use that

with that we can replace the logic of the datastore status with that
function and also use it for root disk usage of the nodes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-09 12:05:39 +02:00
Dietmar Maurer
da84cc52f4 src/tools/systemd.rs: implement escape_unit and unescape_unit 2020-06-09 11:52:06 +02:00
Wolfgang Bumiller
2179359f40 move src/pxar.rs -> src/pxar/mod.rs 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-09 10:49:59 +02:00
Dietmar Maurer
9bb161c881 src/tools/disks.rs: add create_single_linux_partition and create_file_system 2020-06-08 17:43:01 +02:00
Dietmar Maurer
297e600730 cleanup comment 2020-06-08 17:43:01 +02:00
Dietmar Maurer
ed7b3a7de2 src/tools/disks.rs: add get_fs_uuid helper 2020-06-08 17:43:01 +02:00
Dietmar Maurer
0f358204bd src/tools/disks.rs: add helper to list partitions 2020-06-08 17:43:01 +02:00
Dietmar Maurer
ca6124d5fa src/tools/disks.rs: make helpers pub 
So that I can use them with my test code.
 2020-06-08 17:43:01 +02:00
Wolfgang Bumiller
7eacdc765b pxar: split assert_relative_path 
the check for a single component is only required in the dir
stack atm

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-08 15:02:52 +02:00
Wolfgang Bumiller
c443f58b09 switch to external pxar and fuse crates 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-08 13:56:58 +02:00
Wolfgang Bumiller
1e3d9b103d xattr: make xattr_name_fcaps public 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-08 13:56:58 +02:00
Wolfgang Bumiller
386990ba09 tools: add file_get_non_comment_lines 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-08 13:56:58 +02:00
Dietmar Maurer
bc853b028f src/tools/disks.rs: cleanup, remove unused DiskUse bitflag type 2020-06-08 09:43:07 +02:00
Dietmar Maurer
d406de299b src/tools/disks.rs: use dev_t to index zfs/lvm device sets 2020-06-08 09:01:34 +02:00
Dietmar Maurer
dfb31de8f0 proxmox_backup_manager disk list: display gpt column 2020-06-08 07:35:44 +02:00
Dietmar Maurer
7c3aa258f8 src/tools/disks/zfs.rs: allow empty zpool list output 2020-06-08 07:23:04 +02:00
Dietmar Maurer
044055062c src/tools/disks.rs: new helper to reread partition table 2020-06-08 07:22:06 +02:00
Dietmar Maurer
2b388026f8 src/api2/node/disks.rs: correctly use disk_by_name insteadf of disk_by_node 2020-06-08 07:20:59 +02:00
Dietmar Maurer
707974fdb3 src/api2/node/disks.rs: implement initgpt API 2020-06-07 10:30:34 +02:00
Dietmar Maurer
9069debcd8 src/api2/types.rs: define BLOCKDEVICE_NAME_SCHEMA 2020-06-07 07:20:25 +02:00
Dietmar Maurer
fa2bdc1309 src/config/acl.rs: add /system/disks to valid acl paths 2020-06-06 15:48:15 +02:00
Dietmar Maurer
8e40aa63c1 src/bin/proxmox-backup-manager.rs: add disk subcommand 2020-06-06 15:40:28 +02:00
Dietmar Maurer
d2522b2db6 src/tools/disks.rs: fix disk size, add completion helper 2020-06-06 15:39:25 +02:00
Dietmar Maurer
ce8e3de401 move disks api to /node/<node>/disks 2020-06-06 14:43:36 +02:00
Dietmar Maurer
7fa2779559 src/api2/disks.rs: implement smart api 2020-06-06 12:23:11 +02:00
Dietmar Maurer
042afd6e52 src/tools/disks.rs: new helper disk_by_name() 2020-06-06 12:22:38 +02:00
Dietmar Maurer
ff30caeaf8 src/api2/disks.rs - list-disks: add usage-type filter 2020-06-06 11:48:58 +02:00
Dietmar Maurer
553cd12ba6 src/api2/disks.rs: start disks api 2020-06-06 11:38:47 +02:00
Dietmar Maurer
de1e1a9d95 src/tools/disks.rs: use api macro so that we can use those types with the api 2020-06-06 11:37:24 +02:00
Dietmar Maurer
91960d6162 src/tools/disks.rs - get_disks: query smart status 2020-06-06 09:18:20 +02:00
Dietmar Maurer
4c24a48eb3 src/tools/disks/smart.rs: use model.to_string_lossy() to simplify code 2020-06-06 09:05:22 +02:00
Dietmar Maurer
484e761dab src/tools/disks/smart.rs: try to get correct wearout for ATA devices 2020-06-06 09:01:15 +02:00
Dietmar Maurer
059b7a252e src/tools/disks/smart.rs - get_smart_data: use &Disk instead of &str 
So that we can query other device infos easily (model, vendor, ..)
 2020-06-06 08:24:58 +02:00
Dietmar Maurer
eb80aac288 src/tools/disks/smart.rs: parse output from smartctl 2020-06-05 18:30:06 +02:00
Dietmar Maurer
c26aad405f src/tools/disks.rs: implement get_disks (similar to the one in PVE::Diskmanage) 
But no ceph support for now. Also no support for old cciss block devices.
 2020-06-05 10:33:53 +02:00
Dietmar Maurer
f03a0e509e src/tools/disks.rs; use correct subdir to check holders 2020-06-05 10:33:53 +02:00
Dietmar Maurer
4c1e8855cc src/tools/disks.rs: fix disk type detection, remove newline from vendor string 2020-06-05 08:09:52 +02:00
Thomas Lamprecht
a720894ff0 rrd: fix off-by-one in save interval calculation 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-06-04 10:30:47 +02:00
Dominik Csapak
a95a3fb893 fix csum calculation of not 'chunk_size' aligned images 
the last chunk does not have to be as big as the chunk_size,
just use the already available 'chunk_end' function which does the
correct thing

this fixes restoration of images whose sizes are not a multiple of
'chunk_size' as well

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-04 10:18:30 +02:00
Dietmar Maurer
620911b426 src/tools/disks/lvm.rs: implement get_lvm_devices() 2020-06-04 09:12:19 +02:00
Dietmar Maurer
5c264c8d80 src/tools/disks.rs: add/use get_partition_type_info 2020-06-04 07:48:22 +02:00
Dominik Csapak
8d78589969 improve display of 'next run' for sync jobs 
if the last sync job is too far in the past (or there was none at all
for now) we run it at the next iteration, so we want to show that

we now calculate the next_run by using either the real last endtime
as time or 0

then in the frontend, we check if the next_run is < now and show 'pending'
(we do it this way also for replication on pve)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-04 07:03:54 +02:00
Dominik Csapak
eed8a5ad79 tools/systemd/time: fix compute_next_event for weekdays 
two things were wrong here:
* the range (x..y) does not include y, so the range
  (day_num+1..6) goes from (day_num+1) to 5 (but sunday is 6)

* WeekDays.bits() does not return the 'day_num' of that day, but
  the bit value (e.g. 64 for SUNDAY) but was treated as the index of
  the day of the week
  to fix this, we drop the map to WeekDays and use the 'indices'
  directly

this patch makes the test work again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-04 07:02:33 +02:00
Dominik Csapak
538b9c1c27 systemd/time: add tests for all weekdays 
this fails for now

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-06-04 07:02:23 +02:00
Thomas Lamprecht
55919bf141 verify_file: add missing closing parenthesis in error message 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-06-03 19:10:01 +02:00
Dietmar Maurer
456ad0c478 src/tools/disks/zfs.rs: add parser for zpool list output 2020-06-03 12:16:08 +02:00
Dietmar Maurer
c48aa39f3b src/bin/proxmox-backup-client.rs: implement quite flag 2020-06-03 10:11:37 +02:00
Thomas Lamprecht
2d32fe2c04 client restore: don't add server file ending if already specified 
If one executes a client command like
 # proxmox-backup-client files <snapshot> --repository ...
the files shown have already the '.fidx' or '.blob' file ending, so
if a user would just copy paste that one the client would always add
.blob, and the server would not find that file.

So avoid adding file endings if it is already a known OK one.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-06-03 07:03:55 +02:00
Thomas Lamprecht
dc155e9bd7 client restore: factor out archive/type parsing 
will be extended in a next patch.

Also drop a dead else branch, can never get hit as we always add
.blob as fallback

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-06-03 07:03:12 +02:00
Dietmar Maurer
4e14781aec fix typo 2020-06-03 06:59:43 +02:00
Wolfgang Bumiller
a595f0fee0 client: improve connection/new fingerprint query 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-06-02 10:40:31 +02:00
Thomas Lamprecht
add5861e8d typo fixes all over the place 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-05-30 16:39:08 +02:00
Dietmar Maurer
1610c45a86 src/client/pull.rs: also download client.log.blob 2020-05-30 14:51:33 +02:00
Dietmar Maurer
b2387eaa45 avoid compiler warnings 2020-05-30 14:05:33 +02:00
Dietmar Maurer
96d65fbcd0 cleanup: define/use const for predefined blob file names. 2020-05-30 14:04:15 +02:00
Dietmar Maurer
7cc3473a4e src/client/backup_specification.rs: split code into extra file 2020-05-30 10:54:38 +02:00
Dietmar Maurer
4856a21836 src/client/pull.rs: more verbose logging 2020-05-30 08:12:43 +02:00
Dominik Csapak
04b0ca8b59 add owner to group and snapshot listings 
while touching it, make columns and tbar in DataStoreContent.js
declarative members and remove the (now) unnecessary initComponent

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-30 07:24:12 +02:00
Dominik Csapak
d43f86f3f3 api2: add admin/sync endpoint 
this returns the list of syncjobs with status, as opposed to
config/sync (which is just the config)

also adds an api call where users can run the job manually under
/admin/sync/$ID/run

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-29 11:31:32 +02:00
Dominik Csapak
997d7e19fc config/sync: add SyncJobStatus Struct/Schema 
contains the config + status

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-29 11:29:39 +02:00
Dominik Csapak
c67b1fa72f syncjob: change worker type for sync jobs 
'sync' is used for manually pulling a remote datastore
changing it for a scheduled sync to 'syncjob' so that we can
differentiate between both types of syncs

this also adds a seperate task description for it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-29 11:28:04 +02:00
Dominik Csapak
268687ddf0 api2/pull: refactor priv checking and creating pull parameters 
we want to reuse those in the api call for manually running a sync job

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-29 11:27:43 +02:00
Dominik Csapak
426c1e353b api2/config/sync: fix id parameter 
'name' is not the correct parameter for get/post

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-29 11:24:54 +02:00
Dominik Csapak
2888b27f4c create SYNC_SCHEDULE_SCHEMA to adapt description for sync jobs 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-29 11:24:25 +02:00
Dietmar Maurer
934f5bb8ac src/bin/proxmox-backup-proxy.rs: cleanup, move code to src/tools/disks.rs 
And simplify find_mounted_device by using stat.st_dev
 2020-05-29 11:13:36 +02:00
Dominik Csapak
9857472211 fix removing of remotes 
we have to save the remote config after removing the section

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-29 10:48:26 +02:00
Dietmar Maurer
013fa7bbcb rrd: reduce io by saving data only once a minute 2020-05-29 09:16:13 +02:00
Dietmar Maurer
a8d7033cb2 src/bin/proxmox-backup-proxy.rs: add test if last prune job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer
04ad7bc436 src/bin/proxmox-backup-proxy.rs: test if last sync job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer
77ebbefc1a src/server/worker_task.rs: make worker_is_active_local pub 2020-05-29 08:06:48 +02:00
Dietmar Maurer
750252ba2f src/tools/systemd/time.rs: add test for "daily" schedule 2020-05-29 07:52:09 +02:00
Dietmar Maurer
dc58194ebe src/bin/proxmox-backup-proxy.rs: use correct id to lookup sync jobs 2020-05-29 07:50:59 +02:00
Thomas Lamprecht
090decbe76 BACKUP_REPO_URL_REGEX: move to api2::types and allow all valid data store names 
The repo URL consists of
* optional userid
* optional host
* datastore name

All three have defined regex or format, but none of that is used, so
for example not all valid datastore names are accepted.

Move definition of the regex over to api2::types where we can access
all required regexes easily.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-05-29 06:29:23 +02:00
Thomas Lamprecht
c32186595e api2::types: factor out USER_ID regex 
allows for better reuse in a next patch

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-05-29 06:27:38 +02:00
Dietmar Maurer
c94e1f655e rrd stats: improve io delay stats 2020-05-28 19:12:13 +02:00
Dietmar Maurer
91e5bb49f5 src/bin/proxmox-backup-proxy.rs: simplify code 
and gather all stats for the root disk
 2020-05-28 12:30:54 +02:00
Dietmar Maurer
547e3c2f6c src/tools/disks/zfs.rs: use wtime + rtime (wait + run time) 2020-05-28 11:45:34 +02:00
Dietmar Maurer
25c550bc28 src/bin/proxmox-backup-proxy.rs: gather zpool io stats 2020-05-28 10:09:13 +02:00
Dietmar Maurer
0146133b4b src/tools/disks/zfs.rs: helper to read zfs pool io stats 2020-05-28 10:07:52 +02:00
Dietmar Maurer
3eeba68785 depend on proxmox 0.1.38, use new fs helper functions 2020-05-28 10:06:44 +02:00
Dominik Csapak
f5056656b2 use the sync id for the scheduled sync worker task 
this way, multiple sync jobs with the same local store, can get scheduled

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-28 06:26:03 +02:00
Dominik Csapak
8c87743642 fix 'remove_vanished' cli arg again 
since the target side wants this to be a boolean and
serde interprets a None Value as 'null' we have to only
add this when it is really set via cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-28 06:25:30 +02:00
Dominik Csapak
05d755b282 fix inserting of worker tasks 
when starting a new task, we do two things to keep track of tasks
(in that order):
* updating the 'active' file with a list of tasks with
  'update_active_workers'
* updating the WORKER_TASK_LIST

the second also updates the status of running tasks in the file by
checking if it is still running by checking the WORKER_TASK_LIST

since those two things are not locked, it can happend that
we update the file, and before updating the WORKER_TASK_LIST,
another thread calls update_active_workers and tries to
get the status from the task log, which won't have any data yet
so the status is 'unknown'

(we do not update that status ever, likely for performance reasons,
so we have to fix this here)

by switching the order of the two operations, we make sure that only
tasks reach the 'active' file which are inserted in the WORKER_TASK_LIST

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-28 06:24:42 +02:00
Dietmar Maurer
143b654550 src/tools.rs - command_output: add parameter to check exit code 2020-05-27 07:25:39 +02:00
Dietmar Maurer
97fab7aa11 src/tools.rs: new helper to handle command_output (std::process::Output) 2020-05-27 06:53:25 +02:00
Dominik Csapak
40dc103103 fix cli pull api call 
there is no 'delete' parameter, only 'remove-vanished', so fix that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-26 12:39:19 +02:00
Dominik Csapak
de4db62c57 remotes: save passwords as base64 
to avoid having arbitrary characters in the config (e.g. newlines)
note that this breaks existings configs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-26 12:38:06 +02:00
Dietmar Maurer
1a0d3d11d2 src/api2/admin/datastore.rs: add rrd api 2020-05-26 12:26:14 +02:00
Dietmar Maurer
8c03041a2c src/bin/proxmox-backup-proxy.rs: gather block device stats on datastore 2020-05-26 11:20:59 +02:00
Dietmar Maurer
3fcc4b4e5c src/tools/disks.rs: add helper to read block device stats 2020-05-26 11:20:22 +02:00
Dietmar Maurer
3ed07ed2cd src/tools/disks.rs: export read_sys 2020-05-26 09:49:13 +02:00
Dominik Csapak
83fd4b3b1b remote: try to use Struct for api 
with a catch: password is in the struct but we do not want it to return
via the api, so we only 'serialize' it when the string is not empty
(this can only happen when the format is not checked by us, iow.
when its returned from the api) and setting it manually to ""
when we return remotes from the api

this way we can still use the type but do not return the password

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-26 08:55:07 +02:00
Dominik Csapak
5dcdcea293 api2/config/remote: remove password from read_remote 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-26 08:49:12 +02:00
Dominik Csapak
99f443c6ae api2/config/remote: lock and use digest for removal 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-26 08:48:45 +02:00
Dominik Csapak
4f966d0592 api2/config/remote: use rpcenv for digest for read_remote 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-26 08:48:28 +02:00
Dominik Csapak
db0c228719 config/remote: add 'name' to Remote struct 
and use it as section id, like with User

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-26 08:48:05 +02:00
Dietmar Maurer
309ef20d6d src/bin/proxmox-backup-proxy.rs: simplify code 2020-05-25 16:20:32 +02:00
Dietmar Maurer
d0833a70f7 src/bin/proxmox-backup-proxy.rs: gather datastore usage stats 2020-05-25 16:20:32 +02:00
Dietmar Maurer
d8dc281992 www/DataStoreStatus.js: display loadavg stats 2020-05-25 11:54:15 +02:00
Dietmar Maurer
2c66a590c0 src/bin/proxmox-backup-proxy.rs: gather iowait stats 2020-05-25 11:54:15 +02:00
Dietmar Maurer
485841da2c src/bin/proxmox-backup-proxy.rs: gather loadavg stats 2020-05-25 11:40:20 +02:00
Dietmar Maurer
3e930f2bdc www/DataStoreStatus.js: display root disk stats 2020-05-25 11:34:24 +02:00
Dietmar Maurer
dd15c0aa3b src/bin/proxmox-backup-proxy.rs: gather root disk stats 2020-05-25 11:10:07 +02:00
Dietmar Maurer
c1b24fbf0b www/DataStoreStatus.js: display swap stats 2020-05-25 10:39:54 +02:00
Dietmar Maurer
3f23b17298 src/rrd/rrd.rs: do not wrap error and return ErrorKind::NotFound 2020-05-25 10:30:04 +02:00
Dietmar Maurer
c25c9d8dd1 src/bin/proxmox-backup-proxy.rs: gather swap usage stats 2020-05-25 10:25:58 +02:00
Dietmar Maurer
84dc6adcc1 src/rrd/cache.rs: display/log error when RRD load fails 2020-05-25 10:18:53 +02:00
Dietmar Maurer
0c4344650d src/rrd/rrd.rs: store/verify magic number 2020-05-25 09:21:54 +02:00
Dietmar Maurer
4f9513996c src/bin/proxmox-backup-proxy.rs: use block_in_place for rrd update 2020-05-25 08:30:59 +02:00
Dietmar Maurer
736edc7a7e src/rrd/rrd.rs: implement DST_COUNTER 2020-05-25 08:14:30 +02:00
Dietmar Maurer
2b55de407e src/rrd/rrd.rs: correctly compute derived values 
use f64 for time.
 2020-05-25 07:02:04 +02:00
Dietmar Maurer
a608806f65 www/DataStoreStatus.js: display netin/netout 2020-05-24 19:02:35 +02:00
Dietmar Maurer
8f0cec2642 src/bin/proxmox-backup-proxy.rs: gather netin/netout stats 2020-05-24 19:02:35 +02:00
Dietmar Maurer
0ed9a2b3ae src/config/network.rs: implement is_physical_nic() helper 2020-05-24 19:02:35 +02:00
Dietmar Maurer
58edd33d2b src/rrd/rrd.rs: implement DST_DERIVE 2020-05-24 19:02:35 +02:00
Dietmar Maurer
4fb05fde17 src/rrd/rrd.rs: restructure whole code 2020-05-24 16:51:28 +02:00
Dietmar Maurer
daca4f7888 src/rrd/rrd.rs: reduce size by using f64:NAN as UNKNOWN 2020-05-24 09:09:09 +02:00
Dietmar Maurer
4e6585839b src/rrd/rrd.rs: simplify an fix old value deletion 2020-05-24 06:44:06 +02:00
Dietmar Maurer
8c981ae379 rrd: fix display interval, try to avoid numeric errors 2020-05-23 16:03:43 +02:00
Dietmar Maurer
803ab12ad4 rrd: simplify code 2020-05-23 15:37:17 +02:00
Dietmar Maurer
a4a3f7ca5e rrd: pack multiple rrd values into th estat list 2020-05-23 14:03:44 +02:00
Dietmar Maurer
a2f862eed6 add experimental rrd api to get cpu stats 2020-05-23 11:50:53 +02:00
Dietmar Maurer
eaeda365e0 start gathering stats using new rrd module 2020-05-23 10:43:08 +02:00
Dietmar Maurer
6359dc891a add simple rrd implementation 2020-05-23 10:42:48 +02:00
Dietmar Maurer
07ad6470ca src/client/pull.rs: split out pull related code 2020-05-22 08:04:20 +02:00
Dietmar Maurer
a6160cdfeb src/bin/proxmox-backup-proxy.rs: schedule sync jobs 2020-05-22 07:50:59 +02:00
Dietmar Maurer
183125d576 src/api2/pull.rs: aquire try_shared_chunk_store_lock inside pull_store 2020-05-22 07:24:17 +02:00
Dietmar Maurer
a3016d6583 proxmox-backup-manager: add sync-job cli 2020-05-21 11:44:45 +02:00
Dietmar Maurer
b29d046e89 proxmox-backup-manager: split out cert.rs 2020-05-21 11:22:20 +02:00
Dietmar Maurer
380bd7df97 proxmox-backup-manager: split out datastore.rs 2020-05-21 11:14:34 +02:00
Dietmar Maurer
ea6f404e55 proxmox-backup-manager: split out dns.rs 2020-05-21 11:10:58 +02:00
Dietmar Maurer
a35a211d9e proxmox-backup-manager: split out network.rs 2020-05-21 11:08:38 +02:00
Dietmar Maurer
53e14507c1 proxmox-backup-manager: split out acl.rs 2020-05-21 10:56:46 +02:00
Dietmar Maurer
6fa39e53e0 proxmox-backup-manager: split out users.rs 2020-05-21 10:53:06 +02:00
Dietmar Maurer
a220a4564a roxmox-backup-manager: start splitting command into several files 2020-05-21 10:46:07 +02:00
Dietmar Maurer
6f652b1b3a rename 'job' to 'sync' 2020-05-21 10:29:25 +02:00
Dietmar Maurer
b1d4edc769 src/api2/config/job.rs: add job api 2020-05-21 10:16:35 +02:00
Dietmar Maurer
b4900286ce src/config/jobs.rs: use SectionConfig for jobs 2020-05-21 10:16:35 +02:00
Dietmar Maurer
c681885227 src/bin/proxmox-backup-manager.rs: format output of show commands 2020-05-20 16:47:37 +02:00
Dietmar Maurer
ee8b464466 src/tools/systemd.rs: avoid compiler warnings 2020-05-20 16:47:08 +02:00
Dominik Csapak
ce55db66d6 proxmox-backup-manager: add show command for remote and datastore 
to show the data for a single item

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 16:33:07 +02:00
Dominik Csapak
2882c881e9 api2/access/acl: add path and exact parameter to list_acl 
so that we can get only a subset of the acls, filtered by the backed
also return the digest here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 13:44:36 +02:00
Dominik Csapak
12e3895399 api2/access/acl: make update_acl a protected api call 
since we want to set the owner of the acl config to 'root'
which is only possible when using a protected api call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 13:22:41 +02:00
Dominik Csapak
11b6391c83 add 'exact' parameter to extract_acl_node_data 
so that we can return acls for a single path

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 13:22:10 +02:00
Dominik Csapak
b05672579e api2/roles: change return field of role to roleid 
to be compatible with the pve api
with this, we can reuse the ui parts (RoleSelector)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 13:21:47 +02:00
Dominik Csapak
5160c0e986 api2/acl: add privs array to roles 
so that an admin can see which roles have which privileges

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 13:21:37 +02:00
Dominik Csapak
1ad9dd08f4 acls: use constnamemap macro for privileges 
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 13:21:28 +02:00
Dietmar Maurer
25829a879b src/bin/proxmox-backup-proxy.rs: schedule prune jobs 2020-05-20 13:00:53 +02:00
Dietmar Maurer
872062ee9f src/config/datastore.rs_ change prune types from i64 to u64 2020-05-20 13:00:13 +02:00
Dietmar Maurer
67f7ffd0db src/config/datastore.rs: add prune settings 2020-05-20 11:29:59 +02:00
Dietmar Maurer
0fafac2492 src/api2/access/user.rs: remove useless description 
The description is not used at all if we refer to a type.
 2020-05-20 11:27:58 +02:00
Dietmar Maurer
49ff10921c src/api2/types.rs: define PRUNE_SCHEMA_KEEP_* 2020-05-20 10:13:38 +02:00
Dietmar Maurer
479e4932b5 src/tools/systemd/parse_time.rs: improve error message 2020-05-20 09:43:16 +02:00
Dietmar Maurer
dd7a7eae8f src/bin/proxmox-backup-manager.rs: add completion helper for gc-schedule 2020-05-20 09:42:51 +02:00
Dietmar Maurer
8545480a31 src/bin/proxmox-backup-proxy.rs: add simple task scheduler for garbage collection 2020-05-20 08:59:45 +02:00
Dietmar Maurer
d6c28ddf84 src/tools/systemd/time.rs: export parse/verify 2020-05-20 08:38:39 +02:00
Dietmar Maurer
42fdbe5112 src/config/datastore.rs: add gc-schedule property 2020-05-20 08:38:10 +02:00
Dominik Csapak
9c5c383bff user: create default root user as typed struct 
we added a userid attribute to the User struct, but missed that we
created the default user without that attribuet via the json! macro
which lead to a runtime panic on the deserialization

by using the struct directly, such errors will be caught by the compiler
in the future

with this change, we can remove the serde_json import here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-20 06:09:08 +02:00
Dietmar Maurer
7d4e362993 depend on proxmox 0.1.32, src/api2/access/user.rs: simplify code 2020-05-19 12:58:46 +02:00
Dominik Csapak
522c0da0a0 use new 'id_property' for user::User and use it in api calls 
this allows us to return a user::User (or Vec<> of it)
instead of a generic serde value

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-19 09:33:56 +02:00
Dominik Csapak
16c75c580b adapt to changes of SectionConfigPlugin 
it requires not an Option<String> for the optional id_property

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-05-19 09:28:45 +02:00
Dietmar Maurer
07ce44a633 avoid compiler warnings 2020-05-19 07:03:41 +02:00
Dietmar Maurer
e8d1da6a15 depend on proxmox 0.1.31 - use Value to store result metadata 2020-05-18 09:57:35 +02:00
Dietmar Maurer
7a314d18f7 src/tools/systemd/parse_time.rs: check max values 2020-05-16 13:13:50 +02:00
Dietmar Maurer
2d08c97ae2 CalendarEvent - compute_next_value: use change tracking to avoid repeated testing 2020-05-16 10:32:27 +02:00
Dietmar Maurer
50ce1f987d CalendarEvent - compute_next_value: support seconds 2020-05-16 10:21:24 +02:00
Dietmar Maurer
d1a5ffdf78 src/tools/systemd/tm_editor.rs: new helper class 2020-05-16 10:09:41 +02:00
Dietmar Maurer
99baf7afcc CalendarEvent: test and fix repeated values 2020-05-16 07:43:51 +02:00
Dietmar Maurer
fed270bf3f CalendarEvent: speedup/simplify repetition tests 2020-05-16 07:09:53 +02:00
Dietmar Maurer
e05b637c73 src/tools/systemd/parse_time.rs: move parser into separate file 2020-05-16 06:53:15 +02:00
Dietmar Maurer
2ee6b3fdb9 src/tools/systemd/time.rs: implement compute_next_event 2020-05-16 06:33:03 +02:00
Dietmar Maurer
f3a96b2cdb renamed: src/tools/systemd/parser.rs -> src/tools/systemd/config.rs 2020-05-16 06:32:28 +02:00
Dietmar Maurer
a260c74a12 src/tools/systemd/time.rs: add helpers to compute CalendarEvents 2020-05-15 17:55:54 +02:00
Dietmar Maurer
30f577248b src/api2/node/time.rs: avoid custom unsafe readlink implementations 2020-05-15 06:50:07 +02:00
Dietmar Maurer
00491c0230 src/tools/systemd/parser.rs: use different setups for service and timer files, code cleanup 2020-05-14 13:55:13 +02:00
Dietmar Maurer
2ebdbac1c4 depend on nom, add parser for systemd calendar enents and time span 2020-05-14 12:18:30 +02:00
Dietmar Maurer
f486e9e50e add systemd configuration file parser/writer, start job configuration 2020-05-12 13:07:49 +02:00
Dietmar Maurer
65dab0266c proxmox-backup-manager: add completion helper for port list 2020-05-08 17:28:04 +02:00
Dietmar Maurer
525008f7ad proxmox-backup-manager - network list: render ports/slaves 
And render interface name as first comumn.
 2020-05-08 16:07:23 +02:00
Dietmar Maurer
5bef0f43da src/config/network.rs - check_bridge_ports: correctly check vlan ports 2020-05-08 15:51:47 +02:00
Dietmar Maurer
0f6bdbb01f src/config/network.rs - write_config: add more consistency checks 2020-05-08 14:31:38 +02:00
Dietmar Maurer
a4ccb46176 src/config/network.rs: avoid duplicate port usage 2020-05-08 11:15:00 +02:00
Dietmar Maurer
80bf084876 src/config/network.rs: do not combine entries 
It is unclear when and how to write combined entries ...
 2020-05-08 10:20:57 +02:00
Dietmar Maurer
db5672e83e src/config/network.rs: always write bridge_ports and bond_slaves 
So that we can reliable detect the interface type.
 2020-05-08 09:58:03 +02:00
Dietmar Maurer
86a5d56c4e proxmox-backup-manager: add network create command 2020-05-08 09:55:56 +02:00
Dietmar Maurer
3dd27a3bf8 src/api2/node/network.rs: add protected flag to revert 2020-05-08 09:30:25 +02:00
Dietmar Maurer
3aedb73816 src/api2/node/network.rs: pass bridge_ports and slaves a property strings 
To make it compatible with pve.
 2020-05-08 08:49:17 +02:00
Dietmar Maurer
bab5d18c3d src/config/network.rs: implement bond_mode 
and rename bond_slaves to slaves to make it compatible with pve.
 2020-05-07 14:07:45 +02:00
Dietmar Maurer
c2ffc68554 src/api2/node/network.rs: cleanup - factor out check_duplicate_gateway 2020-05-07 11:26:30 +02:00
Dietmar Maurer
9651833130 src/api2/node/network.rs: allow to create bridge and bond 2020-05-07 11:09:12 +02:00
Dietmar Maurer
7b22acd0c2 src/config/network.rs: make it compatible with pve 
and depend on proxmox 0.1.26
 2020-05-07 09:28:25 +02:00
Dietmar Maurer
5751e49566 src/server/worker_task.rs: implement and use status command 2020-05-07 09:27:33 +02:00
Dietmar Maurer
197de83ffa src/server/command_socket.rs: do not abort loop on client errors, allow backup gid 2020-05-07 09:27:33 +02:00
Wolfgang Bumiller
10effc9849 add tools/disks.rs (work in progress...) 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-05-05 10:14:42 +02:00
Dietmar Maurer
99641a6bbb garbage_collect: call fail_on_abort to abort GV when requested. 2020-05-05 09:06:34 +02:00
Dietmar Maurer
74f7240b8d src/bin/proxmox-backup-client.rs: add human readable date to prune list 2020-05-05 07:33:58 +02:00
Dietmar Maurer
db1e061dcb src/bin/proxmox-backup-client.rs: correctly format prune result list. 2020-05-05 06:45:37 +02:00
Dietmar Maurer
b080583ba8 src/bin/proxmox-backup-manager.rs: improve user list output 2020-05-01 16:22:50 +02:00
Dietmar Maurer
ed751dc2ab src/api2/node/status.rs: rework api, implement reboot and shutdown 2020-04-30 11:52:40 +02:00
Dietmar Maurer
ca9dfe5fa4 src/api2/node/tasks.rs: use api macro features for default values 2020-04-30 11:51:56 +02:00
Dietmar Maurer
720af9f69b src/api2/node/tasks.rs: allow users to list/access there own tasks 2020-04-30 10:05:50 +02:00
Dietmar Maurer
f1490da82a use resonable acl paths (fixup) 2020-04-30 09:32:13 +02:00
Dietmar Maurer
74c08a5782 use reasonable acl paths 2020-04-30 09:30:00 +02:00
Dietmar Maurer
bd88dc4116 cached_config: avoid parsing non-existent files multiple times 2020-04-30 07:04:23 +02:00
Dominik Csapak
f9e3b1104e change index to templates using handlebars 
using a handlebars instance in ApiConfig, to cache the templates
as long as possible, this is currently ok, as the index template
can only change when the whole package changes

if we split this in the future, we have to trigger a reload of
the daemon on gui package upgrade (so that the template gets reloaded)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-04-29 17:05:53 +02:00
Dietmar Maurer
bc0d03885c use proxmox 0.1.25, use new EnumEntry feature 2020-04-29 13:01:24 +02:00
Dietmar Maurer
b9f2f761bb avoid problems with missing acl.cfg and user.cfg 2020-04-29 10:40:42 +02:00
Dietmar Maurer
30fb602578 src/api2/admin/datastore.rs - get_datastore_list: only return name and comment 
We dont want to leak the full configuration to users with limited access permission.
Please use the api2::config::datastore api to get the full configuration.
 2020-04-29 09:21:34 +02:00
Dietmar Maurer
0a00f6e01c src/api2/config/datastore.rs_ add delete property to update method 2020-04-29 09:09:59 +02:00
Dietmar Maurer
30003baaa4 src/api2/config/remote.rs: fix white space 2020-04-29 09:09:39 +02:00
Dietmar Maurer
5211705ff1 src/api2/config/remote.rs: add delete parameter to update method 2020-04-29 09:04:17 +02:00
Dietmar Maurer
ec67af9af3 src/api2/pull.rs: require Datastore.Prune if delete flag is set. 2020-04-29 07:19:32 +02:00
Dietmar Maurer
8247db5b39 src/config/acl.rs: introduice privileges and roles for remotes 2020-04-29 07:03:44 +02:00
Dietmar Maurer
409f44247b fix api2::types::ACL_ROLE_SCHEMA 
make sure we list all roles ...
 2020-04-28 13:25:02 +02:00
Dietmar Maurer
dd335b77f5 src/config/acl.rs - fix regression tests 2020-04-28 11:16:15 +02:00
Dietmar Maurer
6f6aa95abb add Datastore.Backup, Datastore.PowerUser and Datastore.Reader role 2020-04-28 11:07:25 +02:00
Dietmar Maurer
54552dda59 implemnt backup ownership, improve datastore access permissions 2020-04-28 10:22:25 +02:00
Dietmar Maurer
1347b1152d src/config/cached_user_info.rs - lookup_privs: correctly handle superuser 2020-04-27 13:22:03 +02:00
Dietmar Maurer
d00e1a216f src/config/acl.rs: introduce more/better datastore privileges 2020-04-27 07:13:50 +02:00
Dietmar Maurer
9c7fe29dfc src/config/acl.rs: rtename PRTIV_DATASTORE_ALLOCATE to PRIV_DATASTORE_MODIFY 2020-04-27 06:50:35 +02:00
Dietmar Maurer
14627d671a src/bin/proxmox-backup-manager.rs: add dns sub command 
Also improved the DNS api, added a --delete option.
 2020-04-26 08:23:23 +02:00
Dietmar Maurer
76227a6acd src/bin/proxmox-backup-manager.rs: fix node parameter handling 2020-04-25 17:20:22 +02:00
Dietmar Maurer
26d9aebc28 move src/api2/config/network.rs to src/api2/node/network.rs 
So that we have the same api path for network config as pve.
 2020-04-25 17:00:38 +02:00
Dietmar Maurer
1ca540a63b src/config/network.rs: auto-add lo, and implement a few regression tests 2020-04-24 12:57:11 +02:00
Wolfgang Bumiller
9094186a57 xattr: cleanup: don't use pxar types in the API 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-04-24 11:23:48 +02:00
Wolfgang Bumiller
27a3decbfe xattr: api cleanup 
Make `flistxattr()` return a `ListXAttr` helper which
provides an iterator over `&CStr`.

This exposes the property that xattr names are a
zero-terminated string without simply being an opaque
"byte vector". Using &[u8] as a type here is too lax.

Also let `fgetxattr` take a `CStr`. While this may be a
burden on the caller, we usually already have
zero-terminated strings on the call site. Currently we only
use this method coming from `flistxattr` after all.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-04-24 10:56:52 +02:00
Wolfgang Bumiller
9af76ef075 xattr: use checked_mul to increase size 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-04-24 10:56:52 +02:00
Wolfgang Bumiller
00ec8d1685 tools: pub use Fd from proxmox crate 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-04-24 10:56:52 +02:00
Dietmar Maurer
fd7c0979b4 src/bin/proxmox-backup-manager.rs: implement netwerk revert 2020-04-24 10:45:49 +02:00
Dietmar Maurer
c67bc9c35c src/bin/proxmox-backup-manager.rs: new command to show pending network changes 2020-04-24 10:27:43 +02:00
Dietmar Maurer
3181f9b625 src/bin/proxmox-backup-manager.rs: only show pending changes with "text" format 2020-04-24 10:16:57 +02:00
Dietmar Maurer
2eefd9aee1 src/config/network.rs: implement network reload, set "changes" attribute 2020-04-24 09:55:46 +02:00
Dietmar Maurer
8a6b86b8a7 src/config/network.rs: use a simple String for comments 2020-04-24 07:46:08 +02:00
Dietmar Maurer
96d9478668 src/config/network/parser.rs: corectly detect vanished interfaces 2020-04-24 07:26:54 +02:00
Dietmar Maurer
10a9be45bd src/api2/config/network.rs: implement update/delete comments 2020-04-23 16:08:35 +02:00
Dietmar Maurer
5f60a58fd5 src/config/network.rs; support interface comments, cleanups 2020-04-23 15:54:30 +02:00
Dietmar Maurer
659c3be3d5 src/config/network.rs: avoid newline after family options 2020-04-23 11:30:41 +02:00
Dietmar Maurer
5e4e88e83f src/api2/config/network.rs: implement update/delete for bridge_ports and bond_slaves 2020-04-23 11:21:27 +02:00
Dietmar Maurer
339965d720 src/api2/config/network.rs: only allow one default gateway 2020-04-23 10:37:40 +02:00
Dietmar Maurer
c38b4bb8b2 src/config/network.rs: do not allow to change interface type 2020-04-23 09:43:38 +02:00
Dietmar Maurer
42fbe91a34 src/config/network.rs: parse bond-slaves 2020-04-23 09:31:10 +02:00
Dietmar Maurer
1d9a68c2fc src/config/network.rs: parse bridge-ports 2020-04-23 09:24:17 +02:00
Dietmar Maurer
02269f3dba src/config/network.rs: introduce NetworkInterfaceType 2020-04-23 08:45:03 +02:00
Dietmar Maurer
d5ca9bd5df src/config/network.rs: cleanup (new helper combine_entry) 2020-04-23 07:54:12 +02:00
Dietmar Maurer
02e36d96ad src/config/network.rs: write changes to interfaces.new 2020-04-23 07:19:29 +02:00
Dietmar Maurer
2c18efd902 src/config/network.rs: use a single mtu setting (instead of mtu_v4 and mtu_v6) 2020-04-23 07:07:14 +02:00
Dietmar Maurer
4cb6bd894c src/bin/proxmox-backup-manager.rs: improve network list output format 2020-04-23 06:44:55 +02:00
Dietmar Maurer
b1564af25a src/bin/proxmox-backup-manager.rs: format datastore list output 2020-04-22 17:37:20 +02:00
Dietmar Maurer
bf004ecd87 src/bin/proxmox-backup-manager.rs: format network list output 2020-04-22 17:14:52 +02:00
Dietmar Maurer
f1026a5aa9 src/api2/config/network.rs: allow to update 'auto' flag 2020-04-22 16:46:46 +02:00
Dietmar Maurer
3fce3bc36e src/config/network/parser.rs: parse MTU settings 2020-04-22 13:44:51 +02:00
Dietmar Maurer
f8e7ac686a src/config/network.rs: only save attriubutes used by configuration method 2020-04-22 12:42:09 +02:00
Dietmar Maurer
c016482c7a src/api2/config/network.rs: implement delete property 2020-04-22 12:19:31 +02:00
Dietmar Maurer
27f2c23049 src/api2/config/network.rs: allow to update configuration method 2020-04-22 11:32:36 +02:00
Dietmar Maurer
df6bb03d0e src/api2/config/network.rs: improve network api 2020-04-22 10:54:07 +02:00
Dietmar Maurer
e2d940b949 src/config/network/parser.rs: remove debug println 2020-04-22 10:53:26 +02:00
Dietmar Maurer
0c226bc173 src/config/network/helper.rs: fix CIDR regex 2020-04-22 10:52:31 +02:00
Dietmar Maurer
76cf5208cf src/api2/types.rs: add schemas for IP/CIDR 2020-04-22 10:28:53 +02:00
Dietmar Maurer
2ea7bf1b3d src/api2/config/datastore.rs_ fix method docs 2020-04-22 08:53:16 +02:00
Dietmar Maurer
8b57cd4441 src/config/network.rs: remove netmask support 
rely on cidr instead.
 2020-04-22 08:45:13 +02:00
Dietmar Maurer
68da20bf62 src/api2/types.rs: define NETWORK_INTERFACE_NAME_SCHEMA 2020-04-21 17:54:52 +02:00
Dietmar Maurer
c357260d09 src/config/network.rs: move type definitions to src/api2/types.rs 2020-04-21 17:25:05 +02:00
Dietmar Maurer
7e02d08cd0 rename ConfigMethod to NetworkConfigMethod 2020-04-21 17:17:57 +02:00
Dietmar Maurer
ca0e534796 src/api2/config/network.rs: start network configuration api 2020-04-21 14:28:26 +02:00
Dietmar Maurer
904e988667 src/config/network.rs: impleement load/save 2020-04-21 12:55:33 +02:00
Dietmar Maurer
3f129233be src/config/network.rs: add Interface flags 'exists' and 'active' 2020-04-21 11:46:56 +02:00
Dietmar Maurer
a9bb491e35 src/config/network.rs: cleanup autostart flag handling 2020-04-21 11:06:22 +02:00
Dietmar Maurer
1ec7f8a0dd src/config/network/helper.rs: new helper get_network_interfaces() 2020-04-21 10:32:54 +02:00
Dietmar Maurer
92310d585c src/config/network.rs: simplify code 2020-04-20 18:10:15 +02:00
Dietmar Maurer
f34d4401f7 src/config/network.rs: read/write /etc/network/interfaces 
Start implementing a recursive descent parser.
 2020-04-20 14:15:57 +02:00
Dietmar Maurer
6e695960ca src/config/cached_user_info.rs: cache it up to 5 seconds 2020-04-18 08:49:20 +02:00
Dietmar Maurer
365f0f720c fix permission tests using non-uri parameters 
We nood to do those tests inside the fuction body instead...
 2020-04-18 08:23:04 +02:00
Dietmar Maurer
a737179eb4 src/config/cached_user_info.rs: new check_privs helper 2020-04-18 08:09:34 +02:00
Dietmar Maurer
bb072ba49c src/api2/access.rs: cleanup 2020-04-18 07:28:25 +02:00
Dietmar Maurer
ff329f970b src/api2/types.rs: use anyhow::Error in test cases 2020-04-18 07:05:31 +02:00
Wolfgang Bumiller
f7d4e4b506 switch from failure to anyhow 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-04-17 18:43:30 +02:00
Dietmar Maurer
404d78c41e src/api2/pull.rs: add access permission 2020-04-17 15:27:04 +02:00
Dietmar Maurer
1bfc1efa50 src/api2/subscription.rs: add access permissions 2020-04-17 15:14:28 +02:00
Dietmar Maurer
73ce1d1146 src/api2/reader.rs: add access permissions 2020-04-17 15:01:56 +02:00
Dietmar Maurer
70e5f2461d src/api2/config/remote.rs: add access permissions 2020-04-17 14:57:26 +02:00
Dietmar Maurer
c0ef209aeb src/api2/config/datastore.rs: impl digest check for delete, add access permissions 2020-04-17 14:51:29 +02:00
Dietmar Maurer
9f9f7eefa3 src/api2/backup.rs: add access permissions 2020-04-17 14:40:20 +02:00
Dietmar Maurer
bb34b58910 src/api2/admin/datastore.rs: add access permissions - first try 
We need to refine this later (introduce backup owner concept?)
 2020-04-17 14:36:27 +02:00
Dietmar Maurer
5972def5ec acl: change path "storage" to "datastore" 2020-04-17 14:15:44 +02:00
Dietmar Maurer
aa90ced3bf src/api2/access/role.rs: use schema ACL_ROLE_SCHEMA 2020-04-17 14:14:06 +02:00
Dietmar Maurer
ca257c8097 move type defs from src/api2/access/acl.rs to src/api2/types.rs 2020-04-17 14:13:15 +02:00
Dietmar Maurer
3fff55b293 src/api2/access/role.rs: new api to list roles 2020-04-17 14:03:24 +02:00
Dietmar Maurer
4f66423fcc src/api2/access/user.rs: add access permissions 2020-04-17 11:04:36 +02:00
Dietmar Maurer
d4f020f4c5 src/api2/access/user.rs: add access permissions 2020-04-17 10:08:45 +02:00
Dietmar Maurer
d28ddb8e04 src/api2/access/acl.rs: add access permissions 2020-04-17 10:03:09 +02:00
Dietmar Maurer
83b6a7cf71 src/api2/node/tasks.rs: use api macro, implement access permissions 2020-04-16 17:47:21 +02:00
Dietmar Maurer
e4681f9f71 src/api2/node/syslog.rs: add access permissions 2020-04-16 17:08:19 +02:00
Dietmar Maurer
b5037fa8ed src/api2/node/status.rs: add access permissions 2020-04-16 17:05:09 +02:00
Dietmar Maurer
9989d2c4e9 src/server/rest.rs: reduce delay for permission error to 500ms 2020-04-16 12:56:34 +02:00
Dietmar Maurer
1cf7bbf412 src/api2/node/services.rs: add access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer
68ed0c629d src/api2/node/journal.rs: add access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer
4b40148caa start impl. access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer
423e656163 src/config/cached_user_info.rs: new helper class 2020-04-16 10:05:16 +02:00
Dietmar Maurer
109d7817cd src/config/user.rs - cached_config: do not store/return digest 2020-04-15 11:35:57 +02:00
Dietmar Maurer
5354511fd0 src/config/acl.rs: implement cached_config 2020-04-15 11:30:47 +02:00
Dietmar Maurer
bd098a7f77 src/api2/node/dns.rs: use api macro (cleanup) 2020-04-15 10:09:18 +02:00
Dietmar Maurer
8d048af2bf acl: improve NoAccess handling 2020-04-15 08:11:43 +02:00
Dietmar Maurer
9a328319dd pxar extract: remove pattern from arg_param, add target instead 2020-04-15 06:41:37 +02:00
Dietmar Maurer
7e3d2e5b41 pxar create: remove exclude from arg_param 2020-04-15 06:31:46 +02:00
Dietmar Maurer
9c06f6c292 fix previous commit - use result. 2020-04-14 17:48:10 +02:00
Dietmar Maurer
9f4e47dd93 acl update: check path 2020-04-14 17:23:48 +02:00
Dietmar Maurer
d83175dd69 acl update: check if user exist. 2020-04-14 13:46:27 +02:00
Dietmar Maurer
68ccdf09a4 src/config/user.rs: implement user config cache 2020-04-14 13:45:45 +02:00
Dietmar Maurer
9765092ede acl api: implement update 2020-04-14 10:16:49 +02:00
Dietmar Maurer
ed3e60ae69 start ACL api 2020-04-13 11:09:44 +02:00
Dietmar Maurer
a83eab3c4d acl: use BTreeMap and BTreeSet to avoid sort() 2020-04-12 17:13:53 +02:00
Dietmar Maurer
0815ec7e65 acl: implement roles(), add regression tests. 2020-04-12 13:06:50 +02:00
Dietmar Maurer
5c6cdf9815 add acl config 2020-04-11 12:24:26 +02:00
Dietmar Maurer
b88f9c5b1e PASSWORD_SCHEMA: set max_length to 1024 (for tickets) 2020-04-09 13:35:58 +02:00
Dietmar Maurer
879546aff6 api: add default property to domain list 2020-04-09 13:35:08 +02:00
Dietmar Maurer
73b40e9b46 api: correctly sort access subdirmap 2020-04-09 13:34:07 +02:00
Dietmar Maurer
708db4b3ae api: add list_domains 2020-04-09 11:36:45 +02:00
Dietmar Maurer
685e13347e api: move config/user to access/users, implement change_password 
To make it similar to the pve api
 2020-04-09 10:21:24 +02:00
Dietmar Maurer
7d817b0358 implement auth framework 2020-04-08 14:06:15 +02:00
Dietmar Maurer
579728c641 add user configiguration 2020-04-08 14:06:15 +02:00
Dietmar Maurer
cf459b1982 gc: log pending removals 2020-04-06 09:50:40 +02:00
Dietmar Maurer
dda7015497 prune api: return a usable result (we run synchronous anyways) 2020-04-01 12:24:28 +02:00
Dietmar Maurer
4299ca727c src/server/rest.rs: use correct formatter 2020-03-26 12:54:20 +01:00
Dominik Csapak
6650a242fb rewrite future select in upgrade_to_backup_protocol using select macro 
and handle all ok/err cases with at least logging

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
 2020-03-18 11:33:59 +01:00
Dietmar Maurer
ca26db1140 delete unused src/storage/config.rs 2020-03-02 13:31:38 +01:00
Dietmar Maurer
90c5239d46 use SectionConfig from proxmox 0.1.18-1 2020-03-02 12:52:11 +01:00
Dietmar Maurer
18deda403d src/bin/proxmox-backup-client.rs - list_groups: render full snapshot path for last-backup column 2020-02-28 09:09:22 +01:00
Dietmar Maurer
6882b3716c src/bin/proxmox-backup-manager.rs: avoid compiler warning 2020-02-28 08:52:12 +01:00
Christian Ebner
32d192a952 catalog: shell: Use the new logic including resolving symlinks for catalog 
and remove the old unused code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:51:01 +01:00
Christian Ebner
fee5528e59 catalog: shell: introduce new CatalogPathStack to navigate in catalog shell. 
This is basically a rewrite of the current logic for navigating the catalog,
but in addition allows to follow symlinks.
Following symlinks introduces the issue that generation of canonical paths
(needed in the actual pxar archive) is more complex, as symlinks have to be
resolved and loops avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:50:43 +01:00
Christian Ebner
4145c36749 catalog: add missing function documentation. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:50:08 +01:00
Christian Ebner
c2f9149461 catalog: introduce is_symlink() to check if DirEntry is a symlink. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:49:55 +01:00
Christian Ebner
11ee5c0563 catalog: derive PartEq for equality checks of two DirEntry's. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:49:39 +01:00
Christian Ebner
536683e73b src/backup/dynamic_index.rs: Add LruCache for chunks. 
In order to improve non-sequential reads of chunks as e.g. in FUSE.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:47:16 +01:00
Christian Ebner
35ddf0b419 catalog: shell: Introduce clear-selected command. 
'clear-selected' allows to clear all the match patterns from the list of
patterns for a subsequent restore.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:40:55 +01:00
Christian Ebner
8e464141cf catalog: shell: Improve list-selected command. 
'list-selected' now shows the filenames matching the patterns for a restore
instead of the patterns themselfs.
The patterns can be displayed by passing the '--pattern' flag.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:40:42 +01:00
Christian Ebner
351b913d1e pxar::decoder: Return the correct symlink target size. 
This eliminates also repeated calls to readlink in fuse, which occur when the
preallocated buffer to store the symlink target path is to small.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 08:38:56 +01:00
Christian Ebner
189996cf4a proxmox-backup-client: expose exclude match patterns to cli. 
Allows to pass exclude match patterns to the 'backup' command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-28 07:56:09 +01:00
Dietmar Maurer
ac3faaf5c0 src/bin/proxmox-backup-manager.rs: cleanup output handling 2020-02-28 07:42:36 +01:00
Dietmar Maurer
4939255fb4 src/tools/format.rs: move output rendering code to this new file 2020-02-28 07:30:35 +01:00
Dietmar Maurer
c81b2b7c70 src/bin/proxmox-backup-client.rs - list_groups: use format_and_print_result_full() 
Depend on proxmox 0.1.16-1
 2020-02-27 13:32:11 +01:00
Dietmar Maurer
c2043614f7 src/bin/proxmox-backup-client.rs: use new output formath helpers from proxmox 0.15.1-1 2020-02-27 12:41:15 +01:00
Dietmar Maurer
e23f586344 src/bin/proxmox-backup-client.rs - status: improve alignment 2020-02-27 11:38:09 +01:00
Dietmar Maurer
be2425ff85 src/bin/proxmox-backup-client.rs - status: do not print header 2020-02-27 11:31:05 +01:00
Dietmar Maurer
f24fc1166b src/bin/proxmox-backup-client.rs - list_snapshots: use format_and_print_result_full() 
Depend on proxmox v0.1.14.
 2020-02-27 11:28:44 +01:00
Christian Ebner
af934f8cf6 tools::lru_cache: Improve access() and insert() by using HashMap::entry(). 
entry() allows to lookup the position where and entry belongs and update/insert
it in the HashMap more efficiently than get_mut() and insert().
Details: https://gankra.github.io/blah/hashbrown-insert/

In addition, use the struct LinkedList and remove the outdated code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-27 06:56:25 +01:00
Christian Ebner
e3ab9a383c tools::lru_cache: Separate LinkedList part of LruCache into own struct. 
In order to keep the separation more aparent and avoid mut borrow conflics.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-27 06:55:48 +01:00
Christian Ebner
75c2ee7bab tools::lru_cache: Make key generic. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-27 06:55:16 +01:00
Christian Ebner
03f779c6f5 catalog: shell: Improve output of stat command. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-27 06:53:17 +01:00
Dietmar Maurer
ea5f547fdc src/bin/proxmox-backup-client.rs - list_snapshot_files: use format_and_print_result_full() 2020-02-26 13:49:47 +01:00
Dietmar Maurer
390c5bdde4 src/bin/proxmox-backup-client.rs - status: use format_and_print_result_full 2020-02-26 13:29:00 +01:00
Dietmar Maurer
03ac286c80 src/bin/proxmox-backup-manager.rs: use crate::tools::runtime::block_on() 2020-02-26 11:37:38 +01:00
Dietmar Maurer
dd04383bb8 src/tools/runtime.rs: call OPENSSL_thread_stop to avoid race with openssl cleanup handlers 2020-02-26 10:38:55 +01:00
Dietmar Maurer
ca2dbb8af1 src/client/backup_writer.rs: also print archive size 2020-02-25 14:12:29 +01:00
Wolfgang Bumiller
0bcbb5434e cleanup unused module 
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-25 11:38:27 +01:00
Christian Ebner
495b238bc1 pxar: Change text of warning for files matched by exclude patterns. 
In addition to the .pxarexclude files, glob match patterns can be passed to pxar
also via cli parameters.
Therefore the warning is rephrased to be more ambiguous.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-02-25 10:47:24 +01:00
Dietmar Maurer
6da73c823f src/client/backup_writer.rs: prefix upload stats with archive name 2020-02-24 13:24:46 +01:00
Dietmar Maurer
e02c3d461f src/client/backup_writer.rs: implement verbose options 2020-02-24 12:48:40 +01:00
Dietmar Maurer
ee5fe978e2 src/client/backup_writer.rs: reduce verbosity 2020-02-24 11:33:00 +01:00
Dietmar Maurer
8ce49a76da src/backup/backup_info.rs: fix SNAPSHOT_PATH_REGEX and GROUP_PATH_REGEX 2020-02-18 13:16:35 +01:00
Wolfgang Bumiller
9e003074cb runtime: fix blocking strategy: 
- do not "double"-block_in_place() (it may not be nested)
- do not call block_in_place() in non-worker threads

is_in_tokio() isn't sufficient, we need to actually know
that we're in a worker-thread, so we do this by remembering
that we're blocking.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-17 09:48:22 +01:00
Wolfgang Bumiller
1283d58ca9 runtime: remove IN_TOKIO thread local variable 
tokio now has Handle::try_current() allowing us to
generally check for a tokio runtime even if spawned by
someone else

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-13 08:56:38 +01:00
Dietmar Maurer
a6e3da987c src/client/http_client.rs: do not store fingerprints passed via options in cache 2020-02-11 11:54:43 +01:00
Dietmar Maurer
1bd6f32b43 src/client/http_client.rs: always convert fingerprint to lowercase 2020-02-11 11:42:03 +01:00
Dietmar Maurer
d1c657276a src/client/http_client.rs: remove useless password_env 2020-02-11 11:42:03 +01:00
Dietmar Maurer
a05c0c6ff6 src/bin/proxmox-backup-client.rs: allows to pass fingerprint via env vars 2020-02-11 11:42:03 +01:00
Thomas Lamprecht
b69b8af26b s/pbsbanner/proxmox-backup-banner/ + move to /usr/lib 
We want to avoid pbs if possible and also avoid placing internal
binaries, not intended for human direct use, in /bin or /sbin paths.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-02-10 14:43:28 +01:00
Thomas Lamprecht
e2b5e75a3e pbsbanner: use match and also return error 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-02-07 18:30:55 +01:00
Thomas Lamprecht
274b0c7bb7 add login banner service 
Modeled after the one from PVE, but using rust instead of perl for
resolving the nodename and writing to /etc/issue

Behavior differs a bit. We write all non-loopback addresses to this
file, as the gui accepts connections from them all, so limiting it to
the first one is not really sensible.
Further an error to resolve, or only getting loopback addresses won't
write out an empty /etc/issue file, but a note about the error at the
place where the address would be displayed.

Named it "pbsbanner", not "proxmox-backup-banner" as it's rather an
internal tool anyway and mirrors pvebanner, pmgbanner

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2020-02-06 22:19:22 +01:00
Christian Ebner
ba050e3788 catalog: shell: Allow two or more successive slashes in path. 
Two or more successive slashes should be allowed and treated as a single slash.
We also do not treat two successive slashes at the beginning of a path any
different.

Details are found here:
https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap04.html#tag_04_11

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-05 09:32:29 +01:00
Christian Ebner
9ff9a00572 pxar::fuse: cleanup: Remove unused code. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-03 13:43:11 +01:00
Christian Ebner
3e69abef02 pxar::fuse: Refactor lookup in order to cache accessed entries. 
Context::find_goodbye_entry() is removed and incorporated into the lookup
callback in order to take advantage of the entry_cache and since it is only used
inside this callback.
All entries read on lookup are cached.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-03 13:43:06 +01:00
Christian Ebner
63698e720c pxar: Improve read performance for fuse. 
By storing the payload start offset in the `DirectoryEntry` and passing this
information to `Decoder::read()`, the payload can be read directly and a repeated
re-reading of the entry information is avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-02-03 12:08:03 +01:00
Dietmar Maurer
032d3ad80f src/bin/proxmox-backup-client.rs: do not use api macro for prune 
So that we can use add_common_prune_prameters!() macro again.
 2020-01-31 10:31:36 +01:00
Dietmar Maurer
b8a192e3b0 src/config/remote.rs: do not serialize empty option 2020-01-31 09:09:24 +01:00
Dietmar Maurer
c43b36bcb6 src/section_config.rs - write: improve error message 2020-01-31 09:08:42 +01:00
Dietmar Maurer
501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Christian Ebner
74616f63b0 proxmox-backup-client: add missing prune options to api definition. 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
 2020-01-31 07:01:56 +01:00
Dietmar Maurer
93fbb4ef0a update proxmox dependency to 0.1.12 2020-01-31 06:59:17 +01:00
Dietmar Maurer
43592f290d src/bin/proxmox-backup-manager.rs: remove stale debug comment 2020-01-30 17:59:13 +01:00
Dietmar Maurer
9e165b5cad src/bin/proxmox-backup-manager.rs - list remotes: do not use client, call directly 2020-01-30 17:57:37 +01:00
Dietmar Maurer
9894469e89 src/bin/proxmox-backup-manager.rs: use new texdt table formatter 2020-01-30 13:31:20 +01:00
Dietmar Maurer
8528fce874 src/api2/node/tasks.rs; make list_tasks public 2020-01-30 13:31:20 +01:00
Dietmar Maurer
70fa1aa3fc fixup 2020-01-30 13:31:20 +01:00
Dietmar Maurer
5eeea607ae src/section_config.rs - convert_to_array: add option to skip properties 2020-01-30 13:26:46 +01:00
Christian Ebner
93a9eb284e pxar::fuse: Do not return fcaps value on listxattr. 
listxattr must only return the name list, no extended attribute values.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-29 09:35:09 +01:00
Christian Ebner
5d2158e89a pxar::fuse: return ACL names in listxattr if present 
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-29 09:35:03 +01:00
Christian Ebner
72677fb0a5 pxar::fuse: add support to read ACLs. 
ACLs are stored separately in the pxar archive. This implements the functionality
needed to read the ACLs and return them as extended attributes in the getxattr
callback.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-29 09:35:00 +01:00
Christian Ebner
0502ce6da3 tools::acl: Add helpers to create extended attribute buffer. 
This helpers are used to construct the extended attributes values from
the ACLs stored in the pxar archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-29 09:34:57 +01:00
Christian Ebner
bcf0d452c9 pxar: move acl helper functions to src/tools/acl.rs 
They are not only needed by the pxar::sequential_decoder but also for the fuse
xattr impl, so it makes more sense to have them there.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-29 09:34:54 +01:00
Wolfgang Bumiller
6f763ae673 tools: LruCache: add dropcheck marker 
See https://doc.rust-lang.org/nomicon/phantom-data.html

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
 2020-01-29 09:18:27 +01:00