Commit Graph

781 Commits

Author SHA1 Message Date
Dietmar Maurer 2995aedf1d src/bin/proxmox_backup_client/mount.rs: fix img name completion 2020-10-06 09:40:00 +02:00
Stefan Reiter 45f9b32e0f client: implement map/unmap commands for .img backups
Allows mapping fixed-index .img files (usually from VM backups) to be
mapped to a local loopback device.

The architecture uses a FUSE-backed temp file mapped to a loopdev:

  /dev/loopX -> FUSE /run/pbs-loopdev/xxx -> backup client -> PBS

Since unmapping requires some cleanup (unmap the loopdev, stop FUSE,
remove the temp files) a special 'unmap' command is added, which uses a
PID file to send SIGINT to the backup-client instance started with
'map', which will handle the cleanup itself.

The polling with select! in mount.rs needs to be split in two, since we
have a chicken and egg problem between running FUSE and setting up the
loop device - so we need to do them concurrently, until the loopdev is
assigned, at which point we can report success and daemonize, and then
continue polling the FUSE loop future.

A loopdev module is added to tools containing all required functions for
mapping a loop device to the FUSE file, with the ioctls moved into an
inline module to avoid exposing them directly.

The client code is placed in the 'mount' module, which, while
admittedly a loose fit, allows reuse of the daemonizing code.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-06 09:08:14 +02:00
Stefan Reiter 1d0b662b42 mount: handle SIGTERM as well
instead of only SIGINT

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-06 09:08:14 +02:00
Dietmar Maurer ef1b436350 paperkey: add html output 2020-09-30 10:49:20 +02:00
Dominik Csapak ba20987ae7 client/remote: add support to specify port number
this adds the ability to add port numbers in the backup repo spec
as well as remotes, so that user that are behind a
NAT/Firewall/Reverse proxy can still use it

also adds some explanation and examples to the docs to make it clearer
for h2 client i left the localhost:8007 part, since it is not
configurable where we bind to

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 10:49:20 +02:00
Dominik Csapak 9a76091785 proxmox-backup-proxy: add task archive rotation
this starts a task once a day at "00:00" that rotates the task log
archive if it is bigger than 500k

if we want, we can make the schedule/size limit/etc. configurable,
but for now it's ok to set fixed values for that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:41:18 +02:00
Dietmar Maurer b02d49ab26 proxmox_backup_client key: allow to generate paperkey for master key 2020-09-29 08:29:42 +02:00
Dietmar Maurer 82a0cd2ad4 proxmox_backup_client key: add new paper-key command 2020-09-29 08:29:42 +02:00
Dietmar Maurer 42fd40a124 src/bin/proxmox_backup_client/benchmark.rs: avoid compiler warning 2020-09-26 16:13:19 +02:00
Dominik Csapak d7a122a026 use jobstate mechanism for verify/garbage_collection schedules
also changes:
* correct comment about reset (replace 'sync' with 'action')
* check schedule change correctly (only when it is actually changed)

with this changes, we can drop the 'lookup_last_worker' method

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:12 +02:00
Dominik Csapak 6c25588e63 proxy: fix error handling in prune scheduling
we rely on the jobstate handling to write the error of the worker
into its state file, but we used '?' here in a block which does not
return the error to the block, but to the function/closure instead

so if a prune job failed because of such an '?', we did not write
into the statefile and got a wrong state there

instead use our try_block! macro that wraps the code in a closure

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:09 +02:00
Dietmar Maurer baae780c99 benchmark: use compressable data to get more realistic result
And add a benchmatrk to test chunk verify speed (decompress+sha256).
2020-09-24 08:58:13 +02:00
Dominik Csapak 9866de5e3d datastore/prune schedules: use JobState for tracking of schedules
like the sync jobs, so that if an admin configures a schedule it
really starts the next time that time is reached not immediately

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-19 06:24:37 +02:00
Hannes Laimer c040ec22f7 add verification scheduling to proxmox-backup-proxy
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:14:05 +02:00
Dietmar Maurer 6a7be83efe avoid chrono dependency, depend on proxmox 0.3.8
- remove chrono dependency

- depend on proxmox 0.3.8

- remove epoch_now, epoch_now_u64 and epoch_now_f64

- remove tm_editor (moved to proxmox crate)

- use new helpers from proxmox 0.3.8
  * epoch_i64 and epoch_f64
  * parse_rfc3339
  * epoch_to_rfc3339_utc
  * strftime_local

- BackupDir changes:
  * store epoch and rfc3339 string instead of DateTime
  * backup_time_to_string now return a Result
  * remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir

- DynamicIndexHeader: change ctime to i64

- FixedIndexHeader: change ctime to i64
2020-09-15 07:12:57 +02:00
Fabian Grünbichler e0e5b4426a BackupDir: make constructor fallible
since converting from i64 epoch timestamp to DateTime is not always
possible. previously, passing invalid backup-time from client to server
(or vice-versa) panicked the corresponding tokio task. now we get proper
error messages including the invalid timestamp.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:49:35 +02:00
Fabian Grünbichler 833eca6d2f use non-panicky timestamp_opt where appropriate
by either printing the original, out-of-range timestamp as-is, or
bailing with a proper error message instead of panicking.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:48:24 +02:00
Fabian Grünbichler 151acf5d96 don't truncate DateTime nanoseconds
where we don't care about them anyway..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:48:10 +02:00
Dietmar Maurer 139bcedc53 benchmark: update TLS reference speed
We are now faster with recent patches.
2020-09-10 12:55:43 +02:00
Hannes Laimer 61d7b5013c add benchmark flag to backup creation for proper cleanup when running a benchmark
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2020-09-10 08:25:24 +02:00
Dylan Whyte 871181d984 mount: fix mount subcommand
fixes the error, "manifest does not contain
file 'X.pxar'", that occurs when trying to mount
a pxar archive with 'proxmox-backup-client mount':

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-09-10 07:21:16 +02:00
Dominik Csapak 83088644da fix #2983: improve tcp performance
by leaving the buffer sizes on default, we get much better tcp performance
for high latency links

throughput is still impacted by latency, but much less so when
leaving the sizes at default.
the disadvantage is slightly higher memory usage of the server
(details below)

my local benchmarks (proxmox-backup-client benchmark):

pbs client:
PVE Host
Epyc 7351P (16core/32thread)
64GB Memory

pbs server:
VM on Host
1 Socket, 4 Cores (Host CPU type)
4GB Memory

average of 3 runs, rounded to MB/s
                    | no delay |     1ms |     5ms |     10ms |    25ms |
without this patch  |  230MB/s |  55MB/s |  13MB/s |    7MB/s |   3MB/s |
with this patch     |  293MB/s | 293MB/s | 249MB/s |  241MB/s | 104MB/s |

memory usage (resident memory) of proxmox-backup-proxy:

                    | peak during benchmarks | after benchmarks |
without this patch  |                  144MB |            100MB |
with this patch     |                  145MB |            130MB |

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-10 07:15:12 +02:00
Thomas Lamprecht abd4c4cb8c ui: add translation support
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-08 08:54:33 +02:00
Dominik Csapak 15ec790a40 tools/systemd/time: convert the resulting timestamp into an option
we want to use dates for the calendarspec, and with that there are some
impossible combinations that cannot be detected during parsing
(e.g. some datetimes do not exist in some timezones, and the timezone
can change after setting the schedule)

so finding no timestamp is not an error anymore but a valid result

we omit logging in that case (since it is not an error anymore)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:28:05 +02:00
Thomas Lamprecht 843880f008 bin/backup-proxy: assert that daemon runs as backup user/group
Because if not, the backups it creates have bogus permissions and may
seem like they got broken once the daemon is started again with the
correct user/group.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 10:30:15 +02:00
Thomas Lamprecht 74f94d0678 bin/backup-proxy: remove outdated perl comments
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 10:27:47 +02:00
Thomas Lamprecht 946c3e8a81 bin/backup-proxy: return error directly in main
anyhow makes this a nice error message, similar to the manual
wrapping used.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 10:27:47 +02:00
Thomas Lamprecht 1ffe030123 various typo fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-25 18:52:31 +02:00
Dylan Whyte 3435f5491b Fix typo in program output
Change "comptation" -> "computation"

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-08-19 09:06:27 +02:00
Dominik Csapak 93bb51fe7e config/jobstate: replace Job:load with create_state_file
it really is not necessary, since the only time we are interested in
loading the state from the file is when we list it, and there
we use JobState::load directly to avoid the lock

we still need to create the file on syncjob creation though, so
that we have the correct time for the schedule

to do this we add a new create_state_file that overwrites it on creation
of a syncjob

for safety, we subtract 30 seconds from the in-memory state in case
the statefile is missing

since we call create_state_file from  proxmox-backup-api,
we have to chown the lock file after creating to the backup user,
else the sync job scheduling cannot aquire the lock

also we remove the lock file on statefile removal

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 06:38:02 +02:00
Dominik Csapak 713b66b6ed cleanup: replace id from do_sync_job with info from job
we already have it inside the job itself

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 06:36:43 +02:00
Dominik Csapak a13573c24a syncjob: use do_sync_job also for scheduled sync jobs
and determine the last runtime with the jobstate

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:51:20 +02:00
Dominik Csapak e6263c2662 config: add JobState helper
this is intended to be a generic helper to (de)serialize job states
(e.g., sync, verify, and so on)

writes a json file into '/var/lib/proxmox-backup/jobstates/TYPE-ID.json'

the api creates the directory with the correct permissions, like
the rrd directory

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:36:10 +02:00
Wolfgang Bumiller 72dc68323c replace and remove old ticket functions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-12 14:28:21 +02:00
Fabian Grünbichler 14f6c9cb8b chunk readers: ensure chunk/index CryptMode matches
an encrypted Index should never reference a plain-text chunk, and an
unencrypted Index should never reference an encrypted chunk.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-11 09:54:22 +02:00
Wolfgang Bumiller e7cb4dc50d introduce Username, Realm and Userid api types
and begin splitting up types.rs as it has grown quite large
already

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 12:05:01 +02:00
Dylan Whyte 5f76ac37b5 fix: master-key: upload RSA encoded key with backup
When uploading an RSA encoded key alongside the backup,
the backup would fail with the error message: "wrong blob
file extension".
Adding the '.blob' extension to rsa-encrypted.key before the
the call to upload_blob_from_data(), rather than after, fixes
the issue.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-08-06 09:34:01 +02:00
Wolfgang Bumiller d9b8e2c795 pxar: better error handling on extract
Errors while applying metadata will not be considered fatal
by default using `pxar extract` unless `--strict` was passed
in which case it'll bail out immediately.

It'll still return an error exit status if something had
failed along the way.

Note that most other errors will still cause it to bail out
(eg. errors creating files, or I/O errors while writing
the contents).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-03 09:40:55 +02:00
Wolfgang Bumiller d44185c4a1 fix #2873: if --pattern is used, default to not extracting
The extraction algorithm has a state (bool) indicating
whether we're currently in a positive or negative match
which has always been initialized to true at the beginning,
but when the user provides a `--pattern` argument we need to
start out with a negative match.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-30 09:33:30 +02:00
Stefan Reiter c9756b40d1 datastore: prevent deletion of snaps in use as "previous backup"
To prevent a race with a background GC operation, do not allow deletion
of backups who's index might currently be referenced as the "known chunk
list" for successive backups. Otherwise the GC could delete chunks it
thinks are no longer referenced, while at the same time telling the
client that it doesn't need to upload said chunks because they already
exist.

Additionally, prevent deletion of whole backup groups, if there are
snapshots contained that appear to be currently in-progress. This is
currently unlikely to trigger, as that function is only used for sync
jobs, but it's a useful safeguard either way.

Deleting a single snapshot has a 'force' parameter, which is necessary
to allow deleting incomplete snapshots on an aborted backup. Pruning
also sets force=true to avoid the check, since it calculates which
snapshots to keep on its own.

To avoid code duplication, the is_finished method is factored out.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:26:01 +02:00
Stoiko Ivanov f2b4b4b9fe fix 2885: bail on duplicate backup target
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-07-24 11:08:56 +02:00
Dominik Csapak 01ca99da2d server/rest: add console to index
register the console template and render it when the 'console' parameter
is given

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 12:06:38 +02:00
Dominik Csapak 2ab5acac5a server/config: add mechanism to update template
instead of exposing handlebars itself, offer a register_template and
a render_template ourselves.

render_template checks if the template file was modified since
the last render and reloads it when necessary

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 11:55:00 +02:00
Thomas Lamprecht 9688f6de0f client: log index.json upload only when verbose
I mean the user expects that we know what archives, fidx or didx, are
in a backup, so this is internal info and should not be logged by
default

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 10:28:38 +02:00
Thomas Lamprecht 5b32820e93 client: don't use debug format for printing BackupRepository
It implements the fmt::Display  trait after all

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 10:28:38 +02:00
Dominik Csapak ec01eeadc6 refactor CertInfo to tools
we want to reuse some of the functionality elsewhere

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-10 11:08:53 +02:00
Wolfgang Bumiller 660a34892d update proxmox crate to 0.2.0
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 11:08:27 +02:00
Dietmar Maurer d86034afec src/bin/proxmox_backup_client/catalog.rs: fix keyfile handling 2020-07-10 10:36:45 +02:00
Wolfgang Bumiller 0eaef8eb84 client: show key path when creating/changing default key
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 09:58:24 +02:00
Thomas Lamprecht e39974afbf client: add simple version command
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-10 09:34:07 +02:00
Dietmar Maurer dde18bbb85 proxmox-backup-client benchmark: improve output format 2020-07-10 09:13:52 +02:00
Dietmar Maurer 4327a8462a proxmox-backup-client benchamrk: add more speed tests 2020-07-09 17:07:22 +02:00
Dietmar Maurer 323b2f3dd6 proxmox-backup-client benchmark: add --verbose flag 2020-07-09 16:16:39 +02:00
Dietmar Maurer dfa517ad6c src/backup/manifest.rs: rename into_string -> to_string
And do not consume self.
2020-07-09 11:28:05 +02:00
Dietmar Maurer b53f637914 src/backup/manifest.rs: cleanup signature generation 2020-07-09 09:20:49 +02:00
Dietmar Maurer 2107a5aebc src/backup/manifest.rs: include signature inside the manifest
This is more flexible, because we can choose what fileds we want to sign.
2020-07-08 16:23:26 +02:00
Dietmar Maurer 3638341aa4 src/backup/file_formats.rs: remove signed chunks
We can include signature in the manifest instead (patch will follow).
2020-07-08 16:23:26 +02:00
Wolfgang Bumiller 0351f23ba4 client: introduce --keyfd parameter
This is a more convenient way to pass along the key when
creating encrypted backups of unprivileged containers in PVE
where the unprivileged user namespace cannot access
`/etc/pve/priv`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 13:56:38 +02:00
Wolfgang Bumiller 05389a0109 more xdg cleanup and encryption parameter improvements
Have a single common function to get the BaseDirectories
instance and a wrapper for `find()` and `place()` which
wrap the error with some context.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 10:57:28 +02:00
Wolfgang Bumiller b65390ebc9 client: xdg usage: place() vs find()
place() is used when creating a file, as it will create
intermediate directories, only use it when actually placing
a new file.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 10:57:28 +02:00
Dietmar Maurer 3bad3e6e52 src/client/backup_writer.rs - upload_stream: add crypt_mode 2020-07-08 10:43:28 +02:00
Wolfgang Bumiller 24be37e3f6 client: fix schema to include --crypt-mode parameter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 10:09:15 +02:00
Wolfgang Bumiller af3a0ae7b1 remove CryptMode::sign_only special method
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 09:19:53 +02:00
Wolfgang Bumiller f28d9088ed introduce a CryptMode enum
This also replaces the recently introduced --encryption
parameter on the client with a --crypt-mode parameter.

This can be "none", "encrypt" or "sign-only".

Note that this introduces various changes in the API types
which previously did not take the above distinction into
account properly:

Both `BackupContent` and the manifest's `FileInfo`:
    lose `encryption: Option<bool>`
    gain `crypt_mode: Option<CryptMode>`

Within the backup manifest itself, the "crypt-mode" property
will always be set.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-07 15:24:19 +02:00
Wolfgang Bumiller 96ee857752 client: add --encryption boolen parameter
This can be used to explicitly disable encryption even if a
default key file exists in ~/.config.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
Wolfgang Bumiller 887018bb79 client: use default encryption key if it is available
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
Wolfgang Bumiller 9696f5193b client: move key management into separate module
and use api macro for methods and Kdf type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
Dietmar Maurer 9de69cdb1a src/bin/proxmox_backup_client/catalog.rs: split out catalog code 2020-07-03 16:45:47 +02:00
Dietmar Maurer e9764238df make ReadChunk not require mutable self.
That way we can reduce lock contentions because we lock for much shorter
times.
2020-07-03 07:37:29 +02:00
Dietmar Maurer cc7995ac40 src/bin/proxmox_backup_client/task.rs: split out task command 2020-07-02 18:04:29 +02:00
Dietmar Maurer 43abba4b4f src/bin/proxmox_backup_client/mount.rs: split out mount code 2020-07-02 17:49:59 +02:00
Dietmar Maurer caea8d611f proxmox-backup-client: add benchmark command
This is just a start, We need to add more useful things here...
2020-07-02 14:01:57 +02:00
Dietmar Maurer b957aa81bd update backup api for incremental backup
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-26 07:17:08 +02:00
Dietmar Maurer 355c055e81 src/bin/proxmox-backup-manager.rs: implement verify 2020-06-24 13:35:21 +02:00
Wolfgang Bumiller a6f8728339 update to pxar 0.1.9, update ReadAt implementations
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-24 11:57:12 +02:00
Dietmar Maurer d6d3b353be cleanup: implement FromStr for BackupGroup 2020-06-23 08:16:56 +02:00
Dietmar Maurer a67f7d0a07 cleanup: implement FromStr for BackupDir 2020-06-23 08:09:52 +02:00
Dietmar Maurer c8137518fe src/bin/proxmox_backup_manager/disk.rs: add renderer for wearout
So that we display the same value as the GUI.
2020-06-23 07:44:09 +02:00
Dominik Csapak 1c090810f5 api2/admin/datastore/snapshos: show encrypted and size info per file
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:39:56 +02:00
Dominik Csapak e181d2f6da add encrypted info to Manifest
we want to save if a file of a backup is encrypted, so that we can
* show that info on the gui
* can later decide if we need to decrypt the backup

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:35:39 +02:00
Dietmar Maurer 1aef491e24 src/bin/proxmox_backup_manager/disk.rs: add cli to create mounted disks 2020-06-17 08:07:54 +02:00
Wolfgang Bumiller 8ecd7c9c21 move api dump binaries back to src/bin for package building
they're required for docs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 14:48:01 +02:00
Wolfgang Bumiller d19c96d507 move test binaries to examples/
These aren't installed and are only used for manual testing,
so there's no reason to force them to be built all the time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 13:32:24 +02:00
Dietmar Maurer 929a13b357 src/api2/node/disks/zfs.rs: add zpool api 2020-06-16 13:25:53 +02:00
Dietmar Maurer ac7513e368 src/tools.rs: add setup_safe_path_env() 2020-06-15 10:38:30 +02:00
Wolfgang Bumiller 3d571d5509 some internal combinator-influenced api cleanup
The download methods used to take the destination by value
and return them again, since this was required when using
combinators before we had `async fn`.
But this is just an ugly left-over now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:46:42 +02:00
Wolfgang Bumiller 8e6e18b77c client: make dump_image async, use async chunk reader
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:40:18 +02:00
Wolfgang Bumiller 4d16badf6f add an AsyncReadChunk trait
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:38:21 +02:00
Dominik Csapak 5d85847f91 client: only start catalog upload if we have one
else we start a dynamic writer and never close it, leading to a backup error

this fixes an issue with backing up vm templates
(and possibly vms without disks)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 10:38:25 +02:00
Dominik Csapak e693818afc refactor time functions to tools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:10 +02:00
Wolfgang Bumiller 5444fa940b turn pxar::flags into bitflags, pxar::Flags
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 11:05:53 +02:00
Wolfgang Bumiller fc6047fcb1 pxar: don't skip list+found by default
This used to be default-off and was accidentally set to
on-by-default with the pxar crate update.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 08:53:10 +02:00
Wolfgang Bumiller 239e49f927 pxar: create .pxarexclude-cli file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-09 13:17:59 +02:00
Dominik Csapak 33070956af let disk_usage return StorageStatus and use it for datastores/nodes
disk_usage returned the same values as defined in StorageStatus,
so simply use that

with that we can replace the logic of the datastore status with that
function and also use it for root disk usage of the nodes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:05:39 +02:00
Wolfgang Bumiller c443f58b09 switch to external pxar and fuse crates
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 13:56:58 +02:00
Dietmar Maurer dfb31de8f0 proxmox_backup_manager disk list: display gpt column 2020-06-08 07:35:44 +02:00
Dietmar Maurer 707974fdb3 src/api2/node/disks.rs: implement initgpt API 2020-06-07 10:30:34 +02:00
Dietmar Maurer 9069debcd8 src/api2/types.rs: define BLOCKDEVICE_NAME_SCHEMA 2020-06-07 07:20:25 +02:00
Dietmar Maurer 8e40aa63c1 src/bin/proxmox-backup-manager.rs: add disk subcommand 2020-06-06 15:40:28 +02:00
Thomas Lamprecht a720894ff0 rrd: fix off-by-one in save interval calculation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-04 10:30:47 +02:00
Dietmar Maurer 5c264c8d80 src/tools/disks.rs: add/use get_partition_type_info 2020-06-04 07:48:22 +02:00
Dietmar Maurer c48aa39f3b src/bin/proxmox-backup-client.rs: implement quite flag 2020-06-03 10:11:37 +02:00
Thomas Lamprecht 2d32fe2c04 client restore: don't add server file ending if already specified
If one executes a client command like
 # proxmox-backup-client files <snapshot> --repository ...
the files shown have already the '.fidx' or '.blob' file ending, so
if a user would just copy paste that one the client would always add
.blob, and the server would not find that file.

So avoid adding file endings if it is already a known OK one.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-03 07:03:55 +02:00
Thomas Lamprecht dc155e9bd7 client restore: factor out archive/type parsing
will be extended in a next patch.

Also drop a dead else branch, can never get hit as we always add
.blob as fallback

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-03 07:03:12 +02:00
Thomas Lamprecht add5861e8d typo fixes all over the place
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-30 16:39:08 +02:00
Dietmar Maurer b2387eaa45 avoid compiler warnings 2020-05-30 14:05:33 +02:00
Dietmar Maurer 7cc3473a4e src/client/backup_specification.rs: split code into extra file 2020-05-30 10:54:38 +02:00
Dominik Csapak c67b1fa72f syncjob: change worker type for sync jobs
'sync' is used for manually pulling a remote datastore
changing it for a scheduled sync to 'syncjob' so that we can
differentiate between both types of syncs

this also adds a seperate task description for it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:28:04 +02:00
Dietmar Maurer 934f5bb8ac src/bin/proxmox-backup-proxy.rs: cleanup, move code to src/tools/disks.rs
And simplify find_mounted_device by using stat.st_dev
2020-05-29 11:13:36 +02:00
Dietmar Maurer 013fa7bbcb rrd: reduce io by saving data only once a minute 2020-05-29 09:16:13 +02:00
Dietmar Maurer a8d7033cb2 src/bin/proxmox-backup-proxy.rs: add test if last prune job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer 04ad7bc436 src/bin/proxmox-backup-proxy.rs: test if last sync job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer dc58194ebe src/bin/proxmox-backup-proxy.rs: use correct id to lookup sync jobs 2020-05-29 07:50:59 +02:00
Dietmar Maurer c94e1f655e rrd stats: improve io delay stats 2020-05-28 19:12:13 +02:00
Dietmar Maurer 91e5bb49f5 src/bin/proxmox-backup-proxy.rs: simplify code
and gather all stats for the root disk
2020-05-28 12:30:54 +02:00
Dietmar Maurer 25c550bc28 src/bin/proxmox-backup-proxy.rs: gather zpool io stats 2020-05-28 10:09:13 +02:00
Dominik Csapak f5056656b2 use the sync id for the scheduled sync worker task
this way, multiple sync jobs with the same local store, can get scheduled

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-28 06:26:03 +02:00
Dominik Csapak 8c87743642 fix 'remove_vanished' cli arg again
since the target side wants this to be a boolean and
serde interprets a None Value as 'null' we have to only
add this when it is really set via cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-28 06:25:30 +02:00
Dominik Csapak 40dc103103 fix cli pull api call
there is no 'delete' parameter, only 'remove-vanished', so fix that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 12:39:19 +02:00
Dietmar Maurer 8c03041a2c src/bin/proxmox-backup-proxy.rs: gather block device stats on datastore 2020-05-26 11:20:59 +02:00
Dietmar Maurer 309ef20d6d src/bin/proxmox-backup-proxy.rs: simplify code 2020-05-25 16:20:32 +02:00
Dietmar Maurer d0833a70f7 src/bin/proxmox-backup-proxy.rs: gather datastore usage stats 2020-05-25 16:20:32 +02:00
Dietmar Maurer 2c66a590c0 src/bin/proxmox-backup-proxy.rs: gather iowait stats 2020-05-25 11:54:15 +02:00
Dietmar Maurer 485841da2c src/bin/proxmox-backup-proxy.rs: gather loadavg stats 2020-05-25 11:40:20 +02:00
Dietmar Maurer dd15c0aa3b src/bin/proxmox-backup-proxy.rs: gather root disk stats 2020-05-25 11:10:07 +02:00
Dietmar Maurer c25c9d8dd1 src/bin/proxmox-backup-proxy.rs: gather swap usage stats 2020-05-25 10:25:58 +02:00
Dietmar Maurer 4f9513996c src/bin/proxmox-backup-proxy.rs: use block_in_place for rrd update 2020-05-25 08:30:59 +02:00
Dietmar Maurer 8f0cec2642 src/bin/proxmox-backup-proxy.rs: gather netin/netout stats 2020-05-24 19:02:35 +02:00
Dietmar Maurer 4fb05fde17 src/rrd/rrd.rs: restructure whole code 2020-05-24 16:51:28 +02:00
Dietmar Maurer a4a3f7ca5e rrd: pack multiple rrd values into th estat list 2020-05-23 14:03:44 +02:00
Dietmar Maurer eaeda365e0 start gathering stats using new rrd module 2020-05-23 10:43:08 +02:00
Dietmar Maurer 07ad6470ca src/client/pull.rs: split out pull related code 2020-05-22 08:04:20 +02:00
Dietmar Maurer a6160cdfeb src/bin/proxmox-backup-proxy.rs: schedule sync jobs 2020-05-22 07:50:59 +02:00
Dietmar Maurer a3016d6583 proxmox-backup-manager: add sync-job cli 2020-05-21 11:44:45 +02:00
Dietmar Maurer b29d046e89 proxmox-backup-manager: split out cert.rs 2020-05-21 11:22:20 +02:00
Dietmar Maurer 380bd7df97 proxmox-backup-manager: split out datastore.rs 2020-05-21 11:14:34 +02:00
Dietmar Maurer ea6f404e55 proxmox-backup-manager: split out dns.rs 2020-05-21 11:10:58 +02:00
Dietmar Maurer a35a211d9e proxmox-backup-manager: split out network.rs 2020-05-21 11:08:38 +02:00
Dietmar Maurer 53e14507c1 proxmox-backup-manager: split out acl.rs 2020-05-21 10:56:46 +02:00
Dietmar Maurer 6fa39e53e0 proxmox-backup-manager: split out users.rs 2020-05-21 10:53:06 +02:00
Dietmar Maurer a220a4564a roxmox-backup-manager: start splitting command into several files 2020-05-21 10:46:07 +02:00
Dietmar Maurer c681885227 src/bin/proxmox-backup-manager.rs: format output of show commands 2020-05-20 16:47:37 +02:00
Dominik Csapak ce55db66d6 proxmox-backup-manager: add show command for remote and datastore
to show the data for a single item

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 16:33:07 +02:00
Dietmar Maurer 25829a879b src/bin/proxmox-backup-proxy.rs: schedule prune jobs 2020-05-20 13:00:53 +02:00
Dietmar Maurer dd7a7eae8f src/bin/proxmox-backup-manager.rs: add completion helper for gc-schedule 2020-05-20 09:42:51 +02:00
Dietmar Maurer 8545480a31 src/bin/proxmox-backup-proxy.rs: add simple task scheduler for garbage collection 2020-05-20 08:59:45 +02:00
Dietmar Maurer e8d1da6a15 depend on proxmox 0.1.31 - use Value to store result metadata 2020-05-18 09:57:35 +02:00
Dietmar Maurer 65dab0266c proxmox-backup-manager: add completion helper for port list 2020-05-08 17:28:04 +02:00
Dietmar Maurer 525008f7ad proxmox-backup-manager - network list: render ports/slaves
And render interface name as first comumn.
2020-05-08 16:07:23 +02:00
Dietmar Maurer 86a5d56c4e proxmox-backup-manager: add network create command 2020-05-08 09:55:56 +02:00
Dietmar Maurer 7b22acd0c2 src/config/network.rs: make it compatible with pve
and depend on proxmox 0.1.26
2020-05-07 09:28:25 +02:00
Dietmar Maurer 74f7240b8d src/bin/proxmox-backup-client.rs: add human readable date to prune list 2020-05-05 07:33:58 +02:00
Dietmar Maurer db1e061dcb src/bin/proxmox-backup-client.rs: correctly format prune result list. 2020-05-05 06:45:37 +02:00
Dietmar Maurer b080583ba8 src/bin/proxmox-backup-manager.rs: improve user list output 2020-05-01 16:22:50 +02:00
Dominik Csapak f9e3b1104e change index to templates using handlebars
using a handlebars instance in ApiConfig, to cache the templates
as long as possible, this is currently ok, as the index template
can only change when the whole package changes

if we split this in the future, we have to trigger a reload of
the daemon on gui package upgrade (so that the template gets reloaded)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-04-29 17:05:53 +02:00
Dietmar Maurer bc0d03885c use proxmox 0.1.25, use new EnumEntry feature 2020-04-29 13:01:24 +02:00
Dietmar Maurer 14627d671a src/bin/proxmox-backup-manager.rs: add dns sub command
Also improved the DNS api, added a --delete option.
2020-04-26 08:23:23 +02:00
Dietmar Maurer 76227a6acd src/bin/proxmox-backup-manager.rs: fix node parameter handling 2020-04-25 17:20:22 +02:00
Dietmar Maurer 26d9aebc28 move src/api2/config/network.rs to src/api2/node/network.rs
So that we have the same api path for network config as pve.
2020-04-25 17:00:38 +02:00
Dietmar Maurer fd7c0979b4 src/bin/proxmox-backup-manager.rs: implement netwerk revert 2020-04-24 10:45:49 +02:00
Dietmar Maurer c67bc9c35c src/bin/proxmox-backup-manager.rs: new command to show pending network changes 2020-04-24 10:27:43 +02:00
Dietmar Maurer 3181f9b625 src/bin/proxmox-backup-manager.rs: only show pending changes with "text" format 2020-04-24 10:16:57 +02:00
Dietmar Maurer 2eefd9aee1 src/config/network.rs: implement network reload, set "changes" attribute 2020-04-24 09:55:46 +02:00
Dietmar Maurer 02269f3dba src/config/network.rs: introduce NetworkInterfaceType 2020-04-23 08:45:03 +02:00
Dietmar Maurer 4cb6bd894c src/bin/proxmox-backup-manager.rs: improve network list output format 2020-04-23 06:44:55 +02:00
Dietmar Maurer b1564af25a src/bin/proxmox-backup-manager.rs: format datastore list output 2020-04-22 17:37:20 +02:00
Dietmar Maurer bf004ecd87 src/bin/proxmox-backup-manager.rs: format network list output 2020-04-22 17:14:52 +02:00
Dietmar Maurer df6bb03d0e src/api2/config/network.rs: improve network api 2020-04-22 10:54:07 +02:00
Dietmar Maurer ca0e534796 src/api2/config/network.rs: start network configuration api 2020-04-21 14:28:26 +02:00
Wolfgang Bumiller f7d4e4b506 switch from failure to anyhow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Dietmar Maurer 9a328319dd pxar extract: remove pattern from arg_param, add target instead 2020-04-15 06:41:37 +02:00
Dietmar Maurer 7e3d2e5b41 pxar create: remove exclude from arg_param 2020-04-15 06:31:46 +02:00
Dietmar Maurer 9765092ede acl api: implement update 2020-04-14 10:16:49 +02:00
Dietmar Maurer ed3e60ae69 start ACL api 2020-04-13 11:09:44 +02:00
Dietmar Maurer 685e13347e api: move config/user to access/users, implement change_password
To make it similar to the pve api
2020-04-09 10:21:24 +02:00
Dietmar Maurer 579728c641 add user configiguration 2020-04-08 14:06:15 +02:00
Dietmar Maurer 18deda403d src/bin/proxmox-backup-client.rs - list_groups: render full snapshot path for last-backup column 2020-02-28 09:09:22 +01:00
Dietmar Maurer 6882b3716c src/bin/proxmox-backup-manager.rs: avoid compiler warning 2020-02-28 08:52:12 +01:00
Christian Ebner 189996cf4a proxmox-backup-client: expose exclude match patterns to cli.
Allows to pass exclude match patterns to the 'backup' command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 07:56:09 +01:00
Dietmar Maurer ac3faaf5c0 src/bin/proxmox-backup-manager.rs: cleanup output handling 2020-02-28 07:42:36 +01:00
Dietmar Maurer 4939255fb4 src/tools/format.rs: move output rendering code to this new file 2020-02-28 07:30:35 +01:00
Dietmar Maurer c81b2b7c70 src/bin/proxmox-backup-client.rs - list_groups: use format_and_print_result_full()
Depend on proxmox 0.1.16-1
2020-02-27 13:32:11 +01:00
Dietmar Maurer c2043614f7 src/bin/proxmox-backup-client.rs: use new output formath helpers from proxmox 0.15.1-1 2020-02-27 12:41:15 +01:00
Dietmar Maurer e23f586344 src/bin/proxmox-backup-client.rs - status: improve alignment 2020-02-27 11:38:09 +01:00
Dietmar Maurer be2425ff85 src/bin/proxmox-backup-client.rs - status: do not print header 2020-02-27 11:31:05 +01:00
Dietmar Maurer f24fc1166b src/bin/proxmox-backup-client.rs - list_snapshots: use format_and_print_result_full()
Depend on proxmox v0.1.14.
2020-02-27 11:28:44 +01:00
Dietmar Maurer ea5f547fdc src/bin/proxmox-backup-client.rs - list_snapshot_files: use format_and_print_result_full() 2020-02-26 13:49:47 +01:00
Dietmar Maurer 390c5bdde4 src/bin/proxmox-backup-client.rs - status: use format_and_print_result_full 2020-02-26 13:29:00 +01:00
Dietmar Maurer 03ac286c80 src/bin/proxmox-backup-manager.rs: use crate::tools::runtime::block_on() 2020-02-26 11:37:38 +01:00
Dietmar Maurer e02c3d461f src/client/backup_writer.rs: implement verbose options 2020-02-24 12:48:40 +01:00
Dietmar Maurer d1c657276a src/client/http_client.rs: remove useless password_env 2020-02-11 11:42:03 +01:00
Dietmar Maurer a05c0c6ff6 src/bin/proxmox-backup-client.rs: allows to pass fingerprint via env vars 2020-02-11 11:42:03 +01:00
Thomas Lamprecht b69b8af26b s/pbsbanner/proxmox-backup-banner/ + move to /usr/lib
We want to avoid pbs if possible and also avoid placing internal
binaries, not intended for human direct use, in /bin or /sbin paths.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-10 14:43:28 +01:00
Thomas Lamprecht e2b5e75a3e pbsbanner: use match and also return error
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-07 18:30:55 +01:00
Thomas Lamprecht 274b0c7bb7 add login banner service
Modeled after the one from PVE, but using rust instead of perl for
resolving the nodename and writing to /etc/issue

Behavior differs a bit. We write all non-loopback addresses to this
file, as the gui accepts connections from them all, so limiting it to
the first one is not really sensible.
Further an error to resolve, or only getting loopback addresses won't
write out an empty /etc/issue file, but a note about the error at the
place where the address would be displayed.

Named it "pbsbanner", not "proxmox-backup-banner" as it's rather an
internal tool anyway and mirrors pvebanner, pmgbanner

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-06 22:19:22 +01:00
Dietmar Maurer 032d3ad80f src/bin/proxmox-backup-client.rs: do not use api macro for prune
So that we can use add_common_prune_prameters!() macro again.
2020-01-31 10:31:36 +01:00
Dietmar Maurer 501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Christian Ebner 74616f63b0 proxmox-backup-client: add missing prune options to api definition.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-31 07:01:56 +01:00
Dietmar Maurer 93fbb4ef0a update proxmox dependency to 0.1.12 2020-01-31 06:59:17 +01:00
Dietmar Maurer 43592f290d src/bin/proxmox-backup-manager.rs: remove stale debug comment 2020-01-30 17:59:13 +01:00