Commit Graph

3784 Commits

Author SHA1 Message Date
Dominik Csapak 38d4675921 fix ipv6 handling for remotes/sync jobs
* add square brackets to ipv6 adresses in BackupRepository if they not
already have some (we save them without in the remote config)

* in get_pull_parameters, we now create a BackupRepository first and use
  those values (which does the [] mapping), this also has the advantage
  that we have one place less were we hardcode 8007 as port

* in the ui, add square brackets for ipv6 adresses for remotes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 13:40:03 +02:00
Dietmar Maurer 7b8aa893fa src/client/pull.rs: log progress 2020-09-30 13:35:09 +02:00
Dominik Csapak df4827f2c0 tasks: improve behaviour on upgrade
when upgrading from a version where we stored all tasks in the 'active' file,
we did not completly account for finished tasks still there

we should update the file when encountering any finished task in
'active' as well as filter them out on the api call (if they get through)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 11:05:50 +02:00
Dietmar Maurer ef1b436350 paperkey: add html output 2020-09-30 10:49:20 +02:00
Dietmar Maurer e64b9f9204 src/tools.rs: make command_output return Vec<u8>
And add a new helper to return output as string.
2020-09-30 10:49:20 +02:00
Dominik Csapak ba20987ae7 client/remote: add support to specify port number
this adds the ability to add port numbers in the backup repo spec
as well as remotes, so that user that are behind a
NAT/Firewall/Reverse proxy can still use it

also adds some explanation and examples to the docs to make it clearer
for h2 client i left the localhost:8007 part, since it is not
configurable where we bind to

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 10:49:20 +02:00
Thomas Lamprecht 729d41fe6a api: disks/zfs: check template exsits before enabling zfs-import service
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-30 09:34:21 +02:00
Stoiko Ivanov 905147a5ee api2/node/disks/zfs: instantiate import service
When creating a new zpool for a datastore, also instantiate an
import-unit for it. This helps in cases where '/etc/zfs/zool.cache'
get corrupted and thus the pool is not imported upon boot.

This patch needs the corresponding addition of 'zfs-import@.service' in
the zfsonlinux repository.

Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-09-30 08:43:38 +02:00
Dominik Csapak 60b9b48e71 require square brackets for ipv6 addresses
we need this, because we append the port to this to get a target url
e.g. we print

format!("https://{}:8007/", address)

if address is now an ipv6 (e.g. fe80::1) it would become

https://fe80::1:8007/ which is a valid ipv6 on its own

by using square brackets we get:

https://[fe80::1]:8007/ which now connects to the correct ip/port

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:16:27 +02:00
Dominik Csapak 7eebe1483e server/worker_task: fix panic on slice range when index is empty
since len() and MAX_INDEX_TASKS are both usize, they underflow
instead of getting negative values

instead check the sizes and set them accordingly

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:11:06 +02:00
Dominik Csapak 9a76091785 proxmox-backup-proxy: add task archive rotation
this starts a task once a day at "00:00" that rotates the task log
archive if it is bigger than 500k

if we want, we can make the schedule/size limit/etc. configurable,
but for now it's ok to set fixed values for that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:41:18 +02:00
Dominik Csapak c386b06fc6 server/worker_task: remove unecessary read_task_list
since there are no users of this anymore and we now have a nicer
TaskListInfoIterator to use, we can drop this function

this also means that 'update_active_workers' does not need to return
a list anymore since we never used that result besides in
read_task_list

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:50 +02:00
Dominik Csapak 6bcfc5c1a4 api2/status: use the TaskListInfoIterator here
this means that limiting with epoch now works correctly
also change the api type to i64, since that is what the starttime is
saved as

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:24 +02:00
Dominik Csapak 768e10d0b3 api2/node/tasks: use TaskListInfoIterator instead of read_task_list
this makes the filtering/limiting much nicer and readable

since we now have potentially an 'infinite' amount of tasks we iterate over,
and cannot now beforehand how many there are, we return the total count
as always 1 higher then requested iff we are not at the end (this is
the case when the amount of entries is smaller than the requested limit)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:02 +02:00
Dominik Csapak e7244387c7 server/worker_task: add TaskListInfoIterator
this is an iterator that reads/parses/updates the task list as
necessary and returns the tasks in descending order (newest first)

it does this by using our logrotate iterator and using a vecdeque

we can use this to iterate over all tasks, even if they are in the
archive and even if the archive is logrotated but only read
as much as we need

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:39:16 +02:00
Dominik Csapak 5ade6c25f3 server/worker_task: write older tasks into archive file
instead of removing tasks beyond the 1000 that are in the index
write them into an archive file by appending them at the end
this way we can later still read them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:38:44 +02:00
Dominik Csapak 784fa1c2e3 server/worker_task: split task list file into two
one for only the active tasks and one for up to 1000 finished tasks

factor out the parsing of a task file (we will later need this again)
and use iterator combinators for easier code

we now sort the tasks ascending (this will become important in a later patch)
but reverse (for now) it to keep compatibility

this code also omits the converting into an intermittent hash
since it cannot really happen that we have duplicate tasks in this list
(since the call is locked by an flock, and it is the only place where we
write into the lists)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:38:28 +02:00
Dominik Csapak 66f4e6a809 server/worker_task: refactor locking of the task list
also add the functionality of having a 'shared' (read) lock for the list
we will need this later

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:37:54 +02:00
Dominik Csapak 8074d2b0c3 tools: add logrotate module
this is a helper to rotate and iterate over log files
there is an iterator for open filehandles as well as
only the filename

also it has the possibilty to rotate them
for compression, zstd is used

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:33:21 +02:00
Dietmar Maurer b02d49ab26 proxmox_backup_client key: allow to generate paperkey for master key 2020-09-29 08:29:42 +02:00
Dietmar Maurer 82a0cd2ad4 proxmox_backup_client key: add new paper-key command 2020-09-29 08:29:42 +02:00
Wolfgang Bumiller ee1a9c3230 parallel_handler: clippy: 'while_let_loop'
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-29 08:13:51 +02:00
Wolfgang Bumiller db24c01106 parallel_handler: explicit Arc::clone
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-28 13:40:03 +02:00
Wolfgang Bumiller ae3cfa8f0d parallel_handler: formatting cleanup, doc comment typo fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-28 13:40:03 +02:00
Dietmar Maurer b56c111e93 depend on proxmox 0.4.2 2020-09-28 10:50:44 +02:00
Dominik Csapak bbeb0256f1 server/worker_task: factor out task list rendering
we will need this later again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-28 07:31:27 +02:00
Dominik Csapak 005a5b9677 api2/node/tasks: move userfilter to function signature
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-28 07:18:13 +02:00
Dietmar Maurer 55bee04856 src/tools/parallel_handler.rs: remove unnecessary Sync bound 2020-09-26 16:16:11 +02:00
Dietmar Maurer 42fd40a124 src/bin/proxmox_backup_client/benchmark.rs: avoid compiler warning 2020-09-26 16:13:19 +02:00
Dietmar Maurer f21508b9e1 src/backup/verify.rs: use ParallelHandler to verify chunks 2020-09-26 11:14:37 +02:00
Dietmar Maurer ee7a308de4 src/backup/verify.rs: cleanup use clause 2020-09-26 10:23:44 +02:00
Dietmar Maurer 636e674ee7 src/client/pull.rs: simplify code 2020-09-26 10:09:51 +02:00
Dietmar Maurer b02b374b46 src/tools/parallel_handler.rs: remove static lifetime bound from handler_fn 2020-09-26 09:26:06 +02:00
Dietmar Maurer 1c13afa8f9 src/tools/parallel_handler.rs: join all threads in drop handler 2020-09-26 08:47:56 +02:00
Dietmar Maurer 69b92fab7e src/tools/parallel_handler.rs: remove unnecessary Sync trait bound 2020-09-26 07:38:44 +02:00
Dietmar Maurer 5441708634 src/client/pull.rs: use new ParallelHandler 2020-09-25 12:58:20 +02:00
Dietmar Maurer 3c9b370255 src/tools/parallel_handler.rs: execute closure inside a thread pool 2020-09-25 12:58:20 +02:00
Dominik Csapak d7a122a026 use jobstate mechanism for verify/garbage_collection schedules
also changes:
* correct comment about reset (replace 'sync' with 'action')
* check schedule change correctly (only when it is actually changed)

with this changes, we can drop the 'lookup_last_worker' method

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:12 +02:00
Dominik Csapak 6c25588e63 proxy: fix error handling in prune scheduling
we rely on the jobstate handling to write the error of the worker
into its state file, but we used '?' here in a block which does not
return the error to the block, but to the function/closure instead

so if a prune job failed because of such an '?', we did not write
into the statefile and got a wrong state there

instead use our try_block! macro that wraps the code in a closure

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-24 17:06:09 +02:00
Dietmar Maurer 998db63933 src/client/pull.rs: decode, verify and write in a separate threads
To maximize throughput.
2020-09-24 13:12:04 +02:00
Dietmar Maurer c0fa14d94a src/backup/data_blob.rs: add is_encrypted helper 2020-09-24 13:00:16 +02:00
Wolfgang Bumiller 6fd129844d remove DummyCatalogWriter
we're using an `Option` instead now

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-24 09:13:54 +02:00
Dietmar Maurer baae780c99 benchmark: use compressable data to get more realistic result
And add a benchmatrk to test chunk verify speed (decompress+sha256).
2020-09-24 08:58:13 +02:00
Dietmar Maurer 09a1da25ed src/backup/data_blob.rs: improve decompress speed 2020-09-24 08:52:35 +02:00
Dietmar Maurer a83e2ffeab src/api2/reader.rs: use std::fs::read instead of tokio::fs::read
Because it is about 10%& faster this way.
2020-09-22 13:27:23 +02:00
Dietmar Maurer ebbe4958c6 src/client/pull.rs: avoid duplicate downloads using in memory HashSet 2020-09-22 12:34:06 +02:00
Dietmar Maurer 73b2cc4977 src/client/pull.rs: allow up to 20 concurrent download streams 2020-09-22 11:39:31 +02:00
Dietmar Maurer 7ecfde8150 remote_chunk_reader.rs: use Arc for cache_hint to make clone faster 2020-09-22 11:39:31 +02:00
Dietmar Maurer e0085e6612 src/client/pull.rs: remove temporary manifest 2020-09-21 14:03:01 +02:00
Dietmar Maurer 194da6f867 src/client/pull.rs: open temporary manifest with truncate(true)
To delete any data if the file already exists.
2020-09-21 13:53:35 +02:00
Dominik Csapak 5e39918fe1 fix #3017: check array boundaries before using
else we panic here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-21 09:22:06 +02:00
Dietmar Maurer 12c65bacf1 src/backup/chunk_store.rs: disable debug output 2020-09-19 15:26:21 +02:00
Dietmar Maurer ba37f3562d src/backup/datastore.rs - open_with_path: use Path instead of str 2020-09-19 10:01:57 +02:00
Dietmar Maurer fce4659388 src/backup/datastore.rs: new method open_with_path
To make testing easier.
2020-09-19 09:55:21 +02:00
Dominik Csapak 9866de5e3d datastore/prune schedules: use JobState for tracking of schedules
like the sync jobs, so that if an admin configures a schedule it
really starts the next time that time is reached not immediately

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-19 06:24:37 +02:00
Fabian Grünbichler be3bd0f90b fix #3015: allow user self-service
listing, updating or deleting a user is now possible for the user
itself, in addition to higher-privileged users that have appropriate
privileges on '/access/users'.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 15:45:11 +02:00
Fabian Grünbichler 3c053adbb5 role api: fix description
wrongly copy-pasted at some point

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 14:55:00 +02:00
Hannes Laimer c040ec22f7 add verification scheduling to proxmox-backup-proxy
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:14:05 +02:00
Hannes Laimer 2b67de2e3f api2: make verify_schedule deletable
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:12:29 +02:00
Hannes Laimer 477859662a api2: add optional verify-schdule field to create/update datastore endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:12:16 +02:00
Hannes Laimer ccd7241e2f add verify_schedule field to DataStoreConfig
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:11:55 +02:00
Hannes Laimer f37ef25bdd api2: add VERIFY_SCHEDULE_SCHEMA
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:11:39 +02:00
Fabian Grünbichler b93bbab454 fix #3014: allow DataStoreAdmins to list DS config
filtered by those they are privileged enough to read individually. this
allows such users to configure prune/GC schedules via the GUI (the API
already allowed it previously).

permission-wise, a user with this privilege can already:
- list all stores they have access to (returns just name/comment)
- read the config of each store they have access to individually
(returns full config of that datastore + digest of whole config)

but combines them to
- read configs of all datastores they have access to (returns full
config of those datastores + digest of whole config)

user that have AUDIT on just /datastore without propagate can now no
longer read all configurations (but this could be added it back, it just
seems to make little sense to me).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 12:09:13 +02:00
Dietmar Maurer 1bc1d81a00 move compute_file_csum to src/tools.rs 2020-09-17 10:27:04 +02:00
Dominik Csapak 8f2f3dd710 fix #2942: implement lacp bond mode and bond_xmit_hash_policy
this was not yet implemented, should be compatible with pve and the gui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 08:36:25 +02:00
Dominik Csapak 85959a99ea api2/network: add bond-primary parameter
needed for 'active-backup' bond mode

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 08:36:14 +02:00
Dominik Csapak 36700a0a87 api2/pull: make pull worker abortable
by selecting between the pull_future and the abort future

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 06:11:33 +02:00
Dominik Csapak dd4b42bac1 fix #2870: renew tickets in HttpClient
by packing the auth into a RwLock and starting a background
future that renews the ticket every 15 minutes

we still use the BroadcastFuture for the first ticket and only
if that is finished we start the scheduled future

we have to store an abort handle for the renewal future and abort it when
the http client is dropped, so we do not request new tickets forever

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 06:09:54 +02:00
Fabian Grünbichler 9626c28619 always allow retrieving (censored) subscription info
like we do for PVE. this is visible on the dashboard, and caused 403 on
each update which bothers me when looking at the dev console.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-17 06:03:25 +02:00
Fabian Grünbichler 463c03462a fix #2957: allow Sys.Audit access to node RRD
this is the same privilege needed to query the node status.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-17 06:03:25 +02:00
Dietmar Maurer 4d431383d3 src/backup/data_blob.rs: expose verify_crc again 2020-09-16 10:43:42 +02:00
Stefan Reiter d10332a15d SnapshotVerifyState: use enum for state
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 13:06:04 +02:00
Stefan Reiter 43772efc6e backup: check all referenced chunks actually exist
A client can omit uploading chunks in the "known_chunks" list, those
then also won't be written on the server side. Check all those chunks
mentioned in the index but not uploaded for existance and report an
error if they don't exist instead of marking a potentially broken backup
as "successful".

This is only important if the base snapshot references corrupted chunks,
but has not been negatively verified. Also, it is important to only
verify this at the end, *after* all index writers are closed, since only
then can it be guaranteed that no GC will sweep referenced chunks away.

If a chunk is found missing, also mark the previous backup with a
verification failure, since we know the missing chunk has to referenced
in it (only way it could have been inserted to known_chunks with
checked=false). This has the benefit of automatically doing a
full-upload backup if the user attempts to retry after seeing the new
error, instead of requiring a manual verify or forget.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 10:00:05 +02:00
Stefan Reiter 0af2da0437 backup: check verify state of previous backup before allowing reuse
Do not allow clients to reuse chunks from the previous backup if it has
a failed validation result. This would result in a new "successful"
backup that potentially references broken chunks.

If the previous backup has not been verified, assume it is fine and
continue on.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 09:59:29 +02:00
Dietmar Maurer d09db6c2e9 rename BackupDir::new_with_group to BackupDir::with_group 2020-09-15 09:40:03 +02:00
Dietmar Maurer bc871bd19d src/backup/backup_info.rs: new BackupDir::with_rfc3339 2020-09-15 09:34:46 +02:00
Dietmar Maurer 6a7be83efe avoid chrono dependency, depend on proxmox 0.3.8
- remove chrono dependency

- depend on proxmox 0.3.8

- remove epoch_now, epoch_now_u64 and epoch_now_f64

- remove tm_editor (moved to proxmox crate)

- use new helpers from proxmox 0.3.8
  * epoch_i64 and epoch_f64
  * parse_rfc3339
  * epoch_to_rfc3339_utc
  * strftime_local

- BackupDir changes:
  * store epoch and rfc3339 string instead of DateTime
  * backup_time_to_string now return a Result
  * remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir

- DynamicIndexHeader: change ctime to i64

- FixedIndexHeader: change ctime to i64
2020-09-15 07:12:57 +02:00
Fabian Grünbichler e0e5b4426a BackupDir: make constructor fallible
since converting from i64 epoch timestamp to DateTime is not always
possible. previously, passing invalid backup-time from client to server
(or vice-versa) panicked the corresponding tokio task. now we get proper
error messages including the invalid timestamp.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:49:35 +02:00
Fabian Grünbichler 7158b304f5 handle invalid mtime when formating entries
otherwise operations like catalog shell panic when viewing pxar archives
containing such entries, e.g. with mtime very far ahead into the future.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:48:43 +02:00
Fabian Grünbichler 833eca6d2f use non-panicky timestamp_opt where appropriate
by either printing the original, out-of-range timestamp as-is, or
bailing with a proper error message instead of panicking.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:48:24 +02:00
Fabian Grünbichler 151acf5d96 don't truncate DateTime nanoseconds
where we don't care about them anyway..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:48:10 +02:00
Fabian Grünbichler 4a363fb4a7 catalog dump: preserve original mtime
even if it can't be handled by chrono. silently replacing it with epoch
0 is confusing..

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-11 15:43:54 +02:00
Dominik Csapak 13bed6226e tools/systemd/parse_time: enable */x syntax for calendar events
we support this in pve, so also support it here to have a more
consistent syntax

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-11 12:17:22 +02:00
Dietmar Maurer 139bcedc53 benchmark: update TLS reference speed
We are now faster with recent patches.
2020-09-10 12:55:43 +02:00
Fabian Grünbichler cf9ea3c4c7 server: set http2 max frame size
else we get the default of 16k, which is quite low for our use case.
this improves the TLS upload benchmark speed by about 30-40% for me.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-10 12:43:51 +02:00
Dietmar Maurer 5656888cc9 verify: fix done count
We need to filter out benchmark group earlier
2020-09-10 09:06:33 +02:00
Dietmar Maurer 5fdc5a6f3d verify: skip benchmark directory 2020-09-10 08:44:18 +02:00
Hannes Laimer 61d7b5013c add benchmark flag to backup creation for proper cleanup when running a benchmark
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2020-09-10 08:25:24 +02:00
Dylan Whyte 871181d984 mount: fix mount subcommand
fixes the error, "manifest does not contain
file 'X.pxar'", that occurs when trying to mount
a pxar archive with 'proxmox-backup-client mount':

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-09-10 07:21:16 +02:00
Dominik Csapak 3be308b949 improve server->client tcp performance for high latency links
similar to the other fix, if we do not set the buffer size manually,
we get better performance for high latency connections

restore benchmark from f.gruenbicher:

no delay, without patch: ~50MB/s
no delay, with patch: ~50MB/s
25ms delay, without patch: ~11MB/s
25ms delay, with path: ~50MB/s

my own restore benchmark:

no delay, without patch: ~1.5GiB/s
no delay, with patch: ~1.5GiB/s
25ms delay, without patch: 30MiB/s
25ms delay, with patch: ~950MiB/s

for some more details about those benchmarks see
https://lists.proxmox.com/pipermail/pbs-devel/2020-September/000600.html

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-10 07:15:25 +02:00
Dominik Csapak 83088644da fix #2983: improve tcp performance
by leaving the buffer sizes on default, we get much better tcp performance
for high latency links

throughput is still impacted by latency, but much less so when
leaving the sizes at default.
the disadvantage is slightly higher memory usage of the server
(details below)

my local benchmarks (proxmox-backup-client benchmark):

pbs client:
PVE Host
Epyc 7351P (16core/32thread)
64GB Memory

pbs server:
VM on Host
1 Socket, 4 Cores (Host CPU type)
4GB Memory

average of 3 runs, rounded to MB/s
                    | no delay |     1ms |     5ms |     10ms |    25ms |
without this patch  |  230MB/s |  55MB/s |  13MB/s |    7MB/s |   3MB/s |
with this patch     |  293MB/s | 293MB/s | 249MB/s |  241MB/s | 104MB/s |

memory usage (resident memory) of proxmox-backup-proxy:

                    | peak during benchmarks | after benchmarks |
without this patch  |                  144MB |            100MB |
with this patch     |                  145MB |            130MB |

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-10 07:15:12 +02:00
Dietmar Maurer 14db8b52dc src/backup/chunk_store.rs: use ? insteadf of unwrap 2020-09-10 06:37:37 +02:00
Stefan Reiter 597427afaf clean up .bad file handling in sweep_unused_chunks
Code cleanup, no functional change intended.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-10 06:31:22 +02:00
Stefan Reiter 3cddfb29be backup: ensure no fixed index writers are left over either
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-10 06:29:38 +02:00
Stefan Reiter 068e526862 backup: touch all chunks, even if they exist
We need to update the atime of chunk files if they already exist,
otherwise a concurrently running GC could sweep them away.

This is protected with ChunkStore.mutex, so the fstat/unlink does not
race with touching.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:51:03 +02:00
Stefan Reiter a9767cf7de gc: remove .bad files on garbage collect
The iterator of get_chunk_iterator is extended with a third parameter
indicating whether the current file is a chunk (false) or a .bad file
(true).

Count their sizes to the total of removed bytes, since it also frees
disk space.

.bad files are only deleted if the corresponding chunk exists, i.e. has
been rewritten. Otherwise we might delete data only marked bad because
of transient errors.

While at it, also clean up and use nix::unistd::unlinkat instead of
unsafe libc calls.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:43:13 +02:00
Dietmar Maurer aadcc2815c cleanup rename_corrupted_chunk: avoid duplicate format macro 2020-09-08 12:29:53 +02:00
Stefan Reiter 0f3b7efa84 verify: rename corrupted chunks with .bad extension
This ensures that following backups will always upload the chunk,
thereby replacing it with a correct version again.

Format for renaming is <digest>.<counter>.bad where <counter> is used if
a chunk is found to be bad again before a GC cleans it up.

Care has been taken to deliberately only rename a chunk in conditions
where it is guaranteed to be an error in the chunk itself. Otherwise a
broken index file could lead to an unwanted mass-rename of chunks.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:20:57 +02:00
Stefan Reiter 7c77e2f94a verify: fix log units
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-08 12:10:19 +02:00
Thomas Lamprecht abd4c4cb8c ui: add translation support
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-08 08:54:33 +02:00
Thomas Lamprecht 09f12d1cf3 tools: rename extract_auth_cookie to extract_cookie
It does nothing specific to authentication..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-08 08:54:33 +02:00
Dominik Csapak 1db4cfb308 tools/sytemd/time: add tests for multivalue fields
we did this wrong earlier, so it makes sense to add regression tests

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-08 07:09:43 +02:00
Dominik Csapak a4c1143664 server/worker_task: fix upid_read_status
a range from high to low in rust results in an empty range
(see std::ops::Range documentation)
so we need to generate the range from 0..data.len() and then reverse it

also, the task log contains a newline at the end, so we have to remove
that (should it exist)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-08 07:06:22 +02:00
Dietmar Maurer 32afd60336 src/tools/systemd/time.rs: derive Clone 2020-09-07 12:37:08 +02:00
Dietmar Maurer 02e47b8d6e SYSTEMD_CALENDAR_EVENT_SCHEMA: fix wrong schema description 2020-09-07 09:07:55 +02:00
Dominik Csapak 44055cac4d tools/systemd/time: enable dates for calendarevents
this implements parsing and calculating calendarevents that have a
basic date component (year-mon-day) with the usual syntax options
(*, ranges, lists)

and some special events:
monthly
yearly/annually (like systemd)
quarterly
semiannually,semi-annually (like systemd)

includes some regression tests

the ~ syntax for days (the last x days of the month) is not yet
implemented

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:36:29 +02:00
Dominik Csapak 1dfc09cb6b tools/systemd/time: fix signed conversion
instead of using 'as' and silently converting wrong,
use the TryInto trait and raise an error if we cannot convert

this should only happen if we have a negative year,
but this is expected (we do not want schedules from before the year 0)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:35:38 +02:00
Dominik Csapak 48c56024aa tools/systemd/tm_editor: add setter/getter for months/years/days
add_* are modeled after add_days

subtract one for set_mon to have a consistent interface for all fields
(i.e. getter/setter return/expect the 'real' number, not the ones
in the tm struct)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:34:27 +02:00
Dominik Csapak cf103266b3 tools/systemd/tm_editor: move conversion of the year into getter and setter
the tm struct contains the year - 1900 but we added that

if we want to use the libc normalization correctly, the tm struct
must have the correct year in it, else the computations for timezones,
etc. fail

instead add a getter that adds the years and a setter that subtracts it again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:34:04 +02:00
Dominik Csapak d5cf8f606c tools/systemd/time: fix selection for multiple options
if we give multiple options/ranges for a value, e.g.
2,4,8
we always choose the biggest, instead of the smallest that is next

this happens because in DateTimeValue::find_next(value)
'next' can be set multiple times and we set it when the new
value was *bigger* than the last found 'next' value, when in reality
we have to choose the *smallest* next we can find

reverse the comparison operator to fix this

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:33:42 +02:00
Dominik Csapak ce7ab28cfa tools/systemd/parse_time: error out on invalid ranges
if the range is reverse (bigger..smaller) we will never find a value,
so error out during parsing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:28:48 +02:00
Dominik Csapak 07ca6f6e66 tools/systemd/tm_editor: remove reset_time from add_days and document it
we never passed 'false' to it anyway so remove it
(we can add it again if we should ever need it)

also remove the adding of wday (gets normalized anyway)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:28:24 +02:00
Dominik Csapak 15ec790a40 tools/systemd/time: convert the resulting timestamp into an option
we want to use dates for the calendarspec, and with that there are some
impossible combinations that cannot be detected during parsing
(e.g. some datetimes do not exist in some timezones, and the timezone
can change after setting the schedule)

so finding no timestamp is not an error anymore but a valid result

we omit logging in that case (since it is not an error anymore)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:28:05 +02:00
Dominik Csapak cb73b2d69c tools/systemd/time: move continue out of the if/else
will be called anyway

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:27:20 +02:00
Dominik Csapak c931c87173 tools/systemd/time: let libc normalize time for us
mktime/gmtime can normalize time and even can handle special timezone
cases like the fact that the time 2:30 on specific day/timezone combos
do not exists

we have to convert the signature of all functions that use
normalize_time since mktime/gmtime can return an EOVERFLOW
but if this happens there is no way we can find a good time anyway

since normalize_time will always set wday according to the rest of the
time, remove set_wday

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:26:40 +02:00
Dominik Csapak 28a0a9343c tools/systemd/tm_editor: remove TMChanges optimization
while it was correct, there was no measurable speed gain
(a benchmark yielded 2.8 ms for a spec that did not find a timestamp either way)
so remove it for simpler code

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-04 15:26:04 +02:00
Dietmar Maurer 56b666458c server/worker_task: fix 'unknown' status for some big task logs
when trying to parse the task status, we seek 8k from the end
which may be into the middle of a line, so the datetime parsing
can fail (when the log message contains ': ')

This patch does a fast search for the last line, and avoid the
'lines' iterator.
2020-09-04 10:41:13 +02:00
Dietmar Maurer cd6ddb5a69 depend on proxmox 0.3.5 2020-09-04 08:11:53 +02:00
Fabian Grünbichler ecd55041a2 fix #2978: allow non-root to view datastore usage
for datastores where the requesting user has read or write permissions,
since the API method itself filters by that already. this is the same
permission setting and filtering that the datastore list API endpoint
does.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-04 06:18:20 +02:00
Dietmar Maurer 8317873c06 gc: improve percentage done logs 2020-09-02 10:04:18 +02:00
Dietmar Maurer deef63699e verify: also fail on server shutdown 2020-09-02 09:50:17 +02:00
Dietmar Maurer 63d9aca96f verify: log progress 2020-09-02 07:43:28 +02:00
Dietmar Maurer 4f09d31085 src/backup/verify.rs: use global hashes (instead of per group)
This makes verify more predictable.
2020-09-01 13:33:04 +02:00
Dietmar Maurer 58d73ddb1d src/backup/data_blob.rs: avoid useless &, data is already a reference 2020-09-01 12:56:25 +02:00
Dietmar Maurer 6b809ff59b src/backup/verify.rs: use separate thread to load data 2020-09-01 12:56:25 +02:00
Thomas Lamprecht 49a92084a9 gc: use human readable units for summary
and avoid the "percentage done: X %" phrase

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-27 16:06:35 +02:00
Thomas Lamprecht 843880f008 bin/backup-proxy: assert that daemon runs as backup user/group
Because if not, the backups it creates have bogus permissions and may
seem like they got broken once the daemon is started again with the
correct user/group.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 10:30:15 +02:00
Thomas Lamprecht a6ed5e1273 backup: add BACKUP_GROUP_NAME const and backup_group helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 10:27:47 +02:00
Thomas Lamprecht 74f94d0678 bin/backup-proxy: remove outdated perl comments
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 10:27:47 +02:00
Thomas Lamprecht 946c3e8a81 bin/backup-proxy: return error directly in main
anyhow makes this a nice error message, similar to the manual
wrapping used.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 10:27:47 +02:00
Thomas Lamprecht 3b2046d263 save last verify result in snapshot manifest
Save the state ("ok" or "failed") and the UPID of the respective
verify task. With this we can easily allow to open the relevant task
log and show when the last verify happened.

As we already load the manifest when listing the snapshots, just add
it there directly.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-26 07:35:13 +02:00
Thomas Lamprecht 1ffe030123 various typo fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-25 18:52:31 +02:00
Thomas Lamprecht 5255e641fa SnapshotListItem: add comment field also to schema
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-25 16:24:36 +02:00
Thomas Lamprecht c86b6f40d7 tools/format: implement from u64 for HumanByte helper type
Could be problematic for systems where usize is 32 bit, but we do not
really support those.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-25 14:18:49 +02:00
Thomas Lamprecht 5a718dce17 api datastore: fix typo in error message
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-08-25 14:16:40 +02:00
Dietmar Maurer 7ae571e7cb verify: speedup - only verify chunks once
We need to do the check before we load the chunk.
2020-08-25 08:52:24 +02:00
Dietmar Maurer 4264c5023b verify: sort backup groups 2020-08-25 08:38:47 +02:00
Dylan Whyte 3435f5491b Fix typo in program output
Change "comptation" -> "computation"

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-08-19 09:06:27 +02:00
Wolfgang Bumiller 11e1e27a42 turn UPID into an API type
It's a string-type.
Implement Serialize via Display, Deserialize via FromStr and
add an API_SCHEMA so that it can be used as a type within
the #[api] macro.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-18 11:54:30 +02:00
Wolfgang Bumiller 4ea831bfa1 style fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-18 08:50:14 +02:00
Wolfgang Bumiller c1d7d708d4 remove map_struct helper
if we ever need this it should be marked as unsafe!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-17 11:53:02 +02:00
Wolfgang Bumiller 3fa2b983c1 add methods to allocate a DynamicIndexHeader
to avoid `map_struct` which is actually unsafe because it
does not verify alignment constraints at all

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-17 11:50:32 +02:00
Dominik Csapak a1e9c05738 api2/node/services: turn service api calls into workers
to be in line with pve/pmg and be able to show the progress in the gui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 12:37:17 +02:00
Dominik Csapak 934deeff2d fix #2904: zpool status: parse vdevs with state but without statistics
some vdevs (e.g. spares) have a 'state' (e.g. AVAIL), but
not statistics like READ/WRITE/etc.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 11:41:32 +02:00
Dominik Csapak c162df60c8 zfs status: add test with spares
this will fail for now, fixed in the next commit

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 11:41:32 +02:00
Dietmar Maurer 98161fddb5 cleanup last patch 2020-08-14 07:30:05 +02:00
Hannes Laimer be614c625f api2/node/../disks/directory: added DELETE endpoint for removal of mount-units
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-08-14 07:06:10 +02:00
Fabian Ebner 87c4cb7419 Fix #2926: parse_iface_attributes: always break on non-{attribitue, comment} token
There is no requirement to have at least
a blank line, attribute or comment in between two
interface definitions, e.g.
iface lo inet loopback
iface lo inet6 loopback

Signed-off-by: Fabian Ebner <f.ebner@proxmox.com>
2020-08-14 06:57:07 +02:00
Dominik Csapak 93bb51fe7e config/jobstate: replace Job:load with create_state_file
it really is not necessary, since the only time we are interested in
loading the state from the file is when we list it, and there
we use JobState::load directly to avoid the lock

we still need to create the file on syncjob creation though, so
that we have the correct time for the schedule

to do this we add a new create_state_file that overwrites it on creation
of a syncjob

for safety, we subtract 30 seconds from the in-memory state in case
the statefile is missing

since we call create_state_file from  proxmox-backup-api,
we have to chown the lock file after creating to the backup user,
else the sync job scheduling cannot aquire the lock

also we remove the lock file on statefile removal

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 06:38:02 +02:00
Dominik Csapak 713b66b6ed cleanup: replace id from do_sync_job with info from job
we already have it inside the job itself

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 06:36:43 +02:00
Dominik Csapak 77bd2a469c cleanup: merge endtime into TaskState
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 06:36:19 +02:00
Dominik Csapak a13573c24a syncjob: use do_sync_job also for scheduled sync jobs
and determine the last runtime with the jobstate

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:51:20 +02:00
Dominik Csapak 02543a5c7f api2/pull: extend do_sync_job to also handle schedule and jobstate
so that we can log if triggered by a schedule, and writing to a jobstatefile
also correctly polls now the abort_future of the worker, so that
users can stop a sync

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:49:28 +02:00
Dominik Csapak 42b68f72e6 api/{pull, sync}: refactor to do_sync_job
and move the pull parameters into the worker, so that the task log
contains the error if there is one

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:40:52 +02:00
Dominik Csapak 664d8a2765 api2/admin/sync: use JobState for faster access to state info
and delete the statefile again on syncjob removal

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:40:00 +02:00
Dominik Csapak e6263c2662 config: add JobState helper
this is intended to be a generic helper to (de)serialize job states
(e.g., sync, verify, and so on)

writes a json file into '/var/lib/proxmox-backup/jobstates/TYPE-ID.json'

the api creates the directory with the correct permissions, like
the rrd directory

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:36:10 +02:00
Dominik Csapak ae197dda23 server/worker_task: let upid_read_status also return the endtime
the endtime should be the timestamp of the last log line
or if there is no log at all, the starttime

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:35:44 +02:00
Dominik Csapak 4c116bafb8 server: change status of a task from a string to an enum
representing a state via an enum makes more sense in this case
we also implement FromStr and Display to make it easy to convet from/to
a string

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:35:19 +02:00
Wolfgang Bumiller df30017ff8 remove unused import
rustc doesn't warn about this kind of import, however,
clippy does

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-13 09:05:15 +02:00
Wolfgang Bumiller 3f3ae19d63 formatting fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-12 14:30:03 +02:00
Wolfgang Bumiller 72dc68323c replace and remove old ticket functions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-12 14:28:21 +02:00
Wolfgang Bumiller 593f917742 introduce Ticket struct
and add tests and compatibility tests

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-12 14:28:21 +02:00
Dietmar Maurer 639419b049 worker_task: new_thread() - remove unused tokio channel 2020-08-12 08:43:09 +02:00
Stefan Reiter 81f293513e backup: lock base snapshot and ensure existance on finish
To prevent forgetting the base snapshot of a running backup, and catch
the case when it still happens (e.g. via manual rm) to at least error
out instead of storing a potentially invalid backup.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:04:47 +02:00
Stefan Reiter 8b5f72b176 Revert "backup: ensure base snapshots are still available after backup"
This reverts commit d53fbe2474.

The HashSet and "register" function are unnecessary, as we already know
which backup is the one we need to check: the last one, stored as
'last_backup'.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:03:53 +02:00
Stefan Reiter f23f75433f backup: flock snapshot on backup start
An flock on the snapshot dir itself is used in addition to the group dir
lock. The lock is used to avoid races with forget and prune, while
having more granularity than the group lock (i.e. the group lock is
necessary to prevent more than one backup per group, but the snapshot
lock still allows backups unrelated to the currently running to be
forgotten/pruned).

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:02:21 +02:00
Stefan Reiter 6d6b4e72d3 datastore: prevent in-use deletion with locks instead of heuristic
Attempt to lock the backup directory to be deleted, if it works keep the
lock until the deletion is complete. This way we ensure that no other
locking operation (e.g. using a snapshot as base for another backup) can
happen concurrently.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 11:00:29 +02:00
Dietmar Maurer e434258592 src/backup/backup_info.rs: remove BackupGroup lock()
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 10:58:35 +02:00
Dietmar Maurer 3dc1a2d5b6 src/tools/fs.rs: new helper lock_dir_noblock
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-11 10:57:48 +02:00
Fabian Grünbichler 882c082369 mark signed manifests as such
for less-confusing display in the web interface

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-11 09:56:53 +02:00
Fabian Grünbichler 9a38fa29c2 verify: also check chunk CryptMode
and in-line verify_stored_chunk to avoid double-loading each chunk.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-11 09:56:20 +02:00
Fabian Grünbichler 14f6c9cb8b chunk readers: ensure chunk/index CryptMode matches
an encrypted Index should never reference a plain-text chunk, and an
unencrypted Index should never reference an encrypted chunk.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-11 09:54:22 +02:00
Fabian Grünbichler 2d55beeca0 datastore api: verify blob/index csum from manifest
when dowloading decoded files.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-11 09:52:45 +02:00
Fabian Grünbichler 9238cdf50d datastore api: only decode unencrypted indices
these checks were already in place for regular downloading of backed up
files, also do them when attempting to decode a catalog, or when
downloading decoded files referenced by a pxar index.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-11 09:51:20 +02:00
Wolfgang Bumiller 5d30f03826 impl PartialEq between Realm and RealmRef
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 12:23:36 +02:00
Wolfgang Bumiller 14263ef989 assert that Username does not impl PartialEq
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 12:21:12 +02:00
Wolfgang Bumiller e7cb4dc50d introduce Username, Realm and Userid api types
and begin splitting up types.rs as it has grown quite large
already

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 12:05:01 +02:00
Wolfgang Bumiller 866c556faf move types.rs to types/mod.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 10:32:31 +02:00
Wolfgang Bumiller 90d515c97d config.rs: sort modules
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 08:33:38 +02:00
Stefan Reiter 4dbe129284 backup: only allow finished backups as base snapshot
If the datastore holds broken backups for some reason, do not attempt to
base following snapshots on those. This would lead to an error on
/previous, leaving the client no choice but to upload all chunks, even
though there might be potential for incremental savings.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-07 07:32:56 +02:00
Dylan Whyte 5f76ac37b5 fix: master-key: upload RSA encoded key with backup
When uploading an RSA encoded key alongside the backup,
the backup would fail with the error message: "wrong blob
file extension".
Adding the '.blob' extension to rsa-encrypted.key before the
the call to upload_blob_from_data(), rather than after, fixes
the issue.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-08-06 09:34:01 +02:00
Stefan Reiter d74edc3d89 finish_backup: mark backup as finished only after checks have passed
Commit 9fa55e09 "finish_backup: test/verify manifest at server side"
moved the finished-marking above some checks, which means if those fail
the backup would still be marked as successful on the server.

Revert that part and comment the line for the future.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-08-06 06:39:34 +02:00
Oguz Bektas 2f57a433b1 fix #2909: handle missing chunks gracefully in garbage collection
instead of bailing and stopping the entire GC process, warn about the
missing chunks and continue.

this results in "TASK WARNINGS: X" as the status.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-08-06 06:36:48 +02:00
Wolfgang Bumiller 98c259b4c1 remove timer and lock functions, fix building with proxmox 0.3.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-04 11:33:02 +02:00
Aaron Lauterer d3d566f7bd GC: use time pre phase1 to calculate min_atime in phase2
Used chunks are marked in phase1 of the garbage collection process by
using the atime property. Each used chunk gets touched so that the atime
gets updated (if older than 24h, see relatime).

Should there ever be a situation in which the phase1 in the GC run needs
a very long time to finish, it could happen that the grace period
calculated in phase2 is not long enough and thus the marking of the
chunks (atime) becomes invalid. This would result in the removal of
needed chunks.

Even though the likelyhood of this happening is very low, using the
timestamp from right before phase1 is started, to calculate the grace
period in phase2 should avoid this situation.

Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-08-04 10:19:05 +02:00
Fabian Grünbichler 2ce159343b sync: verify size and checksum of pulled archives
and not just of previously synced ones.

we can't use BackupManifest::verify_file as the archive is still stored
under the tmp path at this point.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-04 07:27:56 +02:00
Fabian Grünbichler 9e496ff6f1 sync: verify chunk size and digest, if possible
for encrypted chunks this is currently not possible, as we need the key
to decode the chunk.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-04 07:27:56 +02:00
Fabian Grünbichler 8819d1f2f5 blobs: attempt to verify on decode when possible
regular chunks are only decoded when their contents are accessed, in
which case we need to have the key anyway and want to verify the digest.

for blobs we need to verify beforehand, since their checksums are always
calculated based on their raw content, and stored in the manifest.

manifests are also stored as blobs, but don't have a digest in the
traditional sense (they might have a signature covering parts of their
contents, but that is verified already when loading the manifest).

this commit does not cover pull/sync code which copies blobs and chunks
as-is without decoding them.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-08-04 07:27:56 +02:00
Wolfgang Bumiller 0f9218079a pxar/extract: fixup path stack for errors
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-03 12:20:30 +02:00
Wolfgang Bumiller 1cafbdc70d more whitespace fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-03 12:02:19 +02:00
Wolfgang Bumiller a3eb7b2cea whitespace fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-03 12:00:59 +02:00
Wolfgang Bumiller d9b8e2c795 pxar: better error handling on extract
Errors while applying metadata will not be considered fatal
by default using `pxar extract` unless `--strict` was passed
in which case it'll bail out immediately.

It'll still return an error exit status if something had
failed along the way.

Note that most other errors will still cause it to bail out
(eg. errors creating files, or I/O errors while writing
the contents).

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-03 09:40:55 +02:00
Dominik Csapak 4bd2a9e42d worker_task: add getter for upid
sometimes we need the upid inside the worker itself, so give a
possibilty to get it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-03 08:26:17 +02:00
Dominik Csapak cef03f4149 worker_task: refactor log text generator
we will need this elsewhere, so pull it out

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-03 08:23:13 +02:00
Dominik Csapak eeb19aeb2d systemd/time: fix weekday wrapping on month
the weekday does not change depending on the month, so remove that wrapping

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-03 08:18:42 +02:00
Dominik Csapak 6c96ec418d systemd/time: add tests for weekday month wrapping
this will fail for now, gets fixed in the next commit

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-03 08:15:26 +02:00
Wolfgang Bumiller 30c3c5d66c pxar: create: attempt to use O_NOATIME
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-31 11:46:53 +02:00
Wolfgang Bumiller e51be33807 pxar: create: move common O_ flags to open_file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-31 11:42:15 +02:00
Dietmar Maurer 70030b43d0 list_snapshots: Returns new "comment" property (fisrt line from notes) 2020-07-31 11:34:42 +02:00
Dietmar Maurer ff86ef00a7 cleanup: manifest is always CryptMode::None 2020-07-31 10:25:30 +02:00
Dietmar Maurer 912b3f5bc9 src/api2/admin/datastore.rs: add API to get/set Notes for backus 2020-07-31 10:17:35 +02:00
Dietmar Maurer a4acb6ef84 lock_file: return std::io::Error 2020-07-31 08:53:00 +02:00
Dietmar Maurer d7ee07d838 src/api2/backup/environment.rs: remove debug code 2020-07-31 07:48:53 +02:00
Dietmar Maurer 53705acece src/api2/backup/environment.rs: remove debug code 2020-07-31 07:47:08 +02:00
Dietmar Maurer c8fff67d88 finish_backup: add chunk_upload_stats to manifest 2020-07-31 07:45:47 +02:00
Dietmar Maurer 9fa55e09a7 finish_backup: test/verify manifest at server side
We want to make sure that the client uploaded a readable manifest.
2020-07-31 07:45:47 +02:00
Dietmar Maurer e443902583 src/backup/datastore.rs: add helpers to load/store manifest
We want this to modify the manifest "unprotected" data, for example
to add upload statistics, notes, ...
2020-07-31 07:45:47 +02:00
Dominik Csapak f39a900722 api2/node/termproxy: fix user in worker task
'username' here is without realm, but we really want to use user@realm

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-30 11:57:43 +02:00
Dietmar Maurer 1fc82c41f2 src/api2/backup.rs: aquire backup lock earlier in create_locked_backup_group() 2020-07-30 11:03:05 +02:00
Dominik Csapak d2b0c78e23 api2/node/termproxy: fix zombies on worker abort
tokios kill_on_drop sometimes leaves zombies around, especially
when there is not another tokio::process::Command spawned after

so instead of relying on the 'kill_on_drop' feature, we explicitly
kill the child on a worker abort. to be able to do this
we have to use 'tokio::select' instead of 'futures::select' since
the latter requires the future to be fused, which consumes the
child handle, leaving us no possibility to kill it after fusing.
(tokio::select does not need the futures to be fused, so we
can reuse the child future after the select again)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-30 10:38:14 +02:00
Dominik Csapak adfdc36936 verify: keep track and log which dirs failed the verification
so that we can print a list at the end of the worker which backups
are corrupt.

this is useful if there are many snapshots and some in between had an
error. Before this patch, the task log simply says to 'look in the logs'
but if the log is very long it makes it hard to see what exactly failed.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-30 09:39:37 +02:00
Dominik Csapak d8594d87f1 verify: keep also track of corrupt chunks
so that we do not have to verify a corrupt one multiple times

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-30 09:39:37 +02:00
Dominik Csapak f66f537da9 verify: check all chunks of an index, even if we encounter a corrupt one
this makes it easier to see which chunks are corrupt
(and enables us in the future to build a 'complete' list of
corrupt chunks)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-30 09:39:37 +02:00
Wolfgang Bumiller d44185c4a1 fix #2873: if --pattern is used, default to not extracting
The extraction algorithm has a state (bool) indicating
whether we're currently in a positive or negative match
which has always been initialized to true at the beginning,
but when the user provides a `--pattern` argument we need to
start out with a negative match.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-30 09:33:30 +02:00
Stefan Reiter d53fbe2474 backup: ensure base snapshots are still available after backup
This should never trigger if everything else works correctly, but it is
still a very cheap check to avoid wrongly marking a backup as "OK" when
in fact some chunks might be missing.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:28:54 +02:00
Stefan Reiter 95bda2f25d backup: use flock on backup group to forbid multiple backups at once
Multiple backups within one backup group don't really make sense, but
break all sorts of guarantees (e.g. a second backup started after a
first would use a "known-chunks" list from the previous unfinished one,
which would be empty - but using the list from the last finished one is
not a fix either, as that one could be deleted or pruned once the first
simultaneous backup is finished).

Fix it by only allowing one backup per backup group at one time. This is
done via a flock on the backup group directory, thus remaining intact
even after a reload.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:26:26 +02:00
Stefan Reiter c9756b40d1 datastore: prevent deletion of snaps in use as "previous backup"
To prevent a race with a background GC operation, do not allow deletion
of backups who's index might currently be referenced as the "known chunk
list" for successive backups. Otherwise the GC could delete chunks it
thinks are no longer referenced, while at the same time telling the
client that it doesn't need to upload said chunks because they already
exist.

Additionally, prevent deletion of whole backup groups, if there are
snapshots contained that appear to be currently in-progress. This is
currently unlikely to trigger, as that function is only used for sync
jobs, but it's a useful safeguard either way.

Deleting a single snapshot has a 'force' parameter, which is necessary
to allow deleting incomplete snapshots on an aborted backup. Pruning
also sets force=true to avoid the check, since it calculates which
snapshots to keep on its own.

To avoid code duplication, the is_finished method is factored out.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:26:01 +02:00
Stefan Reiter 8cd29fb24a tools: add nonblocking mode to lock_file
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:18:10 +02:00
Stefan Reiter 505c5f0f76 fix typo: avgerage to average
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 07:08:08 +02:00
Dietmar Maurer 2aaae9705e src/backup/verify.rs: try to verify chunks only once
We use a HashSet (per BackupGroup) to track already verified chunks.
2020-07-29 13:29:13 +02:00
Wolfgang Bumiller 8aa67ee758 bump proxmox to 0.3, cleanup http_err macro usage
Also swap the order of a couple of `.map_err().await` to
`.await.map_err()` since that's generally more efficient.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-29 09:38:36 +02:00
Wolfgang Bumiller 3865e27e96 src/api2/node.rs: 'mod' statement cleanup
split them into groups: `pub`, `pub(crate)` and non-pub

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-29 09:19:57 +02:00
Wolfgang Bumiller f6c6e09a8a update to pxar 0.3 to support negative timestamps
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-29 08:31:37 +02:00
Dominik Csapak be10cdb122 fix #2856: also check whole device for device mapper
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-28 11:03:45 +02:00
Dominik Csapak 7fde1a71ca upload_chunk: allow upload of empty blobs
a blob can be empty (e.g. an empty pct fw conf), so we
have to set the minimum size to the header size

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-28 11:03:36 +02:00
Dietmar Maurer 39f18b30b6 src/backup/data_blob.rs: new load_from_reader(), which verifies the CRC
And make verify_crc private for now. We always call load_from_reader() to
verify the CRC.

Also add load_chunk() to datastore.rs (from chunk_store::read_chunk())
2020-07-28 10:23:16 +02:00
Dietmar Maurer bccdc5fa04 src/backup/manifest.rs: cleanup - again, avoid recursive call to write_canonical_json
And use re-borrow instead of dyn trait casting.
2020-07-27 10:31:34 +02:00
Dietmar Maurer 0bf7ba6c92 src/backup/manifest.rs: cleanup - avoid recursive call to write_canonical_json 2020-07-27 08:48:11 +02:00
Thomas Lamprecht e6b599aa6c services: make reload safer and default to it in gui
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-25 20:23:12 +02:00
Thomas Lamprecht ee15af6bb8 api: service command: fix test for essential service
makes no sense to disallow reload or start (even if start cannot
really happen)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-24 19:35:19 +02:00
Thomas Lamprecht 3da9b7e0dd followup: server/state: rename task_count to internal_task_count
so that the relation with spawn_internal_task is made more clear

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-24 12:11:39 +02:00
Dominik Csapak 33a88dafb9 server/state: add spawn_internal_task and use it for websockets
is a helper to spawn an internal tokio task without it showing up
in the task list

it is still tracked for reload and notifies the last_worker_listeners

this enables the console to survive a reload of proxmox-backup-proxy

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-24 11:17:33 +02:00
Dominik Csapak 224c65f8de termproxy: let users stop the termproxy task
for that we have to do a select on the workers abort_future

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-24 11:17:33 +02:00
Stoiko Ivanov f2b4b4b9fe fix 2885: bail on duplicate backup target
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-07-24 11:08:56 +02:00
Thomas Lamprecht ea9e559fc4 client: log archive upload duration more accurate, fix grammar
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-24 10:15:28 +02:00
Thomas Lamprecht 0cf14984cc client: avoid division by zero in avg speed calculation, be more accurate
using micros vs. as_secs_f64 allows to have it calculated as usize
bytes, easier to handle - this was also used when it still lived in
upload_chunk_info_stream

Co-authored-by: Stoiko Ivanov <s.ivanov@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-24 10:14:40 +02:00
Thomas Lamprecht 3d3670d786 termproxy: cmd: support upgrade
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 13:12:18 +02:00
Thomas Lamprecht e744de0eb0 api: termproxy: fix ACL as /nodes is /system
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 12:57:11 +02:00
Thomas Lamprecht 98b1733760 api: apt: use schema default const for quiet param
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 12:25:28 +02:00
Dominik Csapak 01ca99da2d server/rest: add console to index
register the console template and render it when the 'console' parameter
is given

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 12:06:38 +02:00
Dominik Csapak 1c2f842a98 api2/nodes: add termproxy and vncwebsocket api calls
Even though it has nothing to do with vnc, we keep the name of the api
call for compatibility with our xtermjs client.

termproxy:
verifies that the user is allowed to open a console and starts
termproxy with the correct parameters

starts a TcpListener on "localhost:0" so that the kernel decides the
port (instead of trying to rerserving like in pve). Then it
leaves the fd open for termproxy and gives the number as port
and tells it via '--port-as-fd' that it should interpret this
as an open fd

the vncwebsocket api call checks the 'vncticket' (name for compatibility)
and connects the remote side (after an Upgrade) with a local TcpStream
connecting to the port given via WebSocket from the proxmox crate

to make sure that only the client can connect that called termproxy and
no one can connect to an arbitrary port on the host we have to include
the port in the ticket data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 12:06:38 +02:00
Dominik Csapak a4d1675513 api2/access: implement term ticket
modeled after pves/pmgs vncticket (i substituted the vnc with term)
by putting the path and username as secret data in the ticket

when sending the ticket to /access/ticket it only verifies it,
checks the privs on the path and does not generate a new ticket

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 11:55:00 +02:00
Dominik Csapak 2ab5acac5a server/config: add mechanism to update template
instead of exposing handlebars itself, offer a register_template and
a render_template ourselves.

render_template checks if the template file was modified since
the last render and reloads it when necessary

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 11:55:00 +02:00
Thomas Lamprecht 27fde64794 api: apt update must run protected
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 11:45:52 +02:00
Thomas Lamprecht fa3f0584bb api: apt: support refreshing package index
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 11:21:54 +02:00
Stefan Reiter a4e86972a4 add .../apt/update API call
Depends on patched apt-pkg-native-rs. Changelog-URL detection is
inspired by PVE perl code for now, though marked with fixme to use 'apt
changelog' later on, if/when our repos have APT-compatible changelogs
set up.

list_installed_apt_packages iterates all packages and creates an
APTUpdateInfo with detailed information for every package matched by the
given filter Fn.

Sadly, libapt-pkg has some questionable design choices regarding their
use of 'iterators', which means quite a bit of nesting...

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-23 10:41:14 +02:00
Thomas Lamprecht 3a3af6e2b6 backup manifest: make lookup_file_info public
useful to get info like, was the previous snapshot encrypted in
libproxmox-backup-qemu

Requested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 10:39:21 +02:00
Thomas Lamprecht 9688f6de0f client: log index.json upload only when verbose
I mean the user expects that we know what archives, fidx or didx, are
in a backup, so this is internal info and should not be logged by
default

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 10:28:38 +02:00
Thomas Lamprecht 5b32820e93 client: don't use debug format for printing BackupRepository
It implements the fmt::Display  trait after all

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 10:28:38 +02:00
Thomas Lamprecht f40b4fb05a client writer: do not output chunklist for now on verbose true
Verbosity needs to be a non binary level, as this now is just
debug/development info, for endusers normally to much.

We want to have it available, but with a much higher verbosity level.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 10:28:38 +02:00
Thomas Lamprecht 6e1deb158a client: rework logging upload size, bandwidth, ... info
Track reused size and chunk counts.
Log reused size and use pretty print for all sizes and bandwidth
metrics.
Calculate speed over the actually uploaded size, as else it can be
skewed really bad (showing like terabytes per second)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 10:28:38 +02:00
Thomas Lamprecht 50ec1a8712 tools/format: add struct to pretty print bytes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 09:36:02 +02:00
Thomas Lamprecht a74b026baa systemd/time: document CalendarEvent struct and add TODOs
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-23 07:55:42 +02:00
Thomas Lamprecht 7e42ccdaf2 fixed index: chunk_from_offset: avoid slow modulo operation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-22 17:46:07 +02:00
Stefan Reiter e713ee5c56 remove BufferedFixedReader interface
replaced by AsyncIndexReader

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-22 17:28:49 +02:00
Stefan Reiter ec5f9d3525 implement AsyncSeek for AsyncIndexReader
Requires updating the AsyncRead implementation to cope with byte-wise
seeks to intra-chunk positions.

Uses chunk_from_offset to get locations within chunks, but tries to
avoid it for sequential read to not reduce performance from before.

AsyncSeek needs to use the temporary seek_to_pos to avoid changing the
position in case an invalid seek is given and it needs to error in
poll_complete.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-22 17:28:49 +02:00
Stefan Reiter d0463b67ca add and implement chunk_from_offset for IndexFile
Necessary for byte-wise seeking through chunks in an index.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-22 17:28:49 +02:00
Thomas Lamprecht 2ff4c2cd5f datastore/chunker: fix comment typos
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-22 16:12:49 +02:00
Thomas Lamprecht c3b090ac8a backup: list images: handle walkdir error, catch "lost+found"
We support using an ext4 mountpoint directly as datastore and even do
so ourself when creating one through the disk manage code.

Such ext4 ountpoints have a lost+found directory which only root can
traverse into. As the GC list images is done as backup:backup user
walkdir gets an error.

We cannot ignore just all permission errors, as they could lead to
missing some backup indexes and thus possibly sweeping more chunks
than desired. While *normally* that should not happen through our
stack, we had already user report that they do rsyncs to move a
datastore from old to new server and got the permission wrong.

So for now be still very strict, only allow a "lost+found" directory
as immediate child of the datastore base directory, nothing else.

If deemed safe, this can always be made less strict. Possibly by
filtering the known backup-types on the highest level first.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-22 16:01:55 +02:00
Thomas Lamprecht c47e294ea7 datastore: fix typo
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-22 15:04:14 +02:00
Fabian Grünbichler 25455bd06d fix #2871: close FDs when scanning backup group
otherwise we leak those descriptors and run into EMFILE when a backup
group contains many snapshots.

fcntl::openat and Dir::openat are not the same ;)

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-22 09:19:29 +02:00
Fabian Grünbichler c1c4a18f48 fix #2865: detect and skip vanished snapshots
also when they have been removed/forgotten since we retrieved the
snapshot list for the currently syncing backup group.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-22 09:19:29 +02:00
Fabian Grünbichler 91f5594c08 api: translate ENOTFOUND to 404 for downloads
and percolate the HttpError back up on the client side

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-22 09:19:29 +02:00
Fabian Grünbichler 86f6f74114 fix #2860: skip in-progress snapshots when syncing
they don't have a final manifest yet and are not done, so they can't be
synced either.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-22 09:19:29 +02:00
Stefan Reiter 113c9b5981 move subscription API path to /nodes
This aligns it with PVE and allows the widget toolkit's update window
"refresh" to work without modifications once POST /apt/update is
implemented.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-21 19:33:52 +02:00
Thomas Lamprecht 956295cefe parse_calendar_event: support the weekly special expression
While we do not yet support the date specs for CalendarEvent the left
out "weekly" special expression[0] dies not requires that support.
It is specified to be equivalent with `Mon *-*-* 00:00:00` [0] and
this can be implemented with the weekday and time support we already
have.

[0]: https://www.freedesktop.org/software/systemd/man/systemd.time.html#Calendar%20Events

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-21 13:24:51 +02:00
Dominik Csapak a26c27c8e6 api2/status: fix estimation bug
when a datastore has enough data to calculate the estimated full date,
but always has exactly the same usage, the factor b of the regression
is '0'

return 0 for that case so that the gui can show 'never' instead of
'not enough data'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-21 13:02:08 +02:00
Aaron Lauterer b96b11cdb7 chunk_store: Fix typo in bail message
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-07-21 12:51:41 +02:00
Aaron Lauterer faa8e6948a backup: Fix typos and grammar
Signed-off-by: Aaron Lauterer <a.lauterer@proxmox.com>
2020-07-21 12:51:41 +02:00
Fabian Grünbichler 1b1110581a manifest: revert canonicalization to old behaviour
JSON keys MUST be quoted. this is a one-time break in signature
validation for backups created with the broken canonicalization code.
QEMU backups are not affected, as libproxmox-backup-qemu never linked
the broken versions.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-07-20 11:37:53 +02:00
Thomas Lamprecht 449e4a66fe tools/xattr: a char from C is not universally a rust i8
Make it actually do the correct cast by using `libc::c_char`.

Fixes issues when building on other platforms, e.g., the aarch64
client only build on Arch Linux ARM I tested in my free time.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-19 19:46:27 +02:00
Thomas Lamprecht 217c22c754 server: add path value to NOT_FOUND http error
Especially helpful for requests not coming from browsers (where the
URL is normally easy to find out).

Makes it easier to detect if one triggered a request with an old
client, or so..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-16 12:46:51 +02:00
Dominik Csapak ac5e9e770b catalog_shell: add exit command
it is nice to have a command to exit from the shell instead of
only allowing ctrl+d or ctrl+c

the api method is just for documentation/help purposes and does nothing
by itself, the real logic is directly in the read loop

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-15 12:19:57 +02:00
Wolfgang Bumiller b25deec0be pxar: .pxarexclude: absolute paths and byte based paths
Change the .pxarexclude parser to byte based parsing with
`.split(b'\n')` instead of `.lines()`, to not panic on
non-utf8 paths.

Specially deal with absolute paths by prefixing them with
the current directory.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-15 11:55:48 +02:00
Wolfgang Bumiller cdf1da2872 tools: add strip_ascii_whitespace for byte slices
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-15 11:55:48 +02:00
Thomas Lamprecht 3cfc56f5c2 cached user info: check_privs: print privilege path in error message
As else this is really user unfriendly, and it not printing it has no
advantage. If one doesn't wants to leak resource existence they just
need to *always* check permissions before checking if the requested
resource exists, if that's not done one can leak information also
without getting the path returned (as the system will either print
"resource doesn't exists" or "no permissions" respectively)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-15 08:55:58 +02:00
Wolfgang Bumiller 757d0ccc76 warning fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-14 10:37:14 +02:00
Wolfgang Bumiller 032cd1b862 pxar: restore file attributes, improve errors
and use the correct integer types for these operations

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-14 10:25:45 +02:00
Dietmar Maurer f58233a73a src/backup/data_blob_reader.rs: avoid unwrap() - return error instead 2020-07-10 11:28:19 +02:00
Dominik Csapak bfcef26a99 api2/node/status: add fingerprint
and rename get_usage to get_status (since its not usage only anymore)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-10 11:08:53 +02:00
Dominik Csapak ec01eeadc6 refactor CertInfo to tools
we want to reuse some of the functionality elsewhere

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-10 11:08:53 +02:00
Wolfgang Bumiller 660a34892d update proxmox crate to 0.2.0
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 11:08:27 +02:00
Dietmar Maurer d86034afec src/bin/proxmox_backup_client/catalog.rs: fix keyfile handling 2020-07-10 10:36:45 +02:00
Dietmar Maurer 62593aba1e src/backup/manifest.rs: fix signature (exclude 'signature' property) 2020-07-10 10:36:45 +02:00
Wolfgang Bumiller 0eaef8eb84 client: show key path when creating/changing default key
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-10 09:58:24 +02:00
Thomas Lamprecht e39974afbf client: add simple version command
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-10 09:34:07 +02:00
Dietmar Maurer dde18bbb85 proxmox-backup-client benchmark: improve output format 2020-07-10 09:13:52 +02:00
Dietmar Maurer a40e1b0e8b src/server/rest.rs: avoid compiler warning 2020-07-10 09:13:52 +02:00
Stoiko Ivanov c687da9e8e datastore: chown base dir on creation
When creating a new datastore the basedir is only owned by the backup
user if it did not exist beforehand (create_path chowns only if it
creates the directory), and returns false if it did not create the
directory).

This improves the experience when adding a new datastore on a fresh
disk or existing directory (not owned by backup) - backups/pulls can
be run instead of terminating with EPERM.

Tested on my local testinstall with a new disk, and a existing directory:

Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-07-09 18:20:16 +02:00
Dietmar Maurer 4327a8462a proxmox-backup-client benchamrk: add more speed tests 2020-07-09 17:07:22 +02:00
Dietmar Maurer e193544b8e src/server/rest.rs: disable debug logs 2020-07-09 16:18:14 +02:00
Dietmar Maurer 323b2f3dd6 proxmox-backup-client benchmark: add --verbose flag 2020-07-09 16:16:39 +02:00
Dominik Csapak fae11693f0 fix cross process task listing
it does not make sense to check if the worker is running if we already
have an endtime and state

our 'worker_is_active_local' heuristic returns true for non
process-local tasks, so we got 'running' for all tasks that were not
started by 'our' pid and were still running

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-09 15:30:52 +02:00
Wolfgang Bumiller bf7e2a4648 simpler lost+found pattern
the **/ is not required and currently also mistakenly
doesn't match /lost+found which is probably buggy on the
pathpatterns crate side and needs fixing there

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-09 14:06:42 +02:00
Dominik Csapak 3ec99affc8 get_disks: don't fail on zfs_devices
zfs does not have to be installed, so simply log an error and
continue, users still get an error when clicking directly on
ZFS

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-09 13:47:31 +02:00
Dominik Csapak a9649ddc44 disks/zpool_status: add test for pool with special character
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-09 13:37:31 +02:00
Dominik Csapak 4f9096a211 disks/zpool_list: allow some more characters for pool list
not exhaustive of what zfs allows (space is missing), but this
can be done easily without problems

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-09 13:37:31 +02:00
Dominik Csapak c3a4b5e2e1 zpool_list: add tests for special pool names
those names are allowed for zpools

these will fail for now, but it will be fixed in the next commit

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-09 13:37:31 +02:00
Dominik Csapak 7957fabff2 api: add ZPOOL_NAME_SCHEMA and regex
poolnames can containe spaces and some other special characters

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-09 13:37:31 +02:00
Wolfgang Bumiller 20a4e4e252 minor optimization to 'to_canonical_json'
* don't clone hash keys, just use references
* we don't need a String, stick to Vec<u8> and use
  serde_json::to_writer to avoid a temporary strings
  altogether

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-09 13:32:11 +02:00
Dietmar Maurer 4459ffe30e src/backup/manifest.rs: add default toömake it compatible with older backus 2020-07-09 13:25:38 +02:00
Dietmar Maurer dfa517ad6c src/backup/manifest.rs: rename into_string -> to_string
And do not consume self.
2020-07-09 11:28:05 +02:00
Dietmar Maurer 3dacedce71 src/backup/manifest.rs: use serde_json::from_value() to deserialize data
Also modified from_data compute signature ditectly from json.
2020-07-09 09:50:28 +02:00
Dietmar Maurer b53f637914 src/backup/manifest.rs: cleanup signature generation 2020-07-09 09:20:49 +02:00
Dietmar Maurer 2107a5aebc src/backup/manifest.rs: include signature inside the manifest
This is more flexible, because we can choose what fileds we want to sign.
2020-07-08 16:23:26 +02:00
Dietmar Maurer 3638341aa4 src/backup/file_formats.rs: remove signed chunks
We can include signature in the manifest instead (patch will follow).
2020-07-08 16:23:26 +02:00
Wolfgang Bumiller 0351f23ba4 client: introduce --keyfd parameter
This is a more convenient way to pass along the key when
creating encrypted backups of unprivileged containers in PVE
where the unprivileged user namespace cannot access
`/etc/pve/priv`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 13:56:38 +02:00
Dietmar Maurer c1ff544eff src/backup/crypt_config.rs - compute_digest: make it more secure 2020-07-08 12:53:04 +02:00
Wolfgang Bumiller 05389a0109 more xdg cleanup and encryption parameter improvements
Have a single common function to get the BaseDirectories
instance and a wrapper for `find()` and `place()` which
wrap the error with some context.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 10:57:28 +02:00
Wolfgang Bumiller b65390ebc9 client: xdg usage: place() vs find()
place() is used when creating a file, as it will create
intermediate directories, only use it when actually placing
a new file.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 10:57:28 +02:00
Dietmar Maurer 3bad3e6e52 src/client/backup_writer.rs - upload_stream: add crypt_mode 2020-07-08 10:43:28 +02:00
Wolfgang Bumiller 24be37e3f6 client: fix schema to include --crypt-mode parameter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 10:09:15 +02:00
Wolfgang Bumiller 1008a69a13 pxar: less confusing logic
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 09:58:29 +02:00
Wolfgang Bumiller 521a0acb2e DataStore::load_manifest: also return CryptMode
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 09:19:53 +02:00
Wolfgang Bumiller 3b66040de6 add DataBlob::crypt_mode
and move use statements up

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 09:19:53 +02:00
Wolfgang Bumiller af3a0ae7b1 remove CryptMode::sign_only special method
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-08 09:19:53 +02:00
Dietmar Maurer 4e36f78438 src/backup/manifest.rs: support old encrypted property
Just to avoid confusion.
2020-07-08 08:52:27 +02:00
Wolfgang Bumiller f28d9088ed introduce a CryptMode enum
This also replaces the recently introduced --encryption
parameter on the client with a --crypt-mode parameter.

This can be "none", "encrypt" or "sign-only".

Note that this introduces various changes in the API types
which previously did not take the above distinction into
account properly:

Both `BackupContent` and the manifest's `FileInfo`:
    lose `encryption: Option<bool>`
    gain `crypt_mode: Option<CryptMode>`

Within the backup manifest itself, the "crypt-mode" property
will always be set.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-07 15:24:19 +02:00
Dietmar Maurer f91d58e157 src/tools/runtime.rs: implement get_runtime_with_builder 2020-07-07 10:11:04 +02:00
Dietmar Maurer 7443a6e092 src/client/remote_chunk_reader.rs: implement clone for RemoteChunkReader 2020-07-07 07:34:58 +02:00
Wolfgang Bumiller 96ee857752 client: add --encryption boolen parameter
This can be used to explicitly disable encryption even if a
default key file exists in ~/.config.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
Wolfgang Bumiller 887018bb79 client: use default encryption key if it is available
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
Wolfgang Bumiller 9696f5193b client: move key management into separate module
and use api macro for methods and Kdf type

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 14:36:04 +02:00
Wolfgang Bumiller e13c4f66bb minor style & whitespace fixups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-06 10:55:25 +02:00
Thomas Lamprecht 08cb2038bd api: disks: indentation fixup
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-07-03 17:59:30 +02:00
Dietmar Maurer 9de69cdb1a src/bin/proxmox_backup_client/catalog.rs: split out catalog code 2020-07-03 16:45:47 +02:00
Dietmar Maurer e9764238df make ReadChunk not require mutable self.
That way we can reduce lock contentions because we lock for much shorter
times.
2020-07-03 07:37:29 +02:00
Dietmar Maurer cc7995ac40 src/bin/proxmox_backup_client/task.rs: split out task command 2020-07-02 18:04:29 +02:00
Dietmar Maurer 43abba4b4f src/bin/proxmox_backup_client/mount.rs: split out mount code 2020-07-02 17:49:59 +02:00
Dietmar Maurer caea8d611f proxmox-backup-client: add benchmark command
This is just a start, We need to add more useful things here...
2020-07-02 14:01:57 +02:00
Wolfgang Bumiller 7d0754a6d2 pxar: fixup 'vanished-file' logic a bit
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-30 14:41:42 +02:00
Wolfgang Bumiller 5afa0755ea pxar: fix missing newlines in warnings
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-30 14:37:20 +02:00
Dietmar Maurer 2162e2c15d src/api2/admin/datastore.rs: avoid slash in UPID strings 2020-06-30 13:11:22 +02:00
Dietmar Maurer 4059285649 fix typo 2020-06-29 12:59:25 +02:00
Dietmar Maurer 2e079b8bf2 partially revert commit 1f82f9b7b5
do it backwards compatible. Also, code was wrong because FixedIndexWriter
still computed old style csums...
2020-06-29 12:44:45 +02:00
Wolfgang Bumiller a8e2940ff3 pxar: deal with files changing size during archiving
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-26 11:49:51 +02:00
Dominik Csapak 4eb4e94918 fix test output
field separator for pools is always a tab when using -H

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 10:31:11 +02:00
Dietmar Maurer 817bcda848 src/backup/verify.rs: do not stop on server shutdown
This is a read-only task, so there is no need to stop.
2020-06-26 09:45:59 +02:00
Dominik Csapak f6de2c7359 WorkerTask: add warnings and count them
so that we have one level more between errors and OK

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 09:42:11 +02:00
Dominik Csapak 402c8861d8 fix typo
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 09:12:29 +02:00
Dominik Csapak 3f683799a8 improve 'debug' parameter
instead of checking on '1' or 'true', check that it is there and not
'0' and 'false'. this allows using simply

https://foo:8007/?debug

instead of

https://foo:8007/?debug=1

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-26 09:12:14 +02:00
Dietmar Maurer 1f82f9b7b5 src/backup/index.rs: add compute_csum
And use it for fixed and dynamic index. Please note that this
changes checksums for fixed indexes, so restore older backups
will fails now (not backward compatible).
2020-06-26 09:00:34 +02:00
Dietmar Maurer fe3e65c3ea src/api2/backup.rs: call register_chunk in previous download api 2020-06-26 08:22:46 +02:00
Dietmar Maurer fdaab0df4e src/backup/index.rs: add chunk_info method 2020-06-26 08:14:45 +02:00
Dietmar Maurer b957aa81bd update backup api for incremental backup
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-26 07:17:08 +02:00
Dietmar Maurer 8ea00f6e49 allow to abort verify jobs
And improve job description rendering on gui.
2020-06-25 12:56:36 +02:00
Stefan Reiter e22f4882e7 extract create_download_response API helper
and put it into a new "api2::helpers" module.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-25 11:57:37 +02:00
Dietmar Maurer 355c055e81 src/bin/proxmox-backup-manager.rs: implement verify 2020-06-24 13:35:21 +02:00
Dietmar Maurer c2009e5309 src/api2/admin/datastore.rs: add verify api 2020-06-24 13:35:21 +02:00
Dietmar Maurer 23f74c190e src/backup/backup_info.rs: impl Display for BackupGroup 2020-06-24 13:35:21 +02:00
Wolfgang Bumiller a6f8728339 update to pxar 0.1.9, update ReadAt implementations
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-24 11:57:12 +02:00
Stefan Reiter facd9801cf add incremental backup support
To support incremental backups (where not all chunks are sent to the
server), a new parameter "reuse-csum" is introduced on the
"create_fixed_index" API call. When set and equal to last backups'
checksum, the backup writer clones the data from the last index of this
archive file, and only updates chunks it actually receives.

In incremental mode some checks usually done on closing an index cannot
be made, since they would be inaccurate.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-24 10:01:25 +02:00
Dietmar Maurer 21302088de remove debug println 2020-06-24 09:15:13 +02:00
Stefan Reiter 8268c9d161 fix overflow panic during upload
if *only* data chunks are registered (high chance during incremental
backup), then chunk_count might be one lower then upload_stat.count
because of the zero chunk being unconditionally uploaded but not used.
Thus when subtracting the two, an overflow would occur.

In general, don't let the client make the server panic, instead just set
duplicates to 0.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-06-24 09:07:22 +02:00
Dominik Csapak b91b7d9ffd api2/node/disks/zfs: check if default zfs mount path exists
and if it does bail, because otherwise we would get an
error on mounting and have a zpool that is not imported
and disks that are used

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:22:39 +02:00
Dominik Csapak d33d8f4e6a api2/admin/datastore: add pxar-file-download api call
streams a file from a pxar file of an unencrypted backup

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:21:15 +02:00
Dominik Csapak 5b1cfa01f1 api2/admin/datastore: add 'catalog' api call
returns the dir listing of the given filepath of the backup snapshot
the filepath has to be base64 encoded or 'root'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:16:12 +02:00
Dominik Csapak 05d18b907a add From<&DirEntryAttribute to CatalogEntryType and make it pub(crate)
we want to get a string representation of the DirEntryAttribute
like 'f' for file, etc. and since we have such a mapping already
in the CatalogEntryType, use that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:08:50 +02:00
Dominik Csapak e44fe0c9f5 derive Clone for the LocalChunkReader
this will be necessary for accessing local pxar behind didx files

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:07:28 +02:00
Dominik Csapak 4cf0ced950 add LocalDynamicReadAt
mostly copied from BufferedDynamicReadAt from proxmox-backup-client
but the reader is wrapped in an Arc in addition to the Mutex

we will use this for local access to a pxar behind a didx file

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-24 07:05:31 +02:00
Dietmar Maurer 60f9a6ea8f src/backup/datastore.rs: add new helpers to load blobs and verify chunks 2020-06-24 06:58:14 +02:00
Dietmar Maurer 1090fd4424 src/backup/data_blob.rs: cleanup - improve code reuse 2020-06-24 06:56:48 +02:00
Dietmar Maurer 92c3fd2e22 src/backup/chunk_store.rs: allow to read name()
This is helpful for logging ...
2020-06-24 06:54:21 +02:00
Dietmar Maurer d6d3b353be cleanup: implement FromStr for BackupGroup 2020-06-23 08:16:56 +02:00
Dietmar Maurer a67f7d0a07 cleanup: implement FromStr for BackupDir 2020-06-23 08:09:52 +02:00
Dietmar Maurer c8137518fe src/bin/proxmox_backup_manager/disk.rs: add renderer for wearout
So that we display the same value as the GUI.
2020-06-23 07:44:09 +02:00
Dominik Csapak cbef49bf4f remove absolute paths when executing binaries
we set the paths manually, so this is ok

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-23 07:09:06 +02:00
Dominik Csapak 0b99e5aebc remove debug prints
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-23 06:33:58 +02:00
Dominik Csapak f386f512d0 add AsyncReaderStream
and replace AsyncIndexReader's stream implementation with that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-23 06:33:31 +02:00
Dietmar Maurer 3ddb14889a src/tools/daemon.rs: reopen STDOUT/STDERR journald streams to get correct PID in logs 2020-06-22 13:06:53 +02:00
Wolfgang Bumiller c08fac4d69 tools::daemon: sync with child after MainPid message
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-22 10:58:04 +02:00
Wolfgang Bumiller c40440092d tools: add socketpair helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-22 10:49:56 +02:00
Wolfgang Bumiller dc2ef2b54f tools::daemon: fetch exe name in the beginning
We get the path to our executable via a readlink() on
"/proc/self/exe", which appends a " (deleted)" during
package reloads.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-22 10:31:54 +02:00
Dietmar Maurer f28cfb322a avoid compiler warnings 2020-06-20 07:24:02 +02:00
Dietmar Maurer 3bbe291c51 zpool_status.rs - indented_list_to_tree: do not set name property
This is no necessary. We only touch/set 'children' and 'leaf' properties.
2020-06-20 07:19:25 +02:00
Dietmar Maurer 42d19fdf69 src/api2/node/disks/zfs.rs: always set pool name 2020-06-20 07:15:32 +02:00
Dietmar Maurer 215968e033 src/tools/disks/zpool_status.rs: add 'leaf' attribute to root node, rename 'prev' into 'parent' 2020-06-20 06:49:06 +02:00
Dietmar Maurer eddd1a1b9c src/tools/disks/zpool_status.rs: move use clause top of file 2020-06-20 06:17:22 +02:00
Dietmar Maurer d2ce211899 fixup for previous commit 2020-06-20 06:15:26 +02:00
Dietmar Maurer 1cb46c6f65 src/tools/disks/zpool_status.rs - cleanup: use struct StackItem instead of tuple 2020-06-19 18:58:57 +02:00
Dietmar Maurer 5d88c3a1c8 src/tools/disks/zpool_status.rs: remove unnecessary checks
Thos things can never happen, so simply use unwrap().
2020-06-19 18:27:39 +02:00
Dietmar Maurer 07fb504943 src/tools/disks/zpool_status.rs: simplify code by using serde_json::to_value 2020-06-19 17:51:13 +02:00
Dietmar Maurer f675c5e978 src/tools/disks/zpool_status.rs - add all attributes to the tree 2020-06-19 16:55:28 +02:00
Wolfgang Bumiller 4e37d9ce67 add general indented_list_to_tree implementation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-19 14:37:40 +02:00
Wolfgang Bumiller e303077132 lru_cache: restrict and annotate Send impl
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-19 09:37:34 +02:00
Dominik Csapak 6ef9bb59eb api2/admin/datastore: add download-decoded endpoint
similar to 'download', but streams the decoded file
when it is not encrypted

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:39:15 +02:00
Dominik Csapak eeaa2c212b impl Sync for DataBlobReader
this is safe for the reason explained in the comment

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:37:47 +02:00
Dominik Csapak 4a3adc3de8 add AsyncIndexReader
implements AsyncRead as well as Stream for an IndexFile and a store
that implements AsyncReadChunk

we can use this to asyncread or stream the content of a FixedIndex or
DynamicIndex

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:32:33 +02:00
Dominik Csapak abdb976340 add Display trait to BackupDir
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 08:28:35 +02:00
Dominik Csapak 3b62116ce6 implement AsyncReadChunk for LocalChunkReader
same as the sync ReadChunk but uses tokio::fs::read instead
of file_get_contents

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:54:23 +02:00
Dominik Csapak 1c090810f5 api2/admin/datastore/snapshos: show encrypted and size info per file
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:39:56 +02:00
Dominik Csapak e181d2f6da add encrypted info to Manifest
we want to save if a file of a backup is encrypted, so that we can
* show that info on the gui
* can later decide if we need to decrypt the backup

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:35:39 +02:00
Dominik Csapak 16021f6ab7 use the existing async method for read_raw_chunk
does the same, except the manual drop, but thats handled there by
letting the value go out of scope

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 07:23:17 +02:00
Dominik Csapak ba694720fc api2/admin/datastore: log stream error during file download
the client cannot get an error during an chunked http transfer, so at
least log it server side

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 06:58:55 +02:00
Dominik Csapak bde8e243cf remove unsafe copy code
copy_nonoverlapping is basically a memcpy which can also be done
via copy_from_slice which is not unsafe
(copy_from_slice uses copy_nonoverlapping internally)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-19 06:56:15 +02:00
Dietmar Maurer 3352ee5656 parse_zpool_status_field: handle tabs without copying input 2020-06-18 19:40:01 +02:00
Dietmar Maurer b29cbc414d parse_zpool_status_vdev: consider tabs as 8 spaces 2020-06-18 18:38:56 +02:00
Dietmar Maurer 026dc1d11f src/api2/node/disks/zfs.rs: add zpool_details api 2020-06-18 15:04:46 +02:00
Dietmar Maurer 9438aca6c9 src/tools/disks/zpool_status.rs: improve parser 2020-06-18 14:55:22 +02:00
Dietmar Maurer 547f0c97e4 src/tools/nom.rs: new helper parse_complete_line() for single line parsers
Like parse_complete(), but generates simpler error messages.
2020-06-18 12:57:55 +02:00
Dietmar Maurer 177a2de992 src/tools/nom.rs: move nom helpers into separate file 2020-06-18 12:41:13 +02:00
Dietmar Maurer 0686b1f4db src/tools/disks/zpool_list.rs: split code into separate file 2020-06-18 10:31:07 +02:00
Dietmar Maurer 0727e56a06 src/tools/disks/zpool_status.rs: parse zpool status output 2020-06-18 10:23:15 +02:00
Dietmar Maurer 2fd3d57490 src/tools/disks/zfs.rs: rename ZFSPoolStatus into ZFSPoolInfo, fix error message 2020-06-17 09:08:26 +02:00
Dietmar Maurer 3f851d1321 src/api2/node/disks/directory.rs: add early check if disk is unused 2020-06-17 08:31:11 +02:00
Dietmar Maurer 1aef491e24 src/bin/proxmox_backup_manager/disk.rs: add cli to create mounted disks 2020-06-17 08:07:54 +02:00
Dietmar Maurer d0eccae37d avoid compiler warning 2020-06-17 08:07:42 +02:00
Dietmar Maurer a34154d900 src/tools/disks/zfs.rs: cleanup parse_pool_header 2020-06-17 07:47:11 +02:00
Dietmar Maurer c2cc32b4dd src/tools/disks/zfs.rs: add more parser tests 2020-06-17 07:38:19 +02:00
Dietmar Maurer 46405fa35d src/tools/disks/zfs.rs: add comment 2020-06-17 07:14:26 +02:00
Dietmar Maurer 66af7f51bc src/tools/disks/zfs.rs: make zfs list parser private 2020-06-17 07:00:54 +02:00
Dietmar Maurer c72ccd4e33 src/tools/disks/zfs.rs: add regression tests for parse_zfs_list 2020-06-16 18:14:35 +02:00
Dietmar Maurer 902b2cc278 src/tools/disks/zfs.rs: simplify code 2020-06-16 17:51:17 +02:00
Wolfgang Bumiller 8ecd7c9c21 move api dump binaries back to src/bin for package building
they're required for docs

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 14:48:01 +02:00
Dietmar Maurer fb5a066500 src/api2/node/disks.rs: expose directory api 2020-06-16 13:36:32 +02:00
Wolfgang Bumiller d19c96d507 move test binaries to examples/
These aren't installed and are only used for manual testing,
so there's no reason to force them to be built all the time.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 13:32:24 +02:00
Dietmar Maurer 929a13b357 src/api2/node/disks/zfs.rs: add zpool api 2020-06-16 13:25:53 +02:00
Dietmar Maurer 36c65ee0b0 src/tools/disks/zfs.rs: cleanup (rename usage properties)
And allow to parse zpool list output without -v flag.
2020-06-16 13:25:53 +02:00
Dietmar Maurer 3378fd9fe5 src/tools/disks/zfs.rs: parse more infos (dedup, fragmentation, health) 2020-06-16 13:25:53 +02:00
Dietmar Maurer 58c51cf3d9 avoid compiler warnings 2020-06-16 13:25:53 +02:00
Dietmar Maurer 5509b199fb use new run_command helper 2020-06-16 13:25:53 +02:00
Wolfgang Bumiller bb59df9134 catalog: don't panic on invalid file mtimes
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 11:25:54 +02:00
Wolfgang Bumiller 2564b0834f fix file timestamps in catalog
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 11:25:31 +02:00
Wolfgang Bumiller 9321bbd1f5 pxar: fix missing subdirectories in catalogs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 11:04:56 +02:00
Wolfgang Bumiller 4264e52220 reuse some extractor code in catalog shell
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 10:54:54 +02:00
Wolfgang Bumiller 6988b29bdc use O_EXCL when creating files during extraction
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 10:33:27 +02:00
Wolfgang Bumiller 98c54240e6 pxar: make extractor state more reusable
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 10:32:13 +02:00
Wolfgang Bumiller d30c192589 AsyncReadChunk: require Send
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-16 09:50:29 +02:00
Dietmar Maurer ac7513e368 src/tools.rs: add setup_safe_path_env() 2020-06-15 10:38:30 +02:00
Dietmar Maurer fbbcd85839 src/api2/node/disks/directory.rs: implement add-datastore feature 2020-06-15 10:01:50 +02:00
Wolfgang Bumiller 7a6b549270 dynamic index: make it hard to mess up endianess
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-15 09:19:35 +02:00
Wolfgang Bumiller 0196b9bf5b remove unnecessary .into
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 14:57:58 +02:00
Dominik Csapak 20b3094bcb api2/status: add task list endpoint
for now mostly copy/paste from nodes/nodename/tasks
(without the parameters)
but we should replace the 'read_task_list' with a method
that gives us the tasks since some timestamp

so that we can get a longer list of tasks than for the node
(we could of course embed this then in the nodes/node/task api call and
remove this again as long as the api is not stable)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 14:50:17 +02:00
Dominik Csapak df528ee6fa implement From<TaskListInfo> for TaskListItem
and use it where its convenient

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 14:50:12 +02:00
Wolfgang Bumiller 57e50fb906 use new Mmap helper for dynamic index
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 13:57:56 +02:00
Wolfgang Bumiller 3d571d5509 some internal combinator-influenced api cleanup
The download methods used to take the destination by value
and return them again, since this was required when using
combinators before we had `async fn`.
But this is just an ugly left-over now.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:46:42 +02:00
Wolfgang Bumiller 8e6e18b77c client: make dump_image async, use async chunk reader
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:40:18 +02:00
Wolfgang Bumiller 4d16badf6f add an AsyncReadChunk trait
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:38:21 +02:00
Wolfgang Bumiller a609cf210e more cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 11:01:04 +02:00
Wolfgang Bumiller 1498659b4e cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 10:59:34 +02:00
Wolfgang Bumiller 4482f3fe11 pxar, acl: cleanup acl helper usage
use NixPath for Acl::set_file to avoid memduping the c
string

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 10:52:18 +02:00
Dominik Csapak 5d85847f91 client: only start catalog upload if we have one
else we start a dynamic writer and never close it, leading to a backup error

this fixes an issue with backing up vm templates
(and possibly vms without disks)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 10:38:25 +02:00
Dominik Csapak 476b4acadc BackupEnvironment: do not set finished flag prematurely
we check if all dynamic_writers are closed and if the backup contains
any valid files, we can only mark the backup finished after those
checks, else the backup task gets marked as OK, even though it
is not finished and no cleanups run

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 10:37:52 +02:00
Wolfgang Bumiller cf1bd08131 pxar: fcaps in fuse
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 10:37:48 +02:00
Dominik Csapak ec8f042459 api2/status: use new rrd::extract_cached_data
and drop the now unused extract_lists function

this also fixes a bug, where we did not add the datastore to the list at
all when there was no rrd data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:16 +02:00
Dominik Csapak 431cc7b185 rrd: move creation of serde value into api
there is now a 'extract_cached_data' which just returns
the data of the specified field, and an api function that converts
a list of fields to the correct serde value

this way we do not have to create a serde value in rrd/cache.rs
(makes for a better interface)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:14 +02:00
Dominik Csapak e693818afc refactor time functions to tools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:10 +02:00
Wolfgang Bumiller 3d68536fc2 pxar: support .pxareclude files, error report updates
Report vanished files (instead of erroring out on them),
also only warn about files inaccessible due to permissions
instead of bailing out.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 12:22:18 +02:00
Wolfgang Bumiller 26e78a2efb downgrade some FIXMEs to TODOs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 11:09:23 +02:00
Wolfgang Bumiller 5444fa940b turn pxar::flags into bitflags, pxar::Flags
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 11:05:53 +02:00
Dietmar Maurer d4f2397d4c add api to format disks and create datastores 2020-06-10 11:03:36 +02:00
Wolfgang Bumiller fab2413741 catalog: remove unused SenderWriter
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 10:42:42 +02:00
Dietmar Maurer 669c137fec src/tools/systemd.rs: implement daemon_reload, start_unit, stop_unit and enable_unit 2020-06-10 08:56:04 +02:00
Wolfgang Bumiller fc6047fcb1 pxar: don't skip list+found by default
This used to be default-off and was accidentally set to
on-by-default with the pxar crate update.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 08:53:10 +02:00
Wolfgang Bumiller 3014088684 pxar: sort .pxareclude-cli file and fix its mode
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 08:49:14 +02:00
Dietmar Maurer 144006fade src/tools.rs: add new run_command helper 2020-06-10 07:16:47 +02:00
Dietmar Maurer b9cf6ee797 src/tools/systemd/types.rs: add Mount config 2020-06-09 18:47:10 +02:00
Wolfgang Bumiller cdde66d277 statistics: covariance(): avoid allocation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-09 13:57:27 +02:00
Wolfgang Bumiller 239e49f927 pxar: create .pxarexclude-cli file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-09 13:17:59 +02:00
Dominik Csapak bda48e04da api2: add status/datastore-usages api call
returns a list of the datastores and their usages, a list of usages of
the past month (for the gui) and an estimation of when its full
(using the linear regression)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:20:25 +02:00
Dominik Csapak ba97479848 add statistics module
provides some basic statistics functions (sum, mean, etc.)
and a function to return the parameters of the linear regression of
two variables

implemented using num_traits to be more flexible for the types

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:19:51 +02:00
Dominik Csapak 6cad8ce4ce rrd: add 'extract_lists'
this is an interface to simply get the Vec<Option<f64>> out of rrd
without going through serde values

we return a list of timestamps and a HashMap with the lists we could find
(otherwise it is not in the map)

if no lists could be extracted, the time list is also empty

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:19:06 +02:00
Dominik Csapak 33070956af let disk_usage return StorageStatus and use it for datastores/nodes
disk_usage returned the same values as defined in StorageStatus,
so simply use that

with that we can replace the logic of the datastore status with that
function and also use it for root disk usage of the nodes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:05:39 +02:00
Dietmar Maurer da84cc52f4 src/tools/systemd.rs: implement escape_unit and unescape_unit 2020-06-09 11:52:06 +02:00
Wolfgang Bumiller 2179359f40 move src/pxar.rs -> src/pxar/mod.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-09 10:49:59 +02:00
Dietmar Maurer 9bb161c881 src/tools/disks.rs: add create_single_linux_partition and create_file_system 2020-06-08 17:43:01 +02:00
Dietmar Maurer 297e600730 cleanup comment 2020-06-08 17:43:01 +02:00
Dietmar Maurer ed7b3a7de2 src/tools/disks.rs: add get_fs_uuid helper 2020-06-08 17:43:01 +02:00
Dietmar Maurer 0f358204bd src/tools/disks.rs: add helper to list partitions 2020-06-08 17:43:01 +02:00
Dietmar Maurer ca6124d5fa src/tools/disks.rs: make helpers pub
So that I can use them with my test code.
2020-06-08 17:43:01 +02:00
Wolfgang Bumiller 7eacdc765b pxar: split assert_relative_path
the check for a single component is only required in the dir
stack atm

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 15:02:52 +02:00
Wolfgang Bumiller c443f58b09 switch to external pxar and fuse crates
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 13:56:58 +02:00
Wolfgang Bumiller 1e3d9b103d xattr: make xattr_name_fcaps public
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 13:56:58 +02:00
Wolfgang Bumiller 386990ba09 tools: add file_get_non_comment_lines
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 13:56:58 +02:00
Dietmar Maurer bc853b028f src/tools/disks.rs: cleanup, remove unused DiskUse bitflag type 2020-06-08 09:43:07 +02:00
Dietmar Maurer d406de299b src/tools/disks.rs: use dev_t to index zfs/lvm device sets 2020-06-08 09:01:34 +02:00
Dietmar Maurer dfb31de8f0 proxmox_backup_manager disk list: display gpt column 2020-06-08 07:35:44 +02:00
Dietmar Maurer 7c3aa258f8 src/tools/disks/zfs.rs: allow empty zpool list output 2020-06-08 07:23:04 +02:00
Dietmar Maurer 044055062c src/tools/disks.rs: new helper to reread partition table 2020-06-08 07:22:06 +02:00
Dietmar Maurer 2b388026f8 src/api2/node/disks.rs: correctly use disk_by_name insteadf of disk_by_node 2020-06-08 07:20:59 +02:00
Dietmar Maurer 707974fdb3 src/api2/node/disks.rs: implement initgpt API 2020-06-07 10:30:34 +02:00
Dietmar Maurer 9069debcd8 src/api2/types.rs: define BLOCKDEVICE_NAME_SCHEMA 2020-06-07 07:20:25 +02:00
Dietmar Maurer fa2bdc1309 src/config/acl.rs: add /system/disks to valid acl paths 2020-06-06 15:48:15 +02:00
Dietmar Maurer 8e40aa63c1 src/bin/proxmox-backup-manager.rs: add disk subcommand 2020-06-06 15:40:28 +02:00
Dietmar Maurer d2522b2db6 src/tools/disks.rs: fix disk size, add completion helper 2020-06-06 15:39:25 +02:00
Dietmar Maurer ce8e3de401 move disks api to /node/<node>/disks 2020-06-06 14:43:36 +02:00
Dietmar Maurer 7fa2779559 src/api2/disks.rs: implement smart api 2020-06-06 12:23:11 +02:00
Dietmar Maurer 042afd6e52 src/tools/disks.rs: new helper disk_by_name() 2020-06-06 12:22:38 +02:00
Dietmar Maurer ff30caeaf8 src/api2/disks.rs - list-disks: add usage-type filter 2020-06-06 11:48:58 +02:00
Dietmar Maurer 553cd12ba6 src/api2/disks.rs: start disks api 2020-06-06 11:38:47 +02:00
Dietmar Maurer de1e1a9d95 src/tools/disks.rs: use api macro so that we can use those types with the api 2020-06-06 11:37:24 +02:00
Dietmar Maurer 91960d6162 src/tools/disks.rs - get_disks: query smart status 2020-06-06 09:18:20 +02:00
Dietmar Maurer 4c24a48eb3 src/tools/disks/smart.rs: use model.to_string_lossy() to simplify code 2020-06-06 09:05:22 +02:00
Dietmar Maurer 484e761dab src/tools/disks/smart.rs: try to get correct wearout for ATA devices 2020-06-06 09:01:15 +02:00
Dietmar Maurer 059b7a252e src/tools/disks/smart.rs - get_smart_data: use &Disk instead of &str
So that we can query other device infos easily (model, vendor, ..)
2020-06-06 08:24:58 +02:00
Dietmar Maurer eb80aac288 src/tools/disks/smart.rs: parse output from smartctl 2020-06-05 18:30:06 +02:00
Dietmar Maurer c26aad405f src/tools/disks.rs: implement get_disks (similar to the one in PVE::Diskmanage)
But no ceph support for now. Also no support for old cciss block devices.
2020-06-05 10:33:53 +02:00
Dietmar Maurer f03a0e509e src/tools/disks.rs; use correct subdir to check holders 2020-06-05 10:33:53 +02:00
Dietmar Maurer 4c1e8855cc src/tools/disks.rs: fix disk type detection, remove newline from vendor string 2020-06-05 08:09:52 +02:00
Thomas Lamprecht a720894ff0 rrd: fix off-by-one in save interval calculation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-04 10:30:47 +02:00
Dominik Csapak a95a3fb893 fix csum calculation of not 'chunk_size' aligned images
the last chunk does not have to be as big as the chunk_size,
just use the already available 'chunk_end' function which does the
correct thing

this fixes restoration of images whose sizes are not a multiple of
'chunk_size' as well

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-04 10:18:30 +02:00
Dietmar Maurer 620911b426 src/tools/disks/lvm.rs: implement get_lvm_devices() 2020-06-04 09:12:19 +02:00
Dietmar Maurer 5c264c8d80 src/tools/disks.rs: add/use get_partition_type_info 2020-06-04 07:48:22 +02:00
Dominik Csapak 8d78589969 improve display of 'next run' for sync jobs
if the last sync job is too far in the past (or there was none at all
for now) we run it at the next iteration, so we want to show that

we now calculate the next_run by using either the real last endtime
as time or 0

then in the frontend, we check if the next_run is < now and show 'pending'
(we do it this way also for replication on pve)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-04 07:03:54 +02:00
Dominik Csapak eed8a5ad79 tools/systemd/time: fix compute_next_event for weekdays
two things were wrong here:
* the range (x..y) does not include y, so the range
  (day_num+1..6) goes from (day_num+1) to 5 (but sunday is 6)

* WeekDays.bits() does not return the 'day_num' of that day, but
  the bit value (e.g. 64 for SUNDAY) but was treated as the index of
  the day of the week
  to fix this, we drop the map to WeekDays and use the 'indices'
  directly

this patch makes the test work again

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-04 07:02:33 +02:00
Dominik Csapak 538b9c1c27 systemd/time: add tests for all weekdays
this fails for now

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-04 07:02:23 +02:00
Thomas Lamprecht 55919bf141 verify_file: add missing closing parenthesis in error message
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-03 19:10:01 +02:00
Dietmar Maurer 456ad0c478 src/tools/disks/zfs.rs: add parser for zpool list output 2020-06-03 12:16:08 +02:00
Dietmar Maurer c48aa39f3b src/bin/proxmox-backup-client.rs: implement quite flag 2020-06-03 10:11:37 +02:00
Thomas Lamprecht 2d32fe2c04 client restore: don't add server file ending if already specified
If one executes a client command like
 # proxmox-backup-client files <snapshot> --repository ...
the files shown have already the '.fidx' or '.blob' file ending, so
if a user would just copy paste that one the client would always add
.blob, and the server would not find that file.

So avoid adding file endings if it is already a known OK one.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-03 07:03:55 +02:00
Thomas Lamprecht dc155e9bd7 client restore: factor out archive/type parsing
will be extended in a next patch.

Also drop a dead else branch, can never get hit as we always add
.blob as fallback

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-03 07:03:12 +02:00
Dietmar Maurer 4e14781aec fix typo 2020-06-03 06:59:43 +02:00
Wolfgang Bumiller a595f0fee0 client: improve connection/new fingerprint query
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-02 10:40:31 +02:00
Thomas Lamprecht add5861e8d typo fixes all over the place
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-30 16:39:08 +02:00
Dietmar Maurer 1610c45a86 src/client/pull.rs: also download client.log.blob 2020-05-30 14:51:33 +02:00
Dietmar Maurer b2387eaa45 avoid compiler warnings 2020-05-30 14:05:33 +02:00
Dietmar Maurer 96d65fbcd0 cleanup: define/use const for predefined blob file names. 2020-05-30 14:04:15 +02:00
Dietmar Maurer 7cc3473a4e src/client/backup_specification.rs: split code into extra file 2020-05-30 10:54:38 +02:00
Dietmar Maurer 4856a21836 src/client/pull.rs: more verbose logging 2020-05-30 08:12:43 +02:00
Dominik Csapak 04b0ca8b59 add owner to group and snapshot listings
while touching it, make columns and tbar in DataStoreContent.js
declarative members and remove the (now) unnecessary initComponent

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-30 07:24:12 +02:00
Dominik Csapak d43f86f3f3 api2: add admin/sync endpoint
this returns the list of syncjobs with status, as opposed to
config/sync (which is just the config)

also adds an api call where users can run the job manually under
/admin/sync/$ID/run

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:31:32 +02:00
Dominik Csapak 997d7e19fc config/sync: add SyncJobStatus Struct/Schema
contains the config + status

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:29:39 +02:00
Dominik Csapak c67b1fa72f syncjob: change worker type for sync jobs
'sync' is used for manually pulling a remote datastore
changing it for a scheduled sync to 'syncjob' so that we can
differentiate between both types of syncs

this also adds a seperate task description for it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:28:04 +02:00
Dominik Csapak 268687ddf0 api2/pull: refactor priv checking and creating pull parameters
we want to reuse those in the api call for manually running a sync job

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:27:43 +02:00
Dominik Csapak 426c1e353b api2/config/sync: fix id parameter
'name' is not the correct parameter for get/post

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:24:54 +02:00
Dominik Csapak 2888b27f4c create SYNC_SCHEDULE_SCHEMA to adapt description for sync jobs
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:24:25 +02:00
Dietmar Maurer 934f5bb8ac src/bin/proxmox-backup-proxy.rs: cleanup, move code to src/tools/disks.rs
And simplify find_mounted_device by using stat.st_dev
2020-05-29 11:13:36 +02:00
Dominik Csapak 9857472211 fix removing of remotes
we have to save the remote config after removing the section

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 10:48:26 +02:00
Dietmar Maurer 013fa7bbcb rrd: reduce io by saving data only once a minute 2020-05-29 09:16:13 +02:00
Dietmar Maurer a8d7033cb2 src/bin/proxmox-backup-proxy.rs: add test if last prune job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer 04ad7bc436 src/bin/proxmox-backup-proxy.rs: test if last sync job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer 77ebbefc1a src/server/worker_task.rs: make worker_is_active_local pub 2020-05-29 08:06:48 +02:00
Dietmar Maurer 750252ba2f src/tools/systemd/time.rs: add test for "daily" schedule 2020-05-29 07:52:09 +02:00
Dietmar Maurer dc58194ebe src/bin/proxmox-backup-proxy.rs: use correct id to lookup sync jobs 2020-05-29 07:50:59 +02:00
Thomas Lamprecht 090decbe76 BACKUP_REPO_URL_REGEX: move to api2::types and allow all valid data store names
The repo URL consists of
* optional userid
* optional host
* datastore name

All three have defined regex or format, but none of that is used, so
for example not all valid datastore names are accepted.

Move definition of the regex over to api2::types where we can access
all required regexes easily.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-29 06:29:23 +02:00
Thomas Lamprecht c32186595e api2::types: factor out USER_ID regex
allows for better reuse in a next patch

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-29 06:27:38 +02:00
Dietmar Maurer c94e1f655e rrd stats: improve io delay stats 2020-05-28 19:12:13 +02:00
Dietmar Maurer 91e5bb49f5 src/bin/proxmox-backup-proxy.rs: simplify code
and gather all stats for the root disk
2020-05-28 12:30:54 +02:00
Dietmar Maurer 547e3c2f6c src/tools/disks/zfs.rs: use wtime + rtime (wait + run time) 2020-05-28 11:45:34 +02:00
Dietmar Maurer 25c550bc28 src/bin/proxmox-backup-proxy.rs: gather zpool io stats 2020-05-28 10:09:13 +02:00
Dietmar Maurer 0146133b4b src/tools/disks/zfs.rs: helper to read zfs pool io stats 2020-05-28 10:07:52 +02:00
Dietmar Maurer 3eeba68785 depend on proxmox 0.1.38, use new fs helper functions 2020-05-28 10:06:44 +02:00
Dominik Csapak f5056656b2 use the sync id for the scheduled sync worker task
this way, multiple sync jobs with the same local store, can get scheduled

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-28 06:26:03 +02:00
Dominik Csapak 8c87743642 fix 'remove_vanished' cli arg again
since the target side wants this to be a boolean and
serde interprets a None Value as 'null' we have to only
add this when it is really set via cli

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-28 06:25:30 +02:00
Dominik Csapak 05d755b282 fix inserting of worker tasks
when starting a new task, we do two things to keep track of tasks
(in that order):
* updating the 'active' file with a list of tasks with
  'update_active_workers'
* updating the WORKER_TASK_LIST

the second also updates the status of running tasks in the file by
checking if it is still running by checking the WORKER_TASK_LIST

since those two things are not locked, it can happend that
we update the file, and before updating the WORKER_TASK_LIST,
another thread calls update_active_workers and tries to
get the status from the task log, which won't have any data yet
so the status is 'unknown'

(we do not update that status ever, likely for performance reasons,
so we have to fix this here)

by switching the order of the two operations, we make sure that only
tasks reach the 'active' file which are inserted in the WORKER_TASK_LIST

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-28 06:24:42 +02:00
Dietmar Maurer 143b654550 src/tools.rs - command_output: add parameter to check exit code 2020-05-27 07:25:39 +02:00
Dietmar Maurer 97fab7aa11 src/tools.rs: new helper to handle command_output (std::process::Output) 2020-05-27 06:53:25 +02:00
Dominik Csapak 40dc103103 fix cli pull api call
there is no 'delete' parameter, only 'remove-vanished', so fix that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 12:39:19 +02:00
Dominik Csapak de4db62c57 remotes: save passwords as base64
to avoid having arbitrary characters in the config (e.g. newlines)
note that this breaks existings configs

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 12:38:06 +02:00
Dietmar Maurer 1a0d3d11d2 src/api2/admin/datastore.rs: add rrd api 2020-05-26 12:26:14 +02:00
Dietmar Maurer 8c03041a2c src/bin/proxmox-backup-proxy.rs: gather block device stats on datastore 2020-05-26 11:20:59 +02:00
Dietmar Maurer 3fcc4b4e5c src/tools/disks.rs: add helper to read block device stats 2020-05-26 11:20:22 +02:00
Dietmar Maurer 3ed07ed2cd src/tools/disks.rs: export read_sys 2020-05-26 09:49:13 +02:00
Dominik Csapak 83fd4b3b1b remote: try to use Struct for api
with a catch: password is in the struct but we do not want it to return
via the api, so we only 'serialize' it when the string is not empty
(this can only happen when the format is not checked by us, iow.
when its returned from the api) and setting it manually to ""
when we return remotes from the api

this way we can still use the type but do not return the password

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 08:55:07 +02:00
Dominik Csapak 5dcdcea293 api2/config/remote: remove password from read_remote
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 08:49:12 +02:00
Dominik Csapak 99f443c6ae api2/config/remote: lock and use digest for removal
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 08:48:45 +02:00
Dominik Csapak 4f966d0592 api2/config/remote: use rpcenv for digest for read_remote
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 08:48:28 +02:00
Dominik Csapak db0c228719 config/remote: add 'name' to Remote struct
and use it as section id, like with User

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-26 08:48:05 +02:00
Dietmar Maurer 309ef20d6d src/bin/proxmox-backup-proxy.rs: simplify code 2020-05-25 16:20:32 +02:00
Dietmar Maurer d0833a70f7 src/bin/proxmox-backup-proxy.rs: gather datastore usage stats 2020-05-25 16:20:32 +02:00
Dietmar Maurer d8dc281992 www/DataStoreStatus.js: display loadavg stats 2020-05-25 11:54:15 +02:00
Dietmar Maurer 2c66a590c0 src/bin/proxmox-backup-proxy.rs: gather iowait stats 2020-05-25 11:54:15 +02:00
Dietmar Maurer 485841da2c src/bin/proxmox-backup-proxy.rs: gather loadavg stats 2020-05-25 11:40:20 +02:00
Dietmar Maurer 3e930f2bdc www/DataStoreStatus.js: display root disk stats 2020-05-25 11:34:24 +02:00
Dietmar Maurer dd15c0aa3b src/bin/proxmox-backup-proxy.rs: gather root disk stats 2020-05-25 11:10:07 +02:00
Dietmar Maurer c1b24fbf0b www/DataStoreStatus.js: display swap stats 2020-05-25 10:39:54 +02:00
Dietmar Maurer 3f23b17298 src/rrd/rrd.rs: do not wrap error and return ErrorKind::NotFound 2020-05-25 10:30:04 +02:00
Dietmar Maurer c25c9d8dd1 src/bin/proxmox-backup-proxy.rs: gather swap usage stats 2020-05-25 10:25:58 +02:00
Dietmar Maurer 84dc6adcc1 src/rrd/cache.rs: display/log error when RRD load fails 2020-05-25 10:18:53 +02:00
Dietmar Maurer 0c4344650d src/rrd/rrd.rs: store/verify magic number 2020-05-25 09:21:54 +02:00
Dietmar Maurer 4f9513996c src/bin/proxmox-backup-proxy.rs: use block_in_place for rrd update 2020-05-25 08:30:59 +02:00
Dietmar Maurer 736edc7a7e src/rrd/rrd.rs: implement DST_COUNTER 2020-05-25 08:14:30 +02:00
Dietmar Maurer 2b55de407e src/rrd/rrd.rs: correctly compute derived values
use f64 for time.
2020-05-25 07:02:04 +02:00
Dietmar Maurer a608806f65 www/DataStoreStatus.js: display netin/netout 2020-05-24 19:02:35 +02:00
Dietmar Maurer 8f0cec2642 src/bin/proxmox-backup-proxy.rs: gather netin/netout stats 2020-05-24 19:02:35 +02:00
Dietmar Maurer 0ed9a2b3ae src/config/network.rs: implement is_physical_nic() helper 2020-05-24 19:02:35 +02:00
Dietmar Maurer 58edd33d2b src/rrd/rrd.rs: implement DST_DERIVE 2020-05-24 19:02:35 +02:00
Dietmar Maurer 4fb05fde17 src/rrd/rrd.rs: restructure whole code 2020-05-24 16:51:28 +02:00
Dietmar Maurer daca4f7888 src/rrd/rrd.rs: reduce size by using f64:NAN as UNKNOWN 2020-05-24 09:09:09 +02:00
Dietmar Maurer 4e6585839b src/rrd/rrd.rs: simplify an fix old value deletion 2020-05-24 06:44:06 +02:00
Dietmar Maurer 8c981ae379 rrd: fix display interval, try to avoid numeric errors 2020-05-23 16:03:43 +02:00
Dietmar Maurer 803ab12ad4 rrd: simplify code 2020-05-23 15:37:17 +02:00
Dietmar Maurer a4a3f7ca5e rrd: pack multiple rrd values into th estat list 2020-05-23 14:03:44 +02:00
Dietmar Maurer a2f862eed6 add experimental rrd api to get cpu stats 2020-05-23 11:50:53 +02:00
Dietmar Maurer eaeda365e0 start gathering stats using new rrd module 2020-05-23 10:43:08 +02:00
Dietmar Maurer 6359dc891a add simple rrd implementation 2020-05-23 10:42:48 +02:00
Dietmar Maurer 07ad6470ca src/client/pull.rs: split out pull related code 2020-05-22 08:04:20 +02:00
Dietmar Maurer a6160cdfeb src/bin/proxmox-backup-proxy.rs: schedule sync jobs 2020-05-22 07:50:59 +02:00
Dietmar Maurer 183125d576 src/api2/pull.rs: aquire try_shared_chunk_store_lock inside pull_store 2020-05-22 07:24:17 +02:00
Dietmar Maurer a3016d6583 proxmox-backup-manager: add sync-job cli 2020-05-21 11:44:45 +02:00
Dietmar Maurer b29d046e89 proxmox-backup-manager: split out cert.rs 2020-05-21 11:22:20 +02:00
Dietmar Maurer 380bd7df97 proxmox-backup-manager: split out datastore.rs 2020-05-21 11:14:34 +02:00
Dietmar Maurer ea6f404e55 proxmox-backup-manager: split out dns.rs 2020-05-21 11:10:58 +02:00
Dietmar Maurer a35a211d9e proxmox-backup-manager: split out network.rs 2020-05-21 11:08:38 +02:00
Dietmar Maurer 53e14507c1 proxmox-backup-manager: split out acl.rs 2020-05-21 10:56:46 +02:00
Dietmar Maurer 6fa39e53e0 proxmox-backup-manager: split out users.rs 2020-05-21 10:53:06 +02:00
Dietmar Maurer a220a4564a roxmox-backup-manager: start splitting command into several files 2020-05-21 10:46:07 +02:00
Dietmar Maurer 6f652b1b3a rename 'job' to 'sync' 2020-05-21 10:29:25 +02:00
Dietmar Maurer b1d4edc769 src/api2/config/job.rs: add job api 2020-05-21 10:16:35 +02:00
Dietmar Maurer b4900286ce src/config/jobs.rs: use SectionConfig for jobs 2020-05-21 10:16:35 +02:00
Dietmar Maurer c681885227 src/bin/proxmox-backup-manager.rs: format output of show commands 2020-05-20 16:47:37 +02:00
Dietmar Maurer ee8b464466 src/tools/systemd.rs: avoid compiler warnings 2020-05-20 16:47:08 +02:00
Dominik Csapak ce55db66d6 proxmox-backup-manager: add show command for remote and datastore
to show the data for a single item

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 16:33:07 +02:00
Dominik Csapak 2882c881e9 api2/access/acl: add path and exact parameter to list_acl
so that we can get only a subset of the acls, filtered by the backed
also return the digest here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 13:44:36 +02:00
Dominik Csapak 12e3895399 api2/access/acl: make update_acl a protected api call
since we want to set the owner of the acl config to 'root'
which is only possible when using a protected api call

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 13:22:41 +02:00
Dominik Csapak 11b6391c83 add 'exact' parameter to extract_acl_node_data
so that we can return acls for a single path

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 13:22:10 +02:00
Dominik Csapak b05672579e api2/roles: change return field of role to roleid
to be compatible with the pve api
with this, we can reuse the ui parts (RoleSelector)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 13:21:47 +02:00
Dominik Csapak 5160c0e986 api2/acl: add privs array to roles
so that an admin can see which roles have which privileges

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 13:21:37 +02:00
Dominik Csapak 1ad9dd08f4 acls: use constnamemap macro for privileges
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 13:21:28 +02:00
Dietmar Maurer 25829a879b src/bin/proxmox-backup-proxy.rs: schedule prune jobs 2020-05-20 13:00:53 +02:00
Dietmar Maurer 872062ee9f src/config/datastore.rs_ change prune types from i64 to u64 2020-05-20 13:00:13 +02:00
Dietmar Maurer 67f7ffd0db src/config/datastore.rs: add prune settings 2020-05-20 11:29:59 +02:00
Dietmar Maurer 0fafac2492 src/api2/access/user.rs: remove useless description
The description is not used at all if we refer to a type.
2020-05-20 11:27:58 +02:00
Dietmar Maurer 49ff10921c src/api2/types.rs: define PRUNE_SCHEMA_KEEP_* 2020-05-20 10:13:38 +02:00
Dietmar Maurer 479e4932b5 src/tools/systemd/parse_time.rs: improve error message 2020-05-20 09:43:16 +02:00
Dietmar Maurer dd7a7eae8f src/bin/proxmox-backup-manager.rs: add completion helper for gc-schedule 2020-05-20 09:42:51 +02:00
Dietmar Maurer 8545480a31 src/bin/proxmox-backup-proxy.rs: add simple task scheduler for garbage collection 2020-05-20 08:59:45 +02:00
Dietmar Maurer d6c28ddf84 src/tools/systemd/time.rs: export parse/verify 2020-05-20 08:38:39 +02:00
Dietmar Maurer 42fdbe5112 src/config/datastore.rs: add gc-schedule property 2020-05-20 08:38:10 +02:00
Dominik Csapak 9c5c383bff user: create default root user as typed struct
we added a userid attribute to the User struct, but missed that we
created the default user without that attribuet via the json! macro
which lead to a runtime panic on the deserialization

by using the struct directly, such errors will be caught by the compiler
in the future

with this change, we can remove the serde_json import here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-20 06:09:08 +02:00
Dietmar Maurer 7d4e362993 depend on proxmox 0.1.32, src/api2/access/user.rs: simplify code 2020-05-19 12:58:46 +02:00
Dominik Csapak 522c0da0a0 use new 'id_property' for user::User and use it in api calls
this allows us to return a user::User (or Vec<> of it)
instead of a generic serde value

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-19 09:33:56 +02:00
Dominik Csapak 16c75c580b adapt to changes of SectionConfigPlugin
it requires not an Option<String> for the optional id_property

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-19 09:28:45 +02:00
Dietmar Maurer 07ce44a633 avoid compiler warnings 2020-05-19 07:03:41 +02:00
Dietmar Maurer e8d1da6a15 depend on proxmox 0.1.31 - use Value to store result metadata 2020-05-18 09:57:35 +02:00
Dietmar Maurer 7a314d18f7 src/tools/systemd/parse_time.rs: check max values 2020-05-16 13:13:50 +02:00
Dietmar Maurer 2d08c97ae2 CalendarEvent - compute_next_value: use change tracking to avoid repeated testing 2020-05-16 10:32:27 +02:00
Dietmar Maurer 50ce1f987d CalendarEvent - compute_next_value: support seconds 2020-05-16 10:21:24 +02:00
Dietmar Maurer d1a5ffdf78 src/tools/systemd/tm_editor.rs: new helper class 2020-05-16 10:09:41 +02:00
Dietmar Maurer 99baf7afcc CalendarEvent: test and fix repeated values 2020-05-16 07:43:51 +02:00
Dietmar Maurer fed270bf3f CalendarEvent: speedup/simplify repetition tests 2020-05-16 07:09:53 +02:00
Dietmar Maurer e05b637c73 src/tools/systemd/parse_time.rs: move parser into separate file 2020-05-16 06:53:15 +02:00
Dietmar Maurer 2ee6b3fdb9 src/tools/systemd/time.rs: implement compute_next_event 2020-05-16 06:33:03 +02:00
Dietmar Maurer f3a96b2cdb renamed: src/tools/systemd/parser.rs -> src/tools/systemd/config.rs 2020-05-16 06:32:28 +02:00
Dietmar Maurer a260c74a12 src/tools/systemd/time.rs: add helpers to compute CalendarEvents 2020-05-15 17:55:54 +02:00
Dietmar Maurer 30f577248b src/api2/node/time.rs: avoid custom unsafe readlink implementations 2020-05-15 06:50:07 +02:00
Dietmar Maurer 00491c0230 src/tools/systemd/parser.rs: use different setups for service and timer files, code cleanup 2020-05-14 13:55:13 +02:00
Dietmar Maurer 2ebdbac1c4 depend on nom, add parser for systemd calendar enents and time span 2020-05-14 12:18:30 +02:00
Dietmar Maurer f486e9e50e add systemd configuration file parser/writer, start job configuration 2020-05-12 13:07:49 +02:00
Dietmar Maurer 65dab0266c proxmox-backup-manager: add completion helper for port list 2020-05-08 17:28:04 +02:00
Dietmar Maurer 525008f7ad proxmox-backup-manager - network list: render ports/slaves
And render interface name as first comumn.
2020-05-08 16:07:23 +02:00
Dietmar Maurer 5bef0f43da src/config/network.rs - check_bridge_ports: correctly check vlan ports 2020-05-08 15:51:47 +02:00
Dietmar Maurer 0f6bdbb01f src/config/network.rs - write_config: add more consistency checks 2020-05-08 14:31:38 +02:00
Dietmar Maurer a4ccb46176 src/config/network.rs: avoid duplicate port usage 2020-05-08 11:15:00 +02:00
Dietmar Maurer 80bf084876 src/config/network.rs: do not combine entries
It is unclear when and how to write combined entries ...
2020-05-08 10:20:57 +02:00
Dietmar Maurer db5672e83e src/config/network.rs: always write bridge_ports and bond_slaves
So that we can reliable detect the interface type.
2020-05-08 09:58:03 +02:00
Dietmar Maurer 86a5d56c4e proxmox-backup-manager: add network create command 2020-05-08 09:55:56 +02:00
Dietmar Maurer 3dd27a3bf8 src/api2/node/network.rs: add protected flag to revert 2020-05-08 09:30:25 +02:00
Dietmar Maurer 3aedb73816 src/api2/node/network.rs: pass bridge_ports and slaves a property strings
To make it compatible with pve.
2020-05-08 08:49:17 +02:00
Dietmar Maurer bab5d18c3d src/config/network.rs: implement bond_mode
and rename bond_slaves to slaves to make it compatible with pve.
2020-05-07 14:07:45 +02:00
Dietmar Maurer c2ffc68554 src/api2/node/network.rs: cleanup - factor out check_duplicate_gateway 2020-05-07 11:26:30 +02:00
Dietmar Maurer 9651833130 src/api2/node/network.rs: allow to create bridge and bond 2020-05-07 11:09:12 +02:00
Dietmar Maurer 7b22acd0c2 src/config/network.rs: make it compatible with pve
and depend on proxmox 0.1.26
2020-05-07 09:28:25 +02:00
Dietmar Maurer 5751e49566 src/server/worker_task.rs: implement and use status command 2020-05-07 09:27:33 +02:00
Dietmar Maurer 197de83ffa src/server/command_socket.rs: do not abort loop on client errors, allow backup gid 2020-05-07 09:27:33 +02:00
Wolfgang Bumiller 10effc9849 add tools/disks.rs (work in progress...)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-05-05 10:14:42 +02:00
Dietmar Maurer 99641a6bbb garbage_collect: call fail_on_abort to abort GV when requested. 2020-05-05 09:06:34 +02:00
Dietmar Maurer 74f7240b8d src/bin/proxmox-backup-client.rs: add human readable date to prune list 2020-05-05 07:33:58 +02:00
Dietmar Maurer db1e061dcb src/bin/proxmox-backup-client.rs: correctly format prune result list. 2020-05-05 06:45:37 +02:00
Dietmar Maurer b080583ba8 src/bin/proxmox-backup-manager.rs: improve user list output 2020-05-01 16:22:50 +02:00
Dietmar Maurer ed751dc2ab src/api2/node/status.rs: rework api, implement reboot and shutdown 2020-04-30 11:52:40 +02:00
Dietmar Maurer ca9dfe5fa4 src/api2/node/tasks.rs: use api macro features for default values 2020-04-30 11:51:56 +02:00
Dietmar Maurer 720af9f69b src/api2/node/tasks.rs: allow users to list/access there own tasks 2020-04-30 10:05:50 +02:00
Dietmar Maurer f1490da82a use resonable acl paths (fixup) 2020-04-30 09:32:13 +02:00
Dietmar Maurer 74c08a5782 use reasonable acl paths 2020-04-30 09:30:00 +02:00
Dietmar Maurer bd88dc4116 cached_config: avoid parsing non-existent files multiple times 2020-04-30 07:04:23 +02:00
Dominik Csapak f9e3b1104e change index to templates using handlebars
using a handlebars instance in ApiConfig, to cache the templates
as long as possible, this is currently ok, as the index template
can only change when the whole package changes

if we split this in the future, we have to trigger a reload of
the daemon on gui package upgrade (so that the template gets reloaded)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-04-29 17:05:53 +02:00
Dietmar Maurer bc0d03885c use proxmox 0.1.25, use new EnumEntry feature 2020-04-29 13:01:24 +02:00
Dietmar Maurer b9f2f761bb avoid problems with missing acl.cfg and user.cfg 2020-04-29 10:40:42 +02:00
Dietmar Maurer 30fb602578 src/api2/admin/datastore.rs - get_datastore_list: only return name and comment
We dont want to leak the full configuration to users with limited access permission.
Please use the api2::config::datastore api to get the full configuration.
2020-04-29 09:21:34 +02:00
Dietmar Maurer 0a00f6e01c src/api2/config/datastore.rs_ add delete property to update method 2020-04-29 09:09:59 +02:00
Dietmar Maurer 30003baaa4 src/api2/config/remote.rs: fix white space 2020-04-29 09:09:39 +02:00
Dietmar Maurer 5211705ff1 src/api2/config/remote.rs: add delete parameter to update method 2020-04-29 09:04:17 +02:00
Dietmar Maurer ec67af9af3 src/api2/pull.rs: require Datastore.Prune if delete flag is set. 2020-04-29 07:19:32 +02:00
Dietmar Maurer 8247db5b39 src/config/acl.rs: introduice privileges and roles for remotes 2020-04-29 07:03:44 +02:00
Dietmar Maurer 409f44247b fix api2::types::ACL_ROLE_SCHEMA
make sure we list all roles ...
2020-04-28 13:25:02 +02:00
Dietmar Maurer dd335b77f5 src/config/acl.rs - fix regression tests 2020-04-28 11:16:15 +02:00
Dietmar Maurer 6f6aa95abb add Datastore.Backup, Datastore.PowerUser and Datastore.Reader role 2020-04-28 11:07:25 +02:00
Dietmar Maurer 54552dda59 implemnt backup ownership, improve datastore access permissions 2020-04-28 10:22:25 +02:00
Dietmar Maurer 1347b1152d src/config/cached_user_info.rs - lookup_privs: correctly handle superuser 2020-04-27 13:22:03 +02:00
Dietmar Maurer d00e1a216f src/config/acl.rs: introduce more/better datastore privileges 2020-04-27 07:13:50 +02:00
Dietmar Maurer 9c7fe29dfc src/config/acl.rs: rtename PRTIV_DATASTORE_ALLOCATE to PRIV_DATASTORE_MODIFY 2020-04-27 06:50:35 +02:00
Dietmar Maurer 14627d671a src/bin/proxmox-backup-manager.rs: add dns sub command
Also improved the DNS api, added a --delete option.
2020-04-26 08:23:23 +02:00
Dietmar Maurer 76227a6acd src/bin/proxmox-backup-manager.rs: fix node parameter handling 2020-04-25 17:20:22 +02:00
Dietmar Maurer 26d9aebc28 move src/api2/config/network.rs to src/api2/node/network.rs
So that we have the same api path for network config as pve.
2020-04-25 17:00:38 +02:00
Dietmar Maurer 1ca540a63b src/config/network.rs: auto-add lo, and implement a few regression tests 2020-04-24 12:57:11 +02:00
Wolfgang Bumiller 9094186a57 xattr: cleanup: don't use pxar types in the API
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-24 11:23:48 +02:00
Wolfgang Bumiller 27a3decbfe xattr: api cleanup
Make `flistxattr()` return a `ListXAttr` helper which
provides an iterator over `&CStr`.

This exposes the property that xattr names are a
zero-terminated string without simply being an opaque
"byte vector". Using &[u8] as a type here is too lax.

Also let `fgetxattr` take a `CStr`. While this may be a
burden on the caller, we usually already have
zero-terminated strings on the call site. Currently we only
use this method coming from `flistxattr` after all.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-24 10:56:52 +02:00
Wolfgang Bumiller 9af76ef075 xattr: use checked_mul to increase size
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-24 10:56:52 +02:00
Wolfgang Bumiller 00ec8d1685 tools: pub use Fd from proxmox crate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-24 10:56:52 +02:00
Dietmar Maurer fd7c0979b4 src/bin/proxmox-backup-manager.rs: implement netwerk revert 2020-04-24 10:45:49 +02:00
Dietmar Maurer c67bc9c35c src/bin/proxmox-backup-manager.rs: new command to show pending network changes 2020-04-24 10:27:43 +02:00
Dietmar Maurer 3181f9b625 src/bin/proxmox-backup-manager.rs: only show pending changes with "text" format 2020-04-24 10:16:57 +02:00
Dietmar Maurer 2eefd9aee1 src/config/network.rs: implement network reload, set "changes" attribute 2020-04-24 09:55:46 +02:00
Dietmar Maurer 8a6b86b8a7 src/config/network.rs: use a simple String for comments 2020-04-24 07:46:08 +02:00
Dietmar Maurer 96d9478668 src/config/network/parser.rs: corectly detect vanished interfaces 2020-04-24 07:26:54 +02:00
Dietmar Maurer 10a9be45bd src/api2/config/network.rs: implement update/delete comments 2020-04-23 16:08:35 +02:00
Dietmar Maurer 5f60a58fd5 src/config/network.rs; support interface comments, cleanups 2020-04-23 15:54:30 +02:00
Dietmar Maurer 659c3be3d5 src/config/network.rs: avoid newline after family options 2020-04-23 11:30:41 +02:00
Dietmar Maurer 5e4e88e83f src/api2/config/network.rs: implement update/delete for bridge_ports and bond_slaves 2020-04-23 11:21:27 +02:00
Dietmar Maurer 339965d720 src/api2/config/network.rs: only allow one default gateway 2020-04-23 10:37:40 +02:00
Dietmar Maurer c38b4bb8b2 src/config/network.rs: do not allow to change interface type 2020-04-23 09:43:38 +02:00
Dietmar Maurer 42fbe91a34 src/config/network.rs: parse bond-slaves 2020-04-23 09:31:10 +02:00
Dietmar Maurer 1d9a68c2fc src/config/network.rs: parse bridge-ports 2020-04-23 09:24:17 +02:00
Dietmar Maurer 02269f3dba src/config/network.rs: introduce NetworkInterfaceType 2020-04-23 08:45:03 +02:00
Dietmar Maurer d5ca9bd5df src/config/network.rs: cleanup (new helper combine_entry) 2020-04-23 07:54:12 +02:00
Dietmar Maurer 02e36d96ad src/config/network.rs: write changes to interfaces.new 2020-04-23 07:19:29 +02:00
Dietmar Maurer 2c18efd902 src/config/network.rs: use a single mtu setting (instead of mtu_v4 and mtu_v6) 2020-04-23 07:07:14 +02:00
Dietmar Maurer 4cb6bd894c src/bin/proxmox-backup-manager.rs: improve network list output format 2020-04-23 06:44:55 +02:00
Dietmar Maurer b1564af25a src/bin/proxmox-backup-manager.rs: format datastore list output 2020-04-22 17:37:20 +02:00
Dietmar Maurer bf004ecd87 src/bin/proxmox-backup-manager.rs: format network list output 2020-04-22 17:14:52 +02:00
Dietmar Maurer f1026a5aa9 src/api2/config/network.rs: allow to update 'auto' flag 2020-04-22 16:46:46 +02:00
Dietmar Maurer 3fce3bc36e src/config/network/parser.rs: parse MTU settings 2020-04-22 13:44:51 +02:00
Dietmar Maurer f8e7ac686a src/config/network.rs: only save attriubutes used by configuration method 2020-04-22 12:42:09 +02:00
Dietmar Maurer c016482c7a src/api2/config/network.rs: implement delete property 2020-04-22 12:19:31 +02:00
Dietmar Maurer 27f2c23049 src/api2/config/network.rs: allow to update configuration method 2020-04-22 11:32:36 +02:00
Dietmar Maurer df6bb03d0e src/api2/config/network.rs: improve network api 2020-04-22 10:54:07 +02:00
Dietmar Maurer e2d940b949 src/config/network/parser.rs: remove debug println 2020-04-22 10:53:26 +02:00
Dietmar Maurer 0c226bc173 src/config/network/helper.rs: fix CIDR regex 2020-04-22 10:52:31 +02:00
Dietmar Maurer 76cf5208cf src/api2/types.rs: add schemas for IP/CIDR 2020-04-22 10:28:53 +02:00
Dietmar Maurer 2ea7bf1b3d src/api2/config/datastore.rs_ fix method docs 2020-04-22 08:53:16 +02:00
Dietmar Maurer 8b57cd4441 src/config/network.rs: remove netmask support
rely on cidr instead.
2020-04-22 08:45:13 +02:00
Dietmar Maurer 68da20bf62 src/api2/types.rs: define NETWORK_INTERFACE_NAME_SCHEMA 2020-04-21 17:54:52 +02:00
Dietmar Maurer c357260d09 src/config/network.rs: move type definitions to src/api2/types.rs 2020-04-21 17:25:05 +02:00
Dietmar Maurer 7e02d08cd0 rename ConfigMethod to NetworkConfigMethod 2020-04-21 17:17:57 +02:00
Dietmar Maurer ca0e534796 src/api2/config/network.rs: start network configuration api 2020-04-21 14:28:26 +02:00
Dietmar Maurer 904e988667 src/config/network.rs: impleement load/save 2020-04-21 12:55:33 +02:00
Dietmar Maurer 3f129233be src/config/network.rs: add Interface flags 'exists' and 'active' 2020-04-21 11:46:56 +02:00
Dietmar Maurer a9bb491e35 src/config/network.rs: cleanup autostart flag handling 2020-04-21 11:06:22 +02:00
Dietmar Maurer 1ec7f8a0dd src/config/network/helper.rs: new helper get_network_interfaces() 2020-04-21 10:32:54 +02:00
Dietmar Maurer 92310d585c src/config/network.rs: simplify code 2020-04-20 18:10:15 +02:00
Dietmar Maurer f34d4401f7 src/config/network.rs: read/write /etc/network/interfaces
Start implementing a recursive descent parser.
2020-04-20 14:15:57 +02:00
Dietmar Maurer 6e695960ca src/config/cached_user_info.rs: cache it up to 5 seconds 2020-04-18 08:49:20 +02:00
Dietmar Maurer 365f0f720c fix permission tests using non-uri parameters
We nood to do those tests inside the fuction body instead...
2020-04-18 08:23:04 +02:00
Dietmar Maurer a737179eb4 src/config/cached_user_info.rs: new check_privs helper 2020-04-18 08:09:34 +02:00
Dietmar Maurer bb072ba49c src/api2/access.rs: cleanup 2020-04-18 07:28:25 +02:00
Dietmar Maurer ff329f970b src/api2/types.rs: use anyhow::Error in test cases 2020-04-18 07:05:31 +02:00
Wolfgang Bumiller f7d4e4b506 switch from failure to anyhow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Dietmar Maurer 404d78c41e src/api2/pull.rs: add access permission 2020-04-17 15:27:04 +02:00
Dietmar Maurer 1bfc1efa50 src/api2/subscription.rs: add access permissions 2020-04-17 15:14:28 +02:00
Dietmar Maurer 73ce1d1146 src/api2/reader.rs: add access permissions 2020-04-17 15:01:56 +02:00
Dietmar Maurer 70e5f2461d src/api2/config/remote.rs: add access permissions 2020-04-17 14:57:26 +02:00
Dietmar Maurer c0ef209aeb src/api2/config/datastore.rs: impl digest check for delete, add access permissions 2020-04-17 14:51:29 +02:00
Dietmar Maurer 9f9f7eefa3 src/api2/backup.rs: add access permissions 2020-04-17 14:40:20 +02:00
Dietmar Maurer bb34b58910 src/api2/admin/datastore.rs: add access permissions - first try
We need to refine this later (introduce backup owner concept?)
2020-04-17 14:36:27 +02:00
Dietmar Maurer 5972def5ec acl: change path "storage" to "datastore" 2020-04-17 14:15:44 +02:00
Dietmar Maurer aa90ced3bf src/api2/access/role.rs: use schema ACL_ROLE_SCHEMA 2020-04-17 14:14:06 +02:00
Dietmar Maurer ca257c8097 move type defs from src/api2/access/acl.rs to src/api2/types.rs 2020-04-17 14:13:15 +02:00
Dietmar Maurer 3fff55b293 src/api2/access/role.rs: new api to list roles 2020-04-17 14:03:24 +02:00
Dietmar Maurer 4f66423fcc src/api2/access/user.rs: add access permissions 2020-04-17 11:04:36 +02:00
Dietmar Maurer d4f020f4c5 src/api2/access/user.rs: add access permissions 2020-04-17 10:08:45 +02:00
Dietmar Maurer d28ddb8e04 src/api2/access/acl.rs: add access permissions 2020-04-17 10:03:09 +02:00
Dietmar Maurer 83b6a7cf71 src/api2/node/tasks.rs: use api macro, implement access permissions 2020-04-16 17:47:21 +02:00
Dietmar Maurer e4681f9f71 src/api2/node/syslog.rs: add access permissions 2020-04-16 17:08:19 +02:00
Dietmar Maurer b5037fa8ed src/api2/node/status.rs: add access permissions 2020-04-16 17:05:09 +02:00
Dietmar Maurer 9989d2c4e9 src/server/rest.rs: reduce delay for permission error to 500ms 2020-04-16 12:56:34 +02:00
Dietmar Maurer 1cf7bbf412 src/api2/node/services.rs: add access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer 68ed0c629d src/api2/node/journal.rs: add access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer 4b40148caa start impl. access permissions 2020-04-16 12:47:16 +02:00
Dietmar Maurer 423e656163 src/config/cached_user_info.rs: new helper class 2020-04-16 10:05:16 +02:00
Dietmar Maurer 109d7817cd src/config/user.rs - cached_config: do not store/return digest 2020-04-15 11:35:57 +02:00
Dietmar Maurer 5354511fd0 src/config/acl.rs: implement cached_config 2020-04-15 11:30:47 +02:00
Dietmar Maurer bd098a7f77 src/api2/node/dns.rs: use api macro (cleanup) 2020-04-15 10:09:18 +02:00
Dietmar Maurer 8d048af2bf acl: improve NoAccess handling 2020-04-15 08:11:43 +02:00
Dietmar Maurer 9a328319dd pxar extract: remove pattern from arg_param, add target instead 2020-04-15 06:41:37 +02:00
Dietmar Maurer 7e3d2e5b41 pxar create: remove exclude from arg_param 2020-04-15 06:31:46 +02:00
Dietmar Maurer 9c06f6c292 fix previous commit - use result. 2020-04-14 17:48:10 +02:00
Dietmar Maurer 9f4e47dd93 acl update: check path 2020-04-14 17:23:48 +02:00
Dietmar Maurer d83175dd69 acl update: check if user exist. 2020-04-14 13:46:27 +02:00
Dietmar Maurer 68ccdf09a4 src/config/user.rs: implement user config cache 2020-04-14 13:45:45 +02:00
Dietmar Maurer 9765092ede acl api: implement update 2020-04-14 10:16:49 +02:00
Dietmar Maurer ed3e60ae69 start ACL api 2020-04-13 11:09:44 +02:00
Dietmar Maurer a83eab3c4d acl: use BTreeMap and BTreeSet to avoid sort() 2020-04-12 17:13:53 +02:00
Dietmar Maurer 0815ec7e65 acl: implement roles(), add regression tests. 2020-04-12 13:06:50 +02:00
Dietmar Maurer 5c6cdf9815 add acl config 2020-04-11 12:24:26 +02:00
Dietmar Maurer b88f9c5b1e PASSWORD_SCHEMA: set max_length to 1024 (for tickets) 2020-04-09 13:35:58 +02:00
Dietmar Maurer 879546aff6 api: add default property to domain list 2020-04-09 13:35:08 +02:00
Dietmar Maurer 73b40e9b46 api: correctly sort access subdirmap 2020-04-09 13:34:07 +02:00
Dietmar Maurer 708db4b3ae api: add list_domains 2020-04-09 11:36:45 +02:00
Dietmar Maurer 685e13347e api: move config/user to access/users, implement change_password
To make it similar to the pve api
2020-04-09 10:21:24 +02:00
Dietmar Maurer 7d817b0358 implement auth framework 2020-04-08 14:06:15 +02:00
Dietmar Maurer 579728c641 add user configiguration 2020-04-08 14:06:15 +02:00
Dietmar Maurer cf459b1982 gc: log pending removals 2020-04-06 09:50:40 +02:00
Dietmar Maurer dda7015497 prune api: return a usable result (we run synchronous anyways) 2020-04-01 12:24:28 +02:00
Dietmar Maurer 4299ca727c src/server/rest.rs: use correct formatter 2020-03-26 12:54:20 +01:00
Dominik Csapak 6650a242fb rewrite future select in upgrade_to_backup_protocol using select macro
and handle all ok/err cases with at least logging

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-03-18 11:33:59 +01:00
Dietmar Maurer ca26db1140 delete unused src/storage/config.rs 2020-03-02 13:31:38 +01:00
Dietmar Maurer 90c5239d46 use SectionConfig from proxmox 0.1.18-1 2020-03-02 12:52:11 +01:00
Dietmar Maurer 18deda403d src/bin/proxmox-backup-client.rs - list_groups: render full snapshot path for last-backup column 2020-02-28 09:09:22 +01:00
Dietmar Maurer 6882b3716c src/bin/proxmox-backup-manager.rs: avoid compiler warning 2020-02-28 08:52:12 +01:00
Christian Ebner 32d192a952 catalog: shell: Use the new logic including resolving symlinks for catalog
and remove the old unused code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:51:01 +01:00
Christian Ebner fee5528e59 catalog: shell: introduce new CatalogPathStack to navigate in catalog shell.
This is basically a rewrite of the current logic for navigating the catalog,
but in addition allows to follow symlinks.
Following symlinks introduces the issue that generation of canonical paths
(needed in the actual pxar archive) is more complex, as symlinks have to be
resolved and loops avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:50:43 +01:00
Christian Ebner 4145c36749 catalog: add missing function documentation.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:50:08 +01:00
Christian Ebner c2f9149461 catalog: introduce is_symlink() to check if DirEntry is a symlink.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:49:55 +01:00
Christian Ebner 11ee5c0563 catalog: derive PartEq for equality checks of two DirEntry's.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:49:39 +01:00
Christian Ebner 536683e73b src/backup/dynamic_index.rs: Add LruCache for chunks.
In order to improve non-sequential reads of chunks as e.g. in FUSE.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:47:16 +01:00
Christian Ebner 35ddf0b419 catalog: shell: Introduce clear-selected command.
'clear-selected' allows to clear all the match patterns from the list of
patterns for a subsequent restore.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:40:55 +01:00
Christian Ebner 8e464141cf catalog: shell: Improve list-selected command.
'list-selected' now shows the filenames matching the patterns for a restore
instead of the patterns themselfs.
The patterns can be displayed by passing the '--pattern' flag.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:40:42 +01:00
Christian Ebner 351b913d1e pxar::decoder: Return the correct symlink target size.
This eliminates also repeated calls to readlink in fuse, which occur when the
preallocated buffer to store the symlink target path is to small.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 08:38:56 +01:00
Christian Ebner 189996cf4a proxmox-backup-client: expose exclude match patterns to cli.
Allows to pass exclude match patterns to the 'backup' command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-28 07:56:09 +01:00
Dietmar Maurer ac3faaf5c0 src/bin/proxmox-backup-manager.rs: cleanup output handling 2020-02-28 07:42:36 +01:00
Dietmar Maurer 4939255fb4 src/tools/format.rs: move output rendering code to this new file 2020-02-28 07:30:35 +01:00
Dietmar Maurer c81b2b7c70 src/bin/proxmox-backup-client.rs - list_groups: use format_and_print_result_full()
Depend on proxmox 0.1.16-1
2020-02-27 13:32:11 +01:00
Dietmar Maurer c2043614f7 src/bin/proxmox-backup-client.rs: use new output formath helpers from proxmox 0.15.1-1 2020-02-27 12:41:15 +01:00
Dietmar Maurer e23f586344 src/bin/proxmox-backup-client.rs - status: improve alignment 2020-02-27 11:38:09 +01:00
Dietmar Maurer be2425ff85 src/bin/proxmox-backup-client.rs - status: do not print header 2020-02-27 11:31:05 +01:00
Dietmar Maurer f24fc1166b src/bin/proxmox-backup-client.rs - list_snapshots: use format_and_print_result_full()
Depend on proxmox v0.1.14.
2020-02-27 11:28:44 +01:00
Christian Ebner af934f8cf6 tools::lru_cache: Improve access() and insert() by using HashMap::entry().
entry() allows to lookup the position where and entry belongs and update/insert
it in the HashMap more efficiently than get_mut() and insert().
Details: https://gankra.github.io/blah/hashbrown-insert/

In addition, use the struct LinkedList and remove the outdated code.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:56:25 +01:00
Christian Ebner e3ab9a383c tools::lru_cache: Separate LinkedList part of LruCache into own struct.
In order to keep the separation more aparent and avoid mut borrow conflics.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:55:48 +01:00
Christian Ebner 75c2ee7bab tools::lru_cache: Make key generic.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:55:16 +01:00
Christian Ebner 03f779c6f5 catalog: shell: Improve output of stat command.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-27 06:53:17 +01:00
Dietmar Maurer ea5f547fdc src/bin/proxmox-backup-client.rs - list_snapshot_files: use format_and_print_result_full() 2020-02-26 13:49:47 +01:00
Dietmar Maurer 390c5bdde4 src/bin/proxmox-backup-client.rs - status: use format_and_print_result_full 2020-02-26 13:29:00 +01:00
Dietmar Maurer 03ac286c80 src/bin/proxmox-backup-manager.rs: use crate::tools::runtime::block_on() 2020-02-26 11:37:38 +01:00
Dietmar Maurer dd04383bb8 src/tools/runtime.rs: call OPENSSL_thread_stop to avoid race with openssl cleanup handlers 2020-02-26 10:38:55 +01:00
Dietmar Maurer ca2dbb8af1 src/client/backup_writer.rs: also print archive size 2020-02-25 14:12:29 +01:00
Wolfgang Bumiller 0bcbb5434e cleanup unused module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-25 11:38:27 +01:00
Christian Ebner 495b238bc1 pxar: Change text of warning for files matched by exclude patterns.
In addition to the .pxarexclude files, glob match patterns can be passed to pxar
also via cli parameters.
Therefore the warning is rephrased to be more ambiguous.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-02-25 10:47:24 +01:00
Dietmar Maurer 6da73c823f src/client/backup_writer.rs: prefix upload stats with archive name 2020-02-24 13:24:46 +01:00
Dietmar Maurer e02c3d461f src/client/backup_writer.rs: implement verbose options 2020-02-24 12:48:40 +01:00
Dietmar Maurer ee5fe978e2 src/client/backup_writer.rs: reduce verbosity 2020-02-24 11:33:00 +01:00
Dietmar Maurer 8ce49a76da src/backup/backup_info.rs: fix SNAPSHOT_PATH_REGEX and GROUP_PATH_REGEX 2020-02-18 13:16:35 +01:00
Wolfgang Bumiller 9e003074cb runtime: fix blocking strategy:
- do not "double"-block_in_place() (it may not be nested)
- do not call block_in_place() in non-worker threads

is_in_tokio() isn't sufficient, we need to actually know
that we're in a worker-thread, so we do this by remembering
that we're blocking.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-17 09:48:22 +01:00
Wolfgang Bumiller 1283d58ca9 runtime: remove IN_TOKIO thread local variable
tokio now has Handle::try_current() allowing us to
generally check for a tokio runtime even if spawned by
someone else

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-13 08:56:38 +01:00
Dietmar Maurer a6e3da987c src/client/http_client.rs: do not store fingerprints passed via options in cache 2020-02-11 11:54:43 +01:00
Dietmar Maurer 1bd6f32b43 src/client/http_client.rs: always convert fingerprint to lowercase 2020-02-11 11:42:03 +01:00
Dietmar Maurer d1c657276a src/client/http_client.rs: remove useless password_env 2020-02-11 11:42:03 +01:00
Dietmar Maurer a05c0c6ff6 src/bin/proxmox-backup-client.rs: allows to pass fingerprint via env vars 2020-02-11 11:42:03 +01:00
Thomas Lamprecht b69b8af26b s/pbsbanner/proxmox-backup-banner/ + move to /usr/lib
We want to avoid pbs if possible and also avoid placing internal
binaries, not intended for human direct use, in /bin or /sbin paths.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-10 14:43:28 +01:00
Thomas Lamprecht e2b5e75a3e pbsbanner: use match and also return error
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-07 18:30:55 +01:00
Thomas Lamprecht 274b0c7bb7 add login banner service
Modeled after the one from PVE, but using rust instead of perl for
resolving the nodename and writing to /etc/issue

Behavior differs a bit. We write all non-loopback addresses to this
file, as the gui accepts connections from them all, so limiting it to
the first one is not really sensible.
Further an error to resolve, or only getting loopback addresses won't
write out an empty /etc/issue file, but a note about the error at the
place where the address would be displayed.

Named it "pbsbanner", not "proxmox-backup-banner" as it's rather an
internal tool anyway and mirrors pvebanner, pmgbanner

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-02-06 22:19:22 +01:00
Christian Ebner ba050e3788 catalog: shell: Allow two or more successive slashes in path.
Two or more successive slashes should be allowed and treated as a single slash.
We also do not treat two successive slashes at the beginning of a path any
different.

Details are found here:
https://pubs.opengroup.org/onlinepubs/000095399/basedefs/xbd_chap04.html#tag_04_11

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-05 09:32:29 +01:00
Christian Ebner 9ff9a00572 pxar::fuse: cleanup: Remove unused code.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 13:43:11 +01:00
Christian Ebner 3e69abef02 pxar::fuse: Refactor lookup in order to cache accessed entries.
Context::find_goodbye_entry() is removed and incorporated into the lookup
callback in order to take advantage of the entry_cache and since it is only used
inside this callback.
All entries read on lookup are cached.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 13:43:06 +01:00
Christian Ebner 63698e720c pxar: Improve read performance for fuse.
By storing the payload start offset in the `DirectoryEntry` and passing this
information to `Decoder::read()`, the payload can be read directly and a repeated
re-reading of the entry information is avoided.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-02-03 12:08:03 +01:00
Dietmar Maurer 032d3ad80f src/bin/proxmox-backup-client.rs: do not use api macro for prune
So that we can use add_common_prune_prameters!() macro again.
2020-01-31 10:31:36 +01:00
Dietmar Maurer b8a192e3b0 src/config/remote.rs: do not serialize empty option 2020-01-31 09:09:24 +01:00
Dietmar Maurer c43b36bcb6 src/section_config.rs - write: improve error message 2020-01-31 09:08:42 +01:00
Dietmar Maurer 501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Christian Ebner 74616f63b0 proxmox-backup-client: add missing prune options to api definition.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-31 07:01:56 +01:00
Dietmar Maurer 93fbb4ef0a update proxmox dependency to 0.1.12 2020-01-31 06:59:17 +01:00
Dietmar Maurer 43592f290d src/bin/proxmox-backup-manager.rs: remove stale debug comment 2020-01-30 17:59:13 +01:00
Dietmar Maurer 9e165b5cad src/bin/proxmox-backup-manager.rs - list remotes: do not use client, call directly 2020-01-30 17:57:37 +01:00
Dietmar Maurer 9894469e89 src/bin/proxmox-backup-manager.rs: use new texdt table formatter 2020-01-30 13:31:20 +01:00
Dietmar Maurer 8528fce874 src/api2/node/tasks.rs; make list_tasks public 2020-01-30 13:31:20 +01:00
Dietmar Maurer 70fa1aa3fc fixup 2020-01-30 13:31:20 +01:00
Dietmar Maurer 5eeea607ae src/section_config.rs - convert_to_array: add option to skip properties 2020-01-30 13:26:46 +01:00
Christian Ebner 93a9eb284e pxar::fuse: Do not return fcaps value on listxattr.
listxattr must only return the name list, no extended attribute values.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:09 +01:00
Christian Ebner 5d2158e89a pxar::fuse: return ACL names in listxattr if present
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:03 +01:00
Christian Ebner 72677fb0a5 pxar::fuse: add support to read ACLs.
ACLs are stored separately in the pxar archive. This implements the functionality
needed to read the ACLs and return them as extended attributes in the getxattr
callback.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:35:00 +01:00
Christian Ebner 0502ce6da3 tools::acl: Add helpers to create extended attribute buffer.
This helpers are used to construct the extended attributes values from
the ACLs stored in the pxar archive.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:34:57 +01:00
Christian Ebner bcf0d452c9 pxar: move acl helper functions to src/tools/acl.rs
They are not only needed by the pxar::sequential_decoder but also for the fuse
xattr impl, so it makes more sense to have them there.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:34:54 +01:00
Wolfgang Bumiller 6f763ae673 tools: LruCache: add dropcheck marker
See https://doc.rust-lang.org/nomicon/phantom-data.html

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-29 09:18:27 +01:00
Dietmar Maurer 99384f7933 src/api2/types.rs: define and use struct TaskListItem 2020-01-28 11:23:08 +01:00
Dietmar Maurer 2c4b303c62 src/api2/node/tasks.rs: use api macro for list_tasks 2020-01-28 11:23:08 +01:00
Christian Ebner 3e56c4abd7 pxar::fuse: Introduce LRU caching for goodbye table and directory entries.
The goodbye table of directory entries is cached in a LRU cache to speed up
subsequent accesses.
This is especially important for directories with many entries, as then the
readdirplus callback is called repeatedly because of the limited reply buffer
size.

`DirectoryEntry`s are cached for subsequent access in their own LRU cache,
independent of the goodbye tables.

In order to avoid borrow conflicts, the `Context` provides a fn as_mut_refs
as well as a fn run_with_context_refs.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-28 10:06:53 +01:00
Christian Ebner cc552754c8 pxar::fuse: refactor fn stat
By passing `&DirectoryEntry` to stat, the function interface is simplified.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-28 10:06:51 +01:00
Dietmar Maurer d08bc483db use proxmox 0.1.9 with new cli command helpers 2020-01-27 18:08:43 +01:00
Dietmar Maurer 99d863d770 src/client/http_client.rs: new password_env option 2020-01-27 10:42:37 +01:00
Dietmar Maurer 5030b7cea4 src/client/http_client.rs: new prefix options 2020-01-27 10:20:55 +01:00
Dietmar Maurer 5a74756c15 src/client/http_client.rs: implement fingerprint cache 2020-01-25 15:37:34 +01:00
Dietmar Maurer d59dbeca1b src/client/http_client.rs: new struct HttpClientOptions 2020-01-25 12:18:00 +01:00
Dietmar Maurer 6afbe1d846 src/config/remote.rs: add fingerprint 2020-01-25 09:49:45 +01:00
Dietmar Maurer dcb8db66d9 src/api2/types.rs: add CERT_FINGERPRINT_SHA256_SCHEMA 2020-01-25 09:49:45 +01:00
Christian Ebner 90c815bfe2 src/bin/proxmox-backup-client.rs: fix another typo in property description
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 13:46:08 +01:00
Christian Ebner 5eee6d89b6 src/bin/proxmox-backup-client.rs: fix typo in property description
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 13:45:22 +01:00
Christian Ebner 04350b4c08 tools: add access method to LruCache
This will return a mutable reference just like get_mut, but on a cache miss
it will get and insert the missing value via the fetch method provided via the
Cacher trait.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-24 09:32:09 +01:00
Dietmar Maurer 4191018c79 src/api2/admin/datastore.rs - download_file: use BACKUP_ARCHIVE_NAME_SCHEMA for file-name 2020-01-23 13:49:30 +01:00
Dietmar Maurer a92830dc39 src/api2/types.rs: define and use api type GarbageCollectionStatus 2020-01-23 13:40:12 +01:00
Dietmar Maurer dfc58d47ad src/api2/admin/datastore.rs: use api macro for start_garbage_collection 2020-01-23 13:01:21 +01:00
Dietmar Maurer 1dc117bbba src/api2/types.rs: define and use struct StorageStatus 2020-01-23 12:51:30 +01:00
Dietmar Maurer 09b1f7b202 src/api2/types.rs: define struct BackupContent, and use it with list_snapshot_files 2020-01-23 11:17:38 +01:00
Dietmar Maurer 51534c8de9 src/backup/manifest.rs: check if manifest contains files 2020-01-23 11:16:12 +01:00
Dietmar Maurer 68a6a0ee03 src/api2/admin/datastore.rs: use api macro for delete_snapshot 2020-01-23 10:16:45 +01:00
Dietmar Maurer 6abce6c2bb src/backup/datastore.rs: remove_backup_(group/dir) - return Error instead of io::Error 2020-01-23 10:14:46 +01:00
Dietmar Maurer c425bdc9b2 src/api2/pull.rs: delete fanished snapshots 2020-01-23 09:58:57 +01:00
Dietmar Maurer 8a1d68c8b9 src/backup/datastore.rs: improve error messages 2020-01-23 09:58:14 +01:00
Dietmar Maurer 18cc66ee85 src/api2/pull.rs: use new Datastore::last_successful_backup helper
Do not fail if local group does not exists.
2020-01-22 15:06:28 +01:00
Dietmar Maurer 41b373eced src/backup/datastore.rs: new helpers
last_successful_backup: Returns the time of the last successful backup
group_path: Returns the absolute path for a backup_group
snapshot_path: Returns the absolute path for a backup_dir
2020-01-22 15:05:47 +01:00
Thomas Lamprecht 1ca037d81c followup: commit all changes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-01-22 14:14:52 +01:00
Thomas Lamprecht 44b65b4ec7 api/datastore: do not allow updating path for now
It's a bit dangerous as it points to all the saved backups, so they
would be seemingly lost after updating the path.
Follow our logic from other products, e.g. in PVE we do not allow to
update the backing path/location of a storage either for similar
reasons.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-01-22 13:55:37 +01:00
Christian Ebner 1685c2e341 tools: LRU cache to use for pxar FUSE implementation.
Implements a cache with least recently used cache replacement policy.
Internally the state is tracked by a HashMap (for fast access) and a doubly
linked list (for the access order).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-22 13:11:42 +01:00
Dietmar Maurer f1d99e3f6a src/bin/proxmox-backup-client.rs: use a std channel to write the catalog 2020-01-22 12:49:08 +01:00
Dietmar Maurer 02141b4d9b src/client/pxar_backup_stream.rs: use a channel instead of a pipe 2020-01-22 11:37:16 +01:00
Dietmar Maurer dcd033a53c src/tools/std_channel_writer.rs: new StdChannelWriter helper class 2020-01-22 11:33:57 +01:00
Wolfgang Bumiller 9ea4bce444 bump proxmox crate to 0.1.7
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).

When we need to depend on "parts" of the crate later on
we'll just have to use features.

The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 13:48:37 +01:00
Dietmar Maurer 3f06d6fbbd use proxmox_backup::tools::runtime::main to run async run_cli_command 2020-01-21 11:34:45 +01:00
Wolfgang Bumiller 077ac6f8b3 runtime: actually park on Pending...
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 10:37:16 +01:00
Wolfgang Bumiller 2d51569396 runtime: custom block_on implementation
futures::executor::block_on does not allow nesting, but we
need that...

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 10:24:52 +01:00
Wolfgang Bumiller 650e052299 runtime: drop now not required Send and static restrictions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 14:09:24 +01:00
Wolfgang Bumiller d973aa827c introduce new runtime tokio helpers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 13:12:40 +01:00
Christian Ebner aac9dbf635 pxar::fuse: remove open, opendir and releasedir callbacks
They are not needed as there is no internal state which needs to be tracked.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-20 12:56:21 +01:00
Christian Ebner a8aff3535d pxar: Include symlink target in DirectoryEntry
This allows to read the target path of a symbolic link in the
Decoder::read_directory_entry() function and stores it in the DirectoryEntry.
By this the Decoder::read_link() function becomes obsolete and is therefore
removed.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-20 12:56:10 +01:00
Dietmar Maurer 138910bcd4 src/client/pxar_backup_stream.rs: use std:🧵:Builder to name the thread
To make debugging easier.
2020-01-18 12:21:22 +01:00
Dietmar Maurer 4b4eba0b9e src/api2/pull.rs: implement delete flag for vanished groups 2020-01-17 11:24:55 +01:00
Dietmar Maurer 11d89239c3 src/backup/backup_info.rs: new list_groups helper 2020-01-17 10:42:03 +01:00
Dietmar Maurer b31c8019d7 src/api2/types.rs: define and use GroupListItem 2020-01-17 10:17:18 +01:00
Christian Ebner 2bbbade367 pxar: Refactor fuse and remove unused code.
By ambiguously using the Decoder::read_directory_entry() the code is simplified
and reading of the DirectoryEntry is concentrated into Context::run_in_context().

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
Christian Ebner 95c9460c4a pxar::fuse: Always use start offset of entries as inode.
Previously it was disciminated based on the entry mode.
For directories, the inode was the offset of the corresponding
goodbye tail mark while for all others it was the offset of the filename.

By simply using the start offset as calculated from the corresponding
goodbye table entry (which yields the archive offset of the filename),
the code is simplified and the more ambiguous read_directory_entry()
function can be used.
The disatvantage of this approach is the need to keep track of the
start and end offsets for each entry, as the end offset is needed in
order to access the goodbye table of directory entries.
The root node still has to be treated special, as it's inode is 1 as per fuse
definition and it has no filename as per the pxar file format definition.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
Christian Ebner 6744440714 pxar::decoder: refactor Decoder::lookup()
Use Decoder::read_directory_entry() instead of Decoder::attributes() as this
already returns the needed DirectoryEntry.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-17 09:40:39 +01:00
Dietmar Maurer f357390c15 renamed: src/config/remotes.rs -> src/config/remote.rs
And use 'remote' instead of 'remotes' everywhere.
2020-01-16 14:32:06 +01:00
Dietmar Maurer d2dd827877 src/backup/catalog.rs - SenderWriter: use tokio::task::block_in_place
Make sure we do not block the executor.
2020-01-16 14:24:15 +01:00
Dietmar Maurer 94609e2380 src/api2/pull.rs: use reference to remote configuration 2020-01-16 13:55:19 +01:00
Dietmar Maurer 625d19621f src/client/remote_chunk_reader.rs: use tokio::task::block_in_place to avoid blocking
Else we block the current executor.
2020-01-16 13:55:19 +01:00
Wolfgang Bumiller 6a0dc4a5bc bump proxmox crate version
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 11:40:23 +01:00
Wolfgang Bumiller 619495b25e fix use of deprecated items
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 08:58:54 +01:00
Wolfgang Bumiller 18c0df4c92 import ordering cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-16 08:51:52 +01:00
Christian Ebner ef2d7f2f96 pxar::fuse: remove readdir implementation
By not implementing readdir but only readdirplus, the FUSE_CAP_READDIRPLUS flag
is set while the FUSE_CAP_READDIRPLUS_AUTO flag is not set.
Thereby the kernel will issue only readdirplus calls.
Documentation at:
https://libfuse.github.io/doxygen/fuse-3_88_80_2include_2fuse__common_8h.html#a9b90333ad08d0e1c2ed0134d9305ee87

As the expensive part for accessing and reading the attributes is seeking and
decoding each directory entry, it is usefull to force readdirplus calls.
By this a struct `EntryParam` is returned for each entry, therebye avoiding a
subsequent lookup call.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-15 14:44:53 +01:00
Dietmar Maurer 3be839c61c src/bin/proxmox-backup-manager.rs: add completion function for update commands 2020-01-15 12:42:13 +01:00
Dietmar Maurer 002a191abf src/api2/config: check digest 2020-01-15 12:27:05 +01:00
Dietmar Maurer 347834df25 src/api2/config: correctly lock files 2020-01-15 11:57:12 +01:00
Oguz Bektas 1c79e1017f services: fix indentation
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-15 10:55:04 +01:00
Oguz Bektas cbe01dc5c7 tools: add read_and_verify_password helper
add a helper to perform some basic checks on password prompts.
- verification (asks for a 2nd time)
- check length

also use the new helper where password input in tty is taken to reduce
duplicate code.

this helper should be used when creating keys, changing passphrases etc.

note: this helper can be extended later on to provide better checks for
password strength.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-15 10:52:57 +01:00
Christian Ebner ea5061979b pxar::fuse: Avoid macro to generate filler function, use closure instead.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-15 10:25:16 +01:00
Dietmar Maurer f3ec5dae7f src/api2/config: improve api schema by adding return type 2020-01-14 14:55:58 +01:00
Dietmar Maurer ddc526623f src/bin/proxmox-backup-manager.rs: add datastore update cli 2020-01-14 14:47:26 +01:00
Dietmar Maurer c5799e4043 src/api2/config/datastore.rs: improve api, implement update and read 2020-01-14 14:45:56 +01:00
Dietmar Maurer 08195ac823 src/api2/config/remotes.rs: improve api, implement update and read 2020-01-14 14:20:16 +01:00
Dietmar Maurer 16f04b9d79 src/section_config.rs: add lookup_json() helper 2020-01-14 14:19:22 +01:00
Dietmar Maurer 71805bb7ec src/api2/config/remotes.rs: fix variable name -> s/remote/id/ 2020-01-14 13:20:13 +01:00
Dietmar Maurer d0187a51a9 src/section_config.rs - convert_to_array: optionally add digest
datastore::config() -> also return digest
remotes::config() -> also return digest
2020-01-14 12:57:03 +01:00
Dietmar Maurer 4566303b05 src/section_config.rs - write: improve error message 2020-01-14 12:02:46 +01:00
Dietmar Maurer f51420ba1f src/section_config.rs: do not allow control characters 2020-01-14 11:41:48 +01:00
Dietmar Maurer 7e7b781a18 src/api2/types.rs: also define PASSWORD_FORMAT and use it correctly 2020-01-14 11:32:02 +01:00
Dietmar Maurer da4a15a351 src/api2/types.rs: define and use PASSWORD_REGEX 2020-01-14 11:22:42 +01:00
Dietmar Maurer ff620a3d1a src/api2/types.rs: add regression test for PROXMOX_USER_ID_SCHEMA 2020-01-13 17:26:43 +01:00
Dietmar Maurer ae62c4fe59 src/api2/types.rs: fix regex patterns 2020-01-13 14:38:31 +01:00
Dietmar Maurer 163dc16c0b src/api2/types.rs: define PROXMOX_AUTH_REALM_SCHEMA and PROXMOX_USER_ID_SCHEMA
And try to use nbew schemas with config api...
2020-01-13 14:18:19 +01:00
Dietmar Maurer b25f313d66 src/api2/types.rs: define HOSTNAME_FORMAT, DNS_NAME_FORMAT and DNS_NAME_OR_IP_FORMAT 2020-01-13 13:10:52 +01:00
Dietmar Maurer 8aea35fcff src/config/*: add #[serde(skip_serializing_if="Option::is_none")] to optinal comment 2020-01-13 12:14:14 +01:00
Dietmar Maurer 454c13edce src/api2/types.rs: define SINGLE_LINE_COMMENT_SCHEMA 2020-01-13 12:02:13 +01:00
Dietmar Maurer 167971ed49 src/api2/types.rs: define REMOTE_ID_SCHEMA here 2020-01-13 11:47:07 +01:00
Christian Ebner 6de36b94b1 pxar::fuse: impl readdirplus callback
readdirplus returns the entries together with their `EntryParam`, so subsequent
lookups for each of the entries are avoided.
In order to reduce code duplication, the code for filling the reply buffer is
moved into a macro.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-13 11:17:34 +01:00
Christian Ebner caa201d895 pxar::fuse: Get rid of broken cacheing
Remove the current caching of attrs and goodbye tables as it is broken anyway.
This will be replaced with a LRU cache.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-13 11:16:06 +01:00
Oguz Bektas 6d20a29d73 fix typo for function name load_and_decrypt_key
Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-01-13 11:14:47 +01:00
Dietmar Maurer 50af953e1b src/section_config.rs: make set_data generic 2020-01-11 11:09:27 +01:00
Dietmar Maurer 688fbe07a1 cleanup config api, add remotes config cli interface 2020-01-11 10:42:09 +01:00
Dietmar Maurer 9e9bc6525e src/config/datastore.rs: define DataStoreConfig using api macro 2020-01-11 09:18:42 +01:00
Dietmar Maurer ca44172404 src/api2/config/remotes.rs: cleanup - avoid compiler warnings 2020-01-10 19:25:26 +01:00
Christian Ebner c7257c5539 pxar::fuse: refactor HashMap to store child parent inode relation.
By moving the HashMap into `Context`, the use of lazy_static as well as the
additional Mutex can be avoided (`Context` is already guarded by a Mutex).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:58 +01:00
Christian Ebner 90fc97af6a pxar::decoder::Decoder: include xattrs and payload size in `DirectoryEntry`.
By reading and including xattrs and payload size in struct `DirectoryEntry`,
the tuple of return types is avoided and the code is simpler.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:40 +01:00
Christian Ebner 6fc053ed85 pxar: encoder: limit number of max entries held at once in memory during archive creation.
Limit the total number of entries and therefore the approximate memory
consumption instead of doing this on a per directory basis as it was previously.
This makes more sense as it limits not only the width but also the depth of the
directory tree.

Further, instead of hardcoding this value, allow to pass this information as
additional optional parameter 'entires-max'.
By this, creation of the archive with directories containing a large number of
entries is possible.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-10 13:45:08 +01:00
Dietmar Maurer 141304d64e src/api2/config/remotes.rs: new API to configure remotes 2020-01-10 13:28:15 +01:00
Dietmar Maurer 331b869d96 src/bin/proxmox-backup-manager.rs: implement shell completion for remote datastore name 2020-01-10 11:58:31 +01:00
Dietmar Maurer eb506c830c rename src/api2/sync.rs -> src/api2/pull.rs
Use "pull" instead of "sync", because this also indicates a
direction (like "sync from").
2020-01-10 11:15:16 +01:00
Dietmar Maurer a81af92f9d src/section_config.rs: implement generic lookup 2020-01-09 17:35:44 +01:00
Dietmar Maurer 0eb0e02451 src/bin/proxmox-backup-manager.rs: implement cli sync command 2020-01-09 14:52:29 +01:00
Dietmar Maurer a575320657 src/config/remotes.rs: implement SectionConfig for remote hosts 2020-01-09 14:51:02 +01:00
Christian Ebner ecbc62261c pxar: encoder: warn on lacking read permissions instead of fail.
If during creation of the archive, files/dirs with lacking read permissions are
encountered, the user is displayed a warning and the archive is created without
including the file/dir.
Previously this resulted in an error and the archive creation failed.

In order to implement this also for the .pxarexclude files, the Error type of
MatchPattern::from_file() and MatchPattern::from_line() was adopted accordingly.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-09 10:41:19 +01:00
Christian Ebner 88cee60bad pxar: encoder: fix incorrect goodbye table entries and update displayed path.
If nodes are excluded by feature flags, they must not appear in the goodbye table.
This is fixed by continuing with the next entry in the for loop.

Further the relative path buffer is now poped in order to correctly display the path.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-09 10:39:41 +01:00
Christian Ebner ec906dcf19 src/pxar/encoder.rs: avoid use of macro and initialize Vec with known capacity.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-09 10:38:58 +01:00
Dietmar Maurer 71da3d6a1b src/api2/types.rs: fix SnapshotListItem schema definition 2020-01-09 10:36:19 +01:00
Dietmar Maurer de8ec04123 src/api2/sync.rs: implement remote sync 2020-01-08 14:03:52 +01:00
Dietmar Maurer 27c9affb23 src/bin/proxmox-backup-client.rs: cleanup - factor out api_datastore_latest_snapshot() 2020-01-07 15:18:36 +01:00
Dietmar Maurer 42af4b8f06 src/bin/proxmox-backup-client.rs: cleanup - factor out api_datastore_list_snapshots() 2020-01-07 15:07:57 +01:00
Dietmar Maurer af9d4afc66 src/bin/proxmox-backup-client.rs: use SnapshotListItem 2020-01-07 13:25:25 +01:00
Dietmar Maurer fc189b198c src/api2/types.rs: define and use new api type SnapshotListItem 2020-01-07 12:52:52 +01:00
Dietmar Maurer 8f14e8fe4c src/backup/backup_info.rs: implement last_successful_backup()
A backup is considered successful if there is a manifest file.
2020-01-06 11:35:22 +01:00
Dietmar Maurer 247a8ca5bb src/backup/manifest.rs: impl TryFrom DataBlob 2020-01-05 16:28:18 +01:00
Dietmar Maurer 7759eef552 src/backup/datastore.rs: implement cleanup_backup_dir() 2020-01-05 15:15:12 +01:00
Dietmar Maurer 5e1d1bead1 src/client/backup_reader.rs: make compute_file_csum() public 2020-01-05 14:50:54 +01:00
Dietmar Maurer 3743dee6de src/client/http_client.rs: allow access to AuthInfo fields 2020-01-05 10:31:19 +01:00
Dietmar Maurer 3758b398cd src/backup/read_chunk.rs: add read_raw_chunk()
Allow to read chunk data without decoding.
2020-01-02 13:29:10 +01:00
Dietmar Maurer 2585a8a4e2 src/backup/chunk_store.rs: implement cond_touch_chunk()
This will be used by backup sync to test if a chunk already exists.
2020-01-02 13:26:28 +01:00
Dietmar Maurer 1cf5178ac5 src/backup/datastore.rs: new helper try_shared_chunk_store_lock() 2020-01-02 11:00:33 +01:00
Dietmar Maurer 1e8da0a789 src/backup/manifest.rs: new helper archive_type() 2019-12-31 15:23:41 +01:00
Dietmar Maurer 94f085fab5 src/bin/proxmox-backup-proxy.rs: remove unused use clause 2019-12-30 11:50:24 +01:00
Dietmar Maurer 22be470d90 src/bin/proxmox-backup-api.rs: create cert here 2019-12-27 13:44:21 +01:00
Dietmar Maurer ca6119551d src/client/http_client.rs - start_h2_connection: correctly return error during upgrade 2019-12-27 13:41:31 +01:00
Wolfgang Bumiller a546a8a072 proxy/api: await running workers before shutdown
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-27 12:50:33 +01:00
Dietmar Maurer f8fd5095d8 src/bin/proxmox-backup-proxy.rs: create self signed cert at startup. 2019-12-27 11:21:32 +01:00
Dietmar Maurer afddffc7ec proxmox-backup-manager.rs: remove wrong fixme 2019-12-27 10:17:06 +01:00
Dietmar Maurer e739a8d8b4 src/bin/proxmox-backup-manager.rs: impl. cert info 2019-12-23 13:06:43 +01:00
Dietmar Maurer 550e0d8870 src/bin/proxmox-backup-manager.rs: add cli to generate proxy certificate
Without calling external openssl binary.
2019-12-22 17:35:06 +01:00
Wolfgang Bumiller 5e62d19cfb fix create_datastore parameters:
comment was declared as optional but unwrap()ed, causing
panics when not using it

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-20 12:30:25 +01:00
Dietmar Maurer 645995634a src/api2/config/datastore.rs - create: pass uid and gid instead of User 2019-12-20 09:23:58 +01:00
Dietmar Maurer e67770d496 src/backup/chunk_store.rs - create: pass User instead of CreateOptions 2019-12-20 09:11:40 +01:00
Thomas Lamprecht bca9093520 api/ui: datastore: allow to set simple comment
for now forbid all control characters[0] in the comment value, the
section config writer cannot cope with newlines in the value, it
writes them out literally, allowing "injection" or breaking the whole
config.

In the webinterface use also a textfield, not a textarea.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-19 17:58:01 +01:00
Wolfgang Bumiller afdcfb5bc9 let ChunkStore::create take CreateOptions
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-19 13:14:49 +01:00
Dietmar Maurer 47d4712152 src/bin/proxmox-backup-manager.rs: add task management cli 2019-12-19 11:27:36 +01:00
Dietmar Maurer d6c4a1198a src/bin/proxmox-backup-client.rs: use async fn 2019-12-19 11:20:59 +01:00
Dietmar Maurer f74a03da1f remove tools::getpwnam_ugid, impl. crate::backup::backup_user()
And use new nix::unistd::User struct.
2019-12-19 10:20:13 +01:00
Dietmar Maurer 8bf4559b4e update to nix 0.16 2019-12-19 09:29:44 +01:00
Dietmar Maurer 769f8c9998 src/bin/proxmox-backup-manager.rs: connect to daemon to execute GC
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2019-12-19 08:45:27 +01:00
Dietmar Maurer 25de1c8005 src/client/backup_repo.rs: impl. new() 2019-12-19 08:44:38 +01:00
Dietmar Maurer 94913f35ca src/bin/proxmox-backup-client.rs - start_garbage_collection: add missing output-format parameter 2019-12-19 07:57:53 +01:00
Dietmar Maurer 868c585219 src/backup/datastore.rs: avoid divide by zero 2019-12-19 07:14:23 +01:00
Dietmar Maurer 217170e11e src/server/worker_task.rs: set worker thread name
This gives better logs on thread panic...
2019-12-19 07:07:39 +01:00
Dietmar Maurer 7e210bd0b4 src/backup/chunk_store.rs: create lock file with correct owner 2019-12-19 06:55:53 +01:00
Thomas Lamprecht 2337df7b3d api2: add very basic 'status/usage' endpoint for nodes
For returning the nodes basic "usage status", for now one gets memory
and CPU utilization.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-18 17:24:32 +01:00
Dietmar Maurer bce023c57c src/tools.rs: fix typo 2019-12-18 14:29:29 +01:00
Dietmar Maurer 0b97bc6158 src/backup/chunk_store.rs: use proxmox::tools::fs::create_path 2019-12-18 12:26:43 +01:00
Dietmar Maurer f8ec1473be src/config/datastore.rs: use backup gid instead of uid 2019-12-18 12:21:44 +01:00
Wolfgang Bumiller feaa1ad35f replace file_set_contents with replace_file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-18 11:16:04 +01:00
Dietmar Maurer a66e7920dc src/api2/config/datastore.rs: set protected flags for create/delete API 2019-12-18 10:52:45 +01:00
Dietmar Maurer f0db500808 src/api2/config/datastore.rs: fix api docu 2019-12-18 10:48:19 +01:00
Dietmar Maurer 424766bc3b src/config/datastore.rs: change file owner/permissions
owner(root) => read and write
group(backup) => read only
2019-12-18 10:41:58 +01:00
Oguz Bektas c07b458cac datastore: set correct owner on datastore.cfg
since manager runs as root, we have to specify backup:root owner while
saving config.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-12-17 16:59:10 +01:00
Oguz Bektas 14f1e63067 chunk_store: create parent directories
'datastore create storename /path/to/dir/that/may/not/exist' should
work.

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2019-12-17 15:39:42 +01:00
Christian Ebner ebfb2df22b binary_search_tree: add additional doctest for search_binary_tree_by()
Make sure a start indexes larger or equal to the array size results in a return
value of `None`.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 15:37:02 +01:00
Christian Ebner 48f6d67764 binary_search_tree: fix off by one error in search_binary_tree_by()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 14:57:34 +01:00
Christian Ebner 38d9a69875 catalog: decoder and shell: fix incorrect logic in find matching
The find matching was incorrectly performed starting from the parent directroy
and not as intended from the entries of the parent directory.

Further, the match pattern passed from the catalog shell contains the absolute
path of the search entry point as prefix, so find() must always start from the
archive root. This is because the match pattern has to be stored in the selected
list for a subsequent restore-selected command in the shell.
All matching paths are shown as absolute paths with all contents in the subdir,
equal to what would be restored by the given pattern.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 14:57:34 +01:00
Dietmar Maurer 81cc71c0d7 use proxmox-mini-journalreader to display syslog 2019-12-17 14:06:48 +01:00
Dietmar Maurer 20197f7cf5 src/api2/node/syslog.rs: use api macro 2019-12-17 13:26:49 +01:00
Thomas Lamprecht 9c01e73c2f ui: add "Documentation" button to main view
Similar to PVE and PMG, for quick access when one has the basic
webinterface open anyway. Should move to the "proxmoxHelpButton" once
we have an onlineHelp mapping to the docs.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 11:22:34 +01:00
Thomas Lamprecht 2d694f8f1f add CSS file for PBS ExtJS6 basic ui
some fitting rules copied over from PVE's ext6-pve.css file.
simply place it in the css subfolder where the proxmox-backup-gui.js
file is hosted and add a "css/" alias for that directory, the
formatter gets use the right content type with that.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 11:20:32 +01:00
Christian Ebner 25cdd0e0a1 catalog_shell: impl find for catalog shell.
Implements the find command which allows to find and select files for subsequent
restore.
Files selected for restore are now stored in a Vec instead of a HashSet.
This is needed, since instead of the full paths for each file, selected files are
now identified by a list of match pattern, where ordering matters.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
Christian Ebner 90dfd0a71d catalog: impl find() for file search by match pattern.
find() iterates over the file tree and matches each node against a list of match
patterns provided at function call.
For each matching node, a callback function with the current directroy stack is
called.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
Christian Ebner a333b4ccea pxar: match_pattern: impl invert() to switch the match type of a MatchPattern.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
Christian Ebner 48af80b3ab pxar: match_pattern: impl traits needed for MatchPattern to be comparable.
In order to check if two MatchPattern are identical, which will be used to avoid
duplicates.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
Christian Ebner 255bb86030 pxar: match_pattern: refactor MatchPattern and introduce MatchPatternSlice.
The MatchPattern impl heavily used copies and therefore was inefficient regarding
memory management.
This patch intoduces MatchPatternSlice as struct to avoid copies and perform the
same pattern matching functionality.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-17 11:12:34 +01:00
Wolfgang Bumiller dc08934563 drop Cancellable future in favor of abortable
futures-0.3 has a futures::future::abortable() function
which does the exact same, returns an Abortable future with
an AbortHandle providing an abort() method.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-17 10:52:07 +01:00
Wolfgang Bumiller 8554ac5ec3 fix broadcast_future test case
We used to await all the futures via the runtime's shutdown
method, which doesn't exist anymore, so await all the join
handles instead.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-17 10:47:12 +01:00
Thomas Lamprecht ba3eb88d95 generate authkey: public part needs to be readable by backup group
else the API proxy cannot use it and fails to run..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 10:17:03 +01:00
Thomas Lamprecht 9c18e93522 handle_static_file_download: move from and_then to await
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 08:56:55 +01:00
Thomas Lamprecht 4e8cff1c9d cipherbench: align/format output nicely
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 08:49:34 +01:00
Thomas Lamprecht 4832f53ab5 cipherbench: be conform with rfc7539, use 12 byte nonce
else we get an error from this call, using a 16 byte (128 bit) nonce
is currently only supported by the still in draft
XChaCha20-Poly1305, not the current default specified by RFC 7539[0],
which uses a 12 byte (96 bit) nonce.

Fixes the following error:
> thread 'main' panicked at 'called `Result::unwrap()` on an `Err`
> value: ErrorStack([])', src/libcore/result.rs:1165:5

[0]: https://tools.ietf.org/html/rfc7539

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 08:49:07 +01:00
Dietmar Maurer a47a02ae62 src/bin/proxmox-backup-client.rs: use #[api] macro 2019-12-16 13:49:18 +01:00
Wolfgang Bumiller 78d5436040 client: use api macro for catalog_shell async fn
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-16 11:24:44 +01:00
Dietmar Maurer 87c423751a src/bin/proxmox-backup-client.rs: use ApiHandler::Async for prune and status 2019-12-16 11:00:43 +01:00
Dietmar Maurer 8a8a470316 src/bin/proxmox-backup-client.rs: use new ApiHandler::Async 2019-12-16 10:06:26 +01:00
Dietmar Maurer b9799012cf src/pxar/decoder.rs: require Send 2019-12-16 10:01:51 +01:00
Dietmar Maurer bb084b9c91 api2: update for latest proxmox-api changes
- rename ApiFuture into ApiResponseFuture
- impl. ApiHandler::Async
2019-12-16 10:01:51 +01:00
Dietmar Maurer 236a396aa1 src/api2/admin/datastore.rs - prune: log retention options 2019-12-14 16:32:16 +01:00
Dietmar Maurer 503995c767 src/api2/admin/datastore.rs: fix prune worker id 2019-12-14 16:05:21 +01:00
Wolfgang Bumiller f374ba4c0b update tests
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 12:42:02 +01:00
Wolfgang Bumiller b5c9a706ce fix cancellable future
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 12:10:04 +01:00
Wolfgang Bumiller 8a1028e0e7 update to url/percent-encoding to 2.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:57:01 +01:00
Wolfgang Bumiller db0cb9ce0b update a chunk of stuff to the hyper release
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:24:41 +01:00
Dietmar Maurer 3f1020b732 src/bin/proxmox-backup-client.rs: implement task stop 2019-12-13 09:43:08 +01:00
Dietmar Maurer 1ae5677de0 src/api2/types.rs: use PROXMOX_SAFE_ID_FORMAT instead of FILENAME_FORMAT 2019-12-12 14:15:40 +01:00
Dietmar Maurer d0adf270fb src/api2/types.rs: introduce PROXMOX_SAFE_ID_REGEX, use it for DATASTORE_SCHEMA 2019-12-12 12:37:11 +01:00
Dietmar Maurer 66c49c21c3 src/api2/types.rs: factor out DATASTORE_SCHEMA 2019-12-11 13:45:23 +01:00
Dietmar Maurer 567d3e00fb src/api2/node/tasks.rs: new filter "store" to filter tasks for one store 2019-12-11 12:53:34 +01:00
Christian Ebner d3dbe52f37 pxar: match_pattern: refactor match_filename
The match_filename() in sequentail_decoder and encoder are moved to be static
functions of MatchPattern.
This allows to reuse the code also in the catalog find implementation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-11 10:55:01 +01:00
Christian Ebner f084505ec5 src/backup/catalog_shell.rs: sort output of list-selected
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-11 10:54:20 +01:00
Dietmar Maurer 6d04612955 src/api2/node/tasks.rs: add "running" filter to list only running tasks 2019-12-11 10:52:27 +01:00
Dietmar Maurer 8560fe3e5f src/api2/node/tasks.rs - extract_upid: avoid duplicate error message 2019-12-10 13:45:56 +01:00
Dietmar Maurer 5830c20560 src/bin/proxmox-backup-client.rs: add simple task management cli 2019-12-10 13:43:53 +01:00
Dietmar Maurer 73e57f244e src/backup/datastore.rs - garbage-collection: fix average chunk size 2019-12-10 11:25:29 +01:00
Dietmar Maurer d105176f45 src/bin/proxmox-backup-client.rs: cleanup - factor out view_task_result() 2019-12-09 18:00:22 +01:00
Dietmar Maurer 48ef3c3327 cli: avoid useless .into()
This needs latest proxmox-api updates.
2019-12-09 17:40:34 +01:00
Dietmar Maurer eb7e2ee00b src/bin/proxmox-backup-client.rs: remove unnecessary .to_owned() 2019-12-09 17:11:48 +01:00
Dietmar Maurer e5f7def40e src/bin/proxmox-backup-client.rs - garbage-collection: display task log 2019-12-09 13:29:23 +01:00
Dietmar Maurer 0ec9e1b010 src/bin/proxmox-backup-client.rs: impl. and use complete_pxar_archive_name 2019-12-09 13:14:32 +01:00
Dietmar Maurer 1c6ad6ef90 src/bin/proxmox-backup-client.rs: impl. catalog sub command
catalog dump
catalog shell
2019-12-09 13:07:25 +01:00
Dietmar Maurer 5a0b484bdd src/client/task_log.rs: move display_task_log into extra file 2019-12-08 11:27:15 +01:00
Dietmar Maurer 3b03abfe14 src/api2/admin/datastore.rs: remove test_prune, add dry-run to normal prune
And use display_task_log on the client side.
2019-12-08 10:59:47 +01:00
Dietmar Maurer ec137a99c6 src/bin/proxmox-backup-client.r - display_task_log: use string range instead of as_bytes 2019-12-08 10:25:26 +01:00
Dietmar Maurer b73ff941f6 src/bin/proxmox-backup-client.r - display_task_log: add new strip_date parameter 2019-12-08 10:22:02 +01:00
Dietmar Maurer 163e9bbe91 src/bin/proxmox-backup-client.rs: new display task log helper
New helper to display worker task logs. Use it for prune (experimental).
2019-12-07 16:11:26 +01:00
Dietmar Maurer 6b508dd563 src/server/formatter.rs: impl. new result attribute "active" 2019-12-07 15:29:42 +01:00
Dietmar Maurer 102d8d4136 src/backup/prune.rs: implement --keep-hourly 2019-12-07 11:23:33 +01:00
Dietmar Maurer 052c4aa160 src/backup/prune.rs: prune - correctly use iso_week year 2019-12-06 19:41:38 +01:00
Christian Ebner fa0b950070 pxar: fuse: add support for list- and getxattr
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 16:44:15 +01:00
Christian Ebner 25ad4cbf63 pxar: fuse: cache goodbye table for each directory on opendir and release it on releasedir
Cache not only the goodbye table for the last directory but for each opened
directory.
The opendir fuse callback will fill the cache with the goodbye table and
releasedir will remove it from the cache.
This should reduce the number of chuncks fetched from the server in some cases.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 16:43:59 +01:00
Christian Ebner 02491b8fc6 pxar: fuse: use binary search instead of linear search in goodbye table.
The goodbye table is layed out as binary search tree based on the hash, so use
this to be more efficient when looking up a hash in the table for directories
with a large number of entries.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 16:42:01 +01:00
Dietmar Maurer 7074a0b339 src/bin/proxmox-backup-client.rs: white space cleanup 2019-12-06 13:35:59 +01:00
Dietmar Maurer 74fa81b89c src/bin/proxmox-backup-client.rs: implement prune --dry-run 2019-12-06 13:35:25 +01:00
Dietmar Maurer 1f604f1a5a src/api2/admin/datastore.rs: add new api test_prune() 2019-12-06 13:06:56 +01:00
Dietmar Maurer 0ab08ac980 src/api2/admin/datastore.rs: cleanup - move schema defininition in front of rust method 2019-12-06 12:36:41 +01:00
Dietmar Maurer 1a0678b601 src/api2/admin/datastore.rs: remove prune parameters from status call
Not idea why they are added there.
2019-12-06 12:32:53 +01:00
Dietmar Maurer 9e3f008804 src/backup/prune.rs: add new helper keeps_something() 2019-12-06 12:28:31 +01:00
Dietmar Maurer d75fbb0a21 src/bin/dump-catalog-shell-cli.rs: generate catalog shell docs 2019-12-06 09:54:13 +01:00
Dietmar Maurer 55c3cb69cd rename catalog_shell_api() into catalog_shell_cli() 2019-12-06 09:42:05 +01:00
Dietmar Maurer ecbaa38fa4 src/backup/catalog_shell.rs: factor out catalog_shell_api()
In order to extract documentation ...
2019-12-06 09:35:47 +01:00
Christian Ebner cca67dc0c3 src/backup/readline.rs: removed in favor of new readline impl
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 09:03:04 +01:00
Christian Ebner 6934c6fe77 src/backup/catalog_shell.rs: adapt to use API Schema definition and rustyline
This major refactoring of the catalog based shell utilizes the new API macro and
the API Schema as well as rustyline instead of the old GNU readline C API.

The code now has these 3 main components:
 * The `Shell` which handles the readline loop via rustyline.
 * The shell functions defined via the API macro.
 * The `Context` which holds catalog and decoder instances.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-06 09:02:44 +01:00
Dietmar Maurer 9b78352188 src/backup/prune.rs: define new struct PruneOptions 2019-12-06 08:56:27 +01:00
Dietmar Maurer 92acbd69f7 src/backup/backup_info.rs: remove old prune code 2019-12-06 08:29:27 +01:00
Dietmar Maurer 408434360b src/backup/prune.rs: add PruneMark::KeepPartial
Use separate mark for partial backup - dont mess them up with regular ones.
2019-12-06 08:15:32 +01:00
Dietmar Maurer a8c8366cfa src/backup/prune.rs: factor out code into remove_incomplete_snapshots() 2019-12-06 08:12:08 +01:00
Dietmar Maurer dc18849156 src/backup/prune.rs: moved prune related code into extra file 2019-12-06 08:06:21 +01:00
Dietmar Maurer 6f47dd8a0f src/backup/backup_info.rs - prune: improve algorythm 2019-12-05 18:41:28 +01:00
Dietmar Maurer 8f0b4c1f90 src/api2/admin/datastore.rs - prune: log info about all snapshots 2019-12-05 15:35:11 +01:00
Dietmar Maurer 9ce42759ec src/backup/backup_info.rs - compute_prune_list: remove unfinished backups 2019-12-05 11:18:10 +01:00
Dietmar Maurer 2c034f8d0a src/backup/backup_info.rs: improve prune algorithm 2019-12-05 08:55:19 +01:00
Dietmar Maurer aeeac29bb1 src/backup/backup_info.rs: new compute_prune_list helper
So that we can write regression tests for this.
2019-12-04 15:49:11 +01:00
Dietmar Maurer 3d229a4a01 Revert "test another api macro use"
This reverts commit a9aa52e6a8.

Because we do not want to use macros for the backup protocol for now.

And because it crashes backup tasks for some unknown reason.
2019-12-04 14:18:11 +01:00
Dietmar Maurer 4f1e40a29e src/api2/admin/datastore.rs - read_backup_index: add manifest with size
So that the client can display the size of the index file.
2019-12-04 10:03:52 +01:00
Wolfgang Bumiller a9aa52e6a8 test another api macro use
The api macro now supports hyphens in parameter names and
referencing externally defined `Schema`s, so here's an
example.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-03 13:41:40 +01:00
Christian Ebner de0aa53174 pxar: decoder: fix incorrect read for hardlinks.
Decoders read must check if the file is a hardlink and read data from the
corresponding offset if so.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-03 12:08:23 +01:00
Christian Ebner b7e18004a2 pxar: decoder: return original name in attributes()
The returned filename should be the one of the file given at the offset, not of
the one the hardlink points to.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-12-03 12:08:07 +01:00
Dietmar Maurer 7eea56ca00 use cli from proxmox::api 2019-12-02 11:56:29 +01:00
Dietmar Maurer 524b22df35 src/cli/completion.rs - get_help_completion: fix for "help -" expansion 2019-12-02 11:11:31 +01:00
Dietmar Maurer e8e9bae4d1 src/bin/completion.rs: add echo command 2019-12-02 11:04:40 +01:00
Dietmar Maurer 19d7a59236 src/bin/completion.rs: use api macros 2019-12-02 10:57:19 +01:00
Dietmar Maurer f9a51322cb src/cli/command.rs: use insert_help() 2019-12-02 08:52:08 +01:00
Dietmar Maurer e3e72546dd src/cli/command.rs: cleanup, remove unused top_def parameter 2019-12-02 08:47:03 +01:00
Dietmar Maurer 2348e77edf src/cli/completion.rs - get_simple_completion: remove all_arg_param parameters
Seems useless.
2019-12-02 08:31:03 +01:00
Dietmar Maurer 28c855c0a2 src/cli: improve docs 2019-12-01 16:41:49 +01:00
Dietmar Maurer 3bf920527c src/cli/command.rs: make set_help_context private - no need to expose 2019-12-01 12:54:41 +01:00
Dietmar Maurer bca6d9120a src/cli/command.rs: improve help command description 2019-12-01 12:40:37 +01:00
Dietmar Maurer 2f6ecc1ccd src/cli/completion.rs: moved all completion related code here 2019-12-01 11:44:31 +01:00
Dietmar Maurer a810e05288 src/cli/readline.rs: moved readline related code here 2019-12-01 11:31:35 +01:00
Dietmar Maurer 5d64a0d083 src/cli/shellword.rs: moved shellword related code into this file 2019-12-01 11:17:57 +01:00
Dietmar Maurer fb8a00bc00 src/cli/command.rs - get_nested_completion: allow fuzzy match 2019-12-01 11:06:54 +01:00
Dietmar Maurer 9498fcf9c5 src/cli/command.rs - get_nested_completion: return nothing if first argument does not match 2019-12-01 10:24:07 +01:00
Dietmar Maurer 309e449db2 src/cli/command.rs: improve get_help_completion 2019-12-01 10:18:47 +01:00
Dietmar Maurer b8a82922c7 src/cli/command.rs: handle multilevel sub commands in help 2019-12-01 09:42:51 +01:00
Dietmar Maurer b7bf71dfa1 src/cli/command.rs: fix help completions 2019-11-30 15:43:44 +01:00
Dietmar Maurer 7afc1af7ab src/bin/completion.rs: add history support 2019-11-30 15:08:37 +01:00
Dietmar Maurer 4b8573da65 src/cli/command.rs - handle_command: return Result instead of exit() 2019-11-30 15:08:00 +01:00
Dietmar Maurer 2b691daf6f src/cli.rs: add insert_help() method 2019-11-30 14:56:31 +01:00
Dietmar Maurer 1201abcffa src/cli/command.rs: implement help 2019-11-30 13:02:00 +01:00
Wolfgang Bumiller 8423c1fe64 cli/completion: use match statements and .contains()
`match` is a bit more readable than the if-else chains,
also replace
    space_chars.iter().any(|s| c == *s)
with
    space_chars.contains(&c)
which is also more readable.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-29 13:37:34 +01:00
Dietmar Maurer e399398444 remove shellwords crate 2019-11-29 12:49:44 +01:00
Dietmar Maurer ca60ac13d9 use rustyline to replace readline 2019-11-29 12:49:44 +01:00
Wolfgang Bumiller 41874331ed whitespace fixup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-29 09:52:15 +01:00
Wolfgang Bumiller 2905f2b5e6 update api macro example usage
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-29 09:51:27 +01:00
Dietmar Maurer 8b130e71da src/cli/command.rs: cleanup, make handlers more generic 2019-11-27 14:44:21 +01:00
Wolfgang Bumiller 7b6c41078b update api macro invocation to new style
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-27 14:19:46 +01:00
Christian Ebner 314bb35868 catalog_shell: fix several clippy warnings
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-27 10:13:50 +01:00
Wolfgang Bumiller 6486cb853f first api macro usage test/example
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-27 10:05:37 +01:00
Christian Ebner 951cf17ee3 catalog_shell: major refactoring of cli command definition and parsing
By changing the way shell commands are defined and parsed, this makes it more
straight forward to extend the current functionality.
The readline input is parsed based on the provided command definition and the
given parameters and options are passed to a command specific callback function.
In addition, the provided command definition including its description is used
to generate a help string to display.
The help command shows a list of all supported commands or the help string for
the provided command.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 13:01:00 +01:00
Christian Ebner 59bc6ad676 catalog_shell: check and return on empty list in order to avoid division by zero
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 12:55:33 +01:00
Christian Ebner 4e56b2f792 catalog_shell: do not restore full archive if no entries are selected for restore-selected
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-26 12:55:18 +01:00
Dietmar Maurer 12b4098660 src/cli/format.rs: move doc generation code into extra file 2019-11-26 12:12:53 +01:00
Dietmar Maurer 2f2b87e659 src/pxar/fuse.rs: add impl Context 2019-11-26 11:16:46 +01:00
Dietmar Maurer eedd1f95eb src/pxar/fuse.rs: move init and destroy into Session 2019-11-26 11:08:55 +01:00
Dietmar Maurer 970687c982 src/pxar/fuse.rs: cleanup - remove setup_session (moved into new) 2019-11-26 11:04:46 +01:00
Dietmar Maurer 2a11191072 src/pxar/fuse.rs: rename from_decoder into new, new into from_path 2019-11-26 10:56:41 +01:00
Dietmar Maurer 99b5b6cba9 src/pxar/decoder.rs: use trait object as reader
So that the Decoder is no longer genertic.
2019-11-26 10:45:11 +01:00
Dietmar Maurer f701d0335e src/pxar/sequential_decoder.rs: remove callback from new()
And use an extra functzion  set_callback() to configure that.

Also rewrite pxar/fuse.rs and implement a generic Session (will get
further cleanups with next patches).
2019-11-26 09:56:48 +01:00
Dietmar Maurer 49fddd985c src/cli/command.rs: use static array for arg_param 2019-11-25 11:16:41 +01:00
Dietmar Maurer 329d40b50b rename ApiHandler::Async into ApiHandler::AsyncHttp 2019-11-23 09:03:21 +01:00
Dietmar Maurer 70fbac84da src/server/rest.rs: simplify code 2019-11-22 18:44:14 +01:00
Dietmar Maurer 2bbd835b9b src/server/rest.rs: rename get_request_parameters_async to get_request_parameters 2019-11-22 17:24:16 +01:00
Dietmar Maurer 75a5a68944 src/server/rest.rs - only pass ObjectSchema to get_request_parameters_async() 2019-11-22 17:22:07 +01:00
Dietmar Maurer ad51d02aa9 src/server/rest.rs: cleanup async code 2019-11-22 13:02:05 +01:00
Wolfgang Bumiller be2bb37205 remove api_schema submodule
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-22 09:31:45 +01:00
Dietmar Maurer e57e1cd82d move src/api_schema/config.rs -> src/server/config.rs 2019-11-22 09:23:03 +01:00
Dietmar Maurer 8256b0e417 src/api_schema/registry.rs: remove dead code 2019-11-22 09:04:52 +01:00
Christian Ebner 3cf73c4e53 src/bin/proxmox-backup-client.rs: impl shell subcommand for client
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 08:22:50 +01:00
Christian Ebner f14c96ea38 src/backup/catalog_shell.rs: impl shell to inspect and restore a snapshot via the catalog
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:54:04 +01:00
Christian Ebner 38446a9551 src/backup/readline.rs: impl wrapper for GNU readline
In order to provide the context needed for tab completion via the readline
callback, the needed mut ref is passed via a thread local storage key.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:46:24 +01:00
Christian Ebner 6dba015043 src/backup/catalog.rs: derive Clone for DirEntry and DirEntryAttribute
This is needed in order to explicitly clone the values when needed in the
catalog shell implementation.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:15:39 +01:00
Christian Ebner c9f002213e src/backup/catalog.rs: impl DirEntry::is_directory()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:14:00 +01:00
Christian Ebner d3bcc7f261 src/pxar/sequential_decoder.rs: create all directories in target path
Create the full target path and not fail if an intermediate directory does not
exist.
This is needed in order to restore multiple archives via the catalog, where the
target should further contain each archive name as subdir.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:11:49 +01:00
Christian Ebner b6ca690095 src/pxar/decoder.rs: return also payload size in lookup
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-22 07:05:14 +01:00
Dietmar Maurer 45cc157f7c docs: add backup reader protocol 2019-11-22 06:59:37 +01:00
Wolfgang Bumiller cad540e969 api/compat: remove remaining api_schema references
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:53:15 +01:00
Wolfgang Bumiller a2479cfa1a api/compat: drop more compat imports from api_schema.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:36:28 +01:00
Wolfgang Bumiller cc4ecf80b0 api/compat: drop api_handler submodule
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:18:41 +01:00
Wolfgang Bumiller 4f1963d259 api/comapt: drop schema submodule
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:17:36 +01:00
Wolfgang Bumiller 2194824f77 api: BoxFut -> ApiFuture
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:16:37 +01:00
Wolfgang Bumiller 3d482025b3 move api schema into proxmox::api crate
And leave some compat imports in api_schema.rs to get it to
build with minimal changes.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:14:54 +01:00
Dietmar Maurer 552c225948 sort all property lookup tables
Required, because we use binary sreach to find items.
2019-11-21 13:32:09 +01:00
Dietmar Maurer 255f378a1b use const api definitions 2019-11-21 13:32:09 +01:00
Wolfgang Bumiller c5946faffd drop md5 crate dependency
we already depend on openssl which also provides md5

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-14 11:19:43 +01:00
Wolfgang Bumiller f569acc5e2 drop uuid crate dependency
proxmox::tools now has a Uuid module using the native
libuuid.

Adds build dependency: libuuid1 (which is a Pre-Depends of
util-linux, so always installed anyway).

Drops uuid + 16 more crate dependencies.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-14 10:22:29 +01:00
Christian Ebner b423958d8e catalog: fix issue with DirEntry start
start has to be the parent start - offset

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-12 17:18:13 +01:00
Dietmar Maurer 780dd2b0a1 src/backup/catalog.rs: add some docs 2019-11-12 13:23:40 +01:00
Dietmar Maurer 6aa906b547 src/backup/catalog.rs - CatalogReader: verify magic number 2019-11-12 12:54:06 +01:00
Dietmar Maurer 2ec208aef5 src/backup/catalog.rs - CatalogReader::dump(): use root() 2019-11-12 12:47:21 +01:00
Dietmar Maurer 8f24a9ea18 src/backup/catalog.rs - Direntry::parse: abort if callback return false 2019-11-12 11:41:44 +01:00
Dietmar Maurer 7d017123fd src/backup/catalog.rs: impl read_dir() and lookup(), refactor common code 2019-11-12 11:41:44 +01:00
Dietmar Maurer dc9596de45 src/backup/catalog.rs: move CatalogEntryType from src/pxar/catalog.rs 2019-11-12 11:41:44 +01:00
Dietmar Maurer 55c0b3cc7c src/backup/catalog.rs - DirEntry: factor out name attribute
And avoid allocations inside parse().
2019-11-12 11:41:44 +01:00
Christian Ebner 3f1c5b5e65 catalog: impl std::fmt::Display trait for CatalogEntryType
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-12 07:00:08 +01:00
Dietmar Maurer f89359c2f3 src/backup/catalog.rs: clenup - use single impl block 2019-11-10 11:38:31 +01:00
Dietmar Maurer e8d8fdc61d src/backup/catalog.rs: remove unused code 2019-11-10 11:37:06 +01:00
Dietmar Maurer c74c074bfd src/backup/catalog.rs: write MAGIC at start of file 2019-11-10 11:36:45 +01:00
Dietmar Maurer 265664517a src/backup/catalog.rs: new parse() helper 2019-11-09 14:37:33 +01:00
Dietmar Maurer 5d92935e03 src/backup/catalog.rs - dump_dir: make sure we parsed the whole data block 2019-11-09 13:21:20 +01:00
Dietmar Maurer 12a1975af9 src/backup/catalog.rs: store u64 compressed
This way catalogs get about 1/3 smaller.
2019-11-09 13:07:47 +01:00
Dietmar Maurer bb19af7302 src/client/backup_reader.rs: factor out download_blob() helper 2019-11-08 13:16:41 +01:00
Dietmar Maurer 7205050059 src/client/backup_reader.rs: factor out download_fixed_index() helper 2019-11-08 12:51:52 +01:00
Dietmar Maurer c3d84a2281 src/client/backup_reader.rs: factor out download_dynamic_index() helper 2019-11-08 12:51:52 +01:00
Dietmar Maurer 89245fb530 renamed: src/backup/catalog_blob.rs -> src/backup/catalog.rs 2019-11-08 10:41:00 +01:00
Dietmar Maurer bf6e321744 change catalog format, use dynamic index to store catalog.
In order to remove size restriction of a single blob.
2019-11-08 10:35:48 +01:00
Dietmar Maurer e016f9ff2f src/client/backup_writer.rs: rename force_close to cancel
And do not consume self.
2019-11-08 10:34:16 +01:00
Dietmar Maurer 0997967d65 do no restrict index names too much 2019-11-08 10:28:14 +01:00
Christian Ebner e66621182b src/pxar/decoder.rs: impl lookup for `Decoder`
Allows to lookup an entry in a directory based on the provided `DirectoryEntry`.
This is needed to navigate the filesystem based on `DirectoryEntry`s and similar
to the find_goodbye_entry() function in src/pxar/fuse.rs

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-06 10:00:41 +01:00
Christian Ebner fb2554de29 src/pxar/decoder.rs: fix wrong filename check for Decoder::restore()
As it turns out the original implementation was correct and the start in
`DirectoryEntry` points to the `PxarEntry` and not as wrongly stated to the
filename.
This reverts the incorrect code and adds comments to the fields clarifying this.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-06 09:59:35 +01:00
Christian Ebner 33ad183a40 src/pxar/decoder.rs: allow to pass match pattern to restore
In order to partially restore the archive starting from the directory provided as
parameter.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-05 14:19:09 +01:00
Christian Ebner 5e8d600c71 src/pxar/decoder.rs: fix issue with restore
`Decoder::restore()` calls the `SequentialDecoder::restore()` which expects to
encounter a `PxarEntry` at first. But the start of `DirectoryEntry` points to the
filename (except for the root dir), so skip over it.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-05 14:18:29 +01:00
Christian Ebner 9708b2d98c src/pxar/fuse.rs: refactor Iterator: use find() instead of position()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-11-04 13:59:16 +01:00
Christian Ebner b82a101c86 catalog: store current writer position in CatalogBlobWriter
Counts the bytes written by the CatalogBlobWriter in order to obtain the
stream position, needed to get offset to reference catalog items.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-31 09:11:51 +01:00
Dietmar Maurer 62ee2eb405 avoid some clippy warnings 2019-10-26 11:42:05 +02:00
Dietmar Maurer 834a2f95a0 avoid some clippy warnings 2019-10-25 18:44:51 +02:00
Dietmar Maurer f58f426e86 src/pxar/binary_search_tree.rs: allow(clippy::many_single_char_names) 2019-10-25 18:22:19 +02:00
Dietmar Maurer 11377a47bb avoid some clippy warnings 2019-10-25 18:04:37 +02:00
Dietmar Maurer 9fe2f639b8 allow(clippy::cast_ptr_alignment) 2019-10-25 18:03:55 +02:00
Dietmar Maurer cd69d36b45 src/backup/dynamic_index.rs: use write_all() 2019-10-25 17:22:41 +02:00
Christian Ebner 8abd12f1f3 pxar: decoder: cleanup un-needed Decoder::get_dir()
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 08:02:23 +02:00
Christian Ebner 9c25c0330a pxar: fuse: optimize fuse readdir callback
The expensive call to Decoder::read_directory_entry() can be omitted as
Decoder::attributes() returns all the information the fuse response needs.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 08:02:03 +02:00
Dietmar Maurer ffe171c5db src/pxar/encoder.rs: avoid duplicate check for MAX_DIRECTORY_ENTRIES
Simply by reordering code.
2019-10-23 07:55:56 +02:00
Christian Ebner 62d123e50a pxar: add functionality to pass exclude MatchPatterns on create
This exposes the option to pass a list of exclude MatchPattern via the
'--exclude' option.
The list is encoded as file '.pxarexclude-cli' in the archives root directory.
If such a file is present in the filesystem, it is skipped and not included in
the archive in order to avoid conflicting information.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 07:31:49 +02:00
Christian Ebner ba8165c607 pxar: encoder: add encode_pxar_exclude_cli() function.
This provides the functionality needed to encode MatchPatterns passed on the cli
in the root directory.

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 07:30:30 +02:00
Dietmar Maurer 920243b12d pxar: simplify MatchPattern::to_bytes 2019-10-23 07:29:02 +02:00
Christian Ebner e50a90e010 pxar: Implement to_bytes() for MatchPattern in order to write them to file.
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-23 07:21:11 +02:00
Dietmar Maurer a8f10f849e src/bin/proxmox-backup-client.rs: fix parameters for get_encryption_key_password 2019-10-17 10:28:21 +02:00
Christian Ebner 40c9ff9d81 src/pxar/encoder.rs: include .pxarexclude files into the catalog
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-17 10:25:20 +02:00
Christian Ebner 20feec9fac pxar: fuse: include nsec information in stat
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-10-17 10:24:13 +02:00
Dietmar Maurer 649c5ee7b1 load_and_decrtypt_key: allow function closures to readf passphrase 2019-10-16 13:23:52 +02:00
Dietmar Maurer 60ffa9649c src/backup/manifest.rs: fix property names (use '-' instead of '_') 2019-10-14 12:28:27 +02:00
Dietmar Maurer 4c9f753c56 src/backup/data_blob.rs: define create_zero_chunk as member function
and rename it to build_zero_chunk()
2019-10-14 10:58:26 +02:00
Dietmar Maurer 2745151d93 src/backup/data_blob.rs: add create_zero_chunk() helper 2019-10-14 10:44:46 +02:00
Dietmar Maurer f06b820ac0 src/backup/manifest.rs: add verify_file 2019-10-13 10:09:12 +02:00