Commit Graph

5286 Commits

Author SHA1 Message Date
Oguz Bektas dafe3197ab scanrefs: match all instances of 'onlineHelp' in js files
previously it looked for the first instance. this behavior
became an issue while trying to add multiple onlineHelp buttons

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
2020-10-05 17:00:49 +02:00
Dominik Csapak 90d7425afe ui: refactor render_icon code
we will reuse this later

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-05 16:48:01 +02:00
Dominik Csapak 2d81f7b0c0 task archive rotation: better handle non-existing archive
if the archive file does not exist yet, we cannot rotate it, but it's not
actually an error, so just return Ok(false) to indicate no rotation took
place

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-05 10:04:16 +02:00
Dylan Whyte 04e24b14f0 Restructure docs (more first level headings)
This removes the "Backup Management" first level heading in the docs,
and either uses the sub headings contained within it as first level
headings, or groups previous sections logically under new headings.

The administration-guide.rst file is also removed. Its contents are
instead separated into various files, that relate to their respective
first level heading.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-05 09:23:45 +02:00
Dylan Whyte a2bf852818 Add section "GUI"
Section provides a brief overview of the web interface

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-10-05 09:23:45 +02:00
Thomas Lamprecht 0ac612476a REST server: avoid hard coding world readable API endpoints
while we probably do not add much more to them, it still looks ugly.

If this was made so that adding a World readable API call is "hard"
and not done by accident, it rather should be done as a test on build
time. But, IMO, the API permission schema definitions are easy to
review, and not often changed/added - so any wrong World readable API
call will normally still caught.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-05 08:29:43 +02:00
Thomas Lamprecht 0c6b83d656 bump version to 0.9.0-2
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-02 15:37:59 +02:00
Thomas Lamprecht 4e6dc58727 rest server: cleanup use statements
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-02 13:04:08 +02:00
Dominik Csapak 66bbd4200c ui: RemoteEdit: only send delete on update
the create api call does not understand the 'delete' paramter, so
send it only on update

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-02 10:48:00 +02:00
Thomas Lamprecht 326c835e60 update d/control
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 16:57:33 +02:00
Thomas Lamprecht 1a48cbf164 bump version to 0.9.0-1
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 16:19:49 +02:00
Thomas Lamprecht 3480777d89 d/control: bump versioned dependency of proxmox-widget-toolkit
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 15:30:08 +02:00
Dietmar Maurer a71bc08ff4 src/tools/parallel_handler.rs: remove lifetime hacks, require 'static
In theory, one can do std::mem::forget, and ignore the drop handler. With
the lifetime hack, this could result in a crash.

So we simply require 'static lifetime now (futures also needs that).
2020-10-01 14:52:48 +02:00
Thomas Lamprecht df766e668f d/control: add pve-eslint to build dependencies
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 14:46:30 +02:00
Dietmar Maurer 0a8f3ae0b3 src/tools/parallel_handler.rs: cleanup check_abort code 2020-10-01 14:37:29 +02:00
Stefan Reiter da6e67b321 rrd: fix integer underflow
Causes a panic if last_update is smaller than RRD_DATA_ENTRIES*reso,
which (I believe) can happen when inserting the first value for a DB.

Clamp the value to 0 in that case.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Stefan Reiter dec00364b3 ParallelHandler: check for errors during thread join
Fix a potential bug where errors that happen after the SendHandle has
been dropped while doing the thread join might have been ignored.
Requires internal check_abort to be moved out of 'impl SendHandle' since
we only have the Mutex left, not the SendHandle.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-01 14:30:32 +02:00
Thomas Lamprecht 5637087cc9 www: do incremental lint for development, full for build
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 13:14:03 +02:00
Thomas Lamprecht 5ad4bdc482 eslint fixes
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 13:03:14 +02:00
Thomas Lamprecht 823867f5b7 datastore: gc: avoid unsafe call into libc, use epoch_i64 helper
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 12:38:38 +02:00
Thomas Lamprecht c6772c92b8 datastore: gc: comment exclusive process lock
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 12:38:04 +02:00
Thomas Lamprecht 79f6a79cfc assume correct backup, avoid verifying chunk existance
This can slow things down by a lot on setups with (relatively) high
seek time, in the order of doubling the backup times if cache isn't
populated with the last backups chunk inode info.

Effectively there's nothing known this protects us from in the
codebase. The only thing which was theorized about was the case
where a really long running backup job (over 24 hours) is still
running and writing new chunks, not indexed yet anywhere, then an
update (or manual action) triggers a reload of the proxy. There was
some theory that then a GC in the new daemon would not know about the
oldest writer in the old one, and thus use a less strict atime limit
for chunk sweeping - opening up a window for deleting chunks from the
long running backup.
But, this simply cannot happen as we have a per datastore process
wide flock, which is acquired shared by backup jobs and exclusive by
GC. In the same process GC and backup can both get it, as it has a
process locking granularity. If there's an old daemon with a writer,
that also has the lock open shared, and so no GC in the new process
can get exclusive access to it.

So, with that confirmed we have no need for a "half-assed"
verification in the backup finish step. Rather, we plan to add an
opt-in "full verify each backup on finish" option (see #2988)

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-10-01 12:06:59 +02:00
Dietmar Maurer 4c7f100d22 src/api2/reader.rs: fix speedtest description 2020-10-01 11:16:15 +02:00
Dietmar Maurer 9070d11f4c src/api2/backup.rs: use block_in_place for remove_backup 2020-10-01 11:11:14 +02:00
Dietmar Maurer 124b93f31c upload_chunk: use block_in_place 2020-10-01 11:00:23 +02:00
Dominik Csapak 0f22f53b36 ui: RemoteEdit: remove port field and parse it from host field
use our hostport regexes to parse out a potential port from the host field
and send it individually

this makes for a simpler and cleaner ui

this additionally checks the field for valid input before sending it to
the backend

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-01 10:12:04 +02:00
Dominik Csapak 3784dbf029 ui: RemoteView: improve host columns
do not show the default (8007) port
and only add brackets [] to ipv6 addresses if there is a port

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-01 10:11:31 +02:00
Dominik Csapak 4c95d58c41 api2/types: fix DNS_NAME Regexes
We forgot to put braces around the DNS_NAME regex, and in
DNS_NAME_OR_IP_REGEX

this is wrong because the regex

 ^foo|bar$

matches 'foo' at the beginning and 'bar' at the end, so either

 foobaz
 bazbar

would match. only

 ^(foo|bar)$

 matches only 'foo' and 'bar'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-10-01 06:09:34 +02:00
Dominik Csapak 38d4675921 fix ipv6 handling for remotes/sync jobs
* add square brackets to ipv6 adresses in BackupRepository if they not
already have some (we save them without in the remote config)

* in get_pull_parameters, we now create a BackupRepository first and use
  those values (which does the [] mapping), this also has the advantage
  that we have one place less were we hardcode 8007 as port

* in the ui, add square brackets for ipv6 adresses for remotes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 13:40:03 +02:00
Dietmar Maurer 7b8aa893fa src/client/pull.rs: log progress 2020-09-30 13:35:09 +02:00
Dietmar Maurer fb2678f96e www/index.hbs: add nodename to title 2020-09-30 12:10:04 +02:00
Dominik Csapak 486ed27299 ui: improve running task overlay
by setting a maxHeight+scrollable
(i used 500px to be still visible on our 'min screen size' 1280x720)

and by disabling emptyText deferral, which now shows the text instantly

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 11:07:08 +02:00
Dominik Csapak df4827f2c0 tasks: improve behaviour on upgrade
when upgrading from a version where we stored all tasks in the 'active' file,
we did not completly account for finished tasks still there

we should update the file when encountering any finished task in
'active' as well as filter them out on the api call (if they get through)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 11:05:50 +02:00
Dietmar Maurer ef1b436350 paperkey: add html output 2020-09-30 10:49:20 +02:00
Dietmar Maurer b19b4bfcb0 examples: fix HttpClient::new usage 2020-09-30 10:49:20 +02:00
Dietmar Maurer e64b9f9204 src/tools.rs: make command_output return Vec<u8>
And add a new helper to return output as string.
2020-09-30 10:49:20 +02:00
Dominik Csapak 9c33683c25 ui: add port support for remotes
by adding a field to RemoteEdit and showing it in the grid

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 10:49:20 +02:00
Dominik Csapak ba20987ae7 client/remote: add support to specify port number
this adds the ability to add port numbers in the backup repo spec
as well as remotes, so that user that are behind a
NAT/Firewall/Reverse proxy can still use it

also adds some explanation and examples to the docs to make it clearer
for h2 client i left the localhost:8007 part, since it is not
configurable where we bind to

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 10:49:20 +02:00
Thomas Lamprecht 729d41fe6a api: disks/zfs: check template exsits before enabling zfs-import service
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-30 09:34:21 +02:00
Stoiko Ivanov 905147a5ee api2/node/disks/zfs: instantiate import service
When creating a new zpool for a datastore, also instantiate an
import-unit for it. This helps in cases where '/etc/zfs/zool.cache'
get corrupted and thus the pool is not imported upon boot.

This patch needs the corresponding addition of 'zfs-import@.service' in
the zfsonlinux repository.

Suggested-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
Signed-off-by: Stoiko Ivanov <s.ivanov@proxmox.com>
2020-09-30 08:43:38 +02:00
Dominik Csapak 0c41e0d06b ui: add task description for logrotation
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:17:07 +02:00
Dominik Csapak b37b59b726 ui: RemoteEdit: make comment and fingerprint deletable
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:16:53 +02:00
Dominik Csapak 60b9b48e71 require square brackets for ipv6 addresses
we need this, because we append the port to this to get a target url
e.g. we print

format!("https://{}:8007/", address)

if address is now an ipv6 (e.g. fe80::1) it would become

https://fe80::1:8007/ which is a valid ipv6 on its own

by using square brackets we get:

https://[fe80::1]:8007/ which now connects to the correct ip/port

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:16:27 +02:00
Dominik Csapak abf8b5d475 docs: fix wrong user in repository explanation
we use 'root@pam' by default, not 'root'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:14:36 +02:00
Dominik Csapak 7eebe1483e server/worker_task: fix panic on slice range when index is empty
since len() and MAX_INDEX_TASKS are both usize, they underflow
instead of getting negative values

instead check the sizes and set them accordingly

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-30 06:11:06 +02:00
Dominik Csapak 9a76091785 proxmox-backup-proxy: add task archive rotation
this starts a task once a day at "00:00" that rotates the task log
archive if it is bigger than 500k

if we want, we can make the schedule/size limit/etc. configurable,
but for now it's ok to set fixed values for that

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:41:18 +02:00
Dominik Csapak c386b06fc6 server/worker_task: remove unecessary read_task_list
since there are no users of this anymore and we now have a nicer
TaskListInfoIterator to use, we can drop this function

this also means that 'update_active_workers' does not need to return
a list anymore since we never used that result besides in
read_task_list

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:50 +02:00
Dominik Csapak 6bcfc5c1a4 api2/status: use the TaskListInfoIterator here
this means that limiting with epoch now works correctly
also change the api type to i64, since that is what the starttime is
saved as

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:24 +02:00
Dominik Csapak 768e10d0b3 api2/node/tasks: use TaskListInfoIterator instead of read_task_list
this makes the filtering/limiting much nicer and readable

since we now have potentially an 'infinite' amount of tasks we iterate over,
and cannot now beforehand how many there are, we return the total count
as always 1 higher then requested iff we are not at the end (this is
the case when the amount of entries is smaller than the requested limit)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:40:02 +02:00
Dominik Csapak e7244387c7 server/worker_task: add TaskListInfoIterator
this is an iterator that reads/parses/updates the task list as
necessary and returns the tasks in descending order (newest first)

it does this by using our logrotate iterator and using a vecdeque

we can use this to iterate over all tasks, even if they are in the
archive and even if the archive is logrotated but only read
as much as we need

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:39:16 +02:00