Commit Graph

3197 Commits

Author SHA1 Message Date
Dietmar Maurer
998db63933 src/client/pull.rs: decode, verify and write in a separate threads
To maximize throughput.
2020-09-24 13:12:04 +02:00
Dietmar Maurer
c0fa14d94a src/backup/data_blob.rs: add is_encrypted helper 2020-09-24 13:00:16 +02:00
Wolfgang Bumiller
6fd129844d remove DummyCatalogWriter
we're using an `Option` instead now

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-09-24 09:13:54 +02:00
Dietmar Maurer
baae780c99 benchmark: use compressable data to get more realistic result
And add a benchmatrk to test chunk verify speed (decompress+sha256).
2020-09-24 08:58:13 +02:00
Dietmar Maurer
09a1da25ed src/backup/data_blob.rs: improve decompress speed 2020-09-24 08:52:35 +02:00
Oguz Bektas
298c6aaef6 docs: add onlineHelp to some panels
name sections according to the title or content and add
the respective onlineHelp to the following panels:
- datastore
- user management
- ACL
- backup remote

Signed-off-by: Oguz Bektas <o.bektas@proxmox.com>
Reviewed-By: Dominik Csapak <d.csapak@proxmox.com>
Tested-By: Dominik Csapak <d.csapak@proxmox.com>
2020-09-22 19:48:32 +02:00
Dietmar Maurer
a329324139 bump version to 0.8.19-1 2020-09-22 13:30:52 +02:00
Dietmar Maurer
a83e2ffeab src/api2/reader.rs: use std::fs::read instead of tokio::fs::read
Because it is about 10%& faster this way.
2020-09-22 13:27:23 +02:00
Dietmar Maurer
5d7449a121 bump version to 0.8.18-1 2020-09-22 12:39:47 +02:00
Dietmar Maurer
ebbe4958c6 src/client/pull.rs: avoid duplicate downloads using in memory HashSet 2020-09-22 12:34:06 +02:00
Dietmar Maurer
73b2cc4977 src/client/pull.rs: allow up to 20 concurrent download streams 2020-09-22 11:39:31 +02:00
Dietmar Maurer
7ecfde8150 remote_chunk_reader.rs: use Arc for cache_hint to make clone faster 2020-09-22 11:39:31 +02:00
Thomas Lamprecht
796480a38b docs: add version and date to HTML index
Similar to the PDF output or the Proxmox VE docs.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-22 09:00:12 +02:00
Dietmar Maurer
4ae6aede60 bump version to 0.8.17-1 2020-09-21 14:09:20 +02:00
Dietmar Maurer
e0085e6612 src/client/pull.rs: remove temporary manifest 2020-09-21 14:03:01 +02:00
Dietmar Maurer
194da6f867 src/client/pull.rs: open temporary manifest with truncate(true)
To delete any data if the file already exists.
2020-09-21 13:53:35 +02:00
Dietmar Maurer
3fade35260 bump proxmox version to 0.4.1 2020-09-21 13:51:33 +02:00
Dominik Csapak
5e39918fe1 fix #3017: check array boundaries before using
else we panic here

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-21 09:22:06 +02:00
Dietmar Maurer
f4dc47a805 debian/control: update 2020-09-19 16:22:56 +02:00
Dietmar Maurer
12c65bacf1 src/backup/chunk_store.rs: disable debug output 2020-09-19 15:26:21 +02:00
Dietmar Maurer
ba37f3562d src/backup/datastore.rs - open_with_path: use Path instead of str 2020-09-19 10:01:57 +02:00
Dietmar Maurer
fce4659388 src/backup/datastore.rs: new method open_with_path
To make testing easier.
2020-09-19 09:55:21 +02:00
Dietmar Maurer
0a15870a82 depend on proxmox 0.4.0 2020-09-19 06:40:44 +02:00
Dominik Csapak
9866de5e3d datastore/prune schedules: use JobState for tracking of schedules
like the sync jobs, so that if an admin configures a schedule it
really starts the next time that time is reached not immediately

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-19 06:24:37 +02:00
Dylan Whyte
9d3f183ba9 Admin Guide: Add some more detailed info throughout
- Mention config files for: datastores, users, acl,
  remotes, syncjobs
- Expand a little bit on SMART and smartmontools package
- Explain acl config
- Include line in network stating why a bond would be set up
- Note the use of ifupdown2 for network config, and the potential
  need to install it on other systems
- Add note to PVE integration, specifying where to refer to for VM and
  CT backups

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-09-18 15:51:21 +02:00
Dylan Whyte
fe233f3b3d Small formatting fix up
- Fix permission image.
- Change alt text for ZFS
- Change note block to match the others

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
2020-09-18 15:50:36 +02:00
Fabian Grünbichler
be3bd0f90b fix #3015: allow user self-service
listing, updating or deleting a user is now possible for the user
itself, in addition to higher-privileged users that have appropriate
privileges on '/access/users'.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 15:45:11 +02:00
Fabian Grünbichler
3c053adbb5 role api: fix description
wrongly copy-pasted at some point

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 14:55:00 +02:00
Hannes Laimer
c040ec22f7 add verification scheduling to proxmox-backup-proxy
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:14:05 +02:00
Hannes Laimer
43f627ba92 ui: add verify-schedule field to edit datastore form
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:13:09 +02:00
Hannes Laimer
2b67de2e3f api2: make verify_schedule deletable
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:12:29 +02:00
Hannes Laimer
477859662a api2: add optional verify-schdule field to create/update datastore endpoint
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:12:16 +02:00
Hannes Laimer
ccd7241e2f add verify_schedule field to DataStoreConfig
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:11:55 +02:00
Hannes Laimer
f37ef25bdd api2: add VERIFY_SCHEDULE_SCHEMA
Signed-off-by: Hannes Laimer <h.laimer@proxmox.com>
2020-09-18 12:11:39 +02:00
Fabian Grünbichler
b93bbab454 fix #3014: allow DataStoreAdmins to list DS config
filtered by those they are privileged enough to read individually. this
allows such users to configure prune/GC schedules via the GUI (the API
already allowed it previously).

permission-wise, a user with this privilege can already:
- list all stores they have access to (returns just name/comment)
- read the config of each store they have access to individually
(returns full config of that datastore + digest of whole config)

but combines them to
- read configs of all datastores they have access to (returns full
config of those datastores + digest of whole config)

user that have AUDIT on just /datastore without propagate can now no
longer read all configurations (but this could be added it back, it just
seems to make little sense to me).

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-18 12:09:13 +02:00
Dietmar Maurer
9cebc837d5 depend on pxar 0.6.1 2020-09-18 12:02:17 +02:00
Dietmar Maurer
1bc1d81a00 move compute_file_csum to src/tools.rs 2020-09-17 10:27:04 +02:00
Dietmar Maurer
dda72456d7 depend on proxmox 0.3.9 2020-09-17 08:49:50 +02:00
Dominik Csapak
8f2f3dd710 fix #2942: implement lacp bond mode and bond_xmit_hash_policy
this was not yet implemented, should be compatible with pve and the gui

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 08:36:25 +02:00
Dominik Csapak
85959a99ea api2/network: add bond-primary parameter
needed for 'active-backup' bond mode

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 08:36:14 +02:00
Dominik Csapak
36700a0a87 api2/pull: make pull worker abortable
by selecting between the pull_future and the abort future

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 06:11:33 +02:00
Dominik Csapak
dd4b42bac1 fix #2870: renew tickets in HttpClient
by packing the auth into a RwLock and starting a background
future that renews the ticket every 15 minutes

we still use the BroadcastFuture for the first ticket and only
if that is finished we start the scheduled future

we have to store an abort handle for the renewal future and abort it when
the http client is dropped, so we do not request new tickets forever

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-17 06:09:54 +02:00
Fabian Grünbichler
9626c28619 always allow retrieving (censored) subscription info
like we do for PVE. this is visible on the dashboard, and caused 403 on
each update which bothers me when looking at the dev console.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-17 06:03:25 +02:00
Fabian Grünbichler
463c03462a fix #2957: allow Sys.Audit access to node RRD
this is the same privilege needed to query the node status.

Signed-off-by: Fabian Grünbichler <f.gruenbichler@proxmox.com>
2020-09-17 06:03:25 +02:00
Thomas Lamprecht
a086427a7d docs: fix epilogs fixme comment
restructured text comment syntax, i.e., everything it cannot parse is
a comment, is a real PITA!

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-16 16:36:54 +02:00
Dietmar Maurer
4d431383d3 src/backup/data_blob.rs: expose verify_crc again 2020-09-16 10:43:42 +02:00
Stefan Reiter
d10332a15d SnapshotVerifyState: use enum for state
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 13:06:04 +02:00
Stefan Reiter
43772efc6e backup: check all referenced chunks actually exist
A client can omit uploading chunks in the "known_chunks" list, those
then also won't be written on the server side. Check all those chunks
mentioned in the index but not uploaded for existance and report an
error if they don't exist instead of marking a potentially broken backup
as "successful".

This is only important if the base snapshot references corrupted chunks,
but has not been negatively verified. Also, it is important to only
verify this at the end, *after* all index writers are closed, since only
then can it be guaranteed that no GC will sweep referenced chunks away.

If a chunk is found missing, also mark the previous backup with a
verification failure, since we know the missing chunk has to referenced
in it (only way it could have been inserted to known_chunks with
checked=false). This has the benefit of automatically doing a
full-upload backup if the user attempts to retry after seeing the new
error, instead of requiring a manual verify or forget.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 10:00:05 +02:00
Stefan Reiter
0af2da0437 backup: check verify state of previous backup before allowing reuse
Do not allow clients to reuse chunks from the previous backup if it has
a failed validation result. This would result in a new "successful"
backup that potentially references broken chunks.

If the previous backup has not been verified, assume it is fine and
continue on.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-09-15 09:59:29 +02:00
Dietmar Maurer
d09db6c2e9 rename BackupDir::new_with_group to BackupDir::with_group 2020-09-15 09:40:03 +02:00