Commit Graph

129 Commits

Author SHA1 Message Date
Dominik Csapak 93bb51fe7e config/jobstate: replace Job:load with create_state_file
it really is not necessary, since the only time we are interested in
loading the state from the file is when we list it, and there
we use JobState::load directly to avoid the lock

we still need to create the file on syncjob creation though, so
that we have the correct time for the schedule

to do this we add a new create_state_file that overwrites it on creation
of a syncjob

for safety, we subtract 30 seconds from the in-memory state in case
the statefile is missing

since we call create_state_file from  proxmox-backup-api,
we have to chown the lock file after creating to the backup user,
else the sync job scheduling cannot aquire the lock

also we remove the lock file on statefile removal

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 06:38:02 +02:00
Dominik Csapak 713b66b6ed cleanup: replace id from do_sync_job with info from job
we already have it inside the job itself

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-14 06:36:43 +02:00
Dominik Csapak a13573c24a syncjob: use do_sync_job also for scheduled sync jobs
and determine the last runtime with the jobstate

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-08-13 11:51:20 +02:00
Wolfgang Bumiller e7cb4dc50d introduce Username, Realm and Userid api types
and begin splitting up types.rs as it has grown quite large
already

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-10 12:05:01 +02:00
Stefan Reiter c9756b40d1 datastore: prevent deletion of snaps in use as "previous backup"
To prevent a race with a background GC operation, do not allow deletion
of backups who's index might currently be referenced as the "known chunk
list" for successive backups. Otherwise the GC could delete chunks it
thinks are no longer referenced, while at the same time telling the
client that it doesn't need to upload said chunks because they already
exist.

Additionally, prevent deletion of whole backup groups, if there are
snapshots contained that appear to be currently in-progress. This is
currently unlikely to trigger, as that function is only used for sync
jobs, but it's a useful safeguard either way.

Deleting a single snapshot has a 'force' parameter, which is necessary
to allow deleting incomplete snapshots on an aborted backup. Pruning
also sets force=true to avoid the check, since it calculates which
snapshots to keep on its own.

To avoid code duplication, the is_finished method is factored out.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:26:01 +02:00
Dominik Csapak 01ca99da2d server/rest: add console to index
register the console template and render it when the 'console' parameter
is given

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 12:06:38 +02:00
Dominik Csapak 2ab5acac5a server/config: add mechanism to update template
instead of exposing handlebars itself, offer a register_template and
a render_template ourselves.

render_template checks if the template file was modified since
the last render and reloads it when necessary

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-23 11:55:00 +02:00
Dietmar Maurer ac7513e368 src/tools.rs: add setup_safe_path_env() 2020-06-15 10:38:30 +02:00
Dominik Csapak e693818afc refactor time functions to tools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:10 +02:00
Dominik Csapak 33070956af let disk_usage return StorageStatus and use it for datastores/nodes
disk_usage returned the same values as defined in StorageStatus,
so simply use that

with that we can replace the logic of the datastore status with that
function and also use it for root disk usage of the nodes

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:05:39 +02:00
Thomas Lamprecht a720894ff0 rrd: fix off-by-one in save interval calculation
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-06-04 10:30:47 +02:00
Dietmar Maurer 5c264c8d80 src/tools/disks.rs: add/use get_partition_type_info 2020-06-04 07:48:22 +02:00
Dietmar Maurer b2387eaa45 avoid compiler warnings 2020-05-30 14:05:33 +02:00
Dominik Csapak c67b1fa72f syncjob: change worker type for sync jobs
'sync' is used for manually pulling a remote datastore
changing it for a scheduled sync to 'syncjob' so that we can
differentiate between both types of syncs

this also adds a seperate task description for it

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-29 11:28:04 +02:00
Dietmar Maurer 934f5bb8ac src/bin/proxmox-backup-proxy.rs: cleanup, move code to src/tools/disks.rs
And simplify find_mounted_device by using stat.st_dev
2020-05-29 11:13:36 +02:00
Dietmar Maurer 013fa7bbcb rrd: reduce io by saving data only once a minute 2020-05-29 09:16:13 +02:00
Dietmar Maurer a8d7033cb2 src/bin/proxmox-backup-proxy.rs: add test if last prune job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer 04ad7bc436 src/bin/proxmox-backup-proxy.rs: test if last sync job is still running 2020-05-29 08:06:48 +02:00
Dietmar Maurer dc58194ebe src/bin/proxmox-backup-proxy.rs: use correct id to lookup sync jobs 2020-05-29 07:50:59 +02:00
Dietmar Maurer c94e1f655e rrd stats: improve io delay stats 2020-05-28 19:12:13 +02:00
Dietmar Maurer 91e5bb49f5 src/bin/proxmox-backup-proxy.rs: simplify code
and gather all stats for the root disk
2020-05-28 12:30:54 +02:00
Dietmar Maurer 25c550bc28 src/bin/proxmox-backup-proxy.rs: gather zpool io stats 2020-05-28 10:09:13 +02:00
Dominik Csapak f5056656b2 use the sync id for the scheduled sync worker task
this way, multiple sync jobs with the same local store, can get scheduled

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-05-28 06:26:03 +02:00
Dietmar Maurer 8c03041a2c src/bin/proxmox-backup-proxy.rs: gather block device stats on datastore 2020-05-26 11:20:59 +02:00
Dietmar Maurer 309ef20d6d src/bin/proxmox-backup-proxy.rs: simplify code 2020-05-25 16:20:32 +02:00
Dietmar Maurer d0833a70f7 src/bin/proxmox-backup-proxy.rs: gather datastore usage stats 2020-05-25 16:20:32 +02:00
Dietmar Maurer 2c66a590c0 src/bin/proxmox-backup-proxy.rs: gather iowait stats 2020-05-25 11:54:15 +02:00
Dietmar Maurer 485841da2c src/bin/proxmox-backup-proxy.rs: gather loadavg stats 2020-05-25 11:40:20 +02:00
Dietmar Maurer dd15c0aa3b src/bin/proxmox-backup-proxy.rs: gather root disk stats 2020-05-25 11:10:07 +02:00
Dietmar Maurer c25c9d8dd1 src/bin/proxmox-backup-proxy.rs: gather swap usage stats 2020-05-25 10:25:58 +02:00
Dietmar Maurer 4f9513996c src/bin/proxmox-backup-proxy.rs: use block_in_place for rrd update 2020-05-25 08:30:59 +02:00
Dietmar Maurer 8f0cec2642 src/bin/proxmox-backup-proxy.rs: gather netin/netout stats 2020-05-24 19:02:35 +02:00
Dietmar Maurer 4fb05fde17 src/rrd/rrd.rs: restructure whole code 2020-05-24 16:51:28 +02:00
Dietmar Maurer a4a3f7ca5e rrd: pack multiple rrd values into th estat list 2020-05-23 14:03:44 +02:00
Dietmar Maurer eaeda365e0 start gathering stats using new rrd module 2020-05-23 10:43:08 +02:00
Dietmar Maurer 07ad6470ca src/client/pull.rs: split out pull related code 2020-05-22 08:04:20 +02:00
Dietmar Maurer a6160cdfeb src/bin/proxmox-backup-proxy.rs: schedule sync jobs 2020-05-22 07:50:59 +02:00
Dietmar Maurer 25829a879b src/bin/proxmox-backup-proxy.rs: schedule prune jobs 2020-05-20 13:00:53 +02:00
Dietmar Maurer 8545480a31 src/bin/proxmox-backup-proxy.rs: add simple task scheduler for garbage collection 2020-05-20 08:59:45 +02:00
Dominik Csapak f9e3b1104e change index to templates using handlebars
using a handlebars instance in ApiConfig, to cache the templates
as long as possible, this is currently ok, as the index template
can only change when the whole package changes

if we split this in the future, we have to trigger a reload of
the daemon on gui package upgrade (so that the template gets reloaded)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-04-29 17:05:53 +02:00
Wolfgang Bumiller f7d4e4b506 switch from failure to anyhow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Wolfgang Bumiller 9ea4bce444 bump proxmox crate to 0.1.7
The -sys, -tools and -api crate have now been merged into
the proxmx crate directly. Only macro crates are separate
(but still reexported by the proxmox crate in their
designated locations).

When we need to depend on "parts" of the crate later on
we'll just have to use features.

The reason is mostly that these modules had
inter-dependencies which really make them not independent
enough to be their own crates.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-21 13:48:37 +01:00
Wolfgang Bumiller d973aa827c introduce new runtime tokio helpers
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-01-20 13:12:40 +01:00
Dietmar Maurer 94f085fab5 src/bin/proxmox-backup-proxy.rs: remove unused use clause 2019-12-30 11:50:24 +01:00
Dietmar Maurer 22be470d90 src/bin/proxmox-backup-api.rs: create cert here 2019-12-27 13:44:21 +01:00
Wolfgang Bumiller a546a8a072 proxy/api: await running workers before shutdown
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-27 12:50:33 +01:00
Dietmar Maurer f8fd5095d8 src/bin/proxmox-backup-proxy.rs: create self signed cert at startup. 2019-12-27 11:21:32 +01:00
Thomas Lamprecht 9c01e73c2f ui: add "Documentation" button to main view
Similar to PVE and PMG, for quick access when one has the basic
webinterface open anyway. Should move to the "proxmoxHelpButton" once
we have an onlineHelp mapping to the docs.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 11:22:34 +01:00
Thomas Lamprecht 2d694f8f1f add CSS file for PBS ExtJS6 basic ui
some fitting rules copied over from PVE's ext6-pve.css file.
simply place it in the css subfolder where the proxmox-backup-gui.js
file is hosted and add a "css/" alias for that directory, the
formatter gets use the right content type with that.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2019-12-17 11:20:32 +01:00
Wolfgang Bumiller db0cb9ce0b update a chunk of stuff to the hyper release
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:24:41 +01:00
Dietmar Maurer e57e1cd82d move src/api_schema/config.rs -> src/server/config.rs 2019-11-22 09:23:03 +01:00
Wolfgang Bumiller a2479cfa1a api/compat: drop more compat imports from api_schema.rs
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-21 14:36:28 +01:00
Dietmar Maurer 255f378a1b use const api definitions 2019-11-21 13:32:09 +01:00
Dietmar Maurer 4a7de56e2f avoid injecting ENV vars from Makefile
So that we can run "cargo build" without setting vars manually.
2019-09-09 10:51:08 +02:00
Wolfgang Bumiller 083ff3fd5d update to tokio 0.2.0-alpha.4
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Wolfgang Bumiller fda5797b8a src/bin/proxmox-backup-proxy.rs: switch to async
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00
Dietmar Maurer e18a6c9ee5 update to nix 0.14, use code from proxmox:tools 2019-08-03 13:05:38 +02:00
Dietmar Maurer 6d1f61b208 use hyper/tokio-openssl instead of hyper/tokio-tls
This exposes the complete SSL setup. And download is much faster
now (600MB/s instead of 130MB/s)!
2019-07-02 13:36:28 +02:00
Dietmar Maurer 74d0a6bc23 src/bin/proxmox-backup-proxy.rs: set socket options for max. performance 2019-07-02 10:48:58 +02:00
Christian Ebner 97eeea3b4a src/bin/proxmox-backup-proxy.rs: fix typo in pkcs12 error message
Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2019-06-05 06:29:02 +02:00
Wolfgang Bumiller e888155712 proxy: allow .key/.pem files instead of a pkcs12/.pfx file
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-26 10:47:33 +02:00
Wolfgang Bumiller d98c9a7a38 use service Type=notify
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-04-25 11:01:33 +00:00
Dietmar Maurer 5f550fd99f start hyper server using with_graceful_shutdown()
Without, hyper keeps some futures running, and the server does not
correctly shutdown.
2019-04-10 08:24:32 +02:00
Dietmar Maurer d607b8861b src/server/worker_task.rs: implement task control socket 2019-04-09 12:15:06 +02:00
Dietmar Maurer e3f41f2165 src/tools/daemon.rs: use new ServerState handler 2019-04-08 14:00:23 +02:00
Wolfgang Bumiller a690ecacf0 daemon: simplify daemon creation
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 12:12:54 +01:00
Wolfgang Bumiller e4311382d9 tools: daemon: rename some structs
Reloadable resources are now 'Reloadable' instead of
'ReexecContinue'.

The struct handling the reload is a 'Reloader', not a
'ReexecStore'.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-19 11:09:46 +01:00
Wolfgang Bumiller a2ca713701 add reload support to proxy
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller 1c042cdc6c proxy: listener error handling fixup
If the listening socket goes into some error state we'll get
std::io::Errors rather than higher level errors from the
native_tls::TlsAcceptor, those are usually fatal. (Ran into
this after performing a shutdown() on the file descriptor,
after which the future just endlessly loops in accept().)

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-03-18 12:29:18 +01:00
Wolfgang Bumiller fc7f03523c cleanup
Error::from is already a function taking 1 parameter,
there's no need to wrap it with `|e| Error::from(e)`.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-18 13:21:27 +01:00
Dietmar Maurer dc9a007b11 rename src/api to src/api_schema 2019-02-17 09:59:20 +01:00
Dietmar Maurer a8f268afbb fix configuration dir permission
Carefully set and check permissions ...
2019-02-16 12:19:13 +01:00
Dietmar Maurer 4223d9f800 bin/proxmox-backup-proxy.rs: improve error handling 2019-02-11 14:43:26 +01:00
Wolfgang Bumiller 9d02e5c38a proxy: use <configdir>/server.pfx for tls
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 15:19:32 +01:00
Wolfgang Bumiller 0d176f3681 proxy: use TLS via tokio-tls
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-04 14:56:07 +01:00
Wolfgang Bumiller 6285b251e7 don't hardcode /usr/share paths
Add a defines.mk file defining the standard default layout
for installation (PREFIX, DATAROOTDIR, LIBDIR etc.).

Add a $(JSDIR) variable and use env!("JSDIR") in place of
the hardcoded /usr/share/javascript/proxmox-backup path.

Have defines.mk include an *optional* local.mak to override
JSDIR to www/ during local development.

Add `make install` targets and remove debian/install.

Note that service files are now in $libdir/proxmox-backup/
as there's usually no point in starting them from the
command line, so /usr/sbin makes no sense.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
Wolfgang Bumiller 8e4df103dd remove useless command option from daemons
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-02-01 10:41:54 +01:00
Dietmar Maurer d01e2420f7 load auth keys on startup 2019-01-29 17:21:58 +01:00
Dietmar Maurer 02c7a75520 src/bin/proxmox-backup-proxy.rs: implement unpriviledged server
We want to run the public server as user www-data. Requests needing
root priviledges needs to be proxied to the proxmox-backup.service, which
now listens to 127.0.0.1:82.
2019-01-28 13:29:58 +01:00