Commit Graph

144 Commits

Author SHA1 Message Date
Stefan Reiter 5eb9dd0c8a add tools::http for generic HTTP GET and move HttpsConnector there
...to avoid having the tools:: module depend on api2.

The get_string function is based directly on hyper and thus relatively
simple, not supporting redirects for example.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-21 16:22:08 +02:00
Dominik Csapak 943479f5f6 tools: add AsyncChannelWriter
similar to StdChannelWriter, but implements AsyncWrite and sends
to a tokio::sync::mpsc::Sender

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:22 +02:00
Dominik Csapak fdce52aa99 tools: add zip module
This modules contains the 'ZipEncoder' struct, which wraps an async writer,
to create a ZIP archive on the fly

To create a ZIP file, have a target that implements AsyncWrite,
give it to ZipEncoder::new, add entries via 'add_entry' and
at the end, call 'finish'

for now, this does not implement compression (uses ZIPs STORE mode), and
does not support empty directories or hardlinks (or any other special
files)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-10-21 10:04:18 +02:00
Dietmar Maurer 97168f920e set reasonable TCP keepalive timeout 2020-10-19 14:01:17 +02:00
Stefan Reiter 45f9b32e0f client: implement map/unmap commands for .img backups
Allows mapping fixed-index .img files (usually from VM backups) to be
mapped to a local loopback device.

The architecture uses a FUSE-backed temp file mapped to a loopdev:

  /dev/loopX -> FUSE /run/pbs-loopdev/xxx -> backup client -> PBS

Since unmapping requires some cleanup (unmap the loopdev, stop FUSE,
remove the temp files) a special 'unmap' command is added, which uses a
PID file to send SIGINT to the backup-client instance started with
'map', which will handle the cleanup itself.

The polling with select! in mount.rs needs to be split in two, since we
have a chicken and egg problem between running FUSE and setting up the
loop device - so we need to do them concurrently, until the loopdev is
assigned, at which point we can report success and daemonize, and then
continue polling the FUSE loop future.

A loopdev module is added to tools containing all required functions for
mapping a loop device to the FUSE file, with the ioctls moved into an
inline module to avoid exposing them directly.

The client code is placed in the 'mount' module, which, while
admittedly a loose fit, allows reuse of the daemonizing code.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-10-06 09:08:14 +02:00
Dietmar Maurer e64b9f9204 src/tools.rs: make command_output return Vec<u8>
And add a new helper to return output as string.
2020-09-30 10:49:20 +02:00
Dominik Csapak 8074d2b0c3 tools: add logrotate module
this is a helper to rotate and iterate over log files
there is an iterator for open filehandles as well as
only the filename

also it has the possibilty to rotate them
for compression, zstd is used

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-09-29 08:33:21 +02:00
Dietmar Maurer 3c9b370255 src/tools/parallel_handler.rs: execute closure inside a thread pool 2020-09-25 12:58:20 +02:00
Dietmar Maurer 1bc1d81a00 move compute_file_csum to src/tools.rs 2020-09-17 10:27:04 +02:00
Dietmar Maurer 6a7be83efe avoid chrono dependency, depend on proxmox 0.3.8
- remove chrono dependency

- depend on proxmox 0.3.8

- remove epoch_now, epoch_now_u64 and epoch_now_f64

- remove tm_editor (moved to proxmox crate)

- use new helpers from proxmox 0.3.8
  * epoch_i64 and epoch_f64
  * parse_rfc3339
  * epoch_to_rfc3339_utc
  * strftime_local

- BackupDir changes:
  * store epoch and rfc3339 string instead of DateTime
  * backup_time_to_string now return a Result
  * remove unnecessary TryFrom<(BackupGroup, i64)> for BackupDir

- DynamicIndexHeader: change ctime to i64

- FixedIndexHeader: change ctime to i64
2020-09-15 07:12:57 +02:00
Thomas Lamprecht 09f12d1cf3 tools: rename extract_auth_cookie to extract_cookie
It does nothing specific to authentication..

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-09-08 08:54:33 +02:00
Wolfgang Bumiller c1d7d708d4 remove map_struct helper
if we ever need this it should be marked as unsafe!

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-17 11:53:02 +02:00
Wolfgang Bumiller 98c259b4c1 remove timer and lock functions, fix building with proxmox 0.3.2
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-08-04 11:33:02 +02:00
Dietmar Maurer a4acb6ef84 lock_file: return std::io::Error 2020-07-31 08:53:00 +02:00
Stefan Reiter 8cd29fb24a tools: add nonblocking mode to lock_file
Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-07-30 08:18:10 +02:00
Wolfgang Bumiller cdf1da2872 tools: add strip_ascii_whitespace for byte slices
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-07-15 11:55:48 +02:00
Dominik Csapak ec01eeadc6 refactor CertInfo to tools
we want to reuse some of the functionality elsewhere

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-07-10 11:08:53 +02:00
Wolfgang Bumiller c40440092d tools: add socketpair helper
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-22 10:49:56 +02:00
Dietmar Maurer 177a2de992 src/tools/nom.rs: move nom helpers into separate file 2020-06-18 12:41:13 +02:00
Dietmar Maurer ac7513e368 src/tools.rs: add setup_safe_path_env() 2020-06-15 10:38:30 +02:00
Wolfgang Bumiller 57e50fb906 use new Mmap helper for dynamic index
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-12 13:57:56 +02:00
Dominik Csapak e693818afc refactor time functions to tools
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-10 13:31:10 +02:00
Dietmar Maurer 144006fade src/tools.rs: add new run_command helper 2020-06-10 07:16:47 +02:00
Dominik Csapak ba97479848 add statistics module
provides some basic statistics functions (sum, mean, etc.)
and a function to return the parameters of the linear regression of
two variables

implemented using num_traits to be more flexible for the types

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2020-06-09 12:19:51 +02:00
Wolfgang Bumiller 386990ba09 tools: add file_get_non_comment_lines
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-06-08 13:56:58 +02:00
Thomas Lamprecht add5861e8d typo fixes all over the place
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2020-05-30 16:39:08 +02:00
Dietmar Maurer 143b654550 src/tools.rs - command_output: add parameter to check exit code 2020-05-27 07:25:39 +02:00
Dietmar Maurer 97fab7aa11 src/tools.rs: new helper to handle command_output (std::process::Output) 2020-05-27 06:53:25 +02:00
Dietmar Maurer f486e9e50e add systemd configuration file parser/writer, start job configuration 2020-05-12 13:07:49 +02:00
Wolfgang Bumiller 10effc9849 add tools/disks.rs (work in progress...)
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-05-05 10:14:42 +02:00
Wolfgang Bumiller 00ec8d1685 tools: pub use Fd from proxmox crate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-24 10:56:52 +02:00
Wolfgang Bumiller f7d4e4b506 switch from failure to anyhow
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2020-04-17 18:43:30 +02:00
Dietmar Maurer 4939255fb4 src/tools/format.rs: move output rendering code to this new file 2020-02-28 07:30:35 +01:00
Dietmar Maurer 501f4fa220 depend on proxmox 0.1.13, use new tty helpers from there 2020-01-31 08:16:00 +01:00
Christian Ebner 1685c2e341 tools: LRU cache to use for pxar FUSE implementation.
Implements a cache with least recently used cache replacement policy.
Internally the state is tracked by a HashMap (for fast access) and a doubly
linked list (for the access order).

Signed-off-by: Christian Ebner <c.ebner@proxmox.com>
2020-01-22 13:11:42 +01:00
Dietmar Maurer f1d99e3f6a src/bin/proxmox-backup-client.rs: use a std channel to write the catalog 2020-01-22 12:49:08 +01:00
Dietmar Maurer dcd033a53c src/tools/std_channel_writer.rs: new StdChannelWriter helper class 2020-01-22 11:33:57 +01:00
Dietmar Maurer 002a191abf src/api2/config: check digest 2020-01-15 12:27:05 +01:00
Dietmar Maurer f74a03da1f remove tools::getpwnam_ugid, impl. crate::backup::backup_user()
And use new nix::unistd::User struct.
2019-12-19 10:20:13 +01:00
Dietmar Maurer bce023c57c src/tools.rs: fix typo 2019-12-18 14:29:29 +01:00
Wolfgang Bumiller dc08934563 drop Cancellable future in favor of abortable
futures-0.3 has a futures::future::abortable() function
which does the exact same, returns an Abortable future with
an AbortHandle providing an abort() method.

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-17 10:52:07 +01:00
Wolfgang Bumiller 8a1028e0e7 update to url/percent-encoding to 2.1
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:57:01 +01:00
Wolfgang Bumiller db0cb9ce0b update a chunk of stuff to the hyper release
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-12-13 11:24:41 +01:00
Wolfgang Bumiller c5946faffd drop md5 crate dependency
we already depend on openssl which also provides md5

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-11-14 11:19:43 +01:00
Dietmar Maurer 62ee2eb405 avoid some clippy warnings 2019-10-26 11:42:05 +02:00
Dietmar Maurer e17d5d864d src/backup/manifest.rs: improve parser (try_from) 2019-10-13 08:39:49 +02:00
Wolfgang Bumiller 299a2f18b2 fixup variable naming
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:57:23 +02:00
Wolfgang Bumiller 5a54935bc9 clippy: fix pointer to temporary value
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-11 13:55:56 +02:00
Wolfgang Bumiller daef93f481 tools: add tokio::main() replacement
to deal with block_on() not allowing blocking()

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-03 11:16:30 +02:00
Wolfgang Bumiller 556eb70ea3 add generic either-or AsyncRead/Write type
The HttpsConnector will use this. Instead of implementing a
specialized MaybeTlsStream, this is simply a generic "either
this or that kind of Async Read/Write type".

Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2019-09-02 15:21:26 +02:00