Commit Graph

326 Commits

Author SHA1 Message Date
Dietmar Maurer
c95c1c83b0 move src/backup/snapshot_reader.rs to pbs_datastore crate 2021-09-27 09:58:20 +02:00
Dietmar Maurer
6d5d305d9d move src/backup/datastore.rs into pbs_datastore crate 2021-09-27 09:11:38 +02:00
Dietmar Maurer
619cd5cbcb cleanup WorkerTaskContext 2021-09-24 11:39:30 +02:00
Dietmar Maurer
1ec0d70d09 cleanup worker task logging
In order to avoid name conflicts with WorkerTaskContext

- renamed WorkerTask::log to WorkerTask::log_message

Note: Methods have different fuction signatures

Also renamed WorkerTask::warn to WorkerTask::log_warning for
consistency reasons.

Use the task_log!() and task_warn!() macros more often.
2021-09-24 10:34:11 +02:00
Dietmar Maurer
c8449217dc rename TaskState to WorkerTaskContext 2021-09-24 10:33:49 +02:00
Dietmar Maurer
b9700a9fe5 move worker_task.rs into proxmox-rest-server crate
Also moved pbs-datastore/src/task.rs to pbs-tools, which now depends on 'log'.
2021-09-24 10:28:17 +02:00
Dietmar Maurer
81867f0539 use UPID and systemd helpers from proxmox 0.13.4 2021-09-23 12:01:43 +02:00
Dietmar Maurer
84af82e8cf rename pbs-systemd to proxmox-systemd 2021-09-21 10:06:27 +02:00
Dietmar Maurer
ccc3896ff3 avoid type re-exports 2021-09-14 08:35:43 +02:00
Dietmar Maurer
cef5c72682 move src/tape/helpers/snapshot_reader.rs to src/backup/snapshot_reader.rs 2021-09-14 07:42:06 +02:00
Dietmar Maurer
048b43af24 split tape code into new pbs_tape workspace 2021-09-13 12:54:59 +02:00
Dietmar Maurer
67a5cf4714 fix regression tests 2021-09-10 12:45:06 +02:00
Dietmar Maurer
6227654ad8 more api type cleanups: avoid re-exports 2021-09-10 12:25:32 +02:00
Dietmar Maurer
5839c469c1 move tape_encryption_keys.rs to pbs_config workspace 2021-09-07 10:37:08 +02:00
Dietmar Maurer
bbdda58b35 moved key_derivation.rs from pbs_datastore to pbs-config/src/key_config.rs
Also moved pbs-datastore/src/crypt_config.rs to pbs-tools/src/crypt_config.rs.
We do not want to depend on pbs-api-types there, so I use [u8;32] instead of
Fingerprint.
2021-09-07 10:12:17 +02:00
Dominik Csapak
f46806414a tape/inventory: fix the tape tests as user, by mocking the lock
locking during the tests as regular user failed because we try to
chown to the backup user (which is not always possible).

Instead, do not lock at all, by implementing 'open_backup_lockfile' with
'create_mocked_lock'

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-07 08:42:04 +02:00
Dietmar Maurer
1ce8e905ea move drive config to pbs_config workspace
Also moved the tape type definitions to pbs_api_types.
2021-09-03 09:10:18 +02:00
Dietmar Maurer
2121174827 start new pbs-config workspace
moved src/config/domains.rs
2021-09-02 12:58:20 +02:00
Wolfgang Bumiller
4c1b776168 another import cleanup
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 14:46:01 +02:00
Wolfgang Bumiller
42dad3abd3 fixup imports in tests and examples
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-09-01 12:32:21 +02:00
Wolfgang Bumiller
dd2162f6bd more import cleanups
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-31 14:01:03 +02:00
Wolfgang Bumiller
b2065dc7d2 cleanup proxmox_backup::backup module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-08-30 14:14:04 +02:00
Dominik Csapak
9f58e312d7 tape/pool_writer: fix typo
s/wrinting/writing/

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-09 09:36:38 +02:00
Dietmar Maurer
bb14ed8cab cleanup: simplify next_expired_media() 2021-08-04 11:01:18 +02:00
Dietmar Maurer
1d3ae83359 tape: media_pool: implement guess_next_writable_media() 2021-08-04 09:59:12 +02:00
Dominik Csapak
4bb3876352 tape: lto: increase default timeout to 10 minutes
it seems that for some actions or in some circumstances, two minutes is
simply too short and the command aborts. Increase the default timeout to
10 minutes.

While it should give most commands enough time to finish, in case of a real
failure the procedure now takes up to 5 times longer, but IMHO thats an
OK tradeoff.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-08-03 09:19:13 +02:00
Dietmar Maurer
5ad40a3dd1 tape: media_catalog: add snapshot list cache for catalog
For some parts of the ui, we only need the snapshot list from the catalog,
and reading the whole catalog (can be multiple hundred MiB) is not
really necessary.

Instead, we write the list of snapshots into a seperate .index file. This file
is generated on demand and is much smaller and thus faster to read.
2021-07-29 13:34:31 +02:00
Dietmar Maurer
7116a2d9da tape: lock media_catalog file to to get a consistent view with load_catalog 2021-07-29 13:34:25 +02:00
Dietmar Maurer
0d5e990a62 cleanup: factor out tape catalog path helpers 2021-07-29 13:34:18 +02:00
Dominik Csapak
4f57f4ad84 tape: changer: add tests for decode_element_status_page
a test for a valid status_page, one with excess data
(in the descriptor as well in the page as a whole)
and a test with too little data

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:23:21 +02:00
Dominik Csapak
13e13d836f tape: changer: handle libraries that sends wrong amount of data
if the library sends more data than advertised, simply cut it off,
but if it sends less data, bail out (depending on how much data is
missing, trying to parse it could lead to a panic, so bail out early)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:22:48 +02:00
Dominik Csapak
3ab2432ab6 tape: changer: remove unnecesary inquiry parameter
this is never used, so remove it.
Ok, since they are only non public functions.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-28 12:17:07 +02:00
Dominik Csapak
a2ef36d445 tape: media_catalog: improve chunk_archive interface
instead of having a public start/end_chunk_archive and register_chunks,
simply expose a 'register_chunk_archive' method since we always have
a list of chunks anywhere we want to add them

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:18:13 +02:00
Dominik Csapak
42b010174e tape: changer: handle invalid descriptor data from library in status page
We get the descriptor length from the library and use that in
'chunks_exact', which panics on length 0. Catch that case
and bail out, since that makes no sense here anyway.

This could prevent a panic, in case a library sends wrong data.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-26 10:05:37 +02:00
Dominik Csapak
02631056b8 tape: changer: handle missing dvcid information
the dvcid information is not always available, so skip it if is missing

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 12:00:30 +02:00
Dominik Csapak
131d0f10c2 tape: changer: improve error message on wrong counts
Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 11:37:14 +02:00
Dominik Csapak
f9aa980c7d tape: changer: correctly consume data in decode_element_status_page
instead of 'blindly' trusting the changer to deliver the fields written
in the specification, trust the length data it returns in the header.

we slice the descriptor data into equal sized chunks of the correct
size, then we do not have care bout the len and empty checks anymore

this also makes the code to read the rest of the page obsolete,
since the next descriptor is on the correct offset anyway

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-22 11:37:14 +02:00
Dominik Csapak
c1feb447e8 tape: changer: sg_pt: fix typo
ok, since its a private struct

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 17:02:16 +02:00
Dominik Csapak
62a0e190cb tape: changer: sg_pt: add SCSI_VOLUME_TAG_LEN const
so that we do have less 'magic' constants without description

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-21 17:01:37 +02:00
Dietmar Maurer
7526d86419 use new atomic_open_or_create_file
Factor out open_backup_lockfile() method to acquire locks owned by
user backup with permission 0660.

Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
2021-07-20 18:54:23 +02:00
Dominik Csapak
6c053ffc89 tape: changer: sg_pt: make extra scsi request for dvcid
some libraries cannot handle a request with volume tags and DVCID set at
the same time.

So we make 2 separate requests and merge them, since we want to keep
the vendor/model/serial data.

to not overcomplicate the code, add another special type to ElementType

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-07-16 08:46:06 +02:00
Dietmar Maurer
9ee4c23833 tape: changer: sg_pt: always retry until timeout 2021-07-13 10:39:28 +02:00
Dominik Csapak
0ed40b19c7 tape: changer: sg_pt: query element types separately
Some changers do not like the DVCID bit when querying non-drives,
this includes when querying 'all' elements.

To circumvent this, we query each type by itself (like mtx does it),
and only add the DVCID bit for drives (Data Transfer Elements).

Reported by a user in the forum:
https://forum.proxmox.com/threads/ibm-3584-ts3500-support.92291/

and limit to 1000 elements per request.
(Because some changers limit that request with the options we set)

instead of checking if the data len was equal to the allocation_len
for getting more data, we count the returned elements and compare
that with the number we requested

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-07-12 18:19:26 +02:00
Dietmar Maurer
a0cd0f9cec change tape drive lock path
New kernel has stricter checks on tmpfs with stick-bit on directories, so some
commands (i.e. proxmox-tape changer status) fails when executed as root, because
permission checks fails when locking the drive.

This patch move the drive locks to /run/proxmox-backup/drive-lock.

Note: This is incompatible to old locking mechmanism, so users may not
run tape backups during update (or running backup can fail).
2021-07-12 17:26:49 +02:00
Wolfgang Bumiller
c23192d34e move chunk_store to pbs-datastore
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-07 14:37:47 +02:00
Wolfgang Bumiller
770a36e53a add pbs-tools subcrate
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-06 15:10:37 +02:00
Wolfgang Bumiller
af06decd1b split out pbs-buildcfg module
Signed-off-by: Wolfgang Bumiller <w.bumiller@proxmox.com>
2021-07-06 12:00:14 +02:00
Dominik Csapak
9ac8b73e07 tape/drive: fix logging when requesting media
we try to load the correct media in a loop until we find the correct tape.
when encountering an error or wrong tape, we want to log that (and send
an email if one is set) that requests the correct tape.

while trying to avoid printing the same errors more than once in a row,
we had at least one case (starting with an empty tape in the drive)
which would not print/send any tape request.

reworking that code to use a custom 'TapeRequest' enum, which contains
the state + error message, and a helper that prints and sends an email
when the state changes

this reduces the change check/log to a single variable, instead of 4
(tried, last_media_uuid, last_error, failure_reason)

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-06-30 10:25:48 +02:00
Dietmar Maurer
414be8b675 tape: fix LTO locate_file for HP drives
Add test code to the first locate_file command, compute locate_offset.
Subsequent locate_file commands use that offset.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
2021-06-30 09:08:58 +02:00
Dominik Csapak
3d3769830b tape/helpers/snapshot_reader: sort chunks by inode (per index)
sort the chunks we want to backup to tape by inode, to gain some
speed on spinning disks. this is done per index, not globally.

costs a bit memory, but not too much, about 16 bytes per chunk which
would mean ~4MiB for a 1TiB index with 4MiB chunks.

Signed-off-by: Dominik Csapak <d.csapak@proxmox.com>
2021-06-28 12:16:14 +02:00