tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

docs: language and formatting fixup

Browse Source 
Some minor langague and formatting fixes to sections: Proxmox VE
Integration, pxar Command Line Tool, Managing Remotes, Maintenance
Tasks, Host System Administration, Network Management, and Technical
Overview.

Signed-off-by: Dylan Whyte <d.whyte@proxmox.com>
This commit is contained in:
Dylan Whyte 2021-10-11 17:15:19 +02:00 committed by Dietmar Maurer
parent 5fb852afed
commit 7ccbce03d3
9 changed files with 221 additions and 200 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

133
docs/local-zfs.rst
View File

@ -4,17 +4,17 @@
ZFS on Linux
------------
ZFS is a combined file system and logical volume manager designed by
ZFS is a combined file system and logical volume manager, designed by
Sun Microsystems. There is no need to manually compile ZFS modules - all
packages are included.
By using ZFS, it's possible to achieve maximum enterprise features with
low budget hardware, but also high performance systems by leveraging
SSD caching or even SSD only setups. ZFS can replace cost intense
hardware raid cards by moderate CPU and memory load combined with easy
low budget hardware, and also high performance systems by leveraging
SSD caching or even SSD only setups. ZFS can replace expensive
hardware raid cards with moderate CPU and memory load, combined with easy
management.
General ZFS advantages
General advantages of ZFS:
* Easy configuration and management with GUI and CLI.
* Reliable
@ -34,18 +34,18 @@ General ZFS advantages
Hardware
~~~~~~~~~
ZFS depends heavily on memory, so you need at least 8GB to start. In
practice, use as much you can get for your hardware/budget. To prevent
ZFS depends heavily on memory, so it's recommended to have at least 8GB to
start. In practice, use as much you can get for your hardware/budget. To prevent
data corruption, we recommend the use of high quality ECC RAM.
If you use a dedicated cache and/or log disk, you should use an
enterprise class SSD (e.g. Intel SSD DC S3700 Series). This can
enterprise class SSD (for example, Intel SSD DC S3700 Series). This can
increase the overall performance significantly.
IMPORTANT: Do not use ZFS on top of hardware controller which has its
IMPORTANT: Do not use ZFS on top of a hardware controller which has its
own cache management. ZFS needs to directly communicate with disks. An
HBA adapter is the way to go, or something like LSI controller flashed
in ``IT`` mode.
HBA adapter or something like an LSI controller flashed in ``IT`` mode is
recommended.
ZFS Administration
@ -53,7 +53,7 @@ ZFS Administration
This section gives you some usage examples for common tasks. ZFS
itself is really powerful and provides many options. The main commands
to manage ZFS are `zfs` and `zpool`. Both commands come with great
to manage ZFS are `zfs` and `zpool`. Both commands come with extensive
manual pages, which can be read with:
.. code-block:: console
@ -123,7 +123,7 @@ Create a new pool with cache (L2ARC)
It is possible to use a dedicated cache drive partition to increase
the performance (use SSD).
As `<device>` it is possible to use more devices, like it's shown in
For `<device>`, you can use multiple devices, as is shown in
"Create a new pool with RAID*".
.. code-block:: console
@ -136,7 +136,7 @@ Create a new pool with log (ZIL)
It is possible to use a dedicated cache drive partition to increase
the performance (SSD).
As `<device>` it is possible to use more devices, like it's shown in
For `<device>`, you can use multiple devices, as is shown in
"Create a new pool with RAID*".
.. code-block:: console
@ -146,8 +146,9 @@ As `<device>` it is possible to use more devices, like it's shown in
Add cache and log to an existing pool
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you have a pool without cache and log. First partition the SSD in
2 partition with `parted` or `gdisk`
You can add cache and log devices to a pool after its creation. In this example,
we will use a single drive for both cache and log. First, you need to create
2 partitions on the SSD with `parted` or `gdisk`
.. important:: Always use GPT partition tables.
@ -171,12 +172,12 @@ Changing a failed device
Changing a failed bootable device
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Depending on how Proxmox Backup was installed it is either using `grub` or `systemd-boot`
as bootloader.
Depending on how Proxmox Backup was installed, it is either using `grub` or
`systemd-boot` as a bootloader.
The first steps of copying the partition table, reissuing GUIDs and replacing
the ZFS partition are the same. To make the system bootable from the new disk,
different steps are needed which depend on the bootloader in use.
In either case, the first steps of copying the partition table, reissuing GUIDs
and replacing the ZFS partition are the same. To make the system bootable from
the new disk, different steps are needed which depend on the bootloader in use.
.. code-block:: console
@ -207,7 +208,7 @@ Usually `grub.cfg` is located in `/boot/grub/grub.cfg`
# grub-mkconfig -o /path/to/grub.cfg
Activate E-Mail Notification
Activate e-mail notification
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ZFS comes with an event daemon, which monitors events generated by the
@ -219,24 +220,24 @@ and you can install it using `apt-get`:
# apt-get install zfs-zed
To activate the daemon it is necessary to edit `/etc/zfs/zed.d/zed.rc` with your
favorite editor, and uncomment the `ZED_EMAIL_ADDR` setting:
To activate the daemon, it is necessary to to uncomment the ZED_EMAIL_ADDR
setting, in the file `/etc/zfs/zed.d/zed.rc`.
.. code-block:: console
ZED_EMAIL_ADDR="root"
Please note Proxmox Backup forwards mails to `root` to the email address
Please note that Proxmox Backup forwards mails to `root` to the email address
configured for the root user.
IMPORTANT: The only setting that is required is `ZED_EMAIL_ADDR`. All
other settings are optional.
Limit ZFS Memory Usage
Limit ZFS memory usage
^^^^^^^^^^^^^^^^^^^^^^
It is good to use at most 50 percent (which is the default) of the
system memory for ZFS ARC to prevent performance shortage of the
system memory for ZFS ARC, to prevent performance degradation of the
host. Use your preferred editor to change the configuration in
`/etc/modprobe.d/zfs.conf` and insert:
@ -244,27 +245,42 @@ host. Use your preferred editor to change the configuration in
options zfs zfs_arc_max=8589934592
This example setting limits the usage to 8GB.
The above example limits the usage to 8 GiB ('8 * 2^30^').
.. IMPORTANT:: If your root file system is ZFS you must update your initramfs every time this value changes:
.. IMPORTANT:: In case your desired `zfs_arc_max` value is lower than or equal
to `zfs_arc_min` (which defaults to 1/32 of the system memory), `zfs_arc_max`
will be ignored. Thus, for it to work in this case, you must set
`zfs_arc_min` to at most `zfs_arc_max - 1`. This would require updating the
configuration in `/etc/modprobe.d/zfs.conf`, with:
.. code-block:: console
options zfs zfs_arc_min=8589934591
options zfs zfs_arc_max=8589934592
This example setting limits the usage to 8 GiB ('8 * 2^30^') on
systems with more than 256 GiB of total memory, where simply setting
`zfs_arc_max` alone would not work.
.. IMPORTANT:: If your root file system is ZFS, you must update your initramfs
every time this value changes.
.. code-block:: console
# update-initramfs -u
SWAP on ZFS
Swap on ZFS
^^^^^^^^^^^
Swap-space created on a zvol may generate some troubles, like blocking the
Swap-space created on a zvol may cause some issues, such as blocking the
server or generating a high IO load, often seen when starting a Backup
to an external Storage.
We strongly recommend to use enough memory, so that you normally do not
We strongly recommend using enough memory, so that you normally do not
run into low memory situations. Should you need or want to add swap, it is
preferred to create a partition on a physical disk and use it as swap device.
preferred to create a partition on a physical disk and use it as a swap device.
You can leave some space free for this purpose in the advanced options of the
installer. Additionally, you can lower the `swappiness` value.
installer. Additionally, you can lower the `swappiness` value.
A good value for servers is 10:
.. code-block:: console
@ -291,7 +307,7 @@ an editor of your choice and add the following line:
vm.swappiness = 100 The kernel will swap aggressively.
==================== ===============================================================
ZFS Compression
ZFS compression
^^^^^^^^^^^^^^^
To activate compression:
@ -300,10 +316,11 @@ To activate compression:
# zpool set compression=lz4 <pool>
We recommend using the `lz4` algorithm, since it adds very little CPU overhead.
Other algorithms such as `lzjb` and `gzip-N` (where `N` is an integer `1-9` representing
the compression ratio, 1 is fastest and 9 is best compression) are also available.
Depending on the algorithm and how compressible the data is, having compression enabled can even increase
I/O performance.
Other algorithms such as `lzjb` and `gzip-N` (where `N` is an integer from `1-9`
representing the compression ratio, where 1 is fastest and 9 is best
compression) are also available. Depending on the algorithm and how
compressible the data is, having compression enabled can even increase I/O
performance.
You can disable compression at any time with:
.. code-block:: console
@ -314,26 +331,26 @@ Only new blocks will be affected by this change.
.. _local_zfs_special_device:
ZFS Special Device
ZFS special device
^^^^^^^^^^^^^^^^^^
Since version 0.8.0 ZFS supports `special` devices. A `special` device in a
Since version 0.8.0, ZFS supports `special` devices. A `special` device in a
pool is used to store metadata, deduplication tables, and optionally small
file blocks.
A `special` device can improve the speed of a pool consisting of slow spinning
hard disks with a lot of metadata changes. For example workloads that involve
hard disks with a lot of metadata changes. For example, workloads that involve
creating, updating or deleting a large number of files will benefit from the
presence of a `special` device. ZFS datasets can also be configured to store
whole small files on the `special` device which can further improve the
small files on the `special` device, which can further improve the
performance. Use fast SSDs for the `special` device.
.. IMPORTANT:: The redundancy of the `special` device should match the one of the
pool, since the `special` device is a point of failure for the whole pool.
pool, since the `special` device is a point of failure for the entire pool.
.. WARNING:: Adding a `special` device to a pool cannot be undone!
Create a pool with `special` device and RAID-1:
To create a pool with `special` device and RAID-1:
.. code-block:: console
@ -346,8 +363,8 @@ Adding a `special` device to an existing pool with RAID-1:
# zpool add <pool> special mirror <device1> <device2>
ZFS datasets expose the `special_small_blocks=<size>` property. `size` can be
`0` to disable storing small file blocks on the `special` device or a power of
two in the range between `512B` to `128K`. After setting the property new file
`0` to disable storing small file blocks on the `special` device, or a power of
two in the range between `512B` to `128K`. After setting this property, new file
blocks smaller than `size` will be allocated on the `special` device.
.. IMPORTANT:: If the value for `special_small_blocks` is greater than or equal to
@ -355,10 +372,10 @@ blocks smaller than `size` will be allocated on the `special` device.
the `special` device, so be careful!
Setting the `special_small_blocks` property on a pool will change the default
value of that property for all child ZFS datasets (for example all containers
value of that property for all child ZFS datasets (for example, all containers
in the pool will opt in for small file blocks).
Opt in for all file smaller than 4K-blocks pool-wide:
Opt in for all files smaller than 4K-blocks pool-wide:
.. code-block:: console
@ -379,10 +396,15 @@ Opt out from small file blocks for a single dataset:
Troubleshooting
^^^^^^^^^^^^^^^
Corrupted cachefile
Corrupt cache file
""""""""""""""""""
In case of a corrupted ZFS cachefile, some volumes may not be mounted during
boot until mounted manually later.
`zfs-import-cache.service` imports ZFS pools using the ZFS cache file. If this
file becomes corrupted, the service won't be able to import the pools that it's
unable to read from it.
As a result, in case of a corrupted ZFS cache file, some volumes may not be
mounted during boot and must be mounted manually later.
For each pool, run:
@ -390,16 +412,13 @@ For each pool, run:
# zpool set cachefile=/etc/zfs/zpool.cache POOLNAME
and afterwards update the `initramfs` by running:
then, update the `initramfs` by running:
.. code-block:: console
# update-initramfs -u -k all
and finally reboot your node.
Sometimes the ZFS cachefile can get corrupted, and `zfs-import-cache.service`
doesn't import the pools that aren't present in the cachefile.
and finally, reboot the node.
Another workaround to this problem is enabling the `zfs-import-scan.service`,
which searches and imports pools via device scanning (usually slower).

54
docs/maintenance.rst
View File

@ -14,15 +14,15 @@ following retention options are available:
``keep-hourly <N>``
Keep backups for the last ``<N>`` hours. If there is more than one
backup for a single hour, only the latest is kept.
backup for a single hour, only the latest is retained.
``keep-daily <N>``
Keep backups for the last ``<N>`` days. If there is more than one
backup for a single day, only the latest is kept.
backup for a single day, only the latest is retained.
``keep-weekly <N>``
Keep backups for the last ``<N>`` weeks. If there is more than one
backup for a single week, only the latest is kept.
backup for a single week, only the latest is retained.
.. note:: Weeks start on Monday and end on Sunday. The software
uses the `ISO week date`_ system and handles weeks at
@ -30,17 +30,17 @@ following retention options are available:
``keep-monthly <N>``
Keep backups for the last ``<N>`` months. If there is more than one
backup for a single month, only the latest is kept.
backup for a single month, only the latest is retained.
``keep-yearly <N>``
Keep backups for the last ``<N>`` years. If there is more than one
backup for a single year, only the latest is kept.
backup for a single year, only the latest is retained.
The retention options are processed in the order given above. Each option
only covers backups within its time period. The next option does not take care
of already covered backups. It will only consider older backups.
Unfinished and incomplete backups will be removed by the prune command unless
Unfinished and incomplete backups will be removed by the prune command, unless
they are newer than the last successful backup. In this case, the last failed
backup is retained.
@ -48,7 +48,7 @@ Prune Simulator
^^^^^^^^^^^^^^^
You can use the built-in `prune simulator <prune-simulator/index.html>`_
to explore the effect of different retetion options with various backup
to explore the effect of different retention options with various backup
schedules.
Manual Pruning
@ -59,10 +59,10 @@ Manual Pruning
:align: right
:alt: Prune and garbage collection options
To access pruning functionality for a specific backup group, you can use the
prune command line option discussed in :ref:`backup-pruning`, or navigate to
the **Content** tab of the datastore and click the scissors icon in the
**Actions** column of the relevant backup group.
To manually prune a specific backup group, you can use
``proxmox-backup-client``'s ``prune`` subcommand, discussed in
:ref:`backup-pruning`, or navigate to the **Content** tab of the datastore and
click the scissors icon in the **Actions** column of the relevant backup group.
Prune Schedules
^^^^^^^^^^^^^^^
@ -81,7 +81,7 @@ Retention Settings Example
^^^^^^^^^^^^^^^^^^^^^^^^^^
The backup frequency and retention of old backups may depend on how often data
changes, and how important an older state may be, in a specific work load.
changes and how important an older state may be in a specific workload.
When backups act as a company's document archive, there may also be legal
requirements for how long backup snapshots must be kept.
@ -125,8 +125,8 @@ start garbage collection on an entire datastore and the ``status`` subcommand to
see attributes relating to the :ref:`garbage collection <client_garbage-collection>`.
This functionality can also be accessed in the GUI, by navigating to **Prune &
GC** from the top panel. From here, you can edit the schedule at which garbage
collection runs and manually start the operation.
GC** from the top panel of a datastore. From here, you can edit the schedule at
which garbage collection runs and manually start the operation.
.. _maintenance_verification:
@ -139,13 +139,13 @@ Verification
:align: right
:alt: Adding a verify job
Proxmox Backup offers various verification options to ensure that backup data is
intact. Verification is generally carried out through the creation of verify
jobs. These are scheduled tasks that run verification at a given interval (see
:ref:`calendar-event-scheduling`). With these, you can set whether already verified
snapshots are ignored, as well as set a time period, after which verified jobs
are checked again. The interface for creating verify jobs can be found under the
**Verify Jobs** tab of the datastore.
Proxmox Backup Server offers various verification options to ensure that backup
data is intact. Verification is generally carried out through the creation of
verify jobs. These are scheduled tasks that run verification at a given interval
(see :ref:`calendar-event-scheduling`). With these, you can also set whether
already verified snapshots are ignored, as well as set a time period, after
which snapshots are checked again. The interface for creating verify jobs can be
found under the **Verify Jobs** tab of the datastore.
.. Note:: It is recommended that you reverify all backups at least monthly, even
if a previous verification was successful. This is because physical drives
@ -158,9 +158,9 @@ are checked again. The interface for creating verify jobs can be found under the
data.
Aside from using verify jobs, you can also run verification manually on entire
datastores, backup groups, or snapshots. To do this, navigate to the **Content**
tab of the datastore and either click *Verify All*, or select the *V.* icon from
the *Actions* column in the table.
datastores, backup groups or snapshots. To do this, navigate to the **Content**
tab of the datastore and either click *Verify All* or select the *V.* icon from
the **Actions** column in the table.
.. _maintenance_notification:
@ -170,8 +170,8 @@ Notifications
Proxmox Backup Server can send you notification emails about automatically
scheduled verification, garbage-collection and synchronization tasks results.
By default, notifications are send to the email address configured for the
`root@pam` user. You can set that user for each datastore.
By default, notifications are sent to the email address configured for the
`root@pam` user. You can instead set this user for each datastore.
You can also change the level of notification received per task type, the
following options are available:
@ -179,6 +179,6 @@ following options are available:
* Always: send a notification for any scheduled task, independent of the
outcome
* Errors: send a notification for any scheduled task resulting in an error
* Errors: send a notification for any scheduled task that results in an error
* Never: do not send any notification at all

23
docs/managing-remotes.rst
View File

@ -17,8 +17,8 @@ configuration information for remotes is stored in the file
:align: right
:alt: Add a remote
To add a remote, you need its hostname or IP, a userid and password on the
remote, and its certificate fingerprint. To get the fingerprint, use the
To add a remote, you need its hostname or IP address, a userid and password on
the remote, and its certificate fingerprint. To get the fingerprint, use the
``proxmox-backup-manager cert info`` command on the remote, or navigate to
**Dashboard** in the remote's web interface and select **Show Fingerprint**.
@ -60,12 +60,13 @@ Sync Jobs
Sync jobs are configured to pull the contents of a datastore on a **Remote** to
a local datastore. You can manage sync jobs in the web interface, from the
**Sync Jobs** tab of the datastore which you'd like to set one up for, or using
the ``proxmox-backup-manager sync-job`` command. The configuration information
for sync jobs is stored at ``/etc/proxmox-backup/sync.cfg``. To create a new
sync job, click the add button in the GUI, or use the ``create`` subcommand.
After creating a sync job, you can either start it manually from the GUI or
provide it with a schedule (see :ref:`calendar-event-scheduling`) to run regularly.
**Sync Jobs** tab of the **Datastore** panel or from that of the Datastore
itself. Alternatively, you can manage them with the ``proxmox-backup-manager
sync-job`` command. The configuration information for sync jobs is stored at
``/etc/proxmox-backup/sync.cfg``. To create a new sync job, click the add button
in the GUI, or use the ``create`` subcommand. After creating a sync job, you can
either start it manually from the GUI or provide it with a schedule (see
:ref:`calendar-event-scheduling`) to run regularly.
.. code-block:: console
@ -79,14 +80,14 @@ provide it with a schedule (see :ref:`calendar-event-scheduling`) to run regular
└────────────┴───────┴────────┴──────────────┴───────────┴─────────┘
# proxmox-backup-manager sync-job remove pbs2-local
For setting up sync jobs, the configuring user needs the following permissions:
To set up sync jobs, the configuring user needs the following permissions:
#. ``Remote.Read`` on the ``/remote/{remote}/{remote-store}`` path
#. at least ``Datastore.Backup`` on the local target datastore (``/datastore/{store}``)
#. At least ``Datastore.Backup`` on the local target datastore (``/datastore/{store}``)
If the ``remove-vanished`` option is set, ``Datastore.Prune`` is required on
the local datastore as well. If the ``owner`` option is not set (defaulting to
``root@pam``) or set to something other than the configuring user,
``root@pam``) or is set to something other than the configuring user,
``Datastore.Modify`` is required as well.
.. note:: A sync job can only sync backup groups that the configured remote's

3
docs/network-management.rst
View File

@ -82,7 +82,8 @@ is:
.. note:: This command and corresponding GUI button rely on the ``ifreload``
command, from the package ``ifupdown2``. This package is included within the
Proxmox Backup Server installation, however, you may have to install it yourself,
if you have installed Proxmox Backup Server on top of Debian or Proxmox VE.
if you have installed Proxmox Backup Server on top of Debian or a Proxmox VE
version prior to version 7.
You can also configure DNS settings, from the **DNS** section
of **Configuration** or by using the ``dns`` subcommand of

2
docs/proxmox-backup-proxy/description.rst
View File

@ -1,5 +1,5 @@
This daemon exposes the whole Proxmox Backup Server API on TCP port
8007 using HTTPS. It runs as user ``backup`` and has very limited
permissions. Operation requiring more permissions are forwarded to
permissions. Operations requiring more permissions are forwarded to
the local ``proxmox-backup`` service.

16
docs/pve-integration.rst
View File

@ -3,8 +3,8 @@
`Proxmox VE`_ Integration
-------------------------
A Proxmox Backup Server can be integrated into a Proxmox VE setup by adding the
former as a storage in a Proxmox VE standalone or cluster setup.
Proxmox Backup Server can be integrated into a Proxmox VE standalone or cluster
setup, by adding it as a storage in Proxmox VE.
See also the `Proxmox VE Storage - Proxmox Backup Server
<https://pve.proxmox.com/pve-docs/pve-admin-guide.html#storage_pbs>`_ section
@ -14,8 +14,8 @@ of the Proxmox VE Administration Guide for Proxmox VE specific documentation.
Using the Proxmox VE Web-Interface
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Proxmox VE has native API and web-interface integration of Proxmox Backup
Server since the `Proxmox VE 6.3 release
Proxmox VE has native API and web interface integration of Proxmox Backup
Server as of `Proxmox VE 6.3
<https://pve.proxmox.com/wiki/Roadmap#Proxmox_VE_6.3>`_.
A Proxmox Backup Server can be added under ``Datacenter -> Storage``.
@ -24,8 +24,8 @@ Using the Proxmox VE Command-Line
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You need to define a new storage with type 'pbs' on your `Proxmox VE`_
node. The following example uses ``store2`` as storage name, and
assumes the server address is ``localhost``, and you want to connect
node. The following example uses ``store2`` as the storage's name, and
assumes the server address is ``localhost`` and you want to connect
as ``user1@pbs``.
.. code-block:: console
@ -33,7 +33,7 @@ as ``user1@pbs``.
# pvesm add pbs store2 --server localhost --datastore store2
# pvesm set store2 --username user1@pbs --password <secret>
.. note:: If you would rather not pass your password as plain text, you can pass
.. note:: If you would rather not enter your password as plain text, you can pass
the ``--password`` parameter, without any arguments. This will cause the
program to prompt you for a password upon entering the command.
@ -53,7 +53,7 @@ relationship:
# pvesm set store2 --fingerprint 64:d3:ff:3a:50:38:53:5a:9b:f7:50:...:ab:fe
After that you should be able to see storage status with:
After that, you should be able to view storage status with:
.. code-block:: console

52
docs/pxar/description.rst
View File

@ -1,12 +1,12 @@
``pxar`` is a command line utility to create and manipulate archives in the
``pxar`` is a command line utility for creating and manipulating archives in the
:ref:`pxar-format`.
It is inspired by `casync file archive format
<http://0pointer.net/blog/casync-a-tool-for-distributing-file-system-images.html>`_,
which caters to a similar use-case.
The ``.pxar`` format is adapted to fulfill the specific needs of the Proxmox
Backup Server, for example, efficient storage of hardlinks.
The format is designed to reduce storage space needed on the server by achieving
a high level of deduplication.
Backup Server, for example, efficient storage of hard links.
The format is designed to reduce the required storage on the server by
achieving a high level of deduplication.
Creating an Archive
^^^^^^^^^^^^^^^^^^^
@ -24,10 +24,10 @@ This will create a new archive called ``archive.pxar`` with the contents of the
the same name is already present in the target folder, the creation will
fail.
By default, ``pxar`` will skip certain mountpoints and will not follow device
By default, ``pxar`` will skip certain mount points and will not follow device
boundaries. This design decision is based on the primary use case of creating
archives for backups. It makes sense to not back up the contents of certain
temporary or system specific files.
archives for backups. It makes sense to ignore the contents of certain
temporary or system specific files in a backup.
To alter this behavior and follow device boundaries, use the
``--all-file-systems`` flag.
@ -41,40 +41,38 @@ by running:
# pxar create archive.pxar /path/to/source --exclude '**/*.txt'
Be aware that the shell itself will try to expand all of the glob patterns before
invoking ``pxar``.
In order to avoid this, all globs have to be quoted correctly.
Be aware that the shell itself will try to expand glob patterns before invoking
``pxar``. In order to avoid this, all globs have to be quoted correctly.
It is possible to pass the ``--exclude`` parameter multiple times, in order to
match more than one pattern. This allows you to use more complex
file exclusion/inclusion behavior. However, it is recommended to use
file inclusion/exclusion behavior. However, it is recommended to use
``.pxarexclude`` files instead for such cases.
For example you might want to exclude all ``.txt`` files except for a specific
one from the archive. This is achieved via the negated match pattern, prefixed
by ``!``.
All the glob patterns are relative to the ``source`` directory.
For example you might want to exclude all ``.txt`` files except a specific
one from the archive. This would be achieved via the negated match pattern,
prefixed by ``!``. All the glob patterns are relative to the ``source``
directory.
.. code-block:: console
# pxar create archive.pxar /path/to/source --exclude '**/*.txt' --exclude '!/folder/file.txt'
.. NOTE:: The order of the glob match patterns matters as later ones override
previous ones. Permutations of the same patterns lead to different results.
.. NOTE:: The order of the glob match patterns matters, as later ones override
earlier ones. Permutations of the same patterns lead to different results.
``pxar`` will store the list of glob match patterns passed as parameters via the
command line, in a file called ``.pxarexclude-cli`` at the root of
the archive.
command line, in a file called ``.pxarexclude-cli``, at the root of the archive.
If a file with this name is already present in the source folder during archive
creation, this file is not included in the archive and the file containing the
new patterns is added to the archive instead, the original file is not altered.
creation, this file is not included in the archive, and the file containing the
new patterns is added to the archive instead. The original file is not altered.
A more convenient and persistent way to exclude files from the archive is by
placing the glob match patterns in ``.pxarexclude`` files.
It is possible to create and place these files in any directory of the filesystem
tree.
These files must contain one pattern per line, again later patterns win over
previous ones.
These files must contain one pattern per line, and later patterns override
earlier ones.
The patterns control file exclusions of files present within the given directory
or further below it in the tree.
The behavior is the same as described in :ref:`client_creating_backups`.
@ -89,7 +87,7 @@ with the following command:
# pxar extract archive.pxar /path/to/target
If no target is provided, the content of the archive is extracted to the current
If no target is provided, the contents of the archive is extracted to the current
working directory.
In order to restore only parts of an archive, single files, and/or folders,
@ -116,13 +114,13 @@ run the following command:
# pxar list archive.pxar
This displays the full path of each file or directory with respect to the
archives root.
archive's root.
Mounting an Archive
^^^^^^^^^^^^^^^^^^^
``pxar`` allows you to mount and inspect the contents of an archive via _`FUSE`.
In order to mount an archive named ``archive.pxar`` to the mountpoint ``/mnt``,
In order to mount an archive named ``archive.pxar`` to the mount point ``/mnt``,
run the command:
.. code-block:: console
@ -130,7 +128,7 @@ run the command:
# pxar mount archive.pxar /mnt
Once the archive is mounted, you can access its content under the given
mountpoint.
mount point.
.. code-block:: console

8
docs/sysadmin.rst
View File

@ -4,8 +4,8 @@ Host System Administration
==========================
`Proxmox Backup`_ is based on the famous Debian_ Linux
distribution. That means that you have access to the whole world of
Debian packages, and the base system is well documented. The `Debian
distribution. This means that you have access to the entire range of
Debian packages, and that the base system is well documented. The `Debian
Administrator's Handbook`_ is available online, and provides a
comprehensive introduction to the Debian operating system.
@ -17,11 +17,11 @@ updates to some Debian packages when necessary.
We also deliver a specially optimized Linux kernel, where we enable
all required virtualization and container features. That kernel
includes drivers for ZFS_, and several hardware drivers. For example,
includes drivers for ZFS_, as well as several hardware drivers. For example,
we ship Intel network card drivers to support their newest hardware.
The following sections will concentrate on backup related topics. They
either explain things which are different on `Proxmox Backup`_, or
will explain things which are different on `Proxmox Backup`_, or
tasks which are commonly used on `Proxmox Backup`_. For other topics,
please refer to the standard Debian documentation.

130
docs/technical-overview.rst
View File

@ -8,7 +8,7 @@ Datastores
A Datastore is the logical place where :ref:`Backup Snapshots
<term_backup_snapshot>` and their chunks are stored. Snapshots consist of a
manifest, blobs, dynamic- and fixed-indexes (see :ref:`terms`), and are
manifest, blobs, and dynamic- and fixed-indexes (see :ref:`terms`), and are
stored in the following directory structure:
<datastore-root>/<type>/<id>/<time>/
@ -32,8 +32,8 @@ The chunks of a datastore are found in
<datastore-root>/.chunks/
This chunk directory is further subdivided by the first four byte of the chunks
checksum, so the chunk with the checksum
This chunk directory is further subdivided by the first four bytes of the
chunk's checksum, so a chunk with the checksum
a342e8151cbf439ce65f3df696b54c67a114982cc0aa751f2852c2f7acc19a8b
@ -47,7 +47,7 @@ per directory can be bad for file system performance.
These chunk directories ('0000'-'ffff') will be preallocated when a datastore
is created.
Fixed-sized Chunks
Fixed-Sized Chunks
^^^^^^^^^^^^^^^^^^
For block based backups (like VMs), fixed-sized chunks are used. The content
@ -58,10 +58,10 @@ often tries to allocate files in contiguous pieces, so new files get new
blocks, and changing existing files changes only their own blocks.
As an optimization, VMs in `Proxmox VE`_ can make use of 'dirty bitmaps', which
can track the changed blocks of an image. Since these bitmap are also a
can track the changed blocks of an image. Since these bitmaps are also a
representation of the image split into chunks, there is a direct relation
between dirty blocks of the image and chunks which need to get uploaded, so
only modified chunks of the disk have to be uploaded for a backup.
between the dirty blocks of the image and chunks which need to be uploaded.
Thus, only modified chunks of the disk need to be uploaded to a backup.
Since the image is always split into chunks of the same size, unchanged blocks
will result in identical checksums for those chunks, so such chunks do not need
@ -71,24 +71,24 @@ changed blocks.
For consistency, `Proxmox VE`_ uses a QEMU internal snapshot mechanism, that
does not rely on storage snapshots either.
Dynamically sized Chunks
Dynamically Sized Chunks
^^^^^^^^^^^^^^^^^^^^^^^^
If one does not want to backup block-based systems but rather file-based
systems, using fixed-sized chunks is not a good idea, since every time a file
would change in size, the remaining data gets shifted around and this would
result in many chunks changing, reducing the amount of deduplication.
When working with file-based systems rather than block-based systems,
using fixed-sized chunks is not a good idea, since every time a file
would change in size, the remaining data would be shifted around,
resulting in many chunks changing and the amount of deduplication being reduced.
To improve this, `Proxmox Backup`_ Server uses dynamically sized chunks
instead. Instead of splitting an image into fixed sizes, it first generates a
consistent file archive (:ref:`pxar <pxar-format>`) and uses a rolling hash
over this on-the-fly generated archive to calculate chunk boundaries.
We use a variant of Buzhash which is a cyclic polynomial algorithm. It works
We use a variant of Buzhash which is a cyclic polynomial algorithm. It works
by continuously calculating a checksum while iterating over the data, and on
certain conditions it triggers a hash boundary.
certain conditions, it triggers a hash boundary.
Assuming that most files of the system that is to be backed up have not
Assuming that most files on the system that is to be backed up have not
changed, eventually the algorithm triggers the boundary on the same data as a
previous backup, resulting in chunks that can be reused.
@ -100,8 +100,8 @@ can be encrypted, and they are handled in a slightly different manner than
normal chunks.
The hashes of encrypted chunks are calculated not with the actual (encrypted)
chunk content, but with the plain-text content concatenated with the encryption
key. This way, two chunks of the same data encrypted with different keys
chunk content, but with the plain-text content, concatenated with the encryption
key. This way, two chunks with the same data but encrypted with different keys
generate two different checksums and no collisions occur for multiple
encryption keys.
@ -112,14 +112,14 @@ the previous backup, do not need to be encrypted and uploaded.
Caveats and Limitations
-----------------------
Notes on hash collisions
Notes on Hash Collisions
^^^^^^^^^^^^^^^^^^^^^^^^
Every hashing algorithm has a chance to produce collisions, meaning two (or
more) inputs generate the same checksum. For SHA-256, this chance is
negligible. To calculate such a collision, one can use the ideas of the
'birthday problem' from probability theory. For big numbers, this is actually
infeasible to calculate with regular computers, but there is a good
negligible. To calculate the chances of such a collision, one can use the ideas
of the 'birthday problem' from probability theory. For big numbers, this is
actually unfeasible to calculate with regular computers, but there is a good
approximation:
.. math::
@ -127,7 +127,7 @@ approximation:
p(n, d) = 1 - e^{-n^2/(2d)}
Where `n` is the number of tries, and `d` is the number of possibilities.
For a concrete example lets assume a large datastore of 1 PiB, and an average
For a concrete example, lets assume a large datastore of 1 PiB and an average
chunk size of 4 MiB. That means :math:`n = 268435456` tries, and :math:`d =
2^{256}` possibilities. Inserting those values in the formula from earlier you
will see that the probability of a collision in that scenario is:
@ -136,94 +136,96 @@ will see that the probability of a collision in that scenario is:
3.1115 * 10^{-61}
For context, in a lottery game of guessing 6 out of 45, the chance to correctly
guess all 6 numbers is only :math:`1.2277 * 10^{-7}`, that means the chance of
a collision is about the same as winning 13 such lotto games *in a row*.
For context, in a lottery game of guessing 6 numbers out of 45, the chance to
correctly guess all 6 numbers is only :math:`1.2277 * 10^{-7}`. This means the
chance of a collision is about the same as winning 13 such lottery games *in a
row*.
In conclusion, it is extremely unlikely that such a collision would occur by
accident in a normal datastore.
Additionally, SHA-256 is prone to length extension attacks, but since there is
an upper limit for how big the chunk are, this is not a problem, since a
an upper limit for how big the chunks are, this is not a problem, because a
potential attacker cannot arbitrarily add content to the data beyond that
limit.
File-based Backup
File-Based Backup
^^^^^^^^^^^^^^^^^
Since dynamically sized chunks (for file-based backups) are created on a custom
archive format (pxar) and not over the files directly, there is no relation
between files and the chunks. This means that the Proxmox Backup client has to
between the files and chunks. This means that the Proxmox Backup Client has to
read all files again for every backup, otherwise it would not be possible to
generate a consistent independent pxar archive where the original chunks can be
reused. Note that there will be still only new or change chunks be uploaded.
generate a consistent, independent pxar archive where the original chunks can be
reused. Note that in spite of this, only new or changed chunks will be uploaded.
Verification of encrypted chunks
Verification of Encrypted Chunks
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
For encrypted chunks, only the checksum of the original (plaintext) data is
available, making it impossible for the server (without the encryption key), to
available, making it impossible for the server (without the encryption key) to
verify its content against it. Instead only the CRC-32 checksum gets checked.
Troubleshooting
---------------
Index files(.fidx, .didx) contain information about how to rebuild a file, more
precisely, they contain an ordered list of references to the chunks the original
file was split up in. If there is something wrong with a snapshot it might be
useful to find out which chunks are referenced in this specific snapshot, and
check wheather all of them are present and intact. The command for getting the
list of referenced chunks could look something like this:
Index files(*.fidx*, *.didx*) contain information about how to rebuild a file.
More precisely, they contain an ordered list of references to the chunks that
the original file was split into. If there is something wrong with a snapshot,
it might be useful to find out which chunks are referenced in it, and check
whether they are present and intact. The ``proxmox-backup-debug`` command line
tool can be used to inspect such files and recover their contents. For example,
to get a list of the referenced chunks of a *.fidx* index:
.. code-block:: console
# proxmox-backup-debug inspect file drive-scsi0.img.fidx
The same command can be used to look at .blob file, without ``--decode`` just
the size and the encryption type, if any, is printed. If ``--decode`` is set the
blob file is decoded into the specified file('-' will decode it directly into
stdout).
The same command can be used to inspect *.blob* files. Without the ``--decode``
parameter, just the size and the encryption type, if any, are printed. If
``--decode`` is set, the blob file is decoded into the specified file ('-' will
decode it directly to stdout).
The following example would print the decoded contents of
`qemu-server.conf.blob`. If the file you're trying to inspect is encrypted, a
path to the key file must be provided using ``--keyfile``.
.. code-block:: console
# proxmox-backup-debug inspect file qemu-server.conf.blob --decode -
would print the decoded contents of `qemu-server.conf.blob`. If the file you're
trying to inspect is encrypted, a path to the keyfile has to be provided using
``--keyfile``.
Checking in which index files a specific chunk file is referenced can be done
You can also check in which index files a specific chunk file is referenced
with:
.. code-block:: console
# proxmox-backup-debug inspect chunk b531d3ffc9bd7c65748a61198c060678326a431db7eded874c327b7986e595e0 --reference-filter /path/in/a/datastore/directory
Here ``--reference-filter`` specifies where index files should be searched, this
Here ``--reference-filter`` specifies where index files should be searched. This
can be an arbitrary path. If, for some reason, the filename of the chunk was
changed you can explicitly specify the digest using ``--digest``, by default the
chunk filename is used as the digest to look for. Specifying no
``--reference-filter`` will just print the CRC and encryption status of the
chunk. You can also decode chunks, to do so ``--decode`` has to be set. If the
chunk is encrypted a ``--keyfile`` has to be provided for decoding.
changed, you can explicitly specify the digest using ``--digest``. By default, the
chunk filename is used as the digest to look for. If no ``--reference-filter``
is specified, it will only print the CRC and encryption status of the chunk. You
can also decode chunks, by setting the ``--decode`` flag. If the chunk is
encrypted, a ``--keyfile`` must be provided, in order to decode it.
Restore without a running PBS
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Restore without a Running Proxmox Backup Server
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
It is possible to restore spefiic files of snapshots without a running PBS using
the `recover` sub-command, provided you have access to the intact index and
chunk files. Note that you also need the corresponding key file if the backup
was encrypted.
It's possible to restore specific files from a snapshot, without a running
Proxmox Backup Server instance, using the ``recover`` subcommand, provided you
have access to the intact index and chunk files. Note that you also need the
corresponding key file if the backup was encrypted.
.. code-block:: console
# proxmox-backup-debug recover index drive-scsi0.img.fidx /path/to/.chunks
In above example the `/path/to/.chunks` argument is the path to the directory
that contains contains the chunks, and `drive-scsi0.img.fidx` is the index-file
of the file you'd lile to restore. Both paths can be absolute or relative. With
``--skip-crc`` it is possible to disable the crc checks of the chunks, this will
speed up the process slightly and allows for trying to restore (partially)
In the above example, the `/path/to/.chunks` argument is the path to the
directory that contains the chunks, and `drive-scsi0.img.fidx` is the index file
of the file you'd like to restore. Both paths can be absolute or relative. With
``--skip-crc``, it's possible to disable the CRC checks of the chunks. This
will speed up the process slightly and allow for trying to restore (partially)
corrupt chunks. It's recommended to always try without the skip-CRC option
first.