tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

docs: document data blob format

Browse Source
This commit is contained in:
Dietmar Maurer 2021-02-28 10:32:56 +01:00
parent 5006632550
commit ec07a280ba
2 changed files with 50 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
docs/backup-protocol.rst
View File

@ -38,7 +38,7 @@ Upload Blobs
~~~~~~~~~~~~ ~~~~~~~~~~~~
Uploading blobs is done using ``POST /blob``. The HTTP body contains the Uploading blobs is done using ``POST /blob``. The HTTP body contains the
data encoded as ``DataBlob`` (see source code). data encoded as :ref:`Data Blob <data-blob-format>`).
The file name needs to end with ``.blob``, and is automatically added The file name needs to end with ``.blob``, and is automatically added
to the backup manifest. to the backup manifest.
@ -50,7 +50,7 @@ Upload Chunks
Chunks belong to an index, so you first need to open an index (see Chunks belong to an index, so you first need to open an index (see
below). After that, you can upload chunks using ``POST /fixed_chunk`` below). After that, you can upload chunks using ``POST /fixed_chunk``
and ``POST /dynamic_chunk``. The HTTP body contains the chunk data and ``POST /dynamic_chunk``. The HTTP body contains the chunk data
encoded as ``DataBlob`` (see source code). encoded as :ref:`Data Blob <data-blob-format>`).
Upload Fixed Indexes Upload Fixed Indexes
@ -122,14 +122,14 @@ Download Blobs
~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~
Downloading blobs is done using ``GET /download``. The HTTP body contains the Downloading blobs is done using ``GET /download``. The HTTP body contains the
data encoded as ``DataBlob`` (see source code). data encoded as :ref:`Data Blob <data-blob-format>`.
Download Chunks Download Chunks
~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~
Downloading chunks is done using ``GET /chunk``. The HTTP body contains the Downloading chunks is done using ``GET /chunk``. The HTTP body contains the
data encoded as ``DataBlob`` (see source code). data encoded as :ref:`Data Blob <data-blob-format>`).
Download Index Files Download Index Files

48
docs/file-formats.rst
View File

@ -6,7 +6,51 @@ File Formats
Proxmox File Archive Format (``.pxar``) Proxmox File Archive Format (``.pxar``)
--------------------------------------- ---------------------------------------
.. graphviz:: pxar-format-overview.dot .. graphviz:: pxar-format-overview.dot
.. _data-blob-format:
Data Blob Format
----------------
The data blob format is used to store small binary data. The magic number decides the exact format:
.. list-table::
:widths: auto
* - ``[66, 171, 56, 7, 190, 131, 112, 161]``
- unencrypted
- uncompressed
* - ``[49, 185, 88, 66, 111, 182, 163, 127]``
- unencrypted
- compressed
* - ``[123, 103, 133, 190, 34, 45, 76, 240]``
- encrypted
- uncompressed
* - ``[230, 89, 27, 191, 11, 191, 216, 11]``
- encrypted
- compressed
Compression algorithm is ``zstd``. Encryption cipher is ``AES_256_GCM``.
Unencrypted blobs use the following format:
.. list-table::
:widths: auto
* - ``MAGIC: [u8; 8]``
* - ``CRC32: [u8; 4]``
* - ``Data: (max 16MiB)``
Encrypted blobs additionally contains a 16 byte IV, followed by a 16
byte Authenticated Encyryption (AE) tag, followed by the encrypted
data:
.. list-table::
* - ``MAGIC: [u8; 8]``
* - ``CRC32: [u8; 4]``
* - ``ÌV: [u8; 16]``
* - ``TAG: [u8; 16]``
* - ``Data: (max 16MiB)``