Go to file
Stefan Reiter 4799280ccd http_client: add timeouts for critical connects
Use timeout futures for sections that might hang in certain error
conditions. This is mostly intended to be used as a safeguard, not a
first line of defense - i.e. best-effort avoidance of total hangs.

Not every future used for the HttpClient/H2Client is changed, only those
where a quick response is to be expected. For example, the response
reading futures are left alone, so data transfer is never capped with
timeout, only the initial server connect.

It is also used for upgrading to H2 connections, as that can take a long
time on overloaded servers.

Signed-off-by: Stefan Reiter <s.reiter@proxmox.com>
2020-12-22 13:31:10 +01:00
.cargo cargo: switch to use packaged crates by default 2020-01-03 09:40:33 +01:00
debian depend on proxmox 0.9.0 2020-12-22 08:52:24 +01:00
docs tape: update user docu 2020-12-21 12:13:35 +01:00
etc docs: update package repositories 2020-11-10 13:14:04 +01:00
examples client/remote: allow using ApiToken + secret 2020-10-29 15:14:27 +01:00
src http_client: add timeouts for critical connects 2020-12-22 13:31:10 +01:00
tests tests: verify-api: check AllOf schemas 2020-12-22 07:31:38 +01:00
www tape: add svg icon 2020-12-11 13:02:23 +01:00
zsh-completions Add zsh completion scripts 2020-02-27 06:50:14 +01:00
.gitignore .gitignore: add build directory 2020-07-22 09:19:29 +02:00
Cargo.toml depend on proxmox 0.9.1 2020-12-22 13:30:41 +01:00
defines.mk Add zsh completion scripts 2020-02-27 06:50:14 +01:00
Makefile add daily update and maintenance task 2020-10-31 22:51:26 +01:00
README.rst docs: add more thoughts about chunk size 2020-12-01 10:28:06 +01:00
rustfmt.toml import rustfmt.toml 2019-08-22 13:44:57 +02:00
TODO.rst [chore] bump to using pxar 0.2.0 2020-06-25 09:46:56 +02:00

``rustup`` Toolchain
====================

We normally want to build with the ``rustc`` Debian package. To do that
you can set the following ``rustup`` configuration:

    # rustup toolchain link system /usr
    # rustup default system


Versioning of proxmox helper crates
===================================

To use current git master code of the proxmox* helper crates, add::

   git = "git://git.proxmox.com/git/proxmox"

or::

   path = "../proxmox/proxmox"

to the proxmox dependency, and update the version to reflect the current,
pre-release version number (e.g., "0.1.1-dev.1" instead of "0.1.0").


Local cargo config
==================

This repository ships with a ``.cargo/config`` that replaces the crates.io
registry with packaged crates located in ``/usr/share/cargo/registry``.

A similar config is also applied building with dh_cargo. Cargo.lock needs to be
deleted when switching between packaged crates and crates.io, since the
checksums are not compatible.

To reference new dependencies (or updated versions) that are not yet packaged,
the dependency needs to point directly to a path or git source (e.g., see
example for proxmox crate above).


Build
=====
on Debian Buster

Setup:
  1. # echo 'deb http://download.proxmox.com/debian/devel/ buster main' >> /etc/apt/sources.list.d/proxmox-devel.list
  2. # sudo wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg
  3. # sudo apt update
  4. # sudo apt install devscripts debcargo clang
  5. # git clone git://git.proxmox.com/git/proxmox-backup.git
  6. # sudo mk-build-deps -ir

Note: 2. may be skipped if you already added the PVE or PBS package repository

You are now able to build using the Makefile or cargo itself.


Design Notes
============

Here are some random thought about the software design (unless I find a better place).


Large chunk sizes
-----------------

It is important to notice that large chunk sizes are crucial for
performance. We have a multi-user system, where different people can do
different operations on a datastore at the same time, and most operation
involves reading a series of chunks.

So what is the maximal theoretical speed we can get when reading a
series of chunks? Reading a chunk sequence need the following steps:

- seek to the first chunk start location
- read the chunk data
- seek to the first chunk start location
- read the chunk data
- ...

Lets use the following disk performance metrics:

:AST: Average Seek Time (second)
:MRS: Maximum sequential Read Speed (bytes/second)
:ACS: Average Chunk Size (bytes)

The maximum performance you can get is::

  MAX(ACS) = ACS /(AST + ACS/MRS)

Please note that chunk data is likely to be sequential arranged on disk, but
this it is sort of a best case assumption.

For a typical rotational disk, we assume the following values::

  AST: 10ms
  MRS: 170MB/s

  MAX(4MB)  = 115.37 MB/s
  MAX(1MB)  =  61.85 MB/s;
  MAX(64KB) =   6.02 MB/s;
  MAX(4KB)  =   0.39 MB/s;
  MAX(1KB)  =   0.10 MB/s;

Modern SSD are much faster, lets assume the following::

  max IOPS: 20000 => AST = 0.00005
  MRS: 500Mb/s

  MAX(4MB)  = 474 MB/s
  MAX(1MB)  = 465 MB/s;
  MAX(64KB) = 354 MB/s;
  MAX(4KB)  =  67 MB/s;
  MAX(1KB)  =  18 MB/s;


Also, the average chunk directly relates to the number of chunks produced by
a backup::

  CHUNK_COUNT = BACKUP_SIZE / ACS

Here are some staticics from my developer worstation::

  Disk Usage:       65 GB
  Directories:   58971
  Files:        726314
  Files < 64KB: 617541

As you see, there are really many small files. If we would do file
level deduplication, i.e. generate one chunk per file, we end up with
more than 700000 chunks.

Instead, our current algorithm only produce large chunks with an
average chunks size of 4MB. With above data, this produce about 15000
chunks (factor 50 less chunks).