d91a0f9fc9
glibc's malloc has a misguided heuristic to detect transient allocations that will just result in allocation sizes below 32 MiB never using mmap. That it turn means that those relatively big allocations are on the heap where cleanup and returning memory to the OS is harder to do and easier to be blocked by long living, small allocations at the top (end) of the heap. Observing the malloc size distribution in a file-level backup run: @size: [0] 14 | | [1] 25214 |@@@@@ | [2, 4) 9090 |@ | [4, 8) 12987 |@@ | [8, 16) 93453 |@@@@@@@@@@@@@@@@@@@@ | [16, 32) 30255 |@@@@@@ | [32, 64) 237445 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@| [64, 128) 32692 |@@@@@@@ | [128, 256) 22296 |@@@@ | [256, 512) 16177 |@@@ | [512, 1K) 5139 |@ | [1K, 2K) 3352 | | [2K, 4K) 214 | | [4K, 8K) 1568 | | [8K, 16K) 95 | | [16K, 32K) 3457 | | [32K, 64K) 3175 | | [64K, 128K) 161 | | [128K, 256K) 453 | | [256K, 512K) 93 | | [512K, 1M) 74 | | [1M, 2M) 774 | | [2M, 4M) 319 | | [4M, 8M) 700 | | [8M, 16M) 93 | | [16M, 32M) 18 | | We see that all allocations will be on the heap, and that while most allocations are small, the relatively few big ones will still make up most of the RSS and if blocked from being released back to the OS result in much higher peak and average usage for the program than actually required. Avoiding the "dynamic" mmap-threshold increasement algorithm and fixing it at the original default of 128 KiB reduces RSS size by factor 10-20 when running backups. As with memory mappings other mappings or the heap can never block freeing the memory fully back to the OS. But, the drawback of using mmap is more wasted space for unaligned or small allocation sizes, and the fact that the kernel allegedly zeros out the data before giving it to user space. The former doesn't really matter for us when using it only for allocations bigger than 128 KiB, and the latter is a trade-off, using 10 to 20 times less memory brings its own performance improvement possibilities for the whole system after all ;-) Signed-off-by: Dietmar Maurer <dietmar@proxmox.com> [ Thomas: added to comment & commit message + extra-empty-line fixes ] Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com> |
||
---|---|---|
.cargo | ||
debian | ||
docs | ||
etc | ||
examples | ||
pbs-api-types | ||
pbs-buildcfg | ||
pbs-client | ||
pbs-config | ||
pbs-datastore | ||
pbs-fuse-loop | ||
pbs-tape | ||
pbs-tools | ||
proxmox-backup-banner | ||
proxmox-backup-client | ||
proxmox-file-restore | ||
proxmox-rest-server | ||
proxmox-restore-daemon | ||
proxmox-rrd | ||
pxar-bin | ||
src | ||
tests | ||
www | ||
zsh-completions | ||
.gitignore | ||
Cargo.toml | ||
defines.mk | ||
Makefile | ||
README.rst | ||
rustfmt.toml | ||
TODO.rst |
``rustup`` Toolchain ==================== We normally want to build with the ``rustc`` Debian package. To do that you can set the following ``rustup`` configuration: # rustup toolchain link system /usr # rustup default system Versioning of proxmox helper crates =================================== To use current git master code of the proxmox* helper crates, add:: git = "git://git.proxmox.com/git/proxmox" or:: path = "../proxmox/proxmox" to the proxmox dependency, and update the version to reflect the current, pre-release version number (e.g., "0.1.1-dev.1" instead of "0.1.0"). Local cargo config ================== This repository ships with a ``.cargo/config`` that replaces the crates.io registry with packaged crates located in ``/usr/share/cargo/registry``. A similar config is also applied building with dh_cargo. Cargo.lock needs to be deleted when switching between packaged crates and crates.io, since the checksums are not compatible. To reference new dependencies (or updated versions) that are not yet packaged, the dependency needs to point directly to a path or git source (e.g., see example for proxmox crate above). Build ===== on Debian Buster Setup: 1. # echo 'deb http://download.proxmox.com/debian/devel/ buster main' >> /etc/apt/sources.list.d/proxmox-devel.list 2. # sudo wget http://download.proxmox.com/debian/proxmox-ve-release-6.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-6.x.gpg 3. # sudo apt update 4. # sudo apt install devscripts debcargo clang 5. # git clone git://git.proxmox.com/git/proxmox-backup.git 6. # sudo mk-build-deps -ir Note: 2. may be skipped if you already added the PVE or PBS package repository You are now able to build using the Makefile or cargo itself. Design Notes ============ Here are some random thought about the software design (unless I find a better place). Large chunk sizes ----------------- It is important to notice that large chunk sizes are crucial for performance. We have a multi-user system, where different people can do different operations on a datastore at the same time, and most operation involves reading a series of chunks. So what is the maximal theoretical speed we can get when reading a series of chunks? Reading a chunk sequence need the following steps: - seek to the first chunk start location - read the chunk data - seek to the first chunk start location - read the chunk data - ... Lets use the following disk performance metrics: :AST: Average Seek Time (second) :MRS: Maximum sequential Read Speed (bytes/second) :ACS: Average Chunk Size (bytes) The maximum performance you can get is:: MAX(ACS) = ACS /(AST + ACS/MRS) Please note that chunk data is likely to be sequential arranged on disk, but this it is sort of a best case assumption. For a typical rotational disk, we assume the following values:: AST: 10ms MRS: 170MB/s MAX(4MB) = 115.37 MB/s MAX(1MB) = 61.85 MB/s; MAX(64KB) = 6.02 MB/s; MAX(4KB) = 0.39 MB/s; MAX(1KB) = 0.10 MB/s; Modern SSD are much faster, lets assume the following:: max IOPS: 20000 => AST = 0.00005 MRS: 500Mb/s MAX(4MB) = 474 MB/s MAX(1MB) = 465 MB/s; MAX(64KB) = 354 MB/s; MAX(4KB) = 67 MB/s; MAX(1KB) = 18 MB/s; Also, the average chunk directly relates to the number of chunks produced by a backup:: CHUNK_COUNT = BACKUP_SIZE / ACS Here are some staticics from my developer worstation:: Disk Usage: 65 GB Directories: 58971 Files: 726314 Files < 64KB: 617541 As you see, there are really many small files. If we would do file level deduplication, i.e. generate one chunk per file, we end up with more than 700000 chunks. Instead, our current algorithm only produce large chunks with an average chunks size of 4MB. With above data, this produce about 15000 chunks (factor 50 less chunks).