tyler/proxmox-backup
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
6,100 Commits 2 Branches 54 Tags
1c8efc00626219f1b44e088b166f7f988582d719
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Dietmar Maurer 1c8efc0062 cleanup: move BasicRealmInfo to pbs-api-types 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2022-03-07 08:06:55 +01:00
Dietmar Maurer d6644e29fe move src/shared_rate_limiter.rs to src/tools/shared_rate_limiter.rs 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2022-02-14 14:57:56 +01:00
Dietmar Maurer 260147bd73 ParallelHandler: avoid re-export (cleanup) 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2022-02-14 14:12:39 +01:00
Dietmar Maurer e705b3057f rename cached_traffic_control.rs to traffic_control_cache.rs, improve dev docs 
Keep things inside crate::traffic_control_cache (do not pollute root namespace).

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2022-02-14 13:45:44 +01:00
Dietmar Maurer 192ece47fb rrd_cache: add developer docs 
and make RRD_CACHE private (please use get_rrd_cache instead).

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2022-02-14 12:07:10 +01:00
Dietmar MaurerandThomas Lamprecht d91a0f9fc9 Set MMAP_THRESHOLD to a fixed value (128K) 
glibc's malloc has a misguided heuristic to detect transient allocations that
will just result in allocation sizes below 32 MiB never using mmap.

That it turn means that those relatively big allocations are on the heap where
cleanup and returning memory to the OS is harder to do and easier to be blocked
by long living, small allocations at the top (end) of the heap.

Observing the malloc size distribution in a file-level backup run:

@size:
[0]                   14 |                                                    |
[1]                25214 |@@@@@                                               |
[2, 4)              9090 |@                                                   |
[4, 8)             12987 |@@                                                  |
[8, 16)            93453 |@@@@@@@@@@@@@@@@@@@@                                |
[16, 32)           30255 |@@@@@@                                              |
[32, 64)          237445 |@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@|
[64, 128)          32692 |@@@@@@@                                             |
[128, 256)         22296 |@@@@                                                |
[256, 512)         16177 |@@@                                                 |
[512, 1K)           5139 |@                                                   |
[1K, 2K)            3352 |                                                    |
[2K, 4K)             214 |                                                    |
[4K, 8K)            1568 |                                                    |
[8K, 16K)             95 |                                                    |
[16K, 32K)          3457 |                                                    |
[32K, 64K)          3175 |                                                    |
[64K, 128K)          161 |                                                    |
[128K, 256K)         453 |                                                    |
[256K, 512K)          93 |                                                    |
[512K, 1M)            74 |                                                    |
[1M, 2M)             774 |                                                    |
[2M, 4M)             319 |                                                    |
[4M, 8M)             700 |                                                    |
[8M, 16M)             93 |                                                    |
[16M, 32M)            18 |                                                    |

We see that all allocations will be on the heap, and that while most
allocations are small, the relatively few big ones will still make up most of
the RSS and if blocked from being released back to the OS result in much higher
peak and average usage for the program than actually required.

Avoiding the "dynamic" mmap-threshold increasement algorithm and fixing it at
the original default of 128 KiB reduces RSS size by factor 10-20 when running
backups. As with memory mappings other mappings or the heap can never block
freeing the memory fully back to the OS.

But, the drawback of using mmap is more wasted space for unaligned or small
allocation sizes, and the fact that the kernel allegedly zeros out the data
before giving it to user space. The former doesn't really matter for us when
using it only for allocations bigger than 128 KiB, and the latter is a
trade-off, using 10 to 20 times less memory brings its own performance
improvement possibilities for the whole system after all ;-)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 [ Thomas: added to comment & commit message + extra-empty-line fixes ]
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2022-01-26 14:10:54 +01:00
Dietmar Maurer b44483a853 datastore status: do not count empty groups 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2022-01-07 08:40:22 +01:00
Dietmar Maurer d4877712f8 pbs-client: avoid mut self in http_client methods. 
It is not necessary, so avoid it. The client can now be used
with multiple threads (without using a Mutex).

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-12-04 14:44:05 +01:00
Dietmar Maurer 0e994eb938 pbs-api-types: remove proxmox-sys dependency for target wasm 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-12-01 09:49:52 +01:00
Dietmar Maurer 1a211f0d96 pbs-api-types: remove openssl dependency for target wasm 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-12-01 09:28:47 +01:00
Dietmar Maurer f7fde5c81b pbs-api-types: remove libc dependency 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-12-01 09:10:25 +01:00
Dietmar Maurer af5a55509d pbs-api-types: removbe usused nix dependency 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-12-01 09:08:25 +01:00
Dietmar Maurer 0fee4ff2a4 pbs-tape: do not depend on pbs-tools 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 13:17:58 +01:00
Dietmar Maurer 3dcea3ce33 fix typo in comment 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 13:15:35 +01:00
Dietmar Maurer 726b9d4469 use proxmox-sys 0.2.1 and proxmox-io 1.0.1 
And remove unused code from pbs-tools.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 12:30:03 +01:00
Dietmar Maurer 577095e2f7 move pbs-tools/src/percent_encoding.rs to pbs-api-types/src/percent_encoding.rs 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 11:48:52 +01:00
Dietmar Maurer f35e187f16 fix StdChannelWriter usage 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 11:27:20 +01:00
Dietmar Maurer e2b12ce988 StdChannelWriter: avoid using anyhow::Error 
Use a generic implementation to allow different error types.
 2021-11-25 11:14:56 +01:00
Dietmar Maurer 92ef0b56d8 move pbs-tools/src/str.rs to pbs-client/src/pxar/create.rs 
Code is only used there.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 10:43:22 +01:00
Dietmar Maurer 8a8a1850d0 remove trait BufferedRead from pbs-tools/src/io.rs 
We do not need it.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 09:45:47 +01:00
Dietmar Maurer fddb9bcc3e remove pbs-tools/src/sys.rs 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 09:01:29 +01:00
Dietmar Maurer 0df179c2b4 remove pbs-tools/src/cli.rs 
Code is only used once in src/bin/proxmox_backup_debug/inspect.rs, so
move it into that file.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-25 08:33:10 +01:00
Dietmar Maurer 689ed51397 openid_login: improve error message for disabled users. 2021-11-25 07:29:33 +01:00
Dietmar Maurer 3c56335d7b update debian/control 2021-11-25 06:49:26 +01:00
Dietmar Maurer 9eb58647c1 pbs-datastore: use hex::serde feature 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-24 13:06:14 +01:00
Dietmar Maurer 0ff214bedd debian/control: add librust-proxmox-serde 2021-11-24 10:58:01 +01:00
Dietmar Maurer 25877d05ac update to proxmox-sys 0.2 crate 
- imported pbs-api-types/src/common_regex.rs from old proxmox crate
- use hex crate to generate/parse hex digest
- remove all reference to proxmox crate (use proxmox-sys and
  proxmox-serde instead)

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-24 10:32:27 +01:00
Dietmar Maurer c1e6efa8e1 sync job: correctly apply rate limit 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-23 09:42:18 +01:00
Dietmar Maurer e201104d0b docs: update traffic control docs (use HumanBytes) 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-22 09:07:05 +01:00
Dietmar Maurer d63db0863d proxmox-backup-manager traffic: render data human readable 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-22 09:07:05 +01:00
Dietmar Maurer 7a36833103 fix sync job regression test (add RateLimitConfig) 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-22 08:29:43 +01:00
Dietmar Maurer ca6e66aa5a Fingerprint: add new signature method 
commit c42a54795d introcuded a bug by
using fp.to_string(). Replace this with fp.signature() which correctly
returns the full fingerprint instead of the short version.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-22 08:29:43 +01:00
Dietmar Maurer 2d5287fbbc use RateLimitConfig for HttpClient and pull 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-22 07:49:41 +01:00
Dietmar Maurer 6eb756bcab sync-job: add rate limit 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-22 07:49:41 +01:00
Dietmar Maurer 5647219049 pbs-api-types: split out type RateLimitConfig 2021-11-22 07:49:41 +01:00
Dietmar Maurer 193ec30c2b fix proxmox-backup-manager sync-job list 
Property is called 'group-filter' (not 'groups').
 2021-11-21 09:46:46 +01:00
Dietmar Maurer c94723062c pbs-api-types: fix HumanByte::auto_scale 2021-11-21 09:13:02 +01:00
Dietmar MaurerandThomas Lamprecht 118515dbd0 use HumanByte for traffic-control config 
Signed-off-by: Thomas Lamprecht <t.lamprecht@proxmox.com>
 2021-11-20 19:35:24 +01:00
Dietmar Maurer 92a8f0bc82 depend on proxmox-async 0.2 2021-11-20 17:14:02 +01:00
Dietmar Maurer bf298a16ef proxmox-rest-server: remove pbs-tools dependency 2021-11-19 18:06:54 +01:00
Dietmar Maurer 9a1b24b6b1 use new proxmox-async crate 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-19 18:03:22 +01:00
Dietmar Maurer 6f6df501a0 fix debian/control: include librust-proxmox-sys 2021-11-19 11:35:28 +01:00
Dietmar Maurer d5790a9f27 use new proxmox-sys crate 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-19 11:06:35 +01:00
Dietmar Maurer 860eaec58f use proxmox::tools::fd::fd_change_cloexec from proxmox 0.15.3 
Depend on proxmox 0.15.3

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-18 13:43:41 +01:00
Dietmar Maurer 26e949d5fe traffic-control api: return current traffic with config 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-18 12:24:33 +01:00
Dietmar Maurer 10beed1199 openid: allow to configure scopes, prompt, ACRs and arbitrary username-claim values 
- no longer set prompt to 'login' (makes auto-login possible)
- new prompt configuration
- allow arbitrary username-claim values

Depend on proxmox-openid 0.9.0.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-18 11:20:55 +01:00
Dietmar Maurer 15cc41b6cb proxmox-systemd: remove crate, use new proxmox-time 1.1.0 instead 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-17 13:07:51 +01:00
Dietmar Maurer c42a54795d move fingerprint helpers from pbs-tools to pbs-api-types 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-17 07:07:40 +01:00
Dietmar Maurer 96ec3801a9 docs: add traffic control section 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-15 13:43:11 +01:00
Dietmar Maurer c4707d0c1d depend on proxmox-shared-memory 0.1.1 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-15 11:35:52 +01:00
Dietmar Maurer 24f9af9e0f add missing file from previous commit 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-14 18:49:29 +01:00
Dietmar Maurer a0172d766b traffic-controls: add API/CLI to show current traffic 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-14 17:21:45 +01:00
Dietmar Maurer 09f999337a update to proxmox-http 0.5.4 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-14 08:27:45 +01:00
Dietmar Maurer e3eb062c09 cached_traffic_control: fix regression tests 
Avoid using shared memory in tests because of permission problems.
 2021-11-14 08:05:40 +01:00
Dietmar Maurer de21d4efdc implement rate limiter in shared memory 
This kind of rate limiter can be used among several processes (as long
as all set the same rate/burst).
 2021-11-13 17:49:38 +01:00
Dietmar Maurer d5f58006d3 cached_traffic_control: use ShareableRateLimit trait object 2021-11-13 17:49:38 +01:00
Dietmar Maurer cb80ffc1de pbs-config: use new SharedMemory helpers from proxmox-shared-memory crate 
depend on proxmox-shared-memory crate.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-13 17:49:38 +01:00
Dietmar Maurer 9e7132c0b3 bump version to 2.0.14-1 2021-11-12 08:12:18 +01:00
Dietmar Maurer bf013be1c4 create /var/lib/proxmox-bnackup at server startup 
This was missing in previous patch...

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-12 08:12:18 +01:00
Dietmar Maurer b935209584 fix directory permission problems 
By carefully setting options on all create_path() calls,
and by creating "/var/lib/proxmox-backup" at api server startup.
 2021-11-12 07:29:18 +01:00
Dietmar Maurer efd4ddc17b debian/control: depend on librust-cidr-dev 2021-11-10 12:23:16 +01:00
Dietmar Maurer e511e0e553 proxmox-backup-proxy: implement traffic control 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer 610150a4b4 implement a traffic control cache for fast rate control limiter lockups 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer 485b2438ac traffic_control: use Memcom to track. config versions 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer bfd12e871f Add traffic control configuration config with API 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer 0c136bfab1 DailyDuration: implement time_match() 2021-11-10 10:15:40 +01:00
Dietmar Maurer 245e2aea23 New DailyDuration type with nom parser 
We will use this to specify timesframes for network rate limits (only
apply limite when inside the time frame).

Note: This is not systemd related, but we can reuse some of the parser
method.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer b9d588ffde implement Servive for RateLimitedStream 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer e4bc3e0e8d proxmox-backup-client: add rate/burst parameter to backup CLI 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:40 +01:00
Dietmar Maurer 2419dc0de9 pbs-client: add option to use the new RateLimiter 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-10 10:15:32 +01:00
Dietmar Maurer 68fd9ca6d6 openid_login: vertify that firstname, lastname and email fits our schema definitions 
If not, we do not copy the values to our user.cfg.
 2021-11-10 06:48:40 +01:00
Dietmar Maurer 4beb7d2dbe correctly lock remote config 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-11-06 17:35:10 +01:00
Dietmar Maurer 7a356a748a bump version to 2.0.13-1 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-21 08:36:41 +02:00
Dietmar Maurer 1c402740a2 update dedian/control 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-21 07:59:45 +02:00
Dietmar Maurer e0a19d3313 use new fsync parameter to replace_file and atomic_open_or_create 
Depend on proxmox 0.15.0 and proxmox-openid 0.8.1

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-21 07:28:32 +02:00
Dietmar Maurer d6473f5359 proxmox-rrd: use fsync instead of syncfs 
syncfs can sync unrelated data, and we do not want that.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-20 11:46:59 +02:00
Dietmar Maurer f5f9ec81d2 proxmox-rrd: fix regression tests 2021-10-19 18:41:03 +02:00
Dietmar Maurer fea950155f proxmox-rrd: improve dev docs 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer ef2944bc24 proxmox-rrd: cleanup - impl FromStr for JournalEntry 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer 934c8724e2 proxmox-rrd: add option to avoid page cache for load/save 
use fadvice(.., POSIX_FADV_DONTNEED) for RRD files. We read those files only once,
and always rewrite them.

Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer 98eb435d90 proxmox-rrd: use syncfs after writing rrd files 
Signed-off-by: Dietmar Maurer <dietmar@proxmox.com>
 2021-10-19 11:17:09 +02:00
Dietmar Maurer 7f381a6246 proxmox-rrd: use fine grained locking in commit_journal_impl 
Aquire the rrd_map lock for each file (else we block access for a long time)
 2021-10-18 14:55:47 +02:00
Dietmar Maurer c17fbbbc07 proxmox-rrd: log all errors from apply_and_commit_journal_thread (but only once) 2021-10-18 11:57:19 +02:00
Dietmar Maurer d26865c52c proxmox-rrd: cleanup list_old_journals 2021-10-18 10:00:58 +02:00
Dietmar Maurer 2b05008a11 proxmox-rrd: cleanup - use struct instead of tuple 2021-10-16 12:45:03 +02:00
Dietmar Maurer 45700e2ecf proxmox-rrd: move RRDMap into extra file 2021-10-16 12:45:03 +02:00
Dietmar Maurer f84304235b proxmox-rrd: move JournalState into extra file 2021-10-16 12:45:03 +02:00
Dietmar Maurer 0ca41155b2 proxmox-rrd: implement non blocking journal 
Do not block while applying the journal.
 2021-10-16 12:45:03 +02:00
Dietmar Maurer a291ab59ba proxmox-rrd: rename RRDCacheState to JournalState 2021-10-15 09:35:44 +02:00
Dietmar Maurer fce7cd0d36 proxmox-rrd: avoild blocking readers while applying the journal 
By using and extra RwLock<RRDMap> on the rrd data.
 2021-10-15 09:22:07 +02:00
Dietmar Maurer 658357c5a8 proxmox-rrd: log journal apply/flush times, split apply and flush 
We need to apply the journal only once.
 2021-10-15 07:16:41 +02:00
Dietmar Maurer 7484fce24d proxmox-rrd: cleanup - use slot_end_time() 2021-10-14 16:29:00 +02:00
Dietmar Maurer f28a713e2b proxmox-rrd: cleanup - use staturating_add instead of if/else 2021-10-14 16:10:55 +02:00
Dietmar Maurer a9017805b7 proxmox-rrd: improve dev docs 2021-10-14 11:53:54 +02:00
Dietmar Maurer 2e3f94e12f proxmox-rrd: make rrd load callback configurable 2021-10-14 11:41:26 +02:00
Dietmar Maurer d531c7ae61 proxmox-rrd: add more regression tests 2021-10-14 10:55:12 +02:00
Dietmar Maurer 7df1580fa6 proxmox-rrd: add regression tests and two minor fixes 2021-10-14 10:17:07 +02:00
Dietmar Maurer 58f70bccbb proxmox-rrd: pass time and value to update function 2021-10-14 08:12:56 +02:00
Dietmar Maurer fae4f6c509 cleanup: move rrd cache related code into extra file 2021-10-14 07:57:27 +02:00
Dietmar Maurer ddafb28572 proxmox-rrd: add some integration tests (file format tests) 2021-10-13 18:21:23 +02:00