Dotaz: vsetko zozeru buffery a system swapuje

[asdqweqe]

Zdravim, trapim sa s problemom kedy mi buffer zozerie vsetku RAM a system zacne snapovat. Ma niekto skusenost?

# free -m
             total       used       free     shared    buffers     cached
Mem:         20480      20476          3          0          0         41
-/+ buffers/cache:      20434         45
Swap:          512        443         68

Odpovědi

[Fuky]

Co konkrétně Ti žere "kernel slab cache"? Jaký používáš filesystém?

# slabtop -s c

[asdqweqe]

Je to OpenVZ so ZFS

# slabtop -s c
fopen /proc/slabinfo: No such file or directory

[Fuky]

OK, takže RAM Ti podle mě žere právě ZFS. Na jakou hodnotu máš nastaveno "zfs_arc_max"?

Přidej výpis /proc/spl/kstat/zfs/arcstats a /proc/spl/kmem/slab.

Mrkni na následující odkazy:

ZFSonLinuxMemoryWhere
SlabinfoSlabMerging
ZFSonLinuxWeakAreas
arc size vs memory usage

The have been many discussions about the arc size, but not about what this means for the actual memory usage of ZFS. If you thought this was the same, or a fixed relationship: WRONG! The amount of memory needed is increased in steps for 3 different reasons.

First there is more memory allocated in the slab then the arc cache. I'm not sure what the relationship between the slab objects and arc is, but I would guess different zfs caches use the slab as well.

The second increase is caused by slab overhead. The slab needs to manage the objects, and the overhead for 512B objects can be as much as 100%. The overhead is not fixed, but depends on the object size.

The third increase is caused by slab fragmentation, because slabs can only be released when all objects on the slab are released. So this is the difference between the alloced slabs and the required slabs.

So I've written a slabstats script which shows the overhead introduced by each step, and the factor between the arc size and the full slab size. In my test the memory used was between 1.22 and 3.88!!! times the arc size, and stabilized around 2.

[Michal Kubeček]

Jak jste přišel na to, že "buffer zozerie vsetku RAM"? Jak vypadá obsah /proc/meminfo a /proc/zoneinfo?

[asdqweqe]

# cat /proc/meminfo
MemTotal:       20971520 kB
MemFree:           36316 kB
Cached:            65996 kB
Buffers:               0 kB
Active:           617924 kB
Inactive:         197820 kB
Active(anon):     585672 kB
Inactive(anon):   178532 kB
Active(file):      32252 kB
Inactive(file):    19288 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:        524288 kB
SwapFree:         448992 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        764204 kB
Shmem:             14456 kB
Slab:           20119444 kB
SReclaimable:   20094688 kB
SUnreclaim:        24756 kB
# cat /proc/zoneinfo
cat: /proc/zoneinfo: No such file or directory

[Michal Kubeček]

Takže jde o slab, ale reclaimable, takže diagnóza "systém swapuje" nejspíš nebude úplně správná. Spíš bych tipoval, že zvýšená disková aktivita bude důsledkem toho, že vám zbývá hodně málo paměti pro page cache. V každém případě by to chtělo zjistit, co konkrétně to spotřebuje tolik paměti; zkuste postupovat podle rad výše.

[Michal2]

Podle vseho ti pamet zere ZFS (hod sem vystup cat /proc/spl/kstat/zfs/arcstatscat ).

 

Nepouzivas cirou nahodou deduplikaci? Pokud ano, tak uz dale problem hledat nemusis pac je zrejmy

[asdqweqe]

Toto je vystup z nasho OpenVZ servera.

No netusim ako to obmedzit na urovni VPS :-/

# cat /proc/spl/kstat/zfs/arcstats
6 1 0x01 91 4368 20349008856 11810986092908077
name                            type data
hits                            4    1443626517411
misses                          4    1527227514
demand_data_hits                4    1203502394246
demand_data_misses              4    305169580
demand_metadata_hits            4    7243728585
demand_metadata_misses          4    1071662173
prefetch_data_hits              4    231527868245
prefetch_data_misses            4    108133799
prefetch_metadata_hits          4    1352526338
prefetch_metadata_misses        4    42261962
mru_hits                        4    395875804
mru_ghost_hits                  4    88952407
mfu_hits                        4    1298690331029
mfu_ghost_hits                  4    89323812
deleted                         4    137898097
mutex_miss                      4    28525
evict_skip                      4    259520735
evict_not_enough                4    234463
evict_l2_cached                 4    18384265695744
evict_l2_eligible               4    1502941855744
evict_l2_ineligible             4    644783208448
evict_l2_skip                   4    0
hash_elements                   4    10533706
hash_elements_max               4    10607940
hash_collisions                 4    207769396
hash_chains                     4    1344057
hash_chain_max                  4    7
p                               4    8705097216
c                               4    34359738368
c_min                           4    30064771072
c_max                           4    34359738368
size                            4    34363694696
hdr_size                        4    1330915328
data_size                       4    15384919552
metadata_size                   4    6735291904
other_size                      4    10230584896
anon_size                       4    82886656
anon_evictable_data             4    0
anon_evictable_metadata         4    0
mru_size                        4    2621831680
mru_evictable_data              4    410124288
mru_evictable_metadata          4    3414528
mru_ghost_size                  4    31734104576
mru_ghost_evictable_data        4    1910576128
mru_ghost_evictable_metadata    4    29823528448
mfu_size                        4    19415493120
mfu_evictable_data              4    14915006464
mfu_evictable_metadata          4    236452352
mfu_ghost_size                  4    2608713728
mfu_ghost_evictable_data        4    116488704
mfu_ghost_evictable_metadata    4    2492225024
l2_hits                         4    119369318
l2_misses                       4    1407857959
l2_feeds                        4    11838519
l2_rw_clash                     4    29
l2_read_bytes                   4    992057985024
l2_write_bytes                  4    6782502233600
l2_writes_sent                  4    11707457
l2_writes_done                  4    11707457
l2_writes_error                 4    0
l2_writes_lock_retry            4    4432
l2_evict_lock_retry             4    1117
l2_evict_reading                4    41
l2_evict_l1cached               4    5877717
l2_free_on_write                4    58850
l2_cdata_free_on_write          4    96718
l2_abort_lowmem                 4    0
l2_cksum_bad                    4    0
l2_io_error                     4    0
l2_size                         4    548897384448
l2_asize                        4    260944846848
l2_hdr_size                     4    681983016
l2_compress_successes           4    232226027
l2_compress_zeros               4    0
l2_compress_failures            4    11586358
memory_throttle_count           4    0
duplicate_buffers               4    0
duplicate_buffers_size          4    0
duplicate_reads                 4    9
memory_direct_count             4    0
memory_indirect_count           4    0
arc_no_grow                     4    0
arc_tempreserve                 4    0
arc_loaned_bytes                4    0
arc_prune                       4    0
arc_meta_used                   4    18978775144
arc_meta_limit                  4    34359738368
arc_meta_max                    4    21208047664
arc_meta_min                    4    16777216
arc_need_free                   4    0
arc_sys_free                    4    4223533056

[Michal2]

Takze polozka size = ~ 32 GB coz je pamet vyuzita pres ZFS ARC (vetsina z toho podle parametru c je MRU cache (most recenty used)). Takze bude chybne nastaveny parametr zfs_arc_max

[asdqweqe]

Cely system ma 128GB RAM, kontajner kde mam tento problem ma vyclenenych len 20GB. Je spravne, ak vidim reziu filesystemu v kontajneri?

[asdqweqe]

Ak by to tak bolo, tak by som arcsize musel nastavit na 1MB ... aby to nijako neovplyvnovalo jednotlive VPSky ktore tam mame. Ci je ina cesta?