anyone?
--- Begin Message ---
Hi,

<pre>
> ::status
debugging live kernel (64-bit) on mk-archive-1
operating system: 5.11 snv_123 (i86pc)
> ::system
set noexec_user_stack_log=0x1 [0t1]
set noexec_user_stack=0x1 [0t1]
set snooping=0x1 [0t1]
set zfs:zfs_arc_max=0x280000000 [0t10737418240]
> 
> ::memstat
Page Summary                Pages                MB  %Tot
------------     ----------------  ----------------  ----
Kernel                    1843301              7200   44%
ZFS File Data             1651701              6451   40%
Anon                       202473               790    5%
Exec and libs                 934                 3    0%
Page cache                   4369                17    0%
Free (cachelist)             2084                 8    0%
Free (freelist)            462699              1807   11%

Total                     4167561             16279
Physical                  4167560             16279
> 

We are experiencing out-of-memory issues during the night on this server and 
our apps do not use much memory but are during lots of disk and network IO. The 
server is x4500 with 16GB of memory, the filesystem is of course ZFS. I'm 
concerned with the kernel size here.
I'm getting applications crashing and errors like:

WARNING: Sorry, no swap space to grow stack for pid 5991 (cron)
WARNING: Sorry, no swap space to grow stack for pid 6092 (mv)
WARNING: Sorry, no swap space to grow stack for pid 18481 (ggrep)
WARNING: Sorry, no swap space to grow stack for pid 18562 (cron)
WARNING: /tmp: File system full, swap space limit exceeded
WARNING: /etc/svc/volatile: File system full, swap space limit exceeded

And no, we are not filling /tmp and besides /tmp is limited to 1GB anyway.

Couple of highlights from ::kmastat
[...]
cache                        buf    buf    buf    memory     alloc alloc 
name                        size in use  total    in use   succeed  fail 
------------------------- ------ ------ ------ ---------- --------- ----- 
kmem_magazine_143           1152  53621  55065  75182080B  15082603  
[b]5195[/b] 
[...]
vmem                         memory     memory    memory     alloc alloc 
name                         in use      total    import   succeed  fail 
------------------------- ---------- ----------- ---------- --------- ----- 
heap                      [b]13564526592B[/b] 1077569126400B         0B  
34565103     0 
[...]
    kmem_metadata         586997760B  623640576B 623640576B    912072     0 
        kmem_msb          539336704B  539336704B 539336704B    864680  
[b]5198[/b] 
[...]
    kmem_firewall_va      141729792B  141729792B 141729792B    578915     0 
        kmem_firewall             0B          0B         0B         0     0 
        kmem_oversize     141532041B  141729792B 141729792B    578904    
[b]11[/b] 
[...]
    kmem_va               [b]12148248576B[/b] 12148248576B 12148248576B  
31994594     0 
        kmem_default      5738831872B 5738831872B 5738831872B 187061764     0 
[...]
zfs_file_data             6734151680B 17070817280B         0B  90214511     0 
    zfs_file_data_buf     6734151680B 6734151680B 6734151680B  90214511     0 
[...]

Is it a memory fragmentation or something else?


A full ::kmastat output

> ::kmastat
cache                        buf    buf    buf    memory     alloc alloc 
name                        size in use  total    in use   succeed  fail 
------------------------- ------ ------ ------ ---------- --------- ----- 
kmem_magazine_1               16   2893  32379    528384B  94677416     1 
kmem_magazine_3               32   5111   5875    192512B  11584154     0 
kmem_magazine_7               64  15917  17670   1167360B  24614491     0 
kmem_magazine_15             128   4478   4619    610304B   6863634     0 
kmem_magazine_31             256   7823  12150   3317760B   3485723     0 
kmem_magazine_47             384   2796   3280   1343488B   1616387     0 
kmem_magazine_63             512    550   3353   1961984B   1256504     0 
kmem_magazine_95             768  13035  18095  14823424B  18231914     0 
kmem_magazine_143           1152  53621  55065  75182080B  15082603  5195 
kmem_slab_cache               72 822572 1785630 132980736B 275428877     0 
kmem_bufctl_cache             24 6384507 12425301 304754688B 535280109     0 
kmem_bufctl_audit_cache      192      0      0         0B         0     0 
kmem_va_4096                4096 1091176 1998784 3892051968B  49062569     0 
kmem_va_8192                8192  17696  18896 154796032B   4835863     0 
kmem_va_12288              12288    451   3580  46923776B   9282713     0 
kmem_va_16384              16384  26040 182176 2984771584B 106103361     0 
kmem_va_20480              20480   5974   6984 152567808B   9298381     0 
kmem_va_24576              24576     94    365   9568256B   2731797     0 
kmem_va_28672              28672    270   1700  55705600B   9520293     0 
kmem_va_32768              32768    406    620  20316160B   2796821     0 
kmem_alloc_8                   8 203502 1192613   9711616B 3676794457     0 
kmem_alloc_16                 16  78408  80320   1310720B 218811234     0 
kmem_alloc_24                 24  85915  90681   2224128B 239528690     0 
kmem_alloc_32                 32  18962 130750   4284416B 2210322705     0 
kmem_alloc_40                 40  44414 198800   8142848B 1768939641     0 
kmem_alloc_48                 48  16698 151973   7499776B 3462768216     0 
kmem_alloc_56                 56  16698 231105  13332480B 2448936287     0 
kmem_alloc_64                 64 333602 867814  57331712B 3016865581     0 
kmem_alloc_80                 80 207559 242450  19861504B 1631016503     0 
kmem_alloc_96                 96  61986 826724  82591744B 976177061     0 
kmem_alloc_112               112 146379 777805  91025408B 3671178954     0 
kmem_alloc_128               128  99507 272614  36020224B 1947142266     0 
kmem_alloc_160               160 141768 380850  62398464B 1710806616     0 
kmem_alloc_192               192  34271 231800  47472640B 3578907209     0 
kmem_alloc_224               224   3966 211276  50905088B 2515626747     0 
kmem_alloc_256               256  59192 578595 157995008B 877537912     0 
kmem_alloc_320               320  35930  39648  13533184B 625241367     0 
kmem_alloc_384               384   1161   3310   1355776B 3466976598     0 
kmem_alloc_448               448     72   3016   1544192B 448832839     0 
kmem_alloc_512               512    713   5728   2932736B 1978293717     0 
kmem_alloc_640               640    915   1086    741376B  74092368     0 
kmem_alloc_768               768   1036   1350   1105920B 376107800     0 
kmem_alloc_896               896    410    576    524288B    325845     0 
kmem_alloc_1152             1152    794   1022   1196032B 1445274252     0 
kmem_alloc_1344             1344     82    105    143360B   2552385     0 
kmem_alloc_1600             1600    118    140    229376B     60380     0 
kmem_alloc_2048             2048    343    458    937984B   9129870     0 
kmem_alloc_2688             2688    516    660   1802240B   8478726     0 
kmem_alloc_4096             4096    872   3620  14827520B   3441002     0 
kmem_alloc_8192             8192    876    906   7421952B 830397470     0 
kmem_alloc_12288           12288      5     13    159744B  44093699     0 
kmem_alloc_16384           16384    108    114   1867776B    189645     0 
kmem_alloc_24576           24576     53     56   1376256B      4093     0 
kmem_alloc_32768           32768    347    370  12124160B   1777530     0 
kmem_alloc_40960           40960     42     47   1925120B    781665     0 
kmem_alloc_49152           49152     41     45   2211840B    153790     0 
kmem_alloc_57344           57344     38     38   2179072B      1018     0 
kmem_alloc_65536           65536     46     50   3276800B      2746     0 
kmem_alloc_73728           73728     31     48   3538944B  55873688     0 
kmem_alloc_81920           81920      0      7    573440B    524745     0 
kmem_alloc_90112           90112      1      8    720896B    289424     0 
kmem_alloc_98304           98304      1      9    884736B    619351     0 
kmem_alloc_106496         106496      1      9    958464B    381729     0 
kmem_alloc_114688         114688      0      3    344064B         5     0 
kmem_alloc_122880         122880      1      4    491520B         7     0 
kmem_alloc_131072         131072      4     10   1310720B      5821     0 
streams_mblk                  64  39941  58962   3895296B 2404698757     0 
streams_dblk_16              128    125   9889   1306624B 986159352     0 
streams_dblk_80              192    383   3740    765952B 3265025863     0 
streams_dblk_144             256      1   1545    421888B 3566823209     0 
streams_dblk_208             320    198    768    262144B  41844914     0 
streams_dblk_272             384     45    990    405504B 108177566     0 
streams_dblk_336             448      1    824    421888B 448053424     0 
streams_dblk_528             640      0    306    208896B  39279380     0 
streams_dblk_1040           1152      0    448    524288B  58025983     0 
streams_dblk_1488           1600      1     40     65536B 171309402     0 
streams_dblk_1936           2048      2    238    487424B  26176236     0 
streams_dblk_2576           2688      0    231    630784B 103020162     0 
streams_dblk_3856           3968      0    243    995328B 110212268     0 
streams_dblk_8192            112      0    837    110592B 1201359722     0 
streams_dblk_12048         12160      0     40    491520B  56494616     0 
streams_dblk_16384           112      0     93     12288B  35400768     0 
streams_dblk_20240         20352      0     19    389120B   7219243     0 
streams_dblk_24576           112      0     62      8192B   6517456     0 
streams_dblk_28432         28544      0     42   1204224B  49943955     0 
streams_dblk_32768           112      0    403     53248B 206894845     0 
streams_dblk_36624         36736      0     18    663552B  27476519     0 
streams_dblk_40960           112      0     62      8192B   3332238     0 
streams_dblk_44816         44928      0      7    315392B   2087433     0 
streams_dblk_49152           112      0     62      8192B   7648623     0 
streams_dblk_53008         53120      0     18    958464B  10459305     0 
streams_dblk_57344           112      0     62      8192B    855736     0 
streams_dblk_61200         61312      0      7    430080B    415443     0 
streams_dblk_65536           112      0     62      8192B    596854     0 
streams_dblk_69392         69504      0      7    487424B    377915     0 
streams_dblk_73728           112      0     62      8192B    327790     0 
streams_dblk_esb             112  24576  24645   3256320B 2406897958     0 
streams_fthdr                408      0      0         0B         0     0 
streams_ftblk                376      0      0         0B         0     0 
multidata                    248      0      0         0B         0     0 
multidata_pdslab            7112      0      0         0B         0     0 
multidata_pattbl              32      0      0         0B         0     0 
log_cons_cache                48      8     83      4096B     14626     0 
taskq_ent_cache               56   3132 110547   6377472B 1122638311     0 
taskq_cache                  232    410    425    102400B       518     0 
kmem_io_64G_128              128      0     31      4096B     36573     0 
kmem_io_64G_256              256      0     15      4096B        42     0 
kmem_io_64G_512              512     48     56     28672B     14328     0 
kmem_io_64G_1024            1024      0      0         0B         0     0 
kmem_io_64G_2048            2048     48     48     98304B        48     0 
kmem_io_64G_4096            4096  33792  33792 138412032B     33792     0 
kmem_io_4G_128               128      0     31      4096B        32     0 
kmem_io_4G_256               256      0     15      4096B        59     0 
kmem_io_4G_512               512     50     56     28672B        52     0 
kmem_io_4G_1024             1024      0     16     16384B     87279     0 
kmem_io_4G_2048             2048      0      0         0B         0     0 
kmem_io_4G_4096             4096      0      0         0B         0     0 
kmem_io_2G_128               128      0      0         0B         0     0 
kmem_io_2G_256               256      0      0         0B         0     0 
kmem_io_2G_512               512      0      0         0B         0     0 
kmem_io_2G_1024             1024      4      4      4096B         4     0 
kmem_io_2G_2048             2048      0      0         0B         0     0 
kmem_io_2G_4096             4096      0      0         0B         0     0 
kmem_io_16M_128              128      0      0         0B         0     0 
kmem_io_16M_256              256      0      0         0B         0     0 
kmem_io_16M_512              512      0      0         0B         0     0 
kmem_io_16M_1024            1024      0      0         0B         0     0 
kmem_io_16M_2048            2048      0      0         0B         0     0 
kmem_io_16M_4096            4096      0      0         0B         0     0 
id32_cache                    32      4    125      4096B       323     0 
bp_map_4096                 4096      0      0         0B         0     0 
bp_map_8192                 8192      0     16    131072B      2272     0 
bp_map_12288               12288      0     10    131072B        26     0 
bp_map_16384               16384      0      8    131072B        20     0 
bp_map_20480               20480      0      6    131072B        19     0 
bp_map_24576               24576      0      5    131072B        22     0 
bp_map_28672               28672      0      4    131072B        16     0 
bp_map_32768               32768      0      4    131072B        13     0 
htable_t                      72  14684  22110   1646592B  26749358     0 
hment_t                       64  24309 413540  27320320B 1463129974     0 
hat_t                        176     80    286     53248B   3389231     0 
HatHash                    16384      8     67   1097728B     92941     0 
HatVlpHash                  4096     72     98    401408B   4074911     0 
segkp_4096                  4096     75    192    786432B   2191202     0 
segkp_8192                  8192      0      0         0B         0     0 
segkp_12288                12288      0      0         0B         0     0 
segkp_16384                16384      0      0         0B         0     0 
segkp_20480                20480      0      0         0B         0     0 
umem_np_4096                4096      0     32    131072B      1259     0 
umem_np_8192                8192      0     16    131072B      1227     0 
umem_np_12288              12288      0     10    131072B      1200     0 
umem_np_16384              16384      0      8    131072B        27     0 
umem_np_20480              20480      0      0         0B         0     0 
umem_np_24576              24576      0      5    131072B        12     0 
umem_np_28672              28672      0      0         0B         0     0 
umem_np_32768              32768      0      4    131072B        26     0 
mod_hash_entries              24    268   1002     24576B    116163     0 
ipp_mod                      304      0      0         0B         0     0 
ipp_action                   368      0      0         0B         0     0 
ipp_packet                    64      0      0         0B         0     0 
seg_cache                     96   5440  11726   1171456B 152557707     0 
seg_pcache                   104      2     38      4096B        16     0 
fnode_cache                  176      6     40      8192B    179363     0 
pipe_cache                   320     42     72     24576B   1455248     0 
snode_cache                  152    613    806    126976B  16299434     0 
dv_node_cache                176   2728   2728    507904B      5551     0 
mac_impl_cache              1816      6      9     16384B         6     0 
mac_ring_cache               136      8     29      4096B         8     0 
flow_entry_cache            3800     23     27    110592B        74     0 
flow_tab_cache               216      6     18      4096B         6     0 
mac_soft_ring_cache          344     18     20      8192B        38     0 
mac_srs_cache               1552     20     20     32768B        28     0 
mac_bcast_grp_cache           80     13     50      4096B        45     0 
mac_client_impl_cache        880     10     18     16384B        14     0 
mac_promisc_impl_cache       112      0     70      8192B       145     0 
dls_link_cache               336      6     11      4096B        10     0 
dls_devnet_cache             320      6     12      4096B        10     0 
sdev_node_cache              256   5216   5235   1429504B    204208     0 
dev_info_node_cache          640    269    282    192512B      1492     0 
ndi_fm_entry_cache            32  33800  34625   1134592B 2118692606     0 
thread_cache                 880    275    639    581632B   2513593     0 
lwp_cache                   1424    267    456    700416B    276448     0 
turnstile_cache               64    978   1612    106496B   2062939     0 
tslabel_cache                 48      2     83      4096B         2     0 
cred_cache                   188    107    200     40960B    179898     0 
rctl_cache                    48   1164   3569    176128B  23649014     0 
rctl_val_cache                64   2313   7254    479232B  51752528     0 
task_cache                   128     50    124     16384B     14573     0 
kmem_defrag_cache            216      1     18      4096B         1     0 
kmem_move_cache               56      0  42813   2469888B   1692119     2 
rootnex_dmahdl              2536  44717  45808 117268480B 1614161685     0 
timeout_request              128      1     31      4096B         1     0 
cyclic_id_cache               72     13     55      4096B        83     0 
callout_cache0                80    512    527     69632B       512     0 
callout_lcache0               48    668    682     45056B       668     0 
dnlc_space_cache              24      0      0         0B         0     0 
vfs_cache                    208    400    475    102400B       503     0 
vn_cache                     208 208740 320715  87576576B  58342940     0 
vsk_anchor_cache              40    376    500     20480B       477     0 
file_cache                    56    545   1491     86016B 713744661     0 
stream_head_cache            376    311    510    208896B   3118317     0 
queue_cache                  656    626    960    655360B   3270629     0 
syncq_cache                  160     16     25      4096B        40     0 
qband_cache                   64      2     62      4096B         2     0 
linkinfo_cache                48      9     83      4096B        10     0 
ciputctrl_cache              256      0      0         0B         0     0 
serializer_cache              64     16    186     12288B     31508     0 
as_cache                     232     79    272     65536B   3389201     0 
marker_cache                 120      0     33      4096B    158038     0 
anon_cache                    48 222456 2177007 107433984B 333633635     0 
anonmap_cache                112   3709   7910    925696B  78118896     0 
segvn_cache                  168   5440  11109   1978368B 129319780     0 
segvn_szc_cache1            4096      0      0         0B         0     0 
flk_edges                     48      0     83      4096B       971     0 
fdb_cache                    104      0      0         0B         0     0 
timer_cache                  136      1     29      4096B         1     0 
vmu_bound_cache               56      0      0         0B         0     0 
vmu_object_cache              64      0      0         0B         0     0 
physio_buf_cache             248      0      0         0B         0     0 
kcf_sreq_cache                56      0      0         0B         0     0 
kcf_areq_cache               296      0      0         0B         0     0 
kcf_context_cache            112      0      0         0B         0     0 
clnt_clts_endpnt_cache        88      0      0         0B         0     0 
zio_cache                    720     10 175362 130596864B 3141858379     0 
zio_link_cache                48     11 206006  10166272B 3229509018     0 
zio_buf_512                  512 648858 1230088 629805056B 2762819417     0 
zio_data_buf_512             512     11    632    323584B 367260970     0 
zio_buf_1024                1024    643   1572   1609728B 1387975718     0 
zio_data_buf_1024           1024     32    620    634880B 149066577     0 
zio_buf_1536                1536      0    752   1155072B 568156130     0 
zio_data_buf_1536           1536     35    344    528384B  63623869     0 
zio_buf_2048                2048      1    350    716800B 427819133     0 
zio_data_buf_2048           2048     21    152    311296B  65024130     0 
zio_buf_2560                2560      0    400   1024000B 399819454     0 
zio_data_buf_2560           2560      7    256    655360B  56951172     0 
zio_buf_3072                3072      0    268    823296B 268085885     0 
zio_data_buf_3072           3072      4    132    405504B  42628956     0 
zio_buf_3584                3584      1    304   1089536B 346063882     0 
zio_data_buf_3584           3584     13    168    602112B  46018803     0 
zio_buf_4096                4096     12    303   1241088B 507330816     0 
zio_data_buf_4096           4096     16    145    593920B  49934923     0 
zio_buf_5120                5120      0    288   1474560B 574828044     0 
zio_data_buf_5120           5120      7    156    798720B  62559248     0 
zio_buf_6144                6144      0    222   1363968B 411191177     0 
zio_data_buf_6144           6144      5    138    847872B  37959588     0 
zio_buf_7168                7168      0    240   1720320B 450977597     0 
zio_data_buf_7168           7168      0     92    659456B  33700593     0 
zio_buf_8192                8192      1    137   1122304B 317837892     0 
zio_data_buf_8192           8192      1     57    466944B  24943776     0 
zio_buf_10240              10240      0    136   1392640B 390240085     0 
zio_data_buf_10240         10240      0     80    819200B  46621578     0 
zio_buf_12288              12288      0     76    933888B 171113724     0 
zio_data_buf_12288         12288      0     43    528384B  42181413     0 
zio_buf_14336              14336      0    116   1662976B 183992492     0 
zio_data_buf_14336         14336      0     58    831488B  32826942     0 
zio_buf_16384              16384  26327  26349 431702016B 1475077575     0 
zio_data_buf_16384         16384      1     31    507904B  34693533     0 
zio_buf_20480              20480      0     39    798720B 116000831     0 
zio_data_buf_20480         20480      4     26    532480B  60818986     0 
zio_buf_24576              24576      0     38    933888B  98005806     0 
zio_data_buf_24576         24576      0     30    737280B  53277735     0 
zio_buf_28672              28672      3     44   1261568B  73446392     0 
zio_data_buf_28672         28672      1     24    688128B  38261480     0 
zio_buf_32768              32768     27     41   1343488B 103823042     0 
zio_data_buf_32768         32768      0     12    393216B  43362649     0 
zio_buf_36864              36864      8     27    995328B  75619401     0 
zio_data_buf_36864         36864      1     15    552960B  47341359     0 
zio_buf_40960              40960      2     21    860160B  69059675     0 
zio_data_buf_40960         40960      0     13    532480B  44948125     0 
zio_buf_45056              45056      1     19    856064B  78396957     0 
zio_data_buf_45056         45056      0     14    630784B  37000784     0 
zio_buf_49152              49152      3     20    983040B  51364250     0 
zio_data_buf_49152         49152      1     13    638976B  31934929     0 
zio_buf_53248              53248      2     23   1224704B  53080531     0 
zio_data_buf_53248         53248      0     17    905216B  36220216     0 
zio_buf_57344              57344      4     19   1089536B  53196583     0 
zio_data_buf_57344         57344      1     11    630784B  38224559     0 
zio_buf_61440              61440      6     17   1044480B  80650687     0 
zio_data_buf_61440         61440      0     10    614400B  47464432     0 
zio_buf_65536              65536   7416   7420 486277120B 2448460314     0 
zio_data_buf_65536         65536      0      6    393216B  51806665     0 
zio_buf_69632              69632      7     15   1044480B  62545735     0 
zio_data_buf_69632         69632      0      5    348160B  45664181     0 
zio_buf_73728              73728      5     18   1327104B  69709139     0 
zio_data_buf_73728         73728      0      5    368640B  37959897     0 
zio_buf_77824              77824      3     12    933888B  41988589     0 
zio_data_buf_77824         77824      0      5    389120B  28089149     0 
zio_buf_81920              81920      4     10    819200B  43524251     0 
zio_data_buf_81920         81920      0      5    409600B  29834139     0 
zio_buf_86016              86016      1      8    688128B  36758851     0 
zio_data_buf_86016         86016      0      5    430080B  24980859     0 
zio_buf_90112              90112      1      9    811008B  45062366     0 
zio_data_buf_90112         90112      1      6    540672B  18674505     0 
zio_buf_94208              94208      0     11   1036288B  22491224     0 
zio_data_buf_94208         94208      0      4    376832B  12113007     0 
zio_buf_98304              98304      0     14   1376256B  18508939     0 
zio_data_buf_98304         98304      0      8    786432B   8855853     0 
zio_buf_102400            102400      0      6    614400B  24789623     0 
zio_data_buf_102400       102400      0      5    512000B   6756527     0 
zio_buf_106496            106496      1      6    638976B  19080215     0 
zio_data_buf_106496       106496      0      6    638976B   5434102     0 
zio_buf_110592            110592      0      6    663552B  14236411     0 
zio_data_buf_110592       110592      0      4    442368B   5337259     0 
zio_buf_114688            114688      0     51   5849088B 288894190     0 
zio_data_buf_114688       114688      0      3    344064B   5992812     0 
zio_buf_118784            118784      0     13   1544192B 297510866     0 
zio_data_buf_118784       118784      0      5    593920B    568498     0 
zio_buf_122880            122880      0      7    860160B  28435986     0 
zio_data_buf_122880       122880      0      7    860160B    541110     0 
zio_buf_126976            126976      0      7    888832B  72433991     0 
zio_data_buf_126976       126976      0      4    507904B    882991     0 
zio_buf_131072            131072     15     22   2883584B 585277791     0 
zio_data_buf_131072       131072  50634  51192 2414870528B 606950770     0 
dmu_buf_impl_t               192 722892 4563800 934666240B 1216126074     0 
dnode_t                      672 648742 2570598 1754861568B 3809376641     0 
arc_buf_hdr_t                168 861229 861419 153407488B 1278645589     0 
arc_buf_t                     48  83740 329178  16244736B 1833462145     0 
zil_lwb_cache                200      4    200     40960B     71080     0 
zfs_znode_cache              200 139405 257700  52776960B 722277815     0 
ip_minor_arena_sa_1            1     22    128       128B     30959     0 
ip_minor_arena_la_1            1    129    256       256B       901     0 
ip_conn_cache                504      3     16      8192B        24     0 
tcp_conn_cache              1832     42    104    212992B     54605     0 
udp_conn_cache              1224     22     39     53248B    737274     0 
rawip_conn_cache            1184      2     10     12288B        80     0 
rts_conn_cache               616      5      6      4096B         8     0 
ip_helper_stream_cache        32     51    186     12288B    756202     0 
ire_cache                    336    107    143     53248B     10943     0 
rt_entry                     168      2     23      4096B         8     0 
radix_mask                    32      1    125      4096B         1     0 
radix_node                   120      1     33      4096B         1     0 
ipsec_actions                 72      0     55      4096B      9135     0 
ipsec_selectors               80      0      0         0B         0     0 
ipsec_policy                  72      0      0         0B         0     0 
ipsec_info                   352      0     11      4096B     30297     0 
tcp_timercache                88    157    540     49152B     23298     0 
tcp_sack_info_cache           80     12    200     16384B     38574     0 
tcp_iphc_cache               120     42    264     32768B     53185     0 
squeue_cache                 168     10     20      4096B        10     0 
sctp_conn_cache             2312      1      7     16384B         1     0 
sctp_faddr_cache             176      0      0         0B         0     0 
sctp_set_cache                24      0      0         0B         0     0 
sctp_ftsn_set_cache           16      0      0         0B         0     0 
ire_gw_secattr_cache          32      0      0         0B         0     0 
socket_cache                 536     51    135     73728B    760935     0 
socktpi_cache                880      0      9      8192B         1     0 
socktpi_unix_cache           880      9    153    139264B     33413     0 
sock_sod_cache               648      0      0         0B         0     0 
drv_secobj_cache             296      0      0         0B         0     0 
dld_str_cache                296      6     26      8192B        79     0 
process_cache               3176     81    252    802816B   1943668     0 
exacct_object_cache           40      0      0         0B         0     0 
callout_cache1                80    491    496     65536B       491     0 
callout_lcache1               48    662    682     45056B       662     0 
fctl_cache                   112      0      0         0B         0     0 
kssl_cache                  1608      0      0         0B         0     0 
tl_cache                     432     23    171     77824B     33483     0 
keysock_1                      1      0      0         0B         0     0 
spdsock_1                      1      0     64        64B         1     0 
namefs_inodes_1                1     20     64        64B        20     0 
port_cache                    80      8     50      4096B         8     0 
softmac_cache                568      4      7      4096B         8     0 
softmac_upper_cache          232      0      0         0B         0     0 
idm_buf_cache                240      0      0         0B         0     0 
idm_task_cache              2424      0      0         0B         0     0 
idm_tx_pdu_cache             400      0      0         0B         0     0 
idm_rx_pdu_cache             449      0      0         0B         0     0 
idm_128k_buf_cache        131072      0      0         0B         0     0 
aggr_port_cache              960      4      4      4096B         4     0 
aggr_grp_cache              6016      2      2     12288B         2     0 
dtrace_state_cache         16384      0      7    114688B        37     0 
authkern_cache                72      0      0         0B         0     0 
authnone_cache                72      0      0         0B         0     0 
authloopback_cache            72      0      0         0B         0     0 
authdes_cache_handle          80      0      0         0B         0     0 
rnode_cache                  656      0      0         0B         0     0 
nfs_access_cache              56      0      0         0B         0     0 
client_handle_cache           32      0      0         0B         0     0 
rnode4_cache                 968      0      0         0B         0     0 
svnode_cache                  40      0      0         0B         0     0 
nfs4_access_cache             56      0      0         0B         0     0 
client_handle4_cache          32      0      0         0B         0     0 
nfs4_ace4vals_cache           48      0      0         0B         0     0 
nfs4_ace4_list_cache         264      0      0         0B         0     0 
NFS_idmap_cache               48      0      0         0B         0     0 
Hex0xffffff04e12f2648_minor_1      1      0      0         0B         0     0 
Hex0xffffff04e43ed588_minor_1      1      0      0         0B         0     0 
lnode_cache                   32      1    125      4096B         1     0 
pty_map                       64     10     62      4096B       102     0 
ufs_inode_cache              368      0      0         0B         0     0 
directio_buf_cache           272      0      0         0B         0     0 
lufs_save                     24      0      0         0B         0     0 
lufs_bufs                    256      0      0         0B         0     0 
lufs_mapentry_cache          112      0      0         0B         0     0 
sppptun_map                  440      0      0         0B         0     0 
lm_vnode                     184      0      0         0B         0     0 
lm_xprt                       32      0      0         0B         0     0 
lm_sysid                     160      0      0         0B         0     0 
lm_client                    128      0      0         0B         0     0 
lm_async                      32      0      0         0B         0     0 
lm_sleep                      96      0      0         0B         0     0 
lm_config                     80      2     50      4096B         2     0 
crypto_session_cache         104      0      0         0B         0     0 
vnic_cache                   112      0      0         0B         0     0 
fcsm_job_cache               104      0      0         0B         0     0 
smb_mbc_cache                 56      0      0         0B         0     0 
smb_node_cache               616      0      0         0B         0     0 
smb_txreq                  66592      0      0         0B         0     0 
------------------------- ------ ------ ------ ---------- --------- ----- 
Total [hat_memload]                             28966912B 1489879332     0
Total [kmem_msb]                               539336704B 989813932  5198
Total [kmem_va]                                3021733888B 193631798     0
Total [kmem_default]                           1442766848B 1478055280     0
Total [kmem_io_64G]                            138547200B     84783     0
Total [kmem_io_4G]                                 53248B     87422     0
Total [kmem_io_2G]                                  4096B         4     0
Total [bp_map]                                    917504B      2388     0
Total [umem_np]                                   786432B      3751     0
Total [id32]                                        4096B       323     0
Total [zfs_file_data_buf]                      2439184384B 2525293649     0
Total [segkp]                                     786432B   2191202     0
Total [ip_minor_arena_sa]                            128B     30959     0
Total [ip_minor_arena_la]                            256B       901     0
Total [spdsock]                                       64B         1     0
Total [namefs_inodes]                                 64B        20     0
------------------------- ------ ------ ------ ---------- --------- ----- 

vmem                         memory     memory    memory     alloc alloc 
name                         in use      total    import   succeed  fail 
------------------------- ---------- ----------- ---------- --------- ----- 
heap                      13564526592B 1077569126400B         0B  34565103     
0 
    vmem_metadata         425226240B  425459712B 425459712B     99311     0 
        vmem_seg          401031168B  401031168B 401031168B     97893     0 
        vmem_hash          23927808B   23937024B  23937024B      1334     0 
        vmem_vmem            265720B     294448B    258048B       127     0 
    static                        0B          0B         0B         0     0 
        static_alloc              0B          0B         0B         0     0 
    hat_memload            28966912B   28966912B  28966912B     29889     0 
    kstat                    672128B     696320B    630784B      9486     0 
    kmem_metadata         586997760B  623640576B 623640576B    912072     0 
        kmem_msb          539336704B  539336704B 539336704B    864680  5198 
        kmem_cache           329664B     417792B    417792B       460     0 
        kmem_hash          47233024B   47243264B  47243264B    136888     0 
    kmem_log                 131808B     135168B    135168B         6     0 
    kmem_firewall_va      141729792B  141729792B 141729792B    578915     0 
        kmem_firewall             0B          0B         0B         0     0 
        kmem_oversize     141532041B  141729792B 141729792B    578904    11 
    mod_sysfile                 353B       4096B      4096B         9     0 
    kmem_va               12148248576B 12148248576B 12148248576B  31994594     
0 
        kmem_default      5738831872B 5738831872B 5738831872B 187061764     0 
    kmem_io_64G           138547200B  138547200B 138547200B     90945     0 
    kmem_io_4G                53248B      53248B     53248B        23     0 
    kmem_io_2G                36864B      36864B     36864B       308     0 
    kmem_io_16M                   0B          0B         0B         0     0 
    bp_map                   917504B     917504B    917504B    223533     0 
    umem_np                  786432B     786432B    786432B        74     0 
    ksyms                   2426757B    2465792B   2465792B       322     0 
    ctf                     1174591B    1523712B   1523712B       324     0 
heap_core                   1323008B  931115008B         0B        57     0 
    heaptext               11403264B   67108864B         0B       214     0 
        module_text        11650016B   11874304B  11403264B       322     0 
    id32                       4096B       4096B      4096B         1     0 
    module_data             1383424B    1626112B   1318912B       456     0 
logminor_space                   24B     262137B         0B     14633     0 
taskq_id_arena                  347B 2147483647B         0B       437     0 
zfs_file_data             6734151680B 17070817280B         0B  90214511     0 
    zfs_file_data_buf     6734151680B 6734151680B 6734151680B  90214511     0 
device                      2719744B 1073741824B         0B     33045     0 
segkp                      25583616B 2147483648B         0B     15671     0 
mac_minor_ids                    18B     130070B         0B        19     0 
rctl_ids                         35B      32767B         0B        35     0 
zoneid_space                      0B       9998B         0B         0     0 
taskid_space                     50B     999999B         0B     14208     0 
pool_ids                          0B     999998B         0B         0     0 
contracts                        54B 2147483646B         0B     21237     0 
ip_minor_arena_sa               128B     262140B         0B         2     0 
ip_minor_arena_la               256B 4294705152B         0B         4     0 
lib_va_32                   7778304B 2138849280B         0B        66     0 
tl_minor_space                   23B     262138B         0B     31554     0 
keysock                           0B 4294967295B         0B         0     0 
spdsock                          64B 4294967295B         0B         1     0 
namefs_inodes                    64B      65536B         0B         1     0 
aggr_portids                      4B      65534B         0B         4     0 
aggr_key_ids                      2B      64535B         0B         2     0 
dtrace                        80055B 4294967295B         0B    109724     0 
dtrace_minor                      0B 4294967293B         0B        50     0 
lmsysid_space                     1B      16383B         0B         2     0 
ds_minors                         2B     262140B         0B         2     0 
Hex0xffffff04e12f2648_minor         0B 4294967294B         0B         0     0 
Hex0xffffff04e43ed588_minor         0B 4294967294B         0B         0     0 
lib_va_64                  41492480B 137988696072192B         0B       246     
0 
devfsadm_event_channel            1B        101B         0B         1     0 
devfsadm_event_channel            1B          2B         0B         1     0 
logdmux_minor                     0B        256B         0B         0     0 
ptms_minor                       10B         16B         0B       102     0 
sppptun_minor                     0B         16B         0B         0     0 
syseventd_channel                 1B        101B         0B         1     0 
syseventd_channel                 1B          2B         0B         1     0 
syseventconfd_door                0B        101B         0B         0     0 
syseventconfd_door                1B          2B         0B         1     0 
crypto                            0B         32B         0B      6889     0 
ipnet_minor_space                 4B     262141B         0B     66285     0 
lxa_minor_id                      0B       1021B         0B         0     0 
------------------------- ---------- ----------- ---------- --------- ----- 
> 
</pre>


btw: ::memstat and ::kmastat is *very* fast in this build, it used to take a 
minute and now it is instantaneous :)

-- 

Robert Milkowski
http://milek.blogspot.com
-- 
This message posted from opensolaris.org
_______________________________________________
opensolaris-code mailing list
opensolaris-c...@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/opensolaris-code


--- End Message ---
_______________________________________________
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Reply via email to