Nicolaj,
I was more trying to understand why the kernel is complaining of "Nexthop index allocation failed: regular index space exhausted" In this version the regular index space from kernels perspective is 256k. Before you reach such limit you are running out of resources either on the FPC due to Descriptor space shortage or due to DRAM Memory shortage on the S-Board so this should never we the bottleneck in the version you use. Anyway, I did not got to an answer here and maybe I would need to complete message log and or remote login. The number of rtnexthops is 17411 entries as you can see from the virtual memory point of view. From the data below we can see that the mas next-hop entries was 8106 unicast next-hops on both FPCs. You had double of those entries since you had an aggregate with 2 member links and you can see already the number of next-hops and resources depends on the number of memberlinks of an aggregate. Also the L2 Descriptor space was with 32532 entries below the limit. Any increase you would need to have an Enhanced FPC. SSB1(re0 vty)# show nhdb zones Chip Start Size Rsvd Used/Hi Water/Total Size Name ---- ----- ----- ----- -------------------- ---- ---- 0 20000 01000 00000 0/0/512 8 Multicast RTP 0 21000 02400 00000 0/0/9216 1 Multicast Lists 0 23400 04400 00006 7202/8106/17408 1 Next-Hop Entries 0 27800 18800 0004b 28928/32532/50176 2 L2 Descriptors 0 40400 001ff 00000 3/3/63 8 L2 Programs 1 20000 01000 00000 0/0/512 8 Multicast RTP 1 21000 02400 00000 0/0/9216 1 Multicast Lists 1 23400 04400 00006 0/8106/17408 1 Next-Hop Entries 1 27800 18800 0004b 120/32532/50176 2 L2 Descriptors 1 40400 001ff 00000 2/3/63 8 L2 Programs If you have arp entries only with vlans and without vlans here are quick summary on how far you can go depending on the FPC Type. Those numbers are per FPC and not per system and the DRAM memory of the S-Board is the other part which could limit the numbers. arp entries | ether | with vlan ------------+--------+------------ FPC | 16750 | 13060 | | E-FPC | 54272 | 40722 | | E2-FPC | 61183 | 61183 M10i/M7i | | Since you were not running into those merits you indicated that you were running out of memory resources on the S-Board and your utilization is already about 76%. As a guideline usually you should look at the details between 80-90% where the resources are going to better manage this for future and plan the upgrade or changes. Any transient change might need some high peeks of resources and you should have some room left to make such kind of operations. Once you are at 90% then I usually suggest for an immediate actions in short time frames. In order to illustrate how an aggregate is build and how much memory we need here an example Route -> Aggregate next-hop ---> unicast next-hop | +-----------> unicast next-hop route entry: ============ ipv4 route is 68byte + 4byte(ipv4) = 72byte (If you have uRPF enabled you can add 52byte to each route) unicast nexthop: ================ each of them is 348 byte which includes the max possible L2 header Aggregate next-hop: =================== 92 + (x * 40) where x is the number of links in an aggregate which is in your case 172byte. So in your case with 8k of arp entries and you create an aggregate of 2 member links you need to following more resources on memory space. You double the unicast entries and you add an additional aggregate next-hops for each route. 172byte * 8106 = 1,39MB 384byte * 8106 = 3.11MB So in total about 4,5MB more. If you have the enhanced SSB-E with 256M then this will never be the restriction and you would need to upgrade the FPCs to E-FPC and then you can have 54k entries on each FPCs and getting really high numbers of arp entries. Of course of you have then 100k or 150k on arp entries you would need to think on how to deal with arp-refresh and so on and the regular resources you have on the RE. e.g. RE3.0 is recommended. hope this helps a bit. Josef Sunday, June 10, 2007, 3:24:44 PM, you wrote: NK> NK> NK> Josef Buchsteiner wrote: NK> NK> Hi Josef, NK> >> show version >> show system virtual-memory NK> NK> the output is below. But meanwhile of course I had to fix the problem NK> and therefore moved some Gateway IPs to a different router to decrease NK> the arp count from 7200 to 5611 currently. I had to reboot the SSB-E to NK> free the memory but at least it is working now. NK> NK> [EMAIL PROTECTED]> show version NK> Hostname: re0 NK> Model: m20 NK> JUNOS Base OS boot [7.6R1.10] NK> JUNOS Base OS Software Suite [7.6R1.10] NK> JUNOS Kernel Software Suite [7.6R1.10] NK> JUNOS Packet Forwarding Engine Support (M20/M40) [7.6R1.10] NK> JUNOS Routing Software Suite [7.6R1.10] NK> JUNOS Online Documentation [7.6R1.10] NK> JUNOS Crypto Software Suite [7.6R1.10] NK> NK> NK> NK> [EMAIL PROTECTED]> show system virtual-memory NK> Memory statistics by bucket size NK> Size In Use Free Requests HighWater Couldfree NK> 16 196393 183767 2355256898 1280 1140495 NK> 32 30503 249817 1418360018 640 298805 NK> 64 460785 515407 1405666746 320 1857754 NK> 128 20788 244748 686148930 160 944141 NK> 256 21129 11255 992196792 80 1145025 NK> 512 113 15 340207290 40 0 NK> 1K 81 7 105736 20 52 NK> 2K 195 87 17036 10 3309 NK> 4K 17 10 55228 5 1744 NK> 8K 73 5 87868 5 669 NK> 16K 8 0 15 5 0 NK> 32K 12 0 2281 5 0 NK> 64K 18 0 18 5 0 NK> 128K 5 0 5 5 0 NK> 256K 2 0 2 5 0 NK> 512K 1 0 1 5 0 NK> NK> Memory usage type by bucket size NK> Size Type(s) NK> 16 uc_devlist, nexusdev, iftable, temp, devbuf, atexit, COS, BPF, NK> DEVFS mount, DEVFS node, vnodes, mount, pcb, soname, NK> proc-args, kld, NK> MD disk, rman, ATA generic, bus, sysctl, pfestat, ifstate, NK> pfe_ipc, NK> mkey, rtable, ifmaddr, ipfw, rnode NK> 32 atkbddev, dirrem, mkdir, diradd, freefile, freefrag, indirdep, NK> bmsafemap, newblk, temp, devbuf, tseg_qent, COS, vnodes, NK> cluster_save buffer, pcb, soname, proc-args, sigio, kld, Gzip NK> trees, NK> taskqueue, SWAP, eventhandler, bus, sysctl, uidinfo, subproc, NK> pgrp, NK> ippool, pfestat, itable32, ifstate, pfe_ipc, mkey, rtable, NK> ifmaddr, NK> ipfw, ifdevice, iflist, rnode, rtnexthop NK> 64 isadev, ip6ndp, MFS node, allocindir, allocdirect, pagedep, temp, NK> devbuf, lockf, COS, NULLFS hash, DEVFS name, vnodes, NK> cluster_save buffer, vfscache, pcb, soname, proc-args, file, NK> AR driver, AD driver, Gzip trees, rman, eventhandler, bus, NK> sysctl, NK> subproc, pfestat, pic, ifstate, pfe_ipc, mkey, ifaddr, NK> rtable, ipfw, NK> rnode NK> 128 ZONE, freeblks, inodedep, temp, devbuf, zombie, COS, BPF, DEVFS node, NK> vnodes, mount, cluster_save buffer, pcb, soname, proc-args, ttys, NK> dev_t, timecounter, kld, Gzip trees, ISOFS node, bus, NK> uidinfo, cred, NK> session, pfestat, pic, itable16, ifstate, pfe_ipc, mkey, rtable, NK> ipfw, ifstat, metrics, rtnexthop, iffamily NK> 256 iflogical, iftable, MFS node, FFS node, newblk, temp, devbuf, NK> NFS daemon, vnodes, proc-args, kqueue, file desc, Gzip trees, NK> bus, NK> subproc, itable16, ifstate, pfe_ipc, sysctl, rtable, ipfw, rnode, NK> rtnexthop, iffamily NK> 512 UFS mount, temp, devbuf, mount, BIO buffer, ptys, ttys, file desc, NK> AR driver, Gzip trees, ISOFS mount, msg, ioctlops, ATA NK> generic, bus, NK> proc, pfestat, lr, ifstate, pfe_ipc, rtable, ipfw, ifstat, NK> rtnexthop NK> 1K iftable, temp, devbuf, NQNFS Lease, COS, BIO buffer, kqueue, NK> file desc, kld, AD driver, Gzip trees, sem, MD disk, bus, NK> ifstate, NK> pfe_ipc, mkey, ipfw NK> 2K uc_devlist, iflogical, UFS mount, temp, devbuf, BIO buffer, pcb, NK> AR driver, Gzip trees, ioctlops, bus, pfe_ipc, ipfw, ifstat, NK> rcache NK> 4K memdesc, iftable, UFS mount, temp, devbuf, kld, Gzip trees, sem, msg, NK> bus, iffclassstat NK> 8K iflogical, indirdep, temp, devbuf, syncache, Gzip trees NK> 16K UFS mount, temp, devbuf, shm, msg NK> 32K pagedep, devbuf, BPF, kld, Gzip trees NK> 64K VM pgdata, devbuf, MSDOSFS mount NK> 128K UFS ihash, inodedep, NFS hash, kld, ISOFS mount NK> 256K mbuf, vfscache NK> 512K SWAP NK> NK> Memory statistics by type Type Kern NK> Type InUse MemUse HighUse Limit Requests Limit Limit Size(s) NK> isadev 13 1K 1K127696K 13 0 0 64 NK> atkbddev 2 1K 1K127696K 2 0 0 32 NK> uc_devlist 24 3K 3K127696K 24 0 0 16,2K NK> nexusdev 3 1K 1K127696K 3 0 0 16 NK> memdesc 1 4K 4K127696K 1 0 0 4K NK> mbuf 1 152K 152K127696K 1 0 0 256K NK> iflogical 159 558K 576K127696K 543 0 0 256,2K,8K NK> iftable 16 9K 9K127696K 28 0 0 16,256,1K,4K NK> ip6ndp 0 0K 1K127696K 2 0 0 64 NK> ZONE 15 2K 2K127696K 15 0 0 128 NK> VM pgdata 1 64K 64K127696K 1 0 0 64K NK> UFS mount 15 41K 41K127696K 15 0 0 NK> 512,2K,4K,16K NK> UFS ihash 1 128K 128K127696K 1 0 0 128K NK> MFS node 261 65K 72K127696K 11312 0 0 64,256 NK> FFS node 1212 303K 304K127696K 10321 0 0 256 NK> dirrem 0 0K 4K127696K 41400 0 0 32 NK> mkdir 0 0K 1K127696K 8 0 0 32 NK> diradd 0 0K 2K127696K 41433 0 0 32 NK> freefile 0 0K 1K127696K 7302 0 0 32 NK> freeblks 0 0K 1K127696K 8682 0 0 128 NK> freefrag 0 0K 3K127696K 135820 0 0 32 NK> allocindir 0 0K 248K127696K 406886 0 0 64 NK> indirdep 0 0K 33K127696K 166778 0 0 32,8K NK> allocdirect 0 0K 4K127696K 232268 0 0 64 NK> bmsafemap 0 0K 1K127696K 146306 0 0 32 NK> newblk 1 1K 1K127696K 639155 0 0 32,256 NK> inodedep 1 128K 137K127696K 53025 0 0 128,128K NK> pagedep 1 32K 33K127696K 5813 0 0 64,32K NK> temp 1337 94K 99K127696K 337591 0 0 NK> 16,32,64,128,256,512,1K,2K,4K,8K,16K NK> devbuf 16825 2350K 2671K127696K 1042931 0 0 NK> 16,32,64,128,256,512,1K,2K,4K,8K,16K,32K,64K NK> lockf 42 3K 3K127696K 6112563 0 0 64 NK> atexit 1 1K 1K127696K 1 0 0 16 NK> zombie 0 0K 1K127696K 143910 0 0 128 NK> NFS hash 1 128K 128K127696K 1 0 0 128K NK> NQNFS Lease 1 1K 1K127696K 1 0 0 1K NK> NFS daemon 1 1K 1K127696K 1 0 0 256 NK> syncache 1 8K 8K127696K 1 0 0 8K NK> tseg_qent 0 0K 1K127696K 1452 0 0 32 NK> COS 87 63K 65K127696K 349 0 0 NK> 16,32,64,128,1K NK> BPF 403 7K 69K127696K 524 0 0 16,128,32K NK> MSDOSFS mount 1 64K 64K127696K 1 0 0 64K NK> NULLFS hash 1 1K 1K127696K 1 0 0 64 NK> DEVFS mount 2 1K 1K127696K 2 0 0 16 NK> DEVFS name 493 31K 31K127696K 493 0 0 64 NK> DEVFS node 479 59K 59K127696K 517 0 0 16,128 NK> vnodes 28 7K 7K127696K 429 0 0 NK> 16,32,64,128,256 NK> mount 15 8K 8K127696K 17 0 0 16,128,512 NK> cluster_save buffer 0 0K 1K127696K 16056 0 0 NK> 32,64,128 NK> vfscache 2604 419K 423K127696K 159971 0 0 64,256K NK> BIO buffer 16 32K 505K127696K 18228 0 0 512,1K,2K NK> pcb 174 17K 17K127696K 158237 0 0 NK> 16,32,64,128,2K NK> soname 83 10K 11K127696K 21724454 0 0 16,32,64,128 NK> proc-args 53 2K 3K127696K 122092 0 0 NK> 16,32,64,128,256 NK> ptys 32 16K 16K127696K 32 0 0 512 NK> ttys 187 25K 50K127696K 23168 0 0 128,512 NK> kqueue 4 2K 5K127696K 47137 0 0 256,1K NK> sigio 1 1K 1K127696K 2718 0 0 32 NK> file 521 33K 35K127696K 2102577 0 0 64 NK> file desc 75 20K 24K127696K 149811 0 0 256,512,1K NK> shm 1 12K 12K127696K 1 0 0 16K NK> dev_t 287 36K 36K127696K 287 0 0 128 NK> timecounter 10 2K 2K127696K 10 0 0 128 NK> kld 11 121K 126K127696K 34 0 0 NK> 16,32,128,1K,4K,32K,128K NK> AR driver 1 1K 3K127696K 5 0 0 64,512,2K NK> AD driver 2 2K 3K127696K 1500129 0 0 64,1K NK> Gzip trees 0 0K 46K127696K 156997 0 0 NK> 32,64,128,256,512,1K,2K,4K,8K,32K NK> ISOFS node 1097 138K 138K127696K 1097 0 0 128 NK> ISOFS mount 8 132K 132K127696K 8 0 0 512,128K NK> sem 3 6K 6K127696K 3 0 0 1K,4K NK> MD disk 2 2K 2K127696K 2 0 0 16,1K NK> msg 4 25K 25K127696K 4 0 0 512,4K,16K NK> rman 59 4K 4K127696K 461 0 0 16,64 NK> ioctlops 0 0K 2K127696K 1484 0 0 512,2K NK> taskqueue 2 1K 1K127696K 2 0 0 32 NK> SWAP 2 413K 413K127696K 2 0 0 32,512K NK> ATA generic 6 3K 3K127696K 6 0 0 16,512 NK> eventhandler 18 1K 1K127696K 18 0 0 32,64 NK> bus 359 31K 32K127696K 851 0 0 NK> 16,32,64,128,256,512,1K,2K,4K NK> sysctl 0 0K 1K127696K 19399250 0 0 16,32,64 NK> uidinfo 3 1K 1K127696K 9367 0 0 32,128 NK> cred 14 2K 5K127696K 532789 0 0 128 NK> subproc 149 9K 12K127696K 301375 0 0 32,64,256 NK> proc 2 1K 1K127696K 2 0 0 512 NK> session 10 2K 3K127696K 12648 0 0 128 NK> pgrp 10 1K 1K127696K 12749 0 0 32 NK> ippool 1 1K 1K127696K 1 0 0 32 NK> pfestat 0 0K 1K127696K 4017935 0 0 NK> 16,32,64,128,512 NK> pic 3 1K 1K127696K 10 0 0 64,128 NK> iffclassstat 0 0K 40K127696K 19 0 0 4K NK> lr 1 1K 1K127696K 1 0 0 512 NK> itable32 211 7K 7K127696K 211 0 0 32 NK> itable16 2236 537K 1656K127696K 10157 0 0 128,256 NK> ifstate242815 16288K 88751K127696K1711702529 0 0 NK> 16,32,64,128,256,512,1K NK> pfe_ipc 0 0K 2K127696K4389469970 0 0 NK> 16,32,64,128,256,512,1K,2K NK> mkey 453 8K 9K127696K 351154 0 0 NK> 16,32,64,128,1K NK> ifaddr 223 14K 17K127696K 2476 0 0 64 NK> sysctl 0 0K 1K127696K 35 0 0 256 NK> rtable224823 14056K 27291K127696K348790726 0 0 NK> 16,32,64,128,256,512 NK> ifmaddr 80 2K 2K127696K 295 0 0 16,32 NK> ipfw 28 15K 19K127696K 988 0 0 NK> 16,32,64,128,256,512,1K,2K NK> ifstat 240 99K 100K127696K 894 0 0 128,512,2K NK> ifdevice 9 1K 1K127696K 9 0 0 32 NK> iflist 0 0K 1K127696K 27575 0 0 32 NK> rcache 4 8K 8K127696K 4 0 0 2K NK> rnode214112 3663K 7071K127696K345160759 0 0 16,32,64,256 NK> metrics 21 3K 4K127696K 1417 0 0 128 NK> rtnexthop 17411 4320K 6650K127696K342565086 0 0 NK> 32,128,256,512 NK> iffamily 197 37K 38K127696K 608 0 0 128,256 NK> NK> Memory Totals: In Use Free Requests NK> 44866K 76567K 7198104864 NK> NK> ITEM SIZE LIMIT USED FREE REQUESTS NK> NK> PIPE: 192, 0, 8, 77, 131628 NK> SWAPMETA: 160, 95772, 0, 0, 0 NK> unpcb: 160, 0, 114, 61, 74543 NK> ripcb: 192, 25323, 5, 37, 556 NK> syncache: 128, 15359, 0, 64, 88344 NK> tcpcb: 576, 25323, 36, 28, 126889 NK> udpcb: 192, 25323, 119, 51, 69246 NK> socket: 256, 25323, 369, 47, 338019 NK> KNOTE: 96, 0, 48, 78, 104503 NK> NFSNODE: 352, 0, 0, 0, 0 NK> NFSMOUNT: 544, 0, 0, 0, 0 NK> VNODE: 224, 0, 12104, 90, 12104 NK> NAMEI: 1024, 0, 0, 24, 5823314 NK> VMSPACE: 192, 0, 53, 75, 143957 NK> PROC: 448, 0, 70, 65, 143980 NK> DP fakepg: 64, 0, 0, 0, 0 NK> PV ENTRY: 28, 712366, 121953, 89112, 219769545 NK> MAP ENTRY: 48, 0, 1338, 915, 40350969 NK> KMAP ENTRY: 48, 35635, 210, 174, 173612 NK> MAP: 108, 0, 7, 3, 7 NK> VM OBJECT: 92, 0, 2688, 304, 2771913 NK> NK> 2733497006 cpu context switches NK> 1355685303 device interrupts NK> 474779191 software interrupts NK> 55982873 traps NK> 54543811 system calls NK> 17 kernel threads created NK> 139746 fork() calls NK> 4217 vfork() calls NK> 0 rfork() calls NK> 0 swap pager pageins NK> 0 swap pager pages paged in NK> 0 swap pager pageouts NK> 0 swap pager pages paged out NK> 381 vnode pager pageins NK> 396 vnode pager pages paged in NK> 25526 vnode pager pageouts NK> 95451 vnode pager pages paged out NK> 0 page daemon wakeups NK> 0 pages examined by the page daemon NK> 107 pages reactivated NK> 13014131 copy-on-write faults NK> 0 copy-on-write optimized faults NK> 4163866 zero fill pages zeroed NK> 3997676 zero fill pages prezeroed NK> 2786 intransit blocking page faults NK> 70463176 total VM faults taken NK> 0 pages affected by kernel thread creation NK> 115797006 pages affected by fork() NK> 1620165 pages affected by vfork() NK> 0 pages affected by rfork() NK> 19948709 pages freed NK> 0 pages freed by daemon NK> 15065601 pages freed by exiting processes NK> 45590 pages active NK> 70954 pages inactive NK> 0 pages in VM cache NK> 45593 pages wired down NK> 29407 pages free NK> 4096 bytes per page NK> 24839067 total name lookups NK> cache hits (93% pos + 5% neg) system 0% per-directory NK> deletions 0%, falsehits 0%, toolong 0% NK> interrupt total rate NK> ata0 irq14 2506559 0 NK> mux irq7 4093818368 600 NK> fxp1 irq10 15790 0 NK> sio0 irq4 2518 0 NK> clk irq0 681688963 99 NK> rtc irq8 872620401 127 NK> Total 5650652599 828 NK> NK> NK> NK> NK> NK> NK> -- NK> Accelerated IT Services GmbH NK> Schubertstrasse 10 D-67251 Freinsheim NK> [EMAIL PROTECTED] http://www.accelerated.de/ NK> Telefon: +49 6232 - 672568 Telefax: +49 6353 - 989039 NK> HRB: 60665 - Amtsgericht Ludwigshafen UstID: DE253684415 NK> Geschäftsführende Gesellschafter: Nicolaj Kamensek & Ole Krieger NK> NK> NK> _______________________________________________ juniper-nsp mailing list juniper-nsp@puck.nether.net https://puck.nether.net/mailman/listinfo/juniper-nsp