Hi Ben; In according to your feedback about autosizing I've change my setting as below:
net.core.rmem_default = 536870912 net.core.wmem_default = 536870912 net.core.rmem_max = 536870912 net.core.wmem_max = 536870912 net.ipv4.tcp_mem = 10240 87380 268435456 net.ipv4.tcp_rmem = 10240 87380 268435456 net.ipv4.tcp_wmem = 10240 87380 268435456 My ATS usage case is forward transparent cache. In according to my rtt values and network traffic(350 Mbps) buffer size is 256Mbps (=rtt*bw). > I've increased the some of the queue size as is below: > net.core.netdev_max_backlog = 300000 > net.core.somaxconn = 65535 > net.ipv4.tcp_max_orphans = 268435456 > net.ipv4.tcp_max_syn_backlog = 268435456 > > > CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760 > > > CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760 > > > CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760 > > > CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760 I've used above values to prevent bad segment received error on the output of "netstat -s". This is the my nightmare. The "packets collapsed in receive queue" and "bad segment receive" issues are causing the netwotk traffic increasing on the ats-out traffic. Because there is too much packet retransmitions. Do you have any idea to fix this retransmissions? Best Regards, Ergin On Sun, Oct 26, 2014 at 12:29 PM, Ben <[email protected]> wrote: > On Sun, Oct 26, 2014 at 11:05:51AM +0200, Ergin Ozekes wrote: > > Hi james; > > > > In the beginning of my investigation I've set the all interfaces ring > > buffer size max value of the ethernet cards ( which is 4078 by using > > ethtool -g). And also I've increased the socket buffer size by using > kernel > > parameters as is below. > > net.core.rmem_default=536870912 > > net.core.wmem_default=536870912 > > net.core.rmem_max=536870912 > > net.core.wmem_max=536870912 > > net.ipv4.tcp_mem=268435456 268435456 268435456 > > net.ipv4.tcp_rmem=268435456 268435456 268435456 > > net.ipv4.tcp_wmem=268435456 268435456 268435456 > > You really don't need to do that, and it could make things worse rather > than better. > > > I've increased the some of the queue size as is below: > > net.core.netdev_max_backlog = 300000 > > net.core.somaxconn = 65535 > > net.ipv4.tcp_max_orphans = 268435456 > > net.ipv4.tcp_max_syn_backlog = 268435456 > > > Why raise these so high? > > > Also kernel shared memory size. > > > > kernel.shmmax = 16000000000 > > kernel.shmall = 4000000000 > > > > After these setting I saw the increasing "segment retransmited" and > > "packets collapsed in receive queue due to low socket buffer" lines on > the > > "netstat -st" output. > > Probably because you're running out of RAM. > > net.ipv4.tcp_rmem, and net.ipv4.tcp_wmem are per connection. > > autosizing mostly works well, take advantage of it. > > > I think there are timeout values or other parameter settings which > triggers > > the above failures. > > > > Do you have any idea? > > Best Regards, > > Ergin > > > > > > On Sat, Oct 25, 2014 at 1:46 AM, James Peach <[email protected]> wrote: > > > > > > > > > On Oct 24, 2014, at 12:52 AM, Ergin Ozekes <[email protected]> > > > wrote: > > > > > > > > Hi All; > > > > > > > > My problem's solution is as below: > > > > > > > > > > > > ############################################################################## > > > > # Buffer size > > > > > > > > ############################################################################## > > > > CONFIG proxy.config.net.sock_recv_buffer_size_in INT 10485760 > > > > CONFIG proxy.config.net.sock_recv_buffer_size_out INT 10485760 > > > > CONFIG proxy.config.net.sock_send_buffer_size_in INT 10485760 > > > > CONFIG proxy.config.net.sock_send_buffer_size_out INT 10485760 > > So you're saying that you need 10 megabytes buffer size on input and > output to server and client. > > I can do over 10 gigabit/sec with a single client with 512k buffer easily > on LAN. You only need high > values if you have a really fast connection with a really high latency. > Like 10 megabytes at 100 msec ping > would be 100 megabytes/sec to a single client. > > For normal web loads window sizes never even get very high if you use > autosizing as most objects are small > etc. > > > > Great! You might want to check the buffers on your network adaptors as > > > well (ethtool -g IIRC) > > > > > > > > > > > Best Regards, > > > > Ergin > > > > > > > > On Thu, Oct 23, 2014 at 6:33 PM, Ergin Ozekes < > [email protected]> > > > > wrote: > > > > > > > >> Hi all; > > > >> > > > >> In according to my investigation on the ats read > > > >> performance,NetHandler::mainNetEvent is the top function. On my > system > > > >> lots of buffer over run is exist. In my system net input interface > has > > > 250 > > > >> Mbps traffic but on the client output only 125 Mbps is exist. Half > of > > > the > > > >> downloaded content is returning to client. > > > >> > > > >> Does anyone knows the reason of this case? > > > >> And how can I fixe it? > > > >> > > > >> This is the "perf top" output: > > > >> 4.82% libpcre.so.3.13.1 [.] match > > > >> > > > >> > > > >> 3.27% [kernel] [k] native_write_msr_safe > > > >> > > > >> > > > >> 2.09% [kernel] [k] ipt_do_table > > > >> > > > >> > > > >> 1.60% [kernel] [k] __schedule > > > >> > > > >> > > > >> 1.57% [kernel] [k] menu_select > > > >> > > > >> > > > >> 1.43% libc-2.13.so [.] 0x00000000000e92b0 > > > >> > > > >> > > > >> 1.35% [kernel] [k] find_busiest_group > > > >> > > > >> > > > >> 1.16% traffic_server [.] EThread::execute() > > > >> > > > >> > > > >> 1.12% [kernel] [k] copy_user_generic_string > > > >> > > > >> > > > >> 1.10% [kernel] [k] nf_iterate > > > >> > > > >> > > > >> 1.04% [kernel] [k] _raw_spin_lock_irqsave > > > >> > > > >> > > > >> 1.03% [kernel] [k] int_sqrt > > > >> > > > >> > > > >> *1.02% traffic_server [.] NetHandler::mainNetEvent(int, > > > >> Event*)* > > > >> > > > >> 0.96% [bnx2x] [k] bnx2x_rx_int > > > >> > > > >> > > > >> 0.93% [kernel] [k] _raw_spin_lock > > > >> > > > >> > > > >> 0.90% [kernel] [k] htable_selective_cleanup > > > >> > > > >> > > > >> 0.86% [kernel] [k] cpuidle_enter_state > > > >> > > > >> > > > >> 0.83% [kernel] [k] enqueue_task_fair > > > >> > > > >> > > > >> 0.83% [kernel] [k] tcp_packet > > > >> > > > >> > > > >> 0.76% [kernel] [k] apic_timer_interrupt > > > >> > > > >> > > > >> 0.76% [kernel] [k] timerqueue_add > > > >> > > > >> > > > >> 0.76% [kernel] [k] idle_cpu > > > >> > > > >> > > > >> 0.71% [bnx2x] [k] bnx2x_start_xmit > > > >> > > > >> > > > >> 0.67% [kernel] [k] rb_erase > > > >> > > > >> > > > >> *0.64% traffic_server [.] read_from_net(NetHandler*, > > > >> UnixNetVConnection*, EThread*) * > > > >> > > > >> > > > >> > > > >> On Thu, Oct 23, 2014 at 12:33 PM, Ergin Ozekes < > [email protected]> > > > >> wrote: > > > >> > > > >>> Hi all; > > > >>> > > > >>> My current working network bandwidth is 350 Mbps. > > > >>> My netstat -st output prints out below value increasingly. > > > >>> > > > >>> 1555903887 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555908175 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555912925 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555920054 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555929162 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555938162 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555945682 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555951783 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555959318 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555962474 packets collapsed in receive queue due to low socket > > > buffer > > > >>> 1555969574 packets collapsed in receive queue due to low socket > > > buffer > > > >>> > > > >>> I've increased to socket buffer size, backlog queue size and > memlock > > > >>> value was set to unlimited for ats user. How can I fix it > completely? > > > >>> BTW, How can I increase read from socket(connection) performance > of the > > > >>> ATS? > > > >>> > > > >>> > > > >>> Best Regards, > > > >>> Ergin Ozekes > > > >>> > > > >>> On Thu, Oct 23, 2014 at 12:16 PM, Ergin Ozekes < > [email protected] > > > > > > > >>> wrote: > > > >>> > > > >>>> Hi all; > > > >>>> > > > >>>> My current working network bandwidth is 350 Mbps. > > > >>>> My netstat -st output prints out below value increasingly. > > > >>>> > > > >>>> 1555903887 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555908175 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555912925 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555920054 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555929162 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555938162 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555945682 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555951783 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555959318 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555962474 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> 1555969574 packets collapsed in receive queue due to low socket > > > >>>> buffer > > > >>>> > > > >>>> I've increased to socket buffer size, backlog queue size and > memlock > > > >>>> value was set to unlimited for ats user. How can I fix it > completely? > > > >>>> > > > >>>> Best Regards, > > > >>>> Ergin Ozekes > > > >>>> > > > >>> > > > >>> > > > >> > > > > > > >
