dropping characters in ssh sessions

Charles Sprickman Mon, 11 Oct 2004 19:39:11 -0700

Hi,

I have a box that is primarily acting as a local DNS cache. It's taking a ton of traffic, and also generating a good deal of state information within ipf.

During particularly heavy times, I'm noticing that my ssh sessions will just drop characters. It's completely one way, meaning that everything coming back to me is fine, but stuff going "to" the box drops; ie, I could be editing something and when I type ":wq", the "w" makes it and the "q" doesn't. While it's difficult to pin this on ipf, it does seem like the problem goes away if I temporarily flush the ipf rules. It's become quite annoying lately. We've upped the memory that ipfilter can use to solve another problem (filling up the state table). This was accomplised by editing "ip_state.h" like so:

/* LOCAL change - need larger state numbers
 *
 * both numbers must be primes (http://www.prime-numbers.org/)
 * and the second should be around 70% of the first
 */

#ifndef IPSTATE_SIZE
# define    IPSTATE_SIZE    84991
#endif
#ifndef IPSTATE_MAX
# define    IPSTATE_MAX     59999   /* Maximum number of states held */
#endif

That did fix our problem with dnscache not answering everything that was asked of it, but made no difference in this ssh problem.

Some info:

FreeBSD toolbox.blah.com 4.9-RELEASE-p4 FreeBSD 4.9-RELEASE-p4 #4: Thu May 20 01:49:50 EDT 2004

toolbox[/usr/local/etc/tinydns/root]# ipf -V

ipf: IP Filter: v3.4.31 (336)
Kernel: IP Filter: v3.4.31
Running: yes
Log Flags: 0x20000000 = block
Default: pass all, Logging: available
Active list: 0

toolbox[/usr/local/etc/tinydns/root]# ipfstat IPv6 packets: in 0 out 0 input packets: blocked 130183 passed 503317507 nomatch 0 counted 0 short 5871 output packets: blocked 0 passed 516740828 nomatch 0 counted 0 short 0 input packets logged: blocked 130183 passed 0 output packets logged: blocked 0 passed 0 packets logged: input 0 output 0 log failures: input 0 output 0 fragment state(in): kept 3 lost 0 fragment state(out): kept 0 lost 0 packet state(in): kept 372346 lost 0 packet state(out): kept 80723670 lost 4861954 ICMP replies: 29135 TCP RSTs sent: 0 Invalid source(in): 0 Result cache hits(in): 336956 (out): 100 IN Pullups succeeded: 0 failed: 0 OUT Pullups succeeded: 0 failed: 0 Fastroute successes: 29135 failures: 0 TCP cksum fails(in): 0 (out): 0 Packet log flags set: (0x20000000) packets blocked by filter

toolbox[/usr/local/etc/tinydns/root]# ipfstat -s
IP states added:
        1534638 TCP
        79322189 UDP
        239477 ICMP
        924559038 hits
        148079893 misses
        4315733 maximum
        0 no memory
        20179 bkts in use
        24666 active
        79553320 expired
        1518318 closed

I'm not seeing anything that sticks out here other than the "whoa, it's pretty busy" numbers. I've also seen this problem starting to crop up on another server that is taking a smallish netflow feed (udp) from a Cisco.

Not quite sure where to start with this... Is there any other info that would help?

I also have a tcpdump snippet. Where it's marked "HERE" is where it started dropping characters:

P 5088:5136(48) ack 8913 win 65535 <nop,nop,timestamp 956094730 844165988> (DF)19:06:52.621153 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: . ack 9297 win 65535 <nop,nop,timestamp 956094731 844166011> (DF) 19:06:53.236389 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5136:5184(48) ack 9297 win 65535 <nop,nop,timestamp 956094732 844166011> (DF)19:06:53.420928 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: . ack 9345 win 65535 <nop,nop,timestamp 956094732 844166094> (DF) 19:06:53.660208 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5184:5232(48) ack 9345 win 65535 <nop,nop,timestamp 956094733 844166094> (DF)19:06:53.804136 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5232:5280(48) ack 9393 win 65535 <nop,nop,timestamp 956094733 844166136> (DF)19:06:54.021661 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: . ack 9457 win 65535 <nop,nop,timestamp 956094734 844166153> (DF) 19:06:54.139479 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5280:5328(48) ack 9457 win 65535 <nop,nop,timestamp 956094734 844166153> (DF)19:06:54.452300 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5328:5376(48) ack 9457 win 65535 <nop,nop,timestamp 956094735 844166194> (DF)19:06:54.622144 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: . ack 10161 win 65535 <nop,nop,timestamp 956094735 844166215> (DF) 19:06:56.171466 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5376:5424(48) ack 10161 win 65535 <nop,nop,timestamp 956094738 844166215> (DF)

Here -->>>

19:06:56.300545 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5424:5472(48) ack 10161 win 65535 <nop,nop,timestamp 956094738 844166397> (DF) 19:06:56.412936 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5472:5568(96) ack 10161 win 65535 <nop,nop,timestamp 956094738 844166410> (DF) 19:06:56.522385 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5568:5616(48) ack 10161 win 65535 <nop,nop,timestamp 956094739 844166421> (DF) 19:06:56.660469 host-216-220-116-154.dsl.foo.net.988 > toolbox.blah.com.ssh: P 5616:5664(48) ack 10161 win 65535 <nop,nop,timestamp 956094739 844166432> (DF)

Thanks,

Charles

dropping characters in ssh sessions

Reply via email to