A NOTE has been added to this issue. ====================================================================== http://www.dbmail.org/mantis/view.php?id=834 ====================================================================== Reported By: jasb Assigned To: ====================================================================== Project: DBMail Issue ID: 834 Category: POP3 daemon Reproducibility: have not tried Severity: minor Priority: normal Status: new target: ====================================================================== Date Submitted: 25-Jan-10 17:06 CET Last Modified: 01-Dec-11 18:23 CET ====================================================================== Summary: Too many open files (POP3D) Description: Howdy Paul,
Since I’m using 2.3.6, maybe from November or so, today happened something that never did. The POP3 service just hang. -- lira:~# telnet localhost 110 Trying 127.0.0.1... Connected to localhost.localdomain. Escape character is '^]'. Connection closed by foreign host. lira:~# -- after killing it and start again everything’s working fine. I have no debug logs! I saw on the logs this: -- Jan 20 13:48:56 lira dbmail/pop3d[3717]: [0x9c07f80] Error:[server] server_sock_cb(+449): Too many open files -- Don’t know if this help’s on tracking some leak on POP3D. One thing that may help that I saw looking at the log, this may happen ‘cause of the “attach” of the spammer sending known users to the pop3d , and that may be the reason that make it hang with the error “Too many open files”, so the leak may be there. Log is attached, Anything you need say, Jorge, ====================================================================== ---------------------------------------------------------------------- (0003018) waza123 (reporter) - 01-Feb-10 17:03 http://www.dbmail.org/mantis/view.php?id=834#c3018 ---------------------------------------------------------------------- the same !! Feb 01 09:18:40 www dbmail-pop3d[22070]: [0x8055f60] EMERGENCY:[server] _sock_cb(+487): getpeername::error [Transport endpoint is not connected] ---------------------------------------------------------------------- (0003019) paul (administrator) - 01-Feb-10 16:34 http://www.dbmail.org/mantis/view.php?id=834#c3019 ---------------------------------------------------------------------- Not the same. an emergency exit is *not* the same as a segfault. the getpeername error is an indication that the client has hung up before a connection could be initialized. This should of course not lead to a program exit. ---------------------------------------------------------------------- (0003021) jasb (reporter) - 03-Feb-10 10:38 http://www.dbmail.org/mantis/view.php?id=834#c3021 ---------------------------------------------------------------------- Paul, Happened again. Tell me, the leaks that you have been fixing on HEAD, does anyone has anything to do with POP3? ---------------------------------------------------------------------- (0003022) paul (administrator) - 03-Feb-10 13:16 http://www.dbmail.org/mantis/view.php?id=834#c3022 ---------------------------------------------------------------------- No. Unrelated. Try taking lsof snapshots of the system when the error occurs. ---------------------------------------------------------------------- (0003025) jasb (reporter) - 04-Feb-10 21:41 http://www.dbmail.org/mantis/view.php?id=834#c3025 ---------------------------------------------------------------------- Hum, dificult. Can i ask you to write a bash or perl script, to issue for example 20.000 login attempts againts POP3D, so that i can catch that information? I don't have skill's to write that script, sorry :( ---------------------------------------------------------------------- (0003028) jasb (reporter) - 18-Feb-10 10:43 http://www.dbmail.org/mantis/view.php?id=834#c3028 ---------------------------------------------------------------------- Paul, I asked a friend to make me a perl script to do this, and I did the folowing: -Issue 50.000 connections with wrong username and passwd, server behave OK -Issue 50.000 connection with valid user & passwd, server behave OK now, o could simulate a login, and message retrieve, to do this, which commands should i send to server (without delete any message): 1-login 2-STAT 3-LIST 4-RETR msg_id 5-QUIT confirm me if this OK, so that i can simulate it. ---------------------------------------------------------------------- (0003281) jasb (reporter) - 04-Oct-11 17:01 http://www.dbmail.org/mantis/view.php?id=834#c3281 ---------------------------------------------------------------------- Hi Paul, Since last time I reported this, I got the POP3D hang today, right after a restart, and I just saw that someone tried to login with several users (guessing). Attached there's the log. Is there a way that you can simulate this to see if in the last GIT still has this problem? This can't be the system open file limit 'cause I have it set to 2Milion (2.000.000). ---------------------------------------------------------------------- (0003282) jasb (reporter) - 04-Oct-11 17:03 http://www.dbmail.org/mantis/view.php?id=834#c3282 ---------------------------------------------------------------------- Ah, I did a small strace to the hang pid, but only had this, and since it didn't moved for long time, I just canceled it. lira:~# strace -p 12010 Process 12010 attached - interrupt to quit clock_gettime(CLOCK_MONOTONIC, {611842, 882460720}) = 0 epoll_wait(9, ^C <unfinished ...> Process 12010 detached lira:~# cat /proc/sys/fs/file-max 2000000 lira:~# ---------------------------------------------------------------------- (0003285) paul (administrator) - 05-Oct-11 09:35 http://www.dbmail.org/mantis/view.php?id=834#c3285 ---------------------------------------------------------------------- Jorge, Next time it happens do a 'lsof -p <PID>' to see what files where opened. ---------------------------------------------------------------------- (0003286) paul (administrator) - 05-Oct-11 09:44 http://www.dbmail.org/mantis/view.php?id=834#c3286 ---------------------------------------------------------------------- Just looking at your last log. Smells like a DoS. You should most definitely take a look at fail2ban. ---------------------------------------------------------------------- (0003292) jasb (reporter) - 05-Oct-11 11:35 http://www.dbmail.org/mantis/view.php?id=834#c3292 ---------------------------------------------------------------------- Hi Paul, That, it was what I was trying to say but missed the words, those login attempts. Anyway the POP3D should not hang when this happen, right? I'm going to try to do a script with the same to see if I can "hang" pop3d to get that info for you. ---------------------------------------------------------------------- (0003295) paul (administrator) - 05-Oct-11 12:25 http://www.dbmail.org/mantis/view.php?id=834#c3295 ---------------------------------------------------------------------- DoS is very difficult to fix inside dbmail. I have no trouble believing you can DoS dbmail. You should fix this on the outside, by improving your perimeter defenses: your firewall combined with a tool like fail2ban. ---------------------------------------------------------------------- (0003296) jasb (reporter) - 16-Oct-11 00:27 http://www.dbmail.org/mantis/view.php?id=834#c3296 ---------------------------------------------------------------------- Paul, Besides the part of the DoS, here's the lsof -p for you, just happened again, and again the same type of attack. Check the attached file. ---------------------------------------------------------------------- (0003350) vampyre (reporter) - 30-Nov-11 18:41 http://www.dbmail.org/mantis/view.php?id=834#c3350 ---------------------------------------------------------------------- Hi Paul, Is it only brute force can cause such issue? And if this is so can I find the guilty? (ip of client) ? I found something similar on my machine. It looks like the socket accept the connection while no greeting from dbmail is followed. I've tried to trace it, but no luck, are the any ideas? ==== (gdb) bt http://www.dbmail.org/mantis/view.php?id=0 0x00000033452e6ee3 in epoll_wait () from /lib64/libc.so.6 http://www.dbmail.org/mantis/view.php?id=1 0x000000334ec12eab in ?? () from /usr/lib64/libevent-1.4.so.2 http://www.dbmail.org/mantis/view.php?id=2 0x000000334ec064f3 in event_base_loop () from /usr/lib64/libevent-1.4.so.2 http://www.dbmail.org/mantis/view.php?id=3 0x000000334f048ab1 in server_run (conf=0x7fffd77b3990) at server.c:699 http://www.dbmail.org/mantis/view.php?id=4 0x000000334f04905c in server_mainloop (config=0x7fffd77b3990, service=0x41f7ae "IMAP", servicename=0x41f7fc "dbmail-imapd") at server.c:832 http://www.dbmail.org/mantis/view.php?id=5 0x0000000000418d27 in main (argc=1, argv=0x7fffd77b72f8) at imapd.c:53 (gdb) exit ==== ---------------------------------------------------------------------- (0003351) paul (administrator) - 01-Dec-11 09:49 http://www.dbmail.org/mantis/view.php?id=834#c3351 ---------------------------------------------------------------------- Vampyre, the guilty IP should be in your logs. I don't know your situation, but for Jorge his logs in jasbpop3.zip clearly show what's going on: brute force attack. It would be trivial to configure tools like fail2ban to inject firewall rules when someone repeatedly tries and fails to login from the same IP adress. iptables also supports throttling SYNC/ACK. Below example will limit the number of connections to 10 connections per minute per IP address. /sbin/iptables -t filter -A INPUT --jump ACCEPT -p tcp --dport 110 -m state --state NEW -m recent --set --name POP3 /sbin/iptables -t filter -A INPUT --jump LOG --log-prefix INPUT_DROP: -p tcp --dport 110 -m recent --update --seconds 60 --hitcount 10 --rttl --name POP3 /sbin/iptables -t filter -A INPUT --jump DROP -p tcp --dport 110 -m recent --update --seconds 60 --hitcount 10 --rttl --name POP3 personally I prefer the fail2ban approach because SYN/ACK throttling will also limit valid connections. ---------------------------------------------------------------------- (0003352) vampyre (reporter) - 01-Dec-11 18:23 http://www.dbmail.org/mantis/view.php?id=834#c3352 ---------------------------------------------------------------------- Aha, now I see, there is a debug in pop3.c but no in imap4.c, I am using imap. I've just made a grep along the new dbmail code: === vampyre@duffy:~/dbmail$ grep -r "coming from" * src/dm_misc.h.orig: * \brief discards all input coming from instream src/pop3.c: TRACE(TRACE_ERR, "user [%s] coming from [%s] tried to login with wrong password", src/dm_misc.h: * \brief discards all input coming from instream test/check_dbmail_deliver.c: * \brief discards all input coming from instream === Please add the debug message for imap4.c or correct me if I wrong. Issue History Date Modified Username Field Change ====================================================================== 25-Jan-10 17:06 jasb New Issue 25-Jan-10 17:06 jasb File Added: pop3_too_many_open_files.zip 01-Feb-10 15:49 waza123 Note Added: 0003018 01-Feb-10 16:34 paul Note Added: 0003019 01-Feb-10 17:03 paul Note Edited: 0003018 03-Feb-10 10:38 jasb Note Added: 0003021 03-Feb-10 13:16 paul Note Added: 0003022 04-Feb-10 21:41 jasb Note Added: 0003025 18-Feb-10 10:43 jasb Note Added: 0003028 04-Oct-11 17:01 jasb Note Added: 0003281 04-Oct-11 17:01 jasb File Added: jasbpop3.zip 04-Oct-11 17:03 jasb Note Added: 0003282 05-Oct-11 09:35 paul Note Added: 0003285 05-Oct-11 09:44 paul Note Added: 0003286 05-Oct-11 11:35 jasb Note Added: 0003292 05-Oct-11 12:25 paul Note Added: 0003295 16-Oct-11 00:27 jasb Note Added: 0003296 16-Oct-11 00:30 jasb File Added: pop3_hang.zip 30-Nov-11 18:41 vampyre Note Added: 0003350 01-Dec-11 09:49 paul Note Added: 0003351 01-Dec-11 18:23 vampyre Note Added: 0003352 ====================================================================== _______________________________________________ Dbmail-dev mailing list Dbmail-dev@dbmail.org http://mailman.fastxs.nl/cgi-bin/mailman/listinfo/dbmail-dev
