Chris, Thanks for sharing. Always interesting to see the troubleshooting steps and resolution for production issues.
I've used logentries and papertrail (via Heroku addons) and they have both been great. Papertrail has some nice recommendations regarding Rails log verbosity here: http://help.papertrailapp.com/kb/configuration/controlling-verbosity On Sunday, October 27, 2013 9:17:46 PM UTC-7, Chris McCann wrote: > > SD Ruby, > > This morning I checked on a Rails app I've had in production for many > years and found that the app was dumping a full stacktrace to the screen > instead of either rendering the expected page or showing the "Oops. > Something went wrong" error page. > > I immediately tried to do a cap web:deploy:disable to at least put up the > maintenance page but that failed, too. It was slightly unnerving since > nothing like this had happened to me before. > > Looking into it I saw a MySQL error with an error code of 28 in the > message. A quick Google of that showed that's what MySQL does when it's > out of disk space. > > A quick "df -h" and sure enough, the 6 GB disk showed 0% available. WTF, > I wondered. > > I headed over to the Rails log directory to see what the production log > showed. Well, the first thing it showed was the production.log file was > almost 1 GB. That's not normal, as logrotate is set to rotate the log > every week and I've barely ever seen one even 1/10 that size. > > I gzipped the log and pulled it down locally for a little spelunking. > Before long I found a suspicious IP address hitting the Site#index page > over, and over, and over -- 2,131,853 times, to be exact. That caused the > production log to blow way past the available disk space on the slice. > Truncating the log immediately restored the apps functionality. > > As a quick reaction measure I added the offending IP to be dropped via > iptables. The customer support techs at Rimuhosting.com (who are, frankly, > the most responsive I've ever worked with) were very helpful, and they even > let me know I was eligible for a free upgrade of over 1/2 GB of RAM and 6 > GB of disk space, which they promptly added. > > I modified my logrotate conf file for the app to rotate every 100MB > instead of weekly. If you're rotating on time like I was you're setting > yourself up for what I'm calling a "log-bloat denial of service" attack. > > A slightly scary but instructive lesson, which leads me to my questions: > > - has anyone dealt with a similar style attack in a Rails app? > > - are there some security best practices I'm not implementing? > > - has anyone used fail2ban or similar software with a Rails app to > automatically block nefarious traffic? > > - what are your favorite monitoring solutions to get early warnings of > disk space, RAM, or other impending problems? > > Thanks, > > Chris McCann > > -- -- SD Ruby mailing list [email protected] http://groups.google.com/group/sdruby --- You received this message because you are subscribed to the Google Groups "SD Ruby" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
