Re: Full Logging Solutions
I wonder if RackAMole (http://github.com/derailed/rackamole ), which also has a companion Sinatra app (Wackamole) as its UI, might be to your liking. Maybe you could merge the best features of request_log with RackAMole? Al On Fri, Oct 29, 2010 at 6:14 AM, Peter Marklund wrote: > Hey John! > Thanks for the tip about capped collections, I had missed that. I now > have a 20GB capped collection on MongoHQ for my logging... I've added > a note to the README about this. > > Cheers > > Peter > > On Oct 29, 12:20 am, John Barnette wrote: > > On Oct 28, 2010, at 3:13 PM, Peter Marklund wrote: > > > > > I've experimented with logging to MongoDB lately and I've found it to > > > work really well. The MongoDB performance along with its query > > > capabilities opens up a lot of possibilities. I'm doing my logging to > > > MongoHQ now through a Rack middleware and am monitoring the > > > performance (time overhead) of the logging and it's looking really > > > good so far. I've packaged up my code in a gem: > > > > Good stuff. We're doing some similar things. You may want to look at > Mongo's capped collections for some of your logging if you're not already: > The natural ordering and fixed max size can be really useful. > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- Nothing is impossible to the person who doesn't have to do it himself. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
Hey John! Thanks for the tip about capped collections, I had missed that. I now have a 20GB capped collection on MongoHQ for my logging... I've added a note to the README about this. Cheers Peter On Oct 29, 12:20 am, John Barnette wrote: > On Oct 28, 2010, at 3:13 PM, Peter Marklund wrote: > > > I've experimented with logging to MongoDB lately and I've found it to > > work really well. The MongoDB performance along with its query > > capabilities opens up a lot of possibilities. I'm doing my logging to > > MongoHQ now through a Rack middleware and am monitoring the > > performance (time overhead) of the logging and it's looking really > > good so far. I've packaged up my code in a gem: > > Good stuff. We're doing some similar things. You may want to look at Mongo's > capped collections for some of your logging if you're not already: The > natural ordering and fixed max size can be really useful. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
On Oct 28, 2010, at 3:13 PM, Peter Marklund wrote: > I've experimented with logging to MongoDB lately and I've found it to > work really well. The MongoDB performance along with its query > capabilities opens up a lot of possibilities. I'm doing my logging to > MongoHQ now through a Rack middleware and am monitoring the > performance (time overhead) of the logging and it's looking really > good so far. I've packaged up my code in a gem: Good stuff. We're doing some similar things. You may want to look at Mongo's capped collections for some of your logging if you're not already: The natural ordering and fixed max size can be really useful. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
I've experimented with logging to MongoDB lately and I've found it to work really well. The MongoDB performance along with its query capabilities opens up a lot of possibilities. I'm doing my logging to MongoHQ now through a Rack middleware and am monitoring the performance (time overhead) of the logging and it's looking really good so far. I've packaged up my code in a gem: http://github.com/peter/request_log Cheers Peter On Oct 21, 12:11 am, Eric Anderson wrote: > On Oct 20, 11:18 am, Jeff Schmitz wrote: > > > Another question would be does Heroku need anything written to the rails > > log? Would your patch cause Herkou any issues? > > I would think not as Heroku is more "rack" focused and Rails is just a > framework that can run on "rack". That being said there is some > special mojo for Rails (such as setting up the database connection, > ActionMailer, etc). So the only real way to know is to test. I will be > developing this concept and testing it shortly so will report back > what I find. > > Eric -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
On Oct 20, 11:18 am, Jeff Schmitz wrote: > Another question would be does Heroku need anything written to the rails > log? Would your patch cause Herkou any issues? I would think not as Heroku is more "rack" focused and Rails is just a framework that can run on "rack". That being said there is some special mojo for Rails (such as setting up the database connection, ActionMailer, etc). So the only real way to know is to test. I will be developing this concept and testing it shortly so will report back what I find. Eric -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
I highly recommend having a working (and tested) snippet like this in the logworm docs. Another question would be does Heroku need anything written to the rails log? Would your patch cause Herkou any issues? On Wed, Oct 20, 2010 at 10:06 AM, Eric Anderson wrote: > On Oct 19, 9:14 pm, schapirama wrote: > > No, your Rails.logger.info is not logged to logworm; you need to use > > logworm-specific commands to log with us.work well and any issues you > might > foresee with a setup like this. > > > Hmmm I'm glad Jeff asked this question. I just ASSUMED the Rails > logger messages were recorded. I defiantly need that. Would you see > anything wrong with adjusting the Rails logger to call LogWorm. I put > together an example bit of code (completely untested just to convey > the general concept) at http://gist.github.com/636575. Would love to > know if something like this would work well and any issues you might > foresee with a setup like this. > > Eric > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
On Oct 19, 9:14 pm, schapirama wrote: > No, your Rails.logger.info is not logged to logworm; you need to use > logworm-specific commands to log with us. Hmmm I'm glad Jeff asked this question. I just ASSUMED the Rails logger messages were recorded. I defiantly need that. Would you see anything wrong with adjusting the Rails logger to call LogWorm. I put together an example bit of code (completely untested just to convey the general concept) at http://gist.github.com/636575. Would love to know if something like this would work well and any issues you might foresee with a setup like this. Eric -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
Eric, My apologies for not giving you an account earlier --as you said, since we're still in beta we have to manage the number of accounts we give out. Your questions about the meaning of "beta" are valid. The current setup is fully functional and completely stable. We have approx ~100 live users and are handling tens of transactions per second. The logworm gem, which you install with your application, sends log entries to our servers using a light-weight protocol (not HTTP); the log entries are immediately stored in a queue, and then asynchronously handled by separate workers. From your application point of view, logworm adds just ~1ms. of overhead to the processing of an incoming request. If our workers or our databases go down, your messages will simply accumulate in a queue. We have, indeed, had internal downtimes due to DB migrations, but that was completely hidden from clients and in fact no messages were lost. If, on the other hand, it's the queue itself goes down, then log entries will certainly get lost, and you application will see higher latency as TCP retries the connection to the servers (until it times out). We have a clustered setup of active- active servers (inside the same EC2 zone as Heroku) to make this situation as unlikely as possible. In the 4 months since this setup has been operational, we haven't had a single issue with the queues. The main reason we're limiting the number of beta users is precisely that we want to be able to guarantee good response times whenever there are issues or questions. Now that you have a beta account, you can count on getting prompt responses for us. ;-) I hope this helps. Thanks for trying logworm out! - Agustin On Oct 19, 4:03 pm, Eric Anderson wrote: > On Oct 19, 10:25 am, schapirama wrote: > > > I'm currently running a complete logging solution for Heroku, but the > > add-on is still in beta. See: > > http://www.logworm.com > > > If you're interested, please contact me directly > > (schap...@logworm.com) and I'll create accounts for you, > > Thanks, I will e-mail you at that address. If you look at my post you > see that I actually mentioned your service is right up my alley. I > tried to signup up with your service a month or so ago (using your > signup form) and never heard a response back. So things I am wondering > are: > > * The "beta" label scares me. How stable is your service? I want good > log data but I don't want to have a unstable app to get it. > Specifically I am wondering what happens if your service goes down. > Does it take my service with it or does it bundle up the message in a > queue and send to you when everything is working again. If it takes > down my service do you have any historical data regarding service up- > time. If you are rock solid I can risk it (after all Heroku is up and > down a few times a month itself) but if you take my service down on a > regular basis obviously that won't work. > * The lack of response on the signup also concerns me. I realize > private beta means that not everybody that wants in can get in. But > once I am in can I count on your service to be responsive should > problems show up? > > Other than that I like the look of your setup and look forward to > working with you. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
Jeff, No, your Rails.logger.info is not logged to logworm; you need to use logworm-specific commands to log with us. By default, the logworm gem will automatically log information about each HTTP request that your application receives. If you want to log app-specific information (and you're encouraged to do so), then you need to use the logworm log_in command. BTW, logworm entries are stored in a database as hash-maps, not free-form text, so they can be easily queried. We have built-in mechanisms for daily dumps of your logs to S3 -- contact me if you're interested. The beta account gives you up to 3 logging tables, with 10MB each. That is enough, for example, for ~30K entries recording More complete docs available at: http://www.logworm.com/docs - A -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
On Oct 19, 10:25 am, schapirama wrote: > I'm currently running a complete logging solution for Heroku, but the > add-on is still in beta. See: > http://www.logworm.com > > If you're interested, please contact me directly > (schap...@logworm.com) and I'll create accounts for you, Thanks, I will e-mail you at that address. If you look at my post you see that I actually mentioned your service is right up my alley. I tried to signup up with your service a month or so ago (using your signup form) and never heard a response back. So things I am wondering are: * The "beta" label scares me. How stable is your service? I want good log data but I don't want to have a unstable app to get it. Specifically I am wondering what happens if your service goes down. Does it take my service with it or does it bundle up the message in a queue and send to you when everything is working again. If it takes down my service do you have any historical data regarding service up- time. If you are rock solid I can risk it (after all Heroku is up and down a few times a month itself) but if you take my service down on a regular basis obviously that won't work. * The lack of response on the signup also concerns me. I realize private beta means that not everybody that wants in can get in. But once I am in can I count on your service to be responsive should problems show up? Other than that I like the look of your setup and look forward to working with you. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
Thanks for the offer I took a look and its not clear 1) whether my existing Rails.logger.info "msg" gets logged to logwworm, or do I need to rewrite 2) can I get a full text download of my log 3) how much history is kept Thanks jeff On Tue, Oct 19, 2010 at 9:25 AM, schapirama wrote: > Eric, Jeff, > > I'm currently running a complete logging solution for Heroku, but the > add-on is still in beta. See: >http://www.logworm.com > > If you're interested, please contact me directly > (schap...@logworm.com) and I'll create accounts for you, > > - Agustin > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
Eric, Jeff, I'm currently running a complete logging solution for Heroku, but the add-on is still in beta. See: http://www.logworm.com If you're interested, please contact me directly (schap...@logworm.com) and I'll create accounts for you, - Agustin -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Full Logging Solutions
Eric, totally agree - not being able to get logs is a major drawback. Would love to hear what you find. jeff On Mon, Oct 18, 2010 at 9:35 AM, Eric Anderson wrote: > I am looking for a good reliable way of getting my full logs. > Solutions like Hoptod, New Relic and Get Exceptional all focus > tracking errors or summarizing performance (from what I can tell). But > sometimes I just want to see the actual logs of what an actual user > did when interacting with my app. > > Of course there is "heroku logs" but that just shows you the last few > request. Not to mention I think it may only show one dyno as requests > seem to be missing. LogWorm looks right up my alley but I tried to > sign up for their service a month or two ago and never heard back from > them (they are in private beta). > > So now I am wondering if I need to try to roll my own (http:// > github.com/peburrows/mongo_db_logger look promising) or if there is > already a off-the-shelf solution out there? > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to her...@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Full Logging Solutions
I am looking for a good reliable way of getting my full logs. Solutions like Hoptod, New Relic and Get Exceptional all focus tracking errors or summarizing performance (from what I can tell). But sometimes I just want to see the actual logs of what an actual user did when interacting with my app. Of course there is "heroku logs" but that just shows you the last few request. Not to mention I think it may only show one dyno as requests seem to be missing. LogWorm looks right up my alley but I tried to sign up for their service a month or two ago and never heard back from them (they are in private beta). So now I am wondering if I need to try to roll my own (http:// github.com/peburrows/mongo_db_logger look promising) or if there is already a off-the-shelf solution out there? -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to her...@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
