Re: Reading Email
A friend of mine needed the exact same thing on Heroku. He just created a recursive delayed job - a job that schedules itself to run 1 minute later, when it has executed. Works fine for him :) Cheers, Casper Fabricius On 06/10/2009, at 21.12, Yuri Niyazov wrote: Heroku comes with cron support, but only once an hour. I need it more often than that, so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
Oren, any thoughts on how to regularly receive email to a heroku app? On Oct 7, 2:50 pm, Oren Teich o...@heroku.com wrote: Cron runs on a separate single process. It doesn't matter how many dyno's you have, you'll only have one cron process ever running. If you're seeing other behavior, let us know! Oren On Oct 7, 2009, at 8:13 AM, Yuri Niyazov wrote: Also, I forgot the following fun fact about Heroku's cron service. This was true when I investigated it; might still be true now - not sure. Since your app runs on X Heroku VMs, where X is often 1, then, when you use Heroku's cron, the cronjob is executed on each box simultaneously - unless you do something clever (and I was unable to figure out what that something clever is), X email processor instances run at the same time. If you need guarantee that each email is processed once only, this will screw it up for you. On Wed, Oct 7, 2009 at 11:05 AM, Yuri Niyazov yuri.niya...@gmail.com wrote: I haven't checked out the online cron services yet, but there's another issue that I had to solve, and I don't know whether they would support this or not: Heroku limits the execution time of every request to 30 seconds each, and a request that takes longer than that is abruptly interrupted. This means that the magic URL handler has to be written in such a way that it doesn't take longer than 30 secs; I decided to take the dirty-hack approach to this: the URL handler processes two emails at a time (let's say that 30 seconds is almost always enough to open an IMAP connection, do a search, and download the text of two emails). However, the URL handler checks the total number of messages to be processed, and returns a status code for same. So: upto = 2 msg_id_list = imap.search([NOT, DELETED]) msg_id_list = msg_id_list[0, upto] if upto msg_id_list.each do |msg_id| m = imap.fetch(msg_id, RFC822)[0].attr[RFC822] process m end render :json = msg_id_list.to_json and then in the script on the cron-box: do msg_id_list = call_url.parse_json until msg_id_list.empty? As far as the Google indexing your URL issue: make sure that the GET request returns a blank page, and the POST actually executes the cronjob. And, of course, you can always protect that URL via basic-auth or authenticity-token. On Wed, Oct 7, 2009 at 7:42 AM, Wojciech wojci...@oxos.pl wrote: so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. There are services that do it for you (i.e. periodically call your magic URL): http://www.onlinecronservices.com/ But be careful: this URL could be called by anybody and could even get indexed by Google. You might allow only certain IPs (ip of your online cron service) to call this URL to protect the app. There's also this poor man's cron approach, I've seen in Drupal: http://drupal.org/project/poormanscron- but it's a bit crazy. Cheers, Wojciech On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
It looks like sendgrid can POST the contents of an email to your app: http://wiki.sendgrid.com/doku.php?id=parse_api I haven't used this yet, just noticed it in the docs. Jim Gilliam http://act.ly/ http://twitter.com/jgilliam On Thu, Oct 8, 2009 at 6:04 AM, Chap c...@chap.otherinbox.com wrote: Oren, any thoughts on how to regularly receive email to a heroku app? On Oct 7, 2:50 pm, Oren Teich o...@heroku.com wrote: Cron runs on a separate single process. It doesn't matter how many dyno's you have, you'll only have one cron process ever running. If you're seeing other behavior, let us know! Oren On Oct 7, 2009, at 8:13 AM, Yuri Niyazov wrote: Also, I forgot the following fun fact about Heroku's cron service. This was true when I investigated it; might still be true now - not sure. Since your app runs on X Heroku VMs, where X is often 1, then, when you use Heroku's cron, the cronjob is executed on each box simultaneously - unless you do something clever (and I was unable to figure out what that something clever is), X email processor instances run at the same time. If you need guarantee that each email is processed once only, this will screw it up for you. On Wed, Oct 7, 2009 at 11:05 AM, Yuri Niyazov yuri.niya...@gmail.com wrote: I haven't checked out the online cron services yet, but there's another issue that I had to solve, and I don't know whether they would support this or not: Heroku limits the execution time of every request to 30 seconds each, and a request that takes longer than that is abruptly interrupted. This means that the magic URL handler has to be written in such a way that it doesn't take longer than 30 secs; I decided to take the dirty-hack approach to this: the URL handler processes two emails at a time (let's say that 30 seconds is almost always enough to open an IMAP connection, do a search, and download the text of two emails). However, the URL handler checks the total number of messages to be processed, and returns a status code for same. So: upto = 2 msg_id_list = imap.search([NOT, DELETED]) msg_id_list = msg_id_list[0, upto] if upto msg_id_list.each do |msg_id| m = imap.fetch(msg_id, RFC822)[0].attr[RFC822] process m end render :json = msg_id_list.to_json and then in the script on the cron-box: do msg_id_list = call_url.parse_json until msg_id_list.empty? As far as the Google indexing your URL issue: make sure that the GET request returns a blank page, and the POST actually executes the cronjob. And, of course, you can always protect that URL via basic-auth or authenticity-token. On Wed, Oct 7, 2009 at 7:42 AM, Wojciech wojci...@oxos.pl wrote: so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. There are services that do it for you (i.e. periodically call your magic URL): http://www.onlinecronservices.com/ But be careful: this URL could be called by anybody and could even get indexed by Google. You might allow only certain IPs (ip of your online cron service) to call this URL to protect the app. There's also this poor man's cron approach, I've seen in Drupal: http://drupal.org/project/poormanscron- but it's a bit crazy. Cheers, Wojciech On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. There are services that do it for you (i.e. periodically call your magic URL): http://www.onlinecronservices.com/ But be careful: this URL could be called by anybody and could even get indexed by Google. You might allow only certain IPs (ip of your online cron service) to call this URL to protect the app. There's also this poor man's cron approach, I've seen in Drupal: http://drupal.org/project/poormanscron - but it's a bit crazy. Cheers, Wojciech On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
I haven't checked out the online cron services yet, but there's another issue that I had to solve, and I don't know whether they would support this or not: Heroku limits the execution time of every request to 30 seconds each, and a request that takes longer than that is abruptly interrupted. This means that the magic URL handler has to be written in such a way that it doesn't take longer than 30 secs; I decided to take the dirty-hack approach to this: the URL handler processes two emails at a time (let's say that 30 seconds is almost always enough to open an IMAP connection, do a search, and download the text of two emails). However, the URL handler checks the total number of messages to be processed, and returns a status code for same. So: upto = 2 msg_id_list = imap.search([NOT, DELETED]) msg_id_list = msg_id_list[0, upto] if upto msg_id_list.each do |msg_id| m = imap.fetch(msg_id, RFC822)[0].attr[RFC822] process m end render :json = msg_id_list.to_json and then in the script on the cron-box: do msg_id_list = call_url.parse_json until msg_id_list.empty? As far as the Google indexing your URL issue: make sure that the GET request returns a blank page, and the POST actually executes the cronjob. And, of course, you can always protect that URL via basic-auth or authenticity-token. On Wed, Oct 7, 2009 at 7:42 AM, Wojciech wojci...@oxos.pl wrote: so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. There are services that do it for you (i.e. periodically call your magic URL): http://www.onlinecronservices.com/ But be careful: this URL could be called by anybody and could even get indexed by Google. You might allow only certain IPs (ip of your online cron service) to call this URL to protect the app. There's also this poor man's cron approach, I've seen in Drupal: http://drupal.org/project/poormanscron - but it's a bit crazy. Cheers, Wojciech On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
Also, I forgot the following fun fact about Heroku's cron service. This was true when I investigated it; might still be true now - not sure. Since your app runs on X Heroku VMs, where X is often 1, then, when you use Heroku's cron, the cronjob is executed on each box simultaneously - unless you do something clever (and I was unable to figure out what that something clever is), X email processor instances run at the same time. If you need guarantee that each email is processed once only, this will screw it up for you. On Wed, Oct 7, 2009 at 11:05 AM, Yuri Niyazov yuri.niya...@gmail.com wrote: I haven't checked out the online cron services yet, but there's another issue that I had to solve, and I don't know whether they would support this or not: Heroku limits the execution time of every request to 30 seconds each, and a request that takes longer than that is abruptly interrupted. This means that the magic URL handler has to be written in such a way that it doesn't take longer than 30 secs; I decided to take the dirty-hack approach to this: the URL handler processes two emails at a time (let's say that 30 seconds is almost always enough to open an IMAP connection, do a search, and download the text of two emails). However, the URL handler checks the total number of messages to be processed, and returns a status code for same. So: upto = 2 msg_id_list = imap.search([NOT, DELETED]) msg_id_list = msg_id_list[0, upto] if upto msg_id_list.each do |msg_id| m = imap.fetch(msg_id, RFC822)[0].attr[RFC822] process m end render :json = msg_id_list.to_json and then in the script on the cron-box: do msg_id_list = call_url.parse_json until msg_id_list.empty? As far as the Google indexing your URL issue: make sure that the GET request returns a blank page, and the POST actually executes the cronjob. And, of course, you can always protect that URL via basic-auth or authenticity-token. On Wed, Oct 7, 2009 at 7:42 AM, Wojciech wojci...@oxos.pl wrote: so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. There are services that do it for you (i.e. periodically call your magic URL): http://www.onlinecronservices.com/ But be careful: this URL could be called by anybody and could even get indexed by Google. You might allow only certain IPs (ip of your online cron service) to call this URL to protect the app. There's also this poor man's cron approach, I've seen in Drupal: http://drupal.org/project/poormanscron - but it's a bit crazy. Cheers, Wojciech On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
It almost looks like you should have cron on another box create Delayed Jobs to do this. This solves your 30 second timeout problem but introduces another complexity. Maybe this is a necessary complexity. As for the Google get issue, I would just not put a link on any page to the URL, and put a Disallow in your robots.txt User-agent: * Disallow: /process-emails.html Or just leave it out entirely, that way no one knows it exists. Security through obscurity is usually not good enough but in this case I am not sure that it matters. On Wed, Oct 7, 2009 at 11:13 AM, Yuri Niyazov yuri.niya...@gmail.com wrote: Also, I forgot the following fun fact about Heroku's cron service. This was true when I investigated it; might still be true now - not sure. Since your app runs on X Heroku VMs, where X is often 1, then, when you use Heroku's cron, the cronjob is executed on each box simultaneously - unless you do something clever (and I was unable to figure out what that something clever is), X email processor instances run at the same time. If you need guarantee that each email is processed once only, this will screw it up for you. On Wed, Oct 7, 2009 at 11:05 AM, Yuri Niyazov yuri.niya...@gmail.com wrote: I haven't checked out the online cron services yet, but there's another issue that I had to solve, and I don't know whether they would support this or not: Heroku limits the execution time of every request to 30 seconds each, and a request that takes longer than that is abruptly interrupted. This means that the magic URL handler has to be written in such a way that it doesn't take longer than 30 secs; I decided to take the dirty-hack approach to this: the URL handler processes two emails at a time (let's say that 30 seconds is almost always enough to open an IMAP connection, do a search, and download the text of two emails). However, the URL handler checks the total number of messages to be processed, and returns a status code for same. So: upto = 2 msg_id_list = imap.search([NOT, DELETED]) msg_id_list = msg_id_list[0, upto] if upto msg_id_list.each do |msg_id| m = imap.fetch(msg_id, RFC822)[0].attr[RFC822] process m end render :json = msg_id_list.to_json and then in the script on the cron-box: do msg_id_list = call_url.parse_json until msg_id_list.empty? As far as the Google indexing your URL issue: make sure that the GET request returns a blank page, and the POST actually executes the cronjob. And, of course, you can always protect that URL via basic-auth or authenticity-token. On Wed, Oct 7, 2009 at 7:42 AM, Wojciech wojci...@oxos.pl wrote: so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. There are services that do it for you (i.e. periodically call your magic URL): http://www.onlinecronservices.com/ But be careful: this URL could be called by anybody and could even get indexed by Google. You might allow only certain IPs (ip of your online cron service) to call this URL to protect the app. There's also this poor man's cron approach, I've seen in Drupal: http://drupal.org/project/poormanscron - but it's a bit crazy. Cheers, Wojciech On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
Cron runs on a separate single process. It doesn't matter how many dyno's you have, you'll only have one cron process ever running. If you're seeing other behavior, let us know! Oren On Oct 7, 2009, at 8:13 AM, Yuri Niyazov wrote: Also, I forgot the following fun fact about Heroku's cron service. This was true when I investigated it; might still be true now - not sure. Since your app runs on X Heroku VMs, where X is often 1, then, when you use Heroku's cron, the cronjob is executed on each box simultaneously - unless you do something clever (and I was unable to figure out what that something clever is), X email processor instances run at the same time. If you need guarantee that each email is processed once only, this will screw it up for you. On Wed, Oct 7, 2009 at 11:05 AM, Yuri Niyazov yuri.niya...@gmail.com wrote: I haven't checked out the online cron services yet, but there's another issue that I had to solve, and I don't know whether they would support this or not: Heroku limits the execution time of every request to 30 seconds each, and a request that takes longer than that is abruptly interrupted. This means that the magic URL handler has to be written in such a way that it doesn't take longer than 30 secs; I decided to take the dirty-hack approach to this: the URL handler processes two emails at a time (let's say that 30 seconds is almost always enough to open an IMAP connection, do a search, and download the text of two emails). However, the URL handler checks the total number of messages to be processed, and returns a status code for same. So: upto = 2 msg_id_list = imap.search([NOT, DELETED]) msg_id_list = msg_id_list[0, upto] if upto msg_id_list.each do |msg_id| m = imap.fetch(msg_id, RFC822)[0].attr[RFC822] process m end render :json = msg_id_list.to_json and then in the script on the cron-box: do msg_id_list = call_url.parse_json until msg_id_list.empty? As far as the Google indexing your URL issue: make sure that the GET request returns a blank page, and the POST actually executes the cronjob. And, of course, you can always protect that URL via basic-auth or authenticity-token. On Wed, Oct 7, 2009 at 7:42 AM, Wojciech wojci...@oxos.pl wrote: so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. There are services that do it for you (i.e. periodically call your magic URL): http://www.onlinecronservices.com/ But be careful: this URL could be called by anybody and could even get indexed by Google. You might allow only certain IPs (ip of your online cron service) to call this URL to protect the app. There's also this poor man's cron approach, I've seen in Drupal: http://drupal.org/project/poormanscron - but it's a bit crazy. Cheers, Wojciech On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Reading Email
Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
Re: Reading Email
Heroku comes with cron support, but only once an hour. I need it more often than that, so I have a separate box with actual crond on it, and it has a script that hits a specific URL on my app on heroku every x minutes to process email. On Tue, Oct 6, 2009 at 3:06 PM, Carl Fyffe carl.fy...@gmail.com wrote: Rails makes it so easy to send emails. Recieving emails isn't that difficult either, but requires a cron or daemon. What is the best way to do this on Heroku today? Carl --~--~-~--~~~---~--~~ You received this message because you are subscribed to the Google Groups Heroku group. To post to this group, send email to heroku@googlegroups.com To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/heroku?hl=en -~--~~~~--~~--~--~---
