Re: Block On File Upload
A cautionary tale: We recently attempted a migration from Paperclip to Carrierwave (0.4.10). Despite working fine locally, we hit a snag as soon as we went live with it in production: Carrierwave would block for inordinate amounts of time. Never found out why, it just did. Heroku would timeout the connection after 30 seconds, but the dyno handling it would get stuck for up to half an hour. Rinse and repeat and a large amount of our dynos would end up blocked. The Heroku router (HAProxy, I am guessing) doesn't seem to keep tabs on the blocked dynos, so it would keep proxying requests to them, effectively timing out and giving the dreaded H12 error in the logs. Even though we had a few dynos still available, most requests would hit the blocked dynos and timeout. The situation would clear up after 30 mins or so, when their reaper process would kill the dynos (seems like they are using Libratos' Silverline). Or we would have to manually restart the server ourselves with a `heroku restart` Moral of the story: File upload is a delicate operation, even more so in Heroku. Do your best to monitor your service and understand how the underlying stack works. When possible upload straight to S3. :) -christos On 17 Jan 2011, at 00:18, Oren Teich wrote: > A dyno is the entire stack. A request, the second it hits Heroku, > blocks a dyno. This could be because your application is processing, > or because you are handling an upload. > > If you are doing decent size upload, you should have your app upload > directly to S3. > > Oren > > On Sun, Jan 16, 2011 at 12:33 PM, John Maxwell wrote: >> Yeah - that is talking about the outgoing hop to S3 from Heroku - which is a >> blocking I/O, not the incoming step into Nginx... Any word on this from an >> official source? >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Heroku" group. >> To post to this group, send email to heroku@googlegroups.com. >> To unsubscribe from this group, send email to >> heroku+unsubscr...@googlegroups.com. >> For more options, visit this group at >> http://groups.google.com/group/heroku?hl=en. >> > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
Hi, folks. In my app, when the server was uploading a file to S3 (fired from a first request), we got a request timeout error from heroku when the app received a second request. The solution was to use DJ in order to not block the dyno, and manage the upload from the job in the worker in background, and updating the upload status from the front-end querying the DB whenever you consider. We solved the problem this way. Hope it helps. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
Right, that's more of a limitation that I thought apologies for being wrong (and sure I wasn't!) - I'm not massively familiar with Rack, but could you use a Rack Middleware to detect a file upload and spin up another dyno whilst it is in progress? Or is that not detectable within Rack? -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
A dyno is the entire stack. A request, the second it hits Heroku, blocks a dyno. This could be because your application is processing, or because you are handling an upload. If you are doing decent size upload, you should have your app upload directly to S3. Oren On Sun, Jan 16, 2011 at 12:33 PM, John Maxwell wrote: > Yeah - that is talking about the outgoing hop to S3 from Heroku - which is a > blocking I/O, not the incoming step into Nginx... Any word on this from an > official source? > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
Yeah - that is talking about the outgoing hop to S3 from Heroku - which is a blocking I/O, not the incoming step into Nginx... Any word on this from an official source? -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
there's a bit at the bottom of this page http://docs.heroku.com/s3 that may give an additional clue "This is the preferred approach if you’re working with file uploads bigger than 4mbs. The idea is to skip the hop on your application dyno, making a connection from the end user browser straight to S3." John. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
Having searched the docs for confirmation on this, there isn't anything which explicitly says what the situation is. However, unless I'm being a plonker, which is possible - then nginx only reroutes the request to the dyno once the request is complete - ie the file has been received. Otherwise you'd be streaming data into the dyno, which is not how file uploads work! The dyno would obviously be blocked whilst processing the upload, and transferring to S3 unless you do that in the background, but actually uploading itself shouldn't block it, surely? -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
I wouldn't call it rerouting, but it is likely a response to a TCP transaction. Nginx is just a reverse proxy routing requests to the dynos in the first place. This is what large scale deployments do anyway, so it's nothing new. By the way, this is nothing that isn't written on the Heroku docs.. They explain it much better than I could. Hope that helps, Jimmy On Jan 15, 2011, at 7:47 PM, John Maxwell wrote: > So the dyno then reroutes requests back to Nginx? really? > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
So the dyno then reroutes requests back to Nginx? really? -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
I think it's more precise to say that a dyno is a thin+rack combo, so the dyno *is* the web server receiving the request. Jimmy On Jan 15, 2011, at 2:54 PM, John Maxwell wrote: > So an upload to your app on heroku blocks the whole dyno whilst it is > uploading? Surely that isn't right - as the dyno shouldn't even be aware of > the post request until the web server itself has received the whole file? In > that case its just blocked whilst the app processes it? > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
So an upload to your app on heroku blocks the whole dyno whilst it is uploading? Surely that isn't right - as the dyno shouldn't even be aware of the post request until the web server itself has received the whole file? In that case its just blocked whilst the app processes it? -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
Hi Tobes, Not sure if this overlaps, but something else to consider: http://docs.amazonwebservices.com/AmazonS3/latest/dev/index.html?uploadobjusingmpu.html Good luck, Keenan On Jan 13, 2011, at 3:33 PM, Jimmy Thrasher wrote: > Hi Tobes, > > Unless they've removed the 30s hard limit, you should see those requests > dying after 30s, but yes, in theory if it takes 10 minutes it blocks your > upload. > > An approach I like but have never had the need to implement would be to use > the "upload by POST" technique, described here: > http://aws.amazon.com/articles/1434?_encoding=UTF8&jiveRedirect=1 > > You basically create a signed URL with an expiration date which allows a user > to upload to a bucket you have rights to. There are third party utilities, > often written in Flash, to help you accomplish this. > > Hope that helps, > > Jimmy > > On Thu, Jan 13, 2011 at 3:28 PM, Tobes wrote: > Hi guys > > Sorry, this is a lazy question... > > If we have 1 dyno, and a user uploads a file that takes 10 minutes, > will the other requests to the app be blocked for 10 minutes? > > Looking at our logs it looks like that *is* the case. We do store > files on S3 btw, but our Rails app receives the file and moves it to > S3. It's the blocking we're interested in knowing about (I know Rails > doesn't block usually) > > Tobin > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > > > > -- > +1-919-627-7546 > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Re: Block On File Upload
Hi Tobes, Unless they've removed the 30s hard limit, you should see those requests dying after 30s, but yes, in theory if it takes 10 minutes it blocks your upload. An approach I like but have never had the need to implement would be to use the "upload by POST" technique, described here: http://aws.amazon.com/articles/1434?_encoding=UTF8&jiveRedirect=1 You basically create a signed URL with an expiration date which allows a user to upload to a bucket you have rights to. There are third party utilities, often written in Flash, to help you accomplish this. Hope that helps, Jimmy On Thu, Jan 13, 2011 at 3:28 PM, Tobes wrote: > Hi guys > > Sorry, this is a lazy question... > > If we have 1 dyno, and a user uploads a file that takes 10 minutes, > will the other requests to the app be blocked for 10 minutes? > > Looking at our logs it looks like that *is* the case. We do store > files on S3 btw, but our Rails app receives the file and moves it to > S3. It's the blocking we're interested in knowing about (I know Rails > doesn't block usually) > > Tobin > > -- > You received this message because you are subscribed to the Google Groups > "Heroku" group. > To post to this group, send email to heroku@googlegroups.com. > To unsubscribe from this group, send email to > heroku+unsubscr...@googlegroups.com > . > For more options, visit this group at > http://groups.google.com/group/heroku?hl=en. > > -- +1-919-627-7546 -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
Block On File Upload
Hi guys Sorry, this is a lazy question... If we have 1 dyno, and a user uploads a file that takes 10 minutes, will the other requests to the app be blocked for 10 minutes? Looking at our logs it looks like that *is* the case. We do store files on S3 btw, but our Rails app receives the file and moves it to S3. It's the blocking we're interested in knowing about (I know Rails doesn't block usually) Tobin -- You received this message because you are subscribed to the Google Groups "Heroku" group. To post to this group, send email to heroku@googlegroups.com. To unsubscribe from this group, send email to heroku+unsubscr...@googlegroups.com. For more options, visit this group at http://groups.google.com/group/heroku?hl=en.
