There are a couple ways to do this, for example: POST request with payload -> Machine A within request, machine A takes the payload and adds a message with it to the queue, then returns with some status message, perhaps containing a reference to the task entry.
Machine B, upon pulling the message from the queue, processes the message, updates the database(s), updates caching layers (this is the tricky part), marks the message as processed, and perhaps sends a push notification to the originator of the POST request, probably through a common gateway, or the client just polls. For long file uploads, if you have sticky load balancing you just store the file chunks locally, once assembled upload it to s3 or something, then create a queue message that references the s3 file. If you don't have sticky load balancing, then you need to save the data chunks directly to s3, and keep some status info in redis or something. Note that this is not just for large file uploads, in a lot of cases you do this for normal database requests as well. In addition to that it can get very complicated very fast if you have intermediate caching layers. On Sat, Jul 7, 2012 at 1:39 PM, Angel Java Lopez <ajlopez2...@gmail.com>wrote: > Thanks! Hmm... interesting... but I missed some part, Martin. If I > understand your idea, the consumer(s) of the queue is in another machine. > That's is Ok, it compiles in my neuron ;-) My problem is about "POST > request". > > The initial POST request reach Machine A, from Client 1. At the end, it's > a socket connection, that http module gives to us as req/res. How to attend > that incoming request in Machine B? The client (maybe a browser) knows > Machine A, the open socket channel knows Machine A, not B, and it already > started to send POST data to it, maybe a file upload, I guess. The only > solution I can see: open a socket from A to B, and pipe to it the incoming > request (I apologize... my English, I'm struggling trying to express the > idea ;-). > > But I could misunderstand the initial question.. > > > On Sat, Jul 7, 2012 at 5:23 PM, Martin Wawrusch <mar...@wawrusch.com>wrote: > >> Here are some docs, just a piece of advice - this can easily result in >> overkill, only use when it is really necessary: >> >> http://www.soapatterns.org/synchronous_response_deferred_processing.php >> http://martinfowler.com/bliki/CQRS.html >> >> Also, it might be feasible to check out iron.io - iron worker product, >> which works with node.js too. It basically takes care of message queues, >> worker processes etc and is super fast (< 1s response time) >> >> @Angel >> You do this to prevent clogging of the queue on a single machine, and to >> easier distribute load, and to recover in case of error scenarios. >> >> >> On Sat, Jul 7, 2012 at 1:13 PM, Angel Java Lopez >> <ajlopez2...@gmail.com>wrote: >> >>> Just curious... >>> >>> Why you need to queue the POST request? >>> >>> An in-memory queue is enough? Or you need to process the POST request in >>> another process? (I can't imagine why/how) >>> >>> >>> >>> On Sat, Jul 7, 2012 at 5:04 PM, Alan Hoffmeister < >>> alanhoffmeis...@gmail.com> wrote: >>> >>>> The major problem is the logic, I had mastered amqp already. >>>> >>>> Em Jul 7, 2012, às 4:55 PM, Martin Wawrusch <mar...@wawrusch.com> >>>> escreveu: >>>> >>>> No docs that I know of, but take a look at the amqp npm module, it >>>> works fine with RabbitMQ (at least it used to a year back when I worked >>>> with it). >>>> >>>> Other then that it is simply a matter of defining a message format, >>>> pushing that onto the queue, and have some separate process(es) working the >>>> queue. >>>> >>>> >>>> >>>> On Sat, Jul 7, 2012 at 12:49 PM, Alan Hoffmeister < >>>> alanhoffmeis...@gmail.com> wrote: >>>> >>>>> Hello fellows, >>>>> >>>>> I want to queue my POST requests, someone known a link where I can >>>>> read more about requests, Node.js and queue manager (my preference is >>>>> RabbitMQ). >>>>> >>>>> Thanks! >>>>> >>>>> -- >>>>> Job Board: http://jobs.nodejs.org/ >>>>> Posting guidelines: >>>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>>>> You received this message because you are subscribed to the Google >>>>> Groups "nodejs" group. >>>>> To post to this group, send email to nodejs@googlegroups.com >>>>> To unsubscribe from this group, send email to >>>>> nodejs+unsubscr...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/nodejs?hl=en?hl=en >>>>> >>>> >>>> >>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: >>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to nodejs@googlegroups.com >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscr...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/nodejs?hl=en?hl=en >>>> >>>> -- >>>> Job Board: http://jobs.nodejs.org/ >>>> Posting guidelines: >>>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>>> You received this message because you are subscribed to the Google >>>> Groups "nodejs" group. >>>> To post to this group, send email to nodejs@googlegroups.com >>>> To unsubscribe from this group, send email to >>>> nodejs+unsubscr...@googlegroups.com >>>> For more options, visit this group at >>>> http://groups.google.com/group/nodejs?hl=en?hl=en >>>> >>> >>> -- >>> Job Board: http://jobs.nodejs.org/ >>> Posting guidelines: >>> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >>> You received this message because you are subscribed to the Google >>> Groups "nodejs" group. >>> To post to this group, send email to nodejs@googlegroups.com >>> To unsubscribe from this group, send email to >>> nodejs+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/nodejs?hl=en?hl=en >>> >> >> >> >> -- >> Job Board: http://jobs.nodejs.org/ >> Posting guidelines: >> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines >> You received this message because you are subscribed to the Google >> Groups "nodejs" group. >> To post to this group, send email to nodejs@googlegroups.com >> To unsubscribe from this group, send email to >> nodejs+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/nodejs?hl=en?hl=en >> > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
