If the amount of data is small enough that it can all be kept in ram, a simple custom database using large node buffers would work great.
I would use msgpack or some other compact serialization to store the data in the buffer efficiently. If it's too large for memory, then a single append-only file can work as well. As far as performance, I expect the serialization to be the CPU bottleneck and bandwidth the network bottleneck. If your browser clients are new enough browsers, I would use raw binary websockets with msgpack and avoid a lot of the costs since the data can be copied directly into the store. It would use a lot less bandwidth than JSON over socket.io as well. Feel free to contact me off-list if you want to go down this path. On Tue, Dec 18, 2012 at 2:10 PM, Mark Hahn <m...@hahnca.com> wrote: > @Ciprian, et. al. Thanks very much for the detailed list. I have > investigated about half of those and will look into the others. > > I apologize if I repeat myself but here are my detailed requirements ... > > 0) Installable with npm > 1) Very high bandwidth, my app can use as much as you give it > 2) Multi-writers, including browsers and other servers > 3) One reader on a single server > 4) No channels are needed, my data encapsulates any addressing > 5) Ordered data delivery would be nice but I could handle order > problems in my code > 6) Reliable delivery would be nice not not critical. End use of data > is data-mining to be read by humans and I can show the human an > indication data is missing. It would make my app look bad though. > > The solutions I have studied seem to be overkill for my needs > (rabbitMQ) or more complex enterprise solutions that I'd not want each > user to have to install. > > So unless any of the other solutions mentioned fill the bill, I will > proceed with rolling my own using socket.io and homemade storage. > > P.S. I have discovered 3 or 4 messaging-related projects at apache. > That is weird. > > > On Tue, Dec 18, 2012 at 7:17 AM, Chad Engler <chad.eng...@patlive.com> > wrote: > > Do it in IndexedDB so it can translate to the browser :P > > > > -Chad > > > > -----Original Message----- > > From: nodejs@googlegroups.com [mailto:nodejs@googlegroups.com] On Behalf > > Of Ciprian Dorin Craciun > > Sent: Tuesday, December 18, 2012 6:44 AM > > To: nodejs@googlegroups.com > > Subject: Re: [nodejs] best distributed multi-writer, single-reader > > store? > > > > On Tue, Dec 18, 2012 at 2:24 AM, Mark Hahn <m...@hahnca.com> wrote: > >> I have an app where many servers will need to write to a store but > >> only one server will be reading the store. The data will be somewhat > >> transitory so an in-memory store might make sense. > > > > Do you need the "store" to be distributed or a single instance might > > work? What is the expected load (messages per second, message size, > > total connections, etc.?) What do you use it for (general pointers). > > > > If you are interested in message queues there are a few options, all > > with tradeoffs: > > * http://www.rabbitmq.com/ -- implements AMQP, maybe the most mature > > and complex (or flexible) solution; > > * https://github.com/robey/kestrel (used by Twitter?) -- implements > > a memcache-based protocol (thus usable from almost any programming > > language via a memcache client library); > > * https://github.com/bitly/nsq (used by BitLy) -- implements a > > custom memcache-like protocol; > > * a dozen others (see > > http://wiki.volution.ro/Mosaic/Notes/Solutions#Message_queues for some > > that I've tried to "categorize"); > > > > But you could also use something based on a embedded database like > > BerkeleyDB, LevelDB, etc. Maybe look at TokyoDB which has both a > > memcache-based or HTTP-based access protocol. > > > > Hope it helps, > > Ciprian. > > > > -- > > Job Board: http://jobs.nodejs.org/ > > Posting guidelines: > > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. > > To post to this group, send email to nodejs@googlegroups.com To > > unsubscribe from this group, send email to > > nodejs+unsubscr...@googlegroups.com > > For more options, visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > > > -- > > Job Board: http://jobs.nodejs.org/ > > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > > You received this message because you are subscribed to the Google > > Groups "nodejs" group. > > To post to this group, send email to nodejs@googlegroups.com > > To unsubscribe from this group, send email to > > nodejs+unsubscr...@googlegroups.com > > For more options, visit this group at > > http://groups.google.com/group/nodejs?hl=en?hl=en > > -- > Job Board: http://jobs.nodejs.org/ > Posting guidelines: > https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines > You received this message because you are subscribed to the Google > Groups "nodejs" group. > To post to this group, send email to nodejs@googlegroups.com > To unsubscribe from this group, send email to > nodejs+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/nodejs?hl=en?hl=en > -- Job Board: http://jobs.nodejs.org/ Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines You received this message because you are subscribed to the Google Groups "nodejs" group. To post to this group, send email to nodejs@googlegroups.com To unsubscribe from this group, send email to nodejs+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/nodejs?hl=en?hl=en
