Anyone experienced in using clojure as a "database"
Folks, i would like advice, cause since im starting something that will be eventually big data intensive, from scratch and i really like the options already built in in clojure like STM, parallelizing data and concurrency logic implemented on it i think its wonderfully tuned to use as database... and you can achieve different strategies, like graphs or balanced trees for different data my worries are about the efficiency in serializing it and recovering from/to the disk, java VM as a hungry heap VM, (and with immutable strategies, more heap needed) the benefits are control of sharding mechanism's and parallelizing not only in the unit's cores but between nodes of a cluster.. anyone using clojure not just as database middleware(wich is perfect for) but as database backend too? Thanks all, Fabio Kaminski -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
You might want to look at FleetDB http://fleetdb.org/ http://github.com/mmcgrana/fleetdb On May 25, 2:08 pm, Fabio Kaminski wrote: > Folks, > > i would like advice, > cause since im starting something that will be eventually big data > intensive, from scratch > and i really like the options already built in in clojure like STM, > parallelizing data and concurrency logic implemented on it > i think its wonderfully tuned to use as database... and you can achieve > different strategies, like graphs or balanced trees for different data > > my worries are about the efficiency in serializing it and recovering from/to > the disk, > java VM as a hungry heap VM, > (and with immutable strategies, more heap needed) > > the benefits are control of sharding mechanism's and parallelizing not only > in the unit's cores but between nodes of a cluster.. > > anyone using clojure not just as database middleware(wich is perfect for) > but as database backend too? > > Thanks all, > > Fabio Kaminski > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group > athttp://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
On 05/25/2010 01:36 PM, Sean Devlin wrote: > You might want to look at FleetDB > > http://fleetdb.org/ > http://github.com/mmcgrana/fleetdb I have not found the system to be incredibly well documented, but I have discovered that if you want to embed fleetdb in your app, you will need (use 'fleetdb.embedded) And then create your "client" with init-ephemeral or init-persistent. The code for it is in fleetdb/embedded.clj. After that, it works as the client is illustrated in http://fleetdb.org/docs/introduction.html . I have played with very little of fleetdb so far, but I like what I see. -- Savanni -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
I'd love to see a persistent "table" type together with some common primitives (select, join, union) and optimization capabilities. Currently a "set of maps" does something like that but I have no idea how to, for example, add an "index" to some particular field and use it in other operations. Another thing is tying such a database to some on-disk storage. Is there any database backend that doesn't internally mutate data? Even if not, such a data type could still be useful as a read-only view for other databases. Andrzej On Wed, May 26, 2010 at 3:36 AM, Sean Devlin wrote: > You might want to look at FleetDB > > http://fleetdb.org/ > http://github.com/mmcgrana/fleetdb > > On May 25, 2:08 pm, Fabio Kaminski wrote: >> Folks, >> >> i would like advice, >> cause since im starting something that will be eventually big data >> intensive, from scratch >> and i really like the options already built in in clojure like STM, >> parallelizing data and concurrency logic implemented on it >> i think its wonderfully tuned to use as database... and you can achieve >> different strategies, like graphs or balanced trees for different data >> >> my worries are about the efficiency in serializing it and recovering from/to >> the disk, >> java VM as a hungry heap VM, >> (and with immutable strategies, more heap needed) >> >> the benefits are control of sharding mechanism's and parallelizing not only >> in the unit's cores but between nodes of a cluster.. >> >> anyone using clojure not just as database middleware(wich is perfect for) >> but as database backend too? >> >> Thanks all, >> >> Fabio Kaminski >> >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with your >> first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group >> athttp://groups.google.com/group/clojure?hl=en > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
Clojure is not a great choice for this. It's oriented as a programming language, not a database. It doesn't have any built-in persistence mechanisms, and while it has many virtues, it's a little bit of a memory hog. That isn't really what you want in an in-memory DB. For anything more than a toy app where you don't want to do the work, I think you'll be much more pleased with the results if you hook up a real DB or NoSQL solution. - Luke On May 25, 2:08 pm, Fabio Kaminski wrote: > Folks, > > i would like advice, > cause since im starting something that will be eventually big data > intensive, from scratch > and i really like the options already built in in clojure like STM, > parallelizing data and concurrency logic implemented on it > i think its wonderfully tuned to use as database... and you can achieve > different strategies, like graphs or balanced trees for different data > > my worries are about the efficiency in serializing it and recovering from/to > the disk, > java VM as a hungry heap VM, > (and with immutable strategies, more heap needed) > > the benefits are control of sharding mechanism's and parallelizing not only > in the unit's cores but between nodes of a cluster.. > > anyone using clojure not just as database middleware(wich is perfect for) > but as database backend too? > > Thanks all, > > Fabio Kaminski > > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with your > first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group > athttp://groups.google.com/group/clojure?hl=en -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
On 26 Mai, 15:24, Andrzej wrote: > I'd love to see a persistent "table" type together with some common > primitives (select, join, union) and optimization capabilities. > Currently a "set of maps" does something like that but I have no idea > how to, for example, add an "index" to some particular field and use > it in other operations. Got something around, this is my 3rd or 4th attempt to provide a relational datastructure for clojure. I've often found myself in situations, where I needed some kind of table and look up rows by more than one key, hashmaps don't fit that well. http://gist.github.com/415538 - basically is a hashmap which pretends being a set, and lets you access all 'rows'. Indexes (sorted, hash- map ...), except the primary one, are optional but will of course speed up lookup operations. I have not bothered about persistence yet, maybe in the future, there will be some interface to hook into the clojure STM, or Cells will be become available. Erik -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
On Thu, May 27, 2010 at 3:36 PM, Erik Söhnel wrote: > > Got something around, this is my 3rd or 4th attempt to provide a > relational datastructure for clojure. I've often found myself in > situations, where I needed some kind of table and look up rows by more > than one key, hashmaps don't fit that well. They are simply different abstractions. Perhaps at some point general purpose languages will start embedding table/relation types, just like it has happened for hashmaps or sets. There is no "one size fits all" but there is still a merit for having a reasonable default implementation deeply embedded in the language (just like seqs, vectors, maps are in Clojure), especially if they happen to free people from thinking about side effects. > http://gist.github.com/415538 - basically is a hashmap which pretends > being a set, and lets you access all 'rows'. Indexes (sorted, hash- > map ...), except the primary one, are optional but will of course > speed up lookup operations. > I have not bothered about persistence yet, maybe in the future, there > will be some interface to hook into the clojure STM, or Cells will be > become available. I haven't had a chance to play with your code yet but at first glance it looks good. Does the above mean that all set operations are automatically supported and the accelerated? Do you think it is possible to implement join/union etc so that they produce lazy relations (or just lazy-seq of rows)? That would be very useful in many common use cases (instead of cursors). As for the storage persistence - that's a low priority (at least for me), but it would be good to have at least some method for tying a relation to such storage (like a "tie" function in Perl). Can you tell me more about these hooks to Clojure STM and Cells? Thanks, Andrzej -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
> I haven't had a chance to play with your code yet but at first glance > it looks good. Does the above mean that all set operations are > automatically supported and the accelerated? Unfortunately not for all operations. For example, clojure.set/join uses its own index function, which builds an index for every join invokation. On the other hand, difference, union and intersection use the primary-key index (through .equals), which may be a bit faster than hashing whole tuples. > Do you think it is possible to implement join/union etc so that they > produce lazy relations (or just lazy-seq of rows)? That would be very > useful in many common use cases (instead of cursors). I tried it once, but I was too ambigious. Providing lazy relations and hiding the implementation details (SQL or clojure) while wrapping it in a relational algebra API just didn't work out to be practical :). Maybe that snippet is a good foundation to start a more humble approach. But with clojure you get already a general-purpose, lazy select and project operations (filter and map). Adding lazy joins and set-operations isn't that hard. Making them efficient beyond basic index usage is harder. > As for the storage persistence - that's a low priority (at least for > me), but it would be good to have at least some method for tying a > relation to such storage (like a "tie" function in Perl). > Can you tell me more about these hooks to Clojure STM and Cells? Currently, there are no Hooks, and Cells are just an idea: http://www.assembla.com/wiki/show/clojure/Cells http://groups.google.com/group/clojure/browse_thread/thread/aa22a709501a64ac Erik -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
Hi Erik, I have a question about primary keys. As far as I can see you're currently using the first field of the relation as a primary key. While that's what other databases do (and it is working well), I think it would be better to make _records_ themselves primary keys. Since records are immutable they are guaranteed to be unique and comparing them is essentially free. I have a feeling that would make whole lot of things much easier (no need to check the primary key field for uniqueness, no need to implement auto increment mechanisms, foreign keys would just become actual records). What do you think about it? Andrzej -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
Re: Anyone experienced in using clojure as a "database"
Hi Andrzej, > I have a question about primary keys. As far as I can see you're > currently using the first field of the relation as a primary key. > While that's what other databases do (and it is working well), I think > it would be better to make _records_ themselves primary keys. Since > records are immutable they are guaranteed to be unique and comparing > them is essentially free. Right, and my first "iteration" did exactly this (because I wanted to stay "pure" relational), but it turned out not to be that practical. My #1 issue with primary-key-less relations was, how do I update them efficiently (efficient in terms of LOC and code-complexity, not runtime efficiency)?. To alter a value of a tuple, I had to store the tuple as a reference in every place which initiates an update. That made debugging and working on the repl an extreme unpleasant task. It also made commuting updates hard, since there was no "stable" primary- key to rely on. Additionaly, while praktically working with relations, I hardly found any dataset wich hadn't some kind of unique id. > I have a feeling that would make whole lot > of things much easier (no need to check the primary key field for > uniqueness, no need to implement auto increment mechanisms, foreign > keys would just become actual records). What do you think about it? The good news is, that a relation without a "single" primary key field, is a relation, where every field belongs to the primary-key. Of course, indexes over mutliple fields are currently not implemented, but I do consier this an easy task (just look at the clojure.set/index function, which will generate an index for one or many given fields). Erik -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en
