Persistent key-value-store for D?
Hello everyone, After several unsuccessful attempts over the last couple of years, I made a new attempt at learning D last weekend and this time ... something clicked. So first of all, let me say that I like it very much so far! Over the years, I have fallen into the habit when learning a new programming language to rewrite a certain program I first started working on about 12 years ago. It's not so much that this program is super-useful (it really isn't), but that it is a good way to learn how a given programming language handles things like concurrency, data structures, networking, and user interfaces. One part of this program requires a DBM-like database library, preferably one that handles concurrent read-write-access as transparently as possible. In prior incarnations of this program I have used Berkeley DB and Tokyocabinet. I looked at the DUB package registry and asked Google quite a bit now, but I did not found such a package for D. So my first question is - did I not look hard enough? I found a reimplentation of QDBM [1] (the spiritual ancestor of Tokyocabinet), but it does not seem to handle concurrency at all. Are there other options along those lines? (If there was one that also provides transactions, that would be awesome!) If I understand what I have read so far correctly, it is possible to access libraries written in C or C++ from D - in that case, I could just use Tokyocabinet directly, but I have not found any pointers on how to do this. Is this a feasible option, and if so, where can I find documentation on how to do this? [1] http://forum.dlang.org/thread/mjinam$31cp$2...@digitalmars.com Thank you very much for any insights you feel like sharing with me, Benjamin
Re: Persistent key-value-store for D?
On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote: If I understand what I have read so far correctly, it is possible to access libraries written in C or C++ from D - in that case, I could just use Tokyocabinet directly, but I have not found any pointers on how to do this. Is this a feasible option, and if so, where can I find documentation on how to do this? Welcome to the D community. I've never heard of Tokyocabinet, but I did a search and the header files look like they'd be a straightforward translation to D, for instance http://bazaar.launchpad.net/~tokyocabinet/tokyocabinet/c99-posix-trunk/view/head:/tcutil.h You can call C libraries directly. I'd suggest trying dstep on the header files as a first step. https://github.com/jacob-carlborg/dstep These resources might help: https://wiki.dlang.org/Bind_D_to_C http://dlang.org/spec/interfaceToC.html You only need to write a binding for functions that you actually want to call, which might explain why nobody has made bindings available.
Re: Persistent key-value-store for D?
On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote: [...] If I understand what I have read so far correctly, it is possible to access libraries written in C or C++ from D - in that case, I could just use Tokyocabinet directly, but I have not found any pointers on how to do this. Is this a feasible option, and if so, where can I find documentation on how to do this? I can recommend using LMDB[1] which likely does all you want. It also has D bindings[2]. We use it in the AppStream generator with multithreaded parallelization and it is working great so far (although for the usecase of asgen it's not an ideal database choice, but one which makes deployments of the generator really simple). I explored other solutions including Tokyokabinet and LevelDB and LMDB was by far the fastest. Cheers, Matthias [1]: https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database [2]: https://github.com/ximion/appstream-generator/blob/master/src/asgen/bindings/lmdb.d
Re: Persistent key-value-store for D?
On Wednesday, 26 April 2017 at 19:07:22 UTC, bachmeier wrote: On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote: [...] Welcome to the D community. [...] You can call C libraries directly. I'd suggest trying dstep on the header files as a first step. https://github.com/jacob-carlborg/dstep These resources might help: https://wiki.dlang.org/Bind_D_to_C http://dlang.org/spec/interfaceToC.html You only need to write a binding for functions that you actually want to call, which might explain why nobody has made bindings available. Thank you! I will look into that!
Re: Persistent key-value-store for D?
On Thursday, 27 April 2017 at 01:56:12 UTC, Matthias Klumpp wrote: On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote: [...] If I understand what I have read so far correctly, it is possible to access libraries written in C or C++ from D - in that case, I could just use Tokyocabinet directly, but I have not found any pointers on how to do this. Is this a feasible option, and if so, where can I find documentation on how to do this? I can recommend using LMDB[1] which likely does all you want. It also has D bindings[2]. [...] Cheers, Matthias [1]: https://en.wikipedia.org/wiki/Lightning_Memory-Mapped_Database [2]: https://github.com/ximion/appstream-generator/blob/master/src/asgen/bindings/lmdb.d Thank you very much, that sounds exactly like what I was looking for!
Re: Persistent key-value-store for D?
On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote: I looked at the DUB package registry and asked Google quite a bit now, but I did not found such a package for D. So my first question is - did I not look hard enough? I found a reimplentation of QDBM [1] (the spiritual ancestor of Tokyocabinet), but it does not seem to handle concurrency at all. Are there other options along those lines? (If there was one that also provides transactions, that would be awesome!) If I understand what I have read so far correctly, it is possible to access libraries written in C or C++ from D - in that case, I could just use Tokyocabinet directly, but I have not found any pointers on how to do this. Is this a feasible option, and if so, where can I find documentation on how to do this? i recommend leveldb http://code.dlang.org/packages/d-leveldb its easy to use and mostly faster than tokyocabinet ( only very specifically tuned tokyo btrees outperform leveldb) i used above library with great success. it also shows you how to do c bindings.
Re: Persistent key-value-store for D?
On Friday, 28 April 2017 at 16:01:58 UTC, yawniek wrote: On Wednesday, 26 April 2017 at 17:06:52 UTC, krylon wrote: I looked at the DUB package registry and asked Google quite a bit now, but I did not found such a package for D. So my first question is - did I not look hard enough? I found a reimplentation of QDBM [1] (the spiritual ancestor of Tokyocabinet), but it does not seem to handle concurrency at all. Are there other options along those lines? (If there was one that also provides transactions, that would be awesome!) If I understand what I have read so far correctly, it is possible to access libraries written in C or C++ from D - in that case, I could just use Tokyocabinet directly, but I have not found any pointers on how to do this. Is this a feasible option, and if so, where can I find documentation on how to do this? i recommend leveldb http://code.dlang.org/packages/d-leveldb its easy to use and mostly faster than tokyocabinet ( only very specifically tuned tokyo btrees outperform leveldb) i used above library with great success. it also shows you how to do c bindings. Thank you for the suggestion! For the moment I am using LMDB, as suggested in another reply, and it works well. LevelDB seems to have a cleaner interface, though, at least in D. I'll keep that in mind for future projects.
