Interesting. This seems like a pretty promising direction for the bottom of the big-data stack.
A use case on my mind is sorting a big list of datastructures by key (or some set of keys/paths) . Once the data gets big, you need to do an external sort, which means tons of serialization round trips if implemented naively. Being able to just pluck out the values you need really helps in that case. Besides saving on the serialization overhead, it also cuts down on memory which means you can sort much bigger segments at a time, and complete the overall sort in fewer passes. On Mon, Jul 15, 2013 at 1:40 PM, Zach Tellman <ztell...@gmail.com> wrote: > If you (vertigo.core/wrap "a-file-name"), that will use mmap under the > covers, so if no one's tried it, it's easy enough to start. > > With respect to non-fixed data layouts, that could be supported by a > library which parses the framing information, and then layers Vertigo atop > the actual data. In effect, that's what Gloss [1] is going to become, so > keep watching the skies. > > Zach > > [1] https://github.com/ztellman/gloss > > > On Sun, Jul 14, 2013 at 9:16 PM, kovas boguta <kovas.bog...@gmail.com>wrote: > >> This is pretty neat. >> >> Anyone try using this in conjunction with mmap? >> >> It would be nice to have some way to deal with strings & other >> variable-length data. >> >> I'm also curious if its possible to make the analog of this for fressian, >> basically to avoid unpacking objects that are not necessary for the >> computation at hand. >> >> >> >> >> >> >> On Tue, Jul 9, 2013 at 8:56 PM, Zach Tellman <ztell...@gmail.com> wrote: >> >>> Last year, I gave a talk at the Conj on my attempt to write an AI for >>> the board game Go. Two things I discovered is that it was hard to get >>> predictable performance, but even once I made sure I had all the right type >>> hints, there was still a lot of room at the bottom for performance >>> improvements. Towards the end [1], I mentioned a few ideas for >>> improvements, one of which was simply using ByteBuffers rather than objects >>> to host the data. This would remove all the levels of indirection, giving >>> much better cache coherency, and also allow for fast unsynchronized >>> mutability when the situation called for it. >>> >>> So, ten months and several supporting libraries [2] [3] later, here it >>> is: https://github.com/ztellman/vertigo >>> >>> At a high level, this library is useful whenever your datatype has a >>> fixed layout and is used more than once. Depending on your type, it will >>> give you moderate to large memory savings, and if you're willing to forgo >>> some of core library in favor of Vertigo's operators, you can get >>> significant performance gains on batch operations. And, in the cases where >>> performance doesn't matter, it will behave exactly like any other Clojure >>> data structure. >>> >>> I want to point out that something like this would be more or less >>> impossible in Java; reading from an offset in a ByteBuffer without the >>> compile-time inference and validation provided by this library would be >>> pointlessly risky. There's not a lot of low-level Clojure libraries, but >>> there's an increasing amount of production usage where people are using >>> Clojure for performance-sensitive work. I'm looking forward to seeing what >>> people do with Vertigo and libraries like it. >>> >>> Zach >>> >>> [1] >>> http://www.youtube.com/watch?feature=player_detailpage&v=v5dYE0CMmHQ#t=1828s >>> [2] https://github.com/ztellman/primitive-math >>> [3] https://github.com/ztellman/byte-streams >>> >>> -- >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To post to this group, send email to clojure@googlegroups.com >>> Note that posts from new members are moderated - please be patient with >>> your first post. >>> To unsubscribe from this group, send email to >>> clojure+unsubscr...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/clojure?hl=en >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "Clojure" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to clojure+unsubscr...@googlegroups.com. >>> >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> >> -- >> -- >> You received this message because you are subscribed to the Google >> Groups "Clojure" group. >> To post to this group, send email to clojure@googlegroups.com >> Note that posts from new members are moderated - please be patient with >> your first post. >> To unsubscribe from this group, send email to >> clojure+unsubscr...@googlegroups.com >> For more options, visit this group at >> http://groups.google.com/group/clojure?hl=en >> --- >> You received this message because you are subscribed to a topic in the >> Google Groups "Clojure" group. >> To unsubscribe from this topic, visit >> https://groups.google.com/d/topic/clojure/BayfuaqMzvs/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> clojure+unsubscr...@googlegroups.com. >> >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > > -- > -- > You received this message because you are subscribed to the Google > Groups "Clojure" group. > To post to this group, send email to clojure@googlegroups.com > Note that posts from new members are moderated - please be patient with > your first post. > To unsubscribe from this group, send email to > clojure+unsubscr...@googlegroups.com > For more options, visit this group at > http://groups.google.com/group/clojure?hl=en > --- > You received this message because you are subscribed to the Google Groups > "Clojure" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to clojure+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. > > > -- -- You received this message because you are subscribed to the Google Groups "Clojure" group. To post to this group, send email to clojure@googlegroups.com Note that posts from new members are moderated - please be patient with your first post. To unsubscribe from this group, send email to clojure+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/clojure?hl=en --- You received this message because you are subscribed to the Google Groups "Clojure" group. To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
