Re: [ClojureScript] Re: durable datascript experiments

2018-01-06 Thread Alan Moore 
Christian,

Great to see the progress on this! I’ll join you again on the gitter channel as 
there are some things that might be complimentary, ie. holochain.org. TBD.

Alan

-- 
Note that posts from new members are moderated - please be patient with your 
first post.
--- 
You received this message because you are subscribed to the Google Groups 
"ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojurescript+unsubscr...@googlegroups.com.
To post to this group, send email to clojurescript@googlegroups.com.
Visit this group at https://groups.google.com/group/clojurescript.

Re: [ClojureScript] Re: durable datascript experiments

2018-01-05 Thread Christian Weilbach 
It now has its own repo as datahike and is on clojars to play around.

https://github.com/replikativ/datahike

We are also working on replication of the hitchhiker-tree here:

https://gitlab.com/replikativ/index-sync

So hopefully we will be able to stream replicate a datahike db soon.

Best,
Christian

On 31.12.2017 00:13, Laurens Van Houtven wrote:
> Whoa; this looks awesome. Thanks for publishing this.
> 
> On Sat, Dec 30, 2017 at 2:24 PM, Christian Weilbach
> mailto:whitesp...@polyc0l0r.net>> wrote:
> 
> Hi,
> 
> performance is now a lot better, ~3-5x slower than datascript for
> queries on the fractal index trees on my machine. This still needs
> tuning ofc., but I have tried to make it fast with a lot of profiling
> during idle times at 34C3. The fractal tree causes some overhead though.
> Inserts are a lot more expensive, but I expect them to be written to
> disk anyway. Therefore comparing inserts/deletes with datascript is a
> bit pointless, if you do everything in memory then it will always be
> more performant.
> 
> So it should be usable for small test projects now. GC is still TODO,
> but I would be really happy for feedback! We are developing this as a
> toolbox for distributed dataprocessing architectures.
> 
> Best,
> Christian
> 
> On 26.12.2017 12:18, Christian Weilbach wrote:
> > Hi,
> >
> > I have finally taken the time yesterday in a good end-of-year
> tradition
> > of ambitious hacks to bring the hitchhiker-tree (1) and datascript (2)
> > together. I have only touched the db.cljc namespace in datascript and
> > replaced all calls to the in-memory balanced-tree set (btset) with
> calls
> > to the hitchhiker-tree, the rest is vanilla DataScript. All tests pass
> > and I can store and load databases sufficiently with the three
> trees for
> > the indices (no datascript serialization used).
> >
> > I would like to have early feedback, so feel free to play and throw
> > ideas around. You need to install my version of the
> hitchhiker-tree (1)
> > first. Atm. it is a drop-in replacement for DataScript, but this is a
> > prototype, don't use it for anything serious yet.
> >
> > Here is an example interaction with the store:
> >
> >
> 
> https://github.com/whilo/datascript/blob/hitchhiker_tree_support/test/datascript/test/store.cljc
> 
> 
> >
> > Caveats:
> > 1. There is no working GC for the hitchhiker-trees yet.
> > 2. Performance penalty against DataScript is in the 20-100x range,
> but I
> > haven't done serious profiling yet.
> > 3. There are probably bugs that are not covered by the tests, please
> > open an issue.
> >
> > Goals:
> > This is more of an intermediary goal, although it is very useful
> to have
> > a durable datascript around in itself. The hitchhiker-tree is a
> generic
> > persistent durable datastructure that can be efficiently synched and
> > hence replication is the goal. Next step is a "web-after-tomorrow" (3)
> > architecture similar to datsync, that we are discussing about here:
> >
> > https://gitter.im/metasoarous/datsync
> 
> > https://gitter.im/replikativ/replikativ
> 
> >
> > In the longer run we can allow multiple writers with different
> > conflict-resolution scheme similar to CRDTs and the bloom language
> > (dedalus).
> >
> >
> > Happy hacking :),
> > Christian
> >
> > (1) https://github.com/whilo/hitchhiker-tree
> 
> > (2)
> https://github.com/whilo/datascript/tree/hitchhiker_tree_support
> 
> > (3) http://tonsky.me/blog/the-web-after-tomorrow/
> 
> >
> 
> 
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to cloj...@googlegroups.com
> 
> Note that posts from new members are moderated - please be patient
> with your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> 
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> 
> ---
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to clojure+unsubscr...@googlegroups.com
>

[ClojureScript] Re: durable datascript experiments

2017-12-30 Thread Laurens Van Houtven 
Whoa; this looks awesome. Thanks for publishing this.

On Sat, Dec 30, 2017 at 2:24 PM, Christian Weilbach <
whitesp...@polyc0l0r.net> wrote:

> Hi,
>
> performance is now a lot better, ~3-5x slower than datascript for
> queries on the fractal index trees on my machine. This still needs
> tuning ofc., but I have tried to make it fast with a lot of profiling
> during idle times at 34C3. The fractal tree causes some overhead though.
> Inserts are a lot more expensive, but I expect them to be written to
> disk anyway. Therefore comparing inserts/deletes with datascript is a
> bit pointless, if you do everything in memory then it will always be
> more performant.
>
> So it should be usable for small test projects now. GC is still TODO,
> but I would be really happy for feedback! We are developing this as a
> toolbox for distributed dataprocessing architectures.
>
> Best,
> Christian
>
> On 26.12.2017 12:18, Christian Weilbach wrote:
> > Hi,
> >
> > I have finally taken the time yesterday in a good end-of-year tradition
> > of ambitious hacks to bring the hitchhiker-tree (1) and datascript (2)
> > together. I have only touched the db.cljc namespace in datascript and
> > replaced all calls to the in-memory balanced-tree set (btset) with calls
> > to the hitchhiker-tree, the rest is vanilla DataScript. All tests pass
> > and I can store and load databases sufficiently with the three trees for
> > the indices (no datascript serialization used).
> >
> > I would like to have early feedback, so feel free to play and throw
> > ideas around. You need to install my version of the hitchhiker-tree (1)
> > first. Atm. it is a drop-in replacement for DataScript, but this is a
> > prototype, don't use it for anything serious yet.
> >
> > Here is an example interaction with the store:
> >
> > https://github.com/whilo/datascript/blob/hitchhiker_
> tree_support/test/datascript/test/store.cljc
> >
> > Caveats:
> > 1. There is no working GC for the hitchhiker-trees yet.
> > 2. Performance penalty against DataScript is in the 20-100x range, but I
> > haven't done serious profiling yet.
> > 3. There are probably bugs that are not covered by the tests, please
> > open an issue.
> >
> > Goals:
> > This is more of an intermediary goal, although it is very useful to have
> > a durable datascript around in itself. The hitchhiker-tree is a generic
> > persistent durable datastructure that can be efficiently synched and
> > hence replication is the goal. Next step is a "web-after-tomorrow" (3)
> > architecture similar to datsync, that we are discussing about here:
> >
> > https://gitter.im/metasoarous/datsync
> > https://gitter.im/replikativ/replikativ
> >
> > In the longer run we can allow multiple writers with different
> > conflict-resolution scheme similar to CRDTs and the bloom language
> > (dedalus).
> >
> >
> > Happy hacking :),
> > Christian
> >
> > (1) https://github.com/whilo/hitchhiker-tree
> > (2) https://github.com/whilo/datascript/tree/hitchhiker_tree_support
> > (3) http://tonsky.me/blog/the-web-after-tomorrow/
> >
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "Clojure" group.
> To post to this group, send email to cloj...@googlegroups.com
> Note that posts from new members are moderated - please be patient with
> your first post.
> To unsubscribe from this group, send email to
> clojure+unsubscr...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/clojure?hl=en
> ---
> You received this message because you are subscribed to the Google Groups
> "Clojure" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to clojure+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
Note that posts from new members are moderated - please be patient with your 
first post.
--- 
You received this message because you are subscribed to the Google Groups 
"ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojurescript+unsubscr...@googlegroups.com.
To post to this group, send email to clojurescript@googlegroups.com.
Visit this group at https://groups.google.com/group/clojurescript.

[ClojureScript] Re: durable datascript experiments

2017-12-30 Thread Christian Weilbach 
Hi,

performance is now a lot better, ~3-5x slower than datascript for
queries on the fractal index trees on my machine. This still needs
tuning ofc., but I have tried to make it fast with a lot of profiling
during idle times at 34C3. The fractal tree causes some overhead though.
Inserts are a lot more expensive, but I expect them to be written to
disk anyway. Therefore comparing inserts/deletes with datascript is a
bit pointless, if you do everything in memory then it will always be
more performant.

So it should be usable for small test projects now. GC is still TODO,
but I would be really happy for feedback! We are developing this as a
toolbox for distributed dataprocessing architectures.

Best,
Christian

On 26.12.2017 12:18, Christian Weilbach wrote:
> Hi,
> 
> I have finally taken the time yesterday in a good end-of-year tradition
> of ambitious hacks to bring the hitchhiker-tree (1) and datascript (2)
> together. I have only touched the db.cljc namespace in datascript and
> replaced all calls to the in-memory balanced-tree set (btset) with calls
> to the hitchhiker-tree, the rest is vanilla DataScript. All tests pass
> and I can store and load databases sufficiently with the three trees for
> the indices (no datascript serialization used).
> 
> I would like to have early feedback, so feel free to play and throw
> ideas around. You need to install my version of the hitchhiker-tree (1)
> first. Atm. it is a drop-in replacement for DataScript, but this is a
> prototype, don't use it for anything serious yet.
> 
> Here is an example interaction with the store:
> 
> https://github.com/whilo/datascript/blob/hitchhiker_tree_support/test/datascript/test/store.cljc
> 
> Caveats:
> 1. There is no working GC for the hitchhiker-trees yet.
> 2. Performance penalty against DataScript is in the 20-100x range, but I
> haven't done serious profiling yet.
> 3. There are probably bugs that are not covered by the tests, please
> open an issue.
> 
> Goals:
> This is more of an intermediary goal, although it is very useful to have
> a durable datascript around in itself. The hitchhiker-tree is a generic
> persistent durable datastructure that can be efficiently synched and
> hence replication is the goal. Next step is a "web-after-tomorrow" (3)
> architecture similar to datsync, that we are discussing about here:
> 
> https://gitter.im/metasoarous/datsync
> https://gitter.im/replikativ/replikativ
> 
> In the longer run we can allow multiple writers with different
> conflict-resolution scheme similar to CRDTs and the bloom language
> (dedalus).
> 
> 
> Happy hacking :),
> Christian
> 
> (1) https://github.com/whilo/hitchhiker-tree
> (2) https://github.com/whilo/datascript/tree/hitchhiker_tree_support
> (3) http://tonsky.me/blog/the-web-after-tomorrow/
> 


-- 
Note that posts from new members are moderated - please be patient with your 
first post.
--- 
You received this message because you are subscribed to the Google Groups 
"ClojureScript" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to clojurescript+unsubscr...@googlegroups.com.
To post to this group, send email to clojurescript@googlegroups.com.
Visit this group at https://groups.google.com/group/clojurescript.


signature.asc
Description: OpenPGP digital signature