Re: [go-nuts] KVs store for large strings

[Kiki Sugiaman]

Try boltdb.

- supports disk persistence
- no read lock for read-only ops
- fast key iteration (only as fast as O(n) can be)
- probably will translate 100MB raw into 200MB-ish actual utilization, 
but see for yourself

- no cgo

https://github.com/boltdb/bolt


On 15/06/17 04:45, James Pettyjohn wrote:
I have an application which has all of it's text, multiple languages, 
stored in XML on disk that is merged into templates on the fly. About 
100MB. Templates use dozens of strings for each render.


Currently this is loaded in full into memory in a bunch of tier hash 
maps. They are lazy loaded and using multiple locks to perform reads 
but, unless in dev mode, actually don't change throughout the lifetime 
of the application and should be considered immutable.


While workable at a smaller scale, it's slow at scale. The most 
important factor is concurrent lookup speed, secondary concern is memory 
overhead. And it cannot preclude periodic reload while doing dev.


Is there a data structure or lib that suits this scenarios more than others?

Best,
James

--
You received this message because you are subscribed to the Google 
Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to golang-nuts+unsubscr...@googlegroups.com 
.

For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[go-nuts] KVs store for large strings

[James Pettyjohn]

I have an application which has all of it's text, multiple languages, 
stored in XML on disk that is merged into templates on the fly. About 
100MB. Templates use dozens of strings for each render. 

Currently this is loaded in full into memory in a bunch of tier hash maps. 
They are lazy loaded and using multiple locks to perform reads but, unless 
in dev mode, actually don't change throughout the lifetime of the 
application and should be considered immutable.

While workable at a smaller scale, it's slow at scale. The most important 
factor is concurrent lookup speed, secondary concern is memory overhead. 
And it cannot preclude periodic reload while doing dev.

Is there a data structure or lib that suits this scenarios more than others?

Best,
James

-- 
You received this message because you are subscribed to the Google Groups 
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.