Thanks for the follow-up!
That ended up being the approach I took - map two levels deep (because
that's how the keys come through) and load everything at start up and use
as an immutable structure. While in dev it is using the old code
essentially and that's fine.
I also noted that internal structure in the old code using a struct {x
string, y string} for its first level key which I thought odd. Not sure
what effects that would have on things.
Best,
James
On Thursday, June 15, 2017 at 8:17:47 AM UTC-7, ksug wrote:
>
> There were mistakes with the code I posted. Oh well, in a hurry and not
> the point.
>
>
>
> On 15/06/17 22:41, Kiki Sugiaman wrote:
> > I think I misunderstood what you were trying to accomplish. Let me try
> > to rephrase and see if I'm getting closer:
> >
> > - to accommodate template changes during development, you introduced
> > read-locking to your app.
> > - as a result, your reads in production are slow. Never mind the fact
> > that locking is pointless in production because the templates don't
> change.
> > - ideally, you want your reads to yield to template changes during
> > development, but go lockless in production.
> >
> > If the above are correct, then the solution is not data structure
> > change, but code swapping during init() depending on the environment
> > variable $DEVMODE (or whatever you want to call it).
> >
> > Example:
> > https://play.golang.org/p/wqb9lVIPLa
> > (repeat several times and change the random seed to randomize the
> > behavior a bit)
> >
> > Of course, optimizing the data structure doesn't hurt. And to answer
> > your question, yes map can handle 10's of thousands entries with 100MB
> > raw data with *relative* ease.
> >
> >
> >
> > On 15/06/17 15:45, James Pettyjohn wrote:
> >> I looked at doing boltdb but ended up with a slightly simpler
> >> approach. I don't need the disk persistence in this case.
> >>
> >> I tried radix tree implementation but found it slower than my current
> >> nested map with locks. In that interest I rewrote the nested map
> >> approach considering the key partitioning noted by Robert.
> >>
> >> The primary differences in my approach this time were:
> >> 1) Read the whole dataset once treat it as immutable, no locking
> >> 2) Align the API to no require any manipulation of the keys after the
> >> first load, so every lookup is now 0 allocation.
> >>
> >> The structure is a simplistic map[string]map[string]string - language
> >> -> key. There was a breakdown of the second map further in the
> >> previous implementation, probably bringing it from a few thousand to a
> >> few hundred each. Don't know what the threshold on the benefits there,
> >> based on my experience the map implementation can handle 10s of
> >> thousands with ease.
> >>
> >> This brought the lookups down by about 50%:
> >>
> >> BenchmarkLookupLooseLazy-8 3000000 458 ns/op 32 B/op 1 allocs/op
> >> BenchmarkLookupLooseMap-8 5000000 265 ns/op 0 B/op 0 allocs/op
> >>
> >> Let me know if see anything else worth adjusting.
> >>
> >> Best,
> >> James
> >>
> >>
> >> On Wednesday, June 14, 2017 at 2:01:44 PM UTC-7, Robert Johnstone
> wrote:
> >>
> >> I'm surprised that the memory overhead is significant - 100 MB is
> >> not that much.
> >>
> >> Assuming that you don't need atomic updates to the entire KV store,
> >> partition the keys.
> >>
> >> Does the periodic reload involve changing the keys? If not, you
> >> could map the dataset into nested structs. However, you will still
> >> need to synchronise access if you want to to reload without
> stopping
> >> the server, but that would just be the leaves. Switching just the
> >> top-tier to a struct could help with the contention.
> >>
> >>
> >> On Wednesday, 14 June 2017 14:45:25 UTC-4, James Pettyjohn wrote:
> >>
> >> I have an application which has all of it's text, multiple
> >> languages, stored in XML on disk that is merged into templates
> >> on the fly. About 100MB. Templates use dozens of strings for
> >> each render.
> >>
> >> Currently this is loaded in full into memory in a bunch of tier
> >> hash maps. They are lazy loaded and using multiple locks to
> >> perform reads but, unless in dev mode, actually don't change
> >> throughout the lifetime of the application and should be
> >> considered immutable.
> >>
> >> While workable at a smaller scale, it's slow at scale. The most
> >> important factor is concurrent lookup speed, secondary concern
> >> is memory overhead. And it cannot preclude periodic reload
> while
> >> doing dev.
> >>
> >> Is there a data structure or lib that suits this scenarios more
> >> than others?
> >>
> >> --
> >> You received this message because you are subscribed to the Google
> >> Groups "golang-nuts" group.
> >> To unsubscribe from this group and stop receiving emails from it, send
> >> an email to golang-nuts...@googlegroups.com <javascript:>
> >> <mailto:golang-nuts+unsubscr...@googlegroups.com <javascript:>>.
> >> For more options, visit https://groups.google.com/d/optout.
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
