Héllo all, On Sun, Oct 1, 2017 at 6:09 AM Linas Vepstas <linasveps...@gmail.com> wrote:
> Hi Ed, > > Here's my experience and I would love to get help with it. I am now > building graphs that are so large, that they no longer fit into RAM (on a > machine with 256GB RAM). I'm slowly moving to algorithms that can page in > only the needed subgraphs, on demand and then drop these when not needed. > This is exactly the problem I want (or wanted) to solve using wiredtiger database engine. The problem is that I need help. I need to know: 0) Which headers files are relevant? 1) Which classes must be overridden? 2) How to test that my code works as expected? Obviously, if nobody can guide me a *little* through the codebase, I will not be able to help. I can not convince you I will be successful after all I just another webdev (working on a 200k sloc project) lurking around... The main difference with current postgresql backend, is that with wiredtiger database engine is a library. wiredtiger is embedded in the program that use it. Also, it handles caching for you, so you don't need to load/unload subgraphs manually, it will be done for you by the engine which will know precisely the atomspace data layout and which will lead to efficient caching. BTW, wiredtiger is used by mongodb since 3.2. It's the underlying default engine. That said, it's more powerful than what mongodb expose. Some flaws of mongodb were kept for backward compatibility (I guess). > So I turned the safety features back on, but now access to atoms is maybe > 10x slower than before. Yes, I bought SSD's for database storage, this > helped a lot. Yes, I bought an uninterruptible power supply. For the > short-term, I am good to go. But this is very home-brew. There's a big > difference between tinkering in your garage, and building a factory floor. > > How is postgresql fine tuning is helping? I was under the impression that atomspace loaded the whole database into main memory and dumped it on demand. In the long-term, though, a cloud solution, with high-speed access to a > distributed database is needed. I have no clue what sort of performance > numbers are achievable, or what it would take to improve these (as the > initial attempt is bound to be bad). How much of this would require either > redesign, or large new features in atomese. I might hope "relatively > little" but I am too world-wise to entertain such hopes when I'm not drunk. > idk. -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to opencog+unsubscr...@googlegroups.com. To post to this group, send email to opencog@googlegroups.com. Visit this group at https://groups.google.com/group/opencog. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CAL7_Mo8CRKocSMYNye--O85q3yxnRg0B-S3%3DPtPECb%2BwJ7jMmQ%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
