Re: [opencog-dev] Re: Graphs for the two representations of the knowledge and ideas about the third generation Cog system - MOC - MetaOmegaCog?

[Linas Vepstas]

Hi Alex,

On Wed, Oct 4, 2017 at 3:03 AM, Alex  wrote:

> Well, maybe I am too optimistic...
>
> Regarding the rules, I am investigating the following (not impressive):
> https://github.com/threatgrid/naga
>

Atomese is definitely inspired by datalog. About 10 years ago, we actually
had a datalog API to the atomspace, it was created by one of the early
contributors, and was used by the State of Florida in some web-service.

>
> https://link.springer.com/chapter/10.1007/978-3-319-21542-6_14
>

I keep trying to convince Ben and/or anyone at all to write up the
atomspace and submit for publication at one of these conferences, but no
one ever does.  It would be very cool -- a simple, easy-to-write paper
describing the newest ideas -- and ti would be stuff we've already got code
for. Free advertising for opencog.

>
> https://www.google.com/patents/US20150302300
>

Heh. Well, we've got lots and lots of prior art for that patent. Foo.

>
>
> Rules over graph database is far less popular theme than graph matching
> and mining, but I guess - graph matching is by far the most complex and
> important part of the any rule engine.
>

Yes. Well, once you have the concept of graph matching, then the concept of
a rule engine is (should be) "obvious".   There are many more things that
we've done in the atomspace that are now "obvious" in hindsight, but were
not when going forwards.

I am not surprised that other people are inventing/rediscovering and
implementing these things -- that's what happens when an idea is
"obvious".  I am somewhat frustrated that some of these other projects get
more exposure, more advertising, and become more popular than opencog, and
I wish I could change that.

I wish that some of the energy expended on creating graph databases had
been expended on opencog instead.  I wish that the tinkerpop people had
used the opencog query language, instead of inventing thier own.

Maybe this was the usual java vs. c++ thing. Maybe its the Apache license
vs. the GPL license. Some (many?) decisions are political, not technical.

Anyway, I am pretty sure that the atomspace design is five or six steps
ahead of everyone else, not just one or two.  Most of the world has not yet
discovered why these steps are important and interesting.  Maybe in 5 or 10
years, there will be an Apache Whatever that will also do these things in a
faster, more scalable way.  Maybe someone will patent these ideas, and
we'll have prior art. But right now, there isn't, and I can't wait 5 or 10
years.

What is critically important is to make the atomspace better known, easier
to understand, more popular, easier to use, more scalable, faster, more
efficient.  All of this is hard to do, however.

--linas


> Well - I suggest you not to waste your time on this, I will investigate
> further open source projects and then I will report back. I don't want to
> disturb anyone from his plans.
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+unsubscr...@googlegroups.com.
> To post to this group, send email to opencog@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/opencog/da088fdb-5d63-45e1-935e-01554e22b575%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

-- 
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/CAHrUA34FHKAf%3DzVe6rLZ2JjtvHVhFbX-15YzCq%3Dgx6uXBCnNpQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [opencog-dev] Re: Graphs for the two representations of the knowledge and ideas about the third generation Cog system - MOC - MetaOmegaCog?

['Nil Geisweiller' via opencog]

On 10/03/2017 08:29 PM, Alex wrote:
source projects and tools for matcher in miner and I have found even one 
open source project for rule engine that is implemented on top of graph 


Interesting, which one?

I just wanted to say that maybe relying on (and contributing to) open 
source projects for general features/foundation have some advantages.


Absolutely.

Nil

--
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/4f6bc45c-a27a-3a5b-c914-93b89075fea4%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[opencog-dev] Re: Graphs for the two representations of the knowledge and ideas about the third generation Cog system - MOC - MetaOmegaCog?

[Alex]

> Ou focus is NOT to be "just" a graph storage system, but a graph storage 
> system with many additional services (the MMT page lists many of these)
>
> Our big ones are:
> * the pattern matcher
> * the pattern miner
> * the rule engine
>
> Our smaller ones are:
> * a sparse matrix subsystem
> * a parsing (categorial grammar) subsystem 
>

I did Google search - and there are already extensive research about graph 
pattern mahtching and graph pattern mining and there are some open source 
projects and tools for matcher in miner and I have found even one open 
source project for rule engine that is implemented on top of graph 
database. I am currently investigating these projects and deeper and making 
decisions whether they are appropriate for me and whether they are mature 
enough.

I just wanted to say that maybe relying on (and contributing to) open 
source projects for general features/foundation have some advantages.

-- 
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/6c19299e-6c5e-419f-97b5-a7d7dcfd8420%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [opencog-dev] Re: Graphs for the two representations of the knowledge and ideas about the third generation Cog system - MOC - MetaOmegaCog?

[Linas Vepstas]

Oh, and I forgot to mention: opencog atoms are small -- maybe a few hundred
bytes or so.  The performance of most popular web databases totally sucks
when the data is that small -- they are tuned for storing mp3's and jpeg
files, which are megabytes each, and they are great for that - but they
suck for teeny-weeny atoms.  Been there, done that.

--linas

On Sun, Oct 1, 2017 at 11:39 PM, Linas Vepstas 
wrote:

> Hi Amirouche,
>
> Let me top-post, it will be easier. First: bulk load and bulk save of the
> atomspace is part of the API, but  it's very blunt and ugly and useless.  I
> never-ever bulk-load or bulk-save my data.
>
> The more fine-grained API allows:
> * Specific atoms to be loaded (i.e. the values, truthvalues, etc attached
> to those atoms)
> * The entire incoming set of a specific atom to be loaded.
> * Load only that portion of the incoming set that is some specific type.
> * Load all atoms of a specific type.
> * Save just one specific atom.
>
> Let me give an example: So, first, I load all atoms of type WordNode.
> There are maybe 100K or 200K of these, it depends.  Next, I pick one word,
> lets say (WordNode "the") and load all SectionLinks with that word in it.
> (Sections are link-grammar disjuncts).  For a word like "the", there  might
> be 20K or 50K or maybe more sections.  By loading only the SectionLinks, I
> can avoid loading the word-pairs (of which one word is "the"), because I
> don't need the word-pairs, and there's like maybe 100K of them that I don't
> need clogging up RAM.   Then I run my algo, and then pick a different word,
> and repeat.  Pretty much all words have much much fewer sections than
> "the".  The total number of sections is maybe 25 million or double or
> one-tenth of that (it depends), which is probably too much to load all at
> the same time.  I don't really need all 25M at the same time.
>
> So how can wiredtiger help?  To summarize, here's what I got:
>
> So my algo knows exactly which atoms it wants loaded at any given time,
> and I can also provide fairly strong hints about which ones are no longer
> needed.
>
> I absolutely, totally must have these certain kinds of atoms loaded at
> certain times, otherwise the algo totally fails.  The atomspace API allows
> me to ask for exactly those atoms that I want, when I want them.  The
> current API stalls (does not return to caller) until the requested atoms
> are fully loaded in the atomspace.  For all I care, the loading could be
> done async, BUT the atoms must be there when they are accessed.  (We would
> need to change the API to do this kind of async load, but that's doable.
> Hmmm. good idea, even, I should have done this earlier)
>
> Maybe with wiredtiger, we could making loading async, so that the atoms
> are not fully loaded until they are accessed.  I'm not picky.  I can give
> hints about which ones to load, when.
>
> I have no clue how wiredtiger works, so I don't really know what to
> suggest to you.  I can only point you at the current, actual API and its
> documentation.  It is here. If you want a different API, that's OK, I'm OK
> with that, as long as it can actually work. I do NOT need crazy ideas that
> will never work.
>
> The API is here:
>
> https://github.com/opencog/atomspace/blob/master/opencog/
> atomspace/BackingStore.h
>
> an example implementation is here:
>
> https://github.com/opencog/atomspace/blob/master/opencog/
> persist/sql/multi-driver/SQLAtomStorage.cc
>
> If you try to figure out how the one is wired to the other, you will get
> confused; there is some historical perversity that makes it more stupidly
> complicated than it should be. Oh well.  Just skip that part.
>
> So I will help you make wiredtiger work, if you explain to me how it can
> "magically" load the needed atoms at the right time.  Because otherwise, it
> just seems like magic to me.
>
> If we can expose whizzy features in wiredtiger, that's fine too. but I
> have no clue about that.
>
> --linas
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

-- 
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/CAHrUA37Km3BLZ7YB4vPw_-EUCG7q%2BHvpxLaVfo8d170ZNoYX7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [opencog-dev] Re: Graphs for the two representations of the knowledge and ideas about the third generation Cog system - MOC - MetaOmegaCog?

[Linas Vepstas]

Hi Amirouche,

Let me top-post, it will be easier. First: bulk load and bulk save of the
atomspace is part of the API, but  it's very blunt and ugly and useless.  I
never-ever bulk-load or bulk-save my data.

The more fine-grained API allows:
* Specific atoms to be loaded (i.e. the values, truthvalues, etc attached
to those atoms)
* The entire incoming set of a specific atom to be loaded.
* Load only that portion of the incoming set that is some specific type.
* Load all atoms of a specific type.
* Save just one specific atom.

Let me give an example: So, first, I load all atoms of type WordNode. There
are maybe 100K or 200K of these, it depends.  Next, I pick one word, lets
say (WordNode "the") and load all SectionLinks with that word in it.
(Sections are link-grammar disjuncts).  For a word like "the", there  might
be 20K or 50K or maybe more sections.  By loading only the SectionLinks, I
can avoid loading the word-pairs (of which one word is "the"), because I
don't need the word-pairs, and there's like maybe 100K of them that I don't
need clogging up RAM.   Then I run my algo, and then pick a different word,
and repeat.  Pretty much all words have much much fewer sections than
"the".  The total number of sections is maybe 25 million or double or
one-tenth of that (it depends), which is probably too much to load all at
the same time.  I don't really need all 25M at the same time.

So how can wiredtiger help?  To summarize, here's what I got:

So my algo knows exactly which atoms it wants loaded at any given time, and
I can also provide fairly strong hints about which ones are no longer
needed.

I absolutely, totally must have these certain kinds of atoms loaded at
certain times, otherwise the algo totally fails.  The atomspace API allows
me to ask for exactly those atoms that I want, when I want them.  The
current API stalls (does not return to caller) until the requested atoms
are fully loaded in the atomspace.  For all I care, the loading could be
done async, BUT the atoms must be there when they are accessed.  (We would
need to change the API to do this kind of async load, but that's doable.
Hmmm. good idea, even, I should have done this earlier)

Maybe with wiredtiger, we could making loading async, so that the atoms are
not fully loaded until they are accessed.  I'm not picky.  I can give hints
about which ones to load, when.

I have no clue how wiredtiger works, so I don't really know what to suggest
to you.  I can only point you at the current, actual API and its
documentation.  It is here. If you want a different API, that's OK, I'm OK
with that, as long as it can actually work. I do NOT need crazy ideas that
will never work.

The API is here:

https://github.com/opencog/atomspace/blob/master/opencog/atomspace/BackingStore.h

an example implementation is here:

https://github.com/opencog/atomspace/blob/master/opencog/persist/sql/multi-driver/SQLAtomStorage.cc

If you try to figure out how the one is wired to the other, you will get
confused; there is some historical perversity that makes it more stupidly
complicated than it should be. Oh well.  Just skip that part.

So I will help you make wiredtiger work, if you explain to me how it can
"magically" load the needed atoms at the right time.  Because otherwise, it
just seems like magic to me.

If we can expose whizzy features in wiredtiger, that's fine too. but I have
no clue about that.

--linas

-- 
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/CAHrUA36Xx2%3D3e6%3Dq2DjvBUL7x2ajb3gWFg-TM5NBykWZ45gP-g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Re: [opencog-dev] Re: Graphs for the two representations of the knowledge and ideas about the third generation Cog system - MOC - MetaOmegaCog?

[Linas Vepstas]

Hi Ed,

Thanks. It turns out that I have glanced at JanusGraph in the past. The
main landing page for JanusGraph does make it sound very impressive.

Here's my experience and I would love to get help with it.  I am now
building graphs that are so large, that they no longer fit into RAM (on a
machine with 256GB RAM).  I'm slowly moving to algorithms that can page in
only the needed subgraphs, on demand and then drop these when not needed.

In the past, I used to run postgres without any protections: My default
config for postgres was to disable sync-to-disk, and tune all sorts of
other parameters for performance.  This worked great, or it worked "well
enough".  And then came a thunderstorm, and then another, and I got very
shy about disabling the various writeback and sync features in postgres.
Basically, I experienced data loss and database corruption.  Which is
semi-tolerable (for my current datasets), but quite painful and unpleasant
and nerve-wracking.

So I turned the safety features back on, but now access to atoms is maybe
10x slower than before.   Yes, I bought SSD's for database storage, this
helped a lot.  Yes, I bought an uninterruptible power supply.  For the
short-term, I am good to go.  But this is very home-brew.  There's a big
difference between tinkering in your garage, and building a factory floor.

In the long-term, though, a cloud solution, with high-speed access to a
distributed database is needed.  I have no clue what sort of performance
numbers are achievable, or what it would take to improve these (as the
initial attempt is bound to be bad).  How much of this would require either
redesign, or large new features in atomese. I might hope "relatively
little" but I am too world-wise to entertain such hopes when I'm not drunk.

--linas

On Sun, Oct 1, 2017 at 11:36 AM, Ed Pell  wrote:

> JanusGraph is just a graph store. It has no available reasoners.
>
> It is best developed for Java and is weak for Python. As far as I can tell
> there is no QA process.
>
> JanusGraph is developed largely by IBM and Google as a replacement for
> Titan which has been taken proprietary.
>
> IBM would benefit far more by adopting Atomspace and its reasoners than
> Opencog would benefit from JanusGraph.
>
> --
> You received this message because you are subscribed to the Google Groups
> "opencog" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to opencog+unsubscr...@googlegroups.com.
> To post to this group, send email to opencog@googlegroups.com.
> Visit this group at https://groups.google.com/group/opencog.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/opencog/18ebc255-e1ac-402a-8898-6bc1cb74dfd9%40googlegroups.com
> 
> .
>
> For more options, visit https://groups.google.com/d/optout.
>



-- 
*"The problem is not that artificial intelligence will get too smart and
take over the world," computer scientist Pedro Domingos writes, "the
problem is that it's too stupid and already has." *

-- 
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/CAHrUA36jR1pv326w29%2B4CknLSnGjH3qwgDubR%3DLA7ULFY-Jxqw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[opencog-dev] Re: Graphs for the two representations of the knowledge and ideas about the third generation Cog system - MOC - MetaOmegaCog?

[Ed Pell]

JanusGraph is just a graph store. It has no available reasoners. 

It is best developed for Java and is weak for Python. As far as I can tell 
there is no QA process. 

JanusGraph is developed largely by IBM and Google as a replacement for 
Titan which has been taken proprietary. 

IBM would benefit far more by adopting Atomspace and its reasoners than 
Opencog would benefit from JanusGraph.   

-- 
You received this message because you are subscribed to the Google Groups 
"opencog" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to opencog+unsubscr...@googlegroups.com.
To post to this group, send email to opencog@googlegroups.com.
Visit this group at https://groups.google.com/group/opencog.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/opencog/18ebc255-e1ac-402a-8898-6bc1cb74dfd9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.