And, as far as I'm concerned, the last clause of item 4 and the transition to 5
and 6 clearly demonstrates why Steve seems to be making a lot of progress
compared to everyone else.
----- Original Message -----
From: Stephen Reed
To: [email protected]
Sent: Thursday, April 17, 2008 10:23 AM
Subject: Re: [agi] database access fast enough?
YKY
Here is what I learned from implementing the Texai knowledge base. It
persists symbolic statements about concepts.
1.. I designed an SQL schema to persist OpenCyc in its full CycL form, in
MySQL on SuSE 64-bit Linux. My Java application driving MySQL dramatically
slowed down when the number of rows exceeded 20 million as compared to the
initial load of 5 million rows.
2.. I then tried Oracle Berkeley DB Java Edition (open source) which
provides no SQL query facility, instead one programs directly to its API for
inserts, queries, updates and so forth. It is faster than MySQL for my large
KB, but uses four times as much disk space due to its method of inserting new
rows at the end of the file, and having lots of free space.
3.. I then studied partitioning, which means to break up the monolithic KB
into smaller databases in which accesses are expected to be clustered. And I
studied sharding, which means to slice up a database into logical segments that
are hosted by separate db engines, typically with separate disk filesystems.
4.. I began writing my own storage engine, for a fast, space-efficient,
partitioned and sharded knowledge base, soon realizing that this was far too
big a task for a sole developer.
5.. Revisiting my project object persistence needs, and thinking more about
interoperability with semantic web technologies, I decided to convert my
existing KB to an RDF-compatible form and then to evaluate RDF quad stores.
6.. After some analysis, I chose to evaluate the Sesame 2 RDF store, which
is Java based and open source and thus very compatible with my other
components. In Texai, RDF queries have a simpler form than SQL queries when
retrieving logical statements from a store. For example, in SQL my schema had
to provide separate tables for each object type: concept term, functional
term, string, boolean, long integer, double, statement, arity-1 rule, arity-2
rule, arity-3 rule, arity-4 rule and arity-5 rule. Many of these tables would
have to be joined for a typical query (e.g. what concepts subsume a given
concept?).
7.. My development Linux computer has 4 GB of memory, and Linux has a
feature called tmpfs which permits mounting a directory in RAM. I partitioned
my KB into separate KBs of less than six million rows each. In Sesame these
are less than one GB in size and I can therefore put any one of them in tmpfs -
running that application-relevant part of the KB at RAM speed. Experiments
demonstrate about a 10 times speedup.
8.. When Texai is deployed, I expect that the application will log its
transactions to disk as a background process as a safeguard against losing the
volatile KB in tmpfs.
Hope this information is useful.
-Steve
Stephen L. Reed
Artificial Intelligence Researcher
http://texai.org/blog
http://texai.org
3008 Oak Crest Ave.
Austin, Texas, USA 78704
512.791.7860
----- Original Message ----
From: YKY (Yan King Yin) <[EMAIL PROTECTED]>
To: [email protected]
Sent: Wednesday, April 16, 2008 11:51:35 PM
Subject: [agi] database access fast enough?
For those using database systems for AGI, I'm wondering if the data
retrieval rate would be a problem.
Typically we need to retrieve many nodes from the DB to do inference.
The nodes may be scattered around the DB. So it may require *many*
disk accesses. My impression is that most DBMS are optimized for
complex queries but not for large numbers of simple retrievals -- am I
correct about this?
YKY
-------------------------------------------
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription: http://www.listbox.com/member/?&;
Powered by Listbox: http://www.listbox.com
------------------------------------------------------------------------------
Be a better friend, newshound, and know-it-all with Yahoo! Mobile. Try it now.
------------------------------------------------------------------------------
agi | Archives | Modify Your Subscription
-------------------------------------------
agi
Archives: http://www.listbox.com/member/archive/303/=now
RSS Feed: http://www.listbox.com/member/archive/rss/303/
Modify Your Subscription:
http://www.listbox.com/member/?member_id=8660244&id_secret=101455710-f059c4
Powered by Listbox: http://www.listbox.com
