Hi Carsten,
Your points are valid. Cortical.io does indeed currently rely on a kind of
bag-of-words approach for both the creation of the word-SDRs and for
combining SDRs for phrases and text snippets. This is both a strength and a
weakness, as you describe.
It's important to note that Francisco is not claiming to have "solved" NLP.
He is merely proffering this route as an improvement over competing ideas,
and explaining some of the advantages.
The main reason it works at all is that the sparsification process chooses
the bits which are most representational of the contexts in which the word
appears, and this will come mostly from text snippets in which the word is
used in a positive sense.
The "algebra" is not strictly Boolean - it does not follow DeMorgan's Laws
- mainly because NOT is not implemented in the usual sense.
These are not bugs in the design: they are a deliberate feature which is
derived from HTM's SDR logic, and ultimately from how neurons work.
Pyramidal neurons only encode presence of whatever they represent, and they
work in populations by voting positively to form a positive representation
in an SDR.
NOT in the Boolean sense is not a natural neural concept, or conversely
Boolean algebra is not the "Laws of Thought" as claimed by the Irishman.
The idea of not-ness is only sensible (for neural systems) in two kinds of
context: first, when a positive statement is posed as a question, and then
prediction (from the positive) compared with evidence, leading to a "No"
due to absence of a match; secondly, when not-ness is hidden in atomic
positive concepts, as in "black = not white; white = not black", "dark =
not bright; bright = not dark" and so on. This is illustrated as follows:
if I say "there's no such thing as a pink elephant on a unicycle, juggling
piglets" you have no choice but to imagine that image or movie, and only
then falsify it as implausible.
So, if you want to do real algebra on SDRs, you need to construct competing
SDRs which have low overlap, and see which one wins out when connected to
the evidence.
Note that cortical.io SDRs do not support this directly - they will be more
likely to find "opposites" (eg. "man" vs "woman") to be more similar than
unrelated concepts ("banana" vs "woman"), precisely because they appear in
similar contexts. To compare "man" and "woman" you should nil the common
bits, and so compare "man - woman" vs "woman - man".
The second issue is about combining SDRs for phrases and documents. This is
really two issues (at least). One is that meaning is about how words are
combined, just as much, or more, than simply the identities of those words.
Cortical.io SDRs only provide a starting point for processing syntactic
language, leaving that to some kind of HTM processor.
The second issue is about how to "tag" or fingerprint a piece of text given
the words in it. For searching among a large number of candidate documents
(or educational exercises), the Cortical.io approach can be se shown to be
as good as any known automatic method. This is because you want to retrieve
documents which are similar in the sense that they talk about the same
things.
The other sense of tagging is to identify what the piece of text actually
denotes or implies. This the Cortical.io text-fingerprint only does in a
very weak sense, if at all, and they don't claim it does. To extract
meaning, you need to conserve the individual SDRs for each word (which do
encode meaning in some sense), and again use a processor to work on the
intact sequence. The demos described in the paper are clearly doing
something of this sort, but it's difficult to see how this would work in
general, or even in more subtle extensions of those demos.
We currently don't know how to build a HTM processor for NLP (Numenta and
Cortical.io are working on this, apparently), so in the meantime I would
suggest using these SDRs as inputs to the state-of-the-art Deep Learning
NLP platforms, and comparing them against the vector encodings those
systems currently use. This would, in turn, give us a baseline to aim at
for developing HTM-based processors.
Regards,
Fergal Byrne
On Sun, Nov 29, 2015 at 2:07 AM, Chandan Maruthi <[email protected]>
wrote:
> Carsten,
> This was indeed an interesting question.
> Let me try to answer to my extent of understanding.
>
> a.)Cortical.IO's approach is not bag of words. The finger print of the
> word Fox already as attributes about the kind of things it may eat . So the
> finger print or SDR for Fox and Sheep will differ in the bits that
> represent things they eat.
> b.) During training. I think the folks at Cortical.IO may be using some
> additional NLP methods to identify what entity is in focus and what
> attributes are talking about the entity etc . So those methods may help
> resolve the types of issues you have raised. The value of their approach is
> post learning their fingerprints reduce computing power required to operate
> c.) The fact that fingerprints are pre-computed has big benefit in terms
> of speed of computing , because the reduces to a array
> union/intersection/simlarity computation which allows them to process data
> at scale. thats big
> d.) However, Fransico correct me if I am wrong here. Unlike Nupic where
> the system is continuously learning. I dont think learning occurring during
> operation with cortical IO. This is not necessarily bad. Its just how it
> works.
>
> Chandan
>
>
> On Fri, Nov 27, 2015 at 12:21 AM, Carsten Schnober <
> [email protected]> wrote:
>
>> Dear list,
>> Thanks for the good read. I am happy to (hopefully) start the discussion.
>>
>> The first issue that comes into my mind is syntax: this concerns every
>> bag-of-words approach, including Cortical's.
>> The most obvious source of "natural language misunderstanding" in this
>> context is negation, as easily demonstrated in this example:
>>
>> - fox eat rodent
>> - sheep do not eat rodent
>>
>> I suppose the presented algorithm would learn from this that both fox
>> and sheep do eat rodent, doesn't it? This is probably more harmful when
>> classifying sentences than during learning, because a corpus of
>> reasonable size presumably contains a sufficient amount of examples that
>> are phrased in a more straight-forward way.
>> More complex examples, including more subtle negation, relative clauses
>> etc., will pose much larger challenges.
>>
>> I am quite sure that the syntax issue has been discussed in this
>> context. However, I couldn't find any references, neither in the
>> theoretical nor in the practical part of the whitepaper. I am very
>> interested in Cortical.io's experiences with that problem and what
>> possible (future) solutions might look like.
>>
>> In statistical NLP, this issue has been tackled (more or less
>> successfully) with methods such as recurrent neural networks or by using
>> sliding windows across multiple words. amongst others. Neither of these
>> approaches seem applicable here without taking away a fundamental and
>> very handy property of the SDRs: that they can be efficiently aggregated
>> with boolean operations.
>>
>> Although the syntax issue might be almost irrelevant for many practical
>> use cases such as document classification, I think it raises an
>> interesting theoretical question. How does the human brain process
>> syntax and, more interestingly, how can this be incorporated into the
>> presented theory?
>>
>>
>> A slightly more technical issue I've stumbled across is word inflection.
>> The whitepaper briefly mentions morphemes which are, according to
>> linguistic theory, "the smallest meaningful units" in language. I
>> understand that working on the word level is sufficient in most cases
>> and much easier for practical reasons (tokenization is relatively easy).
>> I wonder how this is handled in practice though, for instance when
>> learning a new "language definition corpus". Are the words automatically
>> lemmatized? What if a new language is learned for which no lemmatizers
>> are available? Is mere stemming applied in that case? What happens if
>> different word forms do express a different meaning?
>>
>> Thanks for any input on these issues!
>> Carsten
>>
>>
>>
>>
>> Am 25.11.2015 um 19:13 schrieb Fergal Byrne:
>> > Nice, Francisco, thanks for letting us know. I've read the paper, very
>> > well put together. Looking forward to discussions and questions on the
>> list.
>> >
>> > --
>> >
>> > Fergal Byrne, Brenter IT
>> >
>> > Author, Real Machine Intelligence with Clortex and NuPIC
>> > https://leanpub.com/realsmartmachines
>> >
>> > Speaking on Clortex and HTM/CLA at euroClojure Krakow, June 2014:
>> > http://euroclojure.com/2014/
>> > and at LambdaJam Chicago, July 2014: http://www.lambdajam.com
>> >
>> > http://inbits.com - Better Living through Thoughtful Technology
>> > http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne
>> >
>> > e:[email protected] t:+353 83 4214179
>> > Join the quest for Machine Intelligence at http://numenta.org
>> > Formerly of Adnet [email protected] http://www.adnet.ie
>> >
>> >
>> > On Wed, Nov 25, 2015 at 4:33 PM, Chandan Maruthi
>> > <[email protected] <mailto:[email protected]>> wrote:
>> >
>> > Francisco
>> >
>> > This is great , looking forward to read this today
>> >
>> > On Wednesday, November 25, 2015, cogmission (David Ray)
>> > <[email protected] <mailto:[email protected]>>
>> wrote:
>> >
>> > Hi Francisco,
>> >
>> > This will make for a very interesting and informative read!
>> > Can't wait!
>> >
>> > Cheers,
>> > David
>> >
>> > On Wed, Nov 25, 2015 at 8:38 AM, Pascal Weinberger
>> > <[email protected]> wrote:
>> >
>> > Great!
>> > I was waiting for this a long time :D
>> > Will make my day! :)
>> >
>> > Thank you!
>> >
>> >
>> >
>> > Best,
>> >
>> > Pascal Weinberger
>> >
>> > ____________________________
>> >
>> > BE THE CHANGE YOU WANT TO SEE IN THE WORLD ...
>> >
>> >
>> > PLEASE NOTE: This email and any file transmitted are
>> > strictly confidential and/or legally privileged and intended
>> > only for the person(s) directly addressed. If you are not
>> > the intended recipient, any use, copying, transmission,
>> > distribution, or other forms of dissemination is strictly
>> > prohibited. If you have received this email in error, please
>> > notify the sender immediately and permanently delete the
>> > email and files, if any.____
>> >
>> > Please consider the environment before printing this
>> message.
>> >
>> > __ __
>> >
>> >
>> >
>> > On 25 Nov 2015, at 14:49, Francisco Webber
>> > <[email protected]> wrote:
>> >
>> >> Hello all,
>> >> For everyone interested in the theoretical background to
>> >> Cortical.io <http://cortical.io>’s technology:
>> >>
>> >> The Semantic Folding white paper is out in its first
>> >> incarnation:
>> >>
>> >> Download full White Paper
>> >> <
>> http://www.cortical.io/static/downloads/semantic-folding-theory-white-paper.pdf
>> >
>> >>
>> >> All the Best
>> >>
>> >> Francisco
>> >>
>> >>
>> >
>> >
>> >
>> > --
>> > /With kind regards,/
>> >
>> > David Ray
>> > Java Solutions Architect
>> >
>> > *Cortical.io <http://cortical.io/>*
>> > Sponsor of: HTM.java <https://github.com/numenta/htm.java>
>> >
>> > [email protected]
>> > http://cortical.io <http://cortical.io/>
>> >
>> >
>> >
>> > --
>> > Regards
>> > Chandan Maruthi
>> >
>> >
>> >
>>
>> --
>> Carsten Schnober
>> Doctoral Researcher
>> Ubiquitous Knowledge Processing (UKP) Lab
>> FB 20 / Computer Science Department
>> Technische Universität Darmstadt
>> Hochschulstr. 10, D-64289 Darmstadt, Germany
>> phone [+49] (0)6151 16-6227, fax -5455, room S2/02/B111
>> [email protected]
>> www.ukp.tu-darmstadt.de
>>
>> Web Research at TU Darmstadt (WeRC): www.werc.tu-darmstadt.de
>> GRK 1994: Adaptive Preparation of Information from Heterogeneous Sources
>> (AIPHES): www.aiphes.tu-darmstadt.de
>> PhD program: Knowledge Discovery in Scientific Literature (KDSL)
>> www.kdsl.tu-darmstadt.de
>>
>>
>
>
> --
> Regards
> Chandan Maruthi
>
>
--
Fergal Byrne, Brenter IT @fergbyrne
http://inbits.com - Better Living through Thoughtful Technology
http://ie.linkedin.com/in/fergbyrne/ - https://github.com/fergalbyrne
Founder of Clortex: HTM in Clojure -
https://github.com/nupic-community/clortex
Co-creator @OccupyStartups Time-Bombed Open License http://occupystartups.me
Author, Real Machine Intelligence with Clortex and NuPIC
Read for free or buy the book at https://leanpub.com/realsmartmachines
e:[email protected] t:+353 83 4214179
Join the quest for Machine Intelligence at http://numenta.org
Formerly of Adnet [email protected] http://www.adnet.ie
