On Sat, Jun 22, 2024 at 7:50 PM Boris Kazachenko <cogno...@gmail.com> wrote: > > On Saturday, June 22, 2024, at 7:18 AM, Rob Freeman wrote: > > But I'm not sure that just sticking to some idea of learned hierarchy, which > is all I remember of your work, without exposing it to criticism, is > necessarily going to get you any further. > > It's perfectly exposed: https://github.com/boris-kz/CogAlg
I see. The readme seems concise. Quite a good way to expose it. Trivial typo in that first paragraph BTW, "pattern recognition (is?) a main focus in ML"? So what's your idea now? I remember we talked long ago and while you were early into networks, I couldn't convince you that the key problem was that meaning could not be fully learned, because meaningful patterns contradict. You were sure all that was needed was learning patterns in hierarchy. Where have you arrived now? You say, "The problem I have with current ML is conceptual consistency." By "conceptual consistency" you mean a distinction between searching for "similarity" with "attention" in transformers, "similarity" being co-occurrence(?), vs "variance" or edges, in CNNs. The solution you are working on is to cluster for both "similarity" and "variance". FWIW I think it is a mistake to equate transformers with "attention". Yeah, "attention" was the immediate trigger of the transformer revolution. But it's a hack to compensate for lack of structure. The real power of transformers is the "embedding". Embedding was just waiting for something to liberate it from a lack of structure. "Attention" did that partially. But it's a hack. The lack of structure is because the interesting encoding, which is the "embedding", attempts global optimization, when instead globally it contradicts in context, and needs to be generated at run-time. If you do "embedding" at run time, it can naturally involve token sequences of different length, and embedding sequences of different length generates a hierarchy, and gives you structure. The structure pulls context together naturally, and "attention" as some crude dot product for relevance, won't be necessary. Ah... Reading on I see you do address embeddings... But you see the problem there as being back-prop causing information loss over many layers. So you think the solution is "lateral" clustering first. You say "This cross-comp and clustering is recursively hierarchical". Yes, that fits with what I'm saying. You get hierarchy from sequence embedding. So what's the problem with these embedding hierarchies in your model? In mine it is that they contradict and must be found at run time. You don't have that. Instead you go back to the combination of "similarity and variance" idea. And you imagine there are some highly complex "nested derivatives"... So the contrast between us is still that you don't see that contradictory patterns prohibit global learning. Compared to what I remember you are addressing "lateral" patterns now. Perhaps as a consequence of the success of transformers? But instead of addressing the historical failure of "lateral", embedding hierarchies as a consequence of contradictory patterns, as I do, you imagine that the solution is some mix of this combination of "similarity" and "variance", combined with some kind of complex "nested derivatives". There's a lot of complexity after that. One sentence jumps out "I see no way evolution could produce (the) proposed algorithm". With which I agree. I'm not sure why you don't think that's an argument against it. This compared with my hypothesis, which sees nested hierarchies appearing naturally, in real time, as synchronized oscillations over predictive symmetries in a sequence network. Can I ask you, have you considered instead my argument, that these "lateral" hierarchical embeddings might be context dependent, and contradict globally, so that they can't be globally learned, and must be generated at run time? Do you have any argument to exclude that? ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T682a307a763c1ced-Mcd56e51f00e643bbf4829174 Delivery options: https://agi.topicbox.com/groups/agi/subscription
