Re: [FRIAM] A very good piece about ChatGPT.

2023-02-10 Thread glen 

I'm curious why it's unanticipated. Is this a (very mild) form of the Nobel 
Effect: https://en.wikipedia.org/wiki/Nobel_Prize_effect? The idea that one's 
been successful in some persnickety sub-domain triggers some dopamine and makes 
them think they'll achieve easier success in some other (equally persnickety) 
sub-domain?

I think this highlights a subtle error in the Sutton article Jochen linked, 
related (sorry for the repetition) to a wise choice of use case(s). Sutton 
writes:

http://www.incompleteideas.net/IncIdeas/BitterLesson.html
"1) AI researchers have often tried to build knowledge into their agents, 2) this 
always helps in the short term, and is personally satisfying to the researcher, but 3) in 
the long run it plateaus and even inhibits further progress, and 4) breakthrough progress 
eventually arrives by an opposing approach based on scaling computation by search and 
learning."

This passage and the whole article *assume* a subset of use cases. At first I thought it 
was simply a lack of distinction between special AI vs. AGI. But it's not. It's the 
assumption of a tiny part of special AI. It's the assumption that AI is equivalent to 
*opaque* AI ... "free lunch" algorithms ... brute force. It might be useful to 
contrast something like ChatGPT with Alpha Zero (or, better yet, Alpha Tensor).

The knowledge is in the scaffolding to the well-formed problem. The knowledge 
isn't in the brute force computation that occurs inside the space constrained 
by the scaffolding. To learn something *is* to build that scaffolding.

On 2/10/23 10:45, Russ Abbott wrote:

The amount fo work required to build the intellectual scaffolding almost always 
seems to be an unanticipated problem when learning something new. In many cases 
the scaffolding is not just intellectual. Think of what it takes to learn to 
play musical instrument!


--
ꙮ Mɥǝu ǝlǝdɥɐuʇs ɟᴉƃɥʇ' ʇɥǝ ƃɹɐss snɟɟǝɹs˙ ꙮ

-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom 
https://bit.ly/virtualfriam
to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com
FRIAM-COMIC http://friam-comic.blogspot.com/
archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/
 1/2003 thru 6/2021  http://friam.383.s1.nabble.com/

Re: [FRIAM] A very good piece about ChatGPT.

2023-02-10 Thread Russ Abbott 
Very nice piece by Rich Sutton!

Re NanoGPT. Even though I'm retired, I wanted to learn more about how the
LLMs work. So I started a project in which a number of students and i work
through Karpathy's nanoGPT. We've just started, but I think it will be
worth the effort. Karpathy uses non-trivial Python. Many students don't
know Python at all. So learning Python along with its non-trivial features
as well as becoming familiar with PyTorch, is something of a challenge for
the project. Students (and I, in fact) were not prepared for the amount of
scaffolding background knowledge one had to build for oneself.

The amount fo work required to build the intellectual scaffolding almost
always seems to be an unanticipated problem when learning something new. In
many cases the scaffolding is not just intellectual. Think of what it takes
to learn to play musical instrument!

-- Russ


On Fri, Feb 10, 2023 at 10:09 AM Jochen Fromm  wrote:

> Interesting article. I have found this link to "GPT in 60 Lines of NumPy"
> on Hacker News
> https://jaykmody.com/blog/gpt-from-scratch/
>
> It is similar to nanoGPT and minGPT from Andrej Karpathy, who has joined
> OpenAI again recently
> https://github.com/karpathy/nanoGPT
>
> The secret is apparently not to use a complicated algorithm but just a
> huge amount of data, as Rich Sutton writes here
> http://www.incompleteideas.net/IncIdeas/BitterLesson.html
>
> -J.
>
>
>  Original message 
> From: Russ Abbott 
> Date: 2/10/23 5:53 PM (GMT+01:00)
> To: The Friday Morning Applied Complexity Coffee Group 
>
> Subject: [FRIAM] A very good piece about ChatGPT.
>
> It's in the New Yorker
> <https://www.newyorker.com/tech/annals-of-technology/chatgpt-is-a-blurry-jpeg-of-the-web>
> and intended for a non-scientific audience.
>
>
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom 
https://bit.ly/virtualfriam
to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com
FRIAM-COMIC http://friam-comic.blogspot.com/
archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/
  1/2003 thru 6/2021  http://friam.383.s1.nabble.com/

Re: [FRIAM] A very good piece about ChatGPT.

2023-02-10 Thread Jochen Fromm 
Interesting article. I have found this link to "GPT in 60 Lines of NumPy" on 
Hacker Newshttps://jaykmody.com/blog/gpt-from-scratch/It is similar to nanoGPT 
and minGPT from Andrej Karpathy, who has joined OpenAI again 
recentlyhttps://github.com/karpathy/nanoGPTThe secret is apparently not to use 
a complicated algorithm but just a huge amount of data, as Rich Sutton writes 
here http://www.incompleteideas.net/IncIdeas/BitterLesson.html-J.
 Original message From: Russ Abbott  
Date: 2/10/23  5:53 PM  (GMT+01:00) To: The Friday Morning Applied Complexity 
Coffee Group  Subject: [FRIAM] A very good piece about 
ChatGPT. It's in the New Yorker and intended for a non-scientific audience.


-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom 
https://bit.ly/virtualfriam
to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com
FRIAM-COMIC http://friam-comic.blogspot.com/
archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/
  1/2003 thru 6/2021  http://friam.383.s1.nabble.com/

Re: [FRIAM] A very good piece about ChatGPT.

2023-02-10 Thread Sarbajit Roy 
Thanks for the share.
This article explains things very well from aspects I had never considered.

On Fri, Feb 10, 2023 at 10:23 PM Russ Abbott  wrote:

> It's in the New Yorker
> 
> and intended for a non-scientific audience.
>
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom 
https://bit.ly/virtualfriam
to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com
FRIAM-COMIC http://friam-comic.blogspot.com/
archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/
  1/2003 thru 6/2021  http://friam.383.s1.nabble.com/

[FRIAM] A very good piece about ChatGPT.

2023-02-10 Thread Russ Abbott 
It's in the New Yorker

and intended for a non-scientific audience.
-. --- - / ...- .- .-.. .. -.. / -- --- .-. ... . / -.-. --- -.. .
FRIAM Applied Complexity Group listserv
Fridays 9a-12p Friday St. Johns Cafe   /   Thursdays 9a-12p Zoom 
https://bit.ly/virtualfriam
to (un)subscribe http://redfish.com/mailman/listinfo/friam_redfish.com
FRIAM-COMIC http://friam-comic.blogspot.com/
archives:  5/2017 thru present https://redfish.com/pipermail/friam_redfish.com/
  1/2003 thru 6/2021  http://friam.383.s1.nabble.com/