Stephen Reed wrote:
Richard,
In a comment to a recent blog post of mine
<http://texai.org/blog/2008/01/20/bootstrap-dialog-system-design>
Evgenii asked for my comments on the following automatic programming
task. I responded with an enumeration of the types of problem-solving
knowledge and behaviors that a solution requires:
[Evgenii] ... I’ll share my use case which my system used. My ultimate
goal for the first stage was to create an automatic programmer who would
be capable of parsing the following specification, and capable to emit a
Java program that implements this specification.
The specification was:
Write a TCP echo server. (Client connects to the server, sends bytes to
the server, and receives them back immediately, until client does
disconnect on behalf of its own). Server must be able to serve more than
one client simultaneously. The server should be implemented in Java.
[Steve] ... I expect that your [Evgenii's] system should have an
adequate commonsense understanding of the application domain, and
relevant domain algorithms:
* TCP socket protocol [at the transport layer, e.g. how to listen,
how to connect, how to read/write, & close a socket]
* Client/Server paradigm
* multithreading, including thread safety
* what “echo” means
It should have an understanding of the generally applicable programming
algorithms that are relevant:
* object-oriented program classes and instances
* variable typing
* statement sequencing and control
* method definition and invocation
* program composition (e.g. initialization, process, finalization,
and exception handling)
* test case generation
* programming safety (i.e. be Friendly and do not cause the host to
crash)
And it should understand the relevant Java concepts:
* how to compose a Java class
* how to map attributes of domain objects to Java instance variables
* how to compose Java statements
* how to use applicable class libraries
* how to compile and execute a Java program
If the Texai dialog system were applied to your [Evgenii's] use case,
then I expect a great deal of the effort would be to acquire the
background algorithm knowledge from human teachers. Furthermore, I think
that the specified program would be developed during a dialog with the
teacher. It would only be after the system has adequately learned the
subject, that it could program automatically from text specifications.
Interesting.
It might be worth sharing my (brief) reaction to this.
I see certain "deep" knowledge that is required to do this task (I will
explain what that is in a moment), and I wonder when and how that kind
of deep knowledge would get into the system, or whether it would be
missed entirely...
For example: if I were the system right now, I would have to do some
work to figure out the almost-trivial question of where to put a
"server" process on my machine, to test it, because writing
client-server code is not my specialty.
Now, assuming that the AP system were coming across a new type of
programming task, in this example (the way that I am), would the AP
system have to be explicitly told such background knowledge about how to
handle client-server tasks, or would it have the ability to look up a
textbook that would give the answer? I suspect it would not be able to
use the textbook.
Second: the software development environment used by Mac developers was
recently rewritten (again) and one result of this is that all the books
that describe how to get into Mac programming are out of date. So out
of date, in fact, that it would take a very experienced developer to
figure out how to relate the layout of the new environment to the
instructions given in the books. Would the AP system be able to
recognize a change like that, and cope with it? Again, I doubt it.
Obviously, this is expecting quite a lot from an AP system at this point
in time, but what worries me is that so much of *real* down-n-dirty
computer programming is either (a) having to go into territory that you
have never been in before, which requires research done under pressure,
or (b) adjusting to massive structural changes, with no guidance, that
an AP system might find itself falling over just because of miniature
versions of the two problems I just described.
It seems to me that most AP approaches are geared toward the "plain
sailing" problems that do not have the nasty little issues described
above, and that the "rough seas" type of problems are postponed for
another day. Since an awful lot of good work can go into solving plain
sailing systems that are very large (Czarnecki & Eisenecker, for
example), it can seem like good progress is being made. I think what
then happens is that people do not realise that, in order to go on to
the nasty type of programming problem (i.e. 99% of all programming
problems), all the previous work may count for nothing, because it did
not even in principle address such deep issues as what to do if the only
way forward is to find the right textbook, read a couple of pages, and
understand it.
Of course, the same criticism can be levelled at many approaches to the
entire AI problem, never mind just the AP problem. :-)
Richard Loosemore
-----
This list is sponsored by AGIRI: http://www.agiri.org/email
To unsubscribe or change your options, please go to:
http://v2.listbox.com/member/?member_id=8660244&id_secret=94201972-9013fa
