Hi Andi, I too have thought about using the human/computer interface as sensors and actuators for an AI. This would accomplish a few things: 1. Grounding of symbols in a real world that is mostly symbolic and precise to begin with. 2. Behaviors and actions could be developed that are useful to the AI developer as well as end users. 3. The sensor and actuator components can be derived from GUI test tools as you indicated.
Some years ago I evaluated and used GUI test components on the Windows platform. It is not too hard to get information about each window, to acquire the content of text fields, and to insert window/keyboard/mouse events into the Windows event handler. But nowadays I am strictly Linux. I've not taken the time to look at the details, as my project ( http://sf.net/projects/texai ) is far from the point of hooking up sensors and actuators. But I would look beyond java.awt.Robot because the facility you describe should be applicable to the full range of GUI/keyboard/mouse percepts and actions available to the human user. To me, this means OCR (optical character recognition) at the framebuffer level because many GUI tools on Linux perform their own font rendering; the operating system frequently cannot determine the text contents of a window. A waypoint on the path to achieving this goal would be sensors and actuators that plug into the OS and application API's that populate a computer desktop. So for starters I am thinking about a sensor for computer performance, memory utilization, running processes, etc. A comparable actuator might be to launch an optional subtask, or to adjust priority. These simple things could ground an AI's notion of how it is running on a particular computer. Thanks for raising this issue that I've already thought some about. -Steve --- Andrew Babian <[EMAIL PROTECTED]> wrote: > I wrote: > > > I just had a notion. The proper sensory input > and motor output for an AI is > > > the computer screen (and sound input and regular > keyboard and mouse input). > > > One thing that needs to exist is a freely > available standard API for these > > > things, so people can work on them, plus > implementations for the platforms > > > that people use. My hope is that it would give > different researchers, > > > especially all those lone wolves out there, > something intercompatible to work > > > with. It also seems possible that this could be > a common mechanism for the > > > different systems to > > > work together, in a sort of extension of the > Blackboard model. And, as a > > > lighter element of it, I'd really like it if > these projects could use video > > > games, because they more and more have become > very sophisticated real-world > > > modelling tools. > > > andi > > And Richard Loosemore asked for clarification: > > Can you be more specific about what this would > entail? I can think > > of several interpretations of what you say, but am > not sure which > > you mean. > > Well, if you can think of several interpretations, > then why don't you pick one > you like? > > I was thinking along the lines of java.awt.Robot. I > only had a vague > recollection of it, and I never used it, and looking > at it again now, I think > it is exactly what I was thinking of. Another > reason I thought of it is that > Stan Franklin's Ida model uses e-mail as a sort of > sensory-motor and that's a > kind of subset of this notion. It seems like the > standard reactions people > have when they wonder what an artificial > intelligence is going to do is either > sit in a box and answer questions or control a > physical robot clunking around > the world. I would simply propose that one other > useful answer is to control > and use a computer the way a person might control a > computer. This would mean > that it could use all manner of existing tools to > multiply whatever power its > additional "intelligence" adds. > > But one of the tricky bits of the idea is having > something sufficiently > general and useful enough to make a contribution. > As I mentioned, there is a > Java class that does the kind of thing I'm > interested in. And it's probably > straightforward to have this kind of thing in other > imperative languages. But > how would you have a neural network system interface > to it? I don't know, > maybe the API idea is foolish. I've never tried to > design one, so I don't > particularly know what's involved or if it's even a > good idea. The really > basic functions I would expect are an ability to > capture a piece of the > screen, to control the mouse, and input keyboard > events. I think a very > valuable addition would be to discover a character > (or piece of text) that's > at a particular location, so reading in text from a > screen would be easier. > We have to do this to use a computer and any agent > using a computer would need > to do this anyway, so it would be just more useful > to add that in at the > beginning. Unfortunately, that could be a tricky > bit of code, but it is miles > away from OCR, so it isn't unreasonable. I also > mentioned having access to > the sound streams. People can get away with not > using the sound on a > computer, so clearly it wouldn't be necessary for an > artificial agent using it > to use it, but it might make a valuable addition. > And it might a useful > feature if part of this interface enabled an AI to > simply watch what a person > (or conceivably another agent) was doing, which > could open opportunities for > some kind of instruction or learning. > > andi > > ----- > This list is sponsored by AGIRI: > http://www.agiri.org/email > To unsubscribe or change your options, please go to: > http://v2.listbox.com/member/[EMAIL PROTECTED] > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com ----- This list is sponsored by AGIRI: http://www.agiri.org/email To unsubscribe or change your options, please go to: http://v2.listbox.com/member/[EMAIL PROTECTED]
