Pedro, Thanks for the link there is a lot of honest, direct and useful feedback in that thread from Dr. LeCun and others.
This is a cautionary tale for our community. Mainline AI/ML research has been maligned so often it now takes extraordinary results to make even modest claims. To garner greater acceptance for NuPIC and CLA we need replicable results at or above anything out there on accepted problems and datasets. Luckily we're open source so no one can claim we're hiding anything. In some ways this is unfortunate as many of CLAs unique strengths (inherently online learning, variable and high order sequence memory, temporal pooling) could be missed or ignored by the wider research field until we tackle some of the classic problems head on. Ian On Oct 28, 2013 12:12 PM, "Pedro Tabacof" <[email protected]> wrote: > Yann LeCun (deep learning expert) offers a skeptical view: > > AI startup Vicarious claims to have a system that can solve CAPTCHAs with >> "succes rate up to 90%". >> Beware: It's a textbook example of AI hype of the worst kind >> Hype is dangerous to AI. Hype killed AI four times in the last five >> decades. AI Hype must be stopped. >> Perhaps Vicarious can get "up to 90%" accuracy on some CAPTCHA dataset >> they cooked up, but >> - (1) breaking CAPTCHAs is hardly an interesting task, unless you are a >> spammer; >> - (2) it's easy to claim success on a dataset you cooked up yourself. >> There is no risk someone else will beat you. >> - (3) recognizing object in images is much, much harder than breaking >> CAPTCHAs. Some deep learning systems can already do this with decent >> accuracy. Some such systems have been deployed by Google and Baidu. >> - (4) doing simultaneous segmentation and recognition of character >> strings is hardly a breakthrough. See demos of a 20 year-old system here: >> http://yann.lecun.com/exdb/lenet/index.html >> The sad thing is that this announcement is being picked up by a number >> publications, including MIT Tech Review, Forbes, etc. >> Here is an advice to scientific/tech journalists: please, please do not >> believe vague claims by AI startupsunless they produce state of the art >> results on widely accepted benchmarks. >> >> This is particularly true for claims in image and speech recognition for >> which good benchmarks exists. For image recognition, a good example of such >> benchmark would be the ImageNet Large Scale Visual Recognition Challenge. >> Whenever a startup claims "90% accuracy" on some random task, do not >> consider this newsworthy. If the company also makes claims like "we are >> developing machine learning software based on the computational principles >> of the human brain" or uses impressive-sounding names like "Recursive >> Cortical Network", be even more suspicious. >> >> There are extremely impressive applications of deep learning out there >> (e.g. deployed by Google, Baidu, Microsoft, IBM, and a few startups), but >> this is not one of them. >> Google's automatic photo tagger and Baidu's image retrieval system are >> much, much more impressive than the system in this announcement. Even if we >> just talk about challenging character recognition tasks, Google's system >> for picking out house numbers in StreetView images is way more impressive >> than this. >> >> AI "died" about four times in five decades because of hype: people made >> wild claims (often to impress potential investors or funding agencies) and >> could not deliver. Backlash ensued. It happened twice with neural nets >> already: once in the late 60's and again in the mid-90's. >> Don't let it happen again. Beware of hype. >> And by the way, no one is interested in breaking CAPTCHAs except spammers >> and computer security researchers. That's why you won't find many computer >> vision papers on the topic. That's also why it would be easy to break >> records, even if a standard dataset existed. > > > Dileep George's rebuttal: > > Hi Yann, >> (1) CAPTCHA contains many of the problems that make general vision >> hardhttp://tinyurl.com/mkhllyu. We will be publishing results on >> standard benchmarks in the future as well. >> (2) We get 90% pass rate on a validation set of 10,000 captchas >> downloaded from reCAPTCHA on Nov 5 at 11:25AM. You can download the data >> for yourself here: >> https://www.dropbox.com/s/sqr7b6ck0bzt0ur/recaptcha10k.zip >> (3) We recognize objects in images too, this is just one demo of our >> system. >> (4) Looks like you linked to the wrong video, because the letters in that >> video look pretty well separated and easily segmented out. I'd like to see >> any current system parse modern CAPTCHAs. >> >> Out of curiosity, did you also have this reaction to the news about >> Watson? It's good to (sometimes) post results that the average person can >> connect with. > > > > Source: https://plus.google.com/104362980539466846301/posts/Qwj9EEkUJXY > > > > On Mon, Oct 28, 2013 at 5:03 PM, Azat <[email protected]> wrote: > >> http://www.kurzweilai.net/vicarious-ai-breaks-captcha-turing-test >> >> Azat >> >> _______________________________________________ >> nupic mailing list >> [email protected] >> http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org >> > > > > -- > Pedro Tabacof, > Unicamp - Eng. de Computação 08. > > _______________________________________________ > nupic mailing list > [email protected] > http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org > >
_______________________________________________ nupic mailing list [email protected] http://lists.numenta.org/mailman/listinfo/nupic_lists.numenta.org
