From: *Travis* Date: Mon, Feb 23, 2009 Subject: Exploring a 'Deep Web' That Google Can't Grasp
*Exploring a 'Deep Web' That Google Can't Grasp* By ALEX WRIGHT * http://www.nytimes.com/2009/02/23/technology/internet/23search.html?_r=1&th=&emc=th&pagewanted=print *<http://www.nytimes.com/2009/02/23/technology/internet/23search.html?_r=1&th=&emc=th&pagewanted=print> One day last summer, *Google*<http://topics.nytimes.com/top/news/business/companies/google_inc/index.html?inline=nyt-org>'s search engine trundled quietly past a milestone. It added the one trillionth address to the list of Web pages it knows about. But as impossibly big as that number may seem, it represents only a fraction of the entire Web. Beyond those trillion pages lies an even vaster Web of hidden data: financial information, shopping catalogs, flight schedules, medical research and all kinds of other material stored in databases that remain largely invisible to search engines. The challenges that the major search engines face in penetrating this so-called Deep Web go a long way toward explaining why they still can't provide satisfying answers to questions like "What's the best fare from New York to London next Thursday?" The answers are readily available — if only the search engines knew how to find them. Now a new breed of technologies is taking shape that will extend the reach of search engines into the Web's hidden corners. When that happens, it will do more than just improve the quality of search results — it may ultimately reshape the way many companies do business online. Search engines rely on programs known as crawlers (or spiders) that gather information by following the trails of hyperlinks that tie the Web together. While that approach works well for the pages that make up the surface Web, these programs have a harder time penetrating databases that are set up to respond to typed queries. "The crawlable Web is the tip of the iceberg," says Anand Rajaraman, co-founder of Kosmix (*www.kosmix.com* <http://www.kosmix.com/>), a Deep Web search start-up whose investors include *Jeffrey P. Bezos*<http://topics.nytimes.com/top/reference/timestopics/people/b/jeffrey_p_bezos/index.html?inline=nyt-per>, chief executive of *Amazon.com* <http://amazon.com/>. Kosmix has developed software that matches searches with the databases most likely to yield relevant information, then returns an overview of the topic drawn from multiple sources. "Most search engines try to help you find a needle in a haystack," Mr. Rajaraman said, "but what we're trying to do is help you explore the haystack." That haystack is infinitely large. With millions of databases connected to the Web, and endless possible permutations of search terms, there is simply no way for any search engine — no matter how powerful — to sift through every possible combination of data on the fly. To extract meaningful data from the Deep Web, search engines have to analyze users' search terms and figure out how to broker those queries to particular databases. For example, if a user types in "Rembrandt," the search engine needs to know which databases are most likely to contain information about art ( say, museum catalogs or auction houses), and what kinds of queries those databases will accept. That approach may sound straightforward in theory, but in practice the vast variety of database structures and possible search terms poses a thorny computational challenge. "This is the most interesting data integration problem imaginable," says Alon Halevy, a former computer science professor at the *University of Washington*<http://topics.nytimes.com/top/reference/timestopics/organizations/u/university_of_washington/index.html?inline=nyt-org> who is now leading a team at Google that is trying to solve the Deep Web conundrum. Google's Deep Web search strategy involves sending out a program to analyze the contents of every database it encounters. For example, if the search engine finds a page with a form related to fine art, it starts guessing likely search terms — "Rembrandt," "Picasso," "Vermeer" and so on — until one of those terms returns a match. The search engine then analyzes the results and develops a predictive model of what the database contains. In a similar vein, Prof. Juliana Freire at the *University of Utah*<http://topics.nytimes.com/top/reference/timestopics/organizations/u/university_of_utah/index.html?inline=nyt-org> is working on an ambitious project called DeepPeep (*www.deeppeep.org*<http://www.deeppeep.org/>) that eventually aims to crawl and index every database on the public Web. Extracting the contents of so many far-flung data sets requires a sophisticated kind of computational guessing game. "The naïve way would be to query all the words in the dictionary," Ms. Freire said. Instead, DeepPeep starts by posing a small number of sample queries, "so we can then use that to build up our understanding of the databases and choose which words to search." Based on that analysis, the program then fires off automated search terms in an effort to dislodge as much data as possible. Ms. Freire claims that her approach retrieves better than 90 percent of the content stored in any given database. Ms. Freire's work has recently attracted overtures from one of the major search engine companies. As the major search engines start to experiment with incorporating Deep Web content into their search results, they must figure out how to present different kinds of data without overcomplicating their pages. This poses a particular quandary for Google, which has long resisted the temptation to make significant changes to its tried-and-true search results format. "Google faces a real challenge," said Chris Sherman, executive editor of the Web site Search Engine Land. "They want to make the experience better, but they have to be supercautious with making changes for fear of alienating their users." Beyond the realm of consumer searches, Deep Web technologies may eventually let businesses use data in new ways. For example, a health site could cross-reference data from pharmaceutical companies with the latest findings from medical researchers, or a local news site could extend its coverage by letting users tap into public records stored in government databases. This level of data integration could eventually point the way toward something like the Semantic Web, the much-promoted — but so far unrealized — vision of a Web of interconnected data. Deep Web technologies hold the promise of achieving similar benefits at a much lower cost, by automating the process of analyzing database structures and cross-referencing the results. "The huge thing is the ability to connect disparate data sources," said Mike Bergman, a computer scientist and consultant who is credited with coining the term Deep Web. Mr. Bergman said the long-term impact of Deep Web search had more to do with transforming business than with satisfying the whims of Web surfers. *(F)AIR USE NOTICE: All original content and/or articles and graphics in this message are copyrighted, unless specifically noted otherwise. All rights to these copyrighted items are reserved. Articles and graphics have been placed within for educational and discussion purposes only, in compliance with "Fair Use" criteria established in Section 107 of the Copyright Act of 1976. The principle of "Fair Use" was established as law by Section 107 of The Copyright Act of 1976. "Fair Use" legally eliminates the need to obtain permission or pay royalties for the use of previously copyrighted materials if the purposes of display include "criticism, comment, news reporting, teaching, scholarship, and research." Section 107 establishes four criteria for determining whether the use of a work in any particular case qualifies as a "fair use". A work used does not necessarily have to satisfy all four criteria to qualify as an instance of "fair use". Rather, "fair use" is determined by the overall extent to which the cited work does or does not substantially satisfy the criteria in their totality. If you wish to use copyrighted material for purposes of your own that go beyond 'fair use,' you must obtain permission from the copyright owner. For more information go to: **http://www.law.cornell.edu/uscode/17/107.shtml*<http://www.law.cornell.edu/uscode/17/107.shtml> ** *THIS DOCUMENT MAY CONTAIN COPYRIGHTED MATERIAL. COPYING AND DISSEMINATION IS PROHIBITED WITHOUT PERMISSION OF THE COPYRIGHT OWNERS.* __._,_.___ Messages<http://groups.yahoo.com/group/grendelreport/messages;_ylc=X3oDMTJmdnBjcmMxBF9TAzk3MzU5NzE0BGdycElkAzIwMTk0ODA2BGdycHNwSWQDMTcwNTMyMzY2NwRzZWMDZnRyBHNsawNtc2dzBHN0aW1lAzEyMzU0MzM0Njk-> MARKETPLACE ------------------------------ >From kitchen basics to easy recipes - join the Group from Kraft Foods <http://us.ard.yahoo.com/SIG=13r0m3v8j/M=493064.12016295.13271503.10835568/D=groups/S=1705323667:MKP1/Y=YAHOO/EXP=1235440669/L=/B=IUh2R0PDhFw-/J=1235433469778565/A=5530388/R=0/SIG=11nuutlas/*http://explore.yahoo.com/groups/kraftmealsmadesimple/> [image: Yahoo! Groups]<http://groups.yahoo.com/;_ylc=X3oDMTJlZTVpMjgwBF9TAzk3NDc2NTkwBGdycElkAzIwMTk0ODA2BGdycHNwSWQDMTcwNTMyMzY2NwRzZWMDZnRyBHNsawNnZnAEc3RpbWUDMTIzNTQzMzQ2OQ--> Change settings via the Web<http://groups.yahoo.com/group/grendelreport/join;_ylc=X3oDMTJnbDFuOGJ1BF9TAzk3NDc2NTkwBGdycElkAzIwMTk0ODA2BGdycHNwSWQDMTcwNTMyMzY2NwRzZWMDZnRyBHNsawNzdG5ncwRzdGltZQMxMjM1NDMzNDY5>(Yahoo! ID required) Change settings via email: Switch delivery to Daily Digest<[email protected]?subject=email+delivery:+Digest>| Switch format to Traditional<[email protected]?subject=change+delivery+format:+Traditional> Visit Your Group <http://groups.yahoo.com/group/grendelreport;_ylc=X3oDMTJlMHE0cDRtBF9TAzk3NDc2NTkwBGdycElkAzIwMTk0ODA2BGdycHNwSWQDMTcwNTMyMzY2NwRzZWMDZnRyBHNsawNocGYEc3RpbWUDMTIzNTQzMzQ2OQ-->| Yahoo! Groups Terms of Use <http://docs.yahoo.com/info/terms/>| Unsubscribe <[email protected]?subject=> Recent Activity - 1 New Members<http://groups.yahoo.com/group/grendelreport/members;_ylc=X3oDMTJnbXE2OHNsBF9TAzk3MzU5NzE0BGdycElkAzIwMTk0ODA2BGdycHNwSWQDMTcwNTMyMzY2NwRzZWMDdnRsBHNsawN2bWJycwRzdGltZQMxMjM1NDMzNDY5> Visit Your Group <http://groups.yahoo.com/group/grendelreport;_ylc=X3oDMTJmOWJodXBvBF9TAzk3MzU5NzE0BGdycElkAzIwMTk0ODA2BGdycHNwSWQDMTcwNTMyMzY2NwRzZWMDdnRsBHNsawN2Z2hwBHN0aW1lAzEyMzU0MzM0Njk-> Yahoo! News Odd News<http://us.ard.yahoo.com/SIG=13oo80mdi/M=493064.12016309.12445701.8674578/D=groups/S=1705323667:NC/Y=YAHOO/EXP=1235440669/L=/B=Ikh2R0PDhFw-/J=1235433469778565/A=3848614/R=0/SIG=12t4qk00m/*http://news.yahoo.com/i/757;_ylt=A9FJqYzfwK5EFCQAswis0NUE;_ylu=X3oDMTA3NW1oMDRpBHNlYwM3NTc-> You won't believe it, but it's true Y! Groups blog the best source<http://us.ard.yahoo.com/SIG=13ogp7jf4/M=493064.12016258.12582637.8674578/D=groups/S=1705323667:NC/Y=YAHOO/EXP=1235440669/L=/B=I0h2R0PDhFw-/J=1235433469778565/A=5191955/R=0/SIG=112mhte3e/*http://www.ygroupsblog.com/blog/> for the latest scoop on Groups. Group Charity Hands On Network<http://us.ard.yahoo.com/SIG=13op3jfi4/M=493064.12016306.12445698.8674578/D=groups/S=1705323667:NC/Y=YAHOO/EXP=1235440669/L=/B=JEh2R0PDhFw-/J=1235433469778565/A=5579904/R=0/SIG=110vban8o/*http://www.handsonnetwork.org/> Volunteering has never been so easy . __,_._,___ -- *~@):~{> -- *~@):~{> --~--~---------~--~----~------------~-------~--~----~ Thanks for being part of "PoliticalForum" at Google Groups. For options & help see http://groups.google.com/group/PoliticalForum * Visit our other community at http://www.PoliticalForum.com/ * It's active and moderated. Register and vote in our polls. * Read the latest breaking news, and more. -~----------~----~----~----~------~----~------~--~---
