Hi, Claudio I am doing something about query clustring. You know, in search engine , people always enter query content with few words, finding the similary queries is my job, and from this similary queries, i'd like to find synonym words.
In order to find the similary queries, only using the query content is impossible, because content have serverl words (two or three words). In relative research papers, we suppose the different queries which have same result documents clicked by users are similar. So, the query and the result documents that clicked by users can be constructed as vector or matrix. Query1: DocID1, DocID2, DocIDX,... Query2: DocIDX, DocIDY, DocIDZ,... QueryX: .......... QueryX is the query content which input by users, DocID is the document clicked by user who input the query QueryX. Throug hadoop, I get the matrix and would like to cluster the similary Queries. I reviewed the sourecode of KMeans in Mahout. I will give it a try. Claudio, i don't konw much about textrand. In google, TextRank: Bringing Order into Texts is found, sounds it is useful for text processing and will helpful for me. Thanks! On Thu, Feb 11, 2010 at 5:09 PM, Claudio Martella < [email protected]> wrote: > I don't know what kind of algorithm you're using, have you every thought > of textrank? pagerank applied to automatic keyword extraction. > > > > Andrew Wang wrote: > > OK, i will give it a try. > > > > ps: The solution for vectorizing document is cool, look forward to it ! > > > > On Thu, Feb 11, 2010 at 4:31 PM, Robin Anil <[email protected]> > wrote: > > > > > >> Thanks for replying. Clustering algorithms do work with 0.19 and in > this > >> coming release we are including a hadoop based solution for vectorizing > >> document. Hope you will like it > >> > >> Robin > >> > >> > >> On Thu, Feb 11, 2010 at 1:46 PM, Andrew Wang < > [email protected] > >> > >>> wrote: > >>> > >>> Hi, Robin > >>> > >>> In my work, i have a lot of query log which produced by search engine > and > >>> we > >>> use hadoop as our tool to analyse those data. Sometimes, i'd like to > some > >>> data mining job such as clustering the similary queries, or classify > >>> > >> them. > >> > >>> At first time, i think the mahout maybe another option for me to do > data > >>> mining job (as you know, the weka is my favorable data mining tool). > But, > >>> as > >>> i try to integrate mahout into my project, i find two major obstacles > to > >>> prevent me moving on further: > >>> > >>> First, in my company, The hadoop with 0.19 is provided as platform for > us > >>> to > >>> do daily jobs. As we know, Mahout is dependent the hadoop with 0.2 or > >>> above. > >>> This prevent me from benefiting from the functions which provided by > >>> mahout. > >>> > >>> Secondly, the input data should be indexed by Lucene firstly( right or > >>> wrong? ), then be imported by the Mahout. It confuse me very much, > >>> > >> because > >> > >>> there are so many data stored by HDFS. In order to use the Mahout, i > have > >>> to > >>> check out all the data firstly ,and indexed by Lucene, and so on. It is > >>> unbelievable for me. > >>> > >>> So, i haven't use the mahout in my daily work. However, i always give > my > >>> attendtion to the Mahout, maybe someday i benefit from it. > >>> > >>> What about other one's idea? > >>> > >>> On Wed, Feb 10, 2010 at 6:19 PM, Robin Anil <[email protected]> > >>> > >> wrote: > >> > >>>> Hi Mahouters > >>>> I am trying to find out how you are using Mahout for your work or > >>>> project, or which among the algorithms in Mahout are more important > for > >>>> > >>> you > >>> > >>>> to do that work. And finally what do you expect to see in Mahout(A > kind > >>>> > >>> of > >>> > >>>> a > >>>> wish list). It wont take much of your time. Please reply with this > >>>> > >>> details. > >>> > >>>> It will help a great deal in figuring out where what we need to > >>>> prioritize. > >>>> > >>>> Thanks > >>>> Robin > >>>> > >>>> > >>> > >>> -- > >>> http://anqiang1900.blog.163.com/ > >>> > >>> > > > > > > > > > > > -- > Claudio Martella > Digital Technologies > Unit Research & Development - Analyst > > TIS innovation park > Via Siemens 19 | Siemensstr. 19 > 39100 Bolzano | 39100 Bozen > Tel. +39 0471 068 123 > Fax +39 0471 068 129 > [email protected] http://www.tis.bz.it > > Short information regarding use of personal data. According to Section 13 > of Italian Legislative Decree no. 196 of 30 June 2003, we inform you that we > process your personal data in order to fulfil contractual and fiscal > obligations and also to send you information regarding our services and > events. Your personal data are processed with and without electronic means > and by respecting data subjects' rights, fundamental freedoms and dignity, > particularly with regard to confidentiality, personal identity and the right > to personal data protection. At any time and without formalities you can > write an e-mail to [email protected] in order to object the processing of > your personal data for the purpose of sending advertising materials and also > to exercise the right to access personal data and other rights referred to > in Section 7 of Decree 196/2003. The data controller is TIS Techno > Innovation Alto Adige, Siemens Street n. 19, Bolzano. You can find the > complete information on the web site www.tis.bz.it. > > > -- http://anqiang1900.blog.163.com/
