[
https://jira.nuxeo.com/browse/NXSEM-7?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=95796#comment-95796
]
Olivier Grisel commented on NXSEM-7:
------------------------------------
pignlproc is now updated to output NTriples and there is a README.md file
explaining how to generate the file to index. Here is a public copy of the
result:
http://dl.dropbox.com/u/5743203/IKS/dbpedia/3.6/topics_merged_abstracts.nt.bz2
> develop hadoop based toolset to build categorized TF-IDF corpora to train
> document classification models
> --------------------------------------------------------------------------------------------------------
>
> Key: NXSEM-7
> URL: https://jira.nuxeo.com/browse/NXSEM-7
> Project: Nuxeo Semantic R&D
> Issue Type: Task
> Reporter: Olivier Grisel
> Assignee: Olivier Grisel
> Priority: Major
> Fix For: 5.4.2
>
>
> The toolset should be packaged to be easily deployable on AWS using the
> Cloudera Distribution for Hadoop 2 AMI [1] and the wikipedia XML dump AWS
> Dataset [2].
> The Mahout project already has some partial implementation of this (lacking
> the TF-IDF [3] part). To avoid having to load a huge dictionary in memory, we
> plan to leverage a hashed representation [4] of the term and document
> frequencies.
> [1] http://archive.cloudera.com/docs/ec2.html
> [2]
> http://aws.typepad.com/aws/2009/09/new-public-data-set-wikipedia-xml-data.html
> [3] http://en.wikipedia.org/wiki/Tf-idf
> [4] http://hunch.net/~jl/projects/hash_reps/index.html
> Edit: the in-memory hashed representation is replaced by sparse persisted
> representation using Solr / Lucene. The TF-IDF k-Nearest Neighbors lookups
> will be implemented using MoreLikeThis queries instead. See comments for
> details.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
_______________________________________________
ECM-tickets mailing list
[email protected]
http://lists.nuxeo.com/mailman/listinfo/ecm-tickets
