[
https://issues.apache.org/jira/browse/MAHOUT-944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13211744#comment-13211744
]
Frank Scholten commented on MAHOUT-944:
---------------------------------------
Whirr Hadoop cluster works again, see WHIRR-518
Now the index is split at the segment level. Each mapper processes one segment.
The downside is that input splits have different sizes and the number of map
tasks
equals the number of segments.
I think is a problem but maybe not in a situation with many shards? If it is a
problem do you have any suggestions? Perhaps a split should be part of a
segment. How should I implement this, by combining this with my earlier
implementation?
> LuceneIndexToSequenceFiles (lucene2seq) utility
> -----------------------------------------------
>
> Key: MAHOUT-944
> URL: https://issues.apache.org/jira/browse/MAHOUT-944
> Project: Mahout
> Issue Type: New Feature
> Components: Integration
> Affects Versions: 0.5
> Reporter: Frank Scholten
> Assignee: Grant Ingersoll
> Priority: Minor
> Fix For: 0.7
>
> Attachments: MAHOUT-944.patch, MAHOUT-944.patch, MAHOUT-944.patch,
> MAHOUT-944.patch
>
>
> Here is a lucene2seq tool I used in a project. It creates sequence files
> based on the stored fields of a lucene index.
> The output from this tool can be then fed into seq2sparse and from there you
> can do text clustering.
> Comes with Java bean configuration.
> Let me know what you think. Some CLI code can be added later on. I used this
> for a small-scale project +- 100.000 docs. Is a MR version useful or is that
> overkill?
> See https://github.com/frankscholten/mahout/tree/lucene2seq for commits and
> review comments from Simon Willnauer (Thanks Simon!)
> or the attached patch.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
