Space: Apache Mahout (https://cwiki.apache.org/confluence/display/MAHOUT)
Page: Data Formats
(https://cwiki.apache.org/confluence/display/MAHOUT/Data+Formats)
Added by Lance Norskog:
---------------------------------------------------------------------
Mahout uses a few file formats quite a bit in its various job implementations.
{toc}
h2. File formats
h3. Raw formats for import
* Text files
** can be parsed into SequenceFiles of:
*** (line number, text of line)
*** (file name, full contents of file)
*** (line number, parts of line extracted with regex patterns)
** can also be parsed into Lucene indexes:
*** precise schema designs ???
* ARFF files
** Weka project text data format
* Mailbox files
** can be parsed into SequenceFiles of:
*** (mail message id, text body of mail message)
*** no html or attachment support
* CSV files
** generally without column or row headers
** no "multiple values per column" options
* Hadoop SequenceFile
** canonical, no variations. Currently no use of metadata.
* Lucene indexes
** translated into SequenceFiles
*** precise mapping ???
h3. Raw formats for export
* SequenceFiles
* Text lines, mostly of the toString() variety
* MatrixWritable for ConfusionMatrix
h2. Who Stores What in a SequenceFile?
h3. "Simple" Text Vectors
Simple text vectors represent documents. The dimensions are the set of terms in
the entire document set. Each document vector stores a number in the position
of each term it contains. This number may be derived from the count of the term
inside the document.
h3. Encoded Text Vectors
Each vector represents a document. However, term dimensions are "collapsed"
stochastically, meaning each term in the full term set is mapped randomly to
several smaller indexes.
h3. Directories
<Integer,Text> or <Long,Text> pairs which match matrix rows to input text keys
like movie names, document file names etc.
h3. Matrices
Matrices are almost universally stored as LongWritable/VectorWritable pairs,
where VectorWritable can be sparse or dense.
h3. Clusters
Clusters are stored in complex data structures.
h3. FPGrowth Clusters
These are stored in a custom data structure.
h2. Life cycle
All Mahout jobs generally assume that files generated have no lifespan. All
Writable formats may change, and some may disappear. There are no file
compatibility requirements.
Change your notification preferences:
https://cwiki.apache.org/confluence/users/viewnotifications.action
