Ioannis Kerkinos created CRUNCH-505:
---------------------------------------
Summary: Store intermediate data in memory only using Tachyon
Key: CRUNCH-505
URL: https://issues.apache.org/jira/browse/CRUNCH-505
Project: Crunch
Issue Type: Improvement
Components: Core
Affects Versions: 0.12.0
Reporter: Ioannis Kerkinos
Assignee: Josh Wills
Tachyon is a memory-centric distributed storage system that enables reliable
data sharing at memory-speed. If used as the storage for intermediate data
(between MR jobs) it should improve performance as you won't have to go to
HDFS. In order to do so, the MUST_CACHE write type of Tachyon can be used. This
will enable data to be persisted in memory only without going to HDFS. So the
intermediate data will be read/written at memory-speed and only the final
result will be written in HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
