[ https://issues.apache.org/jira/browse/SPARK-10949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Reynold Xin resolved SPARK-10949. --------------------------------- Resolution: Fixed Assignee: Josh Rosen Fix Version/s: 1.6.0 > Upgrade Snappy Java to 1.1.2 > ---------------------------- > > Key: SPARK-10949 > URL: https://issues.apache.org/jira/browse/SPARK-10949 > Project: Spark > Issue Type: Improvement > Components: Spark Core > Affects Versions: 1.5.0, 1.5.1 > Reporter: Adam Roberts > Assignee: Josh Rosen > Priority: Minor > Fix For: 1.6.0 > > > Snappy now supports concatenation of serialized streams, this patch contains > a version number change and the "does not support" test is now a "supports" > test. > Note: I do have the pull request for this already created, tested to be OK on > Intel 64 bit Linux, IBM Power 8 LE and Linux on IBM Z Systems (all with IBM > Java 8). Also note that I was required to delete my m2 cache in order to > resolve incompatible class exceptions when building. > Snappy 1.1.2 changelog mentions: > snappy-java-1.1.2 (22 September 2015) > This is a backward compatible release for 1.1.x. > Add AIX (32-bit) support. > There is no upgrade for the native libraries of the other platforms. > A major change since 1.1.1 is a support for reading concatenated results of > SnappyOutputStream(s) > snappy-java-1.1.2-RC2 (18 May 2015) > Fix #107: SnappyOutputStream.close() is not idempotent > snappy-java-1.1.2-RC1 (13 May 2015) > SnappyInputStream now supports reading concatenated compressed results of > SnappyOutputStream > There has been no compressed format change since 1.0.5.x. So You can read the > compressed results interchangeablly between these versions. > Fixes a problem when java.io.tmpdir does not exist. > From https://github.com/xerial/snappy-java/blob/develop/Milestone.md and up > to date at the time of this pull request > Also note https://github.com/xerial/snappy-java/issues/103 > "@xerial not sure how feasible or likely it is for this to happen, but it'd > help tremendously Spark's performance because we are experimenting with a new > shuffle path that uses channel.transferTo to avoid user space copying. > However, for that to work, we'd need the underlying format to support > concatenation. As far we know, LZF has this property, and Snappy might also > have it (but snappy-java implementation doesn't support it)." -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org