[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17817233#comment-17817233 ] Jakub Wozniak commented on SPARK-18105: --- With Spark 3.5.0 running on Yarn Hadoop we had this:

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17511815#comment-17511815 ] hujiahua commented on SPARK-18105: -- It's working in my case by setting spark.file.transferTo=false.

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17452730#comment-17452730 ] Yuming Wang commented on SPARK-18105: - Workaround this issue by set spark.io.compression.codec=zstd.

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17445513#comment-17445513 ] Siddharth Kumar commented on SPARK-18105: - Hi, I saw a similar failure just as [~vladimir.prus].

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17444386#comment-17444386 ] Wei Zhang commented on SPARK-18105: --- In our case, it is strongly related with `spark.file.transferTo`.

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421971#comment-17421971 ] wuyi commented on SPARK-18105: -- [~vladimir.prus] Hi, could you also file a sub-task under

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421969#comment-17421969 ] Vladimir Prus commented on SPARK-18105: --- FYI, we recently started to get a lot of such errors;

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17406622#comment-17406622 ] wuyi commented on SPARK-18105: -- FYI, for users who hit the "Stream is corrupted" error, please try to apply

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17399465#comment-17399465 ] dragonlong commented on SPARK-18105: [~cameron.todd] Hi, is any news? I issue this problem both in

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397898#comment-17397898 ] Cameron Todd commented on SPARK-18105: -- Oh sorry, I meant just a portion of the code can be

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397747#comment-17397747 ] Dongjoon Hyun commented on SPARK-18105: --- I did only the above code I posted because you wrote like

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397603#comment-17397603 ] Cameron Todd commented on SPARK-18105: -- Good to hear. Also the count of 136,935,074 is right. 

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397585#comment-17397585 ] Dongjoon Hyun commented on SPARK-18105: --- BTW, I checked that a single zip file contains 157 snappy

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397584#comment-17397584 ] Dongjoon Hyun commented on SPARK-18105: --- Thank you, [~cameron.todd]. I downloaded and ran the test

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17397301#comment-17397301 ] Cameron Todd commented on SPARK-18105: -- Ok I added the zip file on this public S3 bucket, it holds

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17396823#comment-17396823 ] Dongjoon Hyun commented on SPARK-18105: --- Ya, this is a nice simplification really. Thanks. {code}

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395954#comment-17395954 ] Cameron Todd commented on SPARK-18105: -- [^hashed_data.zip] > LZ4 failed to decompress a stream of

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395953#comment-17395953 ] Cameron Todd commented on SPARK-18105: -- Yep I understand. I have attached my hashed data keeping

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17395278#comment-17395278 ] Dongjoon Hyun commented on SPARK-18105: --- [~cameron.todd]. Thank you. The code itself looks nice.

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17392971#comment-17392971 ] Cameron Todd commented on SPARK-18105: -- Let me know if that's enough info. From my tests if I

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17392968#comment-17392968 ] Cameron Todd commented on SPARK-18105: -- I'll attach a portion of the code that is not proprietary

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17392536#comment-17392536 ] Dongjoon Hyun commented on SPARK-18105: --- It's a good news because you make it consistently. Could

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17392205#comment-17392205 ] Cameron Todd commented on SPARK-18105: -- I'm also facing this same error when scaling up my project

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17390108#comment-17390108 ] Dongjoon Hyun commented on SPARK-18105: --- I agree with [~viirya] that this might be not a LZ4 codec

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17389144#comment-17389144 ] L. C. Hsieh commented on SPARK-18105: - Looked at lz4 codebase and the reported failures. I suspect

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388797#comment-17388797 ] Arghya Saha commented on SPARK-18105: - [~dongjoon] Can we please address this before next release,

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17388794#comment-17388794 ] Arghya Saha commented on SPARK-18105: - I am also facing the same error, I have raised a duplicate

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17333000#comment-17333000 ] Anthony commented on SPARK-18105: - We are seeing similar issues with Spark 3.0.1 as well. Not exactly

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307557#comment-17307557 ] Dongjoon Hyun commented on SPARK-18105: --- Got it. Thank you for the details, [~devaraj]. Let's keep

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17307453#comment-17307453 ] Devaraj Kavali commented on SPARK-18105: [~dongjoon] We are seeing this error while running

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306726#comment-17306726 ] Dongjoon Hyun commented on SPARK-18105: --- [~devaraj]. Do you have a reproducer? BTW, there is a

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17305327#comment-17305327 ] Devaraj Kavali commented on SPARK-18105: We are still seeing the issue with Spark 3.0.1 as well,

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246725#comment-17246725 ] Dongjoon Hyun commented on SPARK-18105: --- Apache Spark 3.x is using lz4-java-1.7.1.jar and this

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17246577#comment-17246577 ] Wenchen Fan commented on SPARK-18105: - Is this still an issue in Spark 3.x? cc [~dongjoon] > LZ4

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16993834#comment-16993834 ] Mala Chikka Kempanna commented on SPARK-18105: -- If you are facing this in spark 2.4.0 ,

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981461#comment-16981461 ] Maksym F commented on SPARK-18105: -- I was able to reproduce with the code as shown below: {code:java}

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16981442#comment-16981442 ] Ivan Dyptan commented on SPARK-18105: - You can recreate the error consistently by forcing disk spill

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866973#comment-16866973 ] M. Le Bihan commented on SPARK-18105: - My trick eventually didn't succeed. And I fall back into the

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861803#comment-16861803 ] M. Le Bihan commented on SPARK-18105: - It _seems_ that exchanging from org.lz4:lz4-java:1.4.0 to

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16856558#comment-16856558 ] Piotr Chowaniec commented on SPARK-18105: - I have a similar issue with Spark 2.3.2. Here is a

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854249#comment-16854249 ] M. Le Bihan commented on SPARK-18105: - I have also a problem involving a corrupted stream by LZ4,

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784164#comment-16784164 ] Lewin Ma commented on SPARK-18105: -- Still hit the same issue in Spark 2.3.1：   {code:java}  

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16217333#comment-16217333 ] Ashwin Shankar commented on SPARK-18105: Hi [~davies] [~cloud_fan] We hit the same issue. What is

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023980#comment-16023980 ] Rupesh Mane commented on SPARK-18105: - For the stack provided earlier, I found the root cause: Issue

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16023072#comment-16023072 ] Wenchen Fan commented on SPARK-18105: - can you try to set {{{spark.file.transferTo}}} to false and

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019413#comment-16019413 ] yue long commented on SPARK-18105: -- I met the same issue in spark 1.5.2, so could somebody help to fix

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998900#comment-15998900 ] Rupesh Mane commented on SPARK-18105: - I'm facing this issue with Spark 2.1.0 but not with Spark

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15948864#comment-15948864 ] Xiaochen Ouyang commented on SPARK-18105: - [~Tagar] Hi, I met this issue occasionally in

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15875115#comment-15875115 ] Jason Moore commented on SPARK-18105: - I've hit the same using a very recent build from branch-2.1

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15612686#comment-15612686 ] Davies Liu commented on SPARK-18105: It turned out that the bug in LZ4 is a false alarm, so close the

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15606849#comment-15606849 ] Apache Spark commented on SPARK-18105: -- User 'davies' has created a pull request for this issue: