[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138547#comment-14138547 ] Saisai Shao commented on SPARK-2926: Looking forward to your feedback :). > Add MR-st

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14138706#comment-14138706 ] Saisai Shao commented on SPARK-3129: Strongly agree with Matei's comment, I think we c

[jira] [Commented] (SPARK-3615) Kafka test should not hard code Zookeeper port

[ https://issues.apache.org/jira/browse/SPARK-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14142855#comment-14142855 ] Saisai Shao commented on SPARK-3615: Hi Patrick, I've submit a PR to fix this issue, m

[jira] [Commented] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

[ https://issues.apache.org/jira/browse/SPARK-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14144258#comment-14144258 ] Saisai Shao commented on SPARK-3032: Hi Matei, thanks for your reply, I will try again

[jira] [Commented] (SPARK-3876) Doing a RDD map/reduce within a DStream map fails with a high enough input rate

[ https://issues.apache.org/jira/browse/SPARK-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166832#comment-14166832 ] Saisai Shao commented on SPARK-3876: Hi [~afilip], is there any specific purpose you n

[jira] [Commented] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170386#comment-14170386 ] Saisai Shao commented on SPARK-3426: Hi [~andrewor14], are you going to fix this issue

[jira] [Commented] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170395#comment-14170395 ] Saisai Shao commented on SPARK-3426: Sorry about that, I just saw the PR (https://gith

[jira] [Created] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

Saisai Shao created SPARK-3948: -- Summary: Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception Key: SPARK-3948 URL: https://issues.apache.org/jira/browse/SPARK-3948

[jira] [Commented] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14170990#comment-14170990 ] Saisai Shao commented on SPARK-3948: I think the issue of SPARK-3630 is related to thi

[jira] [Commented] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171834#comment-14171834 ] Saisai Shao commented on SPARK-3948: Hi Josh, according to my observation, this bug is

[jira] [Updated] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-3948: --- Description: Several exceptions occurred when running TPC-DS queries against latest master branch wit

[jira] [Updated] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-3948: --- Description: Several exceptions occurred when running TPC-DS queries against latest master branch wit

[jira] [Commented] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171861#comment-14171861 ] Saisai Shao commented on SPARK-3948: Thanks for your help :). > Potential file append

[jira] [Comment Edited] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171885#comment-14171885 ] Saisai Shao edited comment on SPARK-3630 at 10/15/14 2:07 AM: --

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14171885#comment-14171885 ] Saisai Shao commented on SPARK-3630: Hi Pactrick, this problem is still existed after

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172055#comment-14172055 ] Saisai Shao commented on SPARK-3948: Hi Josh, thanks for your help. I don't think it's

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172069#comment-14172069 ] Saisai Shao commented on SPARK-3948: Yes, that's correct. Also please see the link (h

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172117#comment-14172117 ] Saisai Shao commented on SPARK-3948: Hi Josh, I think for old code without append, th

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172152#comment-14172152 ] Saisai Shao commented on SPARK-3948: Hi [~mridul], what I observed is that, after writ

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172371#comment-14172371 ] Saisai Shao commented on SPARK-3948: Hi [~mridulm80], thanks a lot for your suggestion

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14172389#comment-14172389 ] Saisai Shao commented on SPARK-3948: So probably a kernel bug in 2.6.32, as you can se

[jira] [Commented] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173199#comment-14173199 ] Saisai Shao commented on SPARK-3958: Hi Josh, have you tried other compression like LZ

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173353#comment-14173353 ] Saisai Shao commented on SPARK-3948: Thanks for your suggestions. I think it would be

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173427#comment-14173427 ] Saisai Shao commented on SPARK-3948: Hi [~mridulm80], thanks a lot for your suggestion

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173434#comment-14173434 ] Saisai Shao commented on SPARK-3948: Besides I think we can add configuration as [~jos

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14173528#comment-14173528 ] Saisai Shao commented on SPARK-3948: Hi [~mridulm80], Thanks a lot for your suggestio

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176540#comment-14176540 ] Saisai Shao commented on SPARK-4002: Hi Ryan, would you mind describing your issue mor

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14176747#comment-14176747 ] Saisai Shao commented on SPARK-3633: >From my test, I think this problem might be exis

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14177984#comment-14177984 ] Saisai Shao commented on SPARK-4002: Hi Ryan, thanks a lot for your investigation. I s

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14179445#comment-14179445 ] Saisai Shao commented on SPARK-4002: Hi Ryan, I've tested using Maven with your hadoop

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14181006#comment-14181006 ] Saisai Shao commented on SPARK-4002: Thanks a lot Ryan for your detailed description,

[jira] [Created] (SPARK-4062) Improve KafkaReceiver to prevent data loss

Saisai Shao created SPARK-4062: -- Summary: Improve KafkaReceiver to prevent data loss Key: SPARK-4062 URL: https://issues.apache.org/jira/browse/SPARK-4062 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-4062) Improve KafkaReceiver to prevent data loss

[ https://issues.apache.org/jira/browse/SPARK-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-4062: --- Attachment: RefactoredKafkaReceiver.pdf > Improve KafkaReceiver to prevent data loss > ---

[jira] [Commented] (SPARK-8297) Scheduler backend is not notified in case node fails in YARN

[ https://issues.apache.org/jira/browse/SPARK-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581383#comment-14581383 ] Saisai Shao commented on SPARK-8297: Hi [~mridulm80], I tried with latest master branc

[jira] [Commented] (SPARK-8297) Scheduler backend is not notified in case node fails in YARN

[ https://issues.apache.org/jira/browse/SPARK-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14582796#comment-14582796 ] Saisai Shao commented on SPARK-8297: OK, thanks [~mridulm80], I will take a try, from

[jira] [Commented] (SPARK-8297) Scheduler backend is not notified in case node fails in YARN

[ https://issues.apache.org/jira/browse/SPARK-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14583115#comment-14583115 ] Saisai Shao commented on SPARK-8297: Hi [~mridulm80], I'm trying to simulate your scen

[jira] [Commented] (SPARK-8384) Can not set checkpointDuration or Interval in spark 1.3 and later

[ https://issues.apache.org/jira/browse/SPARK-8384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14587515#comment-14587515 ] Saisai Shao commented on SPARK-8384: Hi [~nhe150], I'm not sure why you need to set ch

[jira] [Commented] (SPARK-8389) Expose KafkaRDDs offsetRange in Java and Python

[ https://issues.apache.org/jira/browse/SPARK-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591152#comment-14591152 ] Saisai Shao commented on SPARK-8389: I think for python API, it is not easy to add equ

[jira] [Created] (SPARK-8424) Add blacklist mechanism for task scheduler and Yarn container allocation

Saisai Shao created SPARK-8424: -- Summary: Add blacklist mechanism for task scheduler and Yarn container allocation Key: SPARK-8424 URL: https://issues.apache.org/jira/browse/SPARK-8424 Project: Spark

[jira] [Created] (SPARK-8425) Add blacklist mechanism for task scheduling

Saisai Shao created SPARK-8425: -- Summary: Add blacklist mechanism for task scheduling Key: SPARK-8425 URL: https://issues.apache.org/jira/browse/SPARK-8425 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-8426) Add blacklist mechanism for YARN container allocation

Saisai Shao created SPARK-8426: -- Summary: Add blacklist mechanism for YARN container allocation Key: SPARK-8426 URL: https://issues.apache.org/jira/browse/SPARK-8426 Project: Spark Issue Type: S

[jira] [Commented] (SPARK-8424) Add blacklist mechanism for task scheduler and Yarn container allocation

[ https://issues.apache.org/jira/browse/SPARK-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591173#comment-14591173 ] Saisai Shao commented on SPARK-8424: Here is the design doc in Google doc, any comment

[jira] [Commented] (SPARK-8389) Expose KafkaRDDs offsetRange in Java and Python

[ https://issues.apache.org/jira/browse/SPARK-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591211#comment-14591211 ] Saisai Shao commented on SPARK-8389: I'm not saying its impossible to do it, from my u

[jira] [Commented] (SPARK-8383) Spark History Server shows Last Updated as 1969/12/31 when SparkPI application completed

[ https://issues.apache.org/jira/browse/SPARK-8383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591478#comment-14591478 ] Saisai Shao commented on SPARK-8383: Hi [~irina.easterl...@teradata.com], I tried to r

[jira] [Comment Edited] (SPARK-8383) Spark History Server shows Last Updated as 1969/12/31 when SparkPI application completed

[ https://issues.apache.org/jira/browse/SPARK-8383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14591478#comment-14591478 ] Saisai Shao edited comment on SPARK-8383 at 6/18/15 8:43 AM: -

[jira] [Commented] (SPARK-8389) Expose KafkaRDDs offsetRange in Java and Python

[ https://issues.apache.org/jira/browse/SPARK-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14592879#comment-14592879 ] Saisai Shao commented on SPARK-8389: OK, let me think a bit on this, but this can only

[jira] [Commented] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598681#comment-14598681 ] Saisai Shao commented on SPARK-8337: Hi [~juanrh], will you also address {{OffsetRange

[jira] [Issue Comment Deleted] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-8337: --- Comment: was deleted (was: OK, well, I'd like to take a crack at it :).) > KafkaUtils.createDirectStr

[jira] [Commented] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600468#comment-14600468 ] Saisai Shao commented on SPARK-8337: OK, well, I'd like to take a crack at it :). > K

[jira] [Commented] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600469#comment-14600469 ] Saisai Shao commented on SPARK-8337: OK, well, I'd like to take a crack at it :). > K

[jira] [Commented] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600470#comment-14600470 ] Saisai Shao commented on SPARK-8337: OK, well, I'd like to take a crack at it :). > K

[jira] [Issue Comment Deleted] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-8337: --- Comment: was deleted (was: OK, well, I'd like to take a crack at it :).) > KafkaUtils.createDirectStr

[jira] [Commented] (SPARK-8424) Add blacklist mechanism for task scheduler and Yarn container allocation

[ https://issues.apache.org/jira/browse/SPARK-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605001#comment-14605001 ] Saisai Shao commented on SPARK-8424: OK, I will close this JIRA. > Add blacklist mech

[jira] [Closed] (SPARK-8424) Add blacklist mechanism for task scheduler and Yarn container allocation

[ https://issues.apache.org/jira/browse/SPARK-8424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-8424. -- Resolution: Duplicate > Add blacklist mechanism for task scheduler and Yarn container allocation > -

[jira] [Commented] (SPARK-8337) KafkaUtils.createDirectStream for python is lacking API/feature parity with the Scala/Java version

[ https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14605241#comment-14605241 ] Saisai Shao commented on SPARK-8337: Hi [~juanrh], I think the best choice is to keep

[jira] [Commented] (SPARK-8827) pyspark.DStream top method

[ https://issues.apache.org/jira/browse/SPARK-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614562#comment-14614562 ] Saisai Shao commented on SPARK-8827: I think you could get top values with follow patt

[jira] [Comment Edited] (SPARK-8827) pyspark.DStream top method

[ https://issues.apache.org/jira/browse/SPARK-8827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614562#comment-14614562 ] Saisai Shao edited comment on SPARK-8827 at 7/6/15 5:22 AM: I

[jira] [Updated] (SPARK-7050) Fix Python Kafka test assembly jar not found issue under Maven build

[ https://issues.apache.org/jira/browse/SPARK-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-7050: --- Summary: Fix Python Kafka test assembly jar not found issue under Maven build (was: Kafka-assembly sh

[jira] [Commented] (SPARK-7050) Fix Python Kafka test assembly jar not found issue under Maven build

[ https://issues.apache.org/jira/browse/SPARK-7050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14614698#comment-14614698 ] Saisai Shao commented on SPARK-7050: Thanks [~srowen], how about this title? > Fix Py

[jira] [Commented] (SPARK-8882) A New Receiver Scheduling Mechanism

[ https://issues.apache.org/jira/browse/SPARK-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621540#comment-14621540 ] Saisai Shao commented on SPARK-8882: Seems the document requires authorization to revi

[jira] [Commented] (SPARK-8882) A New Receiver Scheduling Mechanism

[ https://issues.apache.org/jira/browse/SPARK-8882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14621552#comment-14621552 ] Saisai Shao commented on SPARK-8882: Cool, works now. > A New Receiver Scheduling Mec

[jira] [Created] (SPARK-11457) Yarn AM proxy filter configuration should be reloaded when recovered from checkpoint

Saisai Shao created SPARK-11457: --- Summary: Yarn AM proxy filter configuration should be reloaded when recovered from checkpoint Key: SPARK-11457 URL: https://issues.apache.org/jira/browse/SPARK-11457 Pr

[jira] [Commented] (SPARK-11472) SparkContext creation error after sc.stop() when Spark is compiled for Hive

[ https://issues.apache.org/jira/browse/SPARK-11472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14987314#comment-14987314 ] Saisai Shao commented on SPARK-11472: - Yes, agree with [~srowen], there might be a lo

[jira] [Commented] (SPARK-5934) DStreamGraph.clearMetadata attempts to unpersist the same RDD multiple times

[ https://issues.apache.org/jira/browse/SPARK-5934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14991353#comment-14991353 ] Saisai Shao commented on SPARK-5934: I think here in your code: {code} val output = i

[jira] [Created] (SPARK-11632) Filter out empty partition for KafkaRDD

Saisai Shao created SPARK-11632: --- Summary: Filter out empty partition for KafkaRDD Key: SPARK-11632 URL: https://issues.apache.org/jira/browse/SPARK-11632 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-11718) Explicit killing executor dies silent without get response information

Saisai Shao created SPARK-11718: --- Summary: Explicit killing executor dies silent without get response information Key: SPARK-11718 URL: https://issues.apache.org/jira/browse/SPARK-11718 Project: Spark

[jira] [Commented] (SPARK-2960) Spark executables fail to start via symlinks

[ https://issues.apache.org/jira/browse/SPARK-2960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003598#comment-15003598 ] Saisai Shao commented on SPARK-2960: Hi [~yinxusen], from script's view point, it cann

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003601#comment-15003601 ] Saisai Shao commented on SPARK-11701: - Hi [~tgraves], I think I also hit this issue,

[jira] [Commented] (SPARK-11701) YARN - dynamic allocation and speculation active task accounting wrong

[ https://issues.apache.org/jira/browse/SPARK-11701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15003996#comment-15003996 ] Saisai Shao commented on SPARK-11701: - Hi [~tgraves] I think I'm referring to your la

[jira] [Commented] (SPARK-11744) bin/pyspark --version doesn't return version and exit

[ https://issues.apache.org/jira/browse/SPARK-11744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15005862#comment-15005862 ] Saisai Shao commented on SPARK-11744: - Yes, it is actually a small bug in Launcher, l

[jira] [Commented] (SPARK-11909) Spark Standalone's master URL accepts URLs without port (assuming default 7077)

[ https://issues.apache.org/jira/browse/SPARK-11909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021669#comment-15021669 ] Saisai Shao commented on SPARK-11909: - The master will print the master url in web UI

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028325#comment-15028325 ] Saisai Shao commented on SPARK-12009: - A interesting thing is that AM is shutting dow

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028325#comment-15028325 ] Saisai Shao edited comment on SPARK-12009 at 11/26/15 7:58 AM:

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028340#comment-15028340 ] Saisai Shao commented on SPARK-12009: - Looking at the code again, {{onDisconnected}}

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028370#comment-15028370 ] Saisai Shao commented on SPARK-12009: - If I understood correctly, this log should com

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028375#comment-15028375 ] Saisai Shao commented on SPARK-12009: - Looks like you're running on yarn-cluster mode

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029416#comment-15029416 ] Saisai Shao commented on SPARK-12009: - So what actually version of Spark you're runni

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029416#comment-15029416 ] Saisai Shao edited comment on SPARK-12009 at 11/27/15 3:30 AM:

[jira] [Commented] (SPARK-12002) offsetRanges attribute missing in Kafka RDD when resuming from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-12002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029421#comment-15029421 ] Saisai Shao commented on SPARK-12002: - Looks like because Python `KafkaTransformDStre

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029425#comment-15029425 ] Saisai Shao commented on SPARK-12009: - So I guess your problem is that after you call

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029431#comment-15029431 ] Saisai Shao commented on SPARK-12009: - Alright, my code is master branch. Anyway I un

[jira] [Comment Edited] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029431#comment-15029431 ] Saisai Shao edited comment on SPARK-12009 at 11/27/15 3:50 AM:

[jira] [Commented] (SPARK-12009) Avoid re-allocate yarn container while driver want to stop all Executors

[ https://issues.apache.org/jira/browse/SPARK-12009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15031308#comment-15031308 ] Saisai Shao commented on SPARK-12009: - OK, I see. So how about setting {{YarnAllocato

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15032806#comment-15032806 ] Saisai Shao commented on SPARK-12059: - Thanks a lot [~andrewor14], I will look into t

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15033268#comment-15033268 ] Saisai Shao commented on SPARK-12059: - Hi [~andrewor14], when will this be happened?

[jira] [Comment Edited] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034997#comment-15034997 ] Saisai Shao edited comment on SPARK-12059 at 12/2/15 12:47 AM:

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15034997#comment-15034997 ] Saisai Shao commented on SPARK-12059: - A simple solution is to loose the condition or

[jira] [Commented] (SPARK-12059) Standalone Master assertion error

[ https://issues.apache.org/jira/browse/SPARK-12059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15035350#comment-15035350 ] Saisai Shao commented on SPARK-12059: - I see, so I will relax the condition to avoid

[jira] [Commented] (SPARK-10123) Cannot set "--deploy-mode" in default configuration

[ https://issues.apache.org/jira/browse/SPARK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041260#comment-15041260 ] Saisai Shao commented on SPARK-10123: - Hi [~vanzin], would you mind letting me take a

[jira] [Commented] (SPARK-12103) KafkaUtils createStream with multiple topics -- does not work as expected

[ https://issues.apache.org/jira/browse/SPARK-12103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15041301#comment-15041301 ] Saisai Shao commented on SPARK-12103: - I think I had a proposal of message handler (r

[jira] [Commented] (SPARK-10123) Cannot set "--deploy-mode" in default configuration

[ https://issues.apache.org/jira/browse/SPARK-10123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15042598#comment-15042598 ] Saisai Shao commented on SPARK-10123: - Just confirm if it is on your plan, in case du

[jira] [Commented] (SPARK-12178) Expose reporting of StreamInputInfo for custom made streams

[ https://issues.apache.org/jira/browse/SPARK-12178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15046602#comment-15046602 ] Saisai Shao commented on SPARK-12178: - This is a good idea to make it generic if ther

[jira] [Commented] (SPARK-9059) Update Python Direct Kafka Word count examples to show the use of HasOffsetRanges

[ https://issues.apache.org/jira/browse/SPARK-9059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049804#comment-15049804 ] Saisai Shao commented on SPARK-9059: HasOffsetRanges also has python version, which is

[jira] [Commented] (SPARK-6735) Provide options to make maximum executor failure count ( which kills the application ) relative to a window duration or disable it.

[ https://issues.apache.org/jira/browse/SPARK-6735?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15049958#comment-15049958 ] Saisai Shao commented on SPARK-6735: I've submitted a patch to continue this work (ht

[jira] [Commented] (SPARK-12176) SparkLauncher's setConf() does not support configs containing spaces

[ https://issues.apache.org/jira/browse/SPARK-12176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15055508#comment-15055508 ] Saisai Shao commented on SPARK-12176: - It is OK in my local test against latest maste

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059569#comment-15059569 ] Saisai Shao commented on SPARK-12345: - It is OK in my local test when I followed the

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059778#comment-15059778 ] Saisai Shao commented on SPARK-12345: - A simple solution is to change the scripts to

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059937#comment-15059937 ] Saisai Shao commented on SPARK-12345: - I think by default Spark Mesos implementation

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15059954#comment-15059954 ] Saisai Shao commented on SPARK-12345: - Having a quick test by not exporting {{SPARK_H

[jira] [Commented] (SPARK-12345) Mesos cluster mode is broken

[ https://issues.apache.org/jira/browse/SPARK-12345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15060035#comment-15060035 ] Saisai Shao commented on SPARK-12345: - Here is the one solution (https://github.com/

<    1   2   3   4   5   6   7   8   9   10   >