[jira] [Created] (SPARK-1510) Add Spark Streaming metrics source for metrics system

Saisai Shao created SPARK-1510: -- Summary: Add Spark Streaming metrics source for metrics system Key: SPARK-1510 URL: https://issues.apache.org/jira/browse/SPARK-1510 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2044) Pluggable interface for shuffles

[ https://issues.apache.org/jira/browse/SPARK-2044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14019511#comment-14019511 ] Saisai Shao commented on SPARK-2044: Hi Matei, it's great to see you guys have plan on

[jira] [Created] (SPARK-2122) Move aggregation into shuffle implementation

Saisai Shao created SPARK-2122: -- Summary: Move aggregation into shuffle implementation Key: SPARK-2122 URL: https://issues.apache.org/jira/browse/SPARK-2122 Project: Spark Issue Type:

[jira] [Commented] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028833#comment-14028833 ] Saisai Shao commented on SPARK-2125: Hi Matei, for moving sort into hash shuffle

[jira] [Commented] (SPARK-2124) Move aggregation into ShuffleManager implementations

[ https://issues.apache.org/jira/browse/SPARK-2124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14028922#comment-14028922 ] Saisai Shao commented on SPARK-2124: PR submitted

[jira] [Commented] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041809#comment-14041809 ] Saisai Shao commented on SPARK-2125: Hi Matei, I have some basic implementations about

[jira] [Comment Edited] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14041809#comment-14041809 ] Saisai Shao edited comment on SPARK-2125 at 6/24/14 7:26 AM: -

[jira] [Commented] (SPARK-2125) Add sorting flag to ShuffleManager, and implement it in HashShuffleManager

[ https://issues.apache.org/jira/browse/SPARK-2125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14044285#comment-14044285 ] Saisai Shao commented on SPARK-2125: PR submitted:

[jira] [Commented] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045430#comment-14045430 ] Saisai Shao commented on SPARK-2104: Ok, got it. I will try to fix this issue :)

[jira] [Commented] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045519#comment-14045519 ] Saisai Shao commented on SPARK-2104: Hi Reynold, thanks a lot for your code. At first

[jira] [Commented] (SPARK-2104) RangePartitioner should use user specified serializer to serialize range bounds

[ https://issues.apache.org/jira/browse/SPARK-2104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14045524#comment-14045524 ] Saisai Shao commented on SPARK-2104: OK, got it. Thanks a lot RangePartitioner

[jira] [Created] (SPARK-2402) DiskBlockObjectWriter should update the initial position when reusing this object

Saisai Shao created SPARK-2402: -- Summary: DiskBlockObjectWriter should update the initial position when reusing this object Key: SPARK-2402 URL: https://issues.apache.org/jira/browse/SPARK-2402 Project:

[jira] [Commented] (SPARK-2402) DiskBlockObjectWriter should update the initial position when reusing this object

[ https://issues.apache.org/jira/browse/SPARK-2402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14054686#comment-14054686 ] Saisai Shao commented on SPARK-2402: PR submitted,

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14062053#comment-14062053 ] Saisai Shao commented on SPARK-2492: PR submitted:

[jira] [Commented] (SPARK-2045) Sort-based shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063006#comment-14063006 ] Saisai Shao commented on SPARK-2045: Hi Matei, great to see your design doc, a simple

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14063063#comment-14063063 ] Saisai Shao commented on SPARK-2492: Hi TD, The parameter auto.offset.reset is

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064636#comment-14064636 ] Saisai Shao commented on SPARK-2492: Hi TD, I revisit the Kafka's ConsoleConsumer

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064721#comment-14064721 ] Saisai Shao commented on SPARK-2492: Hi TD, Also I did some experiments on the

[jira] [Commented] (SPARK-2383) With auto.offset.reset, KafkaReceiver potentially deletes Consumer nodes from Zookeeper

[ https://issues.apache.org/jira/browse/SPARK-2383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069692#comment-14069692 ] Saisai Shao commented on SPARK-2383: Hi Tobias, I've also noticed this problem,

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14069727#comment-14069727 ] Saisai Shao commented on SPARK-2492: Hi Tobias, I agree with you. Though I do not

[jira] [Commented] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072749#comment-14072749 ] Saisai Shao commented on SPARK-2492: Hi Tobias, I've updated the code, mind taking a

[jira] [Comment Edited] (SPARK-2492) KafkaReceiver minor changes to align with Kafka 0.8

[ https://issues.apache.org/jira/browse/SPARK-2492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14072749#comment-14072749 ] Saisai Shao edited comment on SPARK-2492 at 7/24/14 3:27 AM: -

[jira] [Commented] (SPARK-2780) Create a StreamingContext.setLocalProperty for setting local property of jobs launched by streaming

[ https://issues.apache.org/jira/browse/SPARK-2780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081947#comment-14081947 ] Saisai Shao commented on SPARK-2780: Hi TD, I think the fair scheduler setting can be

[jira] [Created] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

Saisai Shao created SPARK-2926: -- Summary: Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle Key: SPARK-2926 URL: https://issues.apache.org/jira/browse/SPARK-2926 Project: Spark

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Attachment: SortBasedShuffleRead.pdf A rough design doc is uploaded. Any comments would be greatly

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Description: Currently Spark has already integrated sort-based shuffle write, which greatly improve

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091684#comment-14091684 ] Saisai Shao commented on SPARK-2926: Hi Sandy, Thanks a lot for your comments, basic

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091753#comment-14091753 ] Saisai Shao commented on SPARK-2926: Hi Matei, thanks a lot for your comments. The

[jira] [Comment Edited] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14091753#comment-14091753 ] Saisai Shao edited comment on SPARK-2926 at 8/9/14 2:09 PM: Hi

[jira] [Created] (SPARK-2967) Several SQL unit test failed when sort-based shuffle is enabled

Saisai Shao created SPARK-2967: -- Summary: Several SQL unit test failed when sort-based shuffle is enabled Key: SPARK-2967 URL: https://issues.apache.org/jira/browse/SPARK-2967 Project: Spark

[jira] [Commented] (SPARK-2978) Provide an MR-style shuffle transformation

[ https://issues.apache.org/jira/browse/SPARK-2978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093866#comment-14093866 ] Saisai Shao commented on SPARK-2978: Hi Sandy, A simple question: do you mean to add

[jira] [Commented] (SPARK-2967) Several SQL unit test failed when sort-based shuffle is enabled

[ https://issues.apache.org/jira/browse/SPARK-2967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14093874#comment-14093874 ] Saisai Shao commented on SPARK-2967: Hi Matei and Michael, thanks a lot for looking

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Attachment: Spark Shuffle Test Report.pdf Add MR-style (merge-sort) SortShuffleReader for

[jira] [Comment Edited] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14096608#comment-14096608 ] Saisai Shao edited comment on SPARK-2926 at 8/14/14 7:12 AM: -

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14096688#comment-14096688 ] Saisai Shao commented on SPARK-2926: I think this prototype can easily offer the

[jira] [Created] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

Saisai Shao created SPARK-3032: -- Summary: Potential bug when running sort-based shuffle with sorting using TimSort Key: SPARK-3032 URL: https://issues.apache.org/jira/browse/SPARK-3032 Project: Spark

[jira] [Created] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

Saisai Shao created SPARK-3146: -- Summary: Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM Key: SPARK-3146 URL:

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14103486#comment-14103486 ] Saisai Shao commented on SPARK-3146: This issue can actually solve the problem

[jira] [Updated] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-3146: --- Description: Currently Spark Streaming Kafka API stores the key and value of each message into BM

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14105086#comment-14105086 ] Saisai Shao commented on SPARK-3129: Hi Hari, I have some high level questions about

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14119128#comment-14119128 ] Saisai Shao commented on SPARK-3146: Hi [~tdas], Sorry for late response, thanks a

[jira] [Commented] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120834#comment-14120834 ] Saisai Shao commented on SPARK-3292: Hi [~guowei], did you test the scenario with

[jira] [Commented] (SPARK-3292) Shuffle Tasks run incessantly even though there's no inputs

[ https://issues.apache.org/jira/browse/SPARK-3292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120839#comment-14120839 ] Saisai Shao commented on SPARK-3292: I think these two tickets address the same

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121160#comment-14121160 ] Saisai Shao commented on SPARK-3129: Hi [~hshreedharan], one more question: Is your

[jira] [Commented] (SPARK-3129) Prevent data loss in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122274#comment-14122274 ] Saisai Shao commented on SPARK-3129: Hi [~hshreedharan]], thanks for your reply, is

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122292#comment-14122292 ] Saisai Shao commented on SPARK-2926: Hi Matei, sorry for late response, I will test

[jira] [Commented] (SPARK-2122) Move aggregation into shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14126471#comment-14126471 ] Saisai Shao commented on SPARK-2122: Yes, this is a duplicated ticket, it is fixed in

[jira] [Closed] (SPARK-2122) Move aggregation into shuffle implementation

[ https://issues.apache.org/jira/browse/SPARK-2122?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao closed SPARK-2122. -- Resolution: Duplicate Move aggregation into shuffle implementation

[jira] [Updated] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-2926: --- Attachment: Spark Shuffle Test Report(contd).pdf Add MR-style (merge-sort) SortShuffleReader for

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14131314#comment-14131314 ] Saisai Shao commented on SPARK-2926: Hi Reynold, thanks a lot for your watching this,

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132598#comment-14132598 ] Saisai Shao commented on SPARK-2926: Ok, I will take a try and let you know then it is

[jira] [Comment Edited] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14132598#comment-14132598 ] Saisai Shao edited comment on SPARK-2926 at 9/13/14 8:09 AM: -

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14133694#comment-14133694 ] Saisai Shao commented on SPARK-2926: Hey [~rxin], here is the branch rebased on your

[jira] [Commented] (SPARK-3563) Shuffle data not always be cleaned

[ https://issues.apache.org/jira/browse/SPARK-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14137246#comment-14137246 ] Saisai Shao commented on SPARK-3563: In my thought, I think it relies on JVM's GC

[jira] [Commented] (SPARK-3563) Shuffle data not always be cleaned

[ https://issues.apache.org/jira/browse/SPARK-3563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138389#comment-14138389 ] Saisai Shao commented on SPARK-3563: I think it relies on JVM's GC strategy to treat

[jira] [Commented] (SPARK-2926) Add MR-style (merge-sort) SortShuffleReader for sort-based shuffle

[ https://issues.apache.org/jira/browse/SPARK-2926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14138547#comment-14138547 ] Saisai Shao commented on SPARK-2926: Looking forward to your feedback :). Add

[jira] [Commented] (SPARK-3615) Kafka test should not hard code Zookeeper port

[ https://issues.apache.org/jira/browse/SPARK-3615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14142855#comment-14142855 ] Saisai Shao commented on SPARK-3615: Hi Patrick, I've submit a PR to fix this issue,

[jira] [Commented] (SPARK-3032) Potential bug when running sort-based shuffle with sorting using TimSort

[ https://issues.apache.org/jira/browse/SPARK-3032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14144258#comment-14144258 ] Saisai Shao commented on SPARK-3032: Hi Matei, thanks for your reply, I will try again

[jira] [Commented] (SPARK-3876) Doing a RDD map/reduce within a DStream map fails with a high enough input rate

[ https://issues.apache.org/jira/browse/SPARK-3876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14166832#comment-14166832 ] Saisai Shao commented on SPARK-3876: Hi [~afilip], is there any specific purpose you

[jira] [Commented] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170386#comment-14170386 ] Saisai Shao commented on SPARK-3426: Hi [~andrewor14], are you going to fix this

[jira] [Commented] (SPARK-3426) Sort-based shuffle compression behavior is inconsistent

[ https://issues.apache.org/jira/browse/SPARK-3426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14170395#comment-14170395 ] Saisai Shao commented on SPARK-3426: Sorry about that, I just saw the PR

[jira] [Created] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

Saisai Shao created SPARK-3948: -- Summary: Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception Key: SPARK-3948 URL: https://issues.apache.org/jira/browse/SPARK-3948

[jira] [Commented] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171834#comment-14171834 ] Saisai Shao commented on SPARK-3948: Hi Josh, according to my observation, this bug is

[jira] [Updated] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-3948: --- Description: Several exceptions occurred when running TPC-DS queries against latest master branch

[jira] [Updated] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-3948: --- Description: Several exceptions occurred when running TPC-DS queries against latest master branch

[jira] [Commented] (SPARK-3948) Potential file append bugs in ExternalSorter which leads to sort-based shuffle unexpected exception

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171861#comment-14171861 ] Saisai Shao commented on SPARK-3948: Thanks for your help :). Potential file append

[jira] [Comment Edited] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171885#comment-14171885 ] Saisai Shao edited comment on SPARK-3630 at 10/15/14 2:07 AM: --

[jira] [Commented] (SPARK-3630) Identify cause of Kryo+Snappy PARSING_ERROR

[ https://issues.apache.org/jira/browse/SPARK-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14171885#comment-14171885 ] Saisai Shao commented on SPARK-3630: Hi Pactrick, this problem is still existed after

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172055#comment-14172055 ] Saisai Shao commented on SPARK-3948: Hi Josh, thanks for your help. I don't think it's

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172117#comment-14172117 ] Saisai Shao commented on SPARK-3948: Hi Josh, I think for old code without append,

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172152#comment-14172152 ] Saisai Shao commented on SPARK-3948: Hi [~mridul], what I observed is that, after

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14172371#comment-14172371 ] Saisai Shao commented on SPARK-3948: Hi [~mridulm80], thanks a lot for your

[jira] [Commented] (SPARK-3958) Possible stream-corruption issues in TorrentBroadcast

[ https://issues.apache.org/jira/browse/SPARK-3958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173199#comment-14173199 ] Saisai Shao commented on SPARK-3958: Hi Josh, have you tried other compression like

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173427#comment-14173427 ] Saisai Shao commented on SPARK-3948: Hi [~mridulm80], thanks a lot for your

[jira] [Commented] (SPARK-3948) Sort-based shuffle can lead to assorted stream-corruption exceptions

[ https://issues.apache.org/jira/browse/SPARK-3948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14173528#comment-14173528 ] Saisai Shao commented on SPARK-3948: Hi [~mridulm80], Thanks a lot for your

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176540#comment-14176540 ] Saisai Shao commented on SPARK-4002: Hi Ryan, would you mind describing your issue

[jira] [Commented] (SPARK-3633) Fetches failure observed after SPARK-2711

[ https://issues.apache.org/jira/browse/SPARK-3633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14176747#comment-14176747 ] Saisai Shao commented on SPARK-3633: From my test, I think this problem might be

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14177984#comment-14177984 ] Saisai Shao commented on SPARK-4002: Hi Ryan, thanks a lot for your investigation. I

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14179445#comment-14179445 ] Saisai Shao commented on SPARK-4002: Hi Ryan, I've tested using Maven with your hadoop

[jira] [Commented] (SPARK-4002) JavaKafkaStreamSuite.testKafkaStream fails on OSX

[ https://issues.apache.org/jira/browse/SPARK-4002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14181006#comment-14181006 ] Saisai Shao commented on SPARK-4002: Thanks a lot Ryan for your detailed description,

[jira] [Created] (SPARK-4062) Improve KafkaReceiver to prevent data loss

Saisai Shao created SPARK-4062: -- Summary: Improve KafkaReceiver to prevent data loss Key: SPARK-4062 URL: https://issues.apache.org/jira/browse/SPARK-4062 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-4062) Improve KafkaReceiver to prevent data loss

[ https://issues.apache.org/jira/browse/SPARK-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-4062: --- Attachment: RefactoredKafkaReceiver.pdf Improve KafkaReceiver to prevent data loss

[jira] [Created] (SPARK-4381) User should get warned when set spark.master with local in Spark Streaming

Saisai Shao created SPARK-4381: -- Summary: User should get warned when set spark.master with local in Spark Streaming Key: SPARK-4381 URL: https://issues.apache.org/jira/browse/SPARK-4381 Project: Spark

[jira] [Commented] (SPARK-4537) Add 'processing delay' and 'totalDelay' to the metrics reported by the Spark Streaming subsystem

[ https://issues.apache.org/jira/browse/SPARK-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14222677#comment-14222677 ] Saisai Shao commented on SPARK-4537: Hi [~gmaas], are you going to fix this issue? If

[jira] [Created] (SPARK-4595) Spark MetricsServlet is not enabled because of initialization ordering

Saisai Shao created SPARK-4595: -- Summary: Spark MetricsServlet is not enabled because of initialization ordering Key: SPARK-4595 URL: https://issues.apache.org/jira/browse/SPARK-4595 Project: Spark

[jira] [Updated] (SPARK-4595) Spark MetricsServlet is not worked because of initialization ordering

[ https://issues.apache.org/jira/browse/SPARK-4595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-4595: --- Summary: Spark MetricsServlet is not worked because of initialization ordering (was: Spark

[jira] [Commented] (SPARK-4537) Add 'processing delay' and 'totalDelay' to the metrics reported by the Spark Streaming subsystem

[ https://issues.apache.org/jira/browse/SPARK-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225561#comment-14225561 ] Saisai Shao commented on SPARK-4537: Thanks TD, I'm going to fix this issue. Add

[jira] [Created] (SPARK-4671) Streaming block need not to replicate 2 copies when WAL is enabled

Saisai Shao created SPARK-4671: -- Summary: Streaming block need not to replicate 2 copies when WAL is enabled Key: SPARK-4671 URL: https://issues.apache.org/jira/browse/SPARK-4671 Project: Spark

[jira] [Updated] (SPARK-4740) Netty's network bandwidth is much lower than NIO in spark-perf and Netty takes longer running time

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-4740: --- Affects Version/s: 1.2.0 Netty's network bandwidth is much lower than NIO in spark-perf and Netty

[jira] [Commented] (SPARK-4740) Netty's network bandwidth is much lower than NIO in spark-perf and Netty takes longer running time

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234876#comment-14234876 ] Saisai Shao commented on SPARK-4740: We also tested with small dataset like 40GB, the

[jira] [Commented] (SPARK-4740) Netty's network bandwidth is much lower than NIO in spark-perf and Netty takes longer running time

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234902#comment-14234902 ] Saisai Shao commented on SPARK-4740: Besides we also tested with 24 cores WSM cpu, the

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235036#comment-14235036 ] Saisai Shao commented on SPARK-4740: Hi [~rxin], the difference between NIO and Netty

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14236552#comment-14236552 ] Saisai Shao commented on SPARK-4740: I will test it on my 24 cores and 12 HDDs cluster

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14236777#comment-14236777 ] Saisai Shao commented on SPARK-4740: Hi Reynold, I just tested your patch with

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240627#comment-14240627 ] Saisai Shao commented on SPARK-4740: Hi Aaron, would you mind giving us your system

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14240813#comment-14240813 ] Saisai Shao commented on SPARK-4740: Thanks Aaron, we will try to use ramdisk to

[jira] [Commented] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241973#comment-14241973 ] Saisai Shao commented on SPARK-4740: Hi Reynold, the code I pasted is just the

[jira] [Comment Edited] (SPARK-4740) Netty's network throughput is about 1/2 of NIO's in spark-perf sortByKey

[ https://issues.apache.org/jira/browse/SPARK-4740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241973#comment-14241973 ] Saisai Shao edited comment on SPARK-4740 at 12/11/14 1:34 AM: --

[jira] [Created] (SPARK-4847) extraStrategies cannot take effect in SQLContext

Saisai Shao created SPARK-4847: -- Summary: extraStrategies cannot take effect in SQLContext Key: SPARK-4847 URL: https://issues.apache.org/jira/browse/SPARK-4847 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-3146) Improve the flexibility of Spark Streaming Kafka API to offer user the ability to process message before storing into BM

[ https://issues.apache.org/jira/browse/SPARK-3146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14252742#comment-14252742 ] Saisai Shao commented on SPARK-3146: Hi all, thanks a lot for your comments. My

  1   2   3   4   5   6   7   8   9   10   >