[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001409#comment-16001409 ] Shixiong Zhu commented on SPARK-13747: -- [~revolucion09] I don't know who created ForkJoinPool but

[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001389#comment-16001389 ] Shixiong Zhu commented on SPARK-13747: -- [~revolucion09] If you are not using ForkJoinPool, I'm 100%

[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001365#comment-16001365 ] Shixiong Zhu commented on SPARK-13747: -- [~mousa] This is is because Spark uses ThreadLocal in a

[jira] [Updated] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

[ https://issues.apache.org/jira/browse/SPARK-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20603: - Affects Version/s: 2.1.1 2.1.0 > Flaky test:

[jira] [Resolved] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

[ https://issues.apache.org/jira/browse/SPARK-20603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20603. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.2 > Flaky test:

[jira] [Commented] (SPARK-18971) Netty issue may cause the shuffle client hang

[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998654#comment-15998654 ] Shixiong Zhu commented on SPARK-18971: -- [~tgraves] No, as far as I known. But since Spark 2.2.0 has

[jira] [Comment Edited] (SPARK-18971) Netty issue may cause the shuffle client hang

[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998654#comment-15998654 ] Shixiong Zhu edited comment on SPARK-18971 at 5/5/17 5:49 PM: -- [~tgraves]

[jira] [Updated] (SPARK-19690) Join a streaming DataFrame with a batch DataFrame may not work

[ https://issues.apache.org/jira/browse/SPARK-19690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19690: - Target Version/s: 2.3.0 (was: 2.2.0) > Join a streaming DataFrame with a batch DataFrame may

[jira] [Created] (SPARK-20603) Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0

Shixiong Zhu created SPARK-20603: Summary: Flaky test: o.a.s.sql.kafka010.KafkaSourceSuite deserialization of initial offset with Spark 2.1.0 Key: SPARK-20603 URL:

[jira] [Commented] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997240#comment-15997240 ] Shixiong Zhu commented on SPARK-20599: -- Good point. Yeah, we can just change it to be a longer

[jira] [Updated] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20599: - Affects Version/s: (was: 2.3.0) 2.2.0 > KafkaSourceProvider should

[jira] [Commented] (SPARK-20599) KafkaSourceProvider should work with ConsoleSink

[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997222#comment-15997222 ] Shixiong Zhu commented on SPARK-20599: -- Looks like we just need to provide a better message.

[jira] [Updated] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20600: - Affects Version/s: (was: 2.3.0) 2.2.0 > KafkaRelation should be

[jira] [Updated] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20600: - Component/s: (was: SQL) > KafkaRelation should be pretty printed in web UI (Details for

[jira] [Commented] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997217#comment-15997217 ] Shixiong Zhu commented on SPARK-20600: -- Could you submit a PR to fix its "toString" method? >

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997133#comment-15997133 ] Shixiong Zhu edited comment on SPARK-18057 at 5/4/17 5:50 PM: -- [~helena_e]

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15997133#comment-15997133 ] Shixiong Zhu commented on SPARK-18057: -- [~helena_e] I'm curious why you cannot just update the Kafka

[jira] [Comment Edited] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab

[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995923#comment-15995923 ] Shixiong Zhu edited comment on SPARK-20213 at 5/4/17 12:02 AM: --- I tested

[jira] [Commented] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab

[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995923#comment-15995923 ] Shixiong Zhu commented on SPARK-20213: -- I tested the master branch, and I can see "insertInto" in

[jira] [Updated] (SPARK-20213) DataFrameWriter operations do not show up in SQL tab

[ https://issues.apache.org/jira/browse/SPARK-20213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20213: - Attachment: Screen Shot 2017-05-03 at 5.00.19 PM.png > DataFrameWriter operations do not show up

[jira] [Commented] (SPARK-20568) Delete files after processing in structured streaming

[ https://issues.apache.org/jira/browse/SPARK-20568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995507#comment-15995507 ] Shixiong Zhu commented on SPARK-20568: -- [~srowen] Structured Streaming's Source has a "commit"

[jira] [Resolved] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

[ https://issues.apache.org/jira/browse/SPARK-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19965. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0 > DataFrame batch

[jira] [Assigned] (SPARK-20529) Worker should not use the received Master address

[ https://issues.apache.org/jira/browse/SPARK-20529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-20529: Assignee: Shixiong Zhu > Worker should not use the received Master address >

[jira] [Updated] (SPARK-20529) Worker should not use the received Master address

[ https://issues.apache.org/jira/browse/SPARK-20529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20529: - Affects Version/s: 2.2.0 1.6.3 2.0.2 > Worker

[jira] [Resolved] (SPARK-20531) Spark master shouldn't send its address back to the workers over proxied connections

[ https://issues.apache.org/jira/browse/SPARK-20531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20531. -- Resolution: Duplicate > Spark master shouldn't send its address back to the workers over

[jira] [Updated] (SPARK-20436) NullPointerException when restart from checkpoint file

[ https://issues.apache.org/jira/browse/SPARK-20436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20436: - Issue Type: Question (was: Bug) > NullPointerException when restart from checkpoint file >

[jira] [Commented] (SPARK-20436) NullPointerException when restart from checkpoint file

[ https://issues.apache.org/jira/browse/SPARK-20436?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993663#comment-15993663 ] Shixiong Zhu commented on SPARK-20436: -- [~ffbin] the issue is this line {{val words2 =

[jira] [Resolved] (SPARK-20436) NullPointerException when restart from checkpoint file

[ https://issues.apache.org/jira/browse/SPARK-20436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20436. -- Resolution: Not A Problem > NullPointerException when restart from checkpoint file >

[jira] [Updated] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20547: - Target Version/s: (was: 2.2.0) > ExecutorClassLoader's findClass may not work correctly when a

[jira] [Updated] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20547: - Priority: Minor (was: Blocker) > ExecutorClassLoader's findClass may not work correctly when a

[jira] [Commented] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15993426#comment-15993426 ] Shixiong Zhu commented on SPARK-20547: -- Did some investigation using the reproducer. Looks like it’s

[jira] [Updated] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20547: - Component/s: (was: Spark Core) Spark Shell > ExecutorClassLoader's

[jira] [Comment Edited] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991794#comment-15991794 ] Shixiong Zhu edited comment on SPARK-20547 at 5/1/17 11:37 PM: --- This is a

[jira] [Commented] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15991794#comment-15991794 ] Shixiong Zhu commented on SPARK-20547: -- This is a reproducer:

[jira] [Updated] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20547: - Description: ExecutorClassLoader's findClass may throw some transient exception. For example,

[jira] [Updated] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20547: - Priority: Blocker (was: Major) > ExecutorClassLoader's findClass may not work correctly when a

[jira] [Created] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

Shixiong Zhu created SPARK-20547: Summary: ExecutorClassLoader's findClass may not work correctly when a task is cancelled. Key: SPARK-20547 URL: https://issues.apache.org/jira/browse/SPARK-20547

[jira] [Resolved] (SPARK-20464) Add a job group and an informative description for streaming queries

[ https://issues.apache.org/jira/browse/SPARK-20464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20464. -- Resolution: Fixed Assignee: Kunal Khamar Fix Version/s: 2.2.0 > Add a job

[jira] [Created] (SPARK-20529) Worker should not use the received Master address

Shixiong Zhu created SPARK-20529: Summary: Worker should not use the received Master address Key: SPARK-20529 URL: https://issues.apache.org/jira/browse/SPARK-20529 Project: Spark Issue

[jira] [Resolved] (SPARK-19525) Enable Compression of RDD Checkpoints

[ https://issues.apache.org/jira/browse/SPARK-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19525. -- Resolution: Fixed Assignee: Aaditya Ramesh > Enable Compression of RDD Checkpoints >

[jira] [Updated] (SPARK-19525) Enable Compression of RDD Checkpoints

[ https://issues.apache.org/jira/browse/SPARK-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19525: - Fix Version/s: 2.2.0 > Enable Compression of RDD Checkpoints >

[jira] [Updated] (SPARK-20489) Different results in local mode and yarn mode when working with dates (race condition with SimpleDateFormat?)

[ https://issues.apache.org/jira/browse/SPARK-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20489: - Component/s: (was: Shuffle) (was: Spark Core) > Different results in

[jira] [Commented] (SPARK-20489) Different results in local mode and yarn mode when working with dates (race condition with SimpleDateFormat?)

[ https://issues.apache.org/jira/browse/SPARK-20489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987848#comment-15987848 ] Shixiong Zhu commented on SPARK-20489: -- Could you show the results of `loadDateResult.show(false)`?

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983809#comment-15983809 ] Shixiong Zhu commented on SPARK-18057: -- I prefer to just wait. The user can still use Kafka 0.10.2.0

[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983615#comment-15983615 ] Shixiong Zhu commented on SPARK-13747: -- [~dnaumenko] Unfortunately, Spark uses ThreadLocal variables

[jira] [Updated] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-13747: - Fix Version/s: (was: 2.2.0) > Concurrent execution in SQL doesn't work with Scala

[jira] [Reopened] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-13747: -- > Concurrent execution in SQL doesn't work with Scala ForkJoinPool >

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983396#comment-15983396 ] Shixiong Zhu edited comment on SPARK-18057 at 4/25/17 6:29 PM: --- [~guozhang]

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15983396#comment-15983396 ] Shixiong Zhu commented on SPARK-18057: -- [~guozhang] We have a stress test to test Spark Kafka

[jira] [Created] (SPARK-20461) CachedKafkaConsumer may hang forever when it's interrupted

Shixiong Zhu created SPARK-20461: Summary: CachedKafkaConsumer may hang forever when it's interrupted Key: SPARK-20461 URL: https://issues.apache.org/jira/browse/SPARK-20461 Project: Spark

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982180#comment-15982180 ] Shixiong Zhu commented on SPARK-18057: -- [~ijuma] it's not a regression. In Kafka 0.10.0.1, deleting

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15982180#comment-15982180 ] Shixiong Zhu edited comment on SPARK-18057 at 4/25/17 12:34 AM: [~ijuma]

[jira] [Created] (SPARK-20452) Cancel a batch Kafka query and rerun the same DataFrame may cause ConcurrentModificationException

Shixiong Zhu created SPARK-20452: Summary: Cancel a batch Kafka query and rerun the same DataFrame may cause ConcurrentModificationException Key: SPARK-20452 URL: https://issues.apache.org/jira/browse/SPARK-20452

[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15977132#comment-15977132 ] Shixiong Zhu commented on SPARK-13747: -- [~mousa] could you try the master branch? This issue will be

[jira] [Resolved] (SPARK-20397) Flaky Test: test_streaming.R.Terminated by error

[ https://issues.apache.org/jira/browse/SPARK-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20397. -- Resolution: Fixed Fix Version/s: 2.2.0 > Flaky Test: test_streaming.R.Terminated by

[jira] [Commented] (SPARK-20397) Flaky Test: test_streaming.R.Terminated by error

[ https://issues.apache.org/jira/browse/SPARK-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975269#comment-15975269 ] Shixiong Zhu commented on SPARK-20397: -- I saw the other tests use 5 sec and so just to make it

[jira] [Updated] (SPARK-20397) Flaky Test: test_streaming.R.Terminated by error

[ https://issues.apache.org/jira/browse/SPARK-20397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20397: - Labels: flaky-test (was: ) > Flaky Test: test_streaming.R.Terminated by error >

[jira] [Created] (SPARK-20397) Flaky Test: test_streaming.R.Terminated by error

Shixiong Zhu created SPARK-20397: Summary: Flaky Test: test_streaming.R.Terminated by error Key: SPARK-20397 URL: https://issues.apache.org/jira/browse/SPARK-20397 Project: Spark Issue Type:

[jira] [Updated] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19968: - Target Version/s: 2.3.0 (was: 2.2.0) > Use a cached instance of KafkaProducer for writing to

[jira] [Updated] (SPARK-20370) create external table on read only location fails

[ https://issues.apache.org/jira/browse/SPARK-20370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20370: - Component/s: (was: Spark Core) SQL > create external table on read only

[jira] [Updated] (SPARK-20367) Spark silently escapes partition column names

[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20367: - Component/s: (was: Spark Core) SQL > Spark silently escapes partition

[jira] [Commented] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM

[ https://issues.apache.org/jira/browse/SPARK-20340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969680#comment-15969680 ] Shixiong Zhu commented on SPARK-20340: -- I think it's just a trade off between accuracy and

[jira] [Updated] (SPARK-20341) Support BigIngeger values > 19 precision

[ https://issues.apache.org/jira/browse/SPARK-20341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20341: - Component/s: (was: Spark Core) SQL > Support BigIngeger values > 19

[jira] [Updated] (SPARK-16900) Complete-mode output for file sinks

[ https://issues.apache.org/jira/browse/SPARK-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-16900: - Component/s: (was: DStreams) Structured Streaming > Complete-mode output

[jira] [Updated] (SPARK-20312) query optimizer calls udf with null values when it doesn't expect them

[ https://issues.apache.org/jira/browse/SPARK-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20312: - Component/s: (was: Spark Core) SQL > query optimizer calls udf with null

[jira] [Commented] (SPARK-20321) Spark UI cannot be shutdown in spark streaming app

[ https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968290#comment-15968290 ] Shixiong Zhu commented on SPARK-20321: -- You cannot stop a StreamingContext in foreachRDD. For

[jira] [Resolved] (SPARK-20131) Flaky Test: o.a.s.streaming.StreamingContextSuite.SPARK-18560 Receiver data should be deserialized properly

[ https://issues.apache.org/jira/browse/SPARK-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20131. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-20131) Flaky Test: o.a.s.streaming.StreamingContextSuite.SPARK-18560 Receiver data should be deserialized properly

[ https://issues.apache.org/jira/browse/SPARK-20131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20131: - Summary: Flaky Test: o.a.s.streaming.StreamingContextSuite.SPARK-18560 Receiver data should be

[jira] [Resolved] (SPARK-20282) Flaky test: org.apache.spark.sql.streaming/StreamingQuerySuite/OneTime_trigger__commit_log__and_exception

[ https://issues.apache.org/jira/browse/SPARK-20282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20282. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.2.0 > Flaky test: >

[jira] [Updated] (SPARK-20285) Flaky test: pyspark.streaming.tests.BasicOperationTests.test_cogroup

[ https://issues.apache.org/jira/browse/SPARK-20285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20285: - Affects Version/s: 2.1.1 2.0.3 > Flaky test:

[jira] [Updated] (SPARK-20285) Flaky test: pyspark.streaming.tests.BasicOperationTests.test_cogroup

[ https://issues.apache.org/jira/browse/SPARK-20285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20285: - Affects Version/s: (was: 2.1.1) (was: 2.0.3)

[jira] [Resolved] (SPARK-20285) Flaky test: pyspark.streaming.tests.BasicOperationTests.test_cogroup

[ https://issues.apache.org/jira/browse/SPARK-20285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20285. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 2.0.3

[jira] [Commented] (SPARK-20285) Flaky test: pyspark.streaming.tests.BasicOperationTests.test_cogroup

[ https://issues.apache.org/jira/browse/SPARK-20285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15963422#comment-15963422 ] Shixiong Zhu commented on SPARK-20285: -- https://github.com/apache/spark/pull/17597 > Flaky test:

[jira] [Issue Comment Deleted] (SPARK-20285) Flaky test: pyspark.streaming.tests.BasicOperationTests.test_cogroup

[ https://issues.apache.org/jira/browse/SPARK-20285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20285: - Comment: was deleted (was: https://github.com/apache/spark/pull/17597) > Flaky test:

[jira] [Created] (SPARK-20285) Flaky test:

Shixiong Zhu created SPARK-20285: Summary: Flaky test: Key: SPARK-20285 URL: https://issues.apache.org/jira/browse/SPARK-20285 Project: Spark Issue Type: Bug Components: Tests

[jira] [Updated] (SPARK-20285) Flaky test: pyspark.streaming.tests.BasicOperationTests.test_cogroup

[ https://issues.apache.org/jira/browse/SPARK-20285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20285: - Summary: Flaky test: pyspark.streaming.tests.BasicOperationTests.test_cogroup (was: Flaky test:

[jira] [Updated] (SPARK-20282) Flaky test: org.apache.spark.sql.streaming/StreamingQuerySuite/OneTime_trigger__commit_log__and_exception

[ https://issues.apache.org/jira/browse/SPARK-20282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20282: - Issue Type: Test (was: Bug) > Flaky test: >

[jira] [Created] (SPARK-20282) Flaky test: org.apache.spark.sql.streaming/StreamingQuerySuite/OneTime_trigger__commit_log__and_exception

Shixiong Zhu created SPARK-20282: Summary: Flaky test: org.apache.spark.sql.streaming/StreamingQuerySuite/OneTime_trigger__commit_log__and_exception Key: SPARK-20282 URL:

[jira] [Commented] (SPARK-18971) Netty issue may cause the shuffle client hang

[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15956209#comment-15956209 ] Shixiong Zhu commented on SPARK-18971: -- It's not backported because there are two many changes in

[jira] [Updated] (SPARK-18971) Netty issue may cause the shuffle client hang

[ https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18971: - Description: Check https://github.com/netty/netty/issues/6153 for details You should be able to

[jira] [Resolved] (SPARK-19721) Good error message for version mismatch in log files

[ https://issues.apache.org/jira/browse/SPARK-19721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19721. -- Resolution: Fixed Fix Version/s: 2.1.1 > Good error message for version mismatch in log

[jira] [Created] (SPARK-19986) Make pyspark.streaming.tests.CheckpointTests more stable

Shixiong Zhu created SPARK-19986: Summary: Make pyspark.streaming.tests.CheckpointTests more stable Key: SPARK-19986 URL: https://issues.apache.org/jira/browse/SPARK-19986 Project: Spark

[jira] [Assigned] (SPARK-19721) Good error message for version mismatch in log files

[ https://issues.apache.org/jira/browse/SPARK-19721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-19721: Assignee: Liwei Lin > Good error message for version mismatch in log files >

[jira] [Updated] (SPARK-19721) Good error message for version mismatch in log files

[ https://issues.apache.org/jira/browse/SPARK-19721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19721: - Fix Version/s: 2.2.0 > Good error message for version mismatch in log files >

[jira] [Commented] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

[ https://issues.apache.org/jira/browse/SPARK-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15928657#comment-15928657 ] Shixiong Zhu commented on SPARK-19965: -- [~lwlin] I think we can just ignore “_spark_metadata” in

[jira] [Commented] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

[ https://issues.apache.org/jira/browse/SPARK-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927572#comment-15927572 ] Shixiong Zhu commented on SPARK-19965: -- [~lwlin] Go ahead. I guess the root cause probably is using

[jira] [Commented] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

[ https://issues.apache.org/jira/browse/SPARK-19965?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15927333#comment-15927333 ] Shixiong Zhu commented on SPARK-19965: -- This is because inferring partitions doesn't ignore the

[jira] [Created] (SPARK-19965) DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output

Shixiong Zhu created SPARK-19965: Summary: DataFrame batch reader may fail to infer partitions when reading FileStreamSink's output Key: SPARK-19965 URL: https://issues.apache.org/jira/browse/SPARK-19965

[jira] [Resolved] (SPARK-19853) Uppercase Kafka topics fail when startingOffsets are SpecificOffsets

[ https://issues.apache.org/jira/browse/SPARK-19853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19853. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19831) Sending the heartbeat master from worker maybe blocked by other rpc messages

[ https://issues.apache.org/jira/browse/SPARK-19831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19831. -- Resolution: Fixed Assignee: hustfxj Fix Version/s: 2.2.0 > Sending the

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905886#comment-15905886 ] Shixiong Zhu commented on SPARK-18057: -- > Based on previous kafka client upgrades I wouldn't expect

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905716#comment-15905716 ] Shixiong Zhu edited comment on SPARK-18057 at 3/10/17 9:21 PM: --- I did some

[jira] [Comment Edited] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905716#comment-15905716 ] Shixiong Zhu edited comment on SPARK-18057 at 3/10/17 9:21 PM: --- I did some

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905716#comment-15905716 ] Shixiong Zhu commented on SPARK-18057: -- I did some investigation yesterday, and found one issue in

[jira] [Resolved] (SPARK-19891) Await Batch Lock not signaled on stream execution exit

[ https://issues.apache.org/jira/browse/SPARK-19891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19891. -- Resolution: Fixed Assignee: Tyson Condie Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19886) reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource

[ https://issues.apache.org/jira/browse/SPARK-19886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19886. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > reportDataLoss cause

[jira] [Resolved] (SPARK-19861) watermark should not be a negative time.

[ https://issues.apache.org/jira/browse/SPARK-19861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19861. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19715) Option to Strip Paths in FileSource

[ https://issues.apache.org/jira/browse/SPARK-19715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19715. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0 > Option to Strip

[jira] [Created] (SPARK-19885) The config ignoreCorruptFiles doesn't work for CSV

Shixiong Zhu created SPARK-19885: Summary: The config ignoreCorruptFiles doesn't work for CSV Key: SPARK-19885 URL: https://issues.apache.org/jira/browse/SPARK-19885 Project: Spark Issue

[jira] [Resolved] (SPARK-19874) Hide API docs for "org.apache.spark.sql.internal"

[ https://issues.apache.org/jira/browse/SPARK-19874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19874. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Hide API docs for

<    4   5   6   7   8   9   10   11   12   13   >