[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[ https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21123: - Description: Right now options for file stream source are documented with file sink. We should

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[ https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21123: - Description: !Screen Shot 2017-06-15 at 11.25.49 AM.png|thumbnail!

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[ https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21123: - Description: !attachment-name.jpg|thumbnail! > Options for file stream source are in a wrong

[jira] [Created] (SPARK-21123) Options for file stream source are in a wrong table

Shixiong Zhu created SPARK-21123: Summary: Options for file stream source are in a wrong table Key: SPARK-21123 URL: https://issues.apache.org/jira/browse/SPARK-21123 Project: Spark Issue

[jira] [Updated] (SPARK-21123) Options for file stream source are in a wrong table

[ https://issues.apache.org/jira/browse/SPARK-21123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21123: - Attachment: Screen Shot 2017-06-15 at 11.25.49 AM.png > Options for file stream source are in a

[jira] [Updated] (SPARK-20979) Add a rate source to generate values for tests and benchmark

[ https://issues.apache.org/jira/browse/SPARK-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20979: - Fix Version/s: 2.2.0 > Add a rate source to generate values for tests and benchmark >

[jira] [Updated] (SPARK-21069) Add rate source to programming guide

[ https://issues.apache.org/jira/browse/SPARK-21069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21069: - Labels: starter (was: ) > Add rate source to programming guide >

[jira] [Updated] (SPARK-20979) Add a rate source to generate values for tests and benchmark

[ https://issues.apache.org/jira/browse/SPARK-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20979: - Affects Version/s: (was: 2.2.0) 2.3.0 > Add a rate source to generate

[jira] [Created] (SPARK-21069) Add rate source to programming guide

Shixiong Zhu created SPARK-21069: Summary: Add rate source to programming guide Key: SPARK-21069 URL: https://issues.apache.org/jira/browse/SPARK-21069 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-20979) Add a rate source to generate values for tests and benchmark

[ https://issues.apache.org/jira/browse/SPARK-20979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20979. -- Resolution: Fixed Fix Version/s: 2.3.0 > Add a rate source to generate values for tests

[jira] [Commented] (SPARK-20927) Add cache operator to Unsupported Operations in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16046740#comment-16046740 ] Shixiong Zhu commented on SPARK-20927: -- Do nothing except logging a warning > Add cache operator to

[jira] [Updated] (SPARK-21029) All StreamingQuery should be stopped when the SparkSession is stopped

[ https://issues.apache.org/jira/browse/SPARK-21029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21029: - Issue Type: Improvement (was: Bug) > All StreamingQuery should be stopped when the SparkSession

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043732#comment-16043732 ] Shixiong Zhu commented on SPARK-20952: -- For `ParquetFileFormat#readFootersInParallel`, I would

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043710#comment-16043710 ] Shixiong Zhu commented on SPARK-20952: -- Although I don't know what you plan to do, you can save the

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043708#comment-16043708 ] Shixiong Zhu commented on SPARK-20952: -- Why it needs TaskContext? > TaskContext should be an

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043694#comment-16043694 ] Shixiong Zhu commented on SPARK-20952: -- [~robert3005] could you show me your codes? Are you

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043655#comment-16043655 ] Shixiong Zhu commented on SPARK-20952: -- If TaskContext is not inheritable, we can always find a way

[jira] [Issue Comment Deleted] (SPARK-21022) RDD.foreach swallows exceptions

[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21022: - Comment: was deleted (was: ~~Good catch...~~) > RDD.foreach swallows exceptions >

[jira] [Commented] (SPARK-21022) RDD.foreach swallows exceptions

[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043266#comment-16043266 ] Shixiong Zhu commented on SPARK-21022: -- Wait. I also checked `foreach` method. It does throw the

[jira] [Comment Edited] (SPARK-21022) RDD.foreach swallows exceptions

[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043223#comment-16043223 ] Shixiong Zhu edited comment on SPARK-21022 at 6/8/17 7:05 PM: -- ~~Good

[jira] [Commented] (SPARK-21022) RDD.foreach swallows exceptions

[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043247#comment-16043247 ] Shixiong Zhu commented on SPARK-21022: -- By the way, `foreachPartition` doesn't have the issue. It's

[jira] [Assigned] (SPARK-21022) RDD.foreach swallows exceptions

[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-21022: Assignee: Shixiong Zhu > RDD.foreach swallows exceptions >

[jira] [Commented] (SPARK-21022) RDD.foreach swallows exceptions

[ https://issues.apache.org/jira/browse/SPARK-21022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043223#comment-16043223 ] Shixiong Zhu commented on SPARK-21022: -- Good catch... > RDD.foreach swallows exceptions >

[jira] [Commented] (SPARK-20971) Purge the metadata log for FileStreamSource

[ https://issues.apache.org/jira/browse/SPARK-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16043150#comment-16043150 ] Shixiong Zhu commented on SPARK-20971: -- FileStreamSource saves the seen files in the disk/HDFS, we

[jira] [Updated] (SPARK-21008) Streaming applications read stale credentials file when recovering from checkpoint.

[ https://issues.apache.org/jira/browse/SPARK-21008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21008: - Component/s: (was: Structured Streaming) DStreams > Streaming applications

[jira] [Assigned] (SPARK-20991) BROADCAST_TIMEOUT conf should be a timeoutConf

[ https://issues.apache.org/jira/browse/SPARK-20991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-20991: Assignee: Feng Liu > BROADCAST_TIMEOUT conf should be a timeoutConf >

[jira] [Resolved] (SPARK-20991) BROADCAST_TIMEOUT conf should be a timeoutConf

[ https://issues.apache.org/jira/browse/SPARK-20991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20991. -- Resolution: Fixed Fix Version/s: 2.3.0 > BROADCAST_TIMEOUT conf should be a timeoutConf

[jira] [Created] (SPARK-20979) Add a rate source to generate values for tests and benchmark

Shixiong Zhu created SPARK-20979: Summary: Add a rate source to generate values for tests and benchmark Key: SPARK-20979 URL: https://issues.apache.org/jira/browse/SPARK-20979 Project: Spark

[jira] [Created] (SPARK-20971) Purge the metadata log for FileStreamSource

Shixiong Zhu created SPARK-20971: Summary: Purge the metadata log for FileStreamSource Key: SPARK-20971 URL: https://issues.apache.org/jira/browse/SPARK-20971 Project: Spark Issue Type:

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16035302#comment-16035302 ] Shixiong Zhu commented on SPARK-20952: -- What I'm concerned about is global thread pools, such as

[jira] [Resolved] (SPARK-20955) A lot of duplicated "executorId" strings in "TaskUIData"s

[ https://issues.apache.org/jira/browse/SPARK-20955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20955. -- Resolution: Fixed Fix Version/s: 2.2.0 > A lot of duplicated "executorId" strings in

[jira] [Created] (SPARK-20957) Flaky Test: o.a.s.sql.streaming.StreamingQueryManagerSuite listing

Shixiong Zhu created SPARK-20957: Summary: Flaky Test: o.a.s.sql.streaming.StreamingQueryManagerSuite listing Key: SPARK-20957 URL: https://issues.apache.org/jira/browse/SPARK-20957 Project: Spark

[jira] [Comment Edited] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033915#comment-16033915 ] Shixiong Zhu edited comment on SPARK-20952 at 6/1/17 11:43 PM: ---

[jira] [Commented] (SPARK-20952) TaskContext should be an InheritableThreadLocal

[ https://issues.apache.org/jira/browse/SPARK-20952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033915#comment-16033915 ] Shixiong Zhu commented on SPARK-20952: -- InheritableThreadLocal only works when creating a new

[jira] [Assigned] (SPARK-20894) Error while checkpointing to HDFS

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-20894: Assignee: Shixiong Zhu > Error while checkpointing to HDFS >

[jira] [Updated] (SPARK-20894) Error while checkpointing to HDFS

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20894: - Fix Version/s: 2.3.0 > Error while checkpointing to HDFS > - > >

[jira] [Commented] (SPARK-20927) Add cache operator to Unsupported Operations in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033731#comment-16033731 ] Shixiong Zhu commented on SPARK-20927: -- neither cache nor checkpoint makes sense for a streaming

[jira] [Updated] (SPARK-20927) Add cache operator to Unsupported Operations in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-20927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20927: - Affects Version/s: (was: 2.3.0) 2.2.0 > Add cache operator to

[jira] [Updated] (SPARK-20935) A daemon thread, "BatchedWriteAheadLog Writer", left behind after terminating StreamingContext.

[ https://issues.apache.org/jira/browse/SPARK-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20935: - Component/s: (was: Structured Streaming) DStreams > A daemon thread,

[jira] [Commented] (SPARK-20955) A lot of duplicated "executorId" strings in "TaskUIData"s

[ https://issues.apache.org/jira/browse/SPARK-20955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16033664#comment-16033664 ] Shixiong Zhu commented on SPARK-20955: -- In [this

[jira] [Created] (SPARK-20955) A lot of duplicated "executorId" strings in "TaskUIData"s

Shixiong Zhu created SPARK-20955: Summary: A lot of duplicated "executorId" strings in "TaskUIData"s Key: SPARK-20955 URL: https://issues.apache.org/jira/browse/SPARK-20955 Project: Spark

[jira] [Resolved] (SPARK-20940) AccumulatorV2 should not throw IllegalAccessError

[ https://issues.apache.org/jira/browse/SPARK-20940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20940. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.2 2.0.3

[jira] [Created] (SPARK-20940) AccumulatorV2 should not throw IllegalAccessError

Shixiong Zhu created SPARK-20940: Summary: AccumulatorV2 should not throw IllegalAccessError Key: SPARK-20940 URL: https://issues.apache.org/jira/browse/SPARK-20940 Project: Spark Issue

[jira] [Resolved] (SPARK-20883) Improve StateStore APIs for efficiency

[ https://issues.apache.org/jira/browse/SPARK-20883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20883. -- Resolution: Fixed Fix Version/s: 2.3.0 > Improve StateStore APIs for efficiency >

[jira] [Commented] (SPARK-20894) Error while checkpointing to HDFS (similar to JIRA SPARK-19268)

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030160#comment-16030160 ] Shixiong Zhu commented on SPARK-20894: -- The root issue here is the driver uses the local file system

[jira] [Updated] (SPARK-20894) Error while checkpointing to HDFS

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20894: - Summary: Error while checkpointing to HDFS (was: Error while checkpointing to HDFS (similar to

[jira] [Updated] (SPARK-20894) Error while checkpointing to HDFS (similar to JIRA SPARK-19268)

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20894: - Issue Type: Improvement (was: Bug) > Error while checkpointing to HDFS (similar to JIRA

[jira] [Reopened] (SPARK-20894) Error while checkpointing to HDFS (similar to JIRA SPARK-19268)

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-20894: -- > Error while checkpointing to HDFS (similar to JIRA SPARK-19268) >

[jira] [Updated] (SPARK-20597) KafkaSourceProvider falls back on path as synonym for topic

[ https://issues.apache.org/jira/browse/SPARK-20597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20597: - Labels: starter (was: ) > KafkaSourceProvider falls back on path as synonym for topic >

[jira] [Updated] (SPARK-20599) ConsoleSink should work with write (batch)

[ https://issues.apache.org/jira/browse/SPARK-20599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20599: - Labels: starter (was: ) > ConsoleSink should work with write (batch) >

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer using guava cache.

[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20919: - Affects Version/s: (was: 2.3.0) 2.2.0 Target Version/s: 2.3.0 >

[jira] [Updated] (SPARK-20919) Simplificaiton of CachedKafkaConsumer using guava cache.

[ https://issues.apache.org/jira/browse/SPARK-20919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20919: - Issue Type: Improvement (was: Bug) > Simplificaiton of CachedKafkaConsumer using guava cache. >

[jira] [Resolved] (SPARK-19968) Use a cached instance of KafkaProducer for writing to kafka via KafkaSink.

[ https://issues.apache.org/jira/browse/SPARK-19968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19968. -- Resolution: Fixed Assignee: Prashant Sharma Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-20907) Use testQuietly for test suites that generate long log output

[ https://issues.apache.org/jira/browse/SPARK-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20907. -- Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.2.0 > Use

[jira] [Updated] (SPARK-20907) Use testQuietly for test suites that generate long log output

[ https://issues.apache.org/jira/browse/SPARK-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20907: - Affects Version/s: (was: 2.3.0) > Use testQuietly for test suites that generate long log

[jira] [Updated] (SPARK-20907) Use testQuietly for test suites that generate long log output

[ https://issues.apache.org/jira/browse/SPARK-20907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20907: - Priority: Minor (was: Major) > Use testQuietly for test suites that generate long log output >

[jira] [Updated] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19372: - Fix Version/s: 2.2.0 > Code generation for Filter predicate including many OR conditions exceeds

[jira] [Resolved] (SPARK-20843) Cannot gracefully kill drivers which take longer than 10 seconds to die

[ https://issues.apache.org/jira/browse/SPARK-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20843. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-20894) Error while checkpointing to HDFS (similar to JIRA SPARK-19268)

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026846#comment-16026846 ] Shixiong Zhu commented on SPARK-20894: -- Thanks for reporting it. I'm wondering if you can provide

[jira] [Updated] (SPARK-20894) Error while checkpointing to HDFS (similar to JIRA SPARK-19268)

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20894: - Docs Text: (was: Using Spark's default log4j profile:

[jira] [Commented] (SPARK-20894) Error while checkpointing to HDFS (similar to JIRA SPARK-19268)

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026848#comment-16026848 ] Shixiong Zhu commented on SPARK-20894: -- By the way, you can click "More" -> "Attach Files" to upload

[jira] [Commented] (SPARK-20843) Cannot gracefully kill drivers which take longer than 10 seconds to die

[ https://issues.apache.org/jira/browse/SPARK-20843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026809#comment-16026809 ] Shixiong Zhu commented on SPARK-20843: -- [~michael] Will a per-cluster config be enough for your

[jira] [Resolved] (SPARK-20014) Optimize mergeSpillsWithFileStream method

[ https://issues.apache.org/jira/browse/SPARK-20014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20014. -- Resolution: Fixed Assignee: Sital Kedia Fix Version/s: 2.3.0 > Optimize

[jira] [Resolved] (SPARK-20844) Remove experimental from API and docs

[ https://issues.apache.org/jira/browse/SPARK-20844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20844. -- Resolution: Fixed Assignee: Michael Armbrust Fix Version/s: 2.2.0 > Remove

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16026595#comment-16026595 ] Shixiong Zhu commented on SPARK-20882: -- [~cenyuhai] do you mind to share what's the root cause? >

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025588#comment-16025588 ] Shixiong Zhu commented on SPARK-20882: -- [~cenyuhai] Did you see this log "logger.error("Still have

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025506#comment-16025506 ] Shixiong Zhu commented on SPARK-20882: -- [~cenyuhai] Is it possible to reproduce it and get logs of

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025377#comment-16025377 ] Shixiong Zhu commented on SPARK-20882: -- NVM. I saw NettyBlockTransferService in the log. > Executor

[jira] [Commented] (SPARK-20882) Executor is waiting for ShuffleBlockFetcherIterator

[ https://issues.apache.org/jira/browse/SPARK-20882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16025227#comment-16025227 ] Shixiong Zhu commented on SPARK-20882: -- [~cenyuhai] are you using shuffle service? > Executor is

[jira] [Resolved] (SPARK-20874) The "examples" project doesn't depend on Structured Streaming Kafka source

[ https://issues.apache.org/jira/browse/SPARK-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20874. -- Resolution: Fixed > The "examples" project doesn't depend on Structured Streaming Kafka source

[jira] [Updated] (SPARK-20874) The "examples" project doesn't depend on Structured Streaming Kafka source

[ https://issues.apache.org/jira/browse/SPARK-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20874: - Fix Version/s: 2.2.0 2.1.2 > The "examples" project doesn't depend on

[jira] [Updated] (SPARK-20874) The "examples" project doesn't depend on Structured Streaming Kafka source

[ https://issues.apache.org/jira/browse/SPARK-20874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20874: - Affects Version/s: 2.2.0 > The "examples" project doesn't depend on Structured Streaming Kafka

[jira] [Created] (SPARK-20874) The "examples" project doesn't depend on Structured Streaming Kafka source

Shixiong Zhu created SPARK-20874: Summary: The "examples" project doesn't depend on Structured Streaming Kafka source Key: SPARK-20874 URL: https://issues.apache.org/jira/browse/SPARK-20874 Project:

[jira] [Updated] (SPARK-20799) Unable to infer schema for ORC on reading ORC from S3

[ https://issues.apache.org/jira/browse/SPARK-20799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20799: - Component/s: (was: Spark Core) SQL > Unable to infer schema for ORC on

[jira] [Resolved] (SPARK-20792) Support same timeout operations in mapGroupsWithState function in batch queries as in streaming queries

[ https://issues.apache.org/jira/browse/SPARK-20792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20792. -- Resolution: Fixed > Support same timeout operations in mapGroupsWithState function in batch >

[jira] [Resolved] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-13747. -- Resolution: Fixed Fix Version/s: 2.2.0 Target Version/s: 2.1.0, 2.0.2 (was:

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16014799#comment-16014799 ] Shixiong Zhu commented on SPARK-17463: -- [~allengeorge] it's protected by

[jira] [Resolved] (SPARK-20788) Fix the Executor task reaper's false alarm warning logs

[ https://issues.apache.org/jira/browse/SPARK-20788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20788. -- Resolution: Fixed Fix Version/s: 2.2.0 > Fix the Executor task reaper's false alarm

[jira] [Created] (SPARK-20788) Fix the Executor task reaper's false alarm warning logs

Shixiong Zhu created SPARK-20788: Summary: Fix the Executor task reaper's false alarm warning logs Key: SPARK-20788 URL: https://issues.apache.org/jira/browse/SPARK-20788 Project: Spark

[jira] [Assigned] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-19372: Assignee: Kazuaki Ishizaki > Code generation for Filter predicate including many OR

[jira] [Resolved] (SPARK-19372) Code generation for Filter predicate including many OR conditions exceeds JVM method size limit

[ https://issues.apache.org/jira/browse/SPARK-19372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19372. -- Resolution: Fixed Fix Version/s: 2.3.0 > Code generation for Filter predicate including

[jira] [Created] (SPARK-20774) BroadcastExchangeExec doesn't cancel the Spark job if broadcasting a relation timeouts.

Shixiong Zhu created SPARK-20774: Summary: BroadcastExchangeExec doesn't cancel the Spark job if broadcasting a relation timeouts. Key: SPARK-20774 URL: https://issues.apache.org/jira/browse/SPARK-20774

[jira] [Resolved] (SPARK-20529) Worker should not use the received Master address

[ https://issues.apache.org/jira/browse/SPARK-20529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20529. -- Resolution: Fixed Fix Version/s: 2.2.0 > Worker should not use the received Master

[jira] [Updated] (SPARK-20666) Flaky test - SparkListenerBus randomly failing java.lang.IllegalAccessError

[ https://issues.apache.org/jira/browse/SPARK-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20666: - Fix Version/s: (was: 2.2.1) (was: 2.3.0) 2.2.0 >

[jira] [Updated] (SPARK-20666) Flaky test - SparkListenerBus randomly failing java.lang.IllegalAccessError

[ https://issues.apache.org/jira/browse/SPARK-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20666: - Affects Version/s: (was: 2.3.0) 2.2.0 > Flaky test - SparkListenerBus

[jira] [Issue Comment Deleted] (SPARK-20666) Flaky test - SparkListenerBus randomly failing java.lang.IllegalAccessError

[ https://issues.apache.org/jira/browse/SPARK-20666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20666: - Comment: was deleted (was: User 'shivaram' has created a pull request for this issue:

[jira] [Resolved] (SPARK-20716) StateStore.abort() should not throw further exception

[ https://issues.apache.org/jira/browse/SPARK-20716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20716. -- Resolution: Fixed > StateStore.abort() should not throw further exception >

[jira] [Resolved] (SPARK-20717) Tweak MapGroupsWithState update function behavior

[ https://issues.apache.org/jira/browse/SPARK-20717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20717. -- Resolution: Fixed > Tweak MapGroupsWithState update function behavior >

[jira] [Resolved] (SPARK-20714) Fix match error when watermark is set with timeout = no timeout / processing timeout

[ https://issues.apache.org/jira/browse/SPARK-20714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20714. -- Resolution: Fixed > Fix match error when watermark is set with timeout = no timeout /

[jira] [Resolved] (SPARK-20702) TaskContextImpl.markTaskCompleted should not hide the original error

[ https://issues.apache.org/jira/browse/SPARK-20702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20702. -- Resolution: Fixed Fix Version/s: 2.2.0 > TaskContextImpl.markTaskCompleted should not

[jira] [Resolved] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20600. -- Resolution: Fixed Assignee: Jacek Laskowski Fix Version/s: 2.3.0

[jira] [Created] (SPARK-20702) TaskContextImpl.markTaskCompleted should not hide the original error

Shixiong Zhu created SPARK-20702: Summary: TaskContextImpl.markTaskCompleted should not hide the original error Key: SPARK-20702 URL: https://issues.apache.org/jira/browse/SPARK-20702 Project: Spark

[jira] [Assigned] (SPARK-20373) Batch queries with 'Dataset/DataFrame.withWatermark()` does not execute

[ https://issues.apache.org/jira/browse/SPARK-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-20373: Assignee: Genmao Yu > Batch queries with 'Dataset/DataFrame.withWatermark()` does not

[jira] [Updated] (SPARK-20373) Batch queries with 'Dataset/DataFrame.withWatermark()` does not execute

[ https://issues.apache.org/jira/browse/SPARK-20373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20373: - Fix Version/s: 2.3.0 2.2.1 > Batch queries with

[jira] [Comment Edited] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001680#comment-16001680 ] Shixiong Zhu edited comment on SPARK-20600 at 5/8/17 10:38 PM: ---

[jira] [Commented] (SPARK-20600) KafkaRelation should be pretty printed in web UI (Details for Query)

[ https://issues.apache.org/jira/browse/SPARK-20600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001680#comment-16001680 ] Shixiong Zhu commented on SPARK-20600: -- [~jlaskowski] Hope you want do it soon. Then we can put it

[jira] [Commented] (SPARK-19268) File does not exist: /tmp/temporary-157b89c1-27bb-49f3-a70c-ca1b75022b4d/state/0/2/1.delta

[ https://issues.apache.org/jira/browse/SPARK-19268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001631#comment-16001631 ] Shixiong Zhu commented on SPARK-19268: -- [~skrishna] could you provide your codes, or the output of

[jira] [Commented] (SPARK-18057) Update structured streaming kafka from 10.0.1 to 10.2.0

[ https://issues.apache.org/jira/browse/SPARK-18057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001474#comment-16001474 ] Shixiong Zhu commented on SPARK-18057: -- [~helena_e] I didn't mean for Spark. Even in Spark, the

[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001426#comment-16001426 ] Shixiong Zhu commented on SPARK-13747: -- [~revolucion09] The default dispatcher uses ForkJoinPool.

[jira] [Commented] (SPARK-13747) Concurrent execution in SQL doesn't work with Scala ForkJoinPool

[ https://issues.apache.org/jira/browse/SPARK-13747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16001415#comment-16001415 ] Shixiong Zhu commented on SPARK-13747: -- Okey, I see, the thread name is

<    3   4   5   6   7   8   9   10   11   12   >