[jira] [Resolved] (SPARK-21988) Add default stats to StreamingExecutionRelation

[ https://issues.apache.org/jira/browse/SPARK-21988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21988. -- Resolution: Fixed > Add default stats to StreamingExecutionRelation > -

[jira] [Updated] (SPARK-21988) Add default stats to StreamingRelation and StreamingExecutionRelation

[ https://issues.apache.org/jira/browse/SPARK-21988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21988: - Summary: Add default stats to StreamingRelation and StreamingExecutionRelation (was: Add default

[jira] [Updated] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-22366: - Issue Type: Improvement (was: Bug) > Support ignoreMissingFiles flag parallel to ignoreCorruptFi

[jira] [Updated] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-22366: - Component/s: (was: Spark Core) SQL > Support ignoreMissingFiles flag paralle

[jira] [Commented] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16221432#comment-16221432 ] Shixiong Zhu commented on SPARK-22305: -- [~Yuval.Itzchakov] how many batches per 1 mi

[jira] [Resolved] (SPARK-22366) Support ignoreMissingFiles flag parallel to ignoreCorruptFiles

[ https://issues.apache.org/jira/browse/SPARK-22366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22366. -- Resolution: Fixed Assignee: Jose Torres Fix Version/s: 2.3.0 > Support ignoreMi

[jira] [Commented] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223000#comment-16223000 ] Shixiong Zhu commented on SPARK-22305: -- Why not just delete the whole checkpoint dir

[jira] [Commented] (SPARK-22403) StructuredKafkaWordCount example fails in YARN cluster mode

[ https://issues.apache.org/jira/browse/SPARK-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227266#comment-16227266 ] Shixiong Zhu commented on SPARK-22403: -- Yeah, Spark creates a temp directory for you

[jira] [Commented] (SPARK-22403) StructuredKafkaWordCount example fails in YARN cluster mode

[ https://issues.apache.org/jira/browse/SPARK-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16227321#comment-16227321 ] Shixiong Zhu commented on SPARK-22403: -- Yeah, feel free to submit a PR to improve th

[jira] [Resolved] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22305. -- Resolution: Fixed Fix Version/s: 2.3.0 > HDFSBackedStateStoreProvider fails with StackOv

[jira] [Assigned] (SPARK-22305) HDFSBackedStateStoreProvider fails with StackOverflowException when attempting to recover state

[ https://issues.apache.org/jira/browse/SPARK-22305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-22305: Assignee: Jose Torres > HDFSBackedStateStoreProvider fails with StackOverflowException whe

[jira] [Updated] (SPARK-21930) When the number of attempting to restart receiver greater than 0,spark do nothing in 'else'

[ https://issues.apache.org/jira/browse/SPARK-21930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21930: - Component/s: (was: Structured Streaming) DStreams > When the number of atte

[jira] [Commented] (SPARK-19644) Memory leak in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16234638#comment-16234638 ] Shixiong Zhu commented on SPARK-19644: -- I happened to investigate a similar issue an

[jira] [Commented] (SPARK-19644) Memory leak in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16234643#comment-16234643 ] Shixiong Zhu commented on SPARK-19644: -- By the way, you can confirm this issue by ch

[jira] [Updated] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19644: - Summary: Memory leak in Spark Streaming (Encoder/Scala Reflection) (was: Memory leak in Spark St

[jira] [Updated] (SPARK-19644) Memory leak in Spark Streaming

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19644: - Component/s: SQL > Memory leak in Spark Streaming > -- > >

[jira] [Updated] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19644: - Component/s: Structured Streaming > Memory leak in Spark Streaming (Encoder/Scala Reflection) > -

[jira] [Updated] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19644: - Description: I am using streaming on the production for some aggregation and fetching data from

[jira] [Updated] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19644: - Description: I am using streaming on the production for some aggregation and fetching data from

[jira] [Commented] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16234655#comment-16234655 ] Shixiong Zhu commented on SPARK-19644: -- I added more components since it also affect

[jira] [Updated] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19644: - Description: I am using streaming on the production for some aggregation and fetching data from

[jira] [Resolved] (SPARK-22243) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22243. -- Resolution: Fixed Fix Version/s: 2.3.0 > streaming job failed to restart from checkpoint

[jira] [Assigned] (SPARK-22243) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-22243: Assignee: StephenZou > streaming job failed to restart from checkpoint > -

[jira] [Updated] (SPARK-22429) Streaming checkpointing code does not retry after failure due to NullPointerException

[ https://issues.apache.org/jira/browse/SPARK-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-22429: - Component/s: (was: Structured Streaming) DStreams > Streaming checkpointing

[jira] [Resolved] (SPARK-22403) StructuredKafkaWordCount example fails in YARN cluster mode

[ https://issues.apache.org/jira/browse/SPARK-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22403. -- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > StructuredKafkaWordCou

[jira] [Assigned] (SPARK-22403) StructuredKafkaWordCount example fails in YARN cluster mode

[ https://issues.apache.org/jira/browse/SPARK-22403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-22403: Assignee: Wing Yew Poon > StructuredKafkaWordCount example fails in YARN cluster mode > --

[jira] [Resolved] (SPARK-22294) Reset spark.driver.bindAddress when starting a Checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-22294. -- Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > Reset spark.driver.bin

[jira] [Updated] (SPARK-22243) streaming job failed to restart from checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-22243: - Fix Version/s: 2.2.1 > streaming job failed to restart from checkpoint >

[jira] [Assigned] (SPARK-22294) Reset spark.driver.bindAddress when starting a Checkpoint

[ https://issues.apache.org/jira/browse/SPARK-22294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-22294: Assignee: Santiago Saavedra > Reset spark.driver.bindAddress when starting a Checkpoint >

[jira] [Resolved] (SPARK-19644) Memory leak in Spark Streaming (Encoder/Scala Reflection)

[ https://issues.apache.org/jira/browse/SPARK-19644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19644. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.3.0

[jira] [Resolved] (SPARK-21667) ConsoleSink should not fail streaming query with checkpointLocation option

[ https://issues.apache.org/jira/browse/SPARK-21667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-21667. -- Resolution: Fixed Assignee: Rekha Joshi Fix Version/s: 2.3.0

[jira] [Created] (SPARK-19140) Allow update mode for non-aggregation streaming queries

Shixiong Zhu created SPARK-19140: Summary: Allow update mode for non-aggregation streaming queries Key: SPARK-19140 URL: https://issues.apache.org/jira/browse/SPARK-19140 Project: Spark Issue

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813205#comment-15813205 ] Shixiong Zhu commented on SPARK-17463: -- [~sunil.rangwani] could you have a simple re

[jira] [Commented] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813405#comment-15813405 ] Shixiong Zhu commented on SPARK-18905: -- Sorry for the late reply. Yeah, good catch.

[jira] [Commented] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15813455#comment-15813455 ] Shixiong Zhu commented on SPARK-18905: -- [~CodingCat] I think `pendingTime` is the jo

[jira] [Commented] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15815815#comment-15815815 ] Shixiong Zhu commented on SPARK-18905: -- Sure. Please go ahead. > Potential Issue of

[jira] [Resolved] (SPARK-19137) Garbage left in source tree after SQL tests are run

[ https://issues.apache.org/jira/browse/SPARK-19137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19137. -- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19113: - Fix Version/s: 2.1.1 > Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source

[jira] [Commented] (SPARK-19147) netty throw NPE

[ https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15816170#comment-15816170 ] Shixiong Zhu commented on SPARK-19147: -- I think there is a race condition when the e

[jira] [Updated] (SPARK-19102) Accuracy error of spark SQL results

[ https://issues.apache.org/jira/browse/SPARK-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19102: - Component/s: (was: Spark Core) > Accuracy error of spark SQL results > --

[jira] [Resolved] (SPARK-19140) Allow update mode for non-aggregation streaming queries

[ https://issues.apache.org/jira/browse/SPARK-19140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19140. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Allow update mode for

[jira] [Created] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs

Shixiong Zhu created SPARK-19182: Summary: Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs Key: SPARK-19182 URL: https://issues.apache.org/jira/browse/SPARK-19182

[jira] [Updated] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs

[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19182: - Description: When DStreamGraph is generating a job, it will hold a lock and block other APIs. Be

[jira] [Updated] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block UI when generating Streaming jobs

[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19182: - Summary: Optimize the lock in StreamingJobProgressListener to not block UI when generating Stream

[jira] [Reopened] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-19113: -- Reopened it as it's still flaky > Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors fro

[jira] [Updated] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19113: - Fix Version/s: (was: 2.1.1) (was: 2.2.0) > Fix flaky test: o.a.s.sql.s

[jira] [Resolved] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18905. -- Resolution: Fixed Assignee: Nan Zhu Fix Version/s: 2.2.0 2.1.

[jira] [Updated] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18905: - Affects Version/s: 1.5.1 1.6.3 > Potential Issue of Semantics of BatchComp

[jira] [Updated] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18905: - Affects Version/s: 1.5.2 1.6.1 1.6.2 > Potential Is

[jira] [Updated] (SPARK-19065) dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute replacement

[ https://issues.apache.org/jira/browse/SPARK-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19065: - Summary: dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute

[jira] [Updated] (SPARK-19065) dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute replacement

[ https://issues.apache.org/jira/browse/SPARK-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19065: - Description: Right now if you use .dropDuplicates in a stream you get an exception because attri

[jira] [Reopened] (SPARK-17866) Dataset.dropDuplicates (i.e., distinct) should not change the output of child plan

[ https://issues.apache.org/jira/browse/SPARK-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-17866: -- > Dataset.dropDuplicates (i.e., distinct) should not change the output of child > plan > -

[jira] [Resolved] (SPARK-17866) Dataset.dropDuplicates (i.e., distinct) should not change the output of child plan

[ https://issues.apache.org/jira/browse/SPARK-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17866. -- Resolution: Won't Fix > Dataset.dropDuplicates (i.e., distinct) should not change the output of

[jira] [Commented] (SPARK-17866) Dataset.dropDuplicates (i.e., distinct) should not change the output of child plan

[ https://issues.apache.org/jira/browse/SPARK-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826579#comment-15826579 ] Shixiong Zhu commented on SPARK-17866: -- Although this one was fixed in 2.1.0, it bro

[jira] [Resolved] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

[ https://issues.apache.org/jira/browse/SPARK-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18907. -- Resolution: Duplicate Closing it as this one actually is caused by SPARK-18908. > Fix flaky te

[jira] [Commented] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

[ https://issues.apache.org/jira/browse/SPARK-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15826923#comment-15826923 ] Shixiong Zhu commented on SPARK-18907: -- [~iamshrek] Thanks for looking at it. I forg

[jira] [Updated] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

[ https://issues.apache.org/jira/browse/SPARK-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18907: - Component/s: Tests Structured Streaming > Fix flaky test: o.a.s.sql.streaming.Fi

[jira] [Created] (SPARK-19267) Fix a race condition when stopping StateStore

Shixiong Zhu created SPARK-19267: Summary: Fix a race condition when stopping StateStore Key: SPARK-19267 URL: https://issues.apache.org/jira/browse/SPARK-19267 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-19267) Fix a race condition when stopping StateStore

[ https://issues.apache.org/jira/browse/SPARK-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19267: - Affects Version/s: 2.0.0 2.0.1 2.0.2

[jira] [Updated] (SPARK-19267) Fix a race condition when stopping StateStore

[ https://issues.apache.org/jira/browse/SPARK-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19267: - Priority: Minor (was: Major) > Fix a race condition when stopping StateStore > -

[jira] [Resolved] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19113. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Fix flaky test: o.a.s.

[jira] [Resolved] (SPARK-19168) StateStore should be aborted upon error

[ https://issues.apache.org/jira/browse/SPARK-19168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19168. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0 2.

[jira] [Resolved] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block UI when generating Streaming jobs

[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19182. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0 > Optimize the lock

[jira] [Updated] (SPARK-19268) File does not exist: /tmp/temporary-157b89c1-27bb-49f3-a70c-ca1b75022b4d/state/0/2/1.delta

[ https://issues.apache.org/jira/browse/SPARK-19268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19268: - Priority: Critical (was: Major) > File does not exist: > /tmp/temporary-157b89c1-27bb-49f3-a70c

[jira] [Commented] (SPARK-19268) File does not exist: /tmp/temporary-157b89c1-27bb-49f3-a70c-ca1b75022b4d/state/0/2/1.delta

[ https://issues.apache.org/jira/browse/SPARK-19268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830706#comment-15830706 ] Shixiong Zhu commented on SPARK-19268: -- Right now Structured Streaming doesn't suppo

[jira] [Commented] (SPARK-19275) Spark Streaming, Kafka receiver, "Failed to get records for ... after polling for 512"

[ https://issues.apache.org/jira/browse/SPARK-19275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830976#comment-15830976 ] Shixiong Zhu commented on SPARK-19275: -- This error usually means Spark cannot fetch

[jira] [Commented] (SPARK-19233) Inconsistent Behaviour of Spark Streaming Checkpoint

[ https://issues.apache.org/jira/browse/SPARK-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15830989#comment-15830989 ] Shixiong Zhu commented on SPARK-19233: -- I don't think filtering "generatedRDDs" will

[jira] [Updated] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19280: - Priority: Critical (was: Major) > Failed Recovery from checkpoint caused by the multi-threads is

[jira] [Commented] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15831018#comment-15831018 ] Shixiong Zhu commented on SPARK-19280: -- Good catch and nice explanation. I think ma

[jira] [Commented] (SPARK-19300) Executor is waiting for lock

[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15832658#comment-15832658 ] Shixiong Zhu commented on SPARK-19300: -- Could you provide the full thread dump? Look

[jira] [Commented] (SPARK-19300) Executor is waiting for lock

[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15833244#comment-15833244 ] Shixiong Zhu commented on SPARK-19300: -- Could you check if there is any thread havin

[jira] [Resolved] (SPARK-19300) Executor is waiting for lock

[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19300. -- Resolution: Duplicate Thanks for confirming it. Closing this one as it's a duplicated issue. >

[jira] [Resolved] (SPARK-19139) AES-based authentication mechanism for Spark

[ https://issues.apache.org/jira/browse/SPARK-19139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19139. -- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.2.0 > AES-based aut

[jira] [Resolved] (SPARK-10651) Flaky test: BroadcastSuite

[ https://issues.apache.org/jira/browse/SPARK-10651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-10651. -- Resolution: Fixed Fix Version/s: 2.2.0 The root cause of the failure is SPARK-17755, whi

[jira] [Resolved] (SPARK-19330) Also show tooltip for successful batches

[ https://issues.apache.org/jira/browse/SPARK-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19330. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0 2.

[jira] [Created] (SPARK-19365) Optimize RequestMessage serialization

Shixiong Zhu created SPARK-19365: Summary: Optimize RequestMessage serialization Key: SPARK-19365 URL: https://issues.apache.org/jira/browse/SPARK-19365 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-19365) Optimize RequestMessage serialization

[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19365: - Description: Right now Netty PRC serializes RequestMessage using Java serialization, and the siz

[jira] [Resolved] (SPARK-19365) Optimize RequestMessage serialization

[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19365. -- Resolution: Fixed Fix Version/s: 2.2.0 > Optimize RequestMessage serialization > ---

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

Title: Message Title Shixiong Zhu commented on SPARK-19407

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

Title: Message Title Shixiong Zhu edited a comment on SPARK-19407

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

Title: Message Title Shixiong Zhu updated an issue

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847558#comment-15847558 ] Shixiong Zhu commented on SPARK-19414: -- You also need to override `org.apache.spark

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19414. -- Resolution: Not A Bug > Inferring schema in a structured streaming source > ---

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19414: - Issue Type: Question (was: Bug) > Inferring schema in a structured streaming source > --

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847571#comment-15847571 ] Shixiong Zhu commented on SPARK-19414: -- [~samelamin] I would suggest that you take a

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847574#comment-15847574 ] Shixiong Zhu commented on SPARK-19414: -- Oh, I see, You cannot infer schema when gett

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847578#comment-15847578 ] Shixiong Zhu commented on SPARK-19414: -- Since you can get the DataFrame, the place t

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847598#comment-15847598 ] Shixiong Zhu commented on SPARK-19414: -- [~samelamin] I know little about BigQuery. D

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15847606#comment-15847606 ] Shixiong Zhu commented on SPARK-19414: -- Yes. > Inferring schema in a structured str

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19414. -- Resolution: Not A Bug > Inferring schema in a structured streaming source > ---

[jira] [Resolved] (SPARK-19377) Killed tasks should have the status as KILLED

[ https://issues.apache.org/jira/browse/SPARK-19377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19377. -- Resolution: Fixed Assignee: Devaraj K Fix Version/s: 2.2.0 2.

[jira] [Created] (SPARK-19432) Fix an unexpected failure when connecting timeout

Shixiong Zhu created SPARK-19432: Summary: Fix an unexpected failure when connecting timeout Key: SPARK-19432 URL: https://issues.apache.org/jira/browse/SPARK-19432 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-19432) Fix an unexpected failure when connecting timeout

[ https://issues.apache.org/jira/browse/SPARK-19432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19432. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Fix an unexpected fail

[jira] [Updated] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

[ https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19407: - Target Version/s: 2.1.1, 2.2.0 > defaultFS is used FileSystem.get instead of getting it from uri

[jira] [Resolved] (SPARK-19437) ExecutorId in HearbeatReceiverSuite is incorrect.

[ https://issues.apache.org/jira/browse/SPARK-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19437. -- Resolution: Fixed Assignee: jin xing Fix Version/s: 2.2.0 > ExecutorId in Hearb

[jira] [Created] (SPARK-19481) Fix flaky test: o.a.s.repl.ReplSuite should clone and clean line object in ClosureCleaner

Shixiong Zhu created SPARK-19481: Summary: Fix flaky test: o.a.s.repl.ReplSuite should clone and clean line object in ClosureCleaner Key: SPARK-19481 URL: https://issues.apache.org/jira/browse/SPARK-19481

[jira] [Resolved] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

[ https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19407. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0 2.

[jira] [Resolved] (SPARK-18682) Batch Source for Kafka

[ https://issues.apache.org/jira/browse/SPARK-18682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18682. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Batch Source for Kafka

[jira] [Closed] (SPARK-18386) Batch mode SQL source for Kafka

[ https://issues.apache.org/jira/browse/SPARK-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu closed SPARK-18386. Resolution: Duplicate > Batch mode SQL source for Kafka > --- > >

[jira] [Resolved] (SPARK-19413) Basic mapGroupsWithState API

[ https://issues.apache.org/jira/browse/SPARK-19413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19413. -- Resolution: Fixed Fix Version/s: 2.2.0 > Basic mapGroupsWithState API >

<    2   3   4   5   6   7   8   9   10   11   >