[jira] [Updated] (SPARK-19593) Records read per each kinesis transaction

[ https://issues.apache.org/jira/browse/SPARK-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19593: - Component/s: (was: Structured Streaming) (was: Spark Core)

[jira] [Commented] (SPARK-19594) StreamingQueryListener fails to handle QueryTerminatedEvent if more then one listeners exists

[ https://issues.apache.org/jira/browse/SPARK-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15866778#comment-15866778 ] Shixiong Zhu commented on SPARK-19594: -- Good catch. Would you like to submit a PR to fix it? >

[jira] [Updated] (SPARK-19523) Spark streaming+ insert into table leaves bunch of trash in table directory

[ https://issues.apache.org/jira/browse/SPARK-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19523: - Issue Type: Question (was: Improvement) > Spark streaming+ insert into table leaves bunch of

[jira] [Resolved] (SPARK-19523) Spark streaming+ insert into table leaves bunch of trash in table directory

[ https://issues.apache.org/jira/browse/SPARK-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19523. -- Resolution: Not A Bug > Spark streaming+ insert into table leaves bunch of trash in table

[jira] [Assigned] (SPARK-19497) dropDuplicates with watermark

[ https://issues.apache.org/jira/browse/SPARK-19497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-19497: Assignee: Shixiong Zhu > dropDuplicates with watermark > - >

[jira] [Created] (SPARK-19599) Clean up HDFSMetadataLog for Hadoop 2.6+

Shixiong Zhu created SPARK-19599: Summary: Clean up HDFSMetadataLog for Hadoop 2.6+ Key: SPARK-19599 URL: https://issues.apache.org/jira/browse/SPARK-19599 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-19552) Upgrade Netty version to 4.1.8 final

[ https://issues.apache.org/jira/browse/SPARK-19552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19552. -- Resolution: Later > Upgrade Netty version to 4.1.8 final >

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864850#comment-15864850 ] Shixiong Zhu commented on SPARK-19528: -- The external shuffle service runs inside the node manager.

[jira] [Commented] (SPARK-19528) external shuffle service would close while still have request from executor when dynamic allocation is enabled

[ https://issues.apache.org/jira/browse/SPARK-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864814#comment-15864814 ] Shixiong Zhu commented on SPARK-19528: -- This error is because the executor cannot connect to the

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Attachment: SPARK-19517ProposalforfixingKafkaOffsetMetadata.pdf > KafkaSource fails to

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Attachment: (was: SPARK-19517ProposalforfixingKafkaOffsetMetadata.pdf) > KafkaSource fails

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Attachment: SPARK-19517ProposalforfixingKafkaOffsetMetadata.pdf > KafkaSource fails to

[jira] [Commented] (SPARK-15857) Add Caller Context in Spark

[ https://issues.apache.org/jira/browse/SPARK-15857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15864333#comment-15864333 ] Shixiong Zhu commented on SPARK-15857: -- Can we close this one now? > Add Caller Context in Spark >

[jira] [Updated] (SPARK-17714) ClassCircularityError is thrown when using org.apache.spark.util.Utils.classForName 

[ https://issues.apache.org/jira/browse/SPARK-17714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17714: - Component/s: Spark Core > ClassCircularityError is thrown when using >

[jira] [Resolved] (SPARK-17714) ClassCircularityError is thrown when using org.apache.spark.util.Utils.classForName 

[ https://issues.apache.org/jira/browse/SPARK-17714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17714. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19559) Fix flaky KafkaSourceSuite.subscribing topic by pattern with topic deletions

[ https://issues.apache.org/jira/browse/SPARK-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19559. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19564) KafkaOffsetReader's consumers should not be in the same group

[ https://issues.apache.org/jira/browse/SPARK-19564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19564. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-19559) Fix flaky KafkaSourceSuite.subscribing topic by pattern with topic deletions

[ https://issues.apache.org/jira/browse/SPARK-19559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19559: - Affects Version/s: (was: 2.1.0) 2.2.0 2.1.1 >

[jira] [Created] (SPARK-19542) Delete the temp checkpoint if a query is stopped without errors

Shixiong Zhu created SPARK-19542: Summary: Delete the temp checkpoint if a query is stopped without errors Key: SPARK-19542 URL: https://issues.apache.org/jira/browse/SPARK-19542 Project: Spark

[jira] [Commented] (SPARK-19525) Enable Compression of Spark Streaming Checkpoints

[ https://issues.apache.org/jira/browse/SPARK-19525?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860236#comment-15860236 ] Shixiong Zhu commented on SPARK-19525: -- [~rameshaaditya117] Sounds a good idea. I thought the

[jira] [Commented] (SPARK-19523) Spark streaming+ insert into table leaves bunch of trash in table directory

[ https://issues.apache.org/jira/browse/SPARK-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15860028#comment-15860028 ] Shixiong Zhu commented on SPARK-19523: -- You can create a HiveContext before creating

[jira] [Commented] (SPARK-19523) Spark streaming+ insert into table leaves bunch of trash in table directory

[ https://issues.apache.org/jira/browse/SPARK-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15858716#comment-15858716 ] Shixiong Zhu commented on SPARK-19523: -- These files are temp files created by HiveContext. You can

[jira] [Updated] (SPARK-19523) Spark streaming+ insert into table leaves bunch of trash in table directory

[ https://issues.apache.org/jira/browse/SPARK-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19523: - Component/s: DStreams > Spark streaming+ insert into table leaves bunch of trash in table

[jira] [Updated] (SPARK-19523) Spark streaming+ insert into table leaves bunch of trash in table directory

[ https://issues.apache.org/jira/browse/SPARK-19523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19523: - Component/s: (was: Structured Streaming) > Spark streaming+ insert into table leaves bunch

[jira] [Updated] (SPARK-19524) newFilesOnly does not work according to docs.

[ https://issues.apache.org/jira/browse/SPARK-19524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19524: - Component/s: (was: Structured Streaming) DStreams > newFilesOnly does not

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Target Version/s: 2.1.1, 2.2.0 > KafkaSource fails to initialize partition offsets >

[jira] [Updated] (SPARK-19517) KafkaSource fails to initialize partition offsets

[ https://issues.apache.org/jira/browse/SPARK-19517?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19517: - Priority: Blocker (was: Critical) > KafkaSource fails to initialize partition offsets >

[jira] [Updated] (SPARK-19413) Basic mapGroupsWithState API

[ https://issues.apache.org/jira/browse/SPARK-19413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19413: - Fix Version/s: 2.1.1 > Basic mapGroupsWithState API > > >

[jira] [Resolved] (SPARK-19499) Add more notes in the comments of Sink.addBatch()

[ https://issues.apache.org/jira/browse/SPARK-19499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19499. -- Resolution: Fixed Assignee: Nan Zhu Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19413) Basic mapGroupsWithState API

[ https://issues.apache.org/jira/browse/SPARK-19413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19413. -- Resolution: Fixed Fix Version/s: 2.2.0 > Basic mapGroupsWithState API >

[jira] [Closed] (SPARK-18386) Batch mode SQL source for Kafka

[ https://issues.apache.org/jira/browse/SPARK-18386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu closed SPARK-18386. Resolution: Duplicate > Batch mode SQL source for Kafka > --- > >

[jira] [Resolved] (SPARK-18682) Batch Source for Kafka

[ https://issues.apache.org/jira/browse/SPARK-18682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18682. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Batch Source for

[jira] [Resolved] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

[ https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19407. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0

[jira] [Created] (SPARK-19481) Fix flaky test: o.a.s.repl.ReplSuite should clone and clean line object in ClosureCleaner

Shixiong Zhu created SPARK-19481: Summary: Fix flaky test: o.a.s.repl.ReplSuite should clone and clean line object in ClosureCleaner Key: SPARK-19481 URL: https://issues.apache.org/jira/browse/SPARK-19481

[jira] [Resolved] (SPARK-19437) ExecutorId in HearbeatReceiverSuite is incorrect.

[ https://issues.apache.org/jira/browse/SPARK-19437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19437. -- Resolution: Fixed Assignee: jin xing Fix Version/s: 2.2.0 > ExecutorId in

[jira] [Updated] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

[ https://issues.apache.org/jira/browse/SPARK-19407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19407: - Target Version/s: 2.1.1, 2.2.0 > defaultFS is used FileSystem.get instead of getting it from uri

[jira] [Resolved] (SPARK-19432) Fix an unexpected failure when connecting timeout

[ https://issues.apache.org/jira/browse/SPARK-19432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19432. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Fix an unexpected

[jira] [Created] (SPARK-19432) Fix an unexpected failure when connecting timeout

Shixiong Zhu created SPARK-19432: Summary: Fix an unexpected failure when connecting timeout Key: SPARK-19432 URL: https://issues.apache.org/jira/browse/SPARK-19432 Project: Spark Issue

[jira] [Resolved] (SPARK-19377) Killed tasks should have the status as KILLED

[ https://issues.apache.org/jira/browse/SPARK-19377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19377. -- Resolution: Fixed Assignee: Devaraj K Fix Version/s: 2.2.0

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19414. -- Resolution: Not A Bug > Inferring schema in a structured streaming source >

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847606#comment-15847606 ] Shixiong Zhu commented on SPARK-19414: -- Yes. > Inferring schema in a structured streaming source >

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847598#comment-15847598 ] Shixiong Zhu commented on SPARK-19414: -- [~samelamin] I know little about BigQuery. Does it provide

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847578#comment-15847578 ] Shixiong Zhu commented on SPARK-19414: -- Since you can get the DataFrame, the place to create this

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847574#comment-15847574 ] Shixiong Zhu commented on SPARK-19414: -- Oh, I see, You cannot infer schema when getting the

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847571#comment-15847571 ] Shixiong Zhu commented on SPARK-19414: -- [~samelamin] I would suggest that you take a look at the

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19414: - Issue Type: Question (was: Bug) > Inferring schema in a structured streaming source >

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19414. -- Resolution: Not A Bug > Inferring schema in a structured streaming source >

[jira] (SPARK-19414) Inferring schema in a structured streaming source

[ https://issues.apache.org/jira/browse/SPARK-19414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15847558#comment-15847558 ] Shixiong Zhu commented on SPARK-19414: -- You also need to override

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

Title: Message Title Shixiong Zhu updated an issue

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

Title: Message Title Shixiong Zhu edited a comment on SPARK-19407

[jira] (SPARK-19407) defaultFS is used FileSystem.get instead of getting it from uri scheme

Title: Message Title Shixiong Zhu commented on SPARK-19407

[jira] [Resolved] (SPARK-19365) Optimize RequestMessage serialization

[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19365. -- Resolution: Fixed Fix Version/s: 2.2.0 > Optimize RequestMessage serialization >

[jira] [Updated] (SPARK-19365) Optimize RequestMessage serialization

[ https://issues.apache.org/jira/browse/SPARK-19365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19365: - Description: Right now Netty PRC serializes RequestMessage using Java serialization, and the

[jira] [Created] (SPARK-19365) Optimize RequestMessage serialization

Shixiong Zhu created SPARK-19365: Summary: Optimize RequestMessage serialization Key: SPARK-19365 URL: https://issues.apache.org/jira/browse/SPARK-19365 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-19330) Also show tooltip for successful batches

[ https://issues.apache.org/jira/browse/SPARK-19330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19330. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19139) AES-based authentication mechanism for Spark

[ https://issues.apache.org/jira/browse/SPARK-19139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19139. -- Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 2.2.0 > AES-based

[jira] [Resolved] (SPARK-10651) Flaky test: BroadcastSuite

[ https://issues.apache.org/jira/browse/SPARK-10651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-10651. -- Resolution: Fixed Fix Version/s: 2.2.0 The root cause of the failure is SPARK-17755,

[jira] [Resolved] (SPARK-19300) Executor is waiting for lock

[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19300. -- Resolution: Duplicate Thanks for confirming it. Closing this one as it's a duplicated issue.

[jira] [Commented] (SPARK-19300) Executor is waiting for lock

[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833244#comment-15833244 ] Shixiong Zhu commented on SPARK-19300: -- Could you check if there is any thread having the similar

[jira] [Commented] (SPARK-19300) Executor is waiting for lock

[ https://issues.apache.org/jira/browse/SPARK-19300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15832658#comment-15832658 ] Shixiong Zhu commented on SPARK-19300: -- Could you provide the full thread dump? Looks like there is

[jira] [Commented] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15831018#comment-15831018 ] Shixiong Zhu commented on SPARK-19280: -- Good catch and nice explanation. I think maybe 2) is the

[jira] [Updated] (SPARK-19280) Failed Recovery from checkpoint caused by the multi-threads issue in Spark Streaming scheduler

[ https://issues.apache.org/jira/browse/SPARK-19280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19280: - Priority: Critical (was: Major) > Failed Recovery from checkpoint caused by the multi-threads

[jira] [Commented] (SPARK-19233) Inconsistent Behaviour of Spark Streaming Checkpoint

[ https://issues.apache.org/jira/browse/SPARK-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830989#comment-15830989 ] Shixiong Zhu commented on SPARK-19233: -- I don't think filtering "generatedRDDs" will work.

[jira] [Commented] (SPARK-19275) Spark Streaming, Kafka receiver, "Failed to get records for ... after polling for 512"

[ https://issues.apache.org/jira/browse/SPARK-19275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830976#comment-15830976 ] Shixiong Zhu commented on SPARK-19275: -- This error usually means Spark cannot fetch records from

[jira] [Commented] (SPARK-19268) File does not exist: /tmp/temporary-157b89c1-27bb-49f3-a70c-ca1b75022b4d/state/0/2/1.delta

[ https://issues.apache.org/jira/browse/SPARK-19268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15830706#comment-15830706 ] Shixiong Zhu commented on SPARK-19268: -- Right now Structured Streaming doesn't support

[jira] [Updated] (SPARK-19268) File does not exist: /tmp/temporary-157b89c1-27bb-49f3-a70c-ca1b75022b4d/state/0/2/1.delta

[ https://issues.apache.org/jira/browse/SPARK-19268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19268: - Priority: Critical (was: Major) > File does not exist: >

[jira] [Resolved] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block UI when generating Streaming jobs

[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19182. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.2.0 > Optimize the lock

[jira] [Resolved] (SPARK-19168) StateStore should be aborted upon error

[ https://issues.apache.org/jira/browse/SPARK-19168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19168. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.2.0

[jira] [Resolved] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19113. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Fix flaky test:

[jira] [Updated] (SPARK-19267) Fix a race condition when stopping StateStore

[ https://issues.apache.org/jira/browse/SPARK-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19267: - Priority: Minor (was: Major) > Fix a race condition when stopping StateStore >

[jira] [Updated] (SPARK-19267) Fix a race condition when stopping StateStore

[ https://issues.apache.org/jira/browse/SPARK-19267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19267: - Affects Version/s: 2.0.0 2.0.1 2.0.2

[jira] [Created] (SPARK-19267) Fix a race condition when stopping StateStore

Shixiong Zhu created SPARK-19267: Summary: Fix a race condition when stopping StateStore Key: SPARK-19267 URL: https://issues.apache.org/jira/browse/SPARK-19267 Project: Spark Issue Type:

[jira] [Updated] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

[ https://issues.apache.org/jira/browse/SPARK-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18907: - Component/s: Tests Structured Streaming > Fix flaky test:

[jira] [Commented] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

[ https://issues.apache.org/jira/browse/SPARK-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826923#comment-15826923 ] Shixiong Zhu commented on SPARK-18907: -- [~iamshrek] Thanks for looking at it. I forgot to close this

[jira] [Resolved] (SPARK-18907) Fix flaky test: o.a.s.sql.streaming.FileStreamSourceSuite max files per trigger - incorrect values

[ https://issues.apache.org/jira/browse/SPARK-18907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18907. -- Resolution: Duplicate Closing it as this one actually is caused by SPARK-18908. > Fix flaky

[jira] [Commented] (SPARK-17866) Dataset.dropDuplicates (i.e., distinct) should not change the output of child plan

[ https://issues.apache.org/jira/browse/SPARK-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15826579#comment-15826579 ] Shixiong Zhu commented on SPARK-17866: -- Although this one was fixed in 2.1.0, it broke SPARK-19065.

[jira] [Resolved] (SPARK-17866) Dataset.dropDuplicates (i.e., distinct) should not change the output of child plan

[ https://issues.apache.org/jira/browse/SPARK-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17866. -- Resolution: Won't Fix > Dataset.dropDuplicates (i.e., distinct) should not change the output

[jira] [Reopened] (SPARK-17866) Dataset.dropDuplicates (i.e., distinct) should not change the output of child plan

[ https://issues.apache.org/jira/browse/SPARK-17866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-17866: -- > Dataset.dropDuplicates (i.e., distinct) should not change the output of child > plan >

[jira] [Updated] (SPARK-19065) dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute replacement

[ https://issues.apache.org/jira/browse/SPARK-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19065: - Description: Right now if you use .dropDuplicates in a stream you get an exception because

[jira] [Updated] (SPARK-19065) dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute replacement

[ https://issues.apache.org/jira/browse/SPARK-19065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19065: - Summary: dropDuplicates uses the same expression id for Alias and Attribute and breaks attribute

[jira] [Updated] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18905: - Affects Version/s: 1.5.2 1.6.1 1.6.2 > Potential

[jira] [Updated] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18905: - Affects Version/s: 1.5.1 1.6.3 > Potential Issue of Semantics of

[jira] [Resolved] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18905. -- Resolution: Fixed Assignee: Nan Zhu Fix Version/s: 2.2.0

[jira] [Updated] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19113: - Fix Version/s: (was: 2.1.1) (was: 2.2.0) > Fix flaky test:

[jira] [Reopened] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-19113: -- Reopened it as it's still flaky > Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors

[jira] [Updated] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block UI when generating Streaming jobs

[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19182: - Summary: Optimize the lock in StreamingJobProgressListener to not block UI when generating

[jira] [Updated] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs

[ https://issues.apache.org/jira/browse/SPARK-19182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19182: - Description: When DStreamGraph is generating a job, it will hold a lock and block other APIs.

[jira] [Created] (SPARK-19182) Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs

Shixiong Zhu created SPARK-19182: Summary: Optimize the lock in StreamingJobProgressListener to not block when generating Streaming jobs Key: SPARK-19182 URL: https://issues.apache.org/jira/browse/SPARK-19182

[jira] [Resolved] (SPARK-19140) Allow update mode for non-aggregation streaming queries

[ https://issues.apache.org/jira/browse/SPARK-19140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19140. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Allow update mode for

[jira] [Updated] (SPARK-19102) Accuracy error of spark SQL results

[ https://issues.apache.org/jira/browse/SPARK-19102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19102: - Component/s: (was: Spark Core) > Accuracy error of spark SQL results >

[jira] [Commented] (SPARK-19147) netty throw NPE

[ https://issues.apache.org/jira/browse/SPARK-19147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15816170#comment-15816170 ] Shixiong Zhu commented on SPARK-19147: -- I think there is a race condition when the executor is

[jira] [Updated] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

[ https://issues.apache.org/jira/browse/SPARK-19113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-19113: - Fix Version/s: 2.1.1 > Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a

[jira] [Resolved] (SPARK-19137) Garbage left in source tree after SQL tests are run

[ https://issues.apache.org/jira/browse/SPARK-19137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-19137. -- Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.2.0

[jira] [Commented] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815815#comment-15815815 ] Shixiong Zhu commented on SPARK-18905: -- Sure. Please go ahead. > Potential Issue of Semantics of

[jira] [Commented] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813455#comment-15813455 ] Shixiong Zhu commented on SPARK-18905: -- [~CodingCat] I think `pendingTime` is the jobs that have

[jira] [Commented] (SPARK-18905) Potential Issue of Semantics of BatchCompleted

[ https://issues.apache.org/jira/browse/SPARK-18905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813405#comment-15813405 ] Shixiong Zhu commented on SPARK-18905: -- Sorry for the late reply. Yeah, good catch. However, even if

[jira] [Commented] (SPARK-17463) Serialization of accumulators in heartbeats is not thread-safe

[ https://issues.apache.org/jira/browse/SPARK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15813205#comment-15813205 ] Shixiong Zhu commented on SPARK-17463: -- [~sunil.rangwani] could you have a simple reproducer? I ran

[jira] [Created] (SPARK-19140) Allow update mode for non-aggregation streaming queries

Shixiong Zhu created SPARK-19140: Summary: Allow update mode for non-aggregation streaming queries Key: SPARK-19140 URL: https://issues.apache.org/jira/browse/SPARK-19140 Project: Spark

[jira] [Created] (SPARK-19113) Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user

Shixiong Zhu created SPARK-19113: Summary: Fix flaky test: o.a.s.sql.streaming.StreamSuite fatal errors from a source should be sent to the user Key: SPARK-19113 URL:

[jira] [Commented] (SPARK-19013) java.util.ConcurrentModificationException when using s3 path as checkpointLocation

[ https://issues.apache.org/jira/browse/SPARK-19013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15799931#comment-15799931 ] Shixiong Zhu commented on SPARK-19013: -- Thanks, [~zzztimbo] That must be caused by the negative

<    6   7   8   9   10   11   12   13   14   15   >