[jira] [Updated] (SPARK-18751) Deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext

2016-12-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18751: - Affects Version/s: 2.1.0 > Deadlock when SparkContext.stop is called in

[jira] [Resolved] (SPARK-18751) Deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext

2016-12-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18751. -- Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Deadlock when

[jira] [Created] (SPARK-18774) Ignore non-existing files when ignoreCorruptFiles is enabled

2016-12-07 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18774: Summary: Ignore non-existing files when ignoreCorruptFiles is enabled Key: SPARK-18774 URL: https://issues.apache.org/jira/browse/SPARK-18774 Project: Spark

[jira] [Resolved] (SPARK-18764) Add a warning log when skipping a corrupted file

2016-12-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18764. -- Resolution: Fixed Fix Version/s: 2.1.0 > Add a warning log when skipping a corrupted

[jira] [Updated] (SPARK-18764) Add a warning log when skipping a corrupted file

2016-12-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18764: - Priority: Minor (was: Major) > Add a warning log when skipping a corrupted file >

[jira] [Updated] (SPARK-18764) Add a warning log when skipping a corrupted file

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18764: - Affects Version/s: 2.1.0 > Add a warning log when skipping a corrupted file >

[jira] [Created] (SPARK-18764) Add a warning log when skipping a corrupted file

2016-12-06 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18764: Summary: Add a warning log when skipping a corrupted file Key: SPARK-18764 URL: https://issues.apache.org/jira/browse/SPARK-18764 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-18734) Represent timestamp in StreamingQueryProgress as formatted string instead of millis

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18734. -- Resolution: Fixed Fix Version/s: 2.1.0 > Represent timestamp in StreamingQueryProgress

[jira] [Commented] (SPARK-18753) Inconsistent behavior after writing to parquet files

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15727199#comment-15727199 ] Shixiong Zhu commented on SPARK-18753: -- cc [~liancheng] > Inconsistent behavior after writing to

[jira] [Updated] (SPARK-18753) Inconsistent behavior after writing to parquet files

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18753: - Description: Found an inconsistent behavior when using parquet. {code} scala> val ds =

[jira] [Updated] (SPARK-18753) Inconsistent behavior after writing to parquet files

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18753: - Description: Found an inconsistent behavior when using parquet. {code} scala> val ds =

[jira] [Updated] (SPARK-18753) Inconsistent behavior after writing to parquet files

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18753?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18753: - Description: Found an inconsistent behavior when using parquet. {code} scala> val ds =

[jira] [Created] (SPARK-18753) Inconsistent behavior after writing to parquet files

2016-12-06 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18753: Summary: Inconsistent behavior after writing to parquet files Key: SPARK-18753 URL: https://issues.apache.org/jira/browse/SPARK-18753 Project: Spark Issue

[jira] [Created] (SPARK-18751) Deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext

2016-12-06 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18751: Summary: Deadlock when SparkContext.stop is called in Utils.tryOrStopSparkContext Key: SPARK-18751 URL: https://issues.apache.org/jira/browse/SPARK-18751 Project:

[jira] [Updated] (SPARK-18744) Remove workaround for Netty memory leak

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18744: - Affects Version/s: 2.1.0 > Remove workaround for Netty memory leak >

[jira] [Resolved] (SPARK-18744) Remove workaround for Netty memory leak

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18744. -- Resolution: Fixed Fix Version/s: 2.2.0 > Remove workaround for Netty memory leak >

[jira] [Resolved] (SPARK-18671) Add tests to ensure stability of that all Structured Streaming log formats

2016-12-06 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18671. -- Resolution: Fixed Assignee: Tathagata Das Fix Version/s: 2.1.0 > Add tests to

[jira] [Created] (SPARK-18744) Remove workaround for Netty memory leak

2016-12-06 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18744: Summary: Remove workaround for Netty memory leak Key: SPARK-18744 URL: https://issues.apache.org/jira/browse/SPARK-18744 Project: Spark Issue Type:

[jira] [Created] (SPARK-18729) MemorySink should not call DataFrame.collect when holding a lock

2016-12-05 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18729: Summary: MemorySink should not call DataFrame.collect when holding a lock Key: SPARK-18729 URL: https://issues.apache.org/jira/browse/SPARK-18729 Project: Spark

[jira] [Updated] (SPARK-18729) MemorySink should not call DataFrame.collect when holding a lock

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18729: - Issue Type: Improvement (was: Bug) > MemorySink should not call DataFrame.collect when holding

[jira] [Updated] (SPARK-18721) ForeachSink breaks Watermark in append mode

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18721: - Target Version/s: 2.1.0 > ForeachSink breaks Watermark in append mode >

[jira] [Resolved] (SPARK-18694) Add StreamingQuery.explain and exception to Python and fix StreamingQueryException

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18694. -- Resolution: Fixed Fix Version/s: 2.1.0 > Add StreamingQuery.explain and exception to

[jira] [Assigned] (SPARK-18721) ForeachSink breaks Watermark in append mode

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18721: Assignee: Shixiong Zhu > ForeachSink breaks Watermark in append mode >

[jira] [Created] (SPARK-18722) Move no data rate limit from StreamExecution to ProgressReporter

2016-12-05 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18722: Summary: Move no data rate limit from StreamExecution to ProgressReporter Key: SPARK-18722 URL: https://issues.apache.org/jira/browse/SPARK-18722 Project: Spark

[jira] [Commented] (SPARK-18142) Spark Master tries to launch workers 145 times within 1 minute

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723252#comment-15723252 ] Shixiong Zhu commented on SPARK-18142: -- Looks like we need a blacklist mechanism for workers. >

[jira] [Updated] (SPARK-18470) Provide Spark Streaming Monitor Rest Api

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18470: - Component/s: (was: Structured Streaming) DStreams > Provide Spark Streaming

[jira] [Reopened] (SPARK-18560) Receiver data can not be dataSerialized properly.

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-18560: -- > Receiver data can not be dataSerialized properly. >

[jira] [Updated] (SPARK-18560) Receiver data can not be dataSerialized properly.

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18560: - Assignee: (was: Genmao Yu) > Receiver data can not be dataSerialized properly. >

[jira] [Resolved] (SPARK-18560) Receiver data can not be dataSerialized properly.

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18560. -- Resolution: Duplicate > Receiver data can not be dataSerialized properly. >

[jira] [Updated] (SPARK-18560) Receiver data can not be dataSerialized properly.

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18560: - Component/s: (was: Spark Core) DStreams > Receiver data can not be

[jira] [Resolved] (SPARK-18560) Receiver data can not be dataSerialized properly.

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18560. -- Resolution: Fixed Assignee: Genmao Yu Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-18709) Automatic null conversion bug (instead of throwing error) when creating a Spark Datarame with incompatible types for fields.

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15723100#comment-15723100 ] Shixiong Zhu commented on SPARK-18709: -- [~dongjoon] Is it already resolved? If so, could you close

[jira] [Updated] (SPARK-18709) Automatic null conversion bug (instead of throwing error) when creating a Spark Datarame with incompatible types for fields.

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18709: - Component/s: (was: Spark Core) SQL > Automatic null conversion bug (instead

[jira] [Resolved] (SPARK-18470) Provide Spark Streaming Monitor Rest Api

2016-12-05 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18470. -- Resolution: Duplicate > Provide Spark Streaming Monitor Rest Api >

[jira] [Updated] (SPARK-18694) Add StreamingQuery.explain and exception to Python and fix StreamingQueryException

2016-12-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18694?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18694: - Priority: Blocker (was: Major) > Add StreamingQuery.explain and exception to Python and fix >

[jira] [Created] (SPARK-18694) Add StreamingQuery.explain and exception to Python and fix StreamingQueryException

2016-12-02 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18694: Summary: Add StreamingQuery.explain and exception to Python and fix StreamingQueryException Key: SPARK-18694 URL: https://issues.apache.org/jira/browse/SPARK-18694

[jira] [Commented] (SPARK-13287) Standalone REST API throttling?

2016-12-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713742#comment-15713742 ] Shixiong Zhu commented on SPARK-13287: -- Created SPARK-18683. But I don't have time to work on it

[jira] [Created] (SPARK-18683) REST APIs for standalone Master and Workers

2016-12-01 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18683: Summary: REST APIs for standalone Master and Workers Key: SPARK-18683 URL: https://issues.apache.org/jira/browse/SPARK-18683 Project: Spark Issue Type:

[jira] [Commented] (SPARK-13287) Standalone REST API throttling?

2016-12-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15713735#comment-15713735 ] Shixiong Zhu commented on SPARK-13287: -- Right now there is no REST API for master. You use the REST

[jira] [Resolved] (SPARK-13287) Standalone REST API throttling?

2016-12-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-13287. -- Resolution: Not A Bug > Standalone REST API throttling? > --- > >

[jira] [Resolved] (SPARK-18655) Ignore Structured Streaming 2.0.2 logs in history server

2016-11-30 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18655. -- Resolution: Fixed Fix Version/s: 2.1.0 > Ignore Structured Streaming 2.0.2 logs in

[jira] [Updated] (SPARK-18563) mapWithState: initialState should have a timeout setting per record

2016-11-30 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18563: - Component/s: (was: Structured Streaming) DStreams > mapWithState:

[jira] [Updated] (SPARK-18655) Ignore Structured Streaming 2.0.2 logs in history server

2016-11-30 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18655: - Fix Version/s: 2.1.0 > Ignore Structured Streaming 2.0.2 logs in history server >

[jira] [Updated] (SPARK-18655) Ignore Structured Streaming 2.0.2 logs in history server

2016-11-30 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18655: - Priority: Blocker (was: Major) > Ignore Structured Streaming 2.0.2 logs in history server >

[jira] [Created] (SPARK-18655) Ignore Structured Streaming 2.0.2 logs in history server

2016-11-30 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18655: Summary: Ignore Structured Streaming 2.0.2 logs in history server Key: SPARK-18655 URL: https://issues.apache.org/jira/browse/SPARK-18655 Project: Spark

[jira] [Resolved] (SPARK-18188) Add checksum for block of broadcast

2016-11-29 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18188. -- Resolution: Fixed Fix Version/s: 2.1.0 > Add checksum for block of broadcast >

[jira] [Resolved] (SPARK-18547) Decouple I/O encryption key propagation from UserGroupInformation

2016-11-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18547. -- Resolution: Fixed Fix Version/s: 2.1.0 > Decouple I/O encryption key propagation from

[jira] [Assigned] (SPARK-18588) KafkaSourceStressForDontFailOnDataLossSuite is flaky

2016-11-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18588: Assignee: Shixiong Zhu > KafkaSourceStressForDontFailOnDataLossSuite is flaky >

[jira] [Commented] (SPARK-18552) Watermark should not rely on sinks to proceed

2016-11-23 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15691098#comment-15691098 ] Shixiong Zhu commented on SPARK-18552: -- As per the current doc "the actual watermark used is only

[jira] [Resolved] (SPARK-18530) Kafka timestamp should be TimestampType

2016-11-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18530. -- Resolution: Fixed Fix Version/s: 2.1.0 > Kafka timestamp should be TimestampType >

[jira] [Updated] (SPARK-18513) Record and recover watermark

2016-11-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18513: - Assignee: Tyson Condie > Record and recover watermark > > >

[jira] [Comment Edited] (SPARK-14146) Imported implicits can't be found in Spark REPL in some cases

2016-11-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687597#comment-15687597 ] Shixiong Zhu edited comment on SPARK-14146 at 11/22/16 7:10 PM: Hm...,

[jira] [Commented] (SPARK-14146) Imported implicits can't be found in Spark REPL in some cases

2016-11-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687597#comment-15687597 ] Shixiong Zhu commented on SPARK-14146: -- Hm..., this issue seems worse than I thought. It will ignore

[jira] [Assigned] (SPARK-18529) Timeouts shouldn't be AssertionErrors

2016-11-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18529: Assignee: Shixiong Zhu > Timeouts shouldn't be AssertionErrors >

[jira] [Resolved] (SPARK-18425) Test `CompactibleFileStreamLog` directly

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18425. -- Resolution: Fixed Assignee: Liwei Lin Fix Version/s: 2.1.0 > Test

[jira] [Resolved] (SPARK-18493) Add withWatermark and checkpoint to python dataframe

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18493. -- Resolution: Fixed Assignee: Burak Yavuz Fix Version/s: 2.1.0 > Add

[jira] [Commented] (SPARK-18512) FileNotFoundException on _temporary directory with Spark Streaming 2.0.1 and S3A

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684863#comment-15684863 ] Shixiong Zhu commented on SPARK-18512: -- Did you enable speculation? > FileNotFoundException on

[jira] [Assigned] (SPARK-18530) Kafka timestamp should be TimestampType

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18530: Assignee: Shixiong Zhu > Kafka timestamp should be TimestampType >

[jira] [Commented] (SPARK-18529) Timeouts shouldn't be AssertionErrors

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684703#comment-15684703 ] Shixiong Zhu commented on SPARK-18529: -- This will be fixed in

[jira] [Commented] (SPARK-17850) HadoopRDD should not swallow EOFException

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15684699#comment-15684699 ] Shixiong Zhu commented on SPARK-17850: -- [~mgrover] you're right. This is only in 2.1. Removed 2.0.2.

[jira] [Updated] (SPARK-17850) HadoopRDD should not swallow EOFException

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17850: - Fix Version/s: (was: 2.0.2) > HadoopRDD should not swallow EOFException >

[jira] [Updated] (SPARK-17850) HadoopRDD should not swallow EOFException

2016-11-21 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17850: - Target Version/s: 2.1.0 (was: 2.0.2, 2.1.0) > HadoopRDD should not swallow EOFException >

[jira] [Commented] (SPARK-17755) Master may ask a worker to launch an executor before the worker actually got the response of registration

2016-11-20 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15682778#comment-15682778 ] Shixiong Zhu commented on SPARK-17755: -- It's not easy to fix. The root cause is messages are sent

[jira] [Assigned] (SPARK-18497) ForeachSink fails with "assertion failed: No plan for EventTimeWatermark"

2016-11-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18497: Assignee: Shixiong Zhu > ForeachSink fails with "assertion failed: No plan for

[jira] [Resolved] (SPARK-18187) CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch

2016-11-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18187. -- Resolution: Fixed Assignee: Tyson Condie Fix Version/s: 2.1.0 >

[jira] [Commented] (SPARK-18187) CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch

2016-11-18 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677492#comment-15677492 ] Shixiong Zhu commented on SPARK-18187: -- Resolved by https://github.com/apache/spark/pull/15852 >

[jira] [Created] (SPARK-18477) Enable interrupts for HDFS in HDFSMetadataLog

2016-11-16 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18477: Summary: Enable interrupts for HDFS in HDFSMetadataLog Key: SPARK-18477 URL: https://issues.apache.org/jira/browse/SPARK-18477 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-18459) Rename triggerId to batchId in StreamingQueryStatus.triggerDetails

2016-11-16 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18459. -- Resolution: Fixed > Rename triggerId to batchId in StreamingQueryStatus.triggerDetails >

[jira] [Resolved] (SPARK-18423) ReceiverTracker should close checkpoint dir when stopped even if it was not started

2016-11-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18423. -- Resolution: Fixed Assignee: Hyukjin Kwon Fix Version/s: 2.1.0 >

[jira] [Assigned] (SPARK-13027) Add API for updateStateByKey to provide batch time as input

2016-11-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-13027: Assignee: Shixiong Zhu > Add API for updateStateByKey to provide batch time as input >

[jira] [Updated] (SPARK-13027) Add API for updateStateByKey to provide batch time as input

2016-11-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-13027: - Assignee: Aaditya Ramesh (was: Shixiong Zhu) > Add API for updateStateByKey to provide batch

[jira] [Resolved] (SPARK-13027) Add API for updateStateByKey to provide batch time as input

2016-11-15 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-13027. -- Resolution: Fixed Fix Version/s: 2.1.0 > Add API for updateStateByKey to provide batch

[jira] [Resolved] (SPARK-17510) Set Streaming MaxRate Independently For Multiple Streams

2016-11-14 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17510. -- Resolution: Fixed Assignee: Cody Koeninger Fix Version/s: 2.1.0 > Set

[jira] [Resolved] (SPARK-18416) State Store leaks temporary files

2016-11-14 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18416. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.3 > State Store leaks

[jira] [Resolved] (SPARK-17829) Stable format for offset log

2016-11-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-17829. -- Resolution: Fixed > Stable format for offset log > > >

[jira] [Updated] (SPARK-17829) Stable format for offset log

2016-11-09 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-17829: - Fix Version/s: 2.1.0 > Stable format for offset log > > >

[jira] [Created] (SPARK-18373) Make KafkaSource's failOnDataLoss=false work with Spark jobs

2016-11-08 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18373: Summary: Make KafkaSource's failOnDataLoss=false work with Spark jobs Key: SPARK-18373 URL: https://issues.apache.org/jira/browse/SPARK-18373 Project: Spark

[jira] [Resolved] (SPARK-18280) Potential deadlock in `StandaloneSchedulerBackend.dead`

2016-11-08 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18280. -- Resolution: Fixed Assignee: Shixiong Zhu Fix Version/s: 2.1.0

[jira] [Commented] (SPARK-18335) Add a numSlices parameter to SparkR's createDataFrame

2016-11-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645503#comment-15645503 ] Shixiong Zhu commented on SPARK-18335: -- cc [~falaki] > Add a numSlices parameter to SparkR's

[jira] [Created] (SPARK-18335) Add a numSlices parameter to SparkR's createDataFrame

2016-11-07 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18335: Summary: Add a numSlices parameter to SparkR's createDataFrame Key: SPARK-18335 URL: https://issues.apache.org/jira/browse/SPARK-18335 Project: Spark Issue

[jira] [Commented] (SPARK-18272) Test topic addition for subscribePattern on Kafka DStream and Structured Stream

2016-11-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645460#comment-15645460 ] Shixiong Zhu commented on SPARK-18272: -- KafkaSourceStressSuite covers this case. However, I agree we

[jira] [Resolved] (SPARK-18283) Add a test to make sure the default starting offset is latest

2016-11-07 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18283. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.3 > Add a test to make

[jira] [Created] (SPARK-18280) Potential deadlock in `StandaloneSchedulerBackend.dead`

2016-11-04 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18280: Summary: Potential deadlock in `StandaloneSchedulerBackend.dead` Key: SPARK-18280 URL: https://issues.apache.org/jira/browse/SPARK-18280 Project: Spark

[jira] [Updated] (SPARK-18280) Potential deadlock in `StandaloneSchedulerBackend.dead`

2016-11-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18280: - Affects Version/s: 1.6.2 2.0.0 2.0.1 > Potential

[jira] [Updated] (SPARK-18260) from_json can throw a better exception when it can't find the column or be nullSafe

2016-11-03 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18260: - Component/s: SQL > from_json can throw a better exception when it can't find the column or be >

[jira] [Commented] (SPARK-18212) Flaky test: org.apache.spark.sql.kafka010.KafkaSourceSuite.assign from specific offsets

2016-11-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15630567#comment-15630567 ] Shixiong Zhu commented on SPARK-18212: -- [~c...@koeninger.org] Sounds good to me. > Flaky test:

[jira] [Resolved] (SPARK-18144) StreamingQueryListener.QueryStartedEvent is not written to event log

2016-11-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18144. -- Resolution: Fixed Assignee: Nan Zhu Fix Version/s: 2.1.0

[jira] [Updated] (SPARK-18144) StreamingQueryListener.QueryStartedEvent is not written to event log

2016-11-02 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18144: - Affects Version/s: 2.0.2 > StreamingQueryListener.QueryStartedEvent is not written to event log

[jira] [Commented] (SPARK-18187) CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch

2016-11-01 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627104#comment-15627104 ] Shixiong Zhu commented on SPARK-18187: -- [~lwlin] I don't have a sold idea. Feel free to propose your

[jira] [Created] (SPARK-18187) CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch

2016-10-31 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18187: Summary: CompactibleFileStreamLog should not rely on "compactInterval" to detect a compaction batch Key: SPARK-18187 URL: https://issues.apache.org/jira/browse/SPARK-18187

[jira] [Resolved] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

2016-10-31 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18030. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.2 > Flaky test:

[jira] [Assigned] (SPARK-18030) Flaky test: org.apache.spark.sql.streaming.FileStreamSourceSuite

2016-10-31 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-18030: Assignee: Shixiong Zhu (was: Tathagata Das) > Flaky test:

[jira] [Updated] (SPARK-18164) ForeachSink should fail the Spark job if `process` throws exception

2016-10-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18164: - Issue Type: Sub-task (was: Bug) Parent: SPARK-8360 > ForeachSink should fail the Spark

[jira] [Resolved] (SPARK-18164) ForeachSink should fail the Spark job if `process` throws exception

2016-10-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-18164. -- Resolution: Fixed Fix Version/s: 2.1.0 2.0.3 > ForeachSink should

[jira] [Updated] (SPARK-18144) StreamingQueryListener.QueryStartedEvent is not written to event log

2016-10-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18144: - Priority: Minor (was: Major) > StreamingQueryListener.QueryStartedEvent is not written to event

[jira] [Updated] (SPARK-18143) History Server is broken because of the refactoring work in Structured Streaming

2016-10-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18143: - Priority: Blocker (was: Major) > History Server is broken because of the refactoring work in

[jira] [Created] (SPARK-18164) ForeachSink should fail the Spark job if `process` throws exception

2016-10-28 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18164: Summary: ForeachSink should fail the Spark job if `process` throws exception Key: SPARK-18164 URL: https://issues.apache.org/jira/browse/SPARK-18164 Project: Spark

[jira] [Updated] (SPARK-18144) StreamingQueryListener.QueryStartedEvent is not written to event log

2016-10-27 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18144: - Affects Version/s: 2.0.0 2.0.1 > StreamingQueryListener.QueryStartedEvent

[jira] [Created] (SPARK-18144) StreamingQueryListener.QueryStartedEvent is not written to event log

2016-10-27 Thread Shixiong Zhu (JIRA)
Shixiong Zhu created SPARK-18144: Summary: StreamingQueryListener.QueryStartedEvent is not written to event log Key: SPARK-18144 URL: https://issues.apache.org/jira/browse/SPARK-18144 Project: Spark

[jira] [Updated] (SPARK-18143) History Server is broken because of the refactoring work in Structured Streaming

2016-10-27 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-18143: - Issue Type: Sub-task (was: Bug) Parent: SPARK-8360 > History Server is broken because

<    8   9   10   11   12   13   14   15   16   17   >