[jira] [Created] (SPARK-21977) SinglePartition optimizations break certain StateStore requirements

2017-09-11 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21977: --- Summary: SinglePartition optimizations break certain StateStore requirements Key: SPARK-21977 URL: https://issues.apache.org/jira/browse/SPARK-21977 Project: Spark

[jira] [Assigned] (SPARK-21977) SinglePartition optimizations break certain StateStore requirements

2017-09-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-21977: --- Assignee: Burak Yavuz > SinglePartition optimizations break certain StateStore requirements

[jira] [Updated] (SPARK-21977) SinglePartition optimizations break certain StateStore requirements

2017-09-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21977: Description: This is a bit hard to explain as there are several issues here > SinglePartition

[jira] [Created] (SPARK-21925) Update trigger interval documentation in docs with behavior change in Spark 2.2

2017-09-05 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21925: --- Summary: Update trigger interval documentation in docs with behavior change in Spark 2.2 Key: SPARK-21925 URL: https://issues.apache.org/jira/browse/SPARK-21925

[jira] [Updated] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-08 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21590: Issue Type: Improvement (was: Bug) > Structured Streaming window start time should support

[jira] [Commented] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-08 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16118560#comment-16118560 ] Burak Yavuz commented on SPARK-21590: - Ah, I apologize, I thought the other way around (you had data

[jira] [Commented] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-06 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16115998#comment-16115998 ] Burak Yavuz commented on SPARK-21590: - [~KevinZwx] there's nothing wrong. It works as it needs to

[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE

2017-08-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113519#comment-16113519 ] Burak Yavuz commented on SPARK-21588: - that's what I was proposing. `null` seemed more familiar than

[jira] [Commented] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE

2017-08-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16113422#comment-16113422 ] Burak Yavuz commented on SPARK-21588: - [~vinodkc] [~aokolnychyi] It happens when the config has a

[jira] [Commented] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-02 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111460#comment-16111460 ] Burak Yavuz commented on SPARK-21590: - There are tests to make sure it supports "negative" timestamps

[jira] [Commented] (SPARK-21590) Structured Streaming window start time should support negative values to adjust time zone

2017-08-02 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16111458#comment-16111458 ] Burak Yavuz commented on SPARK-21590: - Does it output incorrect results if you provide `+16 hours`

[jira] [Created] (SPARK-21588) SQLContext.getConf(key, null) should return null, but it throws NPE

2017-07-31 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21588: --- Summary: SQLContext.getConf(key, null) should return null, but it throws NPE Key: SPARK-21588 URL: https://issues.apache.org/jira/browse/SPARK-21588 Project: Spark

[jira] [Assigned] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-18 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-21463: --- Assignee: Burak Yavuz > Output of StructuredStreaming tables don't respect user specified

[jira] [Created] (SPARK-21463) Output of StructuredStreaming tables don't respect user specified schema when reading back the table

2017-07-18 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21463: --- Summary: Output of StructuredStreaming tables don't respect user specified schema when reading back the table Key: SPARK-21463 URL:

[jira] [Assigned] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-21445: --- Assignee: Burak Yavuz > NotSerializableException thrown by UTF8String.IntWrapper >

[jira] [Created] (SPARK-21445) NotSerializableException thrown by UTF8String.IntWrapper

2017-07-17 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21445: --- Summary: NotSerializableException thrown by UTF8String.IntWrapper Key: SPARK-21445 URL: https://issues.apache.org/jira/browse/SPARK-21445 Project: Spark Issue

[jira] [Updated] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21370: Description: During Streaming Aggregation, we have two StateStores per task, one used as

[jira] [Reopened] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reopened SPARK-21370: - > Avoid doing anything on HDFSBackedStateStore.abort() when there are no > updates to commit >

[jira] [Updated] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-21370: Issue Type: Test (was: Improvement) > Avoid doing anything on HDFSBackedStateStore.abort() when

[jira] [Closed] (SPARK-21370) Avoid doing anything on HDFSBackedStateStore.abort() when there are no updates to commit

2017-07-10 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz closed SPARK-21370. --- Resolution: Not A Problem > Avoid doing anything on HDFSBackedStateStore.abort() when there are no

[jira] [Created] (SPARK-21370) Clarify In-Memory State Store purpose (read-only, read-write) with an additional state

2017-07-10 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21370: --- Summary: Clarify In-Memory State Store purpose (read-only, read-write) with an additional state Key: SPARK-21370 URL: https://issues.apache.org/jira/browse/SPARK-21370

[jira] [Created] (SPARK-21216) Streaming DataFrames fail to join with Hive tables

2017-06-26 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21216: --- Summary: Streaming DataFrames fail to join with Hive tables Key: SPARK-21216 URL: https://issues.apache.org/jira/browse/SPARK-21216 Project: Spark Issue Type:

[jira] [Created] (SPARK-21153) Time windowing for tumbling windows can use a project instead of expand + filter

2017-06-20 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-21153: --- Summary: Time windowing for tumbling windows can use a project instead of expand + filter Key: SPARK-21153 URL: https://issues.apache.org/jira/browse/SPARK-21153

[jira] [Resolved] (SPARK-20708) Make `addExclusionRules` up-to-date

2017-05-31 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-20708. - Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.3.0 > Make

[jira] [Commented] (SPARK-20708) Make `addExclusionRules` up-to-date

2017-05-31 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16032475#comment-16032475 ] Burak Yavuz commented on SPARK-20708: - Resolved by https://github.com/apache/spark/pull/17947 > Make

[jira] [Resolved] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries

2017-05-16 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-20140. - Resolution: Fixed Fix Version/s: 2.3.0 2.2.1 > Remove hardcoded

[jira] [Assigned] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries

2017-05-16 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-20140: --- Assignee: Yash Sharma > Remove hardcoded kinesis retry wait and max retries >

[jira] [Commented] (SPARK-20140) Remove hardcoded kinesis retry wait and max retries

2017-05-16 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16013191#comment-16013191 ] Burak Yavuz commented on SPARK-20140: - resolved by https://github.com/apache/spark/pull/17467 >

[jira] [Created] (SPARK-20775) from_json should also have an API where the schema is specified with a string

2017-05-16 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-20775: --- Summary: from_json should also have an API where the schema is specified with a string Key: SPARK-20775 URL: https://issues.apache.org/jira/browse/SPARK-20775 Project:

[jira] [Commented] (SPARK-20571) Flaky SparkR StructuredStreaming tests

2017-05-05 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998680#comment-15998680 ] Burak Yavuz commented on SPARK-20571: - Thanks! > Flaky SparkR StructuredStreaming tests >

[jira] [Resolved] (SPARK-20441) Within the same streaming query, one StreamingRelation should only be transformed to one StreamingExecutionRelation

2017-05-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-20441. - Resolution: Fixed Resolved with https://github.com/apache/spark/pull/17735 > Within the same

[jira] [Closed] (SPARK-20432) Unioning two identical Streaming DataFrames fails during attribute resolution

2017-05-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz closed SPARK-20432. --- Resolution: Duplicate > Unioning two identical Streaming DataFrames fails during attribute

[jira] [Commented] (SPARK-20571) Flaky SparkR StructuredStreaming tests

2017-05-02 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15994185#comment-15994185 ] Burak Yavuz commented on SPARK-20571: - cc [~felixcheung] > Flaky SparkR StructuredStreaming tests >

[jira] [Created] (SPARK-20571) Flaky SparkR StructuredStreaming tests

2017-05-02 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-20571: --- Summary: Flaky SparkR StructuredStreaming tests Key: SPARK-20571 URL: https://issues.apache.org/jira/browse/SPARK-20571 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-20549) java.io.CharConversionException: Invalid UTF-32 in JsonToStructs

2017-05-01 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-20549: --- Summary: java.io.CharConversionException: Invalid UTF-32 in JsonToStructs Key: SPARK-20549 URL: https://issues.apache.org/jira/browse/SPARK-20549 Project: Spark

[jira] [Resolved] (SPARK-20496) KafkaWriter Uses Unanalyzed Logical Plan

2017-04-28 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-20496. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.2 Resolved with

[jira] [Assigned] (SPARK-20496) KafkaWriter Uses Unanalyzed Logical Plan

2017-04-28 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-20496: --- Assignee: Bill Chambers > KafkaWriter Uses Unanalyzed Logical Plan >

[jira] [Created] (SPARK-20432) Unioning two identical Streaming DataFrames fails during attribute resolution

2017-04-21 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-20432: --- Summary: Unioning two identical Streaming DataFrames fails during attribute resolution Key: SPARK-20432 URL: https://issues.apache.org/jira/browse/SPARK-20432 Project:

[jira] [Created] (SPARK-20301) Flakiness in StreamingAggregationSuite

2017-04-11 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-20301: --- Summary: Flakiness in StreamingAggregationSuite Key: SPARK-20301 URL: https://issues.apache.org/jira/browse/SPARK-20301 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-20301) Flakiness in StreamingAggregationSuite

2017-04-11 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-20301: Labels: flaky-test (was: ) > Flakiness in StreamingAggregationSuite >

[jira] [Created] (SPARK-20230) FetchFailedExceptions should invalidate file caches in MapOutputTracker even if newer stages are launched

2017-04-05 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-20230: --- Summary: FetchFailedExceptions should invalidate file caches in MapOutputTracker even if newer stages are launched Key: SPARK-20230 URL:

[jira] [Resolved] (SPARK-19911) Add builder interface for Kinesis DStreams

2017-03-24 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-19911. - Resolution: Fixed Assignee: Adam Budde Fix Version/s: 2.2.0 Target

[jira] [Created] (SPARK-19886) reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource

2017-03-09 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-19886: --- Summary: reportDataLoss cause != null check is wrong for Structured Streaming KafkaSource Key: SPARK-19886 URL: https://issues.apache.org/jira/browse/SPARK-19886

[jira] [Resolved] (SPARK-19813) maxFilesPerTrigger combo latestFirst may miss old files in combination with maxFileAge in FileStreamSource

2017-03-08 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-19813. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > maxFilesPerTrigger

[jira] [Resolved] (SPARK-19304) Kinesis checkpoint recovery is 10x slow

2017-03-06 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-19304. - Resolution: Fixed Fix Version/s: 2.2.0 Target Version/s: 2.2.0 Resolved by:

[jira] [Assigned] (SPARK-19304) Kinesis checkpoint recovery is 10x slow

2017-03-06 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-19304: --- Assignee: Gaurav Shah > Kinesis checkpoint recovery is 10x slow >

[jira] [Resolved] (SPARK-19595) from_json produces only a single row when input is a json array

2017-03-05 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-19595. - Resolution: Fixed Fix Version/s: 2.2.0 Resolved by

[jira] [Assigned] (SPARK-19595) from_json produces only a single row when input is a json array

2017-03-05 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz reassigned SPARK-19595: --- Assignee: Hyukjin Kwon > from_json produces only a single row when input is a json array >

[jira] [Created] (SPARK-19813) maxFilesPerTrigger combo latestFirst may miss old files in combination with maxFileAge in FileStreamSource

2017-03-03 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-19813: --- Summary: maxFilesPerTrigger combo latestFirst may miss old files in combination with maxFileAge in FileStreamSource Key: SPARK-19813 URL:

[jira] [Created] (SPARK-19774) StreamExecution should call stop() on sources when a stream fails

2017-02-28 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-19774: --- Summary: StreamExecution should call stop() on sources when a stream fails Key: SPARK-19774 URL: https://issues.apache.org/jira/browse/SPARK-19774 Project: Spark

[jira] [Resolved] (SPARK-19405) Add support to KinesisUtils for cross-account Kinesis reads via STS

2017-02-22 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-19405. - Resolution: Fixed Assignee: Adam Budde Fix Version/s: 2.2.0 Resolved with:

[jira] [Created] (SPARK-19637) add to_json APIs to SQL

2017-02-16 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-19637: --- Summary: add to_json APIs to SQL Key: SPARK-19637 URL: https://issues.apache.org/jira/browse/SPARK-19637 Project: Spark Issue Type: New Feature

[jira] [Resolved] (SPARK-19542) Delete the temp checkpoint if a query is stopped without errors

2017-02-13 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-19542. - Resolution: Fixed Fix Version/s: 2.2.0 2.1.1 > Delete the temp

[jira] [Created] (SPARK-19543) from_json fails when the input row is empty

2017-02-09 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-19543: --- Summary: from_json fails when the input row is empty Key: SPARK-19543 URL: https://issues.apache.org/jira/browse/SPARK-19543 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-18218) Optimize BlockMatrix multiplication, which may cause OOM and low parallelism usage problem in several cases

2017-01-26 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18218: Assignee: Weichen Xu > Optimize BlockMatrix multiplication, which may cause OOM and low

[jira] [Resolved] (SPARK-18218) Optimize BlockMatrix multiplication, which may cause OOM and low parallelism usage problem in several cases

2017-01-26 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-18218. - Resolution: Implemented Fix Version/s: 2.2.0 Resolved by

[jira] [Updated] (SPARK-19378) StateOperator metrics should still return the total number of rows in state even if there was no data for a trigger

2017-01-26 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-19378: Description: If you have a StreamingDataFrame with an aggregation, we report a metric called

[jira] [Created] (SPARK-19378) StateOperator metrics should still return the total number of rows in state even if there was no data for a trigger

2017-01-26 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-19378: --- Summary: StateOperator metrics should still return the total number of rows in state even if there was no data for a trigger Key: SPARK-19378 URL:

[jira] [Updated] (SPARK-18020) Kinesis receiver does not snapshot when shard completes

2017-01-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18020: Assignee: Takeshi Yamamuro > Kinesis receiver does not snapshot when shard completes >

[jira] [Resolved] (SPARK-18020) Kinesis receiver does not snapshot when shard completes

2017-01-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz resolved SPARK-18020. - Resolution: Fixed Fix Version/s: 2.2.0 Resolved by

[jira] [Updated] (SPARK-18952) regex strings not properly escaped in codegen for aggregations

2016-12-20 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18952: Summary: regex strings not properly escaped in codegen for aggregations (was: regex strings not

[jira] [Created] (SPARK-18952) regex strings not properly escaped in codegen

2016-12-20 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18952: --- Summary: regex strings not properly escaped in codegen Key: SPARK-18952 URL: https://issues.apache.org/jira/browse/SPARK-18952 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-18927) MemorySink for StructuredStreaming can't recover from checkpoint if location is provided in conf

2016-12-19 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18927: --- Summary: MemorySink for StructuredStreaming can't recover from checkpoint if location is provided in conf Key: SPARK-18927 URL: https://issues.apache.org/jira/browse/SPARK-18927

[jira] [Created] (SPARK-18900) Flaky Test: StateStoreSuite.maintenance

2016-12-16 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18900: --- Summary: Flaky Test: StateStoreSuite.maintenance Key: SPARK-18900 URL: https://issues.apache.org/jira/browse/SPARK-18900 Project: Spark Issue Type: Test

[jira] [Updated] (SPARK-18888) partitionBy in DataStreamWriter in Python throws _to_seq not defined

2016-12-15 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-1: Affects Version/s: (was: 2.1.0) 2.0.2 > partitionBy in DataStreamWriter

[jira] [Created] (SPARK-18888) partitionBy in DataStreamWriter in Python throws _to_seq not defined

2016-12-15 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-1: --- Summary: partitionBy in DataStreamWriter in Python throws _to_seq not defined Key: SPARK-1 URL: https://issues.apache.org/jira/browse/SPARK-1 Project: Spark

[jira] [Created] (SPARK-18868) Flaky Test: StreamingQueryListenerSuite

2016-12-14 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18868: --- Summary: Flaky Test: StreamingQueryListenerSuite Key: SPARK-18868 URL: https://issues.apache.org/jira/browse/SPARK-18868 Project: Spark Issue Type: Test

[jira] [Created] (SPARK-18811) Stream Source resolution should happen in StreamExecution thread, not main thread

2016-12-09 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18811: --- Summary: Stream Source resolution should happen in StreamExecution thread, not main thread Key: SPARK-18811 URL: https://issues.apache.org/jira/browse/SPARK-18811

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

2016-11-29 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15706788#comment-15706788 ] Burak Yavuz commented on SPARK-18475: - I'd be happy to share performance results. You're right, I

[jira] [Updated] (SPARK-18634) Corruption and Correctness issues with exploding Python UDFs

2016-11-29 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18634: Description: There are some weird issues with exploding Python UDFs in SparkSQL. There are 2

[jira] [Updated] (SPARK-18634) Corruption and Correctness issues with exploding Python UDFs

2016-11-29 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18634: Description: There are some weird issues with exploding Python UDFs in SparkSQL. There are 2

[jira] [Updated] (SPARK-18634) Corruption and Correctness issues with exploding Python UDFs

2016-11-29 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18634: Summary: Corruption and Correctness issues with exploding Python UDFs (was: Issues with exploding

[jira] [Updated] (SPARK-18634) Corruption and Correctness issues with exploding Python UDFs

2016-11-29 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18634: Description: There are some weird issues with exploding Python UDFs in SparkSQL. There are 2

[jira] [Created] (SPARK-18634) Issues with exploding Python UDFs

2016-11-29 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18634: --- Summary: Issues with exploding Python UDFs Key: SPARK-18634 URL: https://issues.apache.org/jira/browse/SPARK-18634 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-18407) Inferred partition columns cause assertion error

2016-11-25 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15696337#comment-15696337 ] Burak Yavuz commented on SPARK-18407: - This is also resolved as part of

[jira] [Commented] (SPARK-18510) Partition schema inference corrupts data

2016-11-20 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15681677#comment-15681677 ] Burak Yavuz commented on SPARK-18510: - No. Working on a separate fix > Partition schema inference

[jira] [Commented] (SPARK-18510) Partition schema inference corrupts data

2016-11-19 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15680328#comment-15680328 ] Burak Yavuz commented on SPARK-18510: - cc [~r...@databricks.com] I marked this as a blocker as it is

[jira] [Updated] (SPARK-18510) Partition schema inference corrupts data

2016-11-19 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18510: Description: Not sure if this is a regression from 2.0 to 2.1. I was investigating this for

[jira] [Created] (SPARK-18510) Partition schema inference corrupts data

2016-11-19 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18510: --- Summary: Partition schema inference corrupts data Key: SPARK-18510 URL: https://issues.apache.org/jira/browse/SPARK-18510 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-18218) Optimize BlockMatrix multiplication, which may cause OOM and low parallelism usage problem in several cases

2016-11-18 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15677367#comment-15677367 ] Burak Yavuz commented on SPARK-18218: - [~WeichenXu123] You are correct, this would be a problem. But

[jira] [Updated] (SPARK-18493) Add withWatermark and checkpoint to python dataframe

2016-11-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18493: Component/s: PySpark > Add withWatermark and checkpoint to python dataframe >

[jira] [Created] (SPARK-18493) Add withWatermark and checkpoint to python dataframe

2016-11-17 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18493: --- Summary: Add withWatermark and checkpoint to python dataframe Key: SPARK-18493 URL: https://issues.apache.org/jira/browse/SPARK-18493 Project: Spark Issue

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

2016-11-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674535#comment-15674535 ] Burak Yavuz commented on SPARK-18475: - [~c...@koeninger.org] I don't see where you may need strict

[jira] [Commented] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

2016-11-17 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15674254#comment-15674254 ] Burak Yavuz commented on SPARK-18475: - [~ofirm] Thanks for your comment. I've seen significant

[jira] [Created] (SPARK-18475) Be able to provide higher parallelization for StructuredStreaming Kafka Source

2016-11-16 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18475: --- Summary: Be able to provide higher parallelization for StructuredStreaming Kafka Source Key: SPARK-18475 URL: https://issues.apache.org/jira/browse/SPARK-18475

[jira] [Created] (SPARK-18465) Uncache Table shouldn't throw an exception when table doesn't exist

2016-11-15 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18465: --- Summary: Uncache Table shouldn't throw an exception when table doesn't exist Key: SPARK-18465 URL: https://issues.apache.org/jira/browse/SPARK-18465 Project: Spark

[jira] [Created] (SPARK-18342) HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail

2016-11-07 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18342: --- Summary: HDFSBackedStateStore can fail to rename files causing snapshotting and recovery to fail Key: SPARK-18342 URL: https://issues.apache.org/jira/browse/SPARK-18342

[jira] [Created] (SPARK-18339) Don't push down current_timestamp for filters in StructuredStreaming

2016-11-07 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18339: --- Summary: Don't push down current_timestamp for filters in StructuredStreaming Key: SPARK-18339 URL: https://issues.apache.org/jira/browse/SPARK-18339 Project: Spark

[jira] [Created] (SPARK-18337) Memory Sink should be able to recover from checkpoints in Complete OutputMode

2016-11-07 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18337: --- Summary: Memory Sink should be able to recover from checkpoints in Complete OutputMode Key: SPARK-18337 URL: https://issues.apache.org/jira/browse/SPARK-18337 Project:

[jira] [Commented] (SPARK-18261) Add statistics to MemorySink for joining

2016-11-03 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15635014#comment-15635014 ] Burak Yavuz commented on SPARK-18261: - Go for it! > Add statistics to MemorySink for joining >

[jira] [Created] (SPARK-18261) Add statistics to MemorySink for joining

2016-11-03 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18261: --- Summary: Add statistics to MemorySink for joining Key: SPARK-18261 URL: https://issues.apache.org/jira/browse/SPARK-18261 Project: Spark Issue Type: New

[jira] [Created] (SPARK-18260) from_json can throw a better exception when it can't find the column or be nullSafe

2016-11-03 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18260: --- Summary: from_json can throw a better exception when it can't find the column or be nullSafe Key: SPARK-18260 URL: https://issues.apache.org/jira/browse/SPARK-18260

[jira] [Commented] (SPARK-17879) Don't compact metadata logs constantly into a single compacted file

2016-11-01 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15627421#comment-15627421 ] Burak Yavuz commented on SPARK-17879: - We should be doing the second. What you said makes sense, we

[jira] [Created] (SPARK-18142) Spark Master tries to launch workers 145 times within 1 minute

2016-10-27 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-18142: --- Summary: Spark Master tries to launch workers 145 times within 1 minute Key: SPARK-18142 URL: https://issues.apache.org/jira/browse/SPARK-18142 Project: Spark

[jira] [Updated] (SPARK-18142) Spark Master tries to launch workers 145 times within 1 minute

2016-10-27 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-18142: Component/s: Spark Core > Spark Master tries to launch workers 145 times within 1 minute >

[jira] [Created] (SPARK-17921) checkpointLocation being set in memory streams fail after restart. Should fail fast

2016-10-13 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17921: --- Summary: checkpointLocation being set in memory streams fail after restart. Should fail fast Key: SPARK-17921 URL: https://issues.apache.org/jira/browse/SPARK-17921

[jira] [Created] (SPARK-17879) Don't compact metadata logs constantly into a single compacted file

2016-10-11 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17879: --- Summary: Don't compact metadata logs constantly into a single compacted file Key: SPARK-17879 URL: https://issues.apache.org/jira/browse/SPARK-17879 Project: Spark

[jira] [Created] (SPARK-17876) Write StructuredStreaming WAL to a stream instead of materializing all at once

2016-10-11 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17876: --- Summary: Write StructuredStreaming WAL to a stream instead of materializing all at once Key: SPARK-17876 URL: https://issues.apache.org/jira/browse/SPARK-17876

[jira] [Updated] (SPARK-17754) DataFrame reader and writer don't show Input/Output metrics in Spark UI

2016-09-30 Thread Burak Yavuz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Burak Yavuz updated SPARK-17754: Component/s: Web UI SQL > DataFrame reader and writer don't show Input/Output

[jira] [Created] (SPARK-17754) DataFrame reader and writer don't show Input/Output metrics in Spark UI

2016-09-30 Thread Burak Yavuz (JIRA)
Burak Yavuz created SPARK-17754: --- Summary: DataFrame reader and writer don't show Input/Output metrics in Spark UI Key: SPARK-17754 URL: https://issues.apache.org/jira/browse/SPARK-17754 Project: Spark

<    1   2   3   4   >