[jira] [Commented] (SPARK-26379) Structured Streaming - Exception on adding column to Dataset

2019-01-21 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748458#comment-16748458 ] Jungtaek Lim commented on SPARK-26379: -- This looks like also occurred on the master branch. Simpler

[jira] [Assigned] (SPARK-25811) Support PyArrow's feature to raise an error for unsafe cast

2019-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-25811: Assignee: Liang-Chi Hsieh > Support PyArrow's feature to raise an error for unsafe cast

[jira] [Resolved] (SPARK-25811) Support PyArrow's feature to raise an error for unsafe cast

2019-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-25811. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22807

[jira] [Commented] (SPARK-26187) Stream-stream left outer join returns outer nulls for already matched rows

2019-01-21 Thread sandeep katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748396#comment-16748396 ] sandeep katta commented on SPARK-26187: --- which branch you used to reproduce this issue ? Master

[jira] [Updated] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-21 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bruce Robbins updated SPARK-26680: -- Description: This Java code results in a StackOverflowError: {code:java} List groupByCols =

[jira] [Assigned] (SPARK-26682) Task attempt ID collision causes lost data

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26682: Assignee: (was: Apache Spark) > Task attempt ID collision causes lost data >

[jira] [Assigned] (SPARK-26682) Task attempt ID collision causes lost data

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26682: Assignee: Apache Spark > Task attempt ID collision causes lost data >

[jira] [Commented] (SPARK-26679) Deconflict spark.executor.pyspark.memory and spark.python.worker.memory

2019-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748307#comment-16748307 ] Hyukjin Kwon commented on SPARK-26679: -- It's fine to have two different configurations to me if

[jira] [Created] (SPARK-26682) Task attempt ID collision causes lost data

2019-01-21 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-26682: - Summary: Task attempt ID collision causes lost data Key: SPARK-26682 URL: https://issues.apache.org/jira/browse/SPARK-26682 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-26681) Support Ammonite scopes in OuterScopes

2019-01-21 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-26681: - Summary: Support Ammonite scopes in OuterScopes Key: SPARK-26681 URL: https://issues.apache.org/jira/browse/SPARK-26681 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-26681) Support Ammonite scopes in OuterScopes

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26681: Assignee: Apache Spark > Support Ammonite scopes in OuterScopes >

[jira] [Assigned] (SPARK-26681) Support Ammonite scopes in OuterScopes

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26681: Assignee: (was: Apache Spark) > Support Ammonite scopes in OuterScopes >

[jira] [Created] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-21 Thread Bruce Robbins (JIRA)
Bruce Robbins created SPARK-26680: - Summary: StackOverflowError if Stream passed to groupBy Key: SPARK-26680 URL: https://issues.apache.org/jira/browse/SPARK-26680 Project: Spark Issue Type:

[jira] [Commented] (SPARK-26680) StackOverflowError if Stream passed to groupBy

2019-01-21 Thread Bruce Robbins (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748277#comment-16748277 ] Bruce Robbins commented on SPARK-26680: --- I will make a PR for this, but I would like to hear any

[jira] [Assigned] (SPARK-26666) DataSourceV2: Add overwrite and dynamic overwrite.

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-2: Assignee: Apache Spark > DataSourceV2: Add overwrite and dynamic overwrite. >

[jira] [Assigned] (SPARK-26666) DataSourceV2: Add overwrite and dynamic overwrite.

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-2: Assignee: (was: Apache Spark) > DataSourceV2: Add overwrite and dynamic overwrite. >

[jira] [Resolved] (SPARK-26676) Make HiveContextSQLTests.test_unbounded_frames test compatible with Python 2 and PyPy

2019-01-21 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-26676. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23604

[jira] [Resolved] (SPARK-26520) data source V2 API refactoring (micro-batch read)

2019-01-21 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-26520. - Resolution: Fixed Fix Version/s: 3.0.0 > data source V2 API refactoring (micro-batch read) >

[jira] [Assigned] (SPARK-26676) Make HiveContextSQLTests.test_unbounded_frames test compatible with Python 2 and PyPy

2019-01-21 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-26676: Assignee: Hyukjin Kwon > Make HiveContextSQLTests.test_unbounded_frames test compatible

[jira] [Created] (SPARK-26679) Deconflict spark.executor.pyspark.memory and spark.python.worker.memory

2019-01-21 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-26679: - Summary: Deconflict spark.executor.pyspark.memory and spark.python.worker.memory Key: SPARK-26679 URL: https://issues.apache.org/jira/browse/SPARK-26679 Project: Spark

[jira] [Updated] (SPARK-26652) Use Proleptic Gregorian calendar in creation of Timestamp/Date literals from strings

2019-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26652: - Target Version/s: (was: 3.0.0) > Use Proleptic Gregorian calendar in creation of

[jira] [Updated] (SPARK-26652) Use Proleptic Gregorian calendar in creation of Timestamp/Date literals from strings

2019-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26652: - Fix Version/s: 3.0.0 > Use Proleptic Gregorian calendar in creation of Timestamp/Date literals

[jira] [Resolved] (SPARK-26652) Use Proleptic Gregorian calendar in creation of Timestamp/Date literals from strings

2019-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26652. -- Resolution: Fixed Assignee: Maxim Gekk Target Version/s: 3.0.0 Fixed in

[jira] [Assigned] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26650: Assignee: (was: Apache Spark) > Yarn Client throws 'ClassNotFoundException: >

[jira] [Assigned] (SPARK-26650) Yarn Client throws 'ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration'

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26650: Assignee: Apache Spark > Yarn Client throws 'ClassNotFoundException: >

[jira] [Created] (SPARK-26678) Empty values end up as quoted empty strings in CSV files

2019-01-21 Thread Robert V (JIRA)
Robert V created SPARK-26678: Summary: Empty values end up as quoted empty strings in CSV files Key: SPARK-26678 URL: https://issues.apache.org/jira/browse/SPARK-26678 Project: Spark Issue Type:

[jira] [Updated] (SPARK-26678) Empty values end up as quoted empty strings in CSV files

2019-01-21 Thread Robert V (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert V updated SPARK-26678: - Description: h1. Problem statement Empty string values were written to CSV as unquoted strings prior

[jira] [Created] (SPARK-26677) Incorrect results of not(eqNullSafe) when data read from Parquet file

2019-01-21 Thread Michal Kapalka (JIRA)
Michal Kapalka created SPARK-26677: -- Summary: Incorrect results of not(eqNullSafe) when data read from Parquet file Key: SPARK-26677 URL: https://issues.apache.org/jira/browse/SPARK-26677 Project:

[jira] [Assigned] (SPARK-26676) Make HiveContextSQLTests.test_unbounded_frames test compatible with Python 2 and PyPy

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26676: Assignee: (was: Apache Spark) > Make HiveContextSQLTests.test_unbounded_frames test

[jira] [Assigned] (SPARK-26676) Make HiveContextSQLTests.test_unbounded_frames test compatible with Python 2 and PyPy

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26676: Assignee: Apache Spark > Make HiveContextSQLTests.test_unbounded_frames test compatible

[jira] [Created] (SPARK-26676) Make HiveContextSQLTests.test_unbounded_frames test compatible with Python 2 and PyPy

2019-01-21 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-26676: Summary: Make HiveContextSQLTests.test_unbounded_frames test compatible with Python 2 and PyPy Key: SPARK-26676 URL: https://issues.apache.org/jira/browse/SPARK-26676

[jira] [Updated] (SPARK-26675) Error happened during creating avro files

2019-01-21 Thread Tony Mao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tony Mao updated SPARK-26675: - Description: Run cmd {code:java} spark-submit --packages org.apache.spark:spark-avro_2.11:2.4.0

[jira] [Created] (SPARK-26675) Error happened during creating avro files

2019-01-21 Thread Tony Mao (JIRA)
Tony Mao created SPARK-26675: Summary: Error happened during creating avro files Key: SPARK-26675 URL: https://issues.apache.org/jira/browse/SPARK-26675 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-25713) Implement copy() for ColumnarArray

2019-01-21 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-25713: -- Priority: Minor (was: Major) > Implement copy() for ColumnarArray >

[jira] [Assigned] (SPARK-26674) Consolidate CompositeByteBuf when reading large frame

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26674: Assignee: Apache Spark > Consolidate CompositeByteBuf when reading large frame >

[jira] [Assigned] (SPARK-26674) Consolidate CompositeByteBuf when reading large frame

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26674: Assignee: (was: Apache Spark) > Consolidate CompositeByteBuf when reading large

[jira] [Created] (SPARK-26674) Consolidate CompositeByteBuf when reading large frame

2019-01-21 Thread liupengcheng (JIRA)
liupengcheng created SPARK-26674: Summary: Consolidate CompositeByteBuf when reading large frame Key: SPARK-26674 URL: https://issues.apache.org/jira/browse/SPARK-26674 Project: Spark Issue

[jira] [Commented] (SPARK-26649) Noop Streaming Sink using DSV2

2019-01-21 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747830#comment-16747830 ] Gabor Somogyi commented on SPARK-26649: --- Just wondering why is it a bug? > Noop Streaming Sink

[jira] [Commented] (SPARK-26028) Design sketch for SPIP: Property Graphs, Cypher Queries, and Algorithms

2019-01-21 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747795#comment-16747795 ] Martin Junghanns commented on SPARK-26028: -- We opted for the current design as it follows

[jira] [Assigned] (SPARK-26673) File source V2 write: create framework and migrate ORC to it

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26673: Assignee: Apache Spark > File source V2 write: create framework and migrate ORC to it >

[jira] [Assigned] (SPARK-26673) File source V2 write: create framework and migrate ORC to it

2019-01-21 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26673: Assignee: (was: Apache Spark) > File source V2 write: create framework and migrate

[jira] [Created] (SPARK-26673) File source V2 write: create framework and migrate ORC to it

2019-01-21 Thread Gengliang Wang (JIRA)
Gengliang Wang created SPARK-26673: -- Summary: File source V2 write: create framework and migrate ORC to it Key: SPARK-26673 URL: https://issues.apache.org/jira/browse/SPARK-26673 Project: Spark

[jira] [Assigned] (SPARK-26022) PySpark Comparison with Pandas

2019-01-21 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-26022: Assignee: Hyukjin Kwon > PySpark Comparison with Pandas > --

[jira] [Created] (SPARK-26672) SinglePartition should not satisfies HashClusteredDistribution/OrderedDistribution

2019-01-21 Thread Wang, Gang (JIRA)
Wang, Gang created SPARK-26672: -- Summary: SinglePartition should not satisfies HashClusteredDistribution/OrderedDistribution Key: SPARK-26672 URL: https://issues.apache.org/jira/browse/SPARK-26672