[jira] [Updated] (SPARK-27669) Refactor DataFrameWriter to resolve datasources in a command

2019-05-09 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-27669: --- Summary: Refactor DataFrameWriter to resolve datasources in a command (was: Refactor

[jira] [Created] (SPARK-27669) Refactor DataFrameWriter to always go through Catalyst for analysis

2019-05-09 Thread Eric Liang (JIRA)
Eric Liang created SPARK-27669: -- Summary: Refactor DataFrameWriter to always go through Catalyst for analysis Key: SPARK-27669 URL: https://issues.apache.org/jira/browse/SPARK-27669 Project: Spark

[jira] [Created] (SPARK-27392) TestHive test tables should be placed in shared test state, not per session

2019-04-04 Thread Eric Liang (JIRA)
Eric Liang created SPARK-27392: -- Summary: TestHive test tables should be placed in shared test state, not per session Key: SPARK-27392 URL: https://issues.apache.org/jira/browse/SPARK-27392 Project:

[jira] [Created] (SPARK-23971) Should not leak Spark sessions across test suites

2018-04-12 Thread Eric Liang (JIRA)
Eric Liang created SPARK-23971: -- Summary: Should not leak Spark sessions across test suites Key: SPARK-23971 URL: https://issues.apache.org/jira/browse/SPARK-23971 Project: Spark Issue Type:

[jira] [Created] (SPARK-23809) Active SparkSession should be set by getOrCreate

2018-03-28 Thread Eric Liang (JIRA)
Eric Liang created SPARK-23809: -- Summary: Active SparkSession should be set by getOrCreate Key: SPARK-23809 URL: https://issues.apache.org/jira/browse/SPARK-23809 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-18727) Support schema evolution as new files are inserted into table

2017-04-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15987966#comment-15987966 ] Eric Liang commented on SPARK-18727: +1 for supporting ALTER TABLE REPLACE COLUMNS > Support schema

[jira] [Updated] (SPARK-18727) Support schema evolution as new files are inserted into table

2017-04-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18727: --- The common case we see is users having a complete schema (e.g. output of ETL pipeline) and wanting to

[jira] [Updated] (SPARK-18727) Support schema evolution as new files are inserted into table

2017-04-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18727: --- Can we add ALTER TABLE SCHEMA to update the entire schema? That would cover any edge cases. >

[jira] [Commented] (SPARK-20450) Unexpected first-query schema inference cost with 2.1.1 RC

2017-04-24 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981753#comment-15981753 ] Eric Liang commented on SPARK-20450: I'm not sure what you mean by new issue, but it's only in the

[jira] [Comment Edited] (SPARK-20450) Unexpected first-query schema inference cost with 2.1.1 RC

2017-04-24 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15981753#comment-15981753 ] Eric Liang edited comment on SPARK-20450 at 4/24/17 7:40 PM: - I'm not sure

[jira] [Created] (SPARK-20450) Unexpected first-query schema inference cost with 2.1.1 RC

2017-04-24 Thread Eric Liang (JIRA)
Eric Liang created SPARK-20450: -- Summary: Unexpected first-query schema inference cost with 2.1.1 RC Key: SPARK-20450 URL: https://issues.apache.org/jira/browse/SPARK-20450 Project: Spark Issue

[jira] [Created] (SPARK-20398) range() operator should include cancellation reason when killed

2017-04-19 Thread Eric Liang (JIRA)
Eric Liang created SPARK-20398: -- Summary: range() operator should include cancellation reason when killed Key: SPARK-20398 URL: https://issues.apache.org/jira/browse/SPARK-20398 Project: Spark

[jira] [Created] (SPARK-20358) Executors failing stage on interrupted exception thrown by cancelled tasks

2017-04-17 Thread Eric Liang (JIRA)
Eric Liang created SPARK-20358: -- Summary: Executors failing stage on interrupted exception thrown by cancelled tasks Key: SPARK-20358 URL: https://issues.apache.org/jira/browse/SPARK-20358 Project:

[jira] [Updated] (SPARK-20217) Executor should not fail stage if killed task throws non-interrupted exception

2017-04-04 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-20217: --- Description: This is reproducible as follows. Run the following, and then use

[jira] [Created] (SPARK-20217) Executor should not fail stage if killed task throws non-interrupted exception

2017-04-04 Thread Eric Liang (JIRA)
Eric Liang created SPARK-20217: -- Summary: Executor should not fail stage if killed task throws non-interrupted exception Key: SPARK-20217 URL: https://issues.apache.org/jira/browse/SPARK-20217 Project:

[jira] [Created] (SPARK-20148) Extend the file commit interface to allow subscribing to task commit messages

2017-03-29 Thread Eric Liang (JIRA)
Eric Liang created SPARK-20148: -- Summary: Extend the file commit interface to allow subscribing to task commit messages Key: SPARK-20148 URL: https://issues.apache.org/jira/browse/SPARK-20148 Project:

[jira] [Created] (SPARK-19820) Allow reason to be specified for task kill

2017-03-04 Thread Eric Liang (JIRA)
Eric Liang created SPARK-19820: -- Summary: Allow reason to be specified for task kill Key: SPARK-19820 URL: https://issues.apache.org/jira/browse/SPARK-19820 Project: Spark Issue Type:

[jira] [Created] (SPARK-19183) Add deleteWithJob hook to internal commit protocol API

2017-01-11 Thread Eric Liang (JIRA)
Eric Liang created SPARK-19183: -- Summary: Add deleteWithJob hook to internal commit protocol API Key: SPARK-19183 URL: https://issues.apache.org/jira/browse/SPARK-19183 Project: Spark Issue

[jira] [Commented] (SPARK-18814) CheckAnalysis rejects TPCDS query 32

2016-12-09 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737148#comment-15737148 ] Eric Liang commented on SPARK-18814: It seems that the references of an Alias expression should

[jira] [Commented] (SPARK-18814) CheckAnalysis rejects TPCDS query 32

2016-12-09 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15737068#comment-15737068 ] Eric Liang commented on SPARK-18814: [~rxin] > CheckAnalysis rejects TPCDS query 32 >

[jira] [Created] (SPARK-18814) CheckAnalysis rejects TPCDS query 32

2016-12-09 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18814: -- Summary: CheckAnalysis rejects TPCDS query 32 Key: SPARK-18814 URL: https://issues.apache.org/jira/browse/SPARK-18814 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-18727) Support schema evolution as new files are inserted into table

2016-12-05 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18727: -- Summary: Support schema evolution as new files are inserted into table Key: SPARK-18727 URL: https://issues.apache.org/jira/browse/SPARK-18727 Project: Spark

[jira] [Updated] (SPARK-18727) Support schema evolution as new files are inserted into table

2016-12-05 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18727: --- Component/s: SQL > Support schema evolution as new files are inserted into table >

[jira] [Created] (SPARK-18726) Filesystem unnecessarily scanned twice during creation of non-catalog table

2016-12-05 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18726: -- Summary: Filesystem unnecessarily scanned twice during creation of non-catalog table Key: SPARK-18726 URL: https://issues.apache.org/jira/browse/SPARK-18726 Project:

[jira] [Created] (SPARK-18725) Creating a datasource table with schema should not scan all files for table

2016-12-05 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18725: -- Summary: Creating a datasource table with schema should not scan all files for table Key: SPARK-18725 URL: https://issues.apache.org/jira/browse/SPARK-18725 Project:

[jira] [Updated] (SPARK-18679) Regression in file listing performance

2016-12-01 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18679: --- Component/s: SQL > Regression in file listing performance > -- >

[jira] [Created] (SPARK-18679) Regression in file listing performance

2016-12-01 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18679: -- Summary: Regression in file listing performance Key: SPARK-18679 URL: https://issues.apache.org/jira/browse/SPARK-18679 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-18679) Regression in file listing performance

2016-12-01 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18679: --- Affects Version/s: 2.1.0 > Regression in file listing performance >

[jira] [Created] (SPARK-18661) Creating a partitioned datasource table should not scan all files in filesystem

2016-11-30 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18661: -- Summary: Creating a partitioned datasource table should not scan all files in filesystem Key: SPARK-18661 URL: https://issues.apache.org/jira/browse/SPARK-18661 Project:

[jira] [Updated] (SPARK-18661) Creating a partitioned datasource table should not scan all files for table

2016-11-30 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18661: --- Summary: Creating a partitioned datasource table should not scan all files for table (was: Creating

[jira] [Updated] (SPARK-18659) Incorrect behaviors in overwrite table for datasource tables

2016-11-30 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18659: --- Description: The first three test cases fail due to a crash in hive client when dropping partitions

[jira] [Updated] (SPARK-18659) Incorrect behaviors in overwrite table for datasource tables

2016-11-30 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18659: --- Description: The following test cases fail due to a crash in hive client when dropping partitions

[jira] [Updated] (SPARK-18659) Incorrect behaviors in overwrite table for datasource tables

2016-11-30 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18659: --- Summary: Incorrect behaviors in overwrite table for datasource tables (was: Crash in overwrite

[jira] [Created] (SPARK-18659) Crash in overwrite table partitions due to hive metastore integration

2016-11-30 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18659: -- Summary: Crash in overwrite table partitions due to hive metastore integration Key: SPARK-18659 URL: https://issues.apache.org/jira/browse/SPARK-18659 Project: Spark

[jira] [Updated] (SPARK-18635) Partition name/values not escaped correctly in some cases

2016-11-29 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18635: --- Target Version/s: 2.1.0 Priority: Critical (was: Major) > Partition name/values not

[jira] [Created] (SPARK-18635) Partition name/values not escaped correctly in some cases

2016-11-29 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18635: -- Summary: Partition name/values not escaped correctly in some cases Key: SPARK-18635 URL: https://issues.apache.org/jira/browse/SPARK-18635 Project: Spark Issue

[jira] [Updated] (SPARK-18545) Verify number of hive client RPCs in PartitionedTablePerfStatsSuite

2016-11-29 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18545: --- Issue Type: Sub-task (was: Test) Parent: SPARK-17861 > Verify number of hive client RPCs in

[jira] [Updated] (SPARK-18507) Major performance regression in SHOW PARTITIONS on partitioned Hive tables

2016-11-29 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18507: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-17861 > Major performance regression in SHOW

[jira] [Updated] (SPARK-18544) Append with df.saveAsTable writes data to wrong location

2016-11-22 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18544: --- Description: When using saveAsTable in append mode, data will be written to the wrong location for

[jira] [Commented] (SPARK-18544) Append with df.saveAsTable writes data to wrong location

2016-11-22 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15687884#comment-15687884 ] Eric Liang commented on SPARK-18544: cc [~yhuai] [~cloud_fan] I'll try to look at this today but

[jira] [Created] (SPARK-18545) Verify number of hive client RPCs in PartitionedTablePerfStatsSuite

2016-11-22 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18545: -- Summary: Verify number of hive client RPCs in PartitionedTablePerfStatsSuite Key: SPARK-18545 URL: https://issues.apache.org/jira/browse/SPARK-18545 Project: Spark

[jira] [Created] (SPARK-18544) Append with df.saveAsTable writes data to wrong location

2016-11-22 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18544: -- Summary: Append with df.saveAsTable writes data to wrong location Key: SPARK-18544 URL: https://issues.apache.org/jira/browse/SPARK-18544 Project: Spark Issue

[jira] [Created] (SPARK-18393) DataFrame pivot output column names should respect aliases

2016-11-09 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18393: -- Summary: DataFrame pivot output column names should respect aliases Key: SPARK-18393 URL: https://issues.apache.org/jira/browse/SPARK-18393 Project: Spark Issue

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-11-09 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15652414#comment-15652414 ] Eric Liang commented on SPARK-17916: In our case, a user wants the empty string (whether actually

[jira] [Commented] (SPARK-17916) CSV data source treats empty string as null no matter what nullValue option is

2016-11-08 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649039#comment-15649039 ] Eric Liang commented on SPARK-17916: We're hitting this as a regression from 2.0 as well. Ideally,

[jira] [Updated] (SPARK-17990) ALTER TABLE ... ADD PARTITION does not play nice with mixed-case partition column names

2016-11-07 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17990: --- Target Version/s: 2.1.0 > ALTER TABLE ... ADD PARTITION does not play nice with mixed-case partition

[jira] [Updated] (SPARK-18333) Revert hacks in parquet and orc reader to support case insensitive resolution

2016-11-07 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18333: --- Target Version/s: 2.1.0 > Revert hacks in parquet and orc reader to support case insensitive

[jira] [Updated] (SPARK-18145) Update documentation for hive partition management in 2.1

2016-11-07 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18145: --- Target Version/s: 2.1.0 > Update documentation for hive partition management in 2.1 >

[jira] [Created] (SPARK-18333) Revert hacks in parquet and orc reader to support case insensitive resolution

2016-11-07 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18333: -- Summary: Revert hacks in parquet and orc reader to support case insensitive resolution Key: SPARK-18333 URL: https://issues.apache.org/jira/browse/SPARK-18333 Project:

[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

2016-11-07 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-17861 > Should fix INSERT OVERWRITE TABLE of

[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

2016-11-07 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Target Version/s: 2.1.0 > Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic

[jira] [Commented] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

2016-11-07 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15645238#comment-15645238 ] Eric Liang commented on SPARK-18185: I'm currently working on this. > Should fix INSERT OVERWRITE

[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

2016-11-03 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Description: As of current 2.1, INSERT OVERWRITE with dynamic partitions against a Datasource table

[jira] [Updated] (SPARK-18185) Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

2016-11-03 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18185: --- Summary: Should fix INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions (was:

[jira] [Created] (SPARK-18185) Should disallow INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions

2016-10-31 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18185: -- Summary: Should disallow INSERT OVERWRITE TABLE of Datasource tables with dynamic partitions Key: SPARK-18185 URL: https://issues.apache.org/jira/browse/SPARK-18185

[jira] [Created] (SPARK-18184) INSERT [INTO|OVERWRITE] TABLE ... PARTITION for Datasource tables cannot handle partitions with custom locations

2016-10-31 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18184: -- Summary: INSERT [INTO|OVERWRITE] TABLE ... PARTITION for Datasource tables cannot handle partitions with custom locations Key: SPARK-18184 URL:

[jira] [Updated] (SPARK-18183) INSERT OVERWRITE TABLE ... PARTITION will overwrite the entire Datasource table instead of just the specified partition

2016-10-31 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18183: --- Component/s: SQL > INSERT OVERWRITE TABLE ... PARTITION will overwrite the entire Datasource >

[jira] [Created] (SPARK-18183) INSERT OVERWRITE TABLE ... PARTITION will overwrite the entire Datasource table instead of just the specified partition

2016-10-31 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18183: -- Summary: INSERT OVERWRITE TABLE ... PARTITION will overwrite the entire Datasource table instead of just the specified partition Key: SPARK-18183 URL:

[jira] [Created] (SPARK-18167) Flaky test when hive partition pruning is enabled

2016-10-28 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18167: -- Summary: Flaky test when hive partition pruning is enabled Key: SPARK-18167 URL: https://issues.apache.org/jira/browse/SPARK-18167 Project: Spark Issue Type:

[jira] [Created] (SPARK-18146) Avoid using Union to chain together create table and repair partition commands

2016-10-27 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18146: -- Summary: Avoid using Union to chain together create table and repair partition commands Key: SPARK-18146 URL: https://issues.apache.org/jira/browse/SPARK-18146 Project:

[jira] [Updated] (SPARK-18145) Update documentation

2016-10-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18145: --- Issue Type: Sub-task (was: Documentation) Parent: SPARK-17861 > Update documentation >

[jira] [Updated] (SPARK-18145) Update documentation for hive partition management in 2.1

2016-10-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18145: --- Component/s: SQL > Update documentation for hive partition management in 2.1 >

[jira] [Updated] (SPARK-18145) Update documentation for hive partition management in 2.1

2016-10-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18145: --- Summary: Update documentation for hive partition management in 2.1 (was: Update documentation) >

[jira] [Created] (SPARK-18145) Update documentation

2016-10-27 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18145: -- Summary: Update documentation Key: SPARK-18145 URL: https://issues.apache.org/jira/browse/SPARK-18145 Project: Spark Issue Type: Documentation

[jira] [Created] (SPARK-18103) Rename *FileCatalog to *FileProvider

2016-10-25 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18103: -- Summary: Rename *FileCatalog to *FileProvider Key: SPARK-18103 URL: https://issues.apache.org/jira/browse/SPARK-18103 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-18101) ExternalCatalogSuite should test with mixed case fields

2016-10-25 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18101: --- Issue Type: Sub-task (was: Test) Parent: SPARK-17861 > ExternalCatalogSuite should test

[jira] [Created] (SPARK-18101) ExternalCatalogSuite should test with mixed case fields

2016-10-25 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18101: -- Summary: ExternalCatalogSuite should test with mixed case fields Key: SPARK-18101 URL: https://issues.apache.org/jira/browse/SPARK-18101 Project: Spark Issue

[jira] [Updated] (SPARK-17183) put hive serde table schema to table properties like data source table

2016-10-24 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17183: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-17861 > put hive serde table schema

[jira] [Created] (SPARK-18087) Optimize insert to not require REPAIR TABLE

2016-10-24 Thread Eric Liang (JIRA)
Eric Liang created SPARK-18087: -- Summary: Optimize insert to not require REPAIR TABLE Key: SPARK-18087 URL: https://issues.apache.org/jira/browse/SPARK-18087 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-18026) should not always lowercase partition columns of partition spec in parser

2016-10-24 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-18026: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-17861 > should not always lowercase

[jira] [Updated] (SPARK-17970) Use metastore for managing filesource table partitions as well

2016-10-24 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17970: --- Summary: Use metastore for managing filesource table partitions as well (was: store partition spec

[jira] [Updated] (SPARK-17994) Add back a file status cache for catalog tables

2016-10-18 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17994: --- Description: In SPARK-16980, we removed the full in-memory cache of table partitions in favor of

[jira] [Updated] (SPARK-17994) Add back a file status cache for catalog tables

2016-10-18 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17994: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-17861 > Add back a file status cache

[jira] [Created] (SPARK-17994) Add back a file status cache for catalog tables

2016-10-18 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17994: -- Summary: Add back a file status cache for catalog tables Key: SPARK-17994 URL: https://issues.apache.org/jira/browse/SPARK-17994 Project: Spark Issue Type:

[jira] [Commented] (SPARK-17983) Can't filter over mixed case parquet columns of converted Hive tables

2016-10-18 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586476#comment-15586476 ] Eric Liang commented on SPARK-17983: Since we already store the original (case-sensitive) schema of

[jira] [Updated] (SPARK-17991) Enable metastore partition pruning for unconverted hive tables by default

2016-10-18 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17991: --- Issue Type: Sub-task (was: Improvement) Parent: SPARK-17861 > Enable metastore partition

[jira] [Created] (SPARK-17991) Enable metastore partition pruning for unconverted hive tables by default

2016-10-18 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17991: -- Summary: Enable metastore partition pruning for unconverted hive tables by default Key: SPARK-17991 URL: https://issues.apache.org/jira/browse/SPARK-17991 Project: Spark

[jira] [Updated] (SPARK-17980) Fix refreshByPath for converted Hive tables

2016-10-18 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17980: --- Issue Type: Sub-task (was: Bug) Parent: SPARK-17861 > Fix refreshByPath for converted Hive

[jira] [Commented] (SPARK-17862) Feature flag SPARK-16980

2016-10-18 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15586210#comment-15586210 ] Eric Liang commented on SPARK-17862: Yes, this is the flag: {code} val

[jira] [Updated] (SPARK-17983) Can't filter over mixed case parquet columns of converted Hive tables

2016-10-17 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17983: --- Description: We should probably revive https://github.com/apache/spark/pull/14750 in order to fix

[jira] [Created] (SPARK-17980) Fix refreshByPath for converted Hive tables

2016-10-17 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17980: -- Summary: Fix refreshByPath for converted Hive tables Key: SPARK-17980 URL: https://issues.apache.org/jira/browse/SPARK-17980 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-17974) Refactor FileCatalog classes to simplify the inheritance tree

2016-10-17 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17974: --- Affects Version/s: 2.1.0 > Refactor FileCatalog classes to simplify the inheritance tree >

[jira] [Created] (SPARK-17974) Refactor FileCatalog classes to simplify the inheritance tree

2016-10-17 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17974: -- Summary: Refactor FileCatalog classes to simplify the inheritance tree Key: SPARK-17974 URL: https://issues.apache.org/jira/browse/SPARK-17974 Project: Spark

[jira] [Updated] (SPARK-17974) Refactor FileCatalog classes to simplify the inheritance tree

2016-10-17 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17974: --- Component/s: SQL > Refactor FileCatalog classes to simplify the inheritance tree >

[jira] [Created] (SPARK-17740) Spark tests should mock / interpose HDFS to ensure that streams are closed

2016-09-29 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17740: -- Summary: Spark tests should mock / interpose HDFS to ensure that streams are closed Key: SPARK-17740 URL: https://issues.apache.org/jira/browse/SPARK-17740 Project:

[jira] [Created] (SPARK-17713) Move row-datasource related tests out of JDBCSuite

2016-09-28 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17713: -- Summary: Move row-datasource related tests out of JDBCSuite Key: SPARK-17713 URL: https://issues.apache.org/jira/browse/SPARK-17713 Project: Spark Issue Type:

[jira] [Created] (SPARK-17701) Refactor DataSourceScanExec so its sameResult call does not compare strings

2016-09-27 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17701: -- Summary: Refactor DataSourceScanExec so its sameResult call does not compare strings Key: SPARK-17701 URL: https://issues.apache.org/jira/browse/SPARK-17701 Project:

[jira] [Updated] (SPARK-17701) Refactor DataSourceScanExec so its sameResult call does not compare strings

2016-09-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17701: --- Component/s: SQL > Refactor DataSourceScanExec so its sameResult call does not compare strings >

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15528145#comment-15528145 ] Eric Liang commented on SPARK-17673: Russell, could you try applying this patch (wip) to see if it

[jira] [Commented] (SPARK-17673) Reused Exchange Aggregations Produce Incorrect Results

2016-09-27 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15527339#comment-15527339 ] Eric Liang commented on SPARK-17673: I'm looking at this now. > Reused Exchange Aggregations Produce

[jira] [Created] (SPARK-17472) Better error message for serialization failures of large objects in Python

2016-09-09 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17472: -- Summary: Better error message for serialization failures of large objects in Python Key: SPARK-17472 URL: https://issues.apache.org/jira/browse/SPARK-17472 Project:

[jira] [Updated] (SPARK-17370) Shuffle service files not invalidated when a slave is lost

2016-09-01 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17370: --- Component/s: Spark Core > Shuffle service files not invalidated when a slave is lost >

[jira] [Created] (SPARK-17371) Resubmitted stage outputs deleted by zombie map tasks on stop()

2016-09-01 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17371: -- Summary: Resubmitted stage outputs deleted by zombie map tasks on stop() Key: SPARK-17371 URL: https://issues.apache.org/jira/browse/SPARK-17371 Project: Spark

[jira] [Created] (SPARK-17370) Shuffle service files not invalidated when a slave is lost

2016-09-01 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17370: -- Summary: Shuffle service files not invalidated when a slave is lost Key: SPARK-17370 URL: https://issues.apache.org/jira/browse/SPARK-17370 Project: Spark Issue

[jira] [Commented] (SPARK-17042) Repl-defined classes cannot be replicated

2016-08-22 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15431616#comment-15431616 ] Eric Liang commented on SPARK-17042: Yeah, my bad. I was trying to split this up but it turns out to

[jira] [Created] (SPARK-17162) Range does not support SQL generation

2016-08-19 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17162: -- Summary: Range does not support SQL generation Key: SPARK-17162 URL: https://issues.apache.org/jira/browse/SPARK-17162 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-17069) Expose spark.range() as table-valued function in SQL

2016-08-15 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17069: -- Summary: Expose spark.range() as table-valued function in SQL Key: SPARK-17069 URL: https://issues.apache.org/jira/browse/SPARK-17069 Project: Spark Issue Type:

[jira] [Updated] (SPARK-17042) Repl-defined classes cannot be replicated

2016-08-12 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-17042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-17042: --- Description: A simple fix is to erase the classTag when using the default serializer, since it's

[jira] [Created] (SPARK-17042) Repl-defined classes cannot be replicated

2016-08-12 Thread Eric Liang (JIRA)
Eric Liang created SPARK-17042: -- Summary: Repl-defined classes cannot be replicated Key: SPARK-17042 URL: https://issues.apache.org/jira/browse/SPARK-17042 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-16884) Move DataSourceScanExec out of ExistingRDD.scala file

2016-08-03 Thread Eric Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Liang updated SPARK-16884: --- Issue Type: Improvement (was: Bug) > Move DataSourceScanExec out of ExistingRDD.scala file >

  1   2   >