[jira] [Resolved] (SPARK-46486) DataSourceV2: Restore createOrReplaceView

2023-12-22 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-46486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved SPARK-46486. Resolution: Not A Problem > DataSourceV2: Restore createOrReplaceView > --

[jira] [Created] (SPARK-46486) DataSourceV2: Restore createOrReplaceView

2023-12-21 Thread John Zhuge (Jira)
John Zhuge created SPARK-46486: -- Summary: DataSourceV2: Restore createOrReplaceView Key: SPARK-46486 URL: https://issues.apache.org/jira/browse/SPARK-46486 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-39911) Optimize global Sort to RepartitionByExpression

2023-12-01 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-39911: --- Fix Version/s: 3.3.1 > Optimize global Sort to RepartitionByExpression > ---

[jira] [Resolved] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved SPARK-45657. Fix Version/s: 3.5.0 Resolution: Fixed The issue is fixed in 3.5.0 > Caching SQL UNION of

[jira] [Updated] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-45657: --- Affects Version/s: 3.4.1 3.4.0 > Caching SQL UNION of different column data t

[jira] [Comment Edited] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge edited comment on SPARK-45657 at 10/25/23 4:55 AM:

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779326#comment-17779326 ] John Zhuge commented on SPARK-45657: It is fixed in main branch {code:java}       __

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779283#comment-17779283 ] John Zhuge commented on SPARK-45657: Interesting, there is warning in Dataset.union

[jira] [Updated] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-45657: --- Affects Version/s: 3.3.2 (was: 3.4.1) > Caching SQL UNION of different co

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779282#comment-17779282 ] John Zhuge commented on SPARK-45657: Checking whether this is still an issue in main

[jira] [Comment Edited] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge edited comment on SPARK-45657 at 10/25/23 12:38 AM: ---

[jira] [Comment Edited] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge edited comment on SPARK-45657 at 10/25/23 12:36 AM: ---

[jira] [Commented] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-45657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17779281#comment-17779281 ] John Zhuge commented on SPARK-45657: Root cause: # SQL UNION of 2 sides with differ

[jira] [Created] (SPARK-45657) Caching SQL UNION of different column data types does not work inside Dataset.union

2023-10-24 Thread John Zhuge (Jira)
John Zhuge created SPARK-45657: -- Summary: Caching SQL UNION of different column data types does not work inside Dataset.union Key: SPARK-45657 URL: https://issues.apache.org/jira/browse/SPARK-45657 Proje

[jira] [Commented] (SPARK-43288) DataSourceV2: CREATE TABLE LIKE

2023-05-05 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719925#comment-17719925 ] John Zhuge commented on SPARK-43288: TODO: fix a few unit tests > DataSourceV2: CRE

[jira] [Commented] (SPARK-43288) DataSourceV2: CREATE TABLE LIKE

2023-05-05 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17719924#comment-17719924 ] John Zhuge commented on SPARK-43288: Please review the WIP PR: https://github.com/ap

[jira] [Updated] (SPARK-43288) DataSourceV2: CREATE TABLE LIKE

2023-04-26 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-43288: --- Issue Type: Improvement (was: New Feature) > DataSourceV2: CREATE TABLE LIKE >

[jira] [Updated] (SPARK-43288) DataSourceV2: CREATE TABLE LIKE

2023-04-26 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-43288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-43288: --- Priority: Major (was: Minor) > DataSourceV2: CREATE TABLE LIKE > --- >

[jira] [Created] (SPARK-43288) DataSourceV2: CREATE TABLE LIKE

2023-04-25 Thread John Zhuge (Jira)
John Zhuge created SPARK-43288: -- Summary: DataSourceV2: CREATE TABLE LIKE Key: SPARK-43288 URL: https://issues.apache.org/jira/browse/SPARK-43288 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-42613) PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default

2023-02-27 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42613: --- Description: Follow up from [https://github.com/apache/spark/pull/40199#discussion_r1119453996] If

[jira] [Updated] (SPARK-42613) PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default

2023-02-27 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42613: --- Summary: PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default

[jira] [Updated] (SPARK-42613) PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default

2023-02-27 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42613: --- Description: Follow up from [https://github.com/apache/spark/pull/40199#discussion_r1119453996]  

[jira] [Updated] (SPARK-42613) PythonRunner should set OMP_NUM_THREADS to task cpus times executor cores by default

2023-02-27 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42613: --- Description: Follow up from [https://github.com/apache/spark/pull/40199#discussion_r1119453996]  

[jira] [Updated] (SPARK-42613) PythonRunner should set OMP_NUM_THREADS to task cpus times executor cores by default

2023-02-27 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42613: --- Summary: PythonRunner should set OMP_NUM_THREADS to task cpus times executor cores by default (was:

[jira] [Updated] (SPARK-42613) PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default

2023-02-27 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42613: --- Description: Follow up from [https://github.com/apache/spark/pull/40199#discussion_r1119453996]  

[jira] [Created] (SPARK-42613) PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default

2023-02-27 Thread John Zhuge (Jira)
John Zhuge created SPARK-42613: -- Summary: PythonRunner should set OMP_NUM_THREADS to task cpus instead of executor cores by default Key: SPARK-42613 URL: https://issues.apache.org/jira/browse/SPARK-42613

[jira] [Created] (SPARK-42607) [MESOS] OMP_NUM_THREADS not set to number of executor cores by default

2023-02-27 Thread John Zhuge (Jira)
John Zhuge created SPARK-42607: -- Summary: [MESOS] OMP_NUM_THREADS not set to number of executor cores by default Key: SPARK-42607 URL: https://issues.apache.org/jira/browse/SPARK-42607 Project: Spark

[jira] [Updated] (SPARK-42596) [YARN] OMP_NUM_THREADS not set to number of executor cores by default

2023-02-26 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42596: --- Description: Run this PySpark script with `spark.executor.cores=1` {code:python} import os from pysp

[jira] [Updated] (SPARK-42596) [YARN] OMP_NUM_THREADS not set to number of executor cores by default

2023-02-26 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42596: --- Description: Run this PySpark script with `spark.executor.cores=1` {code:python} import os from pysp

[jira] [Commented] (SPARK-42596) [YARN] OMP_NUM_THREADS not set to number of executor cores by default

2023-02-26 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17693837#comment-17693837 ] John Zhuge commented on SPARK-42596: Looks like a regression from SPARK-41188 where

[jira] [Created] (SPARK-42596) [YARN] OMP_NUM_THREADS not set to number of executor cores by default

2023-02-26 Thread John Zhuge (Jira)
John Zhuge created SPARK-42596: -- Summary: [YARN] OMP_NUM_THREADS not set to number of executor cores by default Key: SPARK-42596 URL: https://issues.apache.org/jira/browse/SPARK-42596 Project: Spark

[jira] [Updated] (SPARK-42036) Kryo ClassCastException getting task result when JDK versions mismatch

2023-01-12 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42036: --- Description: {noformat} 22/12/21 01:27:12 ERROR TaskResultGetter: Exception while getting task resul

[jira] [Updated] (SPARK-42036) Kryo ClassCastException getting task result when JDK versions mismatch

2023-01-12 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-42036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-42036: --- Summary: Kryo ClassCastException getting task result when JDK versions mismatch (was: Kryo ClassCas

[jira] [Created] (SPARK-42036) Kryo ClassCastException getting task result when JDK version mismatch

2023-01-12 Thread John Zhuge (Jira)
John Zhuge created SPARK-42036: -- Summary: Kryo ClassCastException getting task result when JDK version mismatch Key: SPARK-42036 URL: https://issues.apache.org/jira/browse/SPARK-42036 Project: Spark

[jira] [Commented] (SPARK-41519) Pin versions-maven-plugin version

2022-12-14 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-41519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17647711#comment-17647711 ] John Zhuge commented on SPARK-41519: {noformat} [ERROR] java.nio.file.NoSuchFileExce

[jira] [Created] (SPARK-41519) Pin versions-maven-plugin version

2022-12-14 Thread John Zhuge (Jira)
John Zhuge created SPARK-41519: -- Summary: Pin versions-maven-plugin version Key: SPARK-41519 URL: https://issues.apache.org/jira/browse/SPARK-41519 Project: Spark Issue Type: Task Comp

[jira] [Updated] (SPARK-39800) DataSourceV2: View support

2022-07-17 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-39800: --- Description: Support Data source V2 views. (was: Data source V2 view substitution and resolution.)

[jira] [Updated] (SPARK-39800) DataSourceV2: View support

2022-07-17 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-39800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-39800: --- Summary: DataSourceV2: View support (was: DataSourceV2: View substitution and resolution) > DataSo

[jira] [Created] (SPARK-39800) DataSourceV2: View substitution and resolution

2022-07-17 Thread John Zhuge (Jira)
John Zhuge created SPARK-39800: -- Summary: DataSourceV2: View substitution and resolution Key: SPARK-39800 URL: https://issues.apache.org/jira/browse/SPARK-39800 Project: Spark Issue Type: Sub-ta

[jira] [Created] (SPARK-39799) DataSourceV2: View catalog interface

2022-07-17 Thread John Zhuge (Jira)
John Zhuge created SPARK-39799: -- Summary: DataSourceV2: View catalog interface Key: SPARK-39799 URL: https://issues.apache.org/jira/browse/SPARK-39799 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-31357) DataSourceV2: Catalog API for view metadata

2022-02-10 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-31357: --- Target Version/s: 3.3.0 (was: 3.2.0) > DataSourceV2: Catalog API for view metadata > --

[jira] [Updated] (SPARK-31357) DataSourceV2: Catalog API for view metadata

2022-02-10 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-31357: --- Summary: DataSourceV2: Catalog API for view metadata (was: SPIP: Catalog API for view metadata) >

[jira] [Commented] (SPARK-36664) Log time spent waiting for cluster resources

2021-09-28 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-36664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421824#comment-17421824 ] John Zhuge commented on SPARK-36664: This can be very useful. For example, we'd like

[jira] [Commented] (SPARK-25929) Support metrics with tags

2020-05-02 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17098247#comment-17098247 ] John Zhuge commented on SPARK-25929: Has anyone looked at https://micrometer.io/? Fr

[jira] [Updated] (SPARK-31357) SPIP: Catalog API for view metadata

2020-04-06 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-31357: --- Description: SPARK-24252 added a catalog plugin system and `TableCatalog` API that provided table m

[jira] [Updated] (SPARK-31357) SPIP: Catalog API for view metadata

2020-04-06 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-31357: --- Description: SPARK-24252 added a catalog plugin system and `TableCatalog` API that provided table m

[jira] [Updated] (SPARK-31357) SPIP: Catalog API for view metadata

2020-04-06 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-31357: --- Summary: SPIP: Catalog API for view metadata (was: Catalog API for View Metadata) > SPIP: Catalog

[jira] [Updated] (SPARK-31357) Catalog API for View Metadata

2020-04-06 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-31357: --- Labels: SPIP (was: ) > Catalog API for View Metadata > - > >

[jira] [Created] (SPARK-31357) Catalog API for View Metadata

2020-04-05 Thread John Zhuge (Jira)
John Zhuge created SPARK-31357: -- Summary: Catalog API for View Metadata Key: SPARK-31357 URL: https://issues.apache.org/jira/browse/SPARK-31357 Project: Spark Issue Type: Improvement C

[jira] [Commented] (SPARK-25929) Support metrics with tags

2020-02-11 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17035074#comment-17035074 ] John Zhuge commented on SPARK-25929: Yeah, I can feel the pain. When I ingest into

[jira] [Commented] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-04 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16988078#comment-16988078 ] John Zhuge commented on SPARK-30118: [~cltlfcjin] Thanks for the comment. Do you kno

[jira] [Commented] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987429#comment-16987429 ] John Zhuge commented on SPARK-30118: I am running Spark master with Hive 1.2.1. Same

[jira] [Commented] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16987428#comment-16987428 ] John Zhuge commented on SPARK-30118: {code:java} spark-sql> DESC FORMATTED jzhuge.v1

[jira] [Created] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-03 Thread John Zhuge (Jira)
John Zhuge created SPARK-30118: -- Summary: ALTER VIEW QUERY does not work Key: SPARK-30118 URL: https://issues.apache.org/jira/browse/SPARK-30118 Project: Spark Issue Type: Bug Componen

[jira] [Updated] (SPARK-29030) Simplify lookupV2Relation

2019-09-09 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-29030: --- Description: Simplify the return type for {{lookupV2Relation}} which makes the 3 callers more straig

[jira] [Created] (SPARK-29030) Simplify lookupV2Relation

2019-09-09 Thread John Zhuge (Jira)
John Zhuge created SPARK-29030: -- Summary: Simplify lookupV2Relation Key: SPARK-29030 URL: https://issues.apache.org/jira/browse/SPARK-29030 Project: Spark Issue Type: Improvement Compo

[jira] [Created] (SPARK-28640) Only give warning when session catalog is not defined

2019-08-06 Thread John Zhuge (JIRA)
John Zhuge created SPARK-28640: -- Summary: Only give warning when session catalog is not defined Key: SPARK-28640 URL: https://issues.apache.org/jira/browse/SPARK-28640 Project: Spark Issue Type:

[jira] [Created] (SPARK-28565) DataSourceV2: DataFrameWriter.saveAsTable

2019-07-30 Thread John Zhuge (JIRA)
John Zhuge created SPARK-28565: -- Summary: DataSourceV2: DataFrameWriter.saveAsTable Key: SPARK-28565 URL: https://issues.apache.org/jira/browse/SPARK-28565 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-28565) DataSourceV2: DataFrameWriter.saveAsTable

2019-07-30 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-28565: --- Description: Support multiple catalogs in the following use cases: * DataFrameWriter.saveAsTable("c

[jira] [Updated] (SPARK-28178) DataSourceV2: DataFrameWriter.insertInfo

2019-06-26 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-28178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-28178: --- Description: Support multiple catalogs in the following InsertInto use cases: * DataFrameWriter.ins

[jira] [Created] (SPARK-28178) DataSourceV2: DataFrameWriter.insertInfo

2019-06-26 Thread John Zhuge (JIRA)
John Zhuge created SPARK-28178: -- Summary: DataSourceV2: DataFrameWriter.insertInfo Key: SPARK-28178 URL: https://issues.apache.org/jira/browse/SPARK-28178 Project: Spark Issue Type: Improvement

[jira] [Updated] (SPARK-27845) DataSourceV2: InsertTable

2019-06-26 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27845: --- Description: Support multiple catalogs in the following InsertInto use cases: * INSERT INTO [TABLE]

[jira] [Updated] (SPARK-27845) DataSourceV2: InsertTable

2019-06-26 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27845: --- Description: Support multiple catalogs in the following use cases: * INSERT INTO [TABLE] catalog.db

[jira] [Updated] (SPARK-27845) DataSourceV2: InsertTable

2019-06-10 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27845: --- Summary: DataSourceV2: InsertTable (was: DataSourceV2: Insert into tables in multiple catalogs) >

[jira] [Updated] (SPARK-27322) DataSourceV2 table relation

2019-06-10 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Summary: DataSourceV2 table relation (was: DataSourceV2: Select from multiple catalogs) > DataSour

[jira] [Updated] (SPARK-27845) DataSourceV2: Insert into tables in multiple catalogs

2019-06-10 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27845: --- Summary: DataSourceV2: Insert into tables in multiple catalogs (was: DataSourceV2: InsertInto multi

[jira] [Updated] (SPARK-27961) DataSourceV2Relation should not have refresh method

2019-06-05 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27961: --- Description: The newly added `Refresh` method in [PR #24401|https://github.com/apache/spark/pull/24

[jira] [Commented] (SPARK-27961) DataSourceV2Relation should not have refresh method

2019-06-05 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16856982#comment-16856982 ] John Zhuge commented on SPARK-27961: [~Gengliang.Wang] [~cloud_fan] Could you help?

[jira] [Created] (SPARK-27961) DataSourceV2Relation should not have refresh method

2019-06-05 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27961: -- Summary: DataSourceV2Relation should not have refresh method Key: SPARK-27961 URL: https://issues.apache.org/jira/browse/SPARK-27961 Project: Spark Issue Type: B

[jira] [Updated] (SPARK-27947) ParsedStatement subclass toString may throw ClassCastException

2019-06-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27947: --- Description: In ParsedStatement.productIterator, `case mapArg: Map[_, _]` may match any Map type, t

[jira] [Updated] (SPARK-27947) ParsedStatement subclass toString may throw ClassCastException

2019-06-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27947: --- Description: In ParsedStatement.productIterator, `case mapArg: Map[_, _]` may match any Map type, t

[jira] [Updated] (SPARK-27947) ParsedStatement subclass toString may throw ClassCastException

2019-06-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27947: --- Description: In ParsedStatement.productIterator, `case mapArg: Map[_, _]` may match any Map type, t

[jira] [Updated] (SPARK-27947) ParsedStatement subclass toString may throw ClassCastException

2019-06-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27947: --- Description: In ParsedStatement.productIterator, "case mapArg: Map[_, _]" may match any Map type, t

[jira] [Created] (SPARK-27947) ParsedStatement subclass toString may throw ClassCastException

2019-06-04 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27947: -- Summary: ParsedStatement subclass toString may throw ClassCastException Key: SPARK-27947 URL: https://issues.apache.org/jira/browse/SPARK-27947 Project: Spark I

[jira] [Updated] (SPARK-27947) ParsedStatement subclass toString may throw ClassCastException

2019-06-04 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27947: --- Description: In ParsedStatement.productIterator, "case mapArg: Map[_, _]" may match any Map type, t

[jira] [Created] (SPARK-27845) DataSourceV2: InsertInto multiple catalogs

2019-05-26 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27845: -- Summary: DataSourceV2: InsertInto multiple catalogs Key: SPARK-27845 URL: https://issues.apache.org/jira/browse/SPARK-27845 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-27322) DataSourceV2: Select from multiple catalogs

2019-05-25 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Summary: DataSourceV2: Select from multiple catalogs (was: DataSourceV2: Logical relation in multip

[jira] [Updated] (SPARK-27322) DataSourceV2: Logical relation in multiple catalogs

2019-05-23 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Summary: DataSourceV2: Logical relation in multiple catalogs (was: Select from multiple catalogs)

[jira] [Created] (SPARK-27813) DataSourceV2: Add DropTable logical operation

2019-05-23 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27813: -- Summary: DataSourceV2: Add DropTable logical operation Key: SPARK-27813 URL: https://issues.apache.org/jira/browse/SPARK-27813 Project: Spark Issue Type: Improve

[jira] [Updated] (SPARK-27322) Select from multiple catalogs

2019-05-22 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Description: Support multi-catalog in the following SELECT code paths: * SELECT * FROM catalog.db.t

[jira] [Updated] (SPARK-27322) Select from multiple catalogs

2019-05-19 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Summary: Select from multiple catalogs (was: SELECT from multiple catalogs) > Select from multiple

[jira] [Updated] (SPARK-27739) df.persist should save stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27739: --- Summary: df.persist should save stats from optimized plan (was: Persist should use stats from optim

[jira] [Updated] (SPARK-27739) Persist should use stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27739: --- Summary: Persist should use stats from optimized plan (was: CacheManager.cacheQuery should copy sta

[jira] [Updated] (SPARK-27739) Persist should use stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27739: --- Description: CacheManager.cacheQuery passes the stats for `planToCache` to InMemoryRelation. Since

[jira] [Created] (SPARK-27739) CacheManager.cacheQuery should copy stats from optimized plan

2019-05-15 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27739: -- Summary: CacheManager.cacheQuery should copy stats from optimized plan Key: SPARK-27739 URL: https://issues.apache.org/jira/browse/SPARK-27739 Project: Spark Is

[jira] [Updated] (SPARK-27322) SELECT from multiple catalogs

2019-03-29 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Summary: SELECT from multiple catalogs (was: Support SELECT from multiple catalogs) > SELECT from

[jira] [Updated] (SPARK-27322) Support SELECT from multiple catalogs

2019-03-29 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Summary: Support SELECT from multiple catalogs (was: Support multi-catalog in SELECT) > Support SE

[jira] [Created] (SPARK-27322) Support multi-catalog in SELECT

2019-03-29 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27322: -- Summary: Support multi-catalog in SELECT Key: SPARK-27322 URL: https://issues.apache.org/jira/browse/SPARK-27322 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-27250) Scala 2.11 maven compile should target Java 1.8

2019-03-22 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27250: --- Description: Discovered by https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboa

[jira] [Updated] (SPARK-27251) SPARK-25196 Scala 2.1.1 maven build failure

2019-03-22 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27251: --- Description: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-m

[jira] [Created] (SPARK-27251) SPARK-25196 Scala 2.1.1 maven build failure

2019-03-22 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27251: -- Summary: SPARK-25196 Scala 2.1.1 maven build failure Key: SPARK-27251 URL: https://issues.apache.org/jira/browse/SPARK-27251 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27250) Scala 2.11 maven compile should target Java 1.8

2019-03-22 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27250: --- Description: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-m

[jira] [Created] (SPARK-27250) Scala 2.11 maven compile should target Java 1.8

2019-03-22 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27250: -- Summary: Scala 2.11 maven compile should target Java 1.8 Key: SPARK-27250 URL: https://issues.apache.org/jira/browse/SPARK-27250 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-20314) Inconsistent error handling in JSON parsing SQL functions

2019-02-21 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge resolved SPARK-20314. Resolution: Duplicate Resolve it as duplicate. > Inconsistent error handling in JSON parsing SQL

[jira] [Updated] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-26946: --- Component/s: (was: Spark Core) > Identifiers for multi-catalog Spark > -

[jira] [Updated] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-26946: --- Description: Propose semantics for identifiers and a listing API to support multiple catalogs. [~r

[jira] [Created] (SPARK-26946) Identifiers for multi-catalog Spark

2019-02-20 Thread John Zhuge (JIRA)
John Zhuge created SPARK-26946: -- Summary: Identifiers for multi-catalog Spark Key: SPARK-26946 URL: https://issues.apache.org/jira/browse/SPARK-26946 Project: Spark Issue Type: New Feature

[jira] [Updated] (SPARK-26576) Broadcast hint not applied to partitioned table

2019-01-09 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-26576: --- Summary: Broadcast hint not applied to partitioned table (was: Broadcast hint not applied to partit

[jira] [Updated] (SPARK-26576) Broadcast hint not applied to partitioned Parquet table

2019-01-09 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-26576: --- Affects Version/s: 2.2.2 > Broadcast hint not applied to partitioned Parquet table > ---

[jira] [Comment Edited] (SPARK-26576) Broadcast hint not applied to partitioned Parquet table

2019-01-09 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16737964#comment-16737964 ] John Zhuge edited comment on SPARK-26576 at 1/9/19 5:12 PM:

  1   2   >