[jira] [Commented] (SPARK-30577) StorageLevel.DISK_ONLY_2 causes the data loss

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021679#comment-17021679 ] Hyukjin Kwon commented on SPARK-30577: -- Spark 2.3 is EOL. Can you try it in higher versions? Also,

[jira] [Commented] (SPARK-30580) Why can PySpark persist data only in serialised format?

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021677#comment-17021677 ] Hyukjin Kwon commented on SPARK-30580: -- Let's ask questions to mailing list. You could have a

[jira] [Resolved] (SPARK-30580) Why can PySpark persist data only in serialised format?

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30580. -- Resolution: Invalid > Why can PySpark persist data only in serialised format? >

[jira] [Resolved] (SPARK-30577) StorageLevel.DISK_ONLY_2 causes the data loss

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30577. -- Resolution: Incomplete I am resolving this as incomplete as it targets EOL release. >

[jira] [Commented] (SPARK-30473) PySpark enum subclass crashes when used inside UDF

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021695#comment-17021695 ] Hyukjin Kwon commented on SPARK-30473: -- This was fixed in the upstream master by upgrading

[jira] [Updated] (SPARK-30462) Structured Streaming _spark_metadata fills up Spark Driver memory when having lots of objects

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30462: - Priority: Major (was: Critical) > Structured Streaming _spark_metadata fills up Spark Driver

[jira] [Resolved] (SPARK-30473) PySpark enum subclass crashes when used inside UDF

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30473. -- Resolution: Cannot Reproduce > PySpark enum subclass crashes when used inside UDF >

[jira] [Resolved] (SPARK-30476) NullPointerException when Insert data to hive mongo external table by spark-sql

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30476. -- Resolution: Not A Problem So, it's an issue in MongoDB connector, as you described. It seems

[jira] [Commented] (SPARK-30488) Deadlock between block-manager-slave-async-thread-pool and spark context cleaner

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021692#comment-17021692 ] Hyukjin Kwon commented on SPARK-30488: -- Is that the only place to create? Can you show full

[jira] [Updated] (SPARK-30556) Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext

2020-01-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30556: -- Summary: Copy sparkContext.localproperties to child thread inSubqueryExec.executionContext

[jira] [Resolved] (SPARK-30483) Job History does not show pool properties table

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30483. -- Resolution: Duplicate > Job History does not show pool properties table >

[jira] [Resolved] (SPARK-30487) Hive MetaException

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30487. -- Resolution: Incomplete > Hive MetaException > --- > > Key:

[jira] [Resolved] (SPARK-30484) Job History Storage Tab does not display RDD Table

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30484. -- Resolution: Not A Problem > Job History Storage Tab does not display RDD Table >

[jira] [Resolved] (SPARK-30531) Duplicate query plan on Spark UI SQL page

2020-01-22 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30531. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27238

[jira] [Resolved] (SPARK-30550) Random pyspark-shell applications being generated

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30550. -- Resolution: Cannot Reproduce Seems like no one can reproduce this. > Random pyspark-shell

[jira] [Commented] (SPARK-30557) Add public documentation for SPARK_SUBMIT_OPTS

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021684#comment-17021684 ] Hyukjin Kwon commented on SPARK-30557: -- Yup, and {{spark.driver.extraJavaOptions}} and

[jira] [Resolved] (SPARK-30239) Creating a dataframe with Pandas rather than Numpy datatypes fails

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30239. -- Resolution: Incomplete Resolving by no feedback from reporter. > Creating a dataframe with

[jira] [Commented] (SPARK-30327) Caused by: java.lang.ArrayIndexOutOfBoundsException: -1

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021700#comment-17021700 ] Hyukjin Kwon commented on SPARK-30327: -- Can you show full, self-contained reproducer? > Caused by:

[jira] [Commented] (SPARK-30229) java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783)

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021704#comment-17021704 ] Hyukjin Kwon commented on SPARK-30229: -- [~Ankitraj] were you able to reproduce? [~SeaAndHill] can

[jira] [Resolved] (SPARK-30328) Fail to write local files with RDD.saveTextFile when setting the incorrect Hadoop configuration files

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30328. -- Resolution: Invalid > Fail to write local files with RDD.saveTextFile when setting the

[jira] [Commented] (SPARK-30328) Fail to write local files with RDD.saveTextFile when setting the incorrect Hadoop configuration files

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021699#comment-17021699 ] Hyukjin Kwon commented on SPARK-30328: -- If Hadoop configuration path is set, it should be correct.

[jira] [Commented] (SPARK-30275) Add gitlab-ci.yml file for reproducible builds

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021701#comment-17021701 ] Hyukjin Kwon commented on SPARK-30275: -- Can you send an email to the dev list and ask some

[jira] [Assigned] (SPARK-30609) Allow default merge command resolution to be bypassed by DSv2 sources

2020-01-22 Thread Tathagata Das (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das reassigned SPARK-30609: - Assignee: Tathagata Das > Allow default merge command resolution to be bypassed by

[jira] [Updated] (SPARK-29701) Different answers when empty input given in GROUPING SETS

2020-01-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29701: -- Affects Version/s: 2.4.0 2.4.1 2.4.2

[jira] [Resolved] (SPARK-30463) Move test cases for 'pandas' sub-package

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30463. -- Resolution: Later Let me take a look for this later. Fortunately, the tests are grouped in

[jira] [Resolved] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30542. -- Resolution: Invalid Please ask questions to mailing list. You could have a better answer. See

[jira] [Assigned] (SPARK-30556) SubqueryExec passes local properties to SubqueryExec.executionContext

2020-01-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30556: - Assignee: Ajith S > SubqueryExec passes local properties to

[jira] [Resolved] (SPARK-30513) Question about spark on k8s

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30513. -- Resolution: Invalid Please ask qeustions at dev mailing list or stackoverflow. See

[jira] [Updated] (SPARK-30529) Improve error messages when Executor dies before registering with driver

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30529: - Description: currently when you give a bad configuration for accelerator aware scheduling to

[jira] [Resolved] (SPARK-30526) Can I translate Spark documents into Chinese ?

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30526. -- Resolution: Won't Fix I think it's better to do it in a separate thridparty repository. Not

[jira] [Resolved] (SPARK-30556) SubqueryExec passes local properties to SubqueryExec.executionContext

2020-01-22 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30556. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27267

[jira] [Commented] (SPARK-30613) support hive style REPLACE COLUMN syntax

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021737#comment-17021737 ] Wenchen Fan commented on SPARK-30613: - Hi [~imback82] do you have time to work on it? Thanks! >

[jira] [Created] (SPARK-30613) support hive style REPLACE COLUMN syntax

2020-01-22 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30613: --- Summary: support hive style REPLACE COLUMN syntax Key: SPARK-30613 URL: https://issues.apache.org/jira/browse/SPARK-30613 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-30614) The native ALTER COLUMN syntax should change one thing at a time

2020-01-22 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021781#comment-17021781 ] Terry Kim commented on SPARK-30614: --- [~cloud_fan] Yes, I will work on this. Thanks! > The native

[jira] [Commented] (SPARK-30615) normalize the column name in AlterTable

2020-01-22 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021783#comment-17021783 ] Terry Kim commented on SPARK-30615: --- [~cloud_fan] Yes, I will work on this. Thanks! > normalize the

[jira] [Commented] (SPARK-30546) Make interval type more future-proofing

2020-01-22 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021806#comment-17021806 ] Kent Yao commented on SPARK-30546: -- Thanks [~dongjoon] > Make interval type more future-proofing >

[jira] [Created] (SPARK-30616) Introduce TTL config option for SQL Parquet Metadata Cache

2020-01-22 Thread Yaroslav Tkachenko (Jira)
Yaroslav Tkachenko created SPARK-30616: -- Summary: Introduce TTL config option for SQL Parquet Metadata Cache Key: SPARK-30616 URL: https://issues.apache.org/jira/browse/SPARK-30616 Project: Spark

[jira] [Resolved] (SPARK-30607) overlay wrappers for SparkR and PySpark

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30607. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27325

[jira] [Assigned] (SPARK-30607) overlay wrappers for SparkR and PySpark

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30607: Assignee: Maciej Szymkiewicz > overlay wrappers for SparkR and PySpark >

[jira] [Commented] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021674#comment-17021674 ] Hyukjin Kwon commented on SPARK-30590: -- Seems fixed in the master: {code} scala>

[jira] [Resolved] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30590. -- Resolution: Cannot Reproduce > can't use more than five type-safe user-defined aggregation in

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-01-22 Thread Min Shen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Min Shen updated SPARK-30602: - Description: In a large deployment of a Spark compute infrastructure, Spark shuffle is becoming a

[jira] [Comment Edited] (SPARK-30476) NullPointerException when Insert data to hive mongo external table by spark-sql

2020-01-22 Thread XiongCheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021709#comment-17021709 ] XiongCheng edited comment on SPARK-30476 at 1/23/20 2:49 AM: -

[jira] [Commented] (SPARK-30612) can't resolve qualified column name with v2 tables

2020-01-22 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021752#comment-17021752 ] Terry Kim commented on SPARK-30612: --- [~cloud_fan] Yes, I will work on this. > can't resolve qualified

[jira] [Commented] (SPARK-27871) LambdaVariable should use per-query unique IDs instead of globally unique IDs

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021765#comment-17021765 ] Wenchen Fan commented on SPARK-27871: - This is done with an optimizer rule so we added a public

[jira] [Reopened] (SPARK-30535) Migrate ALTER TABLE commands to the new resolution framework

2020-01-22 Thread Xiao Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reopened SPARK-30535: - Assignee: (was: Terry Kim) > Migrate ALTER TABLE commands to the new resolution framework >

[jira] [Updated] (SPARK-30531) Duplicate query plan on Spark UI SQL page

2020-01-22 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-30531: - Issue Type: Improvement (was: Bug) Priority: Minor (was: Major) > Duplicate query plan

[jira] [Assigned] (SPARK-30531) Duplicate query plan on Spark UI SQL page

2020-01-22 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30531?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30531: Assignee: Enrico Minack > Duplicate query plan on Spark UI SQL page >

[jira] [Commented] (SPARK-30561) start spark applications without a 30second startup penalty

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021682#comment-17021682 ] Hyukjin Kwon commented on SPARK-30561: -- Is any of them safe to remove? > start spark applications

[jira] [Commented] (SPARK-30476) NullPointerException when Insert data to hive mongo external table by spark-sql

2020-01-22 Thread XiongCheng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021709#comment-17021709 ] XiongCheng commented on SPARK-30476: [~hyukjin.kwon]Thanks for your reply~.However, my doubt is why

[jira] [Created] (SPARK-30612) can't resolve qualified column name with v2 tables

2020-01-22 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30612: --- Summary: can't resolve qualified column name with v2 tables Key: SPARK-30612 URL: https://issues.apache.org/jira/browse/SPARK-30612 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30613) support hive style REPLACE COLUMN syntax

2020-01-22 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021753#comment-17021753 ] Terry Kim commented on SPARK-30613: --- [~cloud_fan] Yes, I will work on this. Thanks! > support hive

[jira] [Comment Edited] (SPARK-30614) The native ALTER COLUMN syntax should change one thing at a time

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021761#comment-17021761 ] Wenchen Fan edited comment on SPARK-30614 at 1/23/20 4:22 AM: -- Hi

[jira] [Commented] (SPARK-30614) The native ALTER COLUMN syntax should change one thing at a time

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021761#comment-17021761 ] Wenchen Fan commented on SPARK-30614: - Hi [~imback82], do you have time to work in it? thanks! >

[jira] [Updated] (SPARK-30614) The native ALTER COLUMN syntax should change one thing at a time

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30614: Description: Our native ALTER COLUMN syntax is newly added in 3.0 and almost follows the SQL

[jira] [Created] (SPARK-30614) The native ALTER COLUMN syntax should change one thing at a time

2020-01-22 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30614: --- Summary: The native ALTER COLUMN syntax should change one thing at a time Key: SPARK-30614 URL: https://issues.apache.org/jira/browse/SPARK-30614 Project: Spark

[jira] [Commented] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021823#comment-17021823 ] Hyukjin Kwon commented on SPARK-30590: -- Ah, thanks. I rushed to read. This issue still persists in

[jira] [Updated] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30590: - Affects Version/s: 3.0.0 > can't use more than five type-safe user-defined aggregation in

[jira] [Commented] (SPARK-30049) SQL fails to parse when comment contains an unmatched quote character

2020-01-22 Thread Oleg Bonar (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021832#comment-17021832 ] Oleg Bonar commented on SPARK-30049: [~tgraves], no, i'm not.   > SQL fails to parse when comment

[jira] [Updated] (SPARK-30597) Unable to load properties fine in SparkStandalone HDFS mode

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30597: - Description: We run the spark application in Yarn HDFS/NFS/WebHDFS and standalone HDFS/NFS

[jira] [Updated] (SPARK-30597) Unable to load properties fine in SparkStandalone HDFS mode

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30597: - Description: We run the spark application in Yarn HDFS/NFS/WebHDFS and standalone HDFS/NFS

[jira] [Resolved] (SPARK-30585) scalatest fails for Apache Spark SQL project

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30585. -- Resolution: Incomplete > scalatest fails for Apache Spark SQL project >

[jira] [Commented] (SPARK-30585) scalatest fails for Apache Spark SQL project

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021675#comment-17021675 ] Hyukjin Kwon commented on SPARK-30585: -- and please just don't copy and paste the logs. You should

[jira] [Resolved] (SPARK-30609) Allow default merge command resolution to be bypassed by DSv2 sources

2020-01-22 Thread Tathagata Das (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-30609. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27326

[jira] [Commented] (SPARK-30612) can't resolve qualified column name with v2 tables

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021729#comment-17021729 ] Wenchen Fan commented on SPARK-30612: - Hi [~imback82] do you have time to work on it? Thanks >

[jira] [Commented] (SPARK-30275) Add gitlab-ci.yml file for reproducible builds

2020-01-22 Thread Jim Kleckner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021766#comment-17021766 ] Jim Kleckner commented on SPARK-30275: -- I sent a message to

[jira] [Updated] (SPARK-30360) Avoid Redact classpath entries in History Server UI

2020-01-22 Thread Ajith S (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S updated SPARK-30360: Summary: Avoid Redact classpath entries in History Server UI (was: Redact classpath entries in Spark UI)

[jira] [Updated] (SPARK-30360) Avoid Redact classpath entries in History Server UI

2020-01-22 Thread Ajith S (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajith S updated SPARK-30360: Description: Currently SPARK history server display the classpath entries in the Environment tab with

[jira] [Commented] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Daniel Mantovani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021791#comment-17021791 ] Daniel Mantovani commented on SPARK-30590: -- [~hyukjin.kwon] You tried the wrong thing, with 5

[jira] [Reopened] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Daniel Mantovani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Mantovani reopened SPARK-30590: -- [~hyukjin.kwon] You tried with 5 parameters which works, you should try with 6 to get

[jira] [Issue Comment Deleted] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Daniel Mantovani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Mantovani updated SPARK-30590: - Comment: was deleted (was: [~hyukjin.kwon] You tried the wrong thing, with 5 parameters

[jira] [Comment Edited] (SPARK-30590) can't use more than five type-safe user-defined aggregation in select statement

2020-01-22 Thread Daniel Mantovani (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021792#comment-17021792 ] Daniel Mantovani edited comment on SPARK-30590 at 1/23/20 5:48 AM: ---

[jira] [Updated] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30602: - Summary: SPIP: Support push-based shuffle to improve shuffle efficiency (was: Support

[jira] [Commented] (SPARK-30608) Postgres Column Interval converts to string and cant be written back to postgres

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021669#comment-17021669 ] Hyukjin Kwon commented on SPARK-30608: -- I don't see interval type conversions are supported between

[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021672#comment-17021672 ] Hyukjin Kwon commented on SPARK-30602: -- Can you send the email to the dev list to discuss? If you

[jira] [Resolved] (SPARK-30608) Postgres Column Interval converts to string and cant be written back to postgres

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30608. -- Resolution: Invalid > Postgres Column Interval converts to string and cant be written back to

[jira] [Resolved] (SPARK-30332) When running sql query with limit catalyst throw StackOverFlow exception

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30332?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30332. -- Resolution: Incomplete > When running sql query with limit catalyst throw StackOverFlow

[jira] [Resolved] (SPARK-30442) Write mode ignored when using CodecStreams

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30442. -- Resolution: Incomplete Resolving due to no feedback from the reporter. > Write mode ignored

[jira] [Commented] (SPARK-30444) The same job will be computated for many times when using Dataset.show()

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021697#comment-17021697 ] Hyukjin Kwon commented on SPARK-30444: -- [~aman_omer] have you made some progresses on this? > The

[jira] [Commented] (SPARK-30615) normalize the column name in AlterTable

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17021763#comment-17021763 ] Wenchen Fan commented on SPARK-30615: - Hi [~imback82], do you have time to work on it? Thanks! >

[jira] [Created] (SPARK-30615) normalize the column name in AlterTable

2020-01-22 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30615: --- Summary: normalize the column name in AlterTable Key: SPARK-30615 URL: https://issues.apache.org/jira/browse/SPARK-30615 Project: Spark Issue Type: New

[jira] [Updated] (SPARK-30615) normalize the column name in AlterTable

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30615: Description: Because of case insensitive resolution, the column name in AlterTable may match the

[jira] [Assigned] (SPARK-30601) Add a Google Maven Central as a primary repository

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30601: Assignee: Hyukjin Kwon > Add a Google Maven Central as a primary repository >

[jira] [Resolved] (SPARK-30601) Add a Google Maven Central as a primary repository

2020-01-22 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30601. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27307

[jira] [Updated] (SPARK-30016) Support ownership management for Spark SQL

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30016: Description: Currently, v2 tables/namespaces do not support ownership management. This JIRA aims

[jira] [Updated] (SPARK-30016) Support ownership management for DS v2

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30016: Summary: Support ownership management for DS v2 (was: Support ownership management for Spark

[jira] [Updated] (SPARK-30546) Make interval type more future-proofing

2020-01-22 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-30546: - Description: Before 3.0 we may make some efforts for the current interval type to make it more

[jira] [Updated] (SPARK-30592) Interval type checker for csv and json functions

2020-01-22 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-30592: - Description: to_csv from_csv to_json from_json was:to_csv should not support output intervals as

[jira] [Updated] (SPARK-30019) Add the owner property to v2 table

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30019?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30019: Summary: Add the owner property to v2 table (was: Support ALTER TABLE SET OWNER syntax) > Add

[jira] [Updated] (SPARK-30018) Add the owner property to v2 namespace

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30018: Summary: Add the owner property to v2 namespace (was: Support ALTER DATABASE SET OWNER syntax)

[jira] [Updated] (SPARK-30016) Support ownership for DS v2 tables/namespaces

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-30016: Summary: Support ownership for DS v2 tables/namespaces (was: Support ownership management for DS

[jira] [Created] (SPARK-30603) Keep the reserved properties of namespaces and tables private

2020-01-22 Thread Kent Yao (Jira)
Kent Yao created SPARK-30603: Summary: Keep the reserved properties of namespaces and tables private Key: SPARK-30603 URL: https://issues.apache.org/jira/browse/SPARK-30603 Project: Spark Issue

[jira] [Resolved] (SPARK-30591) Remove the nonstandard SET OWNER syntax for namespaces

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30591. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 27300

[jira] [Assigned] (SPARK-30591) Remove the nonstandard SET OWNER syntax for namespaces

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30591: --- Assignee: Kent Yao > Remove the nonstandard SET OWNER syntax for namespaces >

[jira] [Commented] (SPARK-27951) ANSI SQL: NTH_VALUE function

2020-01-22 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020895#comment-17020895 ] jiaan.geng commented on SPARK-27951: I'm working on. > ANSI SQL: NTH_VALUE function >

[jira] [Commented] (SPARK-28880) ANSI SQL: Bracketed comments

2020-01-22 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020894#comment-17020894 ] jiaan.geng commented on SPARK-28880: I'm working on. > ANSI SQL: Bracketed comments >

[jira] [Comment Edited] (SPARK-30528) DPP issues

2020-01-22 Thread Mayur Bhosale (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17020803#comment-17020803 ] Mayur Bhosale edited comment on SPARK-30528 at 1/22/20 8:59 AM: Thanks

[jira] [Deleted] (SPARK-30021) set ownerName and owner type as reserved properties to support v2 catalog

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan deleted SPARK-30021: > set ownerName and owner type as reserved properties to support v2 catalog >

[jira] [Deleted] (SPARK-30017) ownerName and ownerType support as properties to databases

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan deleted SPARK-30017: > ownerName and ownerType support as properties to databases >

[jira] [Deleted] (SPARK-30020) ownerName and ownerType support as properties to tables

2020-01-22 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan deleted SPARK-30020: > ownerName and ownerType support as properties to tables >

  1   2   3   >