[jira] [Updated] (SPARK-23892) Improve coverage and fix lint error in UTF8String-related Suite

[ https://issues.apache.org/jira/browse/SPARK-23892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23892: - Description: The following code in {{UTF8StringSuite}} has no sense. {code}

[jira] [Updated] (SPARK-23892) Improve coverage and fix lint error in UTF8String-related Suite

[ https://issues.apache.org/jira/browse/SPARK-23892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23892: - Description: The following code in {{UTF8StringSuite}} has no sense. {code}

[jira] [Updated] (SPARK-23892) Improve coverage and fix lint error in UTF8String-related Suite

[ https://issues.apache.org/jira/browse/SPARK-23892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kazuaki Ishizaki updated SPARK-23892: - Summary: Improve coverage and fix lint error in UTF8String-related Suite (was: Improve

[jira] [Created] (SPARK-23892) Improve coverage and fix lint error in UTF8StringSuite

Kazuaki Ishizaki created SPARK-23892: Summary: Improve coverage and fix lint error in UTF8StringSuite Key: SPARK-23892 URL: https://issues.apache.org/jira/browse/SPARK-23892 Project: Spark

[jira] [Updated] (SPARK-23815) Spark writer dynamic partition overwrite mode fails to write output on multi level partition

[ https://issues.apache.org/jira/browse/SPARK-23815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fangshi Li updated SPARK-23815: --- Priority: Major (was: Minor) > Spark writer dynamic partition overwrite mode fails to write output

[jira] [Commented] (SPARK-18092) add type cast to avoid error "Column prediction must be of type DoubleType but was actually FloatType"

[ https://issues.apache.org/jira/browse/SPARK-18092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429134#comment-16429134 ] Joseph K. Bradley commented on SPARK-18092: --- Can you please add a description and make the

[jira] [Commented] (SPARK-23883) Error with conversion to arrow while using pandas_udf

[ https://issues.apache.org/jira/browse/SPARK-23883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429123#comment-16429123 ] Bryan Cutler commented on SPARK-23883: -- I think the problem might be that since the {{pandas_udf}}

[jira] [Updated] (SPARK-23871) add python api for VectorAssembler handleInvalid

[ https://issues.apache.org/jira/browse/SPARK-23871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-23871: - Component/s: PySpark > add python api for VectorAssembler handleInvalid >

[jira] [Created] (SPARK-23891) Debian based Dockerfile

Sercan Karaoglu created SPARK-23891: --- Summary: Debian based Dockerfile Key: SPARK-23891 URL: https://issues.apache.org/jira/browse/SPARK-23891 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-23828) PySpark StringIndexerModel should have constructor from labels

[ https://issues.apache.org/jira/browse/SPARK-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler reassigned SPARK-23828: Assignee: Huaxin Gao > PySpark StringIndexerModel should have constructor from labels >

[jira] [Resolved] (SPARK-23828) PySpark StringIndexerModel should have constructor from labels

[ https://issues.apache.org/jira/browse/SPARK-23828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler resolved SPARK-23828. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20968

[jira] [Commented] (SPARK-16630) Blacklist a node if executors won't launch on it.

[ https://issues.apache.org/jira/browse/SPARK-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428721#comment-16428721 ] Attila Zsolt Piros commented on SPARK-16630: Yes you are right more executors can be

[jira] [Updated] (SPARK-23751) Kolmogorov-Smirnoff test Python API in pyspark.ml

[ https://issues.apache.org/jira/browse/SPARK-23751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley updated SPARK-23751: -- Shepherd: Joseph K. Bradley > Kolmogorov-Smirnoff test Python API in pyspark.ml >

[jira] [Resolved] (SPARK-23859) Initial PR for Instrumentation improvements: UUID and logging levels

[ https://issues.apache.org/jira/browse/SPARK-23859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joseph K. Bradley resolved SPARK-23859. --- Resolution: Fixed Fix Version/s: 2.4.0 Resolved with

[jira] [Updated] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: Description: As part of SPARK-14118, Spark SQL removed support for sending ALTER TABLE CHANGE

[jira] [Commented] (SPARK-23686) Make better usage of org.apache.spark.ml.util.Instrumentation

[ https://issues.apache.org/jira/browse/SPARK-23686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428612#comment-16428612 ] Joseph K. Bradley commented on SPARK-23686: --- I wanted to ping some other active MLlib

[jira] [Updated] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

[ https://issues.apache.org/jira/browse/SPARK-23890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated SPARK-23890: Description: As part of SPARK-14118, Spark SQL removed support for sending ALTER TABLE CHANGE

[jira] [Commented] (SPARK-22390) Aggregate push down

[ https://issues.apache.org/jira/browse/SPARK-22390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428585#comment-16428585 ] Xiao Li commented on SPARK-22390: - [~huaxingao] It is hard to say whether the design is good or not

[jira] [Created] (SPARK-23890) Hive ALTER TABLE CHANGE COLUMN for struct type no longer works

Andrew Otto created SPARK-23890: --- Summary: Hive ALTER TABLE CHANGE COLUMN for struct type no longer works Key: SPARK-23890 URL: https://issues.apache.org/jira/browse/SPARK-23890 Project: Spark

[jira] [Resolved] (SPARK-23882) Is UTF8StringSuite.writeToOutputStreamUnderflow() supported?

[ https://issues.apache.org/jira/browse/SPARK-23882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell resolved SPARK-23882. --- Resolution: Fixed Assignee: Kazuaki Ishizaki Fix Version/s: 2.4.0 >

[jira] [Resolved] (SPARK-23878) unable to import col() or lit()

[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23878. -- Resolution: Not A Problem > unable to import col() or lit() > ---

[jira] [Updated] (SPARK-19480) Higher order functions in SQL

[ https://issues.apache.org/jira/browse/SPARK-19480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19480: Target Version/s: 2.4.0, 3.0.0 > Higher order functions in SQL > - > >

[jira] [Created] (SPARK-23889) Add interfaces to pass required sorting and clustering for writes

Ryan Blue created SPARK-23889: - Summary: Add interfaces to pass required sorting and clustering for writes Key: SPARK-23889 URL: https://issues.apache.org/jira/browse/SPARK-23889 Project: Spark

[jira] [Updated] (SPARK-23889) DataSourceV2: Add interfaces to pass required sorting and clustering for writes

[ https://issues.apache.org/jira/browse/SPARK-23889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-23889: -- Summary: DataSourceV2: Add interfaces to pass required sorting and clustering for writes (was: Add

[jira] [Updated] (SPARK-22231) Support of map, filter, withColumn, dropColumn in nested list of structures

[ https://issues.apache.org/jira/browse/SPARK-22231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22231: Target Version/s: 2.4.0, 3.0.0 > Support of map, filter, withColumn, dropColumn in nested list of

[jira] [Commented] (SPARK-22391) add `MetadataCreationSupport` trait to separate data and metadata handling at write path

[ https://issues.apache.org/jira/browse/SPARK-22391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428552#comment-16428552 ] Ryan Blue commented on SPARK-22391: --- [~cloud_fan], can you explain what you think should be done for

[jira] [Commented] (SPARK-23866) Extend ALTER TABLE DROP PARTITION syntax to use all comparators

[ https://issues.apache.org/jira/browse/SPARK-23866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428537#comment-16428537 ] Apache Spark commented on SPARK-23866: -- User 'mgaido91' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23866) Extend ALTER TABLE DROP PARTITION syntax to use all comparators

[ https://issues.apache.org/jira/browse/SPARK-23866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23866: Assignee: Apache Spark > Extend ALTER TABLE DROP PARTITION syntax to use all comparators

[jira] [Assigned] (SPARK-23866) Extend ALTER TABLE DROP PARTITION syntax to use all comparators

[ https://issues.apache.org/jira/browse/SPARK-23866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23866: Assignee: (was: Apache Spark) > Extend ALTER TABLE DROP PARTITION syntax to use all

[jira] [Assigned] (SPARK-23861) Clarify behavior of default window frame boundaries with and without orderBy clause

[ https://issues.apache.org/jira/browse/SPARK-23861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-23861: Assignee: Li Jin > Clarify behavior of default window frame boundaries with and without

[jira] [Resolved] (SPARK-23861) Clarify behavior of default window frame boundaries with and without orderBy clause

[ https://issues.apache.org/jira/browse/SPARK-23861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-23861. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 20978

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428517#comment-16428517 ] Andrew Davidson commented on SPARK-23878: - yes please mark resolved   > unable to import col()

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428509#comment-16428509 ] Andrew Davidson commented on SPARK-23878: - added screen shot to 

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428508#comment-16428508 ] Hyukjin Kwon commented on SPARK-23878: -- Thanks. these details must be helpful if other guys face the

[jira] [Commented] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428503#comment-16428503 ] Apache Spark commented on SPARK-23888: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23888: Assignee: Apache Spark > speculative task should not run on a given host where another

[jira] [Assigned] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

[ https://issues.apache.org/jira/browse/SPARK-23888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23888: Assignee: (was: Apache Spark) > speculative task should not run on a given host where

[jira] [Commented] (SPARK-23878) unable to import col() or lit()

[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428494#comment-16428494 ] Andrew Davidson commented on SPARK-23878: - Hi Hyukjin many thanks! I am not a python expert. I

[jira] [Created] (SPARK-23888) speculative task should not run on a given host where another attempt is already running on

wuyi created SPARK-23888: Summary: speculative task should not run on a given host where another attempt is already running on Key: SPARK-23888 URL: https://issues.apache.org/jira/browse/SPARK-23888 Project:

[jira] [Updated] (SPARK-23878) unable to import col() or lit()

[ https://issues.apache.org/jira/browse/SPARK-23878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Davidson updated SPARK-23878: Attachment: eclipsePyDevPySparkConfig.png > unable to import col() or lit() >

[jira] [Resolved] (SPARK-23822) Improve error message for Parquet schema mismatches

[ https://issues.apache.org/jira/browse/SPARK-23822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23822. - Resolution: Fixed Assignee: Yuchen Huo Fix Version/s: 2.4.0 2.3.1 >

[jira] [Created] (SPARK-23887) update query progress

Jose Torres created SPARK-23887: --- Summary: update query progress Key: SPARK-23887 URL: https://issues.apache.org/jira/browse/SPARK-23887 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-23886) update query.status

Jose Torres created SPARK-23886: --- Summary: update query.status Key: SPARK-23886 URL: https://issues.apache.org/jira/browse/SPARK-23886 Project: Spark Issue Type: Sub-task Components:

[jira] [Updated] (SPARK-23816) FetchFailedException when killing speculative task

[ https://issues.apache.org/jira/browse/SPARK-23816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid updated SPARK-23816: - Labels: speculation (was: ) > FetchFailedException when killing speculative task >

[jira] [Commented] (SPARK-19185) ConcurrentModificationExceptions with CachedKafkaConsumers when Windowing

[ https://issues.apache.org/jira/browse/SPARK-19185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428321#comment-16428321 ] Apache Spark commented on SPARK-19185: -- User 'gaborgsomogyi' has created a pull request for this

[jira] [Created] (SPARK-23885) trying to spark submit 2.3.0 on minikube

anant pukale created SPARK-23885: Summary: trying to spark submit 2.3.0 on minikube Key: SPARK-23885 URL: https://issues.apache.org/jira/browse/SPARK-23885 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-23884) hasLaunchedTask should be true when launchedAnyTask be true

[ https://issues.apache.org/jira/browse/SPARK-23884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428188#comment-16428188 ] Apache Spark commented on SPARK-23884: -- User 'Ngone51' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23884) hasLaunchedTask should be true when launchedAnyTask be true

[ https://issues.apache.org/jira/browse/SPARK-23884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23884: Assignee: (was: Apache Spark) > hasLaunchedTask should be true when launchedAnyTask

[jira] [Assigned] (SPARK-23884) hasLaunchedTask should be true when launchedAnyTask be true

[ https://issues.apache.org/jira/browse/SPARK-23884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23884: Assignee: Apache Spark > hasLaunchedTask should be true when launchedAnyTask be true >

[jira] [Created] (SPARK-23884) hasLaunchedTask should be true when launchedAnyTask be true

wuyi created SPARK-23884: Summary: hasLaunchedTask should be true when launchedAnyTask be true Key: SPARK-23884 URL: https://issues.apache.org/jira/browse/SPARK-23884 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-23883) Error with conversion to arrow while using pandas_udf

Omri created SPARK-23883: Summary: Error with conversion to arrow while using pandas_udf Key: SPARK-23883 URL: https://issues.apache.org/jira/browse/SPARK-23883 Project: Spark Issue Type: Bug

[jira] [Assigned] (SPARK-23882) Is UTF8StringSuite.writeToOutputStreamUnderflow() supported?

[ https://issues.apache.org/jira/browse/SPARK-23882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23882: Assignee: (was: Apache Spark) > Is UTF8StringSuite.writeToOutputStreamUnderflow()

[jira] [Commented] (SPARK-23882) Is UTF8StringSuite.writeToOutputStreamUnderflow() supported?

[ https://issues.apache.org/jira/browse/SPARK-23882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428083#comment-16428083 ] Apache Spark commented on SPARK-23882: -- User 'kiszk' has created a pull request for this issue:

[jira] [Assigned] (SPARK-23882) Is UTF8StringSuite.writeToOutputStreamUnderflow() supported?

[ https://issues.apache.org/jira/browse/SPARK-23882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23882: Assignee: Apache Spark > Is UTF8StringSuite.writeToOutputStreamUnderflow() supported? >

[jira] [Commented] (SPARK-21898) Feature parity for KolmogorovSmirnovTest in MLlib

[ https://issues.apache.org/jira/browse/SPARK-21898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428074#comment-16428074 ] Apache Spark commented on SPARK-21898: -- User 'ueshin' has created a pull request for this issue:

[jira] [Created] (SPARK-23882) Is UTF8StringSuite.writeToOutputStreamUnderflow() supported?

Kazuaki Ishizaki created SPARK-23882: Summary: Is UTF8StringSuite.writeToOutputStreamUnderflow() supported? Key: SPARK-23882 URL: https://issues.apache.org/jira/browse/SPARK-23882 Project: Spark

[jira] [Assigned] (SPARK-23881) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

[ https://issues.apache.org/jira/browse/SPARK-23881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23881: Assignee: Apache Spark > Flaky test: JobCancellationSuite."interruptible iterator of

[jira] [Assigned] (SPARK-23881) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

[ https://issues.apache.org/jira/browse/SPARK-23881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-23881: Assignee: (was: Apache Spark) > Flaky test: JobCancellationSuite."interruptible

[jira] [Commented] (SPARK-23881) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

[ https://issues.apache.org/jira/browse/SPARK-23881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428071#comment-16428071 ] Apache Spark commented on SPARK-23881: -- User 'jiangxb1987' has created a pull request for this

[jira] [Created] (SPARK-23881) Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader"

Jiang Xingbo created SPARK-23881: Summary: Flaky test: JobCancellationSuite."interruptible iterator of shuffle reader" Key: SPARK-23881 URL: https://issues.apache.org/jira/browse/SPARK-23881 Project:

[jira] [Commented] (SPARK-23779) TaskMemoryManager and UnsafeSorter use MemoryBlock

[ https://issues.apache.org/jira/browse/SPARK-23779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428058#comment-16428058 ] Apache Spark commented on SPARK-23779: -- User 'kiszk' has created a pull request for this issue:

[jira] [Commented] (SPARK-23880) table cache should be lazy and don't trigger any jobs.

[ https://issues.apache.org/jira/browse/SPARK-23880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428018#comment-16428018 ] Takeshi Yamamuro commented on SPARK-23880: -- ya, I'll look into this. Thank! > table cache

[jira] [Commented] (SPARK-23880) table cache should be lazy and don't trigger any jobs.

[ https://issues.apache.org/jira/browse/SPARK-23880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428011#comment-16428011 ] Wenchen Fan commented on SPARK-23880: - cc [~ZenWzh] [~maropu] is any of you interested in doing this?

[jira] [Created] (SPARK-23880) table cache should be lazy and don't trigger any jobs.

Wenchen Fan created SPARK-23880: --- Summary: table cache should be lazy and don't trigger any jobs. Key: SPARK-23880 URL: https://issues.apache.org/jira/browse/SPARK-23880 Project: Spark Issue

[jira] [Commented] (SPARK-10399) Off Heap Memory Access for non-JVM libraries (C++)

[ https://issues.apache.org/jira/browse/SPARK-10399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428000#comment-16428000 ] Apache Spark commented on SPARK-10399: -- User 'kiszk' has created a pull request for this issue:

[jira] [Commented] (SPARK-23879) Introduce MemoryBlock API instead of Platform API with Object

[ https://issues.apache.org/jira/browse/SPARK-23879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428001#comment-16428001 ] Apache Spark commented on SPARK-23879: -- User 'kiszk' has created a pull request for this issue: