[jira] [Issue Comment Deleted] (SPARK-27862) Upgrade json4s-jackson to 3.6.5

2019-05-28 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Izek Greenfield updated SPARK-27862: Comment: was deleted (was: create pull request for 2.4 branch:

[jira] [Comment Edited] (SPARK-27862) Upgrade json4s-jackson to 3.6.5

2019-05-28 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849703#comment-16849703 ] Izek Greenfield edited comment on SPARK-27862 at 5/29/19 5:47 AM: --

[jira] [Assigned] (SPARK-27871) LambdaVariable should use per-query unique IDs instead of globally unique IDs

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27871: Assignee: Apache Spark (was: Wenchen Fan) > LambdaVariable should use per-query unique

[jira] [Assigned] (SPARK-27871) LambdaVariable should use per-query unique IDs instead of globally unique IDs

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27871: Assignee: Wenchen Fan (was: Apache Spark) > LambdaVariable should use per-query unique

[jira] [Created] (SPARK-27871) LambdaVariable should use per-query unique IDs instead of globally unique IDs

2019-05-28 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27871: --- Summary: LambdaVariable should use per-query unique IDs instead of globally unique IDs Key: SPARK-27871 URL: https://issues.apache.org/jira/browse/SPARK-27871 Project:

[jira] [Updated] (SPARK-27870) Flush each batch for pandas UDF (for improving pandas UDFs pipeline)

2019-05-28 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-27870: --- Description: Flush each batch for pandas UDF. This could improve performance when multiple pandas

[jira] [Updated] (SPARK-27870) Flush each batch for pandas UDF (for improving pandas UDFs pipeline)

2019-05-28 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-27870: --- Summary: Flush each batch for pandas UDF (for improving pandas UDFs pipeline) (was: Flush each

[jira] [Updated] (SPARK-27870) Flush each batch for python UDF

2019-05-28 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-27870: --- Description: Flush each batch for python UDF. This could improve performance when multiple python

[jira] [Updated] (SPARK-27870) Flush each batch for python UDF

2019-05-28 Thread Weichen Xu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weichen Xu updated SPARK-27870: --- Summary: Flush each batch for python UDF (was: Flush each batch for pandas UDF) > Flush each

[jira] [Assigned] (SPARK-27870) Flush each batch for pandas UDF

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27870: Assignee: (was: Apache Spark) > Flush each batch for pandas UDF >

[jira] [Assigned] (SPARK-27870) Flush each batch for pandas UDF

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27870: Assignee: Apache Spark > Flush each batch for pandas UDF >

[jira] [Created] (SPARK-27870) Flush each batch for pandas UDF

2019-05-28 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-27870: -- Summary: Flush each batch for pandas UDF Key: SPARK-27870 URL: https://issues.apache.org/jira/browse/SPARK-27870 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-27826) saveAsTable() function case table have "HiveFileFormat" "ParquetFileFormat" format issue

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27826. -- Resolution: Incomplete > saveAsTable() function case table have "HiveFileFormat"

[jira] [Commented] (SPARK-27826) saveAsTable() function case table have "HiveFileFormat" "ParquetFileFormat" format issue

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850302#comment-16850302 ] Hyukjin Kwon commented on SPARK-27826: -- See the doc for {{saveAsTable}} {quote} * When `mode`

[jira] [Resolved] (SPARK-27828) spark job hangs when kryo.serializers.FieldSerializer is called under multi-executor-cores settings

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27828. -- Resolution: Cannot Reproduce > spark job hangs when kryo.serializers.FieldSerializer is

[jira] [Commented] (SPARK-27828) spark job hangs when kryo.serializers.FieldSerializer is called under multi-executor-cores settings

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850296#comment-16850296 ] Hyukjin Kwon commented on SPARK-27828: -- Then, probably it's fixed. I wouldn't likely have more

[jira] [Resolved] (SPARK-27836) Issue with seeded rand() function in Spark SQL

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27836. -- Resolution: Not A Problem > Issue with seeded rand() function in Spark SQL >

[jira] [Resolved] (SPARK-27837) Running rand() in SQL with seed of column results in error (rand(col1))

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27837. -- Resolution: Won't Fix How do other many DBs support column as a seed? I don't see the reason

[jira] [Commented] (SPARK-27842) Inconsistent results of Statistics.corr() and PearsonCorrelation.computeCorrelationMatrix()

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850281#comment-16850281 ] Hyukjin Kwon commented on SPARK-27842: -- Are you able to test it against higher versions of Spark?

[jira] [Resolved] (SPARK-27850) Make SparkPlan#doExecuteBroadcast public

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27850. -- Resolution: Invalid > Make SparkPlan#doExecuteBroadcast public >

[jira] [Commented] (SPARK-27850) Make SparkPlan#doExecuteBroadcast public

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850280#comment-16850280 ] Hyukjin Kwon commented on SPARK-27850: -- They are under execution package which is meant to be

[jira] [Commented] (SPARK-27851) Allow for custom BroadcastMode#transform return values

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850278#comment-16850278 ] Hyukjin Kwon commented on SPARK-27851: -- {{BroadcastMode}} is meant to be a private API under

[jira] [Resolved] (SPARK-27851) Allow for custom BroadcastMode#transform return values

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27851. -- Resolution: Invalid > Allow for custom BroadcastMode#transform return values >

[jira] [Commented] (SPARK-27854) [Spark-SQL] OOM when using unequal join sql

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850276#comment-16850276 ] Hyukjin Kwon commented on SPARK-27854: -- Can you check if this issue still exists in higher version?

[jira] [Resolved] (SPARK-27855) Union failed between 2 datasets of the same type converted from different dataframes

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27855. -- Resolution: Not A Problem > Union failed between 2 datasets of the same type converted from

[jira] [Commented] (SPARK-27861) get_json_object in sql will truncate long value gotten from jsonpath

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850274#comment-16850274 ] Hyukjin Kwon commented on SPARK-27861: -- Can't reproduce in the current master. > get_json_object

[jira] [Resolved] (SPARK-27861) get_json_object in sql will truncate long value gotten from jsonpath

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27861. -- Resolution: Cannot Reproduce > get_json_object in sql will truncate long value gotten from

[jira] [Updated] (SPARK-27861) get_json_object in sql will truncate long value gotten from jsonpath

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27861: - Description: {code:java} select

[jira] [Commented] (SPARK-27864) spark-submit 2.4 cannot run apps compiled with Scala 2.12

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850273#comment-16850273 ] Hyukjin Kwon commented on SPARK-27864: -- Scala 2.12 is experimental in Spark 2.4. Are you able to

[jira] [Comment Edited] (SPARK-27772) SQLTestUtils Refactoring

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850264#comment-16850264 ] Hyukjin Kwon edited comment on SPARK-27772 at 5/29/19 12:58 AM: Ok.

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850264#comment-16850264 ] Hyukjin Kwon commented on SPARK-27772: -- k. please clarify the problem with before/after message to

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

2019-05-28 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850258#comment-16850258 ] William Wong commented on SPARK-27772: -- If a feature is well written, we don’t expect to hit any

[jira] [Assigned] (SPARK-27869) Redact sensitive information in System Properties from UI

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27869: Assignee: Apache Spark > Redact sensitive information in System Properties from UI >

[jira] [Assigned] (SPARK-27869) Redact sensitive information in System Properties from UI

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27869: Assignee: (was: Apache Spark) > Redact sensitive information in System Properties

[jira] [Commented] (SPARK-27869) Redact sensitive information in System Properties from UI

2019-05-28 Thread Aaruna Godthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850243#comment-16850243 ] Aaruna Godthi commented on SPARK-27869: --- https://github.com/apache/spark/pull/24733 > Redact

[jira] [Comment Edited] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-28 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850235#comment-16850235 ] Jungtaek Lim edited comment on SPARK-24815 at 5/28/19 11:22 PM: I'm also

[jira] [Commented] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-28 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850235#comment-16850235 ] Jungtaek Lim commented on SPARK-24815: -- I'm also interested on the design doc, as I'd like to see

[jira] [Assigned] (SPARK-27868) Better document shuffle / RPC listen backlog

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27868: Assignee: (was: Apache Spark) > Better document shuffle / RPC listen backlog >

[jira] [Assigned] (SPARK-27868) Better document shuffle / RPC listen backlog

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27868: Assignee: Apache Spark > Better document shuffle / RPC listen backlog >

[jira] [Created] (SPARK-27869) Redact sensitive information in System Properties from UI

2019-05-28 Thread Aaruna Godthi (JIRA)
Aaruna Godthi created SPARK-27869: - Summary: Redact sensitive information in System Properties from UI Key: SPARK-27869 URL: https://issues.apache.org/jira/browse/SPARK-27869 Project: Spark

[jira] [Created] (SPARK-27868) Better document shuffle / RPC listen backlog

2019-05-28 Thread Marcelo Vanzin (JIRA)
Marcelo Vanzin created SPARK-27868: -- Summary: Better document shuffle / RPC listen backlog Key: SPARK-27868 URL: https://issues.apache.org/jira/browse/SPARK-27868 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-28 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850137#comment-16850137 ] Stavros Kontopoulos edited comment on SPARK-24815 at 5/28/19 8:43 PM:

[jira] [Commented] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-28 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850137#comment-16850137 ] Stavros Kontopoulos commented on SPARK-24815: - That is great news [~Karthik Palaniappan], I

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

2019-05-28 Thread Dhruve Ashar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850096#comment-16850096 ] Dhruve Ashar commented on SPARK-24149: -- Thanks for the missing context. The current behavior

[jira] [Resolved] (SPARK-20547) ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

2019-05-28 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-20547. -- Resolution: Fixed Fix Version/s: 3.0.0 > ExecutorClassLoader's findClass may not work

[jira] [Commented] (SPARK-26388) No support for "alter table .. replace columns" to drop columns

2019-05-28 Thread Emanuele Bardelli (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850090#comment-16850090 ] Emanuele Bardelli commented on SPARK-26388: --- [~thesuperzapper] maybe this could help? 

[jira] [Commented] (SPARK-25588) SchemaParseException: Can't redefine: list when reading from Parquet

2019-05-28 Thread Michael Heuer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850003#comment-16850003 ] Michael Heuer commented on SPARK-25588: --- Note a fix for this issue has been merged upstream in

[jira] [Assigned] (SPARK-27362) Kubernetes support for GPU-aware scheduling

2019-05-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27362: - Assignee: Thomas Graves > Kubernetes support for GPU-aware scheduling >

[jira] [Commented] (SPARK-24815) Structured Streaming should support dynamic allocation

2019-05-28 Thread Karthik Palaniappan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849968#comment-16849968 ] Karthik Palaniappan commented on SPARK-24815: - Since there's discussion about this FR on the

[jira] [Assigned] (SPARK-27725) GPU Scheduling - add an example discovery Script

2019-05-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27725: - Assignee: Thomas Graves > GPU Scheduling - add an example discovery Script >

[jira] [Comment Edited] (SPARK-27785) Introduce .joinWith() overloads for typed inner joins of 3 or more tables

2019-05-28 Thread Swapnil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849880#comment-16849880 ] Swapnil edited comment on SPARK-27785 at 5/28/19 4:09 PM: -- I like this idea. I

[jira] [Commented] (SPARK-27785) Introduce .joinWith() overloads for typed inner joins of 3 or more tables

2019-05-28 Thread Swapnil (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849880#comment-16849880 ] Swapnil commented on SPARK-27785: - I like this idea. I can start working on it if it seems to be

[jira] [Commented] (SPARK-27837) Running rand() in SQL with seed of column results in error (rand(col1))

2019-05-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849871#comment-16849871 ] Liang-Chi Hsieh commented on SPARK-27837: - Btw, I think this is not a bug but like an

[jira] [Resolved] (SPARK-26872) Use a configurable value for final termination in the JobScheduler.stop() method

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-26872. --- Resolution: Won't Fix > Use a configurable value for final termination in the JobScheduler.stop()

[jira] [Resolved] (SPARK-27584) Add 'Mean reciprocal rank' to RankingMetrics

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27584. --- Resolution: Won't Fix > Add 'Mean reciprocal rank' to RankingMetrics >

[jira] [Resolved] (SPARK-27084) Add function alias for Bitwise functions

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27084. --- Resolution: Won't Fix > Add function alias for Bitwise functions >

[jira] [Resolved] (SPARK-27434) memory leak in spark driver

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27434. --- Resolution: Cannot Reproduce > memory leak in spark driver > --- > >

[jira] [Assigned] (SPARK-27858) Fix for avro deserialization on union types with multiple non-null types

2019-05-28 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27858: - Assignee: Gabbi Merz > Fix for avro deserialization on union types with multiple

[jira] [Commented] (SPARK-27837) Running rand() in SQL with seed of column results in error (rand(col1))

2019-05-28 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849836#comment-16849836 ] Liang-Chi Hsieh commented on SPARK-27837: - Ah, I see. MySQL disallows nonconstant argument in

[jira] [Comment Edited] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Ricardo Pinto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849819#comment-16849819 ] Ricardo Pinto edited comment on SPARK-27866 at 5/28/19 2:59 PM: It's the

[jira] [Commented] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Ricardo Pinto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849819#comment-16849819 ] Ricardo Pinto commented on SPARK-27866: --- It's the same, it doesn't connect to the remote hive

[jira] [Assigned] (SPARK-27835) Resource Scheduling: change driver config from addresses to resourcesFile

2019-05-28 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-27835: - Assignee: Thomas Graves > Resource Scheduling: change driver config from addresses to

[jira] [Assigned] (SPARK-27835) Resource Scheduling: change driver config from addresses to resourcesFile

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27835: Assignee: Apache Spark > Resource Scheduling: change driver config from addresses to

[jira] [Assigned] (SPARK-27835) Resource Scheduling: change driver config from addresses to resourcesFile

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27835: Assignee: (was: Apache Spark) > Resource Scheduling: change driver config from

[jira] [Commented] (SPARK-27837) Running rand() in SQL with seed of column results in error (rand(col1))

2019-05-28 Thread Jason Ferrell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849794#comment-16849794 ] Jason Ferrell commented on SPARK-27837: --- Here is an example from Microsoft SQL Server.  I'm pretty

[jira] [Updated] (SPARK-27666) Do not release lock while TaskContext already completed

2019-05-28 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-27666: - Description: {code:java} Exception in thread "Thread-14" java.lang.AssertionError: assertion failed: Block

[jira] [Resolved] (SPARK-27657) ml.util.Instrumentation.logFailure doesn't log error message

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27657. --- Resolution: Fixed Fix Version/s: 2.4.4 3.0.0 Issue resolved by pull

[jira] [Assigned] (SPARK-27657) ml.util.Instrumentation.logFailure doesn't log error message

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27657: - Assignee: Wesley Tang > ml.util.Instrumentation.logFailure doesn't log error message >

[jira] [Updated] (SPARK-27666) Do not release lock while TaskContext already completed

2019-05-28 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-27666: - Description: We're facing an issue reported by SPARK-18406 and SPARK-25139. And

[jira] [Updated] (SPARK-27657) ml.util.Instrumentation.logFailure doesn't log error message

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27657: -- Target Version/s: 2.4.4, 3.0.0 (was: 3.0.0) > ml.util.Instrumentation.logFailure doesn't log error

[jira] [Updated] (SPARK-27657) ml.util.Instrumentation.logFailure doesn't log error message

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27657: -- Target Version/s: 3.0.0 (was: 2.4.4, 3.0.0) > ml.util.Instrumentation.logFailure doesn't log error

[jira] [Updated] (SPARK-27657) ml.util.Instrumentation.logFailure doesn't log error message

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27657: -- Priority: Minor (was: Major) > ml.util.Instrumentation.logFailure doesn't log error message >

[jira] [Updated] (SPARK-27666) Do not release lock while TaskContext already completed

2019-05-28 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-27666: - Description: We're facing an issue reported by SPARK-18406 and SPARK-25139. And

[jira] [Resolved] (SPARK-27776) Avoid duplicate Java reflection in DataSource

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27776. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24647

[jira] [Assigned] (SPARK-27776) Avoid duplicate Java reflection in DataSource

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27776: - Assignee: jiaan.geng > Avoid duplicate Java reflection in DataSource >

[jira] [Commented] (SPARK-21827) Task fail due to executor exception when enable Sasl Encryption

2019-05-28 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-21827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849757#comment-16849757 ] Sébastien BARNOUD commented on SPARK-21827: --- Hi, For HBase i found the exact reason for this

[jira] [Updated] (SPARK-27666) Do not release lock while TaskContext already completed

2019-05-28 Thread wuyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wuyi updated SPARK-27666: - Summary: Do not release lock while TaskContext already completed (was: Stop python runner threads when task

[jira] [Resolved] (SPARK-27852) One updateBytesWritten operaton may be missed in DiskBlockObjectWriter.scala

2019-05-28 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27852. --- Resolution: Not A Problem > One updateBytesWritten operaton may be missed in

[jira] [Comment Edited] (SPARK-13182) Spark Executor retries infinitely

2019-05-28 Thread Atul Anand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849742#comment-16849742 ] Atul Anand edited comment on SPARK-13182 at 5/28/19 1:55 PM: - # Yarn policy

[jira] [Commented] (SPARK-13182) Spark Executor retries infinitely

2019-05-28 Thread Atul Anand (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849742#comment-16849742 ] Atul Anand commented on SPARK-13182: # Yarn policy is to preempt a job in low priority queue for

[jira] [Commented] (SPARK-15060) Fix stack overflow when executing long lineage transform without checkpoint

2019-05-28 Thread Michael Wu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849732#comment-16849732 ] Michael Wu commented on SPARK-15060: I'm late to the party but have you tried using

[jira] [Commented] (SPARK-27862) Upgrade json4s-jackson to 3.6.5

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849707#comment-16849707 ] Apache Spark commented on SPARK-27862: -- User 'igreenfield' has created a pull request for this

[jira] [Assigned] (SPARK-27862) Upgrade json4s-jackson to 3.6.5

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27862: Assignee: (was: Apache Spark) > Upgrade json4s-jackson to 3.6.5 >

[jira] [Assigned] (SPARK-27862) Upgrade json4s-jackson to 3.6.5

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27862: Assignee: Apache Spark > Upgrade json4s-jackson to 3.6.5 >

[jira] [Commented] (SPARK-27862) Upgrade json4s-jackson to 3.6.5

2019-05-28 Thread Izek Greenfield (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849703#comment-16849703 ] Izek Greenfield commented on SPARK-27862: - create pull request for 2.4 branch:

[jira] [Commented] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Ricardo Pinto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849685#comment-16849685 ] Ricardo Pinto commented on SPARK-27866: --- Thanks, I'll compile with -Phadoop-2.7. > Cannot connect

[jira] [Commented] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849683#comment-16849683 ] Yuming Wang commented on SPARK-27866: - Spark 2.4.3 does not support {{hadoop-3.2}} because of

[jira] [Commented] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Ricardo Pinto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849628#comment-16849628 ] Ricardo Pinto commented on SPARK-27866: --- Do I have to add -Phive-thriftserver to the compilation

[jira] [Commented] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Ricardo Pinto (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849627#comment-16849627 ] Ricardo Pinto commented on SPARK-27866: --- It's the same. Is there a way to test the hive metastore

[jira] [Commented] (SPARK-27554) org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS]

2019-05-28 Thread Jepson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849594#comment-16849594 ] Jepson commented on SPARK-27554: [~hyukjin.kwon] Thanks for concern.  The code issue, i can resolve it.

[jira] [Commented] (SPARK-27593) CSV Parser returns 2 DataFrame - Valid and Malformed DFs

2019-05-28 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849585#comment-16849585 ] Hyukjin Kwon commented on SPARK-27593: -- Yea, it's good to file a request and file the discussions.

[jira] [Assigned] (SPARK-27867) RegressionEvaluator cache lastest RegressionMetrics to avoid duplicated computation

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27867: Assignee: (was: Apache Spark) > RegressionEvaluator cache lastest RegressionMetrics

[jira] [Assigned] (SPARK-27867) RegressionEvaluator cache lastest RegressionMetrics to avoid duplicated computation

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27867: Assignee: Apache Spark > RegressionEvaluator cache lastest RegressionMetrics to avoid

[jira] [Commented] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849542#comment-16849542 ] Yuming Wang commented on SPARK-27866: - Could you try to compile the current master branch?, more

[jira] [Created] (SPARK-27867) RegressionEvaluator cache lastest RegressionMetrics to avoid duplicated computation

2019-05-28 Thread zhengruifeng (JIRA)
zhengruifeng created SPARK-27867: Summary: RegressionEvaluator cache lastest RegressionMetrics to avoid duplicated computation Key: SPARK-27867 URL: https://issues.apache.org/jira/browse/SPARK-27867

[jira] [Created] (SPARK-27866) Cannot connect to hive metastore

2019-05-28 Thread Ricardo Pinto (JIRA)
Ricardo Pinto created SPARK-27866: - Summary: Cannot connect to hive metastore Key: SPARK-27866 URL: https://issues.apache.org/jira/browse/SPARK-27866 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-28 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849504#comment-16849504 ] tommy duan commented on SPARK-27648: Hi [~gsomogyi] & [~kabhwan] Previous speculation may have been

[jira] [Updated] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-28 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tommy duan updated SPARK-27648: --- Attachment: houragg_with_state1_state2.xlsx houragg_with_state1_state2.csv > In

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming:The executor storage memory increasing over time

2019-05-28 Thread tommy duan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16849500#comment-16849500 ] tommy duan commented on SPARK-27648: Hi [~gsomogyi]  I have saved two key fields of stateOpeators

[jira] [Assigned] (SPARK-27865) Spark SQL support 1:N sort merge bucket join without shuffle

2019-05-28 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27865: Assignee: (was: Apache Spark) > Spark SQL support 1:N sort merge bucket join without

  1   2   >