[jira] [Commented] (SPARK-13510) Shuffle may throw FetchFailedException: Direct buffer memory

2019-04-25 Thread Mike Chan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825798#comment-16825798 ] Mike Chan commented on SPARK-13510: --- Thanks man > Shuffle may throw FetchFailedExcept

[jira] [Assigned] (SPARK-27557) Add copybutton to spark Python API docs for easier copying of code-blocks

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27557: Assignee: Apache Spark > Add copybutton to spark Python API docs for easier copying of co

[jira] [Assigned] (SPARK-27557) Add copybutton to spark Python API docs for easier copying of code-blocks

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27557: Assignee: (was: Apache Spark) > Add copybutton to spark Python API docs for easier co

[jira] [Commented] (SPARK-27491) SPARK REST API - "org.apache.spark.deploy.SparkSubmit --status" returns empty response! therefore Airflow won't integrate with Spark 2.3.x

2019-04-25 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825804#comment-16825804 ] t oo commented on SPARK-27491: -- my current workaround to get airflow to integrate with spar

[jira] [Assigned] (SPARK-27563) automatically get the latest Spark versions in HiveExternalCatalogVersionsSuite

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27563: Assignee: Apache Spark (was: Wenchen Fan) > automatically get the latest Spark versions

[jira] [Assigned] (SPARK-27563) automatically get the latest Spark versions in HiveExternalCatalogVersionsSuite

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27563: Assignee: Wenchen Fan (was: Apache Spark) > automatically get the latest Spark versions

[jira] [Commented] (SPARK-26268) Decouple shuffle data from Spark deployment

2019-04-25 Thread Chenzhao Guo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825814#comment-16825814 ] Chenzhao Guo commented on SPARK-26268: -- Actually this can be resolved in SPARK-2529

[jira] [Commented] (SPARK-27331) Schema mismatch using MicroBatchReader with columns pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825874#comment-16825874 ] Kineret commented on SPARK-27331: - set the schema to be the full schema after every comm

[jira] [Issue Comment Deleted] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

2019-04-25 Thread Sergey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-26688: --- Comment: was deleted (was: Hi There! I'm very glad that the community paid attention to my question. Let me

[jira] [Resolved] (SPARK-27331) Schema mismatch using MicroBatchReader with columns pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kineret resolved SPARK-27331. - Resolution: Invalid > Schema mismatch using MicroBatchReader with columns pruning >

[jira] [Issue Comment Deleted] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes

2019-04-25 Thread Sergey (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey updated SPARK-26688: --- Comment: was deleted (was: Hi Imran, thanks for you reply. "Meanwhile devs start to apply this willy-nilly

[jira] [Created] (SPARK-27564) 'No plan for EventTimeWatermark' error while using structured streaming with column pruning

2019-04-25 Thread Kineret (JIRA)
Kineret created SPARK-27564: --- Summary: 'No plan for EventTimeWatermark' error while using structured streaming with column pruning Key: SPARK-27564 URL: https://issues.apache.org/jira/browse/SPARK-27564 Pro

[jira] [Updated] (SPARK-27564) 'No plan for EventTimeWatermark' error while using structured streaming with column pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kineret updated SPARK-27564: Description: I get 'No plan for EventTimeWatermark' error while doing a query with columns pruning using

[jira] [Updated] (SPARK-27564) 'No plan for EventTimeWatermark' error while using structured streaming with column pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kineret updated SPARK-27564: Description: I get 'No plan for EventTimeWatermark' error while doing a query with columns pruning using

[jira] [Updated] (SPARK-27564) 'No plan for EventTimeWatermark' error while using structured streaming with column pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kineret updated SPARK-27564: Description: I get 'No plan for EventTimeWatermark' error while doing a query with columns pruning using

[jira] [Updated] (SPARK-27564) 'No plan for EventTimeWatermark' error while using structured streaming with column pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kineret updated SPARK-27564: Description: I get 'No plan for EventTimeWatermark' error while doing a query with columns pruning using

[jira] [Updated] (SPARK-27564) 'No plan for EventTimeWatermark' error while using structured streaming with column pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kineret updated SPARK-27564: Description: I get 'No plan for EventTimeWatermark' error while doing a query with columns pruning using

[jira] [Updated] (SPARK-27564) 'No plan for EventTimeWatermark' error while using structured streaming with column pruning

2019-04-25 Thread Kineret (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kineret updated SPARK-27564: Description: I get 'No plan for EventTimeWatermark' error while doing a query with columns pruning using

[jira] [Commented] (SPARK-4105) FAILED_TO_UNCOMPRESS(5) errors when fetching shuffle data with sort-based shuffle

2019-04-25 Thread Aakash Mandlik (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825901#comment-16825901 ] Aakash Mandlik commented on SPARK-4105: --- I am facing similar issue while persisting

[jira] [Assigned] (SPARK-27440) Optimize uncorrelated predicate subquery

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27440: Assignee: Apache Spark > Optimize uncorrelated predicate subquery > -

[jira] [Assigned] (SPARK-27440) Optimize uncorrelated predicate subquery

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27440: Assignee: (was: Apache Spark) > Optimize uncorrelated predicate subquery > --

[jira] [Updated] (SPARK-27440) Optimize uncorrelated predicate subquery

2019-04-25 Thread Mingcong Han (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingcong Han updated SPARK-27440: - Description: Currently, we rewrite all the predicate subqueries(InSubquery, Exists) as semi-joi

[jira] [Assigned] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27340: Assignee: Apache Spark > Alias on TimeWIndow expression may cause watermark metadata lost

[jira] [Assigned] (SPARK-27340) Alias on TimeWIndow expression may cause watermark metadata lost

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27340: Assignee: (was: Apache Spark) > Alias on TimeWIndow expression may cause watermark me

[jira] [Assigned] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27562: Assignee: Apache Spark > Complete the verification mechanism for shuffle transmitted data

[jira] [Assigned] (SPARK-27562) Complete the verification mechanism for shuffle transmitted data

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27562: Assignee: (was: Apache Spark) > Complete the verification mechanism for shuffle trans

[jira] [Commented] (SPARK-27549) Commit Kafka Source offsets to facilitate external tooling

2019-04-25 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16825994#comment-16825994 ] Gabor Somogyi commented on SPARK-27549: --- Do you mean commit offsets all the time o

[jira] [Assigned] (SPARK-27350) Support create table on data source V2

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27350: Assignee: Apache Spark > Support create table on data source V2 > ---

[jira] [Assigned] (SPARK-27350) Support create table on data source V2

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27350: Assignee: (was: Apache Spark) > Support create table on data source V2 >

[jira] [Created] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread peng bo (JIRA)
peng bo created SPARK-27565: --- Summary: Show job info of WholeStageCodegen node on SparkSQL UI page Key: SPARK-27565 URL: https://issues.apache.org/jira/browse/SPARK-27565 Project: Spark Issue Type

[jira] [Updated] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread peng bo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng bo updated SPARK-27565: Attachment: SPARK-27565.jpg > Show job info of WholeStageCodegen node on SparkSQL UI page > --

[jira] [Updated] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread peng bo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng bo updated SPARK-27565: Description: Currently it's really hard to link SQL plan to Spark jobs on {{SparkSQL}} page. When one job

[jira] [Updated] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread peng bo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng bo updated SPARK-27565: Attachment: (was: SPARK-27565.jpg) > Show job info of WholeStageCodegen node on SparkSQL UI page > ---

[jira] [Updated] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread peng bo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng bo updated SPARK-27565: Attachment: SPARK-27565.jpg > Show job info of WholeStageCodegen node on SparkSQL UI page > --

[jira] [Created] (SPARK-27566) SIGSEV in Spark SQL during broadcast

2019-04-25 Thread Martin Studer (JIRA)
Martin Studer created SPARK-27566: - Summary: SIGSEV in Spark SQL during broadcast Key: SPARK-27566 URL: https://issues.apache.org/jira/browse/SPARK-27566 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27566) SIGSEV in Spark SQL during broadcast

2019-04-25 Thread Martin Studer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Martin Studer updated SPARK-27566: -- Environment: Hortonworks HDP 2.6.5, Spark 2.3.0.2.6.5.1050-37 > SIGSEV in Spark SQL during bro

[jira] [Commented] (SPARK-27529) Spark Streaming consumer dies with kafka.common.OffsetOutOfRangeException

2019-04-25 Thread Dmitry Goldenberg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826072#comment-16826072 ] Dmitry Goldenberg commented on SPARK-27529: --- Hi Hyukjin, I could check althoug

[jira] [Assigned] (SPARK-27547) fix DataFrame self-join problems

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27547: Assignee: Apache Spark (was: Wenchen Fan) > fix DataFrame self-join problems > -

[jira] [Assigned] (SPARK-27547) fix DataFrame self-join problems

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27547: Assignee: Wenchen Fan (was: Apache Spark) > fix DataFrame self-join problems > -

[jira] [Assigned] (SPARK-27536) Code improvements for 3.0: existentials edition

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27536: Assignee: Sean Owen (was: Apache Spark) > Code improvements for 3.0: existentials editio

[jira] [Assigned] (SPARK-27536) Code improvements for 3.0: existentials edition

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27536: Assignee: Apache Spark (was: Sean Owen) > Code improvements for 3.0: existentials editio

[jira] [Commented] (SPARK-27300) Create the new graph projects in Spark and set up build/test

2019-04-25 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826124#comment-16826124 ] Martin Junghanns commented on SPARK-27300: -- As discussed offline with [~mengxr]

[jira] [Updated] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread peng bo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng bo updated SPARK-27565: Description: Currently it's really hard to link SQL plan to Spark jobs on SparkSQL UI page. When one job

[jira] [Updated] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread peng bo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] peng bo updated SPARK-27565: Description: Currently it's really hard to link SQL plan to Spark jobs on SparkSQL UI page. When one job

[jira] [Assigned] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27565: Assignee: (was: Apache Spark) > Show job info of WholeStageCodegen node on SparkSQL U

[jira] [Assigned] (SPARK-27565) Show job info of WholeStageCodegen node on SparkSQL UI page

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27565: Assignee: Apache Spark > Show job info of WholeStageCodegen node on SparkSQL UI page > --

[jira] [Created] (SPARK-27567) Spark Streaming consumers (from Kafka) intermittently die with 'SparkException: Couldn't find leaders for Set'

2019-04-25 Thread Dmitry Goldenberg (JIRA)
Dmitry Goldenberg created SPARK-27567: - Summary: Spark Streaming consumers (from Kafka) intermittently die with 'SparkException: Couldn't find leaders for Set' Key: SPARK-27567 URL: https://issues.apache.org/j

[jira] [Updated] (SPARK-27567) Spark Streaming consumers (from Kafka) intermittently die with 'SparkException: Couldn't find leaders for Set'

2019-04-25 Thread Dmitry Goldenberg (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dmitry Goldenberg updated SPARK-27567: -- Description: Some of our consumers intermittently die with the stack traces I'm includ

[jira] [Commented] (SPARK-27549) Commit Kafka Source offsets to facilitate external tooling

2019-04-25 Thread Sean Glover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826180#comment-16826180 ] Sean Glover commented on SPARK-27549: - Yes, committing offsets would only be for the

[jira] [Comment Edited] (SPARK-27549) Commit Kafka Source offsets to facilitate external tooling

2019-04-25 Thread Sean Glover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826180#comment-16826180 ] Sean Glover edited comment on SPARK-27549 at 4/25/19 3:41 PM:

[jira] [Resolved] (SPARK-27551) Improve error message of mismatched types for CASE WHEN

2019-04-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27551. --- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 3.0.0 This is res

[jira] [Updated] (SPARK-27551) Improve error message of mismatched types for CASE WHEN

2019-04-25 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27551: -- Summary: Improve error message of mismatched types for CASE WHEN (was: Uniformative error me

[jira] [Created] (SPARK-27568) readLock leaked when method take() called on a cached rdd

2019-04-25 Thread wuyi (JIRA)
wuyi created SPARK-27568: Summary: readLock leaked when method take() called on a cached rdd Key: SPARK-27568 URL: https://issues.apache.org/jira/browse/SPARK-27568 Project: Spark Issue Type: Improve

[jira] [Assigned] (SPARK-27248) REFRESH TABLE should recreate cache with same cache name and storage level

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27248: Assignee: (was: Apache Spark) > REFRESH TABLE should recreate cache with same cache n

[jira] [Assigned] (SPARK-27248) REFRESH TABLE should recreate cache with same cache name and storage level

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27248: Assignee: Apache Spark > REFRESH TABLE should recreate cache with same cache name and sto

[jira] [Assigned] (SPARK-27272) Enable blacklisting of node/executor on fetch failures by default

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27272: Assignee: (was: Apache Spark) > Enable blacklisting of node/executor on fetch failure

[jira] [Assigned] (SPARK-27297) Add higher order functions to Scala API

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27297: Assignee: Apache Spark > Add higher order functions to Scala API > --

[jira] [Assigned] (SPARK-27229) GroupBy Placement in Intersect Distinct

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27229: Assignee: Apache Spark > GroupBy Placement in Intersect Distinct > --

[jira] [Assigned] (SPARK-27237) Introduce State schema validation among query restart

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27237: Assignee: (was: Apache Spark) > Introduce State schema validation among query restart

[jira] [Assigned] (SPARK-27297) Add higher order functions to Scala API

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27297: Assignee: (was: Apache Spark) > Add higher order functions to Scala API > ---

[jira] [Assigned] (SPARK-26356) Remove SaveMode from data source v2 API

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26356: Assignee: Apache Spark > Remove SaveMode from data source v2 API > --

[jira] [Assigned] (SPARK-27204) First time Loading application page from History Server is taking time when event log size is huge

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27204: Assignee: Apache Spark > First time Loading application page from History Server is takin

[jira] [Assigned] (SPARK-27232) Ignore file locality in InMemoryFileIndex if spark.locality.wait is set to

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27232: Assignee: (was: Apache Spark) > Ignore file locality in InMemoryFileIndex if spark.lo

[jira] [Assigned] (SPARK-27204) First time Loading application page from History Server is taking time when event log size is huge

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27204: Assignee: (was: Apache Spark) > First time Loading application page from History Serv

[jira] [Assigned] (SPARK-27280) infer filters from Join's OR condition

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27280: Assignee: (was: Apache Spark) > infer filters from Join's OR condition >

[jira] [Assigned] (SPARK-27254) Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27254: Assignee: Apache Spark > Cleanup complete but becoming invalid output files in > Manifes

[jira] [Assigned] (SPARK-27254) Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27254: Assignee: (was: Apache Spark) > Cleanup complete but becoming invalid output files in

[jira] [Assigned] (SPARK-27280) infer filters from Join's OR condition

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27280: Assignee: Apache Spark > infer filters from Join's OR condition > ---

[jira] [Assigned] (SPARK-27229) GroupBy Placement in Intersect Distinct

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27229: Assignee: (was: Apache Spark) > GroupBy Placement in Intersect Distinct > ---

[jira] [Assigned] (SPARK-27237) Introduce State schema validation among query restart

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27237: Assignee: Apache Spark > Introduce State schema validation among query restart >

[jira] [Assigned] (SPARK-27272) Enable blacklisting of node/executor on fetch failures by default

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27272: Assignee: Apache Spark > Enable blacklisting of node/executor on fetch failures by defaul

[jira] [Assigned] (SPARK-27281) Wrong latest offsets returned by DirectKafkaInputDStream#latestOffsets

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27281: Assignee: Apache Spark > Wrong latest offsets returned by DirectKafkaInputDStream#latestO

[jira] [Assigned] (SPARK-27295) Provision to provide initial values for each source node in personalised page rank - Graphx

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27295: Assignee: (was: Apache Spark) > Provision to provide initial values for each source n

[jira] [Assigned] (SPARK-27281) Wrong latest offsets returned by DirectKafkaInputDStream#latestOffsets

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27281: Assignee: (was: Apache Spark) > Wrong latest offsets returned by DirectKafkaInputDStr

[jira] [Assigned] (SPARK-27258) The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27258: Assignee: Apache Spark > The value of "spark.app.name" or "--name" starts with number , w

[jira] [Assigned] (SPARK-27214) Upgrading locality level when lots of pending tasks have been waiting more than locality.wait

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27214: Assignee: Apache Spark > Upgrading locality level when lots of pending tasks have been wa

[jira] [Assigned] (SPARK-27295) Provision to provide initial values for each source node in personalised page rank - Graphx

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27295: Assignee: Apache Spark > Provision to provide initial values for each source node in pers

[jira] [Assigned] (SPARK-27319) Filter out dir based on PathFilter before listing them

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27319: Assignee: Apache Spark > Filter out dir based on PathFilter before listing them > ---

[jira] [Assigned] (SPARK-27232) Ignore file locality in InMemoryFileIndex if spark.locality.wait is set to

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27232: Assignee: Apache Spark > Ignore file locality in InMemoryFileIndex if spark.locality.wait

[jira] [Assigned] (SPARK-27214) Upgrading locality level when lots of pending tasks have been waiting more than locality.wait

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27214: Assignee: (was: Apache Spark) > Upgrading locality level when lots of pending tasks h

[jira] [Assigned] (SPARK-27258) The value of "spark.app.name" or "--name" starts with number , which causes resourceName does not match regular expression

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27258: Assignee: (was: Apache Spark) > The value of "spark.app.name" or "--name" starts with

[jira] [Assigned] (SPARK-27402) Fix hadoop-3.2 test issue(except the hive-thriftserver module)

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27402?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27402: Assignee: Apache Spark > Fix hadoop-3.2 test issue(except the hive-thriftserver module) >

[jira] [Assigned] (SPARK-27343) Use ConfigEntry for hardcoded configs for spark-sql-kafka

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27343: Assignee: Apache Spark > Use ConfigEntry for hardcoded configs for spark-sql-kafka >

[jira] [Assigned] (SPARK-27354) Move incompatible code from the hive-thriftserver module to sql/hive-thriftserver/v1.2.1

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27354: Assignee: (was: Apache Spark) > Move incompatible code from the hive-thriftserver mod

[jira] [Assigned] (SPARK-27355) make query execution more sensitive to epoch message late or lost

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27355: Assignee: Apache Spark > make query execution more sensitive to epoch message late or los

[jira] [Assigned] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27348: Assignee: (was: Apache Spark) > HeartbeatReceiver doesn't remove lost executors from

[jira] [Assigned] (SPARK-27413) Keep the same epoch pace between driver and executor.

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27413: Assignee: (was: Apache Spark) > Keep the same epoch pace between driver and executor.

[jira] [Assigned] (SPARK-27425) Add count_if functions

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27425: Assignee: (was: Apache Spark) > Add count_if functions > -- > >

[jira] [Assigned] (SPARK-27294) Multi-cluster Kafka delegation token support

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27294: Assignee: Apache Spark > Multi-cluster Kafka delegation token support > -

[jira] [Assigned] (SPARK-27347) [MESOS] Fix supervised driver retry logic when agent crashes/restarts

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27347: Assignee: Apache Spark > [MESOS] Fix supervised driver retry logic when agent crashes/res

[jira] [Assigned] (SPARK-27388) expression encoder for avro objects

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27388: Assignee: (was: Apache Spark) > expression encoder for avro objects > ---

[jira] [Assigned] (SPARK-27348) HeartbeatReceiver doesn't remove lost executors from CoarseGrainedSchedulerBackend

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27348?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27348: Assignee: Apache Spark > HeartbeatReceiver doesn't remove lost executors from > CoarseGr

[jira] [Assigned] (SPARK-27441) Add read/write tests to Hive serde tables(include Parquet vectorized reader)

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27441: Assignee: Apache Spark > Add read/write tests to Hive serde tables(include Parquet vector

[jira] [Assigned] (SPARK-27388) expression encoder for avro objects

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27388: Assignee: Apache Spark > expression encoder for avro objects > --

[jira] [Assigned] (SPARK-27366) Spark scheduler internal changes to support GPU scheduling

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27366: Assignee: Apache Spark (was: Xingbo Jiang) > Spark scheduler internal changes to support

[jira] [Assigned] (SPARK-27343) Use ConfigEntry for hardcoded configs for spark-sql-kafka

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27343: Assignee: (was: Apache Spark) > Use ConfigEntry for hardcoded configs for spark-sql-k

[jira] [Assigned] (SPARK-27355) make query execution more sensitive to epoch message late or lost

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27355: Assignee: (was: Apache Spark) > make query execution more sensitive to epoch message

[jira] [Assigned] (SPARK-27024) Executor interface for cluster managers to support GPU resources

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27024: Assignee: Apache Spark (was: Thomas Graves) > Executor interface for cluster managers to

[jira] [Assigned] (SPARK-27299) Design: Property graph construction, save/load, and query APIs

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27299: Assignee: Martin Junghanns (was: Apache Spark) > Design: Property graph construction, sa

[jira] [Assigned] (SPARK-27347) [MESOS] Fix supervised driver retry logic when agent crashes/restarts

2019-04-25 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27347: Assignee: (was: Apache Spark) > [MESOS] Fix supervised driver retry logic when agent

  1   2   >