[jira] [Assigned] (SPARK-27893) Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

[ https://issues.apache.org/jira/browse/SPARK-27893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27893: Assignee: Apache Spark > Create an integrated test base for Python, Scalar Pandas, Scala

[jira] [Assigned] (SPARK-27893) Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

[ https://issues.apache.org/jira/browse/SPARK-27893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27893: Assignee: (was: Apache Spark) > Create an integrated test base for Python, Scalar

[jira] [Created] (SPARK-27893) Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files

Hyukjin Kwon created SPARK-27893: Summary: Create an integrated test base for Python, Scalar Pandas, Scala UDF by sql files Key: SPARK-27893 URL: https://issues.apache.org/jira/browse/SPARK-27893

[jira] [Comment Edited] (SPARK-24815) Structured Streaming should support dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852541#comment-16852541 ] Karthik Palaniappan edited comment on SPARK-24815 at 5/31/19 2:26 AM:

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852573#comment-16852573 ] hemshankar sahu commented on SPARK-27891: - Sure I'll provide for 2.3.1 in some time as it needs

[jira] [Updated] (SPARK-27876) Split large shuffle partition to multi-segments to enable transfer oversize shuffle partition block.

[ https://issues.apache.org/jira/browse/SPARK-27876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27876: Affects Version/s: (was: 3.1.0) (was: 2.4.3) 2.3.2

[jira] [Commented] (SPARK-24815) Structured Streaming should support dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852541#comment-16852541 ] Karthik Palaniappan commented on SPARK-24815: - I was starting to update the JIRA description

[jira] [Updated] (SPARK-24815) Structured Streaming should support dynamic allocation

[ https://issues.apache.org/jira/browse/SPARK-24815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Palaniappan updated SPARK-24815: Description: For batch jobs, dynamic allocation is very useful for adding and

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852508#comment-16852508 ] Hyukjin Kwon commented on SPARK-27884: -- +1 > Deprecate Python 2 support in Spark 3.0 >

[jira] [Assigned] (SPARK-27862) Upgrade json4s-jackson to 3.6.6

[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27862: - Assignee: Izek Greenfield > Upgrade json4s-jackson to 3.6.6 > ---

[jira] [Resolved] (SPARK-27862) Upgrade json4s-jackson to 3.6.6

[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27862. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24736

[jira] [Updated] (SPARK-27892) Saving/loading stages in PipelineModel should be parallel

[ https://issues.apache.org/jira/browse/SPARK-27892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Wang updated SPARK-27892: --- Description: When a PipelineModel is saved/loaded, all the stages are saved/loaded sequentially.

[jira] [Created] (SPARK-27892) Saving/loading stages in PipelineModel should be parallel

Jason Wang created SPARK-27892: -- Summary: Saving/loading stages in PipelineModel should be parallel Key: SPARK-27892 URL: https://issues.apache.org/jira/browse/SPARK-27892 Project: Spark Issue

[jira] [Assigned] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-27684: -- Assignee: Marco Gaido > Reduce ScalaUDF conversion overheads for primitives >

[jira] [Resolved] (SPARK-27684) Reduce ScalaUDF conversion overheads for primitives

[ https://issues.apache.org/jira/browse/SPARK-27684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-27684. Resolution: Fixed Fix Version/s: 3.0.0 Fixed for 3.0 in

[jira] [Assigned] (SPARK-27890) Improve SQL parser error message when missing backquotes for identifiers with hyphens

[ https://issues.apache.org/jira/browse/SPARK-27890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27890: Assignee: Apache Spark > Improve SQL parser error message when missing backquotes for

[jira] [Assigned] (SPARK-27890) Improve SQL parser error message when missing backquotes for identifiers with hyphens

[ https://issues.apache.org/jira/browse/SPARK-27890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27890: Assignee: (was: Apache Spark) > Improve SQL parser error message when missing

[jira] [Updated] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27772: -- Component/s: Tests > SQLTestUtils Refactoring > > >

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852420#comment-16852420 ] Marcelo Vanzin commented on SPARK-27891: Ok, the updated logs show the issue. But they're from

[jira] [Updated] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu updated SPARK-27891: Description: When the spark job runs on a secured cluster for longer then time that is

[jira] [Updated] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu updated SPARK-27891: Attachment: application_1559242207407_0001.log > Long running spark jobs fail because of

[jira] [Commented] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852408#comment-16852408 ] Marcelo Vanzin commented on SPARK-27891: {{container_e48_1559242207407_0001_02_01}} tells me

[jira] [Updated] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu updated SPARK-27891: Description: When the spark job runs on a secured cluster for longer then time that is

[jira] [Reopened] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hemshankar sahu reopened SPARK-27891: - We used following command to submit the job bin/spark-submit --principal acekrbuser

[jira] [Comment Edited] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852402#comment-16852402 ] Dongjoon Hyun edited comment on SPARK-27812 at 5/30/19 10:06 PM: - Thank

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852402#comment-16852402 ] Dongjoon Hyun commented on SPARK-27812: --- Thank you for reporting, [~Andrew HUALI] and thank you

[jira] [Issue Comment Deleted] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov updated SPARK-22151: -- Comment: was deleted (was: Is there is a workaround for this in Apache Livy? We're

[jira] [Resolved] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27891. Resolution: Not A Problem You have to provide a keytab for Spark for this to work.

[jira] [Created] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

hemshankar sahu created SPARK-27891: --- Summary: Long running spark jobs fail because of HDFS delegation token expires Key: SPARK-27891 URL: https://issues.apache.org/jira/browse/SPARK-27891 Project:

[jira] [Resolved] (SPARK-27361) YARN support for GPU-aware scheduling

[ https://issues.apache.org/jira/browse/SPARK-27361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-27361. --- Resolution: Fixed Fix Version/s: 3.0.0 all the subtasks are finished and the parts

[jira] [Commented] (SPARK-22151) PYTHONPATH not picked up from the spark.yarn.appMasterEnv properly

[ https://issues.apache.org/jira/browse/SPARK-22151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852381#comment-16852381 ] Ruslan Dautkhanov commented on SPARK-22151: --- Is there is a workaround for this in Apache Livy?

[jira] [Created] (SPARK-27890) Improve SQL parser error message when missing backquotes for identifiers with hyphens

Yesheng Ma created SPARK-27890: -- Summary: Improve SQL parser error message when missing backquotes for identifiers with hyphens Key: SPARK-27890 URL: https://issues.apache.org/jira/browse/SPARK-27890

[jira] [Assigned] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27872: Assignee: Apache Spark > Driver and executors use a different service account breaking

[jira] [Assigned] (SPARK-27872) Driver and executors use a different service account breaking pull secrets

[ https://issues.apache.org/jira/browse/SPARK-27872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27872: Assignee: (was: Apache Spark) > Driver and executors use a different service account

[jira] [Commented] (SPARK-24149) Automatic namespaces discovery in HDFS federation

[ https://issues.apache.org/jira/browse/SPARK-24149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852365#comment-16852365 ] Dhruve Ashar commented on SPARK-24149: -- IMHO having a consistent way to reason about a behavior is

[jira] [Resolved] (SPARK-27773) Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

[ https://issues.apache.org/jira/browse/SPARK-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27773. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24645

[jira] [Assigned] (SPARK-27773) Add shuffle service metric for number of exceptions caught in ExternalShuffleBlockHandler

[ https://issues.apache.org/jira/browse/SPARK-27773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27773: -- Assignee: Steven Rand > Add shuffle service metric for number of exceptions caught

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852342#comment-16852342 ] Igor Calabria commented on SPARK-27812: --- I believe this was introduced when `kubernetes-client`

[jira] [Resolved] (SPARK-27378) spark-submit requests GPUs in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27378. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24634

[jira] [Assigned] (SPARK-27378) spark-submit requests GPUs in YARN mode

[ https://issues.apache.org/jira/browse/SPARK-27378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-27378: -- Assignee: Thomas Graves > spark-submit requests GPUs in YARN mode >

[jira] [Commented] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852276#comment-16852276 ] Xiangrui Meng commented on SPARK-27886: --- By "at the end of year", you mean year 2020, right? The

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852268#comment-16852268 ] Dongjoon Hyun commented on SPARK-27798: --- Thank you so much for the investigation and update,

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852238#comment-16852238 ] Dongjoon Hyun commented on SPARK-27884: --- Thanks! +1 for this efforts > Deprecate Python 2 support

[jira] [Commented] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852166#comment-16852166 ] Sean Owen commented on SPARK-27886: --- Maybe I misunderstand, but if it's likely Python 2 goes away from

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852149#comment-16852149 ] Sean Owen commented on SPARK-27884: --- Yes that looks fine to me. > Deprecate Python 2 support in Spark

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852148#comment-16852148 ] William Wong commented on SPARK-27772: -- Hi [~hyukjin.kwon], I submitted the PR and add a test case

[jira] [Created] (SPARK-27889) Make development scripts under dev/ support Python 3

Xiangrui Meng created SPARK-27889: - Summary: Make development scripts under dev/ support Python 3 Key: SPARK-27889 URL: https://issues.apache.org/jira/browse/SPARK-27889 Project: Spark Issue

[jira] [Commented] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852136#comment-16852136 ] Gabor Somogyi commented on SPARK-27648: --- I've made my graphs based on the files you've provided

[jira] [Updated] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27772: - Description: The current `SQLTestUtils` created many `withXXX` utility functions to clean up

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852125#comment-16852125 ] Apache Spark commented on SPARK-27772: -- User 'William1104' has created a pull request for this

[jira] [Assigned] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27772: Assignee: Apache Spark > SQLTestUtils Refactoring > > >

[jira] [Assigned] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27772: Assignee: (was: Apache Spark) > SQLTestUtils Refactoring >

[jira] [Commented] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852124#comment-16852124 ] Apache Spark commented on SPARK-27772: -- User 'William1104' has created a pull request for this

[jira] [Updated] (SPARK-27772) SQLTestUtils Refactoring

[ https://issues.apache.org/jira/browse/SPARK-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27772: - Description: The current `SQLTestUtils` created many `withXXX` utility functions to clean up

[jira] [Commented] (SPARK-27785) Introduce .joinWith() overloads for typed inner joins of 3 or more tables

[ https://issues.apache.org/jira/browse/SPARK-27785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852089#comment-16852089 ] Josh Rosen commented on SPARK-27785: I think this might require a little bit more design work before

[jira] [Updated] (SPARK-27885) Announce deprecation of Python 2 support

[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Description: * Draft the message. * Update Spark website and announce deprecation of Python 2

[jira] [Commented] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852088#comment-16852088 ] Xiangrui Meng commented on SPARK-27886: --- cc: [~srowen] [~smilegator] > Add Apache Spark project

[jira] [Assigned] (SPARK-27831) Move Hive test jars to maven dependency

[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27831: Assignee: (was: Apache Spark) > Move Hive test jars to maven dependency >

[jira] [Updated] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27886: -- Description: Add Spark to https://python3statement.org/ and indicate our timeline. I

[jira] [Assigned] (SPARK-27831) Move Hive test jars to maven dependency

[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27831: Assignee: (was: Apache Spark) > Move Hive test jars to maven dependency >

[jira] [Assigned] (SPARK-27831) Move Hive test jars to maven dependency

[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27831: Assignee: Apache Spark > Move Hive test jars to maven dependency >

[jira] [Updated] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

[ https://issues.apache.org/jira/browse/SPARK-27886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27886: -- Description: Add Spark to https://python3statement.org/ and indicate our timeline. I

[jira] [Reopened] (SPARK-27831) Move Hive test jars to maven dependency

[ https://issues.apache.org/jira/browse/SPARK-27831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reopened SPARK-27831: --- Assignee: (was: Yuming Wang) > Move Hive test jars to maven dependency >

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852077#comment-16852077 ] Xiangrui Meng commented on SPARK-27884: --- [~srowen] Could you help review the draft? Feel free to

[jira] [Resolved] (SPARK-27813) DataSourceV2: Add DropTable logical operation

[ https://issues.apache.org/jira/browse/SPARK-27813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27813. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24686

[jira] [Commented] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852071#comment-16852071 ] Xiangrui Meng commented on SPARK-27884: --- Draft message: *Apache Spark's plan for dropping Python

[jira] [Assigned] (SPARK-27813) DataSourceV2: Add DropTable logical operation

[ https://issues.apache.org/jira/browse/SPARK-27813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27813: --- Assignee: John Zhuge > DataSourceV2: Add DropTable logical operation >

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852055#comment-16852055 ] Gengliang Wang commented on SPARK-27798: Also, the issue can be reproduced on latest master if

[jira] [Commented] (SPARK-25557) ORC predicate pushdown for nested fields

[ https://issues.apache.org/jira/browse/SPARK-25557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852053#comment-16852053 ] Dongjoon Hyun commented on SPARK-25557: --- Nope. The nested scheme pruning is independent from

[jira] [Updated] (SPARK-27862) Upgrade json4s-jackson to 3.6.6

[ https://issues.apache.org/jira/browse/SPARK-27862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Izek Greenfield updated SPARK-27862: Summary: Upgrade json4s-jackson to 3.6.6 (was: Upgrade json4s-jackson to 3.6.5) >

[jira] [Updated] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

[ https://issues.apache.org/jira/browse/SPARK-27884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27884: -- Description: Officially deprecate Python 2 support in Spark 3.0. (was: Officially deprecate

[jira] [Updated] (SPARK-27885) Announce deprecation of Python 2 support

[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Description: * Draft the message. * Update Spark website and announce deprecation of Python 2

[jira] [Created] (SPARK-27888) Python 2->3 migration guide for PySpark users

Xiangrui Meng created SPARK-27888: - Summary: Python 2->3 migration guide for PySpark users Key: SPARK-27888 URL: https://issues.apache.org/jira/browse/SPARK-27888 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27885) Announce deprecation of Python 2 support

[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Summary: Announce deprecation of Python 2 support (was: Update Spark website and put

[jira] [Updated] (SPARK-27887) Check python version and print deprecation warning if version < 3

[ https://issues.apache.org/jira/browse/SPARK-27887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27887: -- Description: In Spark 3.0, users should see a deprecation warning if they use PySpark with

[jira] [Created] (SPARK-27887) Check python version and print deprecation warning if version < 3

Xiangrui Meng created SPARK-27887: - Summary: Check python version and print deprecation warning if version < 3 Key: SPARK-27887 URL: https://issues.apache.org/jira/browse/SPARK-27887 Project: Spark

[jira] [Updated] (SPARK-27885) Update Spark website and put deprecation warning

[ https://issues.apache.org/jira/browse/SPARK-27885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27885: -- Description: Update Spark website and announce deprecation of Python 2 support in the next

[jira] [Created] (SPARK-27886) Add Apache Spark project to https://python3statement.org/

Xiangrui Meng created SPARK-27886: - Summary: Add Apache Spark project to https://python3statement.org/ Key: SPARK-27886 URL: https://issues.apache.org/jira/browse/SPARK-27886 Project: Spark

[jira] [Created] (SPARK-27885) Update Spark website and put deprecation warning

Xiangrui Meng created SPARK-27885: - Summary: Update Spark website and put deprecation warning Key: SPARK-27885 URL: https://issues.apache.org/jira/browse/SPARK-27885 Project: Spark Issue

[jira] [Created] (SPARK-27884) Deprecate Python 2 support in Spark 3.0

Xiangrui Meng created SPARK-27884: - Summary: Deprecate Python 2 support in Spark 3.0 Key: SPARK-27884 URL: https://issues.apache.org/jira/browse/SPARK-27884 Project: Spark Issue Type: Story

[jira] [Commented] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851979#comment-16851979 ] Gabor Somogyi commented on SPARK-27742: --- {quote}For the user this means better flexibility per job

[jira] [Comment Edited] (SPARK-27798) from_avro can modify variables in other rows in local mode

[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851966#comment-16851966 ] Gengliang Wang edited comment on SPARK-27798 at 5/30/19 3:07 PM: -

[jira] [Commented] (SPARK-27798) from_avro can modify variables in other rows in local mode

[ https://issues.apache.org/jira/browse/SPARK-27798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851966#comment-16851966 ] Gengliang Wang commented on SPARK-27798: Turning off the rule "ConvertToLocalRelation" should

[jira] [Closed] (SPARK-27706) Add SQL metrics of numOutputRows for BroadcastExchangeExec

[ https://issues.apache.org/jira/browse/SPARK-27706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27706. - > Add SQL metrics of numOutputRows for BroadcastExchangeExec >

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:58 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:57 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:55 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:55 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:53 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:53 PM:

[jira] [Commented] (SPARK-27873) Csv reader, adding a corrupt record column causes error if enforceSchema=false

[ https://issues.apache.org/jira/browse/SPARK-27873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851952#comment-16851952 ] Liang-Chi Hsieh commented on SPARK-27873: - I can prepare a PR if Marcin or Hyukjin Kwon don't

[jira] [Commented] (SPARK-27873) Csv reader, adding a corrupt record column causes error if enforceSchema=false

[ https://issues.apache.org/jira/browse/SPARK-27873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851948#comment-16851948 ] Liang-Chi Hsieh commented on SPARK-27873: - I guess what Marcin meant is: {code} val schema =

[jira] [Updated] (SPARK-27757) Bump Jackson to 2.9.9

[ https://issues.apache.org/jira/browse/SPARK-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27757: -- Priority: Minor (was: Major) > Bump Jackson to 2.9.9 > - > >

[jira] [Resolved] (SPARK-27757) Bump Jackson to 2.9.9

[ https://issues.apache.org/jira/browse/SPARK-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27757. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24646

[jira] [Assigned] (SPARK-27757) Bump Jackson to 2.9.9

[ https://issues.apache.org/jira/browse/SPARK-27757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27757: - Assignee: Fokko Driesprong > Bump Jackson to 2.9.9 > - > >

[jira] [Assigned] (SPARK-27883) Add aggregates.sql - Part2

[ https://issues.apache.org/jira/browse/SPARK-27883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27883: Assignee: (was: Apache Spark) > Add aggregates.sql - Part2 >

[jira] [Assigned] (SPARK-27883) Add aggregates.sql - Part2

[ https://issues.apache.org/jira/browse/SPARK-27883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27883: Assignee: Apache Spark > Add aggregates.sql - Part2 > -- > >

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:28 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:27 PM:

[jira] [Comment Edited] (SPARK-27742) Security Support in Sources and Sinks for SS and Batch

[ https://issues.apache.org/jira/browse/SPARK-27742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851924#comment-16851924 ] Stavros Kontopoulos edited comment on SPARK-27742 at 5/30/19 2:24 PM:

  1   2   >