[jira] [Assigned] (SPARK-33354) New explicit cast syntax rules in ANSI mode

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33354: Assignee: Apache Spark (was: Gengliang Wang) > New explicit cast syntax rules in ANSI

[jira] [Commented] (SPARK-33354) New explicit cast syntax rules in ANSI mode

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226558#comment-17226558 ] Apache Spark commented on SPARK-33354: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-33354) New explicit cast syntax rules in ANSI mode

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33354: Assignee: Gengliang Wang (was: Apache Spark) > New explicit cast syntax rules in ANSI

[jira] [Created] (SPARK-33354) New explicit cast syntax rules in ANSI mode

2020-11-04 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-33354: -- Summary: New explicit cast syntax rules in ANSI mode Key: SPARK-33354 URL: https://issues.apache.org/jira/browse/SPARK-33354 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-33277) Python/Pandas UDF right after off-heap vectorized reader could cause executor crash.

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33277: Assignee: Apache Spark > Python/Pandas UDF right after off-heap vectorized reader could

[jira] [Assigned] (SPARK-33277) Python/Pandas UDF right after off-heap vectorized reader could cause executor crash.

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33277: Assignee: (was: Apache Spark) > Python/Pandas UDF right after off-heap vectorized

[jira] [Reopened] (SPARK-33277) Python/Pandas UDF right after off-heap vectorized reader could cause executor crash.

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-33277: -- Assignee: (was: Takuya Ueshin) Reverted in: master:

[jira] [Updated] (SPARK-33277) Python/Pandas UDF right after off-heap vectorized reader could cause executor crash.

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33277: - Fix Version/s: (was: 3.0.2) (was: 2.4.8) (was:

[jira] [Comment Edited] (SPARK-33331) Limit the number of pending blocks in memory and store blocks that collide

2020-11-04 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226531#comment-17226531 ] wuyi edited comment on SPARK-1 at 11/5/20, 7:12 AM: I like the idea to cache

[jira] [Assigned] (SPARK-33282) Replace Probot Autolabeler with Github Action

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33282: Assignee: Kyle Bendickson > Replace Probot Autolabeler with Github Action >

[jira] [Resolved] (SPARK-33282) Replace Probot Autolabeler with Github Action

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33282. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30244

[jira] [Comment Edited] (SPARK-33331) Limit the number of pending blocks in memory and store blocks that collide

2020-11-04 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226531#comment-17226531 ] wuyi edited comment on SPARK-1 at 11/5/20, 7:09 AM: I like the idea to cache

[jira] [Comment Edited] (SPARK-33331) Limit the number of pending blocks in memory and store blocks that collide

2020-11-04 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226531#comment-17226531 ] wuyi edited comment on SPARK-1 at 11/5/20, 6:59 AM: I like the idea to cache

[jira] [Commented] (SPARK-33331) Limit the number of pending blocks in memory and store blocks that collide

2020-11-04 Thread wuyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226531#comment-17226531 ] wuyi commented on SPARK-1: -- I like the idea to cache the blocks of the worst case instead of throwing

[jira] [Updated] (SPARK-33239) Use pre-built image at GitHub Action SparkR job

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33239: -- Fix Version/s: 3.0.2 > Use pre-built image at GitHub Action SparkR job >

[jira] [Commented] (SPARK-33353) Cache dependencies for Coursier with new sbt in GitHub Actions

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226494#comment-17226494 ] Apache Spark commented on SPARK-33353: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33353) Cache dependencies for Coursier with new sbt in GitHub Actions

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33353: Assignee: Kousuke Saruta (was: Apache Spark) > Cache dependencies for Coursier with new

[jira] [Assigned] (SPARK-33353) Cache dependencies for Coursier with new sbt in GitHub Actions

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33353: Assignee: Apache Spark (was: Kousuke Saruta) > Cache dependencies for Coursier with new

[jira] [Created] (SPARK-33353) Cache dependencies for Coursier with new sbt in GitHub Actions

2020-11-04 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-33353: -- Summary: Cache dependencies for Coursier with new sbt in GitHub Actions Key: SPARK-33353 URL: https://issues.apache.org/jira/browse/SPARK-33353 Project: Spark

[jira] [Commented] (SPARK-33239) Use pre-built image at GitHub Action SparkR job

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226490#comment-17226490 ] Apache Spark commented on SPARK-33239: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Assigned] (SPARK-33316) Support nullable Avro schemas for non-nullable data in Avro writing

2020-11-04 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang reassigned SPARK-33316: -- Assignee: Bo Zhang > Support nullable Avro schemas for non-nullable data in Avro

[jira] [Updated] (SPARK-33162) Use pre-built image at GitHub Action PySpark jobs

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33162: -- Fix Version/s: 3.0.2 > Use pre-built image at GitHub Action PySpark jobs >

[jira] [Resolved] (SPARK-33316) Support nullable Avro schemas for non-nullable data in Avro writing

2020-11-04 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-33316. Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30224

[jira] [Assigned] (SPARK-33351) WithColumn should add a column with specific position

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33351: Assignee: (was: Apache Spark) > WithColumn should add a column with specific

[jira] [Commented] (SPARK-33351) WithColumn should add a column with specific position

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226482#comment-17226482 ] Apache Spark commented on SPARK-33351: -- User 'Karl-WangSK' has created a pull request for this

[jira] [Assigned] (SPARK-33351) WithColumn should add a column with specific position

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33351: Assignee: Apache Spark > WithColumn should add a column with specific position >

[jira] [Commented] (SPARK-33351) WithColumn should add a column with specific position

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226483#comment-17226483 ] Apache Spark commented on SPARK-33351: -- User 'Karl-WangSK' has created a pull request for this

[jira] [Commented] (SPARK-33290) REFRESH TABLE should invalidate cache even though the table itself may not be cached

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226476#comment-17226476 ] Apache Spark commented on SPARK-33290: -- User 'sunchao' has created a pull request for this issue:

[jira] [Commented] (SPARK-33290) REFRESH TABLE should invalidate cache even though the table itself may not be cached

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226475#comment-17226475 ] Apache Spark commented on SPARK-33290: -- User 'sunchao' has created a pull request for this issue:

[jira] [Commented] (SPARK-33352) Fix procedure-like declaration compilation warning in Scala 2.13

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226473#comment-17226473 ] Apache Spark commented on SPARK-33352: -- User 'LuciferYang' has created a pull request for this

[jira] [Commented] (SPARK-33352) Fix procedure-like declaration compilation warning in Scala 2.13

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226472#comment-17226472 ] Apache Spark commented on SPARK-33352: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-33352) Fix procedure-like declaration compilation warning in Scala 2.13

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33352: Assignee: Apache Spark > Fix procedure-like declaration compilation warning in Scala

[jira] [Assigned] (SPARK-33352) Fix procedure-like declaration compilation warning in Scala 2.13

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33352: Assignee: (was: Apache Spark) > Fix procedure-like declaration compilation warning

[jira] [Resolved] (SPARK-33314) Avro reader drops rows

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33314. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30221

[jira] [Assigned] (SPARK-33314) Avro reader drops rows

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-33314: Assignee: Bruce Robbins > Avro reader drops rows > -- > >

[jira] [Created] (SPARK-33352) Fix procedure-like declaration compilation warning in Scala 2.13

2020-11-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-33352: Summary: Fix procedure-like declaration compilation warning in Scala 2.13 Key: SPARK-33352 URL: https://issues.apache.org/jira/browse/SPARK-33352 Project: Spark

[jira] [Updated] (SPARK-33351) WithColumn should add a column with specific position

2020-11-04 Thread karl wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] karl wang updated SPARK-33351: -- Description: In `DataSet`, WithColumn usually add a new col at the end of the DF. But  sometime

[jira] [Created] (SPARK-33351) WithColumn should add a column with specific position

2020-11-04 Thread karl wang (Jira)
karl wang created SPARK-33351: - Summary: WithColumn should add a column with specific position Key: SPARK-33351 URL: https://issues.apache.org/jira/browse/SPARK-33351 Project: Spark Issue Type:

[jira] [Updated] (SPARK-33350) Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data

2020-11-04 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-33350: -- Summary: Add support to DiskBlockManager to create merge directory and to get the local

[jira] [Created] (SPARK-33350) Add support to DiskBlockManager to create merge directory and the ability to get the shuffle merged data

2020-11-04 Thread Chandni Singh (Jira)
Chandni Singh created SPARK-33350: - Summary: Add support to DiskBlockManager to create merge directory and the ability to get the shuffle merged data Key: SPARK-33350 URL:

[jira] [Resolved] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-33343. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30250

[jira] [Resolved] (SPARK-31711) Register the executor source with the metrics system when running in local mode.

2020-11-04 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-31711. --- Fix Version/s: 3.1.0 Resolution: Fixed > Register the executor source with the

[jira] [Assigned] (SPARK-31711) Register the executor source with the metrics system when running in local mode.

2020-11-04 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-31711: - Assignee: Luca Canali > Register the executor source with the metrics system when

[jira] [Commented] (SPARK-33282) Replace Probot Autolabeler with Github Action

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226414#comment-17226414 ] Apache Spark commented on SPARK-33282: -- User 'kbendick' has created a pull request for this issue:

[jira] [Commented] (SPARK-33282) Replace Probot Autolabeler with Github Action

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226413#comment-17226413 ] Apache Spark commented on SPARK-33282: -- User 'kbendick' has created a pull request for this issue:

[jira] [Commented] (SPARK-33162) Use pre-built image at GitHub Action PySpark jobs

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226408#comment-17226408 ] Apache Spark commented on SPARK-33162: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-33162) Use pre-built image at GitHub Action PySpark jobs

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226407#comment-17226407 ] Apache Spark commented on SPARK-33162: -- User 'dongjoon-hyun' has created a pull request for this

[jira] [Commented] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226395#comment-17226395 ] Dongjoon Hyun commented on SPARK-33349: --- I converted this to a subtask of SPARK-33005 to give more

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33349: -- Affects Version/s: 3.1.0 > ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-33349: -- Parent: SPARK-33005 Issue Type: Sub-task (was: Bug) >

[jira] [Commented] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226393#comment-17226393 ] Dongjoon Hyun commented on SPARK-33349: --- Thanks, [~jkleckner]. It looks like a breaking change.

[jira] [Updated] (SPARK-33331) Limit the number of pending blocks in memory and store blocks that collide

2020-11-04 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-1: -- Description: This jira addresses the below two points: 1. In {{RemoteBlockPushResolver}},

[jira] [Updated] (SPARK-33331) Limit the number of pending blocks in memory and store blocks that collide

2020-11-04 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-1: -- Description: This jira addresses the below two points: 1. In {{RemoteBlockPushResolver}},

[jira] [Updated] (SPARK-33331) Limit the number of pending blocks in memory and store blocks that collide

2020-11-04 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-1: -- Summary: Limit the number of pending blocks in memory and store blocks that collide (was:

[jira] [Updated] (SPARK-33331) Limit the number of pending blocks in memory when RemoteBlockPushResolver defers a block

2020-11-04 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-1: -- Description: 1. In {{RemoteBlockPushResolver}}, bytes that cannot be merged immediately are

[jira] [Commented] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Jim Kleckner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226362#comment-17226362 ] Jim Kleckner commented on SPARK-33349: -- This fabric8 kubernetes-client issue/MR looks relevant:

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola Bova updated SPARK-33349: Shepherd: Dongjoon Hyun > ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed >

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola Bova updated SPARK-33349: Description: I launch my spark application with the

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola Bova updated SPARK-33349: Description: I launch my spark application with the

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola Bova updated SPARK-33349: Description: I launch my spark application with the

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola Bova updated SPARK-33349: Description: I launch my spark application with the

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola Bova updated SPARK-33349: Description: I launch my spark application with the

[jira] [Updated] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicola Bova updated SPARK-33349: Description: I launch my spark application with the

[jira] [Created] (SPARK-33349) ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed

2020-11-04 Thread Nicola Bova (Jira)
Nicola Bova created SPARK-33349: --- Summary: ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed Key: SPARK-33349 URL: https://issues.apache.org/jira/browse/SPARK-33349 Project: Spark

[jira] [Created] (SPARK-33348) Use scala.jdk.CollectionConverters replace scala.collection.JavaConverters

2020-11-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-33348: Summary: Use scala.jdk.CollectionConverters replace scala.collection.JavaConverters Key: SPARK-33348 URL: https://issues.apache.org/jira/browse/SPARK-33348 Project: Spark

[jira] [Updated] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-33343: --- Description: With the current master, spark-shell doesn't work if it's built with sbt

[jira] [Updated] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Kousuke Saruta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kousuke Saruta updated SPARK-33343: --- Priority: Major (was: Critical) > Fix the build with sbt to copy hadoop-client-runtime.jar

[jira] [Resolved] (SPARK-33338) GROUP BY using literal map should not fail

2020-11-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-8?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-8. --- Fix Version/s: 2.4.8 3.0.2 3.1.0 Resolution:

[jira] [Assigned] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33347: Assignee: (was: Apache Spark) > Clean up useless variables in MutableApplicationInfo

[jira] [Commented] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226265#comment-17226265 ] Apache Spark commented on SPARK-33347: -- User 'LuciferYang' has created a pull request for this

[jira] [Assigned] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33347: Assignee: Apache Spark > Clean up useless variables in MutableApplicationInfo >

[jira] [Updated] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-33347: - Description:   {code:java} private class MutableApplicationInfo { var id: String = null var name:

[jira] [Updated] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-33347: - Description:   {code:java} private class MutableApplicationInfo { var id: String = null var name:

[jira] [Updated] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yang Jie updated SPARK-33347: - Description:   {code:java} private class MutableApplicationInfo { var id: String = null var name:

[jira] [Created] (SPARK-33347) Clean up useless variables in MutableApplicationInfo

2020-11-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-33347: Summary: Clean up useless variables in MutableApplicationInfo Key: SPARK-33347 URL: https://issues.apache.org/jira/browse/SPARK-33347 Project: Spark Issue Type:

[jira] [Commented] (SPARK-33346) Change the never changed var to val

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226255#comment-17226255 ] Yang Jie commented on SPARK-33346: -- Above case still can't use as val, also throw illegal access error

[jira] [Commented] (SPARK-33285) Too many "Auto-application to `()` is deprecated." related compilation warnings

2020-11-04 Thread Guillaume Martres (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226251#comment-17226251 ] Guillaume Martres commented on SPARK-33285: --- Note that Scala 2.13 has a configurable warning

[jira] [Commented] (SPARK-33285) Too many "Auto-application to `()` is deprecated." related compilation warnings

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226250#comment-17226250 ] Yang Jie commented on SPARK-33285: -- {quote}Replacing {{'foo}} by {{Symbol("foo")}} will get rid of the

[jira] [Commented] (SPARK-33285) Too many "Auto-application to `()` is deprecated." related compilation warnings

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226248#comment-17226248 ] Yang Jie commented on SPARK-33285: -- [~srowen] Yes, "Auto-application to `()` is deprecated."  warnings

[jira] [Commented] (SPARK-29392) Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name")

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226244#comment-17226244 ] Yang Jie commented on SPARK-29392: -- OK > Remove use of deprecated symbol literal " 'name " syntax in

[jira] [Resolved] (SPARK-33341) Remove unnecessary semicolons

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-33341. -- Resolution: Won't Fix > Remove unnecessary semicolons > - > >

[jira] [Commented] (SPARK-29392) Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name")

2020-11-04 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226230#comment-17226230 ] Sean R. Owen commented on SPARK-29392: -- For this one, we already started I suppose, so OK to

[jira] [Comment Edited] (SPARK-33346) Change the never changed var to val

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226228#comment-17226228 ] Yang Jie edited comment on SPARK-33346 at 11/4/20, 3:25 PM: There are some

[jira] [Commented] (SPARK-33346) Change the never changed var to val

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226228#comment-17226228 ] Yang Jie commented on SPARK-33346: -- There are some code and comments as follow: {code:java} // They

[jira] [Commented] (SPARK-33285) Too many "Auto-application to `()` is deprecated." related compilation warnings

2020-11-04 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226189#comment-17226189 ] Sean R. Owen commented on SPARK-33285: -- I think it's OK to fix the Symbol issue (we already started

[jira] [Created] (SPARK-33346) Change the never changed var to val

2020-11-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-33346: Summary: Change the never changed var to val Key: SPARK-33346 URL: https://issues.apache.org/jira/browse/SPARK-33346 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-33345) Batch fix compilation warnings about "Widening conversion from XXX to XXX is deprecated"

2020-11-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-33345: Summary: Batch fix compilation warnings about "Widening conversion from XXX to XXX is deprecated" Key: SPARK-33345 URL: https://issues.apache.org/jira/browse/SPARK-33345

[jira] [Created] (SPARK-33344) Fix Compilation warings of "multiarg infix syntax looks like a tuple and will be deprecated" in Scala 2.13

2020-11-04 Thread Yang Jie (Jira)
Yang Jie created SPARK-33344: Summary: Fix Compilation warings of "multiarg infix syntax looks like a tuple and will be deprecated" in Scala 2.13 Key: SPARK-33344 URL:

[jira] [Commented] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226080#comment-17226080 ] Apache Spark commented on SPARK-33343: -- User 'sarutak' has created a pull request for this issue:

[jira] [Assigned] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33343: Assignee: Apache Spark (was: Kousuke Saruta) > Fix the build with sbt to copy

[jira] [Assigned] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-33343: Assignee: Kousuke Saruta (was: Apache Spark) > Fix the build with sbt to copy

[jira] [Commented] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226079#comment-17226079 ] Apache Spark commented on SPARK-33343: -- User 'sarutak' has created a pull request for this issue:

[jira] [Created] (SPARK-33343) Fix the build with sbt to copy hadoop-client-runtime.jar

2020-11-04 Thread Kousuke Saruta (Jira)
Kousuke Saruta created SPARK-33343: -- Summary: Fix the build with sbt to copy hadoop-client-runtime.jar Key: SPARK-33343 URL: https://issues.apache.org/jira/browse/SPARK-33343 Project: Spark

[jira] [Commented] (SPARK-23086) Spark SQL cannot support high concurrency for lock in HiveMetastoreCatalog

2020-11-04 Thread gaofeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226069#comment-17226069 ] gaofeng commented on SPARK-23086: - I also encountered this problem in the production environment. it's

[jira] [Commented] (SPARK-29392) Remove use of deprecated symbol literal " 'name " syntax in favor Symbol("name")

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226070#comment-17226070 ] Yang Jie commented on SPARK-29392: -- I can take the time to fix them by module, but is this the right

[jira] [Commented] (SPARK-23086) Spark SQL cannot support high concurrency for lock in HiveMetastoreCatalog

2020-11-04 Thread gaofeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-23086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226067#comment-17226067 ] gaofeng commented on SPARK-23086: - hi?how to resolve this problem > Spark SQL cannot support high

[jira] [Updated] (SPARK-33256) Update contribution guide about NumPy documentation style

2020-11-04 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33256?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-33256: - Description: We should document that PySpark uses NumPy documentation style. See also

[jira] [Commented] (SPARK-33285) Too many "Auto-application to `()` is deprecated." related compilation warnings

2020-11-04 Thread Guillaume Martres (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226040#comment-17226040 ] Guillaume Martres commented on SPARK-33285: --- {quote} Similarly, there are many "symbol literal

[jira] [Commented] (SPARK-32894) Timestamp cast in exernal orc table

2020-11-04 Thread Yang Jie (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226016#comment-17226016 ] Yang Jie commented on SPARK-32894: -- What is the data generation process?It seems that the data type is

[jira] [Commented] (SPARK-33325) Spark executors pod are not shutting down when losing driver connection

2020-11-04 Thread Hadrien Kohl (Jira)
[ https://issues.apache.org/jira/browse/SPARK-33325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17226009#comment-17226009 ] Hadrien Kohl commented on SPARK-33325: -- It looks one thread gets stuck here on the awaitTermination

  1   2   >