[jira] [Resolved] (SPARK-30093) Improve error message for creating views

2019-12-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30093. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26731

[jira] [Assigned] (SPARK-30093) Improve error message for creating views

2019-12-04 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30093: --- Assignee: Aman Omer > Improve error message for creating views >

[jira] [Updated] (SPARK-30135) Add documentation for DELETE JAR and DELETE File command

2019-12-04 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-30135: -- Summary: Add documentation for DELETE JAR and DELETE File command (was: Add documentation

[jira] [Commented] (SPARK-30137) Support DELETE file

2019-12-04 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988508#comment-16988508 ] Sandeep Katta commented on SPARK-30137: --- I started working on this feature, I will raise the PR

[jira] [Created] (SPARK-30137) Support DELETE file

2019-12-04 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-30137: - Summary: Support DELETE file Key: SPARK-30137 URL: https://issues.apache.org/jira/browse/SPARK-30137 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-30133) Support DELETE Jar and DELETE File functionality in spark

2019-12-04 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta updated SPARK-30133: -- Summary: Support DELETE Jar and DELETE File functionality in spark (was: Support DELETE Jar

[jira] [Created] (SPARK-30136) DELETE JAR should also remove the jar from executor classPath

2019-12-04 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-30136: - Summary: DELETE JAR should also remove the jar from executor classPath Key: SPARK-30136 URL: https://issues.apache.org/jira/browse/SPARK-30136 Project: Spark

[jira] [Commented] (SPARK-30136) DELETE JAR should also remove the jar from executor classPath

2019-12-04 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988504#comment-16988504 ] Sandeep Katta commented on SPARK-30136: --- I started working on this feature, I will raise the PR

[jira] [Commented] (SPARK-30134) DELETE JAR should remove from addedJars list and from classpath

2019-12-04 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988502#comment-16988502 ] Sandeep Katta commented on SPARK-30134: --- I started working on this feature, I will raise the PR

[jira] [Commented] (SPARK-30135) Add documentation for DELETE JAR command

2019-12-04 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988503#comment-16988503 ] Sandeep Katta commented on SPARK-30135: --- I started working on this feature, I will raise the PR

[jira] [Created] (SPARK-30134) DELETE JAR should remove from addedJars list and from classpath

2019-12-04 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-30134: - Summary: DELETE JAR should remove from addedJars list and from classpath Key: SPARK-30134 URL: https://issues.apache.org/jira/browse/SPARK-30134 Project: Spark

[jira] [Created] (SPARK-30135) Add documentation for DELETE JAR command

2019-12-04 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-30135: - Summary: Add documentation for DELETE JAR command Key: SPARK-30135 URL: https://issues.apache.org/jira/browse/SPARK-30135 Project: Spark Issue Type:

[jira] [Created] (SPARK-30133) Support DELETE Jar functionality in spark

2019-12-04 Thread Sandeep Katta (Jira)
Sandeep Katta created SPARK-30133: - Summary: Support DELETE Jar functionality in spark Key: SPARK-30133 URL: https://issues.apache.org/jira/browse/SPARK-30133 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-29915) spark-py and spark-r images are not created with docker-image-tool.sh

2019-12-04 Thread Veera (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988492#comment-16988492 ] Veera commented on SPARK-29915: --- Yes this option is not included , but you can add manually in docker

[jira] [Commented] (SPARK-29021) NoSuchElementException: key not found: hostPath.spark-local-dir-5.options.path

2019-12-04 Thread Veera (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988487#comment-16988487 ] Veera commented on SPARK-29021: --- Hi    Even im faing the issue while submiitng the spark  

[jira] [Updated] (SPARK-29152) Spark Executor Plugin API shutdown is not proper when dynamic allocation enabled

2019-12-04 Thread Rakesh Raushan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29152?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rakesh Raushan updated SPARK-29152: --- Affects Version/s: 3.0.0 > Spark Executor Plugin API shutdown is not proper when dynamic

[jira] [Updated] (SPARK-30121) Fix memory usage in sbt build script

2019-12-04 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-30121: - Description: 1. the default memory setting is missing in usage instructions {code:java} ``` build/sbt

[jira] [Updated] (SPARK-30121) Fix memory usage in sbt build script

2019-12-04 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-30121: - Summary: Fix memory usage in sbt build script (was: Miss default memory setting for sbt usage) > Fix

[jira] [Resolved] (SPARK-30129) New auth engine does not keep client ID in TransportClient after auth

2019-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30129. --- Fix Version/s: 3.0.0 Resolution: Fixed This is resolved via

[jira] [Assigned] (SPARK-30129) New auth engine does not keep client ID in TransportClient after auth

2019-12-04 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30129: - Assignee: Marcelo Masiero Vanzin > New auth engine does not keep client ID in

[jira] [Created] (SPARK-30132) Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses

2019-12-04 Thread Sean R. Owen (Jira)
Sean R. Owen created SPARK-30132: Summary: Scala 2.13 compile errors from Hadoop LocalFileSystem subclasses Key: SPARK-30132 URL: https://issues.apache.org/jira/browse/SPARK-30132 Project: Spark

[jira] [Commented] (SPARK-30125) Remove PostgreSQL dialect

2019-12-04 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988301#comment-16988301 ] Yuanjian Li commented on SPARK-30125: - [~aman_omer] Sorry, my PR was a little delayed by fixing

[jira] [Assigned] (SPARK-30084) Add docs showing how to automatically rebuild Python API docs

2019-12-04 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-30084: Assignee: Nicholas Chammas > Add docs showing how to automatically rebuild Python API

[jira] [Resolved] (SPARK-30084) Add docs showing how to automatically rebuild Python API docs

2019-12-04 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-30084. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26719

[jira] [Commented] (SPARK-30131) Add array_median function

2019-12-04 Thread Alexander Hagerf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988289#comment-16988289 ] Alexander Hagerf commented on SPARK-30131: -- Created PR for this issue:

[jira] [Updated] (SPARK-30131) Add array_median function

2019-12-04 Thread Alexander Hagerf (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alexander Hagerf updated SPARK-30131: - Description: It is known that there isn't any exact median function in Spark SQL, and

[jira] [Created] (SPARK-30131) Add array_median function

2019-12-04 Thread Alexander Hagerf (Jira)
Alexander Hagerf created SPARK-30131: Summary: Add array_median function Key: SPARK-30131 URL: https://issues.apache.org/jira/browse/SPARK-30131 Project: Spark Issue Type: New Feature

[jira] [Reopened] (SPARK-29640) [K8S] Intermittent "java.net.UnknownHostException: kubernetes.default.svc" in Spark driver

2019-12-04 Thread Andy Grove (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove reopened SPARK-29640: Re-opening this as the issue came back and is ongoing. I am exploring other solutions and will post

[jira] [Commented] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

2019-12-04 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988168#comment-16988168 ] Ankit Raj Boudh commented on SPARK-30130: - i will check this issue > Hardcoded numeric values

[jira] [Resolved] (SPARK-29453) Improve tooltip information for SQL tab

2019-12-04 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29453. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26641

[jira] [Assigned] (SPARK-29453) Improve tooltip information for SQL tab

2019-12-04 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29453: Assignee: Ankit Raj Boudh > Improve tooltip information for SQL tab >

[jira] [Commented] (SPARK-30118) ALTER VIEW QUERY does not work

2019-12-04 Thread John Zhuge (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988078#comment-16988078 ] John Zhuge commented on SPARK-30118: [~cltlfcjin] Thanks for the comment. Do you know which commit

[jira] [Updated] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

2019-12-04 Thread Matt Boegner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt Boegner updated SPARK-30130: - Description: Hardcoded numeric values in common table expressions which utilize GROUP BY are

[jira] [Created] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

2019-12-04 Thread Matt Boegner (Jira)
Matt Boegner created SPARK-30130: Summary: Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions Key: SPARK-30130 URL:

[jira] [Created] (SPARK-30129) New auth engine does not keep client ID in TransportClient after auth

2019-12-04 Thread Marcelo Masiero Vanzin (Jira)
Marcelo Masiero Vanzin created SPARK-30129: -- Summary: New auth engine does not keep client ID in TransportClient after auth Key: SPARK-30129 URL: https://issues.apache.org/jira/browse/SPARK-30129

[jira] [Commented] (SPARK-30125) Remove PostgreSQL dialect

2019-12-04 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988032#comment-16988032 ] Aman Omer commented on SPARK-30125: --- I would like to take this work. > Remove PostgreSQL dialect >

[jira] [Reopened] (SPARK-21488) Make saveAsTable() and createOrReplaceTempView() return dataframe of created table/ created view

2019-12-04 Thread Ruslan Dautkhanov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ruslan Dautkhanov reopened SPARK-21488: --- > Make saveAsTable() and createOrReplaceTempView() return dataframe of created >

[jira] [Commented] (SPARK-21488) Make saveAsTable() and createOrReplaceTempView() return dataframe of created table/ created view

2019-12-04 Thread Ruslan Dautkhanov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-21488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988011#comment-16988011 ] Ruslan Dautkhanov commented on SPARK-21488: --- [~zsxwing] any chance this can be added to Spark

[jira] [Commented] (SPARK-30126) sparkContext.addFile fails when file path contains spaces

2019-12-04 Thread Ankit Raj Boudh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16988001#comment-16988001 ] Ankit Raj Boudh commented on SPARK-30126: - i will check this issue > sparkContext.addFile fails

[jira] [Commented] (SPARK-19842) Informational Referential Integrity Constraints Support in Spark

2019-12-04 Thread Ruslan Dautkhanov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987999#comment-16987999 ] Ruslan Dautkhanov commented on SPARK-19842: --- >From the design document  """ This alternative

[jira] [Updated] (SPARK-30128) Promote remaining "hidden" PySpark DataFrameReader options to load APIs

2019-12-04 Thread Nicholas Chammas (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicholas Chammas updated SPARK-30128: - Description: Following on to SPARK-29903 and similar issues (linked), there are options

[jira] [Created] (SPARK-30128) Promote remaining "hidden" PySpark DataFrameReader options to load APIs

2019-12-04 Thread Nicholas Chammas (Jira)
Nicholas Chammas created SPARK-30128: Summary: Promote remaining "hidden" PySpark DataFrameReader options to load APIs Key: SPARK-30128 URL: https://issues.apache.org/jira/browse/SPARK-30128

[jira] [Created] (SPARK-30127) UDF should work for case class like Dataset operations

2019-12-04 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30127: --- Summary: UDF should work for case class like Dataset operations Key: SPARK-30127 URL: https://issues.apache.org/jira/browse/SPARK-30127 Project: Spark Issue

[jira] [Created] (SPARK-30126) sparkContext.addFile fails when file path contains spaces

2019-12-04 Thread Jan (Jira)
Jan created SPARK-30126: --- Summary: sparkContext.addFile fails when file path contains spaces Key: SPARK-30126 URL: https://issues.apache.org/jira/browse/SPARK-30126 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-30125) Remove PostgreSQL dialect

2019-12-04 Thread Yuanjian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuanjian Li updated SPARK-30125: Summary: Remove PostgreSQL dialect (was: Remove PostgreSQL dialect to reduce codebase

[jira] [Created] (SPARK-30125) Remove PostgreSQL dialect to reduce codebase maintenance cost

2019-12-04 Thread Yuanjian Li (Jira)
Yuanjian Li created SPARK-30125: --- Summary: Remove PostgreSQL dialect to reduce codebase maintenance cost Key: SPARK-30125 URL: https://issues.apache.org/jira/browse/SPARK-30125 Project: Spark

[jira] [Updated] (SPARK-30124) unnecessary persist in PythonMLLibAPI.scala

2019-12-04 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Omer updated SPARK-30124: -- Summary: unnecessary persist in PythonMLLibAPI.scala (was: Improper persist in PythonMLLibAPI.scala)

[jira] [Created] (SPARK-30124) Improper persist in PythonMLLibAPI.scala

2019-12-04 Thread Aman Omer (Jira)
Aman Omer created SPARK-30124: - Summary: Improper persist in PythonMLLibAPI.scala Key: SPARK-30124 URL: https://issues.apache.org/jira/browse/SPARK-30124 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-30123) PartitionPruning should consider more case

2019-12-04 Thread deshanxiao (Jira)
deshanxiao created SPARK-30123: -- Summary: PartitionPruning should consider more case Key: SPARK-30123 URL: https://issues.apache.org/jira/browse/SPARK-30123 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30092) Number of active tasks is negative in Live UI Executors page

2019-12-04 Thread shahid (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987799#comment-16987799 ] shahid commented on SPARK-30092: [~zhongyu09]Do you have any steps for reproducing the issue? > Number

[jira] [Updated] (SPARK-30041) Add Codegen Stage Id to Stage DAG visualization in Web UI

2019-12-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30041: Attachment: (was: Snippet_StagesDags_with_CodegenId.png) > Add Codegen Stage Id to Stage DAG

[jira] [Updated] (SPARK-30041) Add Codegen Stage Id to Stage DAG visualization in Web UI

2019-12-04 Thread Luca Canali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luca Canali updated SPARK-30041: Attachment: Snippet_StagesDags_with_CodegenId _annotated.png > Add Codegen Stage Id to Stage DAG

[jira] [Created] (SPARK-30122) Allow setting serviceAccountName for executor pods

2019-12-04 Thread Jira
Juho Mäkinen created SPARK-30122: Summary: Allow setting serviceAccountName for executor pods Key: SPARK-30122 URL: https://issues.apache.org/jira/browse/SPARK-30122 Project: Spark Issue

[jira] [Issue Comment Deleted] (SPARK-29591) Support data insertion in a different order if you wish or even omit some columns in spark sql also like postgresql

2019-12-04 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Omer updated SPARK-29591: -- Comment: was deleted (was: Thanks for confirming. I will start working on this.) > Support data

[jira] [Created] (SPARK-30121) Miss default memory setting for sbt usage

2019-12-04 Thread Kent Yao (Jira)
Kent Yao created SPARK-30121: Summary: Miss default memory setting for sbt usage Key: SPARK-30121 URL: https://issues.apache.org/jira/browse/SPARK-30121 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-29667) implicitly convert mismatched datatypes on right side of "IN" operator

2019-12-04 Thread Marco Gaido (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987712#comment-16987712 ] Marco Gaido commented on SPARK-29667: - I can agree more with you [~hyukjin.kwon]. I think that

[jira] [Resolved] (SPARK-29914) ML models append metadata in `transform`/`transformSchema`

2019-12-04 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-29914. -- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 26547

[jira] [Assigned] (SPARK-29914) ML models append metadata in `transform`/`transformSchema`

2019-12-04 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng reassigned SPARK-29914: Assignee: zhengruifeng > ML models append metadata in `transform`/`transformSchema` >

[jira] [Commented] (SPARK-18886) Delay scheduling should not delay some executors indefinitely if one task is scheduled before delay timeout

2019-12-04 Thread Nicholas Brett Marcott (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987633#comment-16987633 ] Nicholas Brett Marcott commented on SPARK-18886: Thanks for mentioning the PRs here. My