[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799545#comment-16799545 ] Jungtaek Lim commented on SPARK-27244: -- [~needna78] Sure, go ahead. That's awesome

[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799543#comment-16799543 ] Chakravarthi commented on SPARK-27244: -- [~needna78] ok > Conf Log Displays Passwor

[jira] [Created] (SPARK-27255) Aggregate functions should not be allowed in WHERE

2019-03-22 Thread Mingcong Han (JIRA)
Mingcong Han created SPARK-27255: Summary: Aggregate functions should not be allowed in WHERE Key: SPARK-27255 URL: https://issues.apache.org/jira/browse/SPARK-27255 Project: Spark Issue Type

[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Ninad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799482#comment-16799482 ] Ninad commented on SPARK-27244: --- [~kabhwan] [~chakravarthi] I already have the code chang

[jira] [Resolved] (SPARK-27243) RuleExecutor throws exception when dumping time spent with no rule executed

2019-03-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27243. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24180 [https://gi

[jira] [Assigned] (SPARK-27243) RuleExecutor throws exception when dumping time spent with no rule executed

2019-03-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27243: Assignee: Marco Gaido > RuleExecutor throws exception when dumping time spent with no rul

[jira] [Commented] (SPARK-27251) @volatile var cannot be defined in case class in Scala 2.11

2019-03-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799469#comment-16799469 ] Takeshi Yamamuro commented on SPARK-27251: -- @volatile var cannot be defined in

[jira] [Updated] (SPARK-27251) @volatile var cannot be defined in case class in Scala 2.11

2019-03-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-27251: - Summary: @volatile var cannot be defined in case class in Scala 2.11 (was: @volatile va

[jira] [Updated] (SPARK-27251) @volatile var cannot be defined in case class in Scala v2.11

2019-03-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-27251: - Summary: @volatile var cannot be defined in case class in Scala v2.11 (was: SPARK-25196

[jira] [Assigned] (SPARK-27184) Replace "spark.jars" & "spark.files" with the variables of JARS & FILES in config object

2019-03-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27184: Assignee: hehuiyuan > Replace "spark.jars" & "spark.files" with the variables of JARS & F

[jira] [Resolved] (SPARK-27184) Replace "spark.jars" & "spark.files" with the variables of JARS & FILES in config object

2019-03-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27184. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24123 [https://gi

[jira] [Updated] (SPARK-27251) SPARK-25196 Scala 2.11 maven build failure

2019-03-22 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-27251: - Summary: SPARK-25196 Scala 2.11 maven build failure (was: SPARK-25196 Scala 2.1.1 maven

[jira] [Commented] (SPARK-26047) Py4JNetworkError (on IPV6): An error occurred while trying to connect to the Java server (127.0.0.1

2019-03-22 Thread Abdul Mateen Mohammed (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799419#comment-16799419 ] Abdul Mateen Mohammed commented on SPARK-26047: --- Yes it was Py4J issue  

[jira] [Created] (SPARK-27254) Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted

2019-03-22 Thread Jungtaek Lim (JIRA)
Jungtaek Lim created SPARK-27254: Summary: Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted Key: SPARK-27254 URL: https://issues.apache.org/jira/browse/SPARK-27254

[jira] [Commented] (SPARK-27254) Cleanup complete but becoming invalid output files in ManifestFileCommitProtocol if job is aborted

2019-03-22 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799410#comment-16799410 ] Jungtaek Lim commented on SPARK-27254: -- Patch will be available soon. > Cleanup co

[jira] [Assigned] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

2019-03-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26606: -- Assignee: Jungtaek Lim > parameters passed in extraJavaOptions are not being picked u

[jira] [Resolved] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up

2019-03-22 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26606. Resolution: Fixed Fix Version/s: 2.3.4 2.4.2 3

[jira] [Resolved] (SPARK-27108) Add parsed CreateTable plans to Catalyst

2019-03-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27108. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24029 [https://gith

[jira] [Assigned] (SPARK-27108) Add parsed CreateTable plans to Catalyst

2019-03-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27108: --- Assignee: Ryan Blue > Add parsed CreateTable plans to Catalyst > --

[jira] [Created] (SPARK-27253) SparkSession clone discards SQLConf overrides in favor of SparkConf defaults

2019-03-22 Thread Jose Torres (JIRA)
Jose Torres created SPARK-27253: --- Summary: SparkSession clone discards SQLConf overrides in favor of SparkConf defaults Key: SPARK-27253 URL: https://issues.apache.org/jira/browse/SPARK-27253 Project: S

[jira] [Created] (SPARK-27252) Make current_date() independent from time zones

2019-03-22 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27252: -- Summary: Make current_date() independent from time zones Key: SPARK-27252 URL: https://issues.apache.org/jira/browse/SPARK-27252 Project: Spark Issue Type: Impro

[jira] [Updated] (SPARK-27250) Scala 2.11 maven compile should target Java 1.8

2019-03-22 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27250: --- Description: Discovered by https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboa

[jira] [Updated] (SPARK-27251) SPARK-25196 Scala 2.1.1 maven build failure

2019-03-22 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27251: --- Description: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-m

[jira] [Created] (SPARK-27251) SPARK-25196 Scala 2.1.1 maven build failure

2019-03-22 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27251: -- Summary: SPARK-25196 Scala 2.1.1 maven build failure Key: SPARK-27251 URL: https://issues.apache.org/jira/browse/SPARK-27251 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27250) Scala 2.11 maven compile should target Java 1.8

2019-03-22 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27250: --- Description: https://amplab.cs.berkeley.edu/jenkins/view/Spark%20QA%20Test%20(Dashboard)/job/spark-m

[jira] [Created] (SPARK-27250) Scala 2.11 maven compile should target Java 1.8

2019-03-22 Thread John Zhuge (JIRA)
John Zhuge created SPARK-27250: -- Summary: Scala 2.11 maven compile should target Java 1.8 Key: SPARK-27250 URL: https://issues.apache.org/jira/browse/SPARK-27250 Project: Spark Issue Type: Bug

[jira] [Resolved] (SPARK-27210) Cleanup incomplete output files in ManifestFileCommitProtocol if task is aborted

2019-03-22 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-27210. -- Resolution: Fixed Assignee: Jungtaek Lim Fix Version/s: 3.0.0 > Cleanup incomp

[jira] [Commented] (SPARK-27233) Schema of ArrayType change after saveAsTable and read

2019-03-22 Thread Sandeep Katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799240#comment-16799240 ] Sandeep Katta commented on SPARK-27233: --- ping [~hyukjin.kwon] [~cloud_fan]  [~dong

[jira] [Updated] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2019-03-22 Thread Everett Rush (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Everett Rush updated SPARK-27249: - Description: It would be nice to have a developers' API for dataset transformations that need m

[jira] [Updated] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2019-03-22 Thread Everett Rush (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Everett Rush updated SPARK-27249: - Description: It would be nice to have a developers' API for dataset transformations that need m

[jira] [Updated] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2019-03-22 Thread Everett Rush (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Everett Rush updated SPARK-27249: - Description: It would be nice to have a developers' API for dataset transformers that need more

[jira] [Updated] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2019-03-22 Thread Everett Rush (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Everett Rush updated SPARK-27249: - Description: It would be nice to have a developers' API for dataset transformers that need more

[jira] [Updated] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2019-03-22 Thread Everett Rush (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Everett Rush updated SPARK-27249: - Description: It would be nice to have a developers' API for dataset transformers that need more

[jira] [Created] (SPARK-27249) Developers API for Transformers beyond UnaryTransformer

2019-03-22 Thread Everett Rush (JIRA)
Everett Rush created SPARK-27249: Summary: Developers API for Transformers beyond UnaryTransformer Key: SPARK-27249 URL: https://issues.apache.org/jira/browse/SPARK-27249 Project: Spark Issue

[jira] [Assigned] (SPARK-27174) Support casting integer types to BinaryType

2019-03-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-27174: --- Assignee: Martin Junghanns > Support casting integer types to BinaryType >

[jira] [Resolved] (SPARK-27174) Support casting integer types to BinaryType

2019-03-22 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-27174. - Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24107 [https://gith

[jira] [Created] (SPARK-27248) REFRESH TABLE should recreate cache with same cache name and storage level

2019-03-22 Thread William Wong (JIRA)
William Wong created SPARK-27248: Summary: REFRESH TABLE should recreate cache with same cache name and storage level Key: SPARK-27248 URL: https://issues.apache.org/jira/browse/SPARK-27248 Project: S

[jira] [Commented] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Ohad Raviv (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799163#comment-16799163 ] Ohad Raviv commented on SPARK-18748: [~nimfadora] - thanks, we actually also ended u

[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799124#comment-16799124 ] Chakravarthi commented on SPARK-27244: -- [~kabhwan] working on this,will submit a PR

[jira] [Comment Edited] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799089#comment-16799089 ] Valeria Vasylieva edited comment on SPARK-18748 at 3/22/19 3:22 PM: --

[jira] [Comment Edited] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799089#comment-16799089 ] Valeria Vasylieva edited comment on SPARK-18748 at 3/22/19 3:15 PM: --

[jira] [Comment Edited] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799089#comment-16799089 ] Valeria Vasylieva edited comment on SPARK-18748 at 3/22/19 3:14 PM: --

[jira] [Comment Edited] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799089#comment-16799089 ] Valeria Vasylieva edited comment on SPARK-18748 at 3/22/19 3:14 PM: --

[jira] [Commented] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799089#comment-16799089 ] Valeria Vasylieva commented on SPARK-18748: --- [~uzadude] [~hqb1989] workaround,

[jira] [Issue Comment Deleted] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Valeria Vasylieva updated SPARK-18748: -- Comment: was deleted (was: [~uzadude] [~hqb1989] workaround, described in -SPARK-17728

[jira] [Commented] (SPARK-18748) UDF multiple evaluations causes very poor performance

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799077#comment-16799077 ] Valeria Vasylieva commented on SPARK-18748: --- [~uzadude] [~hqb1989] workaround,

[jira] [Comment Edited] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799068#comment-16799068 ] Emanuele Sabellico edited comment on SPARK-27218 at 3/22/19 2:45 PM: -

[jira] [Comment Edited] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799068#comment-16799068 ] Emanuele Sabellico edited comment on SPARK-27218 at 3/22/19 2:44 PM: -

[jira] [Commented] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799068#comment-16799068 ] Emanuele Sabellico commented on SPARK-27218: Yes, I've double checked: {code

[jira] [Updated] (SPARK-27241) Add map_keys and map_values support to SelectedField in nested schema pruning

2019-03-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-27241: Summary: Add map_keys and map_values support to SelectedField in nested schema pruning (w

[jira] [Updated] (SPARK-27241) Add map_keys and map_values support to SelectedField in nested schema pruning

2019-03-22 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-27241: Description: {{SelectedField}} in nested schema pruning doesn't support map_keys and map_v

[jira] [Commented] (SPARK-27052) Using PySpark udf in transform yields NULL values

2019-03-22 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799017#comment-16799017 ] Herman van Hovell commented on SPARK-27052: --- This is not supported at the mome

[jira] [Commented] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799018#comment-16799018 ] Gabor Somogyi commented on SPARK-27218: --- Just for the sake of double checking are

[jira] [Commented] (SPARK-24299) UI for Structured Streaming Monitoring

2019-03-22 Thread Valeria Vasylieva (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-24299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799013#comment-16799013 ] Valeria Vasylieva commented on SPARK-24299: --- [~TomaszGaweda] I want to try to

[jira] [Comment Edited] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799009#comment-16799009 ] Emanuele Sabellico edited comment on SPARK-27218 at 3/22/19 1:30 PM: -

[jira] [Comment Edited] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799009#comment-16799009 ] Emanuele Sabellico edited comment on SPARK-27218 at 3/22/19 1:28 PM: -

[jira] [Comment Edited] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799009#comment-16799009 ] Emanuele Sabellico edited comment on SPARK-27218 at 3/22/19 1:27 PM: -

[jira] [Commented] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Emanuele Sabellico (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16799009#comment-16799009 ] Emanuele Sabellico commented on SPARK-27218: Hi Gabor, I'm using structured

[jira] [Comment Edited] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798988#comment-16798988 ] Gabor Somogyi edited comment on SPARK-27218 at 3/22/19 1:22 PM: --

[jira] [Commented] (SPARK-27177) Update jenkins locale to en_US.UTF-8

2019-03-22 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798989#comment-16798989 ] Yuming Wang commented on SPARK-27177: - It works. Thank you [~shaneknapp] > Update j

[jira] [Comment Edited] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798988#comment-16798988 ] Gabor Somogyi edited comment on SPARK-27218 at 3/22/19 12:53 PM: -

[jira] [Commented] (SPARK-27218) spark-sql-kafka-0-10 startingOffset=earliest not working as expected with streaming

2019-03-22 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798988#comment-16798988 ] Gabor Somogyi commented on SPARK-27218: --- [~emasab] Don't fully understand your tes

[jira] [Updated] (SPARK-27247) [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly

2019-03-22 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27247: Description: Catalyst cache some logic plan. During relationConversions(Relation conversion from metastore

[jira] [Commented] (SPARK-27233) Schema of ArrayType change after saveAsTable and read

2019-03-22 Thread Kritsada Limpawatkul (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798972#comment-16798972 ] Kritsada Limpawatkul commented on SPARK-27233: -- I found another problem pro

[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798971#comment-16798971 ] Jungtaek Lim commented on SPARK-27244: -- [~chakravarthi] Are you working on it, or p

[jira] [Updated] (SPARK-27247) [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly

2019-03-22 Thread feiwang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] feiwang updated SPARK-27247: Description: Catalyst cache some logic plan. During relationConversions(Relation conversion from metastore

[jira] [Created] (SPARK-27247) [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly

2019-03-22 Thread feiwang (JIRA)
feiwang created SPARK-27247: --- Summary: [SQL]If the logicPlan get from cached is invalid, invalid it and rertry instead of assert error directly Key: SPARK-27247 URL: https://issues.apache.org/jira/browse/SPARK-27247

[jira] [Commented] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798949#comment-16798949 ] Chakravarthi commented on SPARK-27244: -- I will try to analyze this. > Conf Log Dis

[jira] [Created] (SPARK-27246) scalar subquery with no columns throws exception in spark.shell

2019-03-22 Thread Chakravarthi (JIRA)
Chakravarthi created SPARK-27246: Summary: scalar subquery with no columns throws exception in spark.shell Key: SPARK-27246 URL: https://issues.apache.org/jira/browse/SPARK-27246 Project: Spark

[jira] [Updated] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Ninad (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ninad updated SPARK-27244: -- Affects Version/s: 2.3.0 2.3.2 2.3.3 > Conf Log Displays Passwor

[jira] [Created] (SPARK-27245) Optimizer repeat Python UDF calls

2019-03-22 Thread Andrea Rota (JIRA)
Andrea Rota created SPARK-27245: --- Summary: Optimizer repeat Python UDF calls Key: SPARK-27245 URL: https://issues.apache.org/jira/browse/SPARK-27245 Project: Spark Issue Type: Bug Com

[jira] [Created] (SPARK-27244) Conf Log Displays Password In Clear Text

2019-03-22 Thread Ninad (JIRA)
Ninad created SPARK-27244: - Summary: Conf Log Displays Password In Clear Text Key: SPARK-27244 URL: https://issues.apache.org/jira/browse/SPARK-27244 Project: Spark Issue Type: Bug Componen

[jira] [Resolved] (SPARK-27141) Use ConfigEntry for hardcoded configs Yarn

2019-03-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27141. --- Resolution: Fixed Issue resolved by pull request 24103 [https://github.com/apache/spark/pull/24103]

[jira] [Assigned] (SPARK-27141) Use ConfigEntry for hardcoded configs Yarn

2019-03-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-27141: - Assignee: wangjiaochun > Use ConfigEntry for hardcoded configs Yarn > -

[jira] [Created] (SPARK-27243) RuleExecutor throws exception when dumping time spent with no rule executed

2019-03-22 Thread Marco Gaido (JIRA)
Marco Gaido created SPARK-27243: --- Summary: RuleExecutor throws exception when dumping time spent with no rule executed Key: SPARK-27243 URL: https://issues.apache.org/jira/browse/SPARK-27243 Project: Sp

[jira] [Resolved] (SPARK-27235) Remove the Dead code in HashedRelation.scala file from sql core module

2019-03-22 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-27235. --- Resolution: Invalid This isn't dead code. Option(SparkEnv.get) may be None > Remove the Dead code i

[jira] [Updated] (SPARK-27214) Upgrading locality level when lots of pending tasks have been waiting more than locality.wait

2019-03-22 Thread liupengcheng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-27214: - Description: Currently, Spark locality wait mechanism is not friendly for large job, when numbe

[jira] [Created] (SPARK-27242) Avoid using default time zone in formatting TIMESTAMP/DATE literals

2019-03-22 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27242: -- Summary: Avoid using default time zone in formatting TIMESTAMP/DATE literals Key: SPARK-27242 URL: https://issues.apache.org/jira/browse/SPARK-27242 Project: Spark

[jira] [Commented] (SPARK-27233) Schema of ArrayType change after saveAsTable and read

2019-03-22 Thread Sandeep Katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798852#comment-16798852 ] Sandeep Katta commented on SPARK-27233: --- {{asNullable}} should return it as nullab

[jira] [Commented] (SPARK-27228) Spark long delay on close, possible problem with killing executors

2019-03-22 Thread Lukas Waldmann (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16798841#comment-16798841 ] Lukas Waldmann commented on SPARK-27228: Executors management seems to behave st

[jira] [Created] (SPARK-27241) Add map keys support to SelectedField in nested schema pruning

2019-03-22 Thread Liang-Chi Hsieh (JIRA)
Liang-Chi Hsieh created SPARK-27241: --- Summary: Add map keys support to SelectedField in nested schema pruning Key: SPARK-27241 URL: https://issues.apache.org/jira/browse/SPARK-27241 Project: Spark

[jira] [Reopened] (SPARK-27228) Spark long delay on close, possible problem with killing executors

2019-03-22 Thread Lukas Waldmann (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Waldmann reopened SPARK-27228: Added initial issue investigation > Spark long delay on close, possible problem with killing

[jira] [Updated] (SPARK-27228) Spark long delay on close, possible problem with killing executors

2019-03-22 Thread Lukas Waldmann (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Waldmann updated SPARK-27228: --- Description: When using dynamic allocations after all jobs finishes spark delays for several

[jira] [Assigned] (SPARK-27212) Eliminate TimeZone to ZoneId conversion in stringToTimestamp

2019-03-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27212: Assignee: Maxim Gekk > Eliminate TimeZone to ZoneId conversion in stringToTimestamp > ---

[jira] [Updated] (SPARK-27228) Spark long delay on close, possible problem with killing executors

2019-03-22 Thread Lukas Waldmann (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lukas Waldmann updated SPARK-27228: --- Description: When using dynamic allocations after all jobs finishes spark delays for several

[jira] [Resolved] (SPARK-27212) Eliminate TimeZone to ZoneId conversion in stringToTimestamp

2019-03-22 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27212. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24155 [https://gi

[jira] [Closed] (SPARK-26500) Add conf to support ignore hdfs data locality

2019-03-22 Thread EdisonWang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] EdisonWang closed SPARK-26500. -- > Add conf to support ignore hdfs data locality > - > >

[jira] [Updated] (SPARK-27239) Processing Compressed HDFS files with spark failing with error: "java.lang.IllegalArgumentException: requirement failed: length (-1) cannot be negative" from spark 2.2.X

2019-03-22 Thread Rajesh Kumar K (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27239?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajesh Kumar K updated SPARK-27239: --- Description:   >From spark 2.2.x versions, when spark job processing any compressed HDFS fi

[jira] [Created] (SPARK-27240) Use pandas DataFrame for struct type argument in Scalar Pandas UDF.

2019-03-22 Thread Takuya Ueshin (JIRA)
Takuya Ueshin created SPARK-27240: - Summary: Use pandas DataFrame for struct type argument in Scalar Pandas UDF. Key: SPARK-27240 URL: https://issues.apache.org/jira/browse/SPARK-27240 Project: Spark

[jira] [Updated] (SPARK-22601) Data load is getting displayed successful on providing non existing hdfs file path

2019-03-22 Thread Adrian Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-22601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Wang updated SPARK-22601: Fix Version/s: (was: 2.2.1) 2.2.2 > Data load is getting displayed successf

[jira] [Updated] (SPARK-27238) In the same APP, maybe some hive Parquet(ORC) tables can't use the built-in Parquet(ORC) reader and writer

2019-03-22 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-27238: Description: In the same APP, TableA and TableB are both hive Parquet tables, but TableA can't use the bu

[jira] [Updated] (SPARK-27238) In the same APP, maybe some hive Parquet(ORC) tables can't use the built-in Parquet(ORC) reader and writer

2019-03-22 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-27238: Summary: In the same APP, maybe some hive Parquet(ORC) tables can't use the built-in Parquet(ORC) reader a