[jira] [Updated] (SPARK-30231) Support explain mode in PySpark df.explain

2019-12-11 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-30231: - Description: This pr intends to support explain modes implemented in SPARK-30200(#26829)

[jira] [Created] (SPARK-30231) Support explain mode in PySpark df.explain

2019-12-11 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-30231: Summary: Support explain mode in PySpark df.explain Key: SPARK-30231 URL: https://issues.apache.org/jira/browse/SPARK-30231 Project: Spark Issue Type

[jira] [Created] (SPARK-30230) Like ESCAPE syntax can not use '_' and '%'

2019-12-11 Thread ulysses you (Jira)
ulysses you created SPARK-30230: --- Summary: Like ESCAPE syntax can not use '_' and '%' Key: SPARK-30230 URL: https://issues.apache.org/jira/browse/SPARK-30230 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-30208) A race condition when reading from Kafka in PySpark

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994288#comment-16994288 ] Jungtaek Lim edited comment on SPARK-30208 at 12/12/19 6:41 AM: --

[jira] [Commented] (SPARK-30208) A race condition when reading from Kafka in PySpark

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994288#comment-16994288 ] Jungtaek Lim commented on SPARK-30208: -- Please correct me if I'm missing anything h

[jira] [Commented] (SPARK-30137) Support DELETE file

2019-12-11 Thread Sandeep Katta (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994263#comment-16994263 ] Sandeep Katta commented on SPARK-30137: --- yes, I working on this, will raise it by

[jira] [Commented] (SPARK-29640) [K8S] Intermittent "java.net.UnknownHostException: kubernetes.default.svc" in Spark driver

2019-12-11 Thread Andy Grove (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994257#comment-16994257 ] Andy Grove commented on SPARK-29640: We were finally able to get to a root cause on

[jira] [Resolved] (SPARK-29640) [K8S] Intermittent "java.net.UnknownHostException: kubernetes.default.svc" in Spark driver

2019-12-11 Thread Andy Grove (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove resolved SPARK-29640. Resolution: Not A Bug > [K8S] Intermittent "java.net.UnknownHostException: kubernetes.default.svc"

[jira] [Created] (SPARK-30229) java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783)

2019-12-11 Thread SeaAndHill (Jira)
SeaAndHill created SPARK-30229: -- Summary: java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783) Key: SPARK-30229 URL: https://issues.apache.org/jira/browse/

[jira] [Resolved] (SPARK-30228) Update zstd-jni to 1.4.4-3

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30228. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26856 [https://gi

[jira] [Assigned] (SPARK-30228) Update zstd-jni to 1.4.4-3

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30228: Assignee: Dongjoon Hyun > Update zstd-jni to 1.4.4-3 > -- > >

[jira] [Commented] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Joby Joje (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994226#comment-16994226 ] Joby Joje commented on SPARK-30149: --- [~hyukjin.kwon] Sorry didn't know Critical was fo

[jira] [Updated] (SPARK-30096) drop function does not delete the jars file from tmp folder- session is not getting clear

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30096: - Description: Steps: 1. {{spark-master --master yarn}} 2. {{spark-sql> create function addDoubl

[jira] [Updated] (SPARK-30096) drop function does not delete the jars file from tmp folder- session is not getting clear

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30096: - Description: Steps: 1. {{spark-master --master yarn}} 2. {{spark-sql> create function addDoubles

[jira] [Updated] (SPARK-30100) Decimal Precision Inferred from JDBC via Spark

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30100: - Component/s: SQL > Decimal Precision Inferred from JDBC via Spark >

[jira] [Commented] (SPARK-30123) PartitionPruning should consider more case

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994175#comment-16994175 ] Hyukjin Kwon commented on SPARK-30123: -- [~deshanxiao] can you show the plan generat

[jira] [Commented] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994174#comment-16994174 ] Hyukjin Kwon commented on SPARK-30130: -- [~mboegner], can you also clarify which DBM

[jira] [Commented] (SPARK-30137) Support DELETE file

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994172#comment-16994172 ] Hyukjin Kwon commented on SPARK-30137: -- [~sandeep.katta2007] Have you made some pro

[jira] [Commented] (SPARK-30139) get_json_object does not work correctly

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994171#comment-16994171 ] Hyukjin Kwon commented on SPARK-30139: -- [~rakson] Have you made some progresses on

[jira] [Resolved] (SPARK-30140) Code comment error

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30140. -- Resolution: Invalid > Code comment error > --- > > Key: SPARK

[jira] [Commented] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994170#comment-16994170 ] Hyukjin Kwon commented on SPARK-30149: -- Can you try out in Spark 3 preview? > Sche

[jira] [Commented] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994168#comment-16994168 ] Hyukjin Kwon commented on SPARK-30149: -- Please don't set Critical+ which is usually

[jira] [Updated] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30149: - Priority: Major (was: Blocker) > Schema Definition Spark Read > >

[jira] [Commented] (SPARK-19335) Spark should support doing an efficient DataFrame Upsert via JDBC

2019-12-11 Thread kevin yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994166#comment-16994166 ] kevin yu commented on SPARK-19335: -- [~danny-seismic] [~Vdarshankb] [~nstudenski] [~mray

[jira] [Assigned] (SPARK-30213) Remove the mutable status in QueryStage when enable AQE

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30213: - Assignee: Ke Jia > Remove the mutable status in QueryStage when enable AQE > --

[jira] [Resolved] (SPARK-30213) Remove the mutable status in QueryStage when enable AQE

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30213. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26846 [https://

[jira] [Commented] (SPARK-30072) Create dedicated planner for subqueries

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994136#comment-16994136 ] Wenchen Fan commented on SPARK-30072: - Can you be more specified about the problem?

[jira] [Commented] (SPARK-30219) Support Filter expression reference the outer query

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994124#comment-16994124 ] jiaan.geng commented on SPARK-30219: [~hyukjin.kwon] Thank! I see. > Support Filter

[jira] [Assigned] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30199: - Assignee: Aaruna Godthi > Recover spark.ui.port and spark.blockManager.port from checkp

[jira] [Commented] (SPARK-30173) Automatically close stale PRs

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994108#comment-16994108 ] Hyukjin Kwon commented on SPARK-30173: -- [~nchammas] please go ahead! > Automatical

[jira] [Commented] (SPARK-30182) Support nested aggregates

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994106#comment-16994106 ] Hyukjin Kwon commented on SPARK-30182: -- [~beliefer] please check the comments in it

[jira] [Commented] (SPARK-30208) A race condition when reading from Kafka in PySpark

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994105#comment-16994105 ] Hyukjin Kwon commented on SPARK-30208: -- Possibly it's a duplicate of SPARK-22340 >

[jira] [Commented] (SPARK-30219) Support Filter expression reference the outer query

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994102#comment-16994102 ] Hyukjin Kwon commented on SPARK-30219: -- [~beliefer] check the comments in its paren

[jira] [Commented] (SPARK-30220) Support Filter expression uses IN/EXISTS predicate sub-queries

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994101#comment-16994101 ] Hyukjin Kwon commented on SPARK-30220: -- [~beliefer], check you check the comments o

[jira] [Commented] (SPARK-30224) Configurable field and record separators for filters

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994100#comment-16994100 ] Hyukjin Kwon commented on SPARK-30224: -- Can we make it support quote and/or escapin

[jira] [Comment Edited] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994085#comment-16994085 ] Jungtaek Lim edited comment on SPARK-30222 at 12/12/19 2:28 AM: --

[jira] [Comment Edited] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994085#comment-16994085 ] Jungtaek Lim edited comment on SPARK-30222 at 12/12/19 2:28 AM: --

[jira] [Commented] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994093#comment-16994093 ] Jungtaek Lim commented on SPARK-30222: -- Sorry I confused two configurations. "spark

[jira] [Resolved] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30199. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26827 [https://

[jira] [Commented] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994085#comment-16994085 ] Jungtaek Lim commented on SPARK-30222: -- That config is renamed from "spark.kafka.co

[jira] [Resolved] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30222. -- Fix Version/s: (was: 2.4.1) Target Version/s: (was: 2.4.1) Resolution: I

[jira] [Created] (SPARK-30228) Update zstd-jni to 1.4.4-3

2019-12-11 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-30228: - Summary: Update zstd-jni to 1.4.4-3 Key: SPARK-30228 URL: https://issues.apache.org/jira/browse/SPARK-30228 Project: Spark Issue Type: Improvement

[jira] [Comment Edited] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994061#comment-16994061 ] angerszhu edited comment on SPARK-30223 at 12/12/19 1:49 AM: -

[jira] [Comment Edited] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994061#comment-16994061 ] angerszhu edited comment on SPARK-30223 at 12/12/19 1:49 AM: -

[jira] [Commented] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994061#comment-16994061 ] angerszhu commented on SPARK-30223: --- [~cloud_fan] Add ``` SparkSession.setActiveSessi

[jira] [Commented] (SPARK-30227) Add close() on DataWriter interface

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994022#comment-16994022 ] Jungtaek Lim commented on SPARK-30227: -- Working on it. I'll raise a PR soon. > Add

[jira] [Created] (SPARK-30227) Add close() on DataWriter interface

2019-12-11 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-30227: Summary: Add close() on DataWriter interface Key: SPARK-30227 URL: https://issues.apache.org/jira/browse/SPARK-30227 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2019-12-11 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993981#comment-16993981 ] Nasir Ali commented on SPARK-22947: --- [~icexelloss] Is there any update on this issue?

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 1.6.3 > BytesToBytesMap does not grow internal long array as expected > ---

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 2.0.2 2.1.3 > BytesToBytesMap does not grow internal

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 2.2.3 > BytesToBytesMap does not grow internal long array as expected > ---

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 2.3.4 2.4.4 > BytesToBytesMap does not grow internal

[jira] [Resolved] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30198. --- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by pul

[jira] [Assigned] (SPARK-30038) DESCRIBE FUNCTION should look up catalog/table like v2 commands

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30038: - Assignee: Pablo Langa Blanco > DESCRIBE FUNCTION should look up catalog/table like v2 c

[jira] [Resolved] (SPARK-30038) DESCRIBE FUNCTION should look up catalog/table like v2 commands

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30038. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26840 [https://

[jira] [Created] (SPARK-30226) Remove withXXX functions in WriteBuilder

2019-12-11 Thread Ximo Guanter (Jira)
Ximo Guanter created SPARK-30226: Summary: Remove withXXX functions in WriteBuilder Key: SPARK-30226 URL: https://issues.apache.org/jira/browse/SPARK-30226 Project: Spark Issue Type: Improvem

[jira] [Resolved] (SPARK-30195) Some imports, function need more explicit resolution in 2.13

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30195. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26826 [https://

[jira] [Updated] (SPARK-30225) "Stream is corrupted at" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "Stream is corrupted at" exception on reading disk-spilled data of a sh

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993834#comment-16993834 ] Mala Chikka Kempanna commented on SPARK-18105: -- If you are facing this in s

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Issue Comment Deleted] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Comment: was deleted (was: Exception looks like below:    19/12/10 01:51:31

[jira] [Comment Edited] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993827#comment-16993827 ] Mala Chikka Kempanna edited comment on SPARK-30225 at 12/11/19 7:02 PM: --

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Commented] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993827#comment-16993827 ] Mala Chikka Kempanna commented on SPARK-30225: -- Exception looks like below:

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "Stream is corrupt" exception on reading disk-spilled data of a shuffle

[jira] [Updated] (SPARK-30225) "stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "stream is corrupt" exception on reading disk-spilled data of a shuffle

[jira] [Updated] (SPARK-30225) "stream is corrup" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "stream is corrup" exception on reading disk-spilled data of a shuffle

[jira] [Created] (SPARK-30225) "stream is corrup" exception on reading of shuffle disk-spilled data

2019-12-11 Thread Mala Chikka Kempanna (Jira)
Mala Chikka Kempanna created SPARK-30225: Summary: "stream is corrup" exception on reading of shuffle disk-spilled data Key: SPARK-30225 URL: https://issues.apache.org/jira/browse/SPARK-30225

[jira] [Comment Edited] (SPARK-30127) UDF should work for case class like Dataset operations

2019-12-11 Thread Simeon Simeonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993750#comment-16993750 ] Simeon Simeonov edited comment on SPARK-30127 at 12/11/19 6:01 PM: ---

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

2019-12-11 Thread Mairtin Deady (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993759#comment-16993759 ] Mairtin Deady commented on SPARK-25250: --- [~cloud_fan] Has another ticket been open

[jira] [Comment Edited] (SPARK-30127) UDF should work for case class like Dataset operations

2019-12-11 Thread Simeon Simeonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993750#comment-16993750 ] Simeon Simeonov edited comment on SPARK-30127 at 12/11/19 5:42 PM: ---

[jira] [Commented] (SPARK-30127) UDF should work for case class like Dataset operations

2019-12-11 Thread Simeon Simeonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993750#comment-16993750 ] Simeon Simeonov commented on SPARK-30127: - The ability to transform one or more

[jira] [Resolved] (SPARK-29371) Support interval field values with fractional parts

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29371. -- Resolution: Won't Fix > Support interval field values with fractional parts >

[jira] [Resolved] (SPARK-29920) Parsing failure on interval '20 15' day to hour

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29920. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26473 [https://gith

[jira] [Assigned] (SPARK-29920) Parsing failure on interval '20 15' day to hour

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29920: --- Assignee: Maxim Gekk > Parsing failure on interval '20 15' day to hour > --

[jira] [Resolved] (SPARK-29864) Strict parsing of day-time strings to intervals

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29864. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26473 [https://gith

[jira] [Created] (SPARK-30224) Configurable field and record separators for filters

2019-12-11 Thread thierry accart (Jira)
thierry accart created SPARK-30224: -- Summary: Configurable field and record separators for filters Key: SPARK-30224 URL: https://issues.apache.org/jira/browse/SPARK-30224 Project: Spark Issu

[jira] [Created] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30223: --- Summary: queries in thrift server may read wrong SQL configs Key: SPARK-30223 URL: https://issues.apache.org/jira/browse/SPARK-30223 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993645#comment-16993645 ] Wenchen Fan commented on SPARK-30223: - cc [~yumwang] > queries in thrift server may

[jira] [Commented] (SPARK-29455) Improve tooltip information for Stages Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993643#comment-16993643 ] Sean R. Owen commented on SPARK-29455: -- [~pavithraramachandran] this is the last to

[jira] [Updated] (SPARK-29455) Improve tooltip information for Stages Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29455: - Priority: Minor (was: Major) > Improve tooltip information for Stages Tab > ---

[jira] [Resolved] (SPARK-29457) Improve tooltip information for Environment Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29457. -- Resolution: Not A Problem The two columns are "key" and "value". I can't imagine this needs c

[jira] [Resolved] (SPARK-29460) Improve tooltip for Job Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29460. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26384 [https://gi

[jira] [Updated] (SPARK-29460) Improve tooltip for Job Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29460: - Priority: Minor (was: Major) > Improve tooltip for Job Tab > --- > >

[jira] [Assigned] (SPARK-29460) Improve tooltip for Job Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29460: Assignee: pavithra ramachandran > Improve tooltip for Job Tab > -

[jira] [Commented] (SPARK-25466) Documentation does not specify how to set Kafka consumer cache capacity for SS

2019-12-11 Thread Shyam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993557#comment-16993557 ] Shyam commented on SPARK-25466: --- [~gsomogyi] I raised a ticket https://issues.apache.org/j

[jira] [Created] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Shyam (Jira)
Shyam created SPARK-30222: - Summary: Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe Key: SPARK-30222 URL: https://issues.apache.org/jira/browse/SPARK-30222 Project

[jira] [Resolved] (SPARK-30207) Enhance the SQL NULL Semantics document

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30207. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26837 [https://gith

[jira] [Assigned] (SPARK-30207) Enhance the SQL NULL Semantics document

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30207: --- Assignee: Yuanjian Li > Enhance the SQL NULL Semantics document > -

[jira] [Created] (SPARK-30221) Enhanced implementation of PrometheusPushGateWaySink

2019-12-11 Thread Forward Xu (Jira)
Forward Xu created SPARK-30221: -- Summary: Enhanced implementation of PrometheusPushGateWaySink Key: SPARK-30221 URL: https://issues.apache.org/jira/browse/SPARK-30221 Project: Spark Issue Type:

[jira] [Commented] (SPARK-27217) Nested schema pruning doesn't work for aggregation e.g. `sum`.

2019-12-11 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993460#comment-16993460 ] Aman Omer commented on SPARK-27217: --- I am able to reproduce this bug on spark-shell. W

[jira] [Updated] (SPARK-30219) Support Filter expression reference the outer query

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30219: --- Summary: Support Filter expression reference the outer query (was: Support Filter expression refere

[jira] [Updated] (SPARK-30220) Support Filter expression uses IN/EXISTS predicate sub-queries

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30220: --- Description: Spark SQL cannot supports a SQL with nested aggregate as below:   {code:java} select s

[jira] [Created] (SPARK-30220) Support Filter expression uses IN/EXISTS predicate sub-queries

2019-12-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30220: -- Summary: Support Filter expression uses IN/EXISTS predicate sub-queries Key: SPARK-30220 URL: https://issues.apache.org/jira/browse/SPARK-30220 Project: Spark I

[jira] [Updated] (SPARK-30219) Support Filter expression referencing the outer query

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30219: --- Description: Spark SQL cannot supports a SQL with nested aggregate as below:   {code:java} select (

[jira] [Created] (SPARK-30219) Support Filter expression referencing the outer query

2019-12-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30219: -- Summary: Support Filter expression referencing the outer query Key: SPARK-30219 URL: https://issues.apache.org/jira/browse/SPARK-30219 Project: Spark Issue Type:

[jira] [Commented] (SPARK-30218) Columns used in inequality conditions for joins not resolved correctly in case of common lineage

2019-12-11 Thread Francesco Cavrini (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993379#comment-16993379 ] Francesco Cavrini commented on SPARK-30218: --- Potentially related issue https:/

[jira] [Created] (SPARK-30218) Columns used in inequality conditions for joins not resolved correctly in case of common lineage

2019-12-11 Thread Francesco Cavrini (Jira)
Francesco Cavrini created SPARK-30218: - Summary: Columns used in inequality conditions for joins not resolved correctly in case of common lineage Key: SPARK-30218 URL: https://issues.apache.org/jira/browse/SPA

  1   2   >