[jira] [Created] (SPARK-30215) Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex

2019-12-11 Thread Hu Fuwang (Jira)
Hu Fuwang created SPARK-30215: - Summary: Remove PrunedInMemoryFileIndex and merge its functionality into InMemoryFileIndex Key: SPARK-30215 URL: https://issues.apache.org/jira/browse/SPARK-30215 Project:

[jira] [Updated] (SPARK-30214) Support COMMENT ON syntax

2019-12-11 Thread Kent Yao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kent Yao updated SPARK-30214: - Description: https://prestosql.io/docs/current/sql/comment.html https://www.postgresql.org/docs/12/sql-c

[jira] [Created] (SPARK-30216) Use python3 in Docker release image

2019-12-11 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-30216: --- Summary: Use python3 in Docker release image Key: SPARK-30216 URL: https://issues.apache.org/jira/browse/SPARK-30216 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-30104) global temp db name can be used as a table name under v2 catalog

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30104. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26741 [https://gith

[jira] [Assigned] (SPARK-30104) global temp db name can be used as a table name under v2 catalog

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30104: --- Assignee: Terry Kim > global temp db name can be used as a table name under v2 catalog > --

[jira] [Resolved] (SPARK-27506) Function `from_avro` doesn't allow deserialization of data using other compatible schemas

2019-12-11 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang resolved SPARK-27506. Assignee: Fokko Driesprong Resolution: Fixed The issue is resolved by https://github

[jira] [Updated] (SPARK-27506) Function `from_avro` doesn't allow deserialization of data using other compatible schemas

2019-12-11 Thread Fokko Driesprong (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Fokko Driesprong updated SPARK-27506: - Fix Version/s: 3.0.0 > Function `from_avro` doesn't allow deserialization of data using

[jira] [Created] (SPARK-30217) Improve partition pruning rule to enforce idempotence

2019-12-11 Thread Manu Zhang (Jira)
Manu Zhang created SPARK-30217: -- Summary: Improve partition pruning rule to enforce idempotence Key: SPARK-30217 URL: https://issues.apache.org/jira/browse/SPARK-30217 Project: Spark Issue Type:

[jira] [Created] (SPARK-30218) Columns used in inequality conditions for joins not resolved correctly in case of common lineage

2019-12-11 Thread Francesco Cavrini (Jira)
Francesco Cavrini created SPARK-30218: - Summary: Columns used in inequality conditions for joins not resolved correctly in case of common lineage Key: SPARK-30218 URL: https://issues.apache.org/jira/browse/SPA

[jira] [Commented] (SPARK-30218) Columns used in inequality conditions for joins not resolved correctly in case of common lineage

2019-12-11 Thread Francesco Cavrini (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993379#comment-16993379 ] Francesco Cavrini commented on SPARK-30218: --- Potentially related issue https:/

[jira] [Created] (SPARK-30219) Support Filter expression referencing the outer query

2019-12-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30219: -- Summary: Support Filter expression referencing the outer query Key: SPARK-30219 URL: https://issues.apache.org/jira/browse/SPARK-30219 Project: Spark Issue Type:

[jira] [Updated] (SPARK-30219) Support Filter expression referencing the outer query

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30219: --- Description: Spark SQL cannot supports a SQL with nested aggregate as below:   {code:java} select (

[jira] [Created] (SPARK-30220) Support Filter expression uses IN/EXISTS predicate sub-queries

2019-12-11 Thread jiaan.geng (Jira)
jiaan.geng created SPARK-30220: -- Summary: Support Filter expression uses IN/EXISTS predicate sub-queries Key: SPARK-30220 URL: https://issues.apache.org/jira/browse/SPARK-30220 Project: Spark I

[jira] [Updated] (SPARK-30220) Support Filter expression uses IN/EXISTS predicate sub-queries

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30220: --- Description: Spark SQL cannot supports a SQL with nested aggregate as below:   {code:java} select s

[jira] [Updated] (SPARK-30219) Support Filter expression reference the outer query

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-30219: --- Summary: Support Filter expression reference the outer query (was: Support Filter expression refere

[jira] [Commented] (SPARK-27217) Nested schema pruning doesn't work for aggregation e.g. `sum`.

2019-12-11 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993460#comment-16993460 ] Aman Omer commented on SPARK-27217: --- I am able to reproduce this bug on spark-shell. W

[jira] [Created] (SPARK-30221) Enhanced implementation of PrometheusPushGateWaySink

2019-12-11 Thread Forward Xu (Jira)
Forward Xu created SPARK-30221: -- Summary: Enhanced implementation of PrometheusPushGateWaySink Key: SPARK-30221 URL: https://issues.apache.org/jira/browse/SPARK-30221 Project: Spark Issue Type:

[jira] [Assigned] (SPARK-30207) Enhance the SQL NULL Semantics document

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-30207: --- Assignee: Yuanjian Li > Enhance the SQL NULL Semantics document > -

[jira] [Resolved] (SPARK-30207) Enhance the SQL NULL Semantics document

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-30207. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26837 [https://gith

[jira] [Created] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Shyam (Jira)
Shyam created SPARK-30222: - Summary: Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe Key: SPARK-30222 URL: https://issues.apache.org/jira/browse/SPARK-30222 Project

[jira] [Commented] (SPARK-25466) Documentation does not specify how to set Kafka consumer cache capacity for SS

2019-12-11 Thread Shyam (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993557#comment-16993557 ] Shyam commented on SPARK-25466: --- [~gsomogyi] I raised a ticket https://issues.apache.org/j

[jira] [Assigned] (SPARK-29460) Improve tooltip for Job Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen reassigned SPARK-29460: Assignee: pavithra ramachandran > Improve tooltip for Job Tab > -

[jira] [Updated] (SPARK-29460) Improve tooltip for Job Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29460: - Priority: Minor (was: Major) > Improve tooltip for Job Tab > --- > >

[jira] [Resolved] (SPARK-29460) Improve tooltip for Job Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29460. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26384 [https://gi

[jira] [Resolved] (SPARK-29457) Improve tooltip information for Environment Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29457. -- Resolution: Not A Problem The two columns are "key" and "value". I can't imagine this needs c

[jira] [Commented] (SPARK-29455) Improve tooltip information for Stages Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993643#comment-16993643 ] Sean R. Owen commented on SPARK-29455: -- [~pavithraramachandran] this is the last to

[jira] [Updated] (SPARK-29455) Improve tooltip information for Stages Tab

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-29455: - Priority: Minor (was: Major) > Improve tooltip information for Stages Tab > ---

[jira] [Commented] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993645#comment-16993645 ] Wenchen Fan commented on SPARK-30223: - cc [~yumwang] > queries in thrift server may

[jira] [Created] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread Wenchen Fan (Jira)
Wenchen Fan created SPARK-30223: --- Summary: queries in thrift server may read wrong SQL configs Key: SPARK-30223 URL: https://issues.apache.org/jira/browse/SPARK-30223 Project: Spark Issue Type:

[jira] [Created] (SPARK-30224) Configurable field and record separators for filters

2019-12-11 Thread thierry accart (Jira)
thierry accart created SPARK-30224: -- Summary: Configurable field and record separators for filters Key: SPARK-30224 URL: https://issues.apache.org/jira/browse/SPARK-30224 Project: Spark Issu

[jira] [Resolved] (SPARK-29864) Strict parsing of day-time strings to intervals

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29864. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26473 [https://gith

[jira] [Resolved] (SPARK-29920) Parsing failure on interval '20 15' day to hour

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29920. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26473 [https://gith

[jira] [Assigned] (SPARK-29920) Parsing failure on interval '20 15' day to hour

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29920: --- Assignee: Maxim Gekk > Parsing failure on interval '20 15' day to hour > --

[jira] [Resolved] (SPARK-29371) Support interval field values with fractional parts

2019-12-11 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen resolved SPARK-29371. -- Resolution: Won't Fix > Support interval field values with fractional parts >

[jira] [Commented] (SPARK-30127) UDF should work for case class like Dataset operations

2019-12-11 Thread Simeon Simeonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993750#comment-16993750 ] Simeon Simeonov commented on SPARK-30127: - The ability to transform one or more

[jira] [Comment Edited] (SPARK-30127) UDF should work for case class like Dataset operations

2019-12-11 Thread Simeon Simeonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993750#comment-16993750 ] Simeon Simeonov edited comment on SPARK-30127 at 12/11/19 5:42 PM: ---

[jira] [Commented] (SPARK-25250) Race condition with tasks running when new attempt for same stage is created leads to other task in the next attempt running on the same partition id retry multiple ti

2019-12-11 Thread Mairtin Deady (Jira)
[ https://issues.apache.org/jira/browse/SPARK-25250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993759#comment-16993759 ] Mairtin Deady commented on SPARK-25250: --- [~cloud_fan] Has another ticket been open

[jira] [Comment Edited] (SPARK-30127) UDF should work for case class like Dataset operations

2019-12-11 Thread Simeon Simeonov (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993750#comment-16993750 ] Simeon Simeonov edited comment on SPARK-30127 at 12/11/19 6:01 PM: ---

[jira] [Created] (SPARK-30225) "stream is corrup" exception on reading of shuffle disk-spilled data

2019-12-11 Thread Mala Chikka Kempanna (Jira)
Mala Chikka Kempanna created SPARK-30225: Summary: "stream is corrup" exception on reading of shuffle disk-spilled data Key: SPARK-30225 URL: https://issues.apache.org/jira/browse/SPARK-30225

[jira] [Updated] (SPARK-30225) "stream is corrup" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "stream is corrup" exception on reading disk-spilled data of a shuffle

[jira] [Updated] (SPARK-30225) "stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "stream is corrupt" exception on reading disk-spilled data of a shuffle

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "Stream is corrupt" exception on reading disk-spilled data of a shuffle

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Commented] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993827#comment-16993827 ] Mala Chikka Kempanna commented on SPARK-30225: -- Exception looks like below:

[jira] [Comment Edited] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993827#comment-16993827 ] Mala Chikka Kempanna edited comment on SPARK-30225 at 12/11/19 7:02 PM: --

[jira] [Issue Comment Deleted] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Comment: was deleted (was: Exception looks like below:    19/12/10 01:51:31

[jira] [Updated] (SPARK-30225) "Stream is corrupt" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Description: There is issues with spark.unsafe.sorter.spill.read.ahead.enabled i

[jira] [Commented] (SPARK-18105) LZ4 failed to decompress a stream of shuffled data

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-18105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993834#comment-16993834 ] Mala Chikka Kempanna commented on SPARK-18105: -- If you are facing this in s

[jira] [Updated] (SPARK-30225) "Stream is corrupted at" exception on reading disk-spilled data of a shuffle operation

2019-12-11 Thread Mala Chikka Kempanna (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mala Chikka Kempanna updated SPARK-30225: - Summary: "Stream is corrupted at" exception on reading disk-spilled data of a sh

[jira] [Resolved] (SPARK-30195) Some imports, function need more explicit resolution in 2.13

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30195. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26826 [https://

[jira] [Created] (SPARK-30226) Remove withXXX functions in WriteBuilder

2019-12-11 Thread Ximo Guanter (Jira)
Ximo Guanter created SPARK-30226: Summary: Remove withXXX functions in WriteBuilder Key: SPARK-30226 URL: https://issues.apache.org/jira/browse/SPARK-30226 Project: Spark Issue Type: Improvem

[jira] [Assigned] (SPARK-30038) DESCRIBE FUNCTION should look up catalog/table like v2 commands

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30038: - Assignee: Pablo Langa Blanco > DESCRIBE FUNCTION should look up catalog/table like v2 c

[jira] [Resolved] (SPARK-30038) DESCRIBE FUNCTION should look up catalog/table like v2 commands

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30038. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26840 [https://

[jira] [Resolved] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30198. --- Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Issue resolved by pul

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 2.3.4 2.4.4 > BytesToBytesMap does not grow internal

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 2.2.3 > BytesToBytesMap does not grow internal long array as expected > ---

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 2.0.2 2.1.3 > BytesToBytesMap does not grow internal

[jira] [Updated] (SPARK-30198) BytesToBytesMap does not grow internal long array as expected

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-30198: -- Affects Version/s: 1.6.3 > BytesToBytesMap does not grow internal long array as expected > ---

[jira] [Commented] (SPARK-22947) SPIP: as-of join in Spark SQL

2019-12-11 Thread Nasir Ali (Jira)
[ https://issues.apache.org/jira/browse/SPARK-22947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993981#comment-16993981 ] Nasir Ali commented on SPARK-22947: --- [~icexelloss] Is there any update on this issue?

[jira] [Created] (SPARK-30227) Add close() on DataWriter interface

2019-12-11 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-30227: Summary: Add close() on DataWriter interface Key: SPARK-30227 URL: https://issues.apache.org/jira/browse/SPARK-30227 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-30227) Add close() on DataWriter interface

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994022#comment-16994022 ] Jungtaek Lim commented on SPARK-30227: -- Working on it. I'll raise a PR soon. > Add

[jira] [Commented] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994061#comment-16994061 ] angerszhu commented on SPARK-30223: --- [~cloud_fan] Add ``` SparkSession.setActiveSessi

[jira] [Comment Edited] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994061#comment-16994061 ] angerszhu edited comment on SPARK-30223 at 12/12/19 1:49 AM: -

[jira] [Comment Edited] (SPARK-30223) queries in thrift server may read wrong SQL configs

2019-12-11 Thread angerszhu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994061#comment-16994061 ] angerszhu edited comment on SPARK-30223 at 12/12/19 1:49 AM: -

[jira] [Created] (SPARK-30228) Update zstd-jni to 1.4.4-3

2019-12-11 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-30228: - Summary: Update zstd-jni to 1.4.4-3 Key: SPARK-30228 URL: https://issues.apache.org/jira/browse/SPARK-30228 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30222. -- Fix Version/s: (was: 2.4.1) Target Version/s: (was: 2.4.1) Resolution: I

[jira] [Commented] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994085#comment-16994085 ] Jungtaek Lim commented on SPARK-30222: -- That config is renamed from "spark.kafka.co

[jira] [Resolved] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30199. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26827 [https://

[jira] [Commented] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994093#comment-16994093 ] Jungtaek Lim commented on SPARK-30222: -- Sorry I confused two configurations. "spark

[jira] [Comment Edited] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994085#comment-16994085 ] Jungtaek Lim edited comment on SPARK-30222 at 12/12/19 2:28 AM: --

[jira] [Comment Edited] (SPARK-30222) Still getting KafkaConsumer cache hitting max capacity of 64, removing consumer for CacheKe

2019-12-11 Thread Jungtaek Lim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994085#comment-16994085 ] Jungtaek Lim edited comment on SPARK-30222 at 12/12/19 2:28 AM: --

[jira] [Commented] (SPARK-30224) Configurable field and record separators for filters

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994100#comment-16994100 ] Hyukjin Kwon commented on SPARK-30224: -- Can we make it support quote and/or escapin

[jira] [Commented] (SPARK-30220) Support Filter expression uses IN/EXISTS predicate sub-queries

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994101#comment-16994101 ] Hyukjin Kwon commented on SPARK-30220: -- [~beliefer], check you check the comments o

[jira] [Commented] (SPARK-30219) Support Filter expression reference the outer query

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994102#comment-16994102 ] Hyukjin Kwon commented on SPARK-30219: -- [~beliefer] check the comments in its paren

[jira] [Commented] (SPARK-30208) A race condition when reading from Kafka in PySpark

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994105#comment-16994105 ] Hyukjin Kwon commented on SPARK-30208: -- Possibly it's a duplicate of SPARK-22340 >

[jira] [Commented] (SPARK-30182) Support nested aggregates

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994106#comment-16994106 ] Hyukjin Kwon commented on SPARK-30182: -- [~beliefer] please check the comments in it

[jira] [Commented] (SPARK-30173) Automatically close stale PRs

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994108#comment-16994108 ] Hyukjin Kwon commented on SPARK-30173: -- [~nchammas] please go ahead! > Automatical

[jira] [Assigned] (SPARK-30199) Recover spark.ui.port and spark.blockManager.port from checkpoint

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30199: - Assignee: Aaruna Godthi > Recover spark.ui.port and spark.blockManager.port from checkp

[jira] [Commented] (SPARK-30219) Support Filter expression reference the outer query

2019-12-11 Thread jiaan.geng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994124#comment-16994124 ] jiaan.geng commented on SPARK-30219: [~hyukjin.kwon] Thank! I see. > Support Filter

[jira] [Commented] (SPARK-30072) Create dedicated planner for subqueries

2019-12-11 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994136#comment-16994136 ] Wenchen Fan commented on SPARK-30072: - Can you be more specified about the problem?

[jira] [Assigned] (SPARK-30213) Remove the mutable status in QueryStage when enable AQE

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-30213: - Assignee: Ke Jia > Remove the mutable status in QueryStage when enable AQE > --

[jira] [Resolved] (SPARK-30213) Remove the mutable status in QueryStage when enable AQE

2019-12-11 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-30213. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26846 [https://

[jira] [Commented] (SPARK-19335) Spark should support doing an efficient DataFrame Upsert via JDBC

2019-12-11 Thread kevin yu (Jira)
[ https://issues.apache.org/jira/browse/SPARK-19335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994166#comment-16994166 ] kevin yu commented on SPARK-19335: -- [~danny-seismic] [~Vdarshankb] [~nstudenski] [~mray

[jira] [Updated] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30149: - Priority: Major (was: Blocker) > Schema Definition Spark Read > >

[jira] [Commented] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994168#comment-16994168 ] Hyukjin Kwon commented on SPARK-30149: -- Please don't set Critical+ which is usually

[jira] [Commented] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994170#comment-16994170 ] Hyukjin Kwon commented on SPARK-30149: -- Can you try out in Spark 3 preview? > Sche

[jira] [Commented] (SPARK-30139) get_json_object does not work correctly

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994171#comment-16994171 ] Hyukjin Kwon commented on SPARK-30139: -- [~rakson] Have you made some progresses on

[jira] [Resolved] (SPARK-30140) Code comment error

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30140. -- Resolution: Invalid > Code comment error > --- > > Key: SPARK

[jira] [Commented] (SPARK-30137) Support DELETE file

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994172#comment-16994172 ] Hyukjin Kwon commented on SPARK-30137: -- [~sandeep.katta2007] Have you made some pro

[jira] [Commented] (SPARK-30130) Hardcoded numeric values in common table expressions which utilize GROUP BY are interpreted as ordinal positions

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994174#comment-16994174 ] Hyukjin Kwon commented on SPARK-30130: -- [~mboegner], can you also clarify which DBM

[jira] [Commented] (SPARK-30123) PartitionPruning should consider more case

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994175#comment-16994175 ] Hyukjin Kwon commented on SPARK-30123: -- [~deshanxiao] can you show the plan generat

[jira] [Updated] (SPARK-30100) Decimal Precision Inferred from JDBC via Spark

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30100: - Component/s: SQL > Decimal Precision Inferred from JDBC via Spark >

[jira] [Updated] (SPARK-30096) drop function does not delete the jars file from tmp folder- session is not getting clear

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30096: - Description: Steps: 1. {{spark-master --master yarn}} 2. {{spark-sql> create function addDoubl

[jira] [Updated] (SPARK-30096) drop function does not delete the jars file from tmp folder- session is not getting clear

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-30096: - Description: Steps: 1. {{spark-master --master yarn}} 2. {{spark-sql> create function addDoubles

[jira] [Commented] (SPARK-30149) Schema Definition Spark Read

2019-12-11 Thread Joby Joje (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16994226#comment-16994226 ] Joby Joje commented on SPARK-30149: --- [~hyukjin.kwon] Sorry didn't know Critical was fo

[jira] [Resolved] (SPARK-30228) Update zstd-jni to 1.4.4-3

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-30228. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26856 [https://gi

[jira] [Assigned] (SPARK-30228) Update zstd-jni to 1.4.4-3

2019-12-11 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-30228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-30228: Assignee: Dongjoon Hyun > Update zstd-jni to 1.4.4-3 > -- > >

[jira] [Created] (SPARK-30229) java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783)

2019-12-11 Thread SeaAndHill (Jira)
SeaAndHill created SPARK-30229: -- Summary: java.lang.NullPointerException at org.apache.spark.SparkContext.getPreferredLocs(SparkContext.scala:1783) Key: SPARK-30229 URL: https://issues.apache.org/jira/browse/

  1   2   >