[jira] [Created] (SPARK-29650) Discard a NULL constant in LIMIT

2019-10-29 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29650: Summary: Discard a NULL constant in LIMIT Key: SPARK-29650 URL: https://issues.apache.org/jira/browse/SPARK-29650 Project: Spark Issue Type: Sub-task

[jira] [Created] (SPARK-29649) Stop task set if FileAlreadyExistsException was thrown when writing to output file

2019-10-29 Thread L. C. Hsieh (Jira)
L. C. Hsieh created SPARK-29649: --- Summary: Stop task set if FileAlreadyExistsException was thrown when writing to output file Key: SPARK-29649 URL: https://issues.apache.org/jira/browse/SPARK-29649 Proj

[jira] [Created] (SPARK-29648) Add limit.sql

2019-10-29 Thread Takeshi Yamamuro (Jira)
Takeshi Yamamuro created SPARK-29648: Summary: Add limit.sql Key: SPARK-29648 URL: https://issues.apache.org/jira/browse/SPARK-29648 Project: Spark Issue Type: Sub-task Componen

[jira] [Commented] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Suchintak Patnaik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962741#comment-16962741 ] Suchintak Patnaik commented on SPARK-29621: --- [~hyukjin.kwon] count() returns t

[jira] [Assigned] (SPARK-29647) Use Python 3.7 in GitHub Action to recover lint-python

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29647: Assignee: Dongjoon Hyun > Use Python 3.7 in GitHub Action to recover lint-python > --

[jira] [Resolved] (SPARK-29647) Use Python 3.7 in GitHub Action to recover lint-python

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29647. -- Fix Version/s: 2.4.5 Resolution: Fixed Issue resolved by pull request 26308 [https://gi

[jira] [Resolved] (SPARK-29110) Add window.sql - Part 4

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29110. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26238 [https://gi

[jira] [Assigned] (SPARK-29110) Add window.sql - Part 4

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29110: Assignee: Dylan Guedes > Add window.sql - Part 4 > --- > >

[jira] [Commented] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962734#comment-16962734 ] Hyukjin Kwon commented on SPARK-29621: -- the problem here is {{count()}} since inter

[jira] [Commented] (SPARK-28632) Cannot read from a partitioned table by Int

2019-10-29 Thread Sunny Shukla (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962724#comment-16962724 ] Sunny Shukla commented on SPARK-28632: -- The same works for Spark 2.2.x (tested this

[jira] [Updated] (SPARK-29647) Use Python 3.7 in GitHub Action to recover lint-python

2019-10-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-29647: -- Component/s: PySpark > Use Python 3.7 in GitHub Action to recover lint-python > --

[jira] [Created] (SPARK-29647) Use Python 3.7 in GitHub Action to recover lint-python

2019-10-29 Thread Dongjoon Hyun (Jira)
Dongjoon Hyun created SPARK-29647: - Summary: Use Python 3.7 in GitHub Action to recover lint-python Key: SPARK-29647 URL: https://issues.apache.org/jira/browse/SPARK-29647 Project: Spark Issu

[jira] [Created] (SPARK-29646) Allow pyspark version name format `${versionNumber}-preview` in release script

2019-10-29 Thread Xingbo Jiang (Jira)
Xingbo Jiang created SPARK-29646: Summary: Allow pyspark version name format `${versionNumber}-preview` in release script Key: SPARK-29646 URL: https://issues.apache.org/jira/browse/SPARK-29646 Projec

[jira] [Created] (SPARK-29645) ML add param RelativeError

2019-10-29 Thread zhengruifeng (Jira)
zhengruifeng created SPARK-29645: Summary: ML add param RelativeError Key: SPARK-29645 URL: https://issues.apache.org/jira/browse/SPARK-29645 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-29372) Codegen grows beyond 64 KB for more columns in case of SupportsScanColumnarBatch

2019-10-29 Thread Razibul Hossain (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962663#comment-16962663 ] Razibul Hossain commented on SPARK-29372: - I kind of face same issue following s

[jira] [Comment Edited] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Suchintak Patnaik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962658#comment-16962658 ] Suchintak Patnaik edited comment on SPARK-29621 at 10/30/19 3:38 AM: -

[jira] [Commented] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Suchintak Patnaik (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962658#comment-16962658 ] Suchintak Patnaik commented on SPARK-29621: --- [~gurwls223] it should not allow

[jira] [Updated] (SPARK-29644) ShortType is wrongly set as Int in JDBCUtils.scala

2019-10-29 Thread Shiv Prashant Sood (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shiv Prashant Sood updated SPARK-29644: --- Description: @maropu pointed out this issue during  [PR 25344|https://github.com/ap

[jira] [Created] (SPARK-29644) ShortType is wrongly set as Int in JDBCUtils.scala

2019-10-29 Thread Shiv Prashant Sood (Jira)
Shiv Prashant Sood created SPARK-29644: -- Summary: ShortType is wrongly set as Int in JDBCUtils.scala Key: SPARK-29644 URL: https://issues.apache.org/jira/browse/SPARK-29644 Project: Spark

[jira] [Commented] (SPARK-27763) Port test cases from PostgreSQL to Spark SQL

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962588#comment-16962588 ] Hyukjin Kwon commented on SPARK-27763: -- +1 !! > Port test cases from PostgreSQL to

[jira] [Updated] (SPARK-29620) UnsafeKVExternalSorterSuite failure on bigendian system

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29620: - Description: {code} spark/sql/core# ../../build/mvn -Dtest=none -DwildcardSuites=org.apache.spa

[jira] [Commented] (SPARK-29643) ALTER TABLE (DROP PARTITION) should look up catalog/table like v2 commands

2019-10-29 Thread Huaxin Gao (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962581#comment-16962581 ] Huaxin Gao commented on SPARK-29643: I am working on this > ALTER TABLE (DROP PARTI

[jira] [Commented] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962580#comment-16962580 ] Hyukjin Kwon commented on SPARK-29621: -- Filter is fine. it doesn't have any problem

[jira] [Resolved] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29621. -- Resolution: Not A Problem > Querying internal corrupt record column should not be allowed in f

[jira] [Created] (SPARK-29643) ALTER TABLE (DROP PARTITION) should look up catalog/table like v2 commands

2019-10-29 Thread Huaxin Gao (Jira)
Huaxin Gao created SPARK-29643: -- Summary: ALTER TABLE (DROP PARTITION) should look up catalog/table like v2 commands Key: SPARK-29643 URL: https://issues.apache.org/jira/browse/SPARK-29643 Project: Spark

[jira] [Updated] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29621: - Description: As per *https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/ap

[jira] [Updated] (SPARK-29621) Querying internal corrupt record column should not be allowed in filter operation

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29621: - Description: As per *https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/ap

[jira] [Commented] (SPARK-29625) Spark Structure Streaming Kafka Wrong Reset Offset twice

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962576#comment-16962576 ] Hyukjin Kwon commented on SPARK-29625: -- [~sanysand...@gmail.com] Please don't just

[jira] [Updated] (SPARK-29625) Spark Structure Streaming Kafka Wrong Reset Offset twice

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-29625: - Description: Spark Structure Streaming Kafka Reset Offset twice, once with right offsets and se

[jira] [Resolved] (SPARK-29627) array_contains should allow column instances in PySpark

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29627. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26288 [https://gi

[jira] [Resolved] (SPARK-29629) Support typed integer literal expression

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-29629. -- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26291 [https://gi

[jira] [Assigned] (SPARK-29629) Support typed integer literal expression

2019-10-29 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-29629: Assignee: Kent Yao > Support typed integer literal expression > -

[jira] [Updated] (SPARK-29638) Spark handles 'NaN' as 0 in sums

2019-10-29 Thread Dylan Guedes (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dylan Guedes updated SPARK-29638: - Description: Currently, Spark handles 'NaN' as 0 in window functions, such that 3+'NaN'=3. PgSQ

[jira] [Issue Comment Deleted] (SPARK-29120) Add create_view.sql

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Omer updated SPARK-29120: -- Comment: was deleted (was: I will work on this.) > Add create_view.sql > --- > >

[jira] [Commented] (SPARK-29636) Can't parse '11:00 BST' or '2000-10-19 10:23:54+01' signatures to timestamp

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962542#comment-16962542 ] Aman Omer commented on SPARK-29636: --- Thanks [~DylanGuedes] . I am checking this one.

[jira] [Resolved] (SPARK-28746) Add repartitionby hint to support RepartitionByExpression

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-28746. -- Fix Version/s: 3.0.0 Assignee: ulysses you Resolution: Fixed Resolved

[jira] [Commented] (SPARK-29592) ALTER TABLE (set partition location) should look up catalog/table like v2 commands

2019-10-29 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962531#comment-16962531 ] Terry Kim commented on SPARK-29592: --- OK. Thanks. > ALTER TABLE (set partition locatio

[jira] [Commented] (SPARK-29592) ALTER TABLE (set partition location) should look up catalog/table like v2 commands

2019-10-29 Thread Ryan Blue (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962527#comment-16962527 ] Ryan Blue commented on SPARK-29592: --- There is not currently a way to alter the partiti

[jira] [Commented] (SPARK-29592) ALTER TABLE (set partition location) should look up catalog/table like v2 commands

2019-10-29 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962522#comment-16962522 ] Terry Kim commented on SPARK-29592: --- + [~rdblue] as well (for setting partition spec)

[jira] [Commented] (SPARK-27763) Port test cases from PostgreSQL to Spark SQL

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962515#comment-16962515 ] Takeshi Yamamuro commented on SPARK-27763: -- Thanks for the check, [~dongjoon] a

[jira] [Resolved] (SPARK-29639) Spark Kafka connector 0.10.0 generates incorrect end offsets for micro batch

2019-10-29 Thread Abhinav Choudhury (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Choudhury resolved SPARK-29639. --- Resolution: Abandoned Resolved because of lack of enough info and steps to reproduce

[jira] [Created] (SPARK-29642) ContinuousMemoryStream throws error on String type

2019-10-29 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-29642: Summary: ContinuousMemoryStream throws error on String type Key: SPARK-29642 URL: https://issues.apache.org/jira/browse/SPARK-29642 Project: Spark Issue Type

[jira] [Created] (SPARK-29641) Stage Level Sched: Add python api's and test

2019-10-29 Thread Thomas Graves (Jira)
Thomas Graves created SPARK-29641: - Summary: Stage Level Sched: Add python api's and test Key: SPARK-29641 URL: https://issues.apache.org/jira/browse/SPARK-29641 Project: Spark Issue Type: St

[jira] [Updated] (SPARK-29497) Cannot assign instance of java.lang.invoke.SerializedLambda to field

2019-10-29 Thread Rob Russo (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rob Russo updated SPARK-29497: -- Description: Note this is for scala 2.12: There seems to be an issue in spark with serializing a udf

[jira] [Commented] (SPARK-29561) Large Case Statement Code Generation OOM

2019-10-29 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962361#comment-16962361 ] Michael Chen commented on SPARK-29561: -- If I increase the memory, it will run into

[jira] [Comment Edited] (SPARK-29561) Large Case Statement Code Generation OOM

2019-10-29 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962361#comment-16962361 ] Michael Chen edited comment on SPARK-29561 at 10/29/19 7:59 PM: --

[jira] [Comment Edited] (SPARK-29561) Large Case Statement Code Generation OOM

2019-10-29 Thread Michael Chen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962361#comment-16962361 ] Michael Chen edited comment on SPARK-29561 at 10/29/19 7:59 PM: --

[jira] [Commented] (SPARK-29592) ALTER TABLE (set partition location) should look up catalog/table like v2 commands

2019-10-29 Thread Terry Kim (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29592?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962333#comment-16962333 ] Terry Kim commented on SPARK-29592: --- [~cloud_fan], is the spec for setting partitions

[jira] [Resolved] (SPARK-29637) SHS Endpoint /applications//jobs/ doesn't include description

2019-10-29 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin resolved SPARK-29637. Fix Version/s: 3.0.0 2.4.5 Resolution: Fixed Iss

[jira] [Assigned] (SPARK-29637) SHS Endpoint /applications//jobs/ doesn't include description

2019-10-29 Thread Marcelo Masiero Vanzin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Masiero Vanzin reassigned SPARK-29637: -- Assignee: Gabor Somogyi > SHS Endpoint /applications//jobs/ doesn't i

[jira] [Created] (SPARK-29640) [K8S] Make it possible to set DNS option to use TCP instead of UDP

2019-10-29 Thread Andy Grove (Jira)
Andy Grove created SPARK-29640: -- Summary: [K8S] Make it possible to set DNS option to use TCP instead of UDP Key: SPARK-29640 URL: https://issues.apache.org/jira/browse/SPARK-29640 Project: Spark

[jira] [Commented] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-10-29 Thread Steve Loughran (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962255#comment-16962255 ] Steve Loughran commented on SPARK-27676: 1. Well see if you can create a backpo

[jira] [Assigned] (SPARK-29607) Move static methods from CalendarInterval to IntervalUtils

2019-10-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-29607: --- Assignee: Maxim Gekk > Move static methods from CalendarInterval to IntervalUtils > ---

[jira] [Resolved] (SPARK-29607) Move static methods from CalendarInterval to IntervalUtils

2019-10-29 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-29607. - Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 26261 [https://gith

[jira] [Commented] (SPARK-27763) Port test cases from PostgreSQL to Spark SQL

2019-10-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962248#comment-16962248 ] Dongjoon Hyun commented on SPARK-27763: --- +1, [~maropu]! > Port test cases from Po

[jira] [Commented] (SPARK-28938) Move to supported OpenJDK docker image for Kubernetes

2019-10-29 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962243#comment-16962243 ] Dongjoon Hyun commented on SPARK-28938: --- Thank you, [~jkleckner]. It landed to `br

[jira] [Commented] (SPARK-29625) Spark Structure Streaming Kafka Wrong Reset Offset twice

2019-10-29 Thread Sandish Kumar HN (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962228#comment-16962228 ] Sandish Kumar HN commented on SPARK-29625: -- [~tdas] It would be very helpful if

[jira] [Updated] (SPARK-29639) Spark Kafka connector 0.10.0 generates incorrect end offsets for micro batch

2019-10-29 Thread Abhinav Choudhury (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Choudhury updated SPARK-29639: -- Description: We have been running a Spark structured job on production for more than a

[jira] [Updated] (SPARK-29639) Spark Kafka connector 0.10.0 generates incorrect end offsets for micro batch

2019-10-29 Thread Abhinav Choudhury (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Abhinav Choudhury updated SPARK-29639: -- Target Version/s: 2.4.4, 2.4.5 > Spark Kafka connector 0.10.0 generates incorrect end

[jira] [Created] (SPARK-29639) Spark Kafka connector 0.10.0 generates incorrect end offsets for micro batch

2019-10-29 Thread Abhinav Choudhury (Jira)
Abhinav Choudhury created SPARK-29639: - Summary: Spark Kafka connector 0.10.0 generates incorrect end offsets for micro batch Key: SPARK-29639 URL: https://issues.apache.org/jira/browse/SPARK-29639

[jira] [Commented] (SPARK-28938) Move to supported OpenJDK docker image for Kubernetes

2019-10-29 Thread Jim Kleckner (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962131#comment-16962131 ] Jim Kleckner commented on SPARK-28938: -- I created a minor one-line patch fix for th

[jira] [Commented] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962096#comment-16962096 ] Aman Omer commented on SPARK-29595: --- Thanks [~srowen] . Kindly check PR [https://gith

[jira] [Issue Comment Deleted] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aman Omer updated SPARK-29595: -- Comment: was deleted (was: Raised a PR [https://github.com/apache/spark/pull/26293] . Kindly review.

[jira] [Commented] (SPARK-28478) Optimizer rule to remove unnecessary explicit null checks for null-intolerant expressions (e.g. if(x is null, x, f(x)))

2019-10-29 Thread David Vrba (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962029#comment-16962029 ] David Vrba commented on SPARK-28478: If no one is working on this, i will take this

[jira] [Commented] (SPARK-27763) Port test cases from PostgreSQL to Spark SQL

2019-10-29 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16962021#comment-16962021 ] Yuming Wang commented on SPARK-27763: - +1 for porting: {noformat} alter_table.sql cr

[jira] [Updated] (SPARK-26985) Test "access only some column of the all of columns " fails on big endian

2019-10-29 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean R. Owen updated SPARK-26985: - Fix Version/s: 2.4.5 > Test "access only some column of the all of columns " fails on big endian

[jira] [Commented] (SPARK-29628) Forcibly create a temporary view in CREATE VIEW if referencing a temporary view

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961973#comment-16961973 ] Takeshi Yamamuro commented on SPARK-29628: -- Thanks! > Forcibly create a tempor

[jira] [Commented] (SPARK-27763) Port test cases from PostgreSQL to Spark SQL

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961970#comment-16961970 ] Takeshi Yamamuro commented on SPARK-27763: -- Hi, all, I've checked the other le

[jira] [Commented] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961964#comment-16961964 ] Sean R. Owen commented on SPARK-29595: -- As I recall that behavior is intended to ma

[jira] [Updated] (SPARK-29451) Some queries with divisions in SQL windows are failling in Thrift

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-29451: - Parent: SPARK-27764 Issue Type: Sub-task (was: Bug) > Some queries with divisio

[jira] [Updated] (SPARK-29451) Some queries with divisions in SQL windows are failling in Thrift

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-29451: - Parent: (was: SPARK-27763) Issue Type: Bug (was: Sub-task) > Some queries w

[jira] [Commented] (SPARK-27676) InMemoryFileIndex should hard-fail on missing files instead of logging and continuing

2019-10-29 Thread siddartha sagar chinne (Jira)
[ https://issues.apache.org/jira/browse/SPARK-27676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961899#comment-16961899 ] siddartha sagar chinne commented on SPARK-27676: hi [~ste...@apache.org]

[jira] [Commented] (SPARK-29637) SHS Endpoint /applications//jobs/ doesn't include description

2019-10-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961872#comment-16961872 ] Gabor Somogyi commented on SPARK-29637: --- I'm working on this. > SHS Endpoint /app

[jira] [Created] (SPARK-29638) Spark handles 'NaN' as 0 in sums

2019-10-29 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29638: Summary: Spark handles 'NaN' as 0 in sums Key: SPARK-29638 URL: https://issues.apache.org/jira/browse/SPARK-29638 Project: Spark Issue Type: Sub-task

[jira] [Updated] (SPARK-29637) SHS Endpoint /applications//jobs/ doesn't include description

2019-10-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-29637: -- Description: Starting from Spark 2.3, the SHS REST API endpoint /applications//jobs/  is not

[jira] [Updated] (SPARK-29637) SHS Endpoint /applications//jobs/ doesn't include description

2019-10-29 Thread Gabor Somogyi (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-29637: -- Issue Type: Bug (was: Improvement) > SHS Endpoint /applications//jobs/ doesn't include descri

[jira] [Created] (SPARK-29637) SHS Endpoint /applications//jobs/ doesn't include description

2019-10-29 Thread Gabor Somogyi (Jira)
Gabor Somogyi created SPARK-29637: - Summary: SHS Endpoint /applications//jobs/ doesn't include description Key: SPARK-29637 URL: https://issues.apache.org/jira/browse/SPARK-29637 Project: Spark

[jira] [Created] (SPARK-29636) Can't parse '11:00 BST' or '2000-10-19 10:23:54+01' signatures to timestamp

2019-10-29 Thread Dylan Guedes (Jira)
Dylan Guedes created SPARK-29636: Summary: Can't parse '11:00 BST' or '2000-10-19 10:23:54+01' signatures to timestamp Key: SPARK-29636 URL: https://issues.apache.org/jira/browse/SPARK-29636 Project:

[jira] [Commented] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961857#comment-16961857 ] Aman Omer commented on SPARK-29595: --- Raised a PR [https://github.com/apache/spark/pull

[jira] [Comment Edited] (SPARK-29634) spark-sql can't query hive table values with schema Char by equality.

2019-10-29 Thread Zhaoyang Qin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961828#comment-16961828 ] Zhaoyang Qin edited comment on SPARK-29634 at 10/29/19 9:42 AM: --

[jira] [Updated] (SPARK-29634) spark-sql can't query hive table values with schema Char by equality.

2019-10-29 Thread Zhaoyang Qin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaoyang Qin updated SPARK-29634: - Summary: spark-sql can't query hive table values with schema Char by equality. (was: spark-sql

[jira] [Updated] (SPARK-29634) spark-sql can't query hive table values that with schema Char by equality.

2019-10-29 Thread Zhaoyang Qin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhaoyang Qin updated SPARK-29634: - Description: spark-sql can't query hive table values that with schema Char by equality. When i

[jira] [Commented] (SPARK-29634) spark-sql can't query hive table values that with schema Char by equality.

2019-10-29 Thread Zhaoyang Qin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961830#comment-16961830 ] Zhaoyang Qin commented on SPARK-29634: -- Hive doc says: {panel:title=Char} Char type

[jira] [Commented] (SPARK-29634) spark-sql can't query hive table values that with schema Char by equality.

2019-10-29 Thread Zhaoyang Qin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961828#comment-16961828 ] Zhaoyang Qin commented on SPARK-29634: -- About Char datatype,Hive Docs sees: [https

[jira] [Created] (SPARK-29635) Deduplicate test suites between Kafka micro-batch sink and Kafka continuous sink

2019-10-29 Thread Jungtaek Lim (Jira)
Jungtaek Lim created SPARK-29635: Summary: Deduplicate test suites between Kafka micro-batch sink and Kafka continuous sink Key: SPARK-29635 URL: https://issues.apache.org/jira/browse/SPARK-29635 Proj

[jira] [Created] (SPARK-29634) spark-sql can't query hive table values that with schema Char by equality.

2019-10-29 Thread Zhaoyang Qin (Jira)
Zhaoyang Qin created SPARK-29634: Summary: spark-sql can't query hive table values that with schema Char by equality. Key: SPARK-29634 URL: https://issues.apache.org/jira/browse/SPARK-29634 Project: S

[jira] [Commented] (SPARK-29630) Not allowed to create a permanent view by referencing a temporary view in EXISTS

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961797#comment-16961797 ] Aman Omer commented on SPARK-29630: --- cc [~srowen] [~LI,Xiao] [~cloud_fan] > Not allow

[jira] [Comment Edited] (SPARK-29630) Not allowed to create a permanent view by referencing a temporary view in EXISTS

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961785#comment-16961785 ] Takeshi Yamamuro edited comment on SPARK-29630 at 10/29/19 8:19 AM: --

[jira] [Comment Edited] (SPARK-29630) Not allowed to create a permanent view by referencing a temporary view in EXISTS

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961785#comment-16961785 ] Takeshi Yamamuro edited comment on SPARK-29630 at 10/29/19 8:18 AM: --

[jira] [Comment Edited] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783 ] Aman Omer edited comment on SPARK-29595 at 10/29/19 8:18 AM: -

[jira] [Comment Edited] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783 ] Aman Omer edited comment on SPARK-29595 at 10/29/19 8:17 AM: -

[jira] [Updated] (SPARK-29630) Not allowed to create a permanent view by referencing a temporary view in EXISTS

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro updated SPARK-29630: - Parent: SPARK-27764 Issue Type: Sub-task (was: Improvement) > Not allowed to cr

[jira] [Comment Edited] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783 ] Aman Omer edited comment on SPARK-29595 at 10/29/19 8:13 AM: -

[jira] [Commented] (SPARK-29630) Not allowed to create a permanent view by referencing a temporary view in EXISTS

2019-10-29 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961785#comment-16961785 ] Takeshi Yamamuro commented on SPARK-29630: -- Ah, I see. I'll recheck later. > N

[jira] [Commented] (SPARK-29595) Insertion with named_struct should match by name

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961783#comment-16961783 ] Aman Omer commented on SPARK-29595: --- Parameters required for named_struct is _{color:#

[jira] [Commented] (SPARK-29626) notEqual() should return true when the one is null, the other is not null

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961770#comment-16961770 ] Aman Omer commented on SPARK-29626: --- Looking into this one. > notEqual() should retur

[jira] [Commented] (SPARK-29222) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_parameter_convergence

2019-10-29 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961766#comment-16961766 ] huangtianhua commented on SPARK-29222: -- [~hyukjin.kwon], the failure is releated wi

[jira] [Commented] (SPARK-29630) Not allowed to create a permanent view by referencing a temporary view in EXISTS

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961765#comment-16961765 ] Aman Omer commented on SPARK-29630: --- [~maropu] I think when we project some constant i

[jira] [Commented] (SPARK-29106) Add jenkins arm test for spark

2019-10-29 Thread huangtianhua (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961760#comment-16961760 ] huangtianhua commented on SPARK-29106: -- [~shaneknapp], hi, the arm job has built fo

[jira] [Commented] (SPARK-29628) Forcibly create a temporary view in CREATE VIEW if referencing a temporary view

2019-10-29 Thread Aman Omer (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16961758#comment-16961758 ] Aman Omer commented on SPARK-29628: --- Thanks [~maropu] . I will look into this one. >

  1   2   >