[jira] [Updated] (SPARK-28940) Subquery reuse across all subquery levels

[ https://issues.apache.org/jira/browse/SPARK-28940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Toth updated SPARK-28940: --- Summary: Subquery reuse across all subquery levels (was: Subquery reuse accross all subquery levels

[jira] [Commented] (SPARK-28912) MatchError exception in CheckpointWriteHandler

[ https://issues.apache.org/jira/browse/SPARK-28912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920647#comment-16920647 ] Hyukjin Kwon commented on SPARK-28912: -- ping [~avk1] > MatchError exception in Che

[jira] [Resolved] (SPARK-28941) Spark Sql Jobs

[ https://issues.apache.org/jira/browse/SPARK-28941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-28941. -- Resolution: Invalid Please ask questions to mailing lists. > Spark Sql Jobs > --

[jira] [Commented] (SPARK-28943) NoSuchMethodError: shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B

[ https://issues.apache.org/jira/browse/SPARK-28943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920644#comment-16920644 ] Hyukjin Kwon commented on SPARK-28943: -- Does this happen in regular Apache Spark to

[jira] [Updated] (SPARK-28941) Spark Sql Jobs

[ https://issues.apache.org/jira/browse/SPARK-28941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28941: - Target Version/s: (was: 2.4.3) > Spark Sql Jobs > -- > > Key: SPAR

[jira] [Comment Edited] (SPARK-28943) NoSuchMethodError: shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B

[ https://issues.apache.org/jira/browse/SPARK-28943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920644#comment-16920644 ] Hyukjin Kwon edited comment on SPARK-28943 at 9/2/19 6:27 AM:

[jira] [Resolved] (SPARK-27336) Incorrect DataSet.summary() result

[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27336. -- Resolution: Won't Fix > Incorrect DataSet.summary() result > -

[jira] [Commented] (SPARK-28934) Add `spark.sql.compatiblity.mode`

[ https://issues.apache.org/jira/browse/SPARK-28934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920611#comment-16920611 ] Wenchen Fan commented on SPARK-28934: - yea if we have pgsql mode, it's a good reason

[jira] [Created] (SPARK-28946) Add some more information about building SparkR on Windows

Hyukjin Kwon created SPARK-28946: Summary: Add some more information about building SparkR on Windows Key: SPARK-28946 URL: https://issues.apache.org/jira/browse/SPARK-28946 Project: Spark Is

[jira] [Updated] (SPARK-28945) Allow concurrent writes to different partitions with dynamic partition overwrite

[ https://issues.apache.org/jira/browse/SPARK-28945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] koert kuipers updated SPARK-28945: -- Summary: Allow concurrent writes to different partitions with dynamic partition overwrite (wa

[jira] [Commented] (SPARK-28945) Allow concurrent writes to unrelated partitions with dynamic partition overwrite

[ https://issues.apache.org/jira/browse/SPARK-28945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920602#comment-16920602 ] koert kuipers commented on SPARK-28945: --- See also: https://mail-archives.apache.or

[jira] [Created] (SPARK-28945) Allow concurrent writes to unrelated partitions with dynamic partition overwrite

koert kuipers created SPARK-28945: - Summary: Allow concurrent writes to unrelated partitions with dynamic partition overwrite Key: SPARK-28945 URL: https://issues.apache.org/jira/browse/SPARK-28945 Pr

[jira] [Comment Edited] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920601#comment-16920601 ] Qiang Wang edited comment on SPARK-28927 at 9/2/19 3:56 AM:

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920601#comment-16920601 ] Qiang Wang commented on SPARK-28927: I check the commit list since May 31, 2017 befo

[jira] [Updated] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qiang Wang updated SPARK-28927: --- Attachment: image-2019-09-02-11-55-33-596.png > ArrayIndexOutOfBoundsException and Not-stable AUC me

[jira] [Commented] (SPARK-28373) Document JDBC/ODBC Server page

[ https://issues.apache.org/jira/browse/SPARK-28373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920599#comment-16920599 ] zhengruifeng commented on SPARK-28373: -- [~smilegator] [~yumwang]  I am afraid I hav

[jira] [Updated] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-28612: - Fix Version/s: (was: 3.0.0) > DataSourceV2: Add new DataFrameWriter API for v2 > ---

[jira] [Reopened] (SPARK-28612) DataSourceV2: Add new DataFrameWriter API for v2

[ https://issues.apache.org/jira/browse/SPARK-28612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reopened SPARK-28612: -- Assignee: (was: Ryan Blue) Reverted at https://github.com/apache/spark/commit/bd3915e35

[jira] [Updated] (SPARK-28933) Reduce unnecessary shuffle in ALS when initializing factors

[ https://issues.apache.org/jira/browse/SPARK-28933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-28933: -- Priority: Minor (was: Major) > Reduce unnecessary shuffle in ALS when initializing factors >

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920596#comment-16920596 ] Qiang Wang commented on SPARK-28927: I only tested it on version 2.2.1 which is comp

[jira] [Commented] (SPARK-28906) `bin/spark-submit --version` shows incorrect info

[ https://issues.apache.org/jira/browse/SPARK-28906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920594#comment-16920594 ] Kazuaki Ishizaki commented on SPARK-28906: -- For the information on {{git}} coma

[jira] [Commented] (SPARK-28770) Flaky Tests: Test ReplayListenerSuite.End-to-end replay with compression failed

[ https://issues.apache.org/jira/browse/SPARK-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920587#comment-16920587 ] huangtianhua commented on SPARK-28770: -- [~wypoon], thank you for looking into this,

[jira] [Updated] (SPARK-28933) Reduce unnecessary shuffle in ALS when initializing factors

[ https://issues.apache.org/jira/browse/SPARK-28933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh updated SPARK-28933: Fix Version/s: 3.0.0 > Reduce unnecessary shuffle in ALS when initializing factors > -

[jira] [Commented] (SPARK-28933) Reduce unnecessary shuffle in ALS when initializing factors

[ https://issues.apache.org/jira/browse/SPARK-28933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920584#comment-16920584 ] Liang-Chi Hsieh commented on SPARK-28933: - This issue was resolved by [https://g

[jira] [Resolved] (SPARK-28933) Reduce unnecessary shuffle in ALS when initializing factors

[ https://issues.apache.org/jira/browse/SPARK-28933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang-Chi Hsieh resolved SPARK-28933. - Resolution: Resolved > Reduce unnecessary shuffle in ALS when initializing factors > ---

[jira] [Resolved] (SPARK-28923) Deduplicate the codes 'multipartIdentifier' and 'identifierSeq'

[ https://issues.apache.org/jira/browse/SPARK-28923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xianyin Xin resolved SPARK-28923. - Resolution: Invalid > Deduplicate the codes 'multipartIdentifier' and 'identifierSeq' >

[jira] [Created] (SPARK-28944) Expose peak memory of executor in metrics for parameter tuning

deshanxiao created SPARK-28944: -- Summary: Expose peak memory of executor in metrics for parameter tuning Key: SPARK-28944 URL: https://issues.apache.org/jira/browse/SPARK-28944 Project: Spark I

[jira] [Commented] (SPARK-28916) Generated SpecificSafeProjection.apply method grows beyond 64 KB when use SparkSQL

[ https://issues.apache.org/jira/browse/SPARK-28916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920577#comment-16920577 ] MOBIN commented on SPARK-28916: --- [~mgaido]thinks, spark.sql.subexpressionElimination.enabl

[jira] [Commented] (SPARK-27336) Incorrect DataSet.summary() result

[ https://issues.apache.org/jira/browse/SPARK-27336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920557#comment-16920557 ] daile commented on SPARK-27336: --- I will check this issue > Incorrect DataSet.summary() re

[jira] [Commented] (SPARK-28935) Document SQL metrics for Details for Query Plan

[ https://issues.apache.org/jira/browse/SPARK-28935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920550#comment-16920550 ] Liang-Chi Hsieh commented on SPARK-28935: - Thanks! [~smilegator] It should be h

[jira] [Resolved] (SPARK-28790) Document CACHE TABLE statement in SQL Reference.

[ https://issues.apache.org/jira/browse/SPARK-28790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-28790. - Fix Version/s: 3.0.0 Assignee: Huaxin Gao Resolution: Fixed > Document CACHE TABLE state

[jira] [Commented] (SPARK-28594) Allow event logs for running streaming apps to be rolled over.

[ https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920534#comment-16920534 ] Jungtaek Lim commented on SPARK-28594: -- Thanks [~felixcheung] for reviewing and vol

[jira] [Commented] (SPARK-28373) Document JDBC/ODBC Server page

[ https://issues.apache.org/jira/browse/SPARK-28373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920515#comment-16920515 ] Xiao Li commented on SPARK-28373: - [~podongfeng] This is the last one. Could you help fi

[jira] [Commented] (SPARK-28935) Document SQL metrics for Details for Query Plan

[ https://issues.apache.org/jira/browse/SPARK-28935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920507#comment-16920507 ] Xiao Li commented on SPARK-28935: - [https://docs.google.com/spreadsheets/d/11PV6SfkIQ8W_

[jira] [Created] (SPARK-28943) NoSuchMethodError: shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B

Michael Heuer created SPARK-28943: - Summary: NoSuchMethodError: shaded.parquet.org.apache.thrift.EncodingUtils.setBit(BIZ)B Key: SPARK-28943 URL: https://issues.apache.org/jira/browse/SPARK-28943 Pro

[jira] [Commented] (SPARK-28927) ArrayIndexOutOfBoundsException and Not-stable AUC metrics in ALS for datasets with 12 billion instances

[ https://issues.apache.org/jira/browse/SPARK-28927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920480#comment-16920480 ] Liang-Chi Hsieh commented on SPARK-28927: - Does this only happen on 2.2.1? How a

[jira] [Commented] (SPARK-27495) SPIP: Support Stage level resource configuration and scheduling

[ https://issues.apache.org/jira/browse/SPARK-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920479#comment-16920479 ] Felix Cheung commented on SPARK-27495: -- +1 on this.   I've reviewed this. A few q

[jira] [Commented] (SPARK-28902) Spark ML Pipeline with nested Pipelines fails to load when saved from Python

[ https://issues.apache.org/jira/browse/SPARK-28902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920478#comment-16920478 ] Junichi Koizumi commented on SPARK-28902: --- Could you tell a little bit more

[jira] [Commented] (SPARK-28594) Allow event logs for running streaming apps to be rolled over.

[ https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920476#comment-16920476 ] Felix Cheung commented on SPARK-28594: -- Reviewed. looks reasonable to me. I can hel

[jira] [Updated] (SPARK-28594) Allow event logs for running streaming apps to be rolled over.

[ https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Felix Cheung updated SPARK-28594: - Shepherd: Felix Cheung > Allow event logs for running streaming apps to be rolled over. > --

[jira] [Created] (SPARK-28942) Spark in local mode hostname display localhost in the Host Column of Task Summary Page

ABHISHEK KUMAR GUPTA created SPARK-28942: Summary: Spark in local mode hostname display localhost in the Host Column of Task Summary Page Key: SPARK-28942 URL: https://issues.apache.org/jira/browse/SPARK-2

[jira] [Commented] (SPARK-28942) Spark in local mode hostname display localhost in the Host Column of Task Summary Page

[ https://issues.apache.org/jira/browse/SPARK-28942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920427#comment-16920427 ] Shivu Sondur commented on SPARK-28942: -- i will work on this issue > Spark in local

[jira] [Created] (SPARK-28941) Spark Sql Jobs

Brahmendra created SPARK-28941: -- Summary: Spark Sql Jobs Key: SPARK-28941 URL: https://issues.apache.org/jira/browse/SPARK-28941 Project: Spark Issue Type: Improvement Components: SQL

[jira] [Resolved] (SPARK-28855) Remove outdated Experimental, Evolving annotations

[ https://issues.apache.org/jira/browse/SPARK-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-28855. --- Fix Version/s: 3.0.0 Resolution: Fixed Issue resolved by pull request 25558 [https://github.c

[jira] [Resolved] (SPARK-28925) Update Kubernetes-client to 4.4.2 to be compatible with Kubernetes 1.13 and 1.14

[ https://issues.apache.org/jira/browse/SPARK-28925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove resolved SPARK-28925. Resolution: Duplicate > Update Kubernetes-client to 4.4.2 to be compatible with Kubernetes 1.13 an

[jira] [Commented] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16920411#comment-16920411 ] Andy Grove commented on SPARK-28921: [~dongjoon] we are seeing it on both of the EKS

[jira] [Updated] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11.10)

[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove updated SPARK-28921: --- Summary: Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.12.10, 1.11

[jira] [Updated] (SPARK-28921) Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.11.10)

[ https://issues.apache.org/jira/browse/SPARK-28921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andy Grove updated SPARK-28921: --- Summary: Spark jobs failing on latest versions of Kubernetes (1.15.3, 1.14.6, 1,13.10, 1.11.10) (wa

[jira] [Created] (SPARK-28940) Subquery reuse accross all subquery levels

Peter Toth created SPARK-28940: -- Summary: Subquery reuse accross all subquery levels Key: SPARK-28940 URL: https://issues.apache.org/jira/browse/SPARK-28940 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-28939) SQL configuration are not always propagated

[ https://issues.apache.org/jira/browse/SPARK-28939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marco Gaido updated SPARK-28939: Description: The SQL configurations are propagated to executors in order to be effective. Unfortun

[jira] [Created] (SPARK-28939) SQL configuration are not always propagated

Marco Gaido created SPARK-28939: --- Summary: SQL configuration are not always propagated Key: SPARK-28939 URL: https://issues.apache.org/jira/browse/SPARK-28939 Project: Spark Issue Type: Bug