[jira] [Created] (SPARK-31951) Explicitly close iterator in KVStoreView

Jungtaek Lim created SPARK-31951: Summary: Explicitly close iterator in KVStoreView Key: SPARK-31951 URL: https://issues.apache.org/jira/browse/SPARK-31951 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-31946) Failed to register SIGPWR handler on MacOS

[ https://issues.apache.org/jira/browse/SPARK-31946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130059#comment-17130059 ] Jungtaek Lim edited comment on SPARK-31946 at 6/10/20, 4:31 AM: Oh I see

[jira] [Commented] (SPARK-31946) Failed to register SIGPWR handler on MacOS

[ https://issues.apache.org/jira/browse/SPARK-31946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130059#comment-17130059 ] Jungtaek Lim commented on SPARK-31946: -- Oh I see what you meant. Thanks for making it clear. I'm

[jira] [Commented] (SPARK-31946) Failed to register SIGPWR handler on MacOS

[ https://issues.apache.org/jira/browse/SPARK-31946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17130001#comment-17130001 ] Jungtaek Lim commented on SPARK-31946: -- It’s intended to take up non-posix compliant signal, AFAIK.

[jira] [Comment Edited] (SPARK-31946) Failed to register SIGPWR handler on MacOS

[ https://issues.apache.org/jira/browse/SPARK-31946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129955#comment-17129955 ] Jungtaek Lim edited comment on SPARK-31946 at 6/10/20, 2:12 AM: That

[jira] [Commented] (SPARK-31946) Failed to register SIGPWR handler on MacOS

[ https://issues.apache.org/jira/browse/SPARK-31946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129955#comment-17129955 ] Jungtaek Lim commented on SPARK-31946: -- That message is intended. It wasn't showing which feature

[jira] [Comment Edited] (SPARK-31931) When using GCS as checkpoint location for Structured Streaming aggregation pipeline, the Spark writing job is aborted

[ https://issues.apache.org/jira/browse/SPARK-31931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128707#comment-17128707 ] Jungtaek Lim edited comment on SPARK-31931 at 6/8/20, 11:12 PM: Well, I

[jira] [Commented] (SPARK-31931) When using GCS as checkpoint location for Structured Streaming aggregation pipeline, the Spark writing job is aborted

[ https://issues.apache.org/jira/browse/SPARK-31931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128707#comment-17128707 ] Jungtaek Lim commented on SPARK-31931: -- Well, I looked into the attached file, and it doesn't show

[jira] [Commented] (SPARK-31931) When using GCS as checkpoint location for Structured Streaming aggregation pipeline, the Spark writing job is aborted

[ https://issues.apache.org/jira/browse/SPARK-31931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128705#comment-17128705 ] Jungtaek Lim commented on SPARK-31931: -- Critical+ is reserved for committers. Lowering the

[jira] [Updated] (SPARK-31931) When using GCS as checkpoint location for Structured Streaming aggregation pipeline, the Spark writing job is aborted

[ https://issues.apache.org/jira/browse/SPARK-31931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31931: - Priority: Major (was: Blocker) > When using GCS as checkpoint location for Structured

[jira] [Updated] (SPARK-31928) Flaky test: StreamingDeduplicationSuite.test no-data flag

[ https://issues.apache.org/jira/browse/SPARK-31928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31928: - Description: Test failed:

[jira] [Commented] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127965#comment-17127965 ] Jungtaek Lim commented on SPARK-17604: -- The issue is even reported from user group, refer here:

[jira] [Issue Comment Deleted] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-17604: - Comment: was deleted (was: The issue is even reported from user group, refer here:

[jira] [Commented] (SPARK-28594) Allow event logs for running streaming apps to be rolled over

[ https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127788#comment-17127788 ] Jungtaek Lim commented on SPARK-28594: -- Unfortunately that is most probably the guaranteed way if

[jira] [Commented] (SPARK-28594) Allow event logs for running streaming apps to be rolled over

[ https://issues.apache.org/jira/browse/SPARK-28594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17127776#comment-17127776 ] Jungtaek Lim commented on SPARK-28594: -- Actually it has been an issue with almost all of Spark

[jira] [Commented] (SPARK-31812) Spark to support the auto cancelation of delegation token when an Application completes

[ https://issues.apache.org/jira/browse/SPARK-31812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120749#comment-17120749 ] Jungtaek Lim commented on SPARK-31812: -- [~kamrul] In general Spark project doesn't assign the issue

[jira] [Commented] (SPARK-31764) JsonProtocol doesn't write RDDInfo#isBarrier

[ https://issues.apache.org/jira/browse/SPARK-31764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118244#comment-17118244 ] Jungtaek Lim commented on SPARK-31764: -- Thanks for confirming. :) > JsonProtocol doesn't write

[jira] [Commented] (SPARK-31764) JsonProtocol doesn't write RDDInfo#isBarrier

[ https://issues.apache.org/jira/browse/SPARK-31764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118201#comment-17118201 ] Jungtaek Lim commented on SPARK-31764: -- For me this looks to be a bug - the description of PR

[jira] [Commented] (SPARK-31841) Dataset.repartition leverage adaptive execution

[ https://issues.apache.org/jira/browse/SPARK-31841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17118114#comment-17118114 ] Jungtaek Lim commented on SPARK-31841: -- It sounds like a question/feature request which is better

[jira] [Commented] (SPARK-26646) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

[ https://issues.apache.org/jira/browse/SPARK-26646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117400#comment-17117400 ] Jungtaek Lim commented on SPARK-26646: -- Still happening.

[jira] [Commented] (SPARK-29137) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_train_prediction

[ https://issues.apache.org/jira/browse/SPARK-29137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17117399#comment-17117399 ] Jungtaek Lim commented on SPARK-29137: -- Still valid on latest master.

[jira] [Created] (SPARK-31831) Flaky test: org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector)

Jungtaek Lim created SPARK-31831: Summary: Flaky test: org.apache.spark.sql.hive.thriftserver.HiveSessionImplSuite.(It is not a test it is a sbt.testing.SuiteSelector) Key: SPARK-31831 URL:

[jira] [Commented] (SPARK-23539) Add support for Kafka headers in Structured Streaming

[ https://issues.apache.org/jira/browse/SPARK-23539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116653#comment-17116653 ] Jungtaek Lim commented on SPARK-23539: -- You can ignore the affect version in most cases if the type

[jira] [Commented] (SPARK-31794) Incorrect distribution with repartitionByRange and repartition column expression

[ https://issues.apache.org/jira/browse/SPARK-31794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17116470#comment-17116470 ] Jungtaek Lim commented on SPARK-31794: --

[jira] [Updated] (SPARK-31793) Reduce the memory usage in file scan location metadata

[ https://issues.apache.org/jira/browse/SPARK-31793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31793: - Fix Version/s: 3.1.0 > Reduce the memory usage in file scan location metadata >

[jira] [Commented] (SPARK-31794) Incorrect distribution with repartitionByRange and repartition column expression

[ https://issues.apache.org/jira/browse/SPARK-31794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17115080#comment-17115080 ] Jungtaek Lim commented on SPARK-31794: -- Please read through the doc of these methods, which explain

[jira] [Commented] (SPARK-31792) Introduce the structured streaming UI in the Web UI page

[ https://issues.apache.org/jira/browse/SPARK-31792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113791#comment-17113791 ] Jungtaek Lim commented on SPARK-31792: -- Fix version is for tracking the version which contains the

[jira] [Updated] (SPARK-31792) Introduce the structured streaming UI in the Web UI page

[ https://issues.apache.org/jira/browse/SPARK-31792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31792: - Fix Version/s: (was: 3.0.0) > Introduce the structured streaming UI in the Web UI page >

[jira] [Commented] (SPARK-31789) SparkSubmitOperator could not get Exit Code after log stream interrupted by k8s old resource version execption

[ https://issues.apache.org/jira/browse/SPARK-31789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113708#comment-17113708 ] Jungtaek Lim commented on SPARK-31789: -- critical / blocker are tend to be reserved for committers.

[jira] [Updated] (SPARK-31789) SparkSubmitOperator could not get Exit Code after log stream interrupted by k8s old resource version execption

[ https://issues.apache.org/jira/browse/SPARK-31789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31789: - Priority: Major (was: Blocker) > SparkSubmitOperator could not get Exit Code after log stream

[jira] [Comment Edited] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113706#comment-17113706 ] Jungtaek Lim edited comment on SPARK-31761 at 5/22/20, 2:58 AM: Let's

[jira] [Commented] (SPARK-31761) Sql Div operator can result in incorrect output for int_min

[ https://issues.apache.org/jira/browse/SPARK-31761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17113706#comment-17113706 ] Jungtaek Lim commented on SPARK-31761: -- Let's make sure priority is marked properly - sounds like

[jira] [Commented] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17112816#comment-17112816 ] Jungtaek Lim commented on SPARK-31754: -- I can also take a look if the input and checkpoint are

[jira] [Comment Edited] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111696#comment-17111696 ] Jungtaek Lim edited comment on SPARK-31754 at 5/20/20, 2:50 AM: Looks

[jira] [Comment Edited] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111696#comment-17111696 ] Jungtaek Lim edited comment on SPARK-31754 at 5/20/20, 2:21 AM: Looks

[jira] [Commented] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17111696#comment-17111696 ] Jungtaek Lim commented on SPARK-31754: -- Looks like the row itself is null which shouldn't happen.

[jira] [Commented] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110873#comment-17110873 ] Jungtaek Lim commented on SPARK-31754: -- [~puviarasu] Given the error comes from "generated code",

[jira] [Updated] (SPARK-31754) Spark Structured Streaming: NullPointerException in Stream Stream join

[ https://issues.apache.org/jira/browse/SPARK-31754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31754: - Priority: Major (was: Blocker) > Spark Structured Streaming: NullPointerException in Stream

[jira] [Updated] (SPARK-31257) Unify create table syntax to fix ambiguous two different CREATE TABLE syntaxes

[ https://issues.apache.org/jira/browse/SPARK-31257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31257: - Summary: Unify create table syntax to fix ambiguous two different CREATE TABLE syntaxes (was:

[jira] [Updated] (SPARK-31257) Fix ambiguous two different CREATE TABLE syntaxes

[ https://issues.apache.org/jira/browse/SPARK-31257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31257: - Affects Version/s: (was: 3.0.0) 3.1.0 Description: There's

[jira] [Updated] (SPARK-31722) flaky streaming tests

[ https://issues.apache.org/jira/browse/SPARK-31722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31722: - Component/s: (was: Structured Streaming) DStreams > flaky streaming tests

[jira] [Updated] (SPARK-31707) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31707: - Description: According to the latest status of discussion in the dev@ mailing list, [[DISCUSS]

[jira] [Created] (SPARK-31707) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

Jungtaek Lim created SPARK-31707: Summary: Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax Key: SPARK-31707 URL: https://issues.apache.org/jira/browse/SPARK-31707

[jira] [Commented] (SPARK-29046) Possible NPE on SQLConf.get when SparkContext is stopping in another thread

[ https://issues.apache.org/jira/browse/SPARK-29046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17106104#comment-17106104 ] Jungtaek Lim commented on SPARK-29046: -- Sorry I don't know. Also worth noting that Spark 2.3

[jira] [Resolved] (SPARK-31698) NPE on big dataset plans

[ https://issues.apache.org/jira/browse/SPARK-31698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-31698. -- Resolution: Duplicate The weird error message and stack trace is matched with SPARK-29046

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17098723#comment-17098723 ] Jungtaek Lim commented on SPARK-26385: -- [~rajeevkumar] Yes please raise a separate JIRA issue.

[jira] [Resolved] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-31599. -- Resolution: Invalid > Reading from S3 (Structured Streaming Bucket) Fails after Compaction >

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096483#comment-17096483 ] Jungtaek Lim commented on SPARK-31599: -- You understand how file stream sink and file source works

[jira] [Resolved] (SPARK-30261) Should not change owner of hive table for some commands like 'alter' operation

[ https://issues.apache.org/jira/browse/SPARK-30261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30261. -- Target Version/s: 2.4.3, 2.3.0 (was: 2.3.0, 2.4.3) Resolution: Duplicate > Should

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094925#comment-17094925 ] Jungtaek Lim commented on SPARK-31599: -- Oh sorry I should guide to user@ mailing list, my bad.

[jira] [Commented] (SPARK-31599) Reading from S3 (Structured Streaming Bucket) Fails after Compaction

[ https://issues.apache.org/jira/browse/SPARK-31599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094915#comment-17094915 ] Jungtaek Lim commented on SPARK-31599: -- Please post a mail thread on dev@ mailing list. This looks

[jira] [Updated] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-17604: - Affects Version/s: 3.1.0 Labels: (was: bulk-closed) Priority:

[jira] [Reopened] (SPARK-17604) Support purging aged file entry for FileStreamSource metadata log

[ https://issues.apache.org/jira/browse/SPARK-17604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim reopened SPARK-17604: -- Reopening this, as end user reported this in user mailing list recently.

[jira] [Commented] (SPARK-31559) AM starts with initial fetched tokens in any attempt

[ https://issues.apache.org/jira/browse/SPARK-31559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092928#comment-17092928 ] Jungtaek Lim commented on SPARK-31559: -- PR submitted: https://github.com/apache/spark/pull/28336 >

[jira] [Commented] (SPARK-31554) Flaky test suite org.apache.spark.sql.hive.thriftserver.CliSuite

[ https://issues.apache.org/jira/browse/SPARK-31554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17092012#comment-17092012 ] Jungtaek Lim commented on SPARK-31554: -- There're two existing PRs addressing the test suite:

[jira] [Created] (SPARK-31559) AM starts with initial fetched tokens in any attempt

Jungtaek Lim created SPARK-31559: Summary: AM starts with initial fetched tokens in any attempt Key: SPARK-31559 URL: https://issues.apache.org/jira/browse/SPARK-31559 Project: Spark Issue

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091075#comment-17091075 ] Jungtaek Lim commented on SPARK-26385: -- The symptoms are mixed up - please clarify whether the

[jira] [Resolved] (SPARK-27891) Long running spark jobs fail because of HDFS delegation token expires

[ https://issues.apache.org/jira/browse/SPARK-27891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-27891. -- Resolution: Cannot Reproduce SPARK-23361 is in Spark 2.4.0 and the fix is not going to be

[jira] [Commented] (SPARK-31460) spark-sql-kafka source in spark 2.4.4 causes reading stream failure frequently

[ https://issues.apache.org/jira/browse/SPARK-31460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085346#comment-17085346 ] Jungtaek Lim commented on SPARK-31460: -- 1. Please check your app / submit phase doesn't override

[jira] [Commented] (SPARK-26646) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

[ https://issues.apache.org/jira/browse/SPARK-26646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083856#comment-17083856 ] Jungtaek Lim commented on SPARK-26646: -- Looks like still happening on master branch

[jira] [Commented] (SPARK-29222) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_parameter_convergence

[ https://issues.apache.org/jira/browse/SPARK-29222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083857#comment-17083857 ] Jungtaek Lim commented on SPARK-29222: -- Still happening on master (3.1.0-SNAPSHOT)

[jira] [Comment Edited] (SPARK-29137) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_train_prediction

[ https://issues.apache.org/jira/browse/SPARK-29137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083855#comment-17083855 ] Jungtaek Lim edited comment on SPARK-29137 at 4/15/20, 6:58 AM: Still

[jira] [Commented] (SPARK-29137) Flaky test: pyspark.mllib.tests.test_streaming_algorithms.StreamingLinearRegressionWithTests.test_train_prediction

[ https://issues.apache.org/jira/browse/SPARK-29137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083855#comment-17083855 ] Jungtaek Lim commented on SPARK-29137: -- Still valid on latest master (3.1.0-SNAPSHOT).

[jira] [Commented] (SPARK-26385) YARN - Spark Stateful Structured streaming HDFS_DELEGATION_TOKEN not found in cache

[ https://issues.apache.org/jira/browse/SPARK-26385?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083849#comment-17083849 ] Jungtaek Lim commented on SPARK-26385: -- Probably you may need to share the entire log messages

[jira] [Commented] (SPARK-31427) Spark Structure streaming read data twice per every micro-batch.

[ https://issues.apache.org/jira/browse/SPARK-31427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082036#comment-17082036 ] Jungtaek Lim commented on SPARK-31427: -- Could you please check whether using Spark 3.0 preview 2

[jira] [Comment Edited] (SPARK-31376) Non-global sort support for structured streaming

[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077821#comment-17077821 ] Jungtaek Lim edited comment on SPARK-31376 at 4/8/20, 4:46 AM: --- Btw it

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077821#comment-17077821 ] Jungtaek Lim commented on SPARK-31376: -- Btw it would be even better if you initiate the

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077819#comment-17077819 ] Jungtaek Lim commented on SPARK-31376: -- I'm saying that sort is simply unavailable operation for

[jira] [Commented] (SPARK-31376) Non-global sort support for structured streaming

[ https://issues.apache.org/jira/browse/SPARK-31376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17077723#comment-17077723 ] Jungtaek Lim commented on SPARK-31376: -- I'll reflect the question; why do you think not allowing

[jira] [Resolved] (SPARK-30436) CREATE EXTERNAL TABLE doesn't work without STORED AS

[ https://issues.apache.org/jira/browse/SPARK-30436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-30436. -- Resolution: Duplicate > CREATE EXTERNAL TABLE doesn't work without STORED AS >

[jira] [Commented] (SPARK-31312) Transforming Hive simple UDF (using JAR) expression may incur CNFE in later evaluation

[ https://issues.apache.org/jira/browse/SPARK-31312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17073264#comment-17073264 ] Jungtaek Lim commented on SPARK-31312: -- No, it wasn't triggered by SPARK-26560 and should be

[jira] [Created] (SPARK-31312) Transforming Hive simple UDF (using JAR) expression may incur CNFE in later evaluation

Jungtaek Lim created SPARK-31312: Summary: Transforming Hive simple UDF (using JAR) expression may incur CNFE in later evaluation Key: SPARK-31312 URL: https://issues.apache.org/jira/browse/SPARK-31312

[jira] [Created] (SPARK-31257) Fix ambiguous two different CREATE TABLE syntaxes

Jungtaek Lim created SPARK-31257: Summary: Fix ambiguous two different CREATE TABLE syntaxes Key: SPARK-31257 URL: https://issues.apache.org/jira/browse/SPARK-31257 Project: Spark Issue

[jira] [Comment Edited] (SPARK-31136) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058333#comment-17058333 ] Jungtaek Lim edited comment on SPARK-31136 at 3/18/20, 8:14 AM: This

[jira] [Commented] (SPARK-29301) Removing block is not reflected to the driver/executor's storage memory

[ https://issues.apache.org/jira/browse/SPARK-29301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17060004#comment-17060004 ] Jungtaek Lim commented on SPARK-29301: -- Thanks for reminding. Marked as duplicated. > Removing

[jira] [Resolved] (SPARK-27648) In Spark2.4 Structured Streaming：The executor storage memory increasing over time

[ https://issues.apache.org/jira/browse/SPARK-27648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-27648. -- Resolution: Duplicate > In Spark2.4 Structured Streaming：The executor storage memory

[jira] [Resolved] (SPARK-29301) Removing block is not reflected to the driver/executor's storage memory

[ https://issues.apache.org/jira/browse/SPARK-29301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim resolved SPARK-29301. -- Resolution: Duplicate > Removing block is not reflected to the driver/executor's storage

[jira] [Commented] (SPARK-31136) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17059914#comment-17059914 ] Jungtaek Lim commented on SPARK-31136: -- For 2, I just initiated the discussion thread on dev@ list.

[jira] [Commented] (SPARK-31143) Spark 2.4.4 count distinct query much slower than Spark 1.6.2 and Hive 1.2.1

[ https://issues.apache.org/jira/browse/SPARK-31143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058573#comment-17058573 ] Jungtaek Lim commented on SPARK-31143: -- [~shijiezhiai] Could you please leave the information about

[jira] [Comment Edited] (SPARK-31136) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058380#comment-17058380 ] Jungtaek Lim edited comment on SPARK-31136 at 3/13/20, 3:01 AM:

[jira] [Commented] (SPARK-31136) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058380#comment-17058380 ] Jungtaek Lim commented on SPARK-31136: --

[jira] [Comment Edited] (SPARK-31136) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058333#comment-17058333 ] Jungtaek Lim edited comment on SPARK-31136 at 3/13/20, 1:34 AM: This

[jira] [Commented] (SPARK-31136) Revert SPARK-30098 Use default datasource as provider for CREATE TABLE syntax

[ https://issues.apache.org/jira/browse/SPARK-31136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17058333#comment-17058333 ] Jungtaek Lim commented on SPARK-31136: -- This reminds me about my previous PR:

[jira] [Commented] (SPARK-25987) StackOverflowError when executing many operations on a table with many columns

[ https://issues.apache.org/jira/browse/SPARK-25987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057713#comment-17057713 ] Jungtaek Lim commented on SPARK-25987: -- The root cause is the way of how "flowAnalysis" in Janino

[jira] [Created] (SPARK-31115) Lots of columns and distinct aggregation functions triggers compile exception on Janino

Jungtaek Lim created SPARK-31115: Summary: Lots of columns and distinct aggregation functions triggers compile exception on Janino Key: SPARK-31115 URL: https://issues.apache.org/jira/browse/SPARK-31115

[jira] [Commented] (SPARK-31099) Create migration script for metastore_db

[ https://issues.apache.org/jira/browse/SPARK-31099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17056465#comment-17056465 ] Jungtaek Lim commented on SPARK-31099: -- [~dongjoon] Could you elaborate your comment "Apache Spark

[jira] [Created] (SPARK-31101) Upgrade Janino to 3.1.1

Jungtaek Lim created SPARK-31101: Summary: Upgrade Janino to 3.1.1 Key: SPARK-31101 URL: https://issues.apache.org/jira/browse/SPARK-31101 Project: Spark Issue Type: Dependency upgrade

[jira] [Updated] (SPARK-31011) Failed to register signal handler for PWR

[ https://issues.apache.org/jira/browse/SPARK-31011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-31011: - Affects Version/s: (was: 3.0.0) 3.1.0 > Failed to register signal

[jira] [Updated] (SPARK-30993) GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and its sql type has fixed length

[ https://issues.apache.org/jira/browse/SPARK-30993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30993: - Fix Version/s: 2.4.6 > GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and

[jira] [Commented] (SPARK-31011) Failed to register signal handler for PWR

[ https://issues.apache.org/jira/browse/SPARK-31011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049880#comment-17049880 ] Jungtaek Lim commented on SPARK-31011: -- According to the wikipedia, SIGPWR is NOT listed in the

[jira] [Comment Edited] (SPARK-31011) Failed to register signal handler for PWR

[ https://issues.apache.org/jira/browse/SPARK-31011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17049880#comment-17049880 ] Jungtaek Lim edited comment on SPARK-31011 at 3/3/20 4:11 AM: --

[jira] [Created] (SPARK-31014) InMemoryStore: CountingRemoveIfForEach misses to remove key from parentToChildrenMap

Jungtaek Lim created SPARK-31014: Summary: InMemoryStore: CountingRemoveIfForEach misses to remove key from parentToChildrenMap Key: SPARK-31014 URL: https://issues.apache.org/jira/browse/SPARK-31014

[jira] [Commented] (SPARK-30993) GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and its sql type has fixed length

[ https://issues.apache.org/jira/browse/SPARK-30993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048456#comment-17048456 ] Jungtaek Lim commented on SPARK-30993: -- Just confirmed the problem persists in branch-2.3 and

[jira] [Updated] (SPARK-30993) GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and its sql type has fixed length

[ https://issues.apache.org/jira/browse/SPARK-30993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30993: - Affects Version/s: 2.3.4 2.4.5 > GenerateUnsafeRowJoiner corrupts the

[jira] [Commented] (SPARK-30993) GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and its sql type has fixed length

[ https://issues.apache.org/jira/browse/SPARK-30993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048321#comment-17048321 ] Jungtaek Lim commented on SPARK-30993: -- During review phase I'll check which versions are affected

[jira] [Updated] (SPARK-30993) GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and its sql type has fixed length

[ https://issues.apache.org/jira/browse/SPARK-30993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-30993: - Summary: GenerateUnsafeRowJoiner corrupts the value if the datatype is UDF and its sql type has

[jira] [Commented] (SPARK-30993) GenerateUnsafeRowJoiner incorrectly modifies the value if the datatype is UDF and its sql type has fixed length

[ https://issues.apache.org/jira/browse/SPARK-30993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048318#comment-17048318 ] Jungtaek Lim commented on SPARK-30993: -- The reporter uses Spark 2.3.0, and validated it exists on

[jira] [Commented] (SPARK-30993) GenerateUnsafeRowJoiner incorrectly modifies the value if the datatype is UDF and its sql type has fixed length

[ https://issues.apache.org/jira/browse/SPARK-30993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17048317#comment-17048317 ] Jungtaek Lim commented on SPARK-30993: -- Will submit a PR soon. Btw, it looks to be correctness

[jira] [Created] (SPARK-30993) GenerateUnsafeRowJoiner incorrectly modifies the value if the datatype is UDF and its sql type has fixed length

Jungtaek Lim created SPARK-30993: Summary: GenerateUnsafeRowJoiner incorrectly modifies the value if the datatype is UDF and its sql type has fixed length Key: SPARK-30993 URL:

[jira] [Commented] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045011#comment-17045011 ] Jungtaek Lim commented on SPARK-24295: -- [~iqbal_khattra] [~alfredo-gimenez-bv] Hi, if you're open

<    7   8   9   10   11   12   13   14   15   16   >