[jira] [Updated] (SPARK-20449) Upgrade breeze version to 0.13.1

[ https://issues.apache.org/jira/browse/SPARK-20449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-20449: Fix Version/s: (was: 2.3.0) > Upgrade breeze version to 0.13.1 > > >

[jira] [Updated] (SPARK-19378) StateOperator metrics should still return the total number of rows in state even if there was no data for a trigger

[ https://issues.apache.org/jira/browse/SPARK-19378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-19378: Fix Version/s: (was: 3.0.0) 2.2.0 > StateOperator metrics should still return the t

[jira] [Updated] (SPARK-21587) Filter pushdown for EventTime Watermark Operator

[ https://issues.apache.org/jira/browse/SPARK-21587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21587: Fix Version/s: (was: 3.0.0) 2.3.0 > Filter pushdown for EventTime Watermark Operato

[jira] [Updated] (SPARK-21765) Ensure all leaf nodes that are derived from streaming sources have isStreaming=true

[ https://issues.apache.org/jira/browse/SPARK-21765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21765: Fix Version/s: (was: 3.0.0) 2.3.0 > Ensure all leaf nodes that are derived from str

[jira] [Updated] (SPARK-21696) State Store can't handle corrupted snapshots

[ https://issues.apache.org/jira/browse/SPARK-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21696: Fix Version/s: (was: 3.0.0) 2.3.0 > State Store can't handle corrupted snapshots >

[jira] [Updated] (SPARK-21925) Update trigger interval documentation in docs with behavior change in Spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21925: Fix Version/s: (was: 2.3.2) 2.3.0 > Update trigger interval documentation in docs w

[jira] [Updated] (SPARK-20449) Upgrade breeze version to 0.13.1

[ https://issues.apache.org/jira/browse/SPARK-20449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-20449: Fix Version/s: (was: 3.0.0) 2.3.0 > Upgrade breeze version to 0.13.1 >

[jira] [Updated] (SPARK-21464) Minimize deprecation warnings caused by ProcessingTime class

[ https://issues.apache.org/jira/browse/SPARK-21464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21464: Fix Version/s: (was: 3.0.0) 2.3.0 > Minimize deprecation warnings caused by Process

[jira] [Updated] (SPARK-22884) ML test for StructuredStreaming: spark.ml.clustering

[ https://issues.apache.org/jira/browse/SPARK-22884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22884: Fix Version/s: (was: 3.0.0) 2.4.0 > ML test for StructuredStreaming: spark.ml.clust

[jira] [Updated] (SPARK-22018) Catalyst Optimizer does not preserve top-level metadata while collapsing projects

[ https://issues.apache.org/jira/browse/SPARK-22018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22018: Fix Version/s: (was: 3.0.0) 2.4.0 > Catalyst Optimizer does not preserve top-level

[jira] [Updated] (SPARK-22017) watermark evaluation with multi-input stream operators is unspecified

[ https://issues.apache.org/jira/browse/SPARK-22017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22017: Fix Version/s: (was: 3.0.0) 2.4.0 > watermark evaluation with multi-input stream op

[jira] [Updated] (SPARK-22839) Refactor Kubernetes code for configuring driver/executor pods to use consistent and cleaner abstraction

[ https://issues.apache.org/jira/browse/SPARK-22839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22839: Fix Version/s: (was: 3.0.0) 2.4.0 > Refactor Kubernetes code for configuring driver

[jira] [Updated] (SPARK-22908) add basic continuous kafka source

[ https://issues.apache.org/jira/browse/SPARK-22908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22908: Fix Version/s: (was: 3.0.0) > add basic continuous kafka source > - >

[jira] [Updated] (SPARK-22238) EnsureStatefulOpPartitioning shouldn't ask for the child RDD before planning is completed

[ https://issues.apache.org/jira/browse/SPARK-22238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-22238: Fix Version/s: (was: 3.0.0) 2.4.0 > EnsureStatefulOpPartitioning shouldn't ask for

[jira] [Updated] (SPARK-21925) Update trigger interval documentation in docs with behavior change in Spark 2.2

[ https://issues.apache.org/jira/browse/SPARK-21925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-21925: Fix Version/s: (was: 3.0.0) 2.3.2 > Update trigger interval documentation in docs w

[jira] [Updated] (SPARK-23004) Structured Streaming raise "llegalStateException: Cannot remove after already committed or aborted"

[ https://issues.apache.org/jira/browse/SPARK-23004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23004: Fix Version/s: (was: 3.0.0) 2.4.0 > Structured Streaming raise "llegalStateExceptio

[jira] [Updated] (SPARK-23143) Add Python support for continuous trigger

[ https://issues.apache.org/jira/browse/SPARK-23143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23143: Fix Version/s: (was: 3.0.0) > Add Python support for continuous trigger >

[jira] [Updated] (SPARK-23099) Migrate foreach sink

[ https://issues.apache.org/jira/browse/SPARK-23099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23099: Fix Version/s: (was: 3.0.0) > Migrate foreach sink > > > Key: SPA

[jira] [Updated] (SPARK-23096) Migrate rate source to v2

[ https://issues.apache.org/jira/browse/SPARK-23096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23096: Fix Version/s: (was: 3.0.0) 2.4.0 > Migrate rate source to v2 > ---

[jira] [Updated] (SPARK-23033) disable task-level retry for continuous execution

[ https://issues.apache.org/jira/browse/SPARK-23033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23033: Fix Version/s: (was: 3.0.0) > disable task-level retry for continuous execution >

[jira] [Updated] (SPARK-23092) Migrate MemoryStream to DataSource V2

[ https://issues.apache.org/jira/browse/SPARK-23092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23092: Fix Version/s: (was: 3.0.0) 2.4.0 > Migrate MemoryStream to DataSource V2 > ---

[jira] [Updated] (SPARK-23142) Add documentation for Continuous Processing

[ https://issues.apache.org/jira/browse/SPARK-23142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23142: Fix Version/s: (was: 3.0.0) > Add documentation for Continuous Processing > --

[jira] [Updated] (SPARK-23052) Migrate Microbatch ConsoleSink to v2

[ https://issues.apache.org/jira/browse/SPARK-23052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23052: Fix Version/s: (was: 3.0.0) > Migrate Microbatch ConsoleSink to v2 > -

[jira] [Updated] (SPARK-23097) Migrate text socket source to v2

[ https://issues.apache.org/jira/browse/SPARK-23097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23097: Fix Version/s: (was: 3.0.0) 2.4.0 > Migrate text socket source to v2 >

[jira] [Updated] (SPARK-23454) Add Trigger information to the Structured Streaming programming guide

[ https://issues.apache.org/jira/browse/SPARK-23454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23454: Fix Version/s: (was: 3.0.0) 2.4.0 > Add Trigger information to the Structured Strea

[jira] [Updated] (SPARK-23362) Migrate Kafka microbatch source to v2

[ https://issues.apache.org/jira/browse/SPARK-23362?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23362: Fix Version/s: (was: 3.0.0) 2.4.0 > Migrate Kafka microbatch source to v2 > ---

[jira] [Updated] (SPARK-23144) Add console sink for continuous queries

[ https://issues.apache.org/jira/browse/SPARK-23144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23144: Fix Version/s: (was: 3.0.0) > Add console sink for continuous queries > --

[jira] [Updated] (SPARK-23484) Fix possible race condition in KafkaContinuousReader

[ https://issues.apache.org/jira/browse/SPARK-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23484: Fix Version/s: (was: 2.4.0) > Fix possible race condition in KafkaContinuousReader > -

[jira] [Updated] (SPARK-23454) Add Trigger information to the Structured Streaming programming guide

[ https://issues.apache.org/jira/browse/SPARK-23454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23454: Fix Version/s: (was: 2.4.0) > Add Trigger information to the Structured Streaming programming guide >

[jira] [Updated] (SPARK-23408) Flaky test: StreamingOuterJoinSuite.left outer early state exclusion on right

[ https://issues.apache.org/jira/browse/SPARK-23408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23408: Fix Version/s: (was: 3.0.0) > Flaky test: StreamingOuterJoinSuite.left outer early state exclusion on

[jira] [Updated] (SPARK-23559) add epoch ID to data writer factory

[ https://issues.apache.org/jira/browse/SPARK-23559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23559: Fix Version/s: (was: 3.0.0) 2.4.0 > add epoch ID to data writer factory > -

[jira] [Updated] (SPARK-23491) continuous symptom

[ https://issues.apache.org/jira/browse/SPARK-23491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23491: Fix Version/s: (was: 3.0.0) 2.4.0 > continuous symptom > -- > >

[jira] [Updated] (SPARK-23748) Support select from temp tables

[ https://issues.apache.org/jira/browse/SPARK-23748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23748: Fix Version/s: (was: 3.0.0) 2.4.0 > Support select from temp tables > -

[jira] [Updated] (SPARK-23827) StreamingJoinExec should ensure that input data is partitioned into specific number of partitions

[ https://issues.apache.org/jira/browse/SPARK-23827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23827: Fix Version/s: (was: 3.0.0) > StreamingJoinExec should ensure that input data is partitioned into spec

[jira] [Updated] (SPARK-23747) Add EpochCoordinator unit tests

[ https://issues.apache.org/jira/browse/SPARK-23747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23747: Fix Version/s: (was: 3.0.0) 2.4.0 > Add EpochCoordinator unit tests > -

[jira] [Updated] (SPARK-23484) Fix possible race condition in KafkaContinuousReader

[ https://issues.apache.org/jira/browse/SPARK-23484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23484: Fix Version/s: (was: 3.0.0) 2.4.0 > Fix possible race condition in KafkaContinuousR

[jira] [Updated] (SPARK-23503) continuous execution should sequence committed epochs

[ https://issues.apache.org/jira/browse/SPARK-23503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23503: Fix Version/s: (was: 3.0.0) 2.4.0 > continuous execution should sequence committed

[jira] [Updated] (SPARK-24094) Change description strings of v2 streaming sources to reflect the change

[ https://issues.apache.org/jira/browse/SPARK-24094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24094: Fix Version/s: (was: 3.0.0) 2.4.0 > Change description strings of v2 streaming sour

[jira] [Updated] (SPARK-24157) Enable no-data micro batches for streaming aggregation and deduplication

[ https://issues.apache.org/jira/browse/SPARK-24157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24157: Fix Version/s: (was: 3.0.0) 2.4.0 > Enable no-data micro batches for streaming aggr

[jira] [Updated] (SPARK-24039) remove restarting iterators hack

[ https://issues.apache.org/jira/browse/SPARK-24039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24039: Fix Version/s: (was: 3.0.0) 2.4.0 > remove restarting iterators hack >

[jira] [Updated] (SPARK-24155) Instrumentation improvement for clustering

[ https://issues.apache.org/jira/browse/SPARK-24155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24155: Fix Version/s: (was: 3.0.0) 2.4.0 > Instrumentation improvement for clustering > --

[jira] [Updated] (SPARK-24115) improve instrumentation for spark.ml.tuning

[ https://issues.apache.org/jira/browse/SPARK-24115?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24115: Fix Version/s: (was: 3.0.0) 2.4.0 > improve instrumentation for spark.ml.tuning > -

[jira] [Updated] (SPARK-23966) Refactoring all checkpoint file writing logic in a common interface

[ https://issues.apache.org/jira/browse/SPARK-23966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-23966: Fix Version/s: (was: 3.0.0) 2.4.0 > Refactoring all checkpoint file writing logic i

[jira] [Updated] (SPARK-24050) StreamingQuery does not calculate input / processing rates in some cases

[ https://issues.apache.org/jira/browse/SPARK-24050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24050: Fix Version/s: (was: 3.0.0) 2.4.0 > StreamingQuery does not calculate input / proce

[jira] [Updated] (SPARK-24038) refactor continuous write exec to its own class

[ https://issues.apache.org/jira/browse/SPARK-24038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24038: Fix Version/s: (was: 3.0.0) 2.4.0 > refactor continuous write exec to its own class

[jira] [Updated] (SPARK-24056) Make consumer creation lazy in Kafka source for Structured streaming

[ https://issues.apache.org/jira/browse/SPARK-24056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24056: Fix Version/s: (was: 3.0.0) 2.4.0 > Make consumer creation lazy in Kafka source for

[jira] [Updated] (SPARK-24137) [K8s] Mount temporary directories in emptydir volumes

[ https://issues.apache.org/jira/browse/SPARK-24137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24137: Fix Version/s: (was: 3.0.0) 2.4.0 > [K8s] Mount temporary directories in emptydir v

[jira] [Updated] (SPARK-24132) Instrumentation improvement for classification

[ https://issues.apache.org/jira/browse/SPARK-24132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24132: Fix Version/s: (was: 3.0.0) 2.4.0 > Instrumentation improvement for classification

[jira] [Updated] (SPARK-24396) Add Structured Streaming ForeachWriter for python

[ https://issues.apache.org/jira/browse/SPARK-24396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24396: Fix Version/s: (was: 3.0.0) 2.4.0 > Add Structured Streaming ForeachWriter for pyth

[jira] [Updated] (SPARK-24158) Enable no-data micro batches for streaming joins

[ https://issues.apache.org/jira/browse/SPARK-24158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24158: Fix Version/s: (was: 3.0.0) 2.4.0 > Enable no-data micro batches for streaming join

[jira] [Updated] (SPARK-24453) Fix error recovering from the failure in a no-data batch

[ https://issues.apache.org/jira/browse/SPARK-24453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24453: Fix Version/s: (was: 3.0.0) 2.4.0 > Fix error recovering from the failure in a no-d

[jira] [Updated] (SPARK-24231) Python API: Provide evaluateEachIteration method or equivalent for spark.ml GBTs

[ https://issues.apache.org/jira/browse/SPARK-24231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24231: Fix Version/s: (was: 3.0.0) 2.4.0 > Python API: Provide evaluateEachIteration metho

[jira] [Updated] (SPARK-24386) implement continuous processing coalesce(1)

[ https://issues.apache.org/jira/browse/SPARK-24386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24386: Fix Version/s: (was: 3.0.0) 2.4.0 > implement continuous processing coalesce(1) > -

[jira] [Updated] (SPARK-24397) Add TaskContext.getLocalProperties in Python

[ https://issues.apache.org/jira/browse/SPARK-24397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24397: Fix Version/s: (was: 3.0.0) 2.4.0 > Add TaskContext.getLocalProperties in Python >

[jira] [Updated] (SPARK-24234) create the bottom-of-task RDD with row buffer

[ https://issues.apache.org/jira/browse/SPARK-24234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24234: Fix Version/s: (was: 3.0.0) 2.4.0 > create the bottom-of-task RDD with row buffer >

[jira] [Updated] (SPARK-24232) Allow referring to kubernetes secrets as env variable

[ https://issues.apache.org/jira/browse/SPARK-24232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24232: Fix Version/s: (was: 3.0.0) 2.4.0 > Allow referring to kubernetes secrets as env va

[jira] [Updated] (SPARK-24697) Fix the reported start offsets in streaming query progress

[ https://issues.apache.org/jira/browse/SPARK-24697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24697: Fix Version/s: (was: 3.0.0) 2.4.0 > Fix the reported start offsets in streaming que

[jira] [Updated] (SPARK-24662) Structured Streaming should support LIMIT

[ https://issues.apache.org/jira/browse/SPARK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24662: Fix Version/s: (was: 3.0.0) 2.4.0 > Structured Streaming should support LIMIT > ---

[jira] [Updated] (SPARK-24730) Add policy to choose max as global watermark when streaming query has multiple watermarks

[ https://issues.apache.org/jira/browse/SPARK-24730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24730: Fix Version/s: (was: 3.0.0) 2.4.0 > Add policy to choose max as global watermark wh

[jira] [Assigned] (SPARK-24795) Implement barrier execution mode

[ https://issues.apache.org/jira/browse/SPARK-24795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24795: Assignee: (was: Apache Spark) > Implement barrier execution mode > --

[jira] [Assigned] (SPARK-24795) Implement barrier execution mode

[ https://issues.apache.org/jira/browse/SPARK-24795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24795: Assignee: Apache Spark > Implement barrier execution mode > -

[jira] [Commented] (SPARK-24795) Implement barrier execution mode

[ https://issues.apache.org/jira/browse/SPARK-24795?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542538#comment-16542538 ] Apache Spark commented on SPARK-24795: -- User 'jiangxb1987' has created a pull reque

[jira] [Resolved] (SPARK-23486) LookupFunctions should not check the same function name more than once

[ https://issues.apache.org/jira/browse/SPARK-23486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23486. - Resolution: Fixed Assignee: kevin yu Fix Version/s: 2.4.0 > LookupFunctions should not c

[jira] [Assigned] (SPARK-24615) Accelerator aware task scheduling for Spark

[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reassigned SPARK-24615: - Assignee: Saisai Shao > Accelerator aware task scheduling for Spark > -

[jira] [Commented] (SPARK-24568) Code refactoring for DataType equalsXXX methods

[ https://issues.apache.org/jira/browse/SPARK-24568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542516#comment-16542516 ] Swapnil commented on SPARK-24568: - I am working on this. I will create PR very soon. >

[jira] [Assigned] (SPARK-24797) Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build the data source table

[ https://issues.apache.org/jira/browse/SPARK-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24797: Assignee: Apache Spark > Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parqu

[jira] [Assigned] (SPARK-24797) Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build the data source table

[ https://issues.apache.org/jira/browse/SPARK-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24797: Assignee: (was: Apache Spark) > Analyzer should respect spark.sql.hive.convertMetasto

[jira] [Commented] (SPARK-24797) Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build the data source table

[ https://issues.apache.org/jira/browse/SPARK-24797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542437#comment-16542437 ] Apache Spark commented on SPARK-24797: -- User 'CodingCat' has created a pull request

[jira] [Created] (SPARK-24797) Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build the data source table

Nan Zhu created SPARK-24797: --- Summary: Analyzer should respect spark.sql.hive.convertMetastoreOrc/Parquet when build the data source table Key: SPARK-24797 URL: https://issues.apache.org/jira/browse/SPARK-24797

[jira] [Assigned] (SPARK-24537) Add array_remove / array_zip / map_from_arrays / array_distinct

[ https://issues.apache.org/jira/browse/SPARK-24537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-24537: Assignee: Huaxin Gao > Add array_remove / array_zip / map_from_arrays / array_distinct >

[jira] [Resolved] (SPARK-24537) Add array_remove / array_zip / map_from_arrays / array_distinct

[ https://issues.apache.org/jira/browse/SPARK-24537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-24537. -- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21645 [https://gi

[jira] [Resolved] (SPARK-24790) Allow complex aggregate expressions in Pivot

[ https://issues.apache.org/jira/browse/SPARK-24790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24790. - Resolution: Fixed Assignee: Maryann Xue Fix Version/s: 2.4.0 > Allow complex aggregate e

[jira] [Commented] (SPARK-24796) Support GROUPED_AGG_PANDAS_UDF in Pivot

[ https://issues.apache.org/jira/browse/SPARK-24796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542338#comment-16542338 ] Xiao Li commented on SPARK-24796: - cc [~icexelloss] [~bryanc] [~hyukjin.kwon] > Support

[jira] [Created] (SPARK-24796) Support GROUPED_AGG_PANDAS_UDF in Pivot

Xiao Li created SPARK-24796: --- Summary: Support GROUPED_AGG_PANDAS_UDF in Pivot Key: SPARK-24796 URL: https://issues.apache.org/jira/browse/SPARK-24796 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-23007) Add schema evolution test suite for file-based data sources

[ https://issues.apache.org/jira/browse/SPARK-23007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-23007. - Resolution: Fixed Assignee: Dongjoon Hyun Fix Version/s: 2.4.0 > Add schema evolution te

[jira] [Commented] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542207#comment-16542207 ] Xiao Li commented on SPARK-24741: - [~mn-mikke] So far, we do not have a plan to improve

[jira] [Resolved] (SPARK-24741) Have a built-in AVRO data source implementation

[ https://issues.apache.org/jira/browse/SPARK-24741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-24741. - Resolution: Fixed Assignee: Gengliang Wang > Have a built-in AVRO data source implementation > ---

[jira] [Resolved] (SPARK-24610) wholeTextFiles broken for small files

[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-24610. --- Resolution: Fixed Fix Version/s: 2.4.0 > wholeTextFiles broken for small files >

[jira] [Assigned] (SPARK-24610) wholeTextFiles broken for small files

[ https://issues.apache.org/jira/browse/SPARK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-24610: - Assignee: Dhruve Ashar > wholeTextFiles broken for small files > --

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542135#comment-16542135 ] Anirudh Ramanathan commented on SPARK-24793: Great! I'll take a stab at a PR

[jira] [Updated] (SPARK-24724) Discuss necessary info and access in barrier mode + Kubernetes

[ https://issues.apache.org/jira/browse/SPARK-24724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yinan Li updated SPARK-24724: - Component/s: Kubernetes > Discuss necessary info and access in barrier mode + Kubernetes > -

[jira] [Comment Edited] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542109#comment-16542109 ] Yinan Li edited comment on SPARK-24793 at 7/12/18 7:11 PM: --- Oh

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542109#comment-16542109 ] Yinan Li commented on SPARK-24793: -- Oh, yeah, {{kill}} and {{status}} are existing opti

[jira] [Comment Edited] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542102#comment-16542102 ] Erik Erlandson edited comment on SPARK-24793 at 7/12/18 7:01 PM: -

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542102#comment-16542102 ] Erik Erlandson commented on SPARK-24793: Also a good point that {{--kill}} and {

[jira] [Commented] (SPARK-20894) Error while checkpointing to HDFS

[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542006#comment-16542006 ] Robert Reid commented on SPARK-20894: - I'm encountering similar problems as [~aydinc

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16542003#comment-16542003 ] Yinan Li commented on SPARK-24793: -- Good points, Erik. I think [sparkctl|https://githu

[jira] [Updated] (SPARK-24795) Implement barrier execution mode

[ https://issues.apache.org/jira/browse/SPARK-24795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiang Xingbo updated SPARK-24795: - Description: Implement barrier execution mode, as described in SPARK-24582 Include all the API c

[jira] [Created] (SPARK-24795) Implement barrier execution mode

Jiang Xingbo created SPARK-24795: Summary: Implement barrier execution mode Key: SPARK-24795 URL: https://issues.apache.org/jira/browse/SPARK-24795 Project: Spark Issue Type: New Feature

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541990#comment-16541990 ] Anirudh Ramanathan commented on SPARK-24793: Good points Erik. These options

[jira] [Commented] (SPARK-24760) Pandas UDF does not handle NaN correctly

[ https://issues.apache.org/jira/browse/SPARK-24760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541967#comment-16541967 ] Bryan Cutler commented on SPARK-24760: -- Yeah, createDataFrame is inconsistent with

[jira] [Updated] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-20168: -- Priority: Minor (was: Major) > Enable kinesis to start stream from Initial position specified by a ti

[jira] [Resolved] (SPARK-20168) Enable kinesis to start stream from Initial position specified by a timestamp

[ https://issues.apache.org/jira/browse/SPARK-20168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20168. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21541 [https://github.c

[jira] [Commented] (SPARK-24764) Add ServiceLoader implementation for SparkHadoopUtil

[ https://issues.apache.org/jira/browse/SPARK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541932#comment-16541932 ] Apache Spark commented on SPARK-24764: -- User 'shrutig' has created a pull request f

[jira] [Assigned] (SPARK-24764) Add ServiceLoader implementation for SparkHadoopUtil

[ https://issues.apache.org/jira/browse/SPARK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24764: Assignee: Apache Spark (was: Shruti Gumma) > Add ServiceLoader implementation for SparkH

[jira] [Assigned] (SPARK-24764) Add ServiceLoader implementation for SparkHadoopUtil

[ https://issues.apache.org/jira/browse/SPARK-24764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24764: Assignee: Shruti Gumma (was: Apache Spark) > Add ServiceLoader implementation for SparkH

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541930#comment-16541930 ] Erik Erlandson commented on SPARK-24793: Another possible angle (not mutually ex

[jira] [Commented] (SPARK-24793) Make spark-submit more useful with k8s

[ https://issues.apache.org/jira/browse/SPARK-24793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541923#comment-16541923 ] Erik Erlandson commented on SPARK-24793: I am concerned that this is outside the

[jira] [Commented] (SPARK-21962) Distributed Tracing in Spark

[ https://issues.apache.org/jira/browse/SPARK-21962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541915#comment-16541915 ] Steve Loughran commented on SPARK-21962: Yes, assume that: * Htrace goes off t

[jira] [Resolved] (SPARK-24691) Add new API `supportDataType` in FileFormat

[ https://issues.apache.org/jira/browse/SPARK-24691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-24691. - Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 21667 [https://gith

  1   2   >