[jira] [Commented] (SPARK-15689) Data source API v2

[ https://issues.apache.org/jira/browse/SPARK-15689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636307#comment-16636307 ] Geoff Freeman commented on SPARK-15689: --- Thanks Wenchen. I was hoping that we might be able to

[jira] [Assigned] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14681: Assignee: Weichen Xu (was: Apache Spark) > Provide label/impurity stats for spark.ml

[jira] [Assigned] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14681: Assignee: Apache Spark (was: Weichen Xu) > Provide label/impurity stats for spark.ml

[jira] [Resolved] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-14681. --- Resolution: Won't Do Fix Version/s: (was: 2.4.0) Marked the ticket as "won't do"

[jira] [Commented] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636138#comment-16636138 ] Xiangrui Meng commented on SPARK-14681: --- The change were reverted in both branch-2.4 and master to

[jira] [Reopened] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng reopened SPARK-14681: --- > Provide label/impurity stats for spark.ml decision tree nodes >

[jira] [Commented] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636136#comment-16636136 ] Apache Spark commented on SPARK-25321: -- User 'mengxr' has created a pull request for this issue:

[jira] [Commented] (SPARK-14681) Provide label/impurity stats for spark.ml decision tree nodes

[ https://issues.apache.org/jira/browse/SPARK-14681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636135#comment-16636135 ] Apache Spark commented on SPARK-14681: -- User 'mengxr' has created a pull request for this issue:

[jira] [Commented] (SPARK-25321) ML, Graph 2.4 QA: API: New Scala APIs, docs

[ https://issues.apache.org/jira/browse/SPARK-25321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636133#comment-16636133 ] Apache Spark commented on SPARK-25321: -- User 'mengxr' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25484) Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

[ https://issues.apache.org/jira/browse/SPARK-25484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25484: Assignee: (was: Apache Spark) > Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark >

[jira] [Commented] (SPARK-25484) Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

[ https://issues.apache.org/jira/browse/SPARK-25484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16636111#comment-16636111 ] Apache Spark commented on SPARK-25484: -- User 'peter-toth' has created a pull request for this

[jira] [Assigned] (SPARK-25484) Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark

[ https://issues.apache.org/jira/browse/SPARK-25484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25484: Assignee: Apache Spark > Refactor ExternalAppendOnlyUnsafeRowArrayBenchmark >

[jira] [Created] (SPARK-25599) Stateful aggregation in PySpark

Vincent Grosbois created SPARK-25599: Summary: Stateful aggregation in PySpark Key: SPARK-25599 URL: https://issues.apache.org/jira/browse/SPARK-25599 Project: Spark Issue Type: New

[jira] [Commented] (SPARK-25586) toString method of GeneralizedLinearRegressionTrainingSummary runs in infinite loop throwing StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-25586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635968#comment-16635968 ] Apache Spark commented on SPARK-25586: -- User 'ankuriitg' has created a pull request for this issue:

[jira] [Commented] (SPARK-25586) toString method of GeneralizedLinearRegressionTrainingSummary runs in infinite loop throwing StackOverflowError

[ https://issues.apache.org/jira/browse/SPARK-25586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635967#comment-16635967 ] Apache Spark commented on SPARK-25586: -- User 'ankuriitg' has created a pull request for this issue:

[jira] [Created] (SPARK-25598) Remove flume connector in Spark 3

Marcelo Vanzin created SPARK-25598: -- Summary: Remove flume connector in Spark 3 Key: SPARK-25598 URL: https://issues.apache.org/jira/browse/SPARK-25598 Project: Spark Issue Type: Task

[jira] [Commented] (SPARK-25016) remove Support for hadoop 2.6

[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635962#comment-16635962 ] Apache Spark commented on SPARK-25016: -- User 'srowen' has created a pull request for this issue:

[jira] [Commented] (SPARK-25016) remove Support for hadoop 2.6

[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635959#comment-16635959 ] Apache Spark commented on SPARK-25016: -- User 'srowen' has created a pull request for this issue:

[jira] [Assigned] (SPARK-25016) remove Support for hadoop 2.6

[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25016: Assignee: Apache Spark > remove Support for hadoop 2.6 > - >

[jira] [Assigned] (SPARK-25016) remove Support for hadoop 2.6

[ https://issues.apache.org/jira/browse/SPARK-25016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25016: Assignee: (was: Apache Spark) > remove Support for hadoop 2.6 >

[jira] [Assigned] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25561: Assignee: (was: Apache Spark) > HiveClient.getPartitionsByFilter throws an exception

[jira] [Assigned] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25561: Assignee: Apache Spark > HiveClient.getPartitionsByFilter throws an exception if Hive

[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635946#comment-16635946 ] Apache Spark commented on SPARK-25561: -- User 'kmanamcheri' has created a pull request for this

[jira] [Commented] (SPARK-25561) HiveClient.getPartitionsByFilter throws an exception if Hive retries directSql

[ https://issues.apache.org/jira/browse/SPARK-25561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635945#comment-16635945 ] Karthik Manamcheri commented on SPARK-25561: Created PR

[jira] [Commented] (SPARK-7276) withColumn is very slow on dataframe with large number of columns

[ https://issues.apache.org/jira/browse/SPARK-7276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635891#comment-16635891 ] Jacek Tokar commented on SPARK-7276: I confirm Barry's observation. > withColumn is very slow on

[jira] [Comment Edited] (SPARK-25576) Fix lint failure in 2.2

[ https://issues.apache.org/jira/browse/SPARK-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635839#comment-16635839 ] Dongjoon Hyun edited comment on SPARK-25576 at 10/2/18 5:17 PM:

[jira] [Resolved] (SPARK-22275) SparkContext doesn't clean up after itself when "fatal" errors occur

[ https://issues.apache.org/jira/browse/SPARK-22275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-22275. Resolution: Won't Fix Let's leave it like that until it becomes a bigger problem. >

[jira] [Resolved] (SPARK-25576) Fix lint failure in 2.2

[ https://issues.apache.org/jira/browse/SPARK-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25576. --- Resolution: Fixed Fix Version/s: 2.2.3 Issue resolved by pull request 22596

[jira] [Updated] (SPARK-25576) Fix lint failure in 2.2

[ https://issues.apache.org/jira/browse/SPARK-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-25576: -- Attachment: Screen Shot 2018-10-02 at 10.16.52 AM.png > Fix lint failure in 2.2 >

[jira] [Commented] (SPARK-25576) Fix lint failure in 2.2

[ https://issues.apache.org/jira/browse/SPARK-25576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635839#comment-16635839 ] Dongjoon Hyun commented on SPARK-25576: --- [~samdvr]. It's weird. I cannot assign to you. > Fix

[jira] [Updated] (SPARK-24499) Documentation improvement of Spark core and SQL

[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-24499: Target Version/s: 3.0.0 > Documentation improvement of Spark core and SQL >

[jira] [Commented] (SPARK-24499) Documentation improvement of Spark core and SQL

[ https://issues.apache.org/jira/browse/SPARK-24499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635836#comment-16635836 ] Xiao Li commented on SPARK-24499: - [~XuanYuan] Yeah. Let us do the split first, and then discuss how to

[jira] [Resolved] (SPARK-25581) Rename method `benchmark` in BenchmarkBase as benchmarkSuite

[ https://issues.apache.org/jira/browse/SPARK-25581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25581. --- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 22599

[jira] [Assigned] (SPARK-25581) Rename method `benchmark` in BenchmarkBase as benchmarkSuite

[ https://issues.apache.org/jira/browse/SPARK-25581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25581: - Assignee: Gengliang Wang > Rename method `benchmark` in BenchmarkBase as

[jira] [Commented] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635811#comment-16635811 ] Stu (Michael Stewart) commented on SPARK-25461: --- Thanks all; to me the largest issue with

[jira] [Commented] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635762#comment-16635762 ] Bryan Cutler commented on SPARK-25461: -- Thanks for looking into this [~viirya]! You are right that

[jira] [Updated] (SPARK-25414) make it clear that the numRows metrics should be counted for each scan of the source

[ https://issues.apache.org/jira/browse/SPARK-25414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25414: Fix Version/s: (was: 2.5.0) > make it clear that the numRows metrics should be counted for each scan

[jira] [Updated] (SPARK-25426) Remove the duplicate fallback logic in UnsafeProjection

[ https://issues.apache.org/jira/browse/SPARK-25426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25426: Fix Version/s: (was: 2.5.0) > Remove the duplicate fallback logic in UnsafeProjection >

[jira] [Assigned] (SPARK-25381) Stratified sampling by Column argument

[ https://issues.apache.org/jira/browse/SPARK-25381?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li reassigned SPARK-25381: --- Assignee: Maxim Gekk > Stratified sampling by Column argument >

[jira] [Updated] (SPARK-25423) Output "dataFilters" in DataSourceScanExec.metadata

[ https://issues.apache.org/jira/browse/SPARK-25423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25423: Fix Version/s: (was: 2.5.0) > Output "dataFilters" in DataSourceScanExec.metadata >

[jira] [Updated] (SPARK-25415) Make plan change log in RuleExecutor configurable by SQLConf

[ https://issues.apache.org/jira/browse/SPARK-25415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25415: Fix Version/s: (was: 2.5.0) > Make plan change log in RuleExecutor configurable by SQLConf >

[jira] [Updated] (SPARK-25449) Don't send zero accumulators in heartbeats

[ https://issues.apache.org/jira/browse/SPARK-25449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25449: Fix Version/s: (was: 2.5.0) 3.0.0 > Don't send zero accumulators in heartbeats >

[jira] [Updated] (SPARK-25472) Structured Streaming query.stop() doesn't always stop gracefully

[ https://issues.apache.org/jira/browse/SPARK-25472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25472: Fix Version/s: (was: 2.5.0) 3.0.0 > Structured Streaming query.stop() doesn't

[jira] [Updated] (SPARK-25458) Support FOR ALL COLUMNS in ANALYZE TABLE

[ https://issues.apache.org/jira/browse/SPARK-25458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25458: Fix Version/s: (was: 2.5.0) 3.0.0 > Support FOR ALL COLUMNS in ANALYZE TABLE >

[jira] [Updated] (SPARK-25457) IntegralDivide (div) should not always return long

[ https://issues.apache.org/jira/browse/SPARK-25457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25457: Fix Version/s: (was: 2.5.0) 3.0.0 > IntegralDivide (div) should not always return

[jira] [Updated] (SPARK-25429) SparkListenerBus inefficient due to 'LiveStageMetrics#accumulatorIds:Array[Long]' data structure

[ https://issues.apache.org/jira/browse/SPARK-25429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25429: Fix Version/s: (was: 2.5.0) > SparkListenerBus inefficient due to >

[jira] [Updated] (SPARK-25444) Refactor GenArrayData.genCodeToCreateArrayData() method

[ https://issues.apache.org/jira/browse/SPARK-25444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25444: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor

[jira] [Updated] (SPARK-25447) Support JSON options by schema_of_json

[ https://issues.apache.org/jira/browse/SPARK-25447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25447: Fix Version/s: (was: 2.5.0) 3.0.0 > Support JSON options by schema_of_json >

[jira] [Updated] (SPARK-25465) Refactor Parquet test suites in project Hive

[ https://issues.apache.org/jira/browse/SPARK-25465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25465: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor Parquet test suites in project Hive >

[jira] [Updated] (SPARK-25476) Refactor AggregateBenchmark to use main method

[ https://issues.apache.org/jira/browse/SPARK-25476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25476: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor AggregateBenchmark to use main method

[jira] [Updated] (SPARK-25473) PySpark ForeachWriter test fails on Python 3.6 and macOS High Serria

[ https://issues.apache.org/jira/browse/SPARK-25473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25473: Fix Version/s: (was: 2.5.0) 3.0.0 > PySpark ForeachWriter test fails on Python 3.6

[jira] [Updated] (SPARK-25486) Refactor SortBenchmark to use main method

[ https://issues.apache.org/jira/browse/SPARK-25486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25486: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor SortBenchmark to use main method >

[jira] [Updated] (SPARK-25499) Refactor BenchmarkBase and Benchmark

[ https://issues.apache.org/jira/browse/SPARK-25499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25499: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor BenchmarkBase and Benchmark >

[jira] [Updated] (SPARK-25489) Refactor UDTSerializationBenchmark

[ https://issues.apache.org/jira/browse/SPARK-25489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25489: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor UDTSerializationBenchmark >

[jira] [Updated] (SPARK-25481) Refactor ColumnarBatchBenchmark to use main method

[ https://issues.apache.org/jira/browse/SPARK-25481?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25481: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor ColumnarBatchBenchmark to use main

[jira] [Updated] (SPARK-25485) Refactor UnsafeProjectionBenchmark to use main method

[ https://issues.apache.org/jira/browse/SPARK-25485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25485: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor UnsafeProjectionBenchmark to use main

[jira] [Updated] (SPARK-25478) Refactor CompressionSchemeBenchmark to use main method

[ https://issues.apache.org/jira/browse/SPARK-25478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25478: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor CompressionSchemeBenchmark to use

[jira] [Updated] (SPARK-25494) Upgrade Spark's use of Janino to 3.0.10

[ https://issues.apache.org/jira/browse/SPARK-25494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25494: Fix Version/s: (was: 2.5.0) 3.0.0 > Upgrade Spark's use of Janino to 3.0.10 >

[jira] [Updated] (SPARK-25487) Refactor PrimitiveArrayBenchmark

[ https://issues.apache.org/jira/browse/SPARK-25487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25487: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor PrimitiveArrayBenchmark >

[jira] [Updated] (SPARK-25508) Refactor OrcReadBenchmark to use main method

[ https://issues.apache.org/jira/browse/SPARK-25508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25508: Fix Version/s: (was: 2.5.0) 3.0.0 > Refactor OrcReadBenchmark to use main method >

[jira] [Updated] (SPARK-25534) Make `SQLHelper` trait

[ https://issues.apache.org/jira/browse/SPARK-25534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25534: Fix Version/s: (was: 2.5.0) 3.0.0 > Make `SQLHelper` trait >

[jira] [Updated] (SPARK-25510) Create a new trait SqlBasedBenchmark

[ https://issues.apache.org/jira/browse/SPARK-25510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25510: Fix Version/s: (was: 2.5.0) 3.0.0 > Create a new trait SqlBasedBenchmark >

[jira] [Updated] (SPARK-25514) Generating pretty JSON by to_json

[ https://issues.apache.org/jira/browse/SPARK-25514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25514: Fix Version/s: (was: 2.5.0) 3.0.0 > Generating pretty JSON by to_json >

[jira] [Updated] (SPARK-25541) CaseInsensitiveMap should be serializable after '-' operator

[ https://issues.apache.org/jira/browse/SPARK-25541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25541: Fix Version/s: (was: 2.5.0) 3.0.0 > CaseInsensitiveMap should be serializable

[jira] [Updated] (SPARK-25540) Make HiveContext in PySpark behave as the same as Scala.

[ https://issues.apache.org/jira/browse/SPARK-25540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25540: Fix Version/s: (was: 2.5.0) 3.0.0 > Make HiveContext in PySpark behave as the same

[jira] [Updated] (SPARK-25525) Do not update conf for existing SparkContext in SparkSession.getOrCreate.

[ https://issues.apache.org/jira/browse/SPARK-25525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25525: Fix Version/s: (was: 2.5.0) 3.0.0 > Do not update conf for existing SparkContext

[jira] [Updated] (SPARK-25551) Remove unused InSubquery expression

[ https://issues.apache.org/jira/browse/SPARK-25551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25551: Fix Version/s: (was: 2.5.0) 3.0.0 > Remove unused InSubquery expression >

[jira] [Updated] (SPARK-25559) Just remove the unsupported predicates in Parquet

[ https://issues.apache.org/jira/browse/SPARK-25559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25559: Fix Version/s: (was: 2.5.0) 3.0.0 > Just remove the unsupported predicates in

[jira] [Updated] (SPARK-25565) Add scala style checker to check add Locale.ROOT to .toLowerCase and .toUpperCase for internal calls

[ https://issues.apache.org/jira/browse/SPARK-25565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25565: Fix Version/s: (was: 2.5.0) 3.0.0 > Add scala style checker to check add

[jira] [Updated] (SPARK-25575) SQL tab in the spark UI doesn't have option of hiding tables, eventhough other UI tabs has.

[ https://issues.apache.org/jira/browse/SPARK-25575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-25575: Fix Version/s: (was: 2.5.0) 3.0.0 > SQL tab in the spark UI doesn't have option of

[jira] [Resolved] (SPARK-25592) Bump master branch version to 3.0.0-SNAPSHOT

[ https://issues.apache.org/jira/browse/SPARK-25592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li resolved SPARK-25592. - Resolution: Fixed Fix Version/s: 3.0.0 > Bump master branch version to 3.0.0-SNAPSHOT >

[jira] [Commented] (SPARK-25583) Add newly added History server related configurations in the documentation

[ https://issues.apache.org/jira/browse/SPARK-25583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635688#comment-16635688 ] Apache Spark commented on SPARK-25583: -- User 'shahidki31' has created a pull request for this

[jira] [Commented] (SPARK-24958) Add executors' process tree total memory information to heartbeat signals

[ https://issues.apache.org/jira/browse/SPARK-24958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635687#comment-16635687 ] Apache Spark commented on SPARK-24958: -- User 'rezasafi' has created a pull request for this issue:

[jira] [Commented] (SPARK-24958) Add executors' process tree total memory information to heartbeat signals

[ https://issues.apache.org/jira/browse/SPARK-24958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635686#comment-16635686 ] Apache Spark commented on SPARK-24958: -- User 'rezasafi' has created a pull request for this issue:

[jira] [Updated] (SPARK-24958) Add executors' process tree total memory information to heartbeat signals

[ https://issues.apache.org/jira/browse/SPARK-24958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reza Safi updated SPARK-24958: -- Summary: Add executors' process tree total memory information to heartbeat signals (was: Report

[jira] [Commented] (SPARK-25582) Error in Spark logs when using the org.apache.spark:spark-sql_2.11:2.2.0 Java library

[ https://issues.apache.org/jira/browse/SPARK-25582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635655#comment-16635655 ] Thomas Brugiere commented on SPARK-25582: - Hi Marco, Do I need to do anything on my side in

[jira] [Resolved] (SPARK-25583) Add newly added History server related configurations in the documentation

[ https://issues.apache.org/jira/browse/SPARK-25583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-25583. --- Resolution: Fixed Fix Version/s: 2.4.0 Issue resolved by pull request 22601

[jira] [Assigned] (SPARK-25583) Add newly added History server related configurations in the documentation

[ https://issues.apache.org/jira/browse/SPARK-25583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-25583: - Assignee: shahid > Add newly added History server related configurations in the

[jira] [Commented] (SPARK-23153) Support application dependencies in submission client's local file system

[ https://issues.apache.org/jira/browse/SPARK-23153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635534#comment-16635534 ] Rob Vesse commented on SPARK-23153: --- [~cloud_fan][~liyinan926][~mcheah][~eje] Has there been any

[jira] [Commented] (SPARK-25497) limit operation within whole stage codegen should not consume all the inputs

[ https://issues.apache.org/jira/browse/SPARK-25497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635443#comment-16635443 ] Hannu Kröger commented on SPARK-25497: -- Note: This also affects 2.3.0. > limit operation within

[jira] [Resolved] (SPARK-25597) SQL query with limit iterates the whole iterator when WholeStage code generation is enabled

[ https://issues.apache.org/jira/browse/SPARK-25597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hannu Kröger resolved SPARK-25597. -- Resolution: Duplicate > SQL query with limit iterates the whole iterator when WholeStage code

[jira] [Commented] (SPARK-25597) SQL query with limit iterates the whole iterator when WholeStage code generation is enabled

[ https://issues.apache.org/jira/browse/SPARK-25597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635430#comment-16635430 ] Marco Gaido commented on SPARK-25597: - I think this is a duplicate of SPARK-25497. [~hkroger] may

[jira] [Commented] (SPARK-11150) Dynamic partition pruning

[ https://issues.apache.org/jira/browse/SPARK-11150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635349#comment-16635349 ] Dani Rubio commented on SPARK-11150: I would edit it, as I am specifically interested in the case

[jira] [Updated] (SPARK-25597) SQL query with limit iterates the whole iterator when WholeStage code generation is enabled

[ https://issues.apache.org/jira/browse/SPARK-25597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hannu Kröger updated SPARK-25597: - Description: When _SELECT * FROM table LIMIT 1_ is executed, the WholeStageCodegenExec

[jira] [Created] (SPARK-25597) SQL query with limit iterates the whole iterator when WholeStage code generation is enabled

Hannu Kröger created SPARK-25597: Summary: SQL query with limit iterates the whole iterator when WholeStage code generation is enabled Key: SPARK-25597 URL: https://issues.apache.org/jira/browse/SPARK-25597

[jira] [Created] (SPARK-25596) TLS1.3 support

t oo created SPARK-25596: Summary: TLS1.3 support Key: SPARK-25596 URL: https://issues.apache.org/jira/browse/SPARK-25596 Project: Spark Issue Type: Sub-task Components: Project Infra

[jira] [Commented] (SPARK-16418) DataFrame.filter fails if it references a window function

[ https://issues.apache.org/jira/browse/SPARK-16418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635283#comment-16635283 ] belgacea commented on SPARK-16418: -- I'm encountering [this

[jira] [Commented] (SPARK-24417) Build and Run Spark on JDK9+

[ https://issues.apache.org/jira/browse/SPARK-24417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635253#comment-16635253 ] t oo commented on SPARK-24417: -- [http://jdk.java.net/11/]  i guess can skip java9-10 and go to 11 > Build

[jira] [Commented] (SPARK-24422) Add JDK9+ in our Jenkins' build servers

[ https://issues.apache.org/jira/browse/SPARK-24422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635250#comment-16635250 ] t oo commented on SPARK-24422: -- https://jenkins.io/blog/2018/06/17/running-jenkins-with-java10-11/ > Add

[jira] [Commented] (SPARK-16859) History Server storage information is missing

[ https://issues.apache.org/jira/browse/SPARK-16859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635237#comment-16635237 ] ravi commented on SPARK-16859: -- still an issue. Storage tab on SparkUI is empty > History Server storage

[jira] [Commented] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635216#comment-16635216 ] Liang-Chi Hsieh commented on SPARK-25461: - [~hyukjin.kwon] Thanks and no problem at all! You can

[jira] [Commented] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635210#comment-16635210 ] Hyukjin Kwon commented on SPARK-25461: -- [~viirya], I am sorry I missed this. I have been busy this

[jira] [Commented] (SPARK-25595) Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

[ https://issues.apache.org/jira/browse/SPARK-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635157#comment-16635157 ] Apache Spark commented on SPARK-25595: -- User 'gengliangwang' has created a pull request for this

[jira] [Commented] (SPARK-25595) Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

[ https://issues.apache.org/jira/browse/SPARK-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635155#comment-16635155 ] Apache Spark commented on SPARK-25595: -- User 'gengliangwang' has created a pull request for this

[jira] [Assigned] (SPARK-25595) Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

[ https://issues.apache.org/jira/browse/SPARK-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25595: Assignee: (was: Apache Spark) > Ignore corrupt Avro file if flag

[jira] [Assigned] (SPARK-25595) Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

[ https://issues.apache.org/jira/browse/SPARK-25595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25595: Assignee: Apache Spark > Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled >

[jira] [Created] (SPARK-25595) Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled

Gengliang Wang created SPARK-25595: -- Summary: Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES enabled Key: SPARK-25595 URL: https://issues.apache.org/jira/browse/SPARK-25595 Project: Spark

[jira] [Assigned] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-25461: Assignee: Apache Spark > PySpark Pandas UDF outputs incorrect results when input columns

[jira] [Commented] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635146#comment-16635146 ] Apache Spark commented on SPARK-25461: -- User 'viirya' has created a pull request for this issue:

[jira] [Commented] (SPARK-25461) PySpark Pandas UDF outputs incorrect results when input columns contain None

[ https://issues.apache.org/jira/browse/SPARK-25461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635147#comment-16635147 ] Apache Spark commented on SPARK-25461: -- User 'viirya' has created a pull request for this issue:

  1   2   >