[jira] [Assigned] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32638: Assignee: Apache Spark > WidenSetOperationTypes in subquery attribute missing > ---

[jira] [Commented] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180985#comment-17180985 ] Apache Spark commented on SPARK-32638: -- User 'maropu' has created a pull request fo

[jira] [Assigned] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32638: Assignee: (was: Apache Spark) > WidenSetOperationTypes in subquery attribute missin

[jira] [Assigned] (SPARK-32649) Optimize BHJ/SHJ inner and semi join with empty hashed relation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32649: Assignee: (was: Apache Spark) > Optimize BHJ/SHJ inner and semi join with empty hashe

[jira] [Commented] (SPARK-32649) Optimize BHJ/SHJ inner and semi join with empty hashed relation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180978#comment-17180978 ] Apache Spark commented on SPARK-32649: -- User 'c21' has created a pull request for t

[jira] [Assigned] (SPARK-32649) Optimize BHJ/SHJ inner and semi join with empty hashed relation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32649: Assignee: Apache Spark > Optimize BHJ/SHJ inner and semi join with empty hashed relation

[jira] [Commented] (SPARK-32649) Optimize BHJ/SHJ inner and semi join with empty hashed relation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180977#comment-17180977 ] Apache Spark commented on SPARK-32649: -- User 'c21' has created a pull request for t

[jira] [Created] (SPARK-32664) Getting local shuffle block clutters the executor logs

2020-08-19 Thread Chandni Singh (Jira)
Chandni Singh created SPARK-32664: - Summary: Getting local shuffle block clutters the executor logs Key: SPARK-32664 URL: https://issues.apache.org/jira/browse/SPARK-32664 Project: Spark Issu

[jira] [Commented] (SPARK-32662) CountVectorizerModel: Remove requirement for minimum vocabulary size

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180972#comment-17180972 ] Apache Spark commented on SPARK-32662: -- User 'purijatin' has created a pull request

[jira] [Assigned] (SPARK-32662) CountVectorizerModel: Remove requirement for minimum vocabulary size

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32662: Assignee: (was: Apache Spark) > CountVectorizerModel: Remove requirement for minimum

[jira] [Assigned] (SPARK-32662) CountVectorizerModel: Remove requirement for minimum vocabulary size

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32662: Assignee: Apache Spark > CountVectorizerModel: Remove requirement for minimum vocabulary

[jira] [Commented] (SPARK-32662) CountVectorizerModel: Remove requirement for minimum vocabulary size

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180971#comment-17180971 ] Apache Spark commented on SPARK-32662: -- User 'purijatin' has created a pull request

[jira] [Commented] (SPARK-29967) KMeans support instance weighting

2020-08-19 Thread YuQiang Ye (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180970#comment-17180970 ] YuQiang Ye commented on SPARK-29967: {code:java} scala> import org.apache.spark.mlli

[jira] [Resolved] (SPARK-32042) Support UTF8String literals

2020-08-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-32042. - Resolution: Won't Fix > Support UTF8String literals > --- > >

[jira] [Commented] (SPARK-29967) KMeans support instance weighting

2020-08-19 Thread zhengruifeng (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180937#comment-17180937 ] zhengruifeng commented on SPARK-29967: -- [~YuQiang Ye] can you provide more details

[jira] [Commented] (SPARK-29967) KMeans support instance weighting

2020-08-19 Thread Sean R. Owen (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180936#comment-17180936 ] Sean R. Owen commented on SPARK-29967: -- No, why? I don't understand any of that. >

[jira] [Commented] (SPARK-29967) KMeans support instance weighting

2020-08-19 Thread YuQiang Ye (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180932#comment-17180932 ] YuQiang Ye commented on SPARK-29967: {code:java} def run(data: RDD[Vector]): KMean

[jira] [Resolved] (SPARK-32296) Flaky Test: submit a barrier ResultStage that requires more slots than current total under local-cluster mode

2020-08-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32296. -- Resolution: Later I don't see this flakiness anymore. Tentatively resolving. > Flaky Test: su

[jira] [Resolved] (SPARK-32304) Flaky Test: AdaptiveQueryExecSuite.multiple joins

2020-08-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32304. -- Resolution: Later I don't see this flakiness anymore. Tentatively resolving. > Flaky Test: Ad

[jira] [Resolved] (SPARK-32297) Flaky Test: YarnClusterSuite 4 test cases

2020-08-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32297. -- Resolution: Later I don't see this flakiness anymore. Tentatively resolving. > Flaky Test: Ya

[jira] [Resolved] (SPARK-32255) Set jobs that runs periodically for other profile combinations

2020-08-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-32255. -- Resolution: Later I am going to resolve this as a later for now due to the resource problem at

[jira] [Updated] (SPARK-31966) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2020-08-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31966: - Fix Version/s: 2.4.8 > Flaky test: pyspark.mllib.tests.test_streaming_algorithms > StreamingLog

[jira] [Updated] (SPARK-31966) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2020-08-19 Thread Hyukjin Kwon (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-31966: - Fix Version/s: (was: 2.4.8) 2.4.7 > Flaky test: pyspark.mllib.tests.test_

[jira] [Updated] (SPARK-32249) Run Github Actions builds in other branches as well

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32249: -- Fix Version/s: 2.4.7 > Run Github Actions builds in other branches as well > -

[jira] [Commented] (SPARK-31966) Flaky test: pyspark.mllib.tests.test_streaming_algorithms StreamingLogisticRegressionWithSGDTests.test_training_and_prediction

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-31966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180906#comment-17180906 ] Apache Spark commented on SPARK-31966: -- User 'HyukjinKwon' has created a pull reque

[jira] [Assigned] (SPARK-32657) Decommissioning Integration Test checks for old string

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32657: - Assignee: Holden Karau > Decommissioning Integration Test checks for old string > -

[jira] [Resolved] (SPARK-32657) Decommissioning Integration Test checks for old string

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32657. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29479 [https://

[jira] [Updated] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server

2020-08-19 Thread Chandni Singh (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chandni Singh updated SPARK-32663: -- Description: The implementation of {{removeBlocks}} and {{getHostLocalDirs}} in {{ExternalBlo

[jira] [Created] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server

2020-08-19 Thread Chandni Singh (Jira)
Chandni Singh created SPARK-32663: - Summary: TransportClient getting closed when there are outstanding requests to the server Key: SPARK-32663 URL: https://issues.apache.org/jira/browse/SPARK-32663 Pr

[jira] [Comment Edited] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

2020-08-19 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180772#comment-17180772 ] Thomas Graves edited comment on SPARK-32640 at 8/19/20, 7:14 PM: -

[jira] [Comment Edited] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

2020-08-19 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180772#comment-17180772 ] Thomas Graves edited comment on SPARK-32640 at 8/19/20, 7:14 PM: -

[jira] [Commented] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

2020-08-19 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180772#comment-17180772 ] Thomas Graves commented on SPARK-32640: --- hmm, interesting, this is how my test was

[jira] [Updated] (SPARK-32655) Support appId/execId placeholder in K8s SPARK_EXECUTOR_DIRS

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32655: -- Summary: Support appId/execId placeholder in K8s SPARK_EXECUTOR_DIRS (was: Support appId/exec

[jira] [Resolved] (SPARK-32655) Support appId/execId placeholder in K8s SPARK_EXECUTOR_DIRS

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32655. --- Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29472 [https://

[jira] [Commented] (SPARK-28008) Default values & column comments in AVRO schema converters

2020-08-19 Thread Richard Gilmore (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180737#comment-17180737 ] Richard Gilmore commented on SPARK-28008: - hit this issue, this causes issues if

[jira] [Commented] (SPARK-32657) Decommissioning Integration Test checks for old string

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180720#comment-17180720 ] Apache Spark commented on SPARK-32657: -- User 'holdenk' has created a pull request f

[jira] [Commented] (SPARK-32657) Decommissioning Integration Test checks for old string

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180719#comment-17180719 ] Apache Spark commented on SPARK-32657: -- User 'holdenk' has created a pull request f

[jira] [Assigned] (SPARK-32657) Decommissioning Integration Test checks for old string

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32657: Assignee: Apache Spark > Decommissioning Integration Test checks for old string > ---

[jira] [Assigned] (SPARK-32657) Decommissioning Integration Test checks for old string

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32657: Assignee: (was: Apache Spark) > Decommissioning Integration Test checks for old strin

[jira] [Commented] (SPARK-32621) "path" option is added again to input paths during infer()

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180718#comment-17180718 ] Apache Spark commented on SPARK-32621: -- User 'imback82' has created a pull request

[jira] [Updated] (SPARK-32662) CountVectorizerModel: Remove requirement for minimum vocabulary size

2020-08-19 Thread Jatin Puri (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jatin Puri updated SPARK-32662: --- Shepherd: (was: Jatin Puri) > CountVectorizerModel: Remove requirement for minimum vocabulary size

[jira] [Updated] (SPARK-32662) CountVectorizerModel: Remove requirement for minimum vocabulary size

2020-08-19 Thread Jatin Puri (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jatin Puri updated SPARK-32662: --- Shepherd: Jatin Puri > CountVectorizerModel: Remove requirement for minimum vocabulary size > --

[jira] [Created] (SPARK-32662) CountVectorizerModel: Remove requirement for minimum vocabulary size

2020-08-19 Thread Jatin Puri (Jira)
Jatin Puri created SPARK-32662: -- Summary: CountVectorizerModel: Remove requirement for minimum vocabulary size Key: SPARK-32662 URL: https://issues.apache.org/jira/browse/SPARK-32662 Project: Spark

[jira] [Assigned] (SPARK-28863) Add an AlreadyPlanned logical node that skips query planning

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-28863: --- Assignee: Burak Yavuz > Add an AlreadyPlanned logical node that skips query planning >

[jira] [Resolved] (SPARK-28863) Add an AlreadyPlanned logical node that skips query planning

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-28863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-28863. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull re

[jira] [Resolved] (SPARK-32621) "path" option is added again to input paths during infer()

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32621. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29437 [https://gith

[jira] [Assigned] (SPARK-32621) "path" option is added again to input paths during infer()

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32621: --- Assignee: Terry Kim > "path" option is added again to input paths during infer() >

[jira] [Commented] (SPARK-32628) Use bloom filter to improve dynamicPartitionPruning

2020-08-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180605#comment-17180605 ] Yuming Wang commented on SPARK-32628: - {code:scala} import org.apache.spark.benchmar

[jira] [Commented] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180598#comment-17180598 ] Wenchen Fan commented on SPARK-32640: - master branch works too {code:java} Welcome t

[jira] [Updated] (SPARK-32451) Support Apache Arrow 1.0.0 in SparkR

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-32451: -- Fix Version/s: 3.0.1 > Support Apache Arrow 1.0.0 in SparkR >

[jira] [Assigned] (SPARK-32249) Run Github Actions builds in other branches as well

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-32249: - Assignee: Hyukjin Kwon > Run Github Actions builds in other branches as well >

[jira] [Resolved] (SPARK-32249) Run Github Actions builds in other branches as well

2020-08-19 Thread Dongjoon Hyun (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-32249. --- Fix Version/s: 3.0.1 Resolution: Fixed Issue resolved by pull request 29460 [https://

[jira] [Commented] (SPARK-32661) Spark executors on K8S do not request extra memory for off-heap allocations

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180547#comment-17180547 ] Apache Spark commented on SPARK-32661: -- User 'LucaCanali' has created a pull reques

[jira] [Commented] (SPARK-32661) Spark executors on K8S do not request extra memory for off-heap allocations

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180546#comment-17180546 ] Apache Spark commented on SPARK-32661: -- User 'LucaCanali' has created a pull reques

[jira] [Assigned] (SPARK-32661) Spark executors on K8S do not request extra memory for off-heap allocations

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32661: Assignee: (was: Apache Spark) > Spark executors on K8S do not request extra memory fo

[jira] [Assigned] (SPARK-32661) Spark executors on K8S do not request extra memory for off-heap allocations

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32661: Assignee: Apache Spark > Spark executors on K8S do not request extra memory for off-heap

[jira] [Commented] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

2020-08-19 Thread Thomas Graves (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180533#comment-17180533 ] Thomas Graves commented on SPARK-32640: --- The problem is with 3.1.0 not, 3.0.0. The

[jira] [Created] (SPARK-32661) Spark executors on K8S do not request extra memory for off-heap allocations

2020-08-19 Thread Luca Canali (Jira)
Luca Canali created SPARK-32661: --- Summary: Spark executors on K8S do not request extra memory for off-heap allocations Key: SPARK-32661 URL: https://issues.apache.org/jira/browse/SPARK-32661 Project: Sp

[jira] [Assigned] (SPARK-32624) Replace getClass.getName with getClass.getCanonicalName in CodegenContext.addReferenceObj

2020-08-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang reassigned SPARK-32624: --- Assignee: Yuming Wang > Replace getClass.getName with getClass.getCanonicalName in > Codeg

[jira] [Resolved] (SPARK-32624) Replace getClass.getName with getClass.getCanonicalName in CodegenContext.addReferenceObj

2020-08-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang resolved SPARK-32624. - Fix Version/s: 3.1.0 3.0.1 Resolution: Fixed Issue resolved by pull re

[jira] [Commented] (SPARK-32660) Show Avro related API in documentation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180431#comment-17180431 ] Apache Spark commented on SPARK-32660: -- User 'gengliangwang' has created a pull req

[jira] [Assigned] (SPARK-32660) Show Avro related API in documentation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32660: Assignee: Gengliang Wang (was: Apache Spark) > Show Avro related API in documentation >

[jira] [Assigned] (SPARK-32660) Show Avro related API in documentation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32660: Assignee: Apache Spark (was: Gengliang Wang) > Show Avro related API in documentation >

[jira] [Commented] (SPARK-32660) Show Avro related API in documentation

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180430#comment-17180430 ] Apache Spark commented on SPARK-32660: -- User 'gengliangwang' has created a pull req

[jira] [Updated] (SPARK-32660) Show Avro related API in documentation

2020-08-19 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-32660: --- Component/s: SQL > Show Avro related API in documentation >

[jira] [Updated] (SPARK-32660) Show Avro related API in documentation

2020-08-19 Thread Gengliang Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gengliang Wang updated SPARK-32660: --- Affects Version/s: 3.1.0 > Show Avro related API in documentation >

[jira] [Created] (SPARK-32660) Show Avro related API in documentation

2020-08-19 Thread Gengliang Wang (Jira)
Gengliang Wang created SPARK-32660: -- Summary: Show Avro related API in documentation Key: SPARK-32660 URL: https://issues.apache.org/jira/browse/SPARK-32660 Project: Spark Issue Type: Sub-ta

[jira] [Commented] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180415#comment-17180415 ] Takeshi Yamamuro commented on SPARK-32638: -- Looks like a minor bug. I'll make a

[jira] [Updated] (SPARK-32659) Replace Array with Set in InSubqueryExec

2020-08-19 Thread Yuming Wang (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuming Wang updated SPARK-32659: Description: {{Set}}'s contains function has better performance: https://www.baeldung.com/java-has

[jira] [Commented] (SPARK-32640) Spark 3.1 log(NaN) returns null instead of NaN

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180410#comment-17180410 ] Wenchen Fan commented on SPARK-32640: - I quickly tried 3.0.0 {code:java} scala> sql(

[jira] [Comment Edited] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Guojian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180373#comment-17180373 ] Guojian Li edited comment on SPARK-32638 at 8/19/20, 8:56 AM:

[jira] [Commented] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Guojian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180373#comment-17180373 ] Guojian Li commented on SPARK-32638: I reproduce it on 2.4.5 .  and find  it was rep

[jira] [Commented] (SPARK-32659) Replace Array with Set in InSubqueryExec

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180368#comment-17180368 ] Apache Spark commented on SPARK-32659: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-32659) Replace Array with Set in InSubqueryExec

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32659: Assignee: (was: Apache Spark) > Replace Array with Set in InSubqueryExec > --

[jira] [Commented] (SPARK-32659) Replace Array with Set in InSubqueryExec

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180367#comment-17180367 ] Apache Spark commented on SPARK-32659: -- User 'wangyum' has created a pull request f

[jira] [Assigned] (SPARK-32659) Replace Array with Set in InSubqueryExec

2020-08-19 Thread Apache Spark (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-32659: Assignee: Apache Spark > Replace Array with Set in InSubqueryExec > -

[jira] [Updated] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Guojian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guojian Li updated SPARK-32638: --- Affects Version/s: 3.0.0 > WidenSetOperationTypes in subquery attribute missing >

[jira] [Updated] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Guojian Li (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guojian Li updated SPARK-32638: --- Affects Version/s: (was: 3.0.0) 2.4.5 > WidenSetOperationTypes in subquer

[jira] [Commented] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180365#comment-17180365 ] Lantao Jin commented on SPARK-32638: Yes. This problem exists in 3.0 and master. Th

[jira] [Resolved] (SPARK-32600) Unify task name in some logs between driver and executor

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32600. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29418 [https://gith

[jira] [Assigned] (SPARK-32600) Unify task name in some logs between driver and executor

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32600: --- Assignee: wuyi > Unify task name in some logs between driver and executor > ---

[jira] [Created] (SPARK-32659) Replace Array with Set in InSubqueryExec

2020-08-19 Thread Yuming Wang (Jira)
Yuming Wang created SPARK-32659: --- Summary: Replace Array with Set in InSubqueryExec Key: SPARK-32659 URL: https://issues.apache.org/jira/browse/SPARK-32659 Project: Spark Issue Type: Improvemen

[jira] [Updated] (SPARK-32638) WidenSetOperationTypes in subquery attribute missing

2020-08-19 Thread Lantao Jin (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lantao Jin updated SPARK-32638: --- Affects Version/s: 3.0.0 > WidenSetOperationTypes in subquery attribute missing >

[jira] [Assigned] (SPARK-32608) Script Transform DELIMIT value should be formatted

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-32608: --- Assignee: angerszhu > Script Transform DELIMIT value should be formatted > ---

[jira] [Resolved] (SPARK-32608) Script Transform DELIMIT value should be formatted

2020-08-19 Thread Wenchen Fan (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-32608. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29428 [https://gith

[jira] [Resolved] (SPARK-29767) Core dump happening on executors while doing simple union of Data Frames

2020-08-19 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-29767. -- Resolution: Cannot Reproduce > Core dump happening on executors while doing simple uni

[jira] [Commented] (SPARK-29767) Core dump happening on executors while doing simple union of Data Frames

2020-08-19 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-29767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17180353#comment-17180353 ] Takeshi Yamamuro commented on SPARK-29767: -- I'll close this. If you open this,

[jira] [Resolved] (SPARK-32632) Bad partitioning in spark jdbc method with parameter lowerBound and upperBound

2020-08-19 Thread Takeshi Yamamuro (Jira)
[ https://issues.apache.org/jira/browse/SPARK-32632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takeshi Yamamuro resolved SPARK-32632. -- Resolution: Invalid > Bad partitioning in spark jdbc method with parameter lowerBound