[jira] [Created] (SPARK-26926) Disable sparkUI

Carlos Perez created SPARK-26926: Summary: Disable sparkUI Key: SPARK-26926 URL: https://issues.apache.org/jira/browse/SPARK-26926 Project: Spark Issue Type: Bug Components: PySpark

[jira] [Commented] (SPARK-21453) Cached Kafka consumer may be closed too early

[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771714#comment-16771714 ] Jungtaek Lim commented on SPARK-21453: -- Btw, just set higher priority since this bu

[jira] [Comment Edited] (SPARK-21453) Cached Kafka consumer may be closed too early

[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771709#comment-16771709 ] Jungtaek Lim edited comment on SPARK-21453 at 2/19/19 8:39 AM: ---

[jira] [Commented] (SPARK-21453) Cached Kafka consumer may be closed too early

[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771709#comment-16771709 ] Jungtaek Lim commented on SPARK-21453: -- [~ppanero] [~Julescs0] I guess it's too lat

[jira] [Updated] (SPARK-21453) Cached Kafka consumer may be closed too early

[ https://issues.apache.org/jira/browse/SPARK-21453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jungtaek Lim updated SPARK-21453: - Priority: Major (was: Minor) > Cached Kafka consumer may be closed too early >

[jira] [Created] (SPARK-26927) Race condition may cause dynamic allocation not working

liupengcheng created SPARK-26927: Summary: Race condition may cause dynamic allocation not working Key: SPARK-26927 URL: https://issues.apache.org/jira/browse/SPARK-26927 Project: Spark Issue

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Attachment: Selection_042.jpg > Race condition may cause dynamic allocation not working > --

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Attachment: Selection_043.jpg > Race condition may cause dynamic allocation not working > --

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Attachment: Selection_044.jpg > Race condition may cause dynamic allocation not working > --

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Attachment: Selection_045.jpg > Race condition may cause dynamic allocation not working > --

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Description: Recently, we catch a bug that caused our production spark thriftserver hangs: Ther

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Attachment: Selection_046.jpg > Race condition may cause dynamic allocation not working > --

[jira] [Created] (SPARK-26928) Add driver CPU Time to the metrics system

Luca Canali created SPARK-26928: --- Summary: Add driver CPU Time to the metrics system Key: SPARK-26928 URL: https://issues.apache.org/jira/browse/SPARK-26928 Project: Spark Issue Type: Improveme

[jira] [Created] (SPARK-26929) table owner should use user instead of principal when use kerberos

hong dongdong created SPARK-26929: - Summary: table owner should use user instead of principal when use kerberos Key: SPARK-26929 URL: https://issues.apache.org/jira/browse/SPARK-26929 Project: Spark

[jira] [Updated] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated SPARK-26929: -- Description: In kerberos cluster, when use spark-sql or beeline to create table,  the owner w

[jira] [Updated] (SPARK-26927) Race condition may cause dynamic allocation not working

[ https://issues.apache.org/jira/browse/SPARK-26927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liupengcheng updated SPARK-26927: - Description: Recently, we catch a bug that caused our production spark thriftserver hangs: Ther

[jira] [Updated] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated SPARK-26929: -- External issue URL: https://github.com/apache/spark/pull/23837 > table owner should use user i

[jira] [Assigned] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26929: Assignee: (was: Apache Spark) > table owner should use user instead of principal when

[jira] [Assigned] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26929: Assignee: Apache Spark > table owner should use user instead of principal when use kerber

[jira] [Updated] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hong dongdong updated SPARK-26929: -- Fix Version/s: 2.3.0 > table owner should use user instead of principal when use kerberos > --

[jira] [Assigned] (SPARK-26928) Add driver CPU Time to the metrics system

[ https://issues.apache.org/jira/browse/SPARK-26928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26928: Assignee: (was: Apache Spark) > Add driver CPU Time to the metrics system > -

[jira] [Assigned] (SPARK-26928) Add driver CPU Time to the metrics system

[ https://issues.apache.org/jira/browse/SPARK-26928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26928: Assignee: Apache Spark > Add driver CPU Time to the metrics system >

[jira] [Commented] (SPARK-26686) Remove unnecessary KafkaSourceProvider parameter lowercase conversion

[ https://issues.apache.org/jira/browse/SPARK-26686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771805#comment-16771805 ] Gabor Somogyi commented on SPARK-26686: --- As the API is not explicitly stating that

[jira] [Resolved] (SPARK-26686) Remove unnecessary KafkaSourceProvider parameter lowercase conversion

[ https://issues.apache.org/jira/browse/SPARK-26686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi resolved SPARK-26686. --- Resolution: Won't Do > Remove unnecessary KafkaSourceProvider parameter lowercase conversion

[jira] [Created] (SPARK-26930) Several test cases in ParquetFilterSuite are broken

Nandor Kollar created SPARK-26930: - Summary: Several test cases in ParquetFilterSuite are broken Key: SPARK-26930 URL: https://issues.apache.org/jira/browse/SPARK-26930 Project: Spark Issue T

[jira] [Created] (SPARK-26931) Hive orcCompatibility

Bo Hai created SPARK-26931: -- Summary: Hive orcCompatibility Key: SPARK-26931 URL: https://issues.apache.org/jira/browse/SPARK-26931 Project: Spark Issue Type: Documentation Components: Doc

[jira] [Created] (SPARK-26932) Orc compatibility between hive and spark

Bo Hai created SPARK-26932: -- Summary: Orc compatibility between hive and spark Key: SPARK-26932 URL: https://issues.apache.org/jira/browse/SPARK-26932 Project: Spark Issue Type: Documentation

[jira] [Commented] (SPARK-26146) CSV wouln't be ingested in Spark 2.4.0 with Scala 2.12

[ https://issues.apache.org/jira/browse/SPARK-26146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772054#comment-16772054 ] Michael Heuer commented on SPARK-26146: --- Another possible reproducing case [https

[jira] [Commented] (SPARK-26395) Spark Thrift server memory leak

[ https://issues.apache.org/jira/browse/SPARK-26395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772058#comment-16772058 ] Konstantinos Andrikopoulos commented on SPARK-26395: After setting t

[jira] [Commented] (SPARK-26777) SQL worked in 2.3.2 and fails in 2.4.0

[ https://issues.apache.org/jira/browse/SPARK-26777?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772071#comment-16772071 ] Ilya Peysakhov commented on SPARK-26777: [~hyukjin.kwon]  It's the same error/i

[jira] [Updated] (SPARK-26873) FileFormatWriter creates inconsistent MR job IDs

[ https://issues.apache.org/jira/browse/SPARK-26873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-26873: --- Fix Version/s: 2.3.4 > FileFormatWriter creates inconsistent MR job IDs > --

[jira] [Assigned] (SPARK-26788) Remove SchedulerExtensionService

[ https://issues.apache.org/jira/browse/SPARK-26788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26788: Assignee: (was: Apache Spark) > Remove SchedulerExtensionService > --

[jira] [Assigned] (SPARK-26788) Remove SchedulerExtensionService

[ https://issues.apache.org/jira/browse/SPARK-26788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26788: Assignee: Apache Spark > Remove SchedulerExtensionService > -

[jira] [Commented] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772266#comment-16772266 ] Bryan Cutler commented on SPARK-26858: -- [~hyukjin.kwon] actually {{pyarrow.Table.fr

[jira] [Reopened] (SPARK-20977) NPE in CollectionAccumulator

[ https://issues.apache.org/jira/browse/SPARK-20977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reopened SPARK-20977: -- Reopening this issue as I believe I understand the cause. An accumulator is escaped before it's f

[jira] [Commented] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772296#comment-16772296 ] Jungtaek Lim commented on SPARK-24295: -- Please correct me if I'm missing here. I ju

[jira] [Commented] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772322#comment-16772322 ] Jungtaek Lim commented on SPARK-24295: -- FileStreamSinkLog cannot be removed even we

[jira] [Commented] (SPARK-24736) --py-files not functional for non local URLs. It appears to pass non-local URL's into PYTHONPATH directly.

[ https://issues.apache.org/jira/browse/SPARK-24736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772337#comment-16772337 ] Marcelo Vanzin commented on SPARK-24736: I'm going to fork this issue into 2:

[jira] [Created] (SPARK-26933) spark-submit does not make zip files provided with --py-files visible to pyspark

Marcelo Vanzin created SPARK-26933: -- Summary: spark-submit does not make zip files provided with --py-files visible to pyspark Key: SPARK-26933 URL: https://issues.apache.org/jira/browse/SPARK-26933

[jira] [Created] (SPARK-26934) python dependencies with "local:" URIs are not visible to executors

Marcelo Vanzin created SPARK-26934: -- Summary: python dependencies with "local:" URIs are not visible to executors Key: SPARK-26934 URL: https://issues.apache.org/jira/browse/SPARK-26934 Project: Spar

[jira] [Commented] (SPARK-20971) Purge the metadata log for FileStreamSource

[ https://issues.apache.org/jira/browse/SPARK-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772351#comment-16772351 ] Jungtaek Lim commented on SPARK-20971: -- I guess this can be also handled in SPARK-2

[jira] [Resolved] (SPARK-26891) Flaky test:YarnSchedulerBackendSuite."RequestExecutors reflects node blacklist and is serializable"

[ https://issues.apache.org/jira/browse/SPARK-26891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26891. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23801 [https:

[jira] [Assigned] (SPARK-26891) Flaky test:YarnSchedulerBackendSuite."RequestExecutors reflects node blacklist and is serializable"

[ https://issues.apache.org/jira/browse/SPARK-26891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26891: -- Assignee: Attila Zsolt Piros > Flaky test:YarnSchedulerBackendSuite."RequestExecutors

[jira] [Resolved] (SPARK-26882) lint-scala script does not check all components

[ https://issues.apache.org/jira/browse/SPARK-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26882. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23792 [https:

[jira] [Created] (SPARK-26935) Skip DataFrameReader's CSV first line scan when not used

Douglas Colkitt created SPARK-26935: --- Summary: Skip DataFrameReader's CSV first line scan when not used Key: SPARK-26935 URL: https://issues.apache.org/jira/browse/SPARK-26935 Project: Spark

[jira] [Assigned] (SPARK-26935) Skip DataFrameReader's CSV first line scan when not used

[ https://issues.apache.org/jira/browse/SPARK-26935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26935: Assignee: (was: Apache Spark) > Skip DataFrameReader's CSV first line scan when not u

[jira] [Commented] (SPARK-26935) Skip DataFrameReader's CSV first line scan when not used

[ https://issues.apache.org/jira/browse/SPARK-26935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772394#comment-16772394 ] Apache Spark commented on SPARK-26935: -- User 'Mister-Meeseeks' has created a pull r

[jira] [Assigned] (SPARK-26935) Skip DataFrameReader's CSV first line scan when not used

[ https://issues.apache.org/jira/browse/SPARK-26935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26935: Assignee: Apache Spark > Skip DataFrameReader's CSV first line scan when not used > -

[jira] [Commented] (SPARK-26655) Support multiple aggregates in Structured Streaming append mode

[ https://issues.apache.org/jira/browse/SPARK-26655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772415#comment-16772415 ] Arun Mahadevan commented on SPARK-26655: Design doc link: [https://docs.google.

[jira] [Updated] (SPARK-26655) Support multiple aggregates in Structured Streaming append mode

[ https://issues.apache.org/jira/browse/SPARK-26655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Mahadevan updated SPARK-26655: --- Attachment: Watermarks and multiple aggregates in Spark strucutred streaming_v1.pdf > Suppo

[jira] [Resolved] (SPARK-24894) Invalid DNS name due to hostname truncation

[ https://issues.apache.org/jira/browse/SPARK-24894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-24894. Resolution: Fixed Assignee: Marcelo Vanzin Fix Version/s: 3.0.0 > Invalid

[jira] [Resolved] (SPARK-26933) spark-submit does not make zip files provided with --py-files visible to pyspark

[ https://issues.apache.org/jira/browse/SPARK-26933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26933. Resolution: Not A Problem Did something wrong when testing this in the morning. This is wo

[jira] [Commented] (SPARK-23682) Memory issue with Spark structured streaming

[ https://issues.apache.org/jira/browse/SPARK-23682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772471#comment-16772471 ] Yuriy Bondaruk commented on SPARK-23682: [~kabhwan] Unfortunately currently I do

[jira] [Resolved] (SPARK-26931) Hive orcCompatibility

[ https://issues.apache.org/jira/browse/SPARK-26931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Hai resolved SPARK-26931. Resolution: Duplicate > Hive orcCompatibility > - > > Key: SPARK-26931

[jira] [Updated] (SPARK-26643) Spark hive throwing an incorrect analysis exception,when set table properties.

[ https://issues.apache.org/jira/browse/SPARK-26643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiaan.geng updated SPARK-26643: --- Description: When I execute a DDL in spark-sql,throwing a AnalysisException as follows: {code:java}

[jira] [Created] (SPARK-26936) insert overwrite local directory can not create temporary path in local staging directory

jiaan.geng created SPARK-26936: -- Summary: insert overwrite local directory can not create temporary path in local staging directory Key: SPARK-26936 URL: https://issues.apache.org/jira/browse/SPARK-26936

[jira] [Created] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

Xu Jiang created SPARK-26937: Summary: Build Spark 2.4 Support Hadoop-3.1 faild Key: SPARK-26937 URL: https://issues.apache.org/jira/browse/SPARK-26937 Project: Spark Issue Type: Bug Co

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Jiang updated SPARK-26937: - Environment:   Hi, my environmental information is as follows： h2. Operating System CentOS Linux relea

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Jiang updated SPARK-26937: - Environment: h2.   Hi, my environmental information is as follows： h2. Operating System   {code:java}

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Jiang updated SPARK-26937: - Description:  The build command I am running is： {{}} {code:java} ./dev/make-distribution.sh --name jdp

[jira] [Assigned] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24295: Assignee: Apache Spark > Purge Structured streaming FileStreamSinkLog metadata compact fi

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Jiang updated SPARK-26937: - Environment: h2.  Hi, my environmental information is as follows： Operating System  {code:java} CentOS

[jira] [Assigned] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-24295: Assignee: (was: Apache Spark) > Purge Structured streaming FileStreamSinkLog metadata

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772570#comment-16772570 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/20/19 2:56 AM: ---

[jira] [Commented] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772570#comment-16772570 ] Hyukjin Kwon commented on SPARK-26858: -- Yes .. that's matched with what I was think

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Jiang updated SPARK-26937: - Description:  The build command I am running is： {code:java} ./dev/make-distribution.sh --name jdp-spark

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Jiang updated SPARK-26937: - Fix Version/s: (was: 2.4.1) > Build Spark 2.4 Support Hadoop-3.1 faild > ---

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772570#comment-16772570 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/20/19 3:00 AM: ---

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772570#comment-16772570 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/20/19 3:01 AM: ---

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772570#comment-16772570 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/20/19 3:02 AM: ---

[jira] [Commented] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772578#comment-16772578 ] Hyukjin Kwon commented on SPARK-26858: -- {quote} Back on your sequence diagram at st

[jira] [Commented] (SPARK-13446) Spark need to support reading data from Hive 2.0.0 metastore

[ https://issues.apache.org/jira/browse/SPARK-13446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772584#comment-16772584 ] Lewin Ma commented on SPARK-13446: -- I got the same problem as [~elgalu] > Spark need t

[jira] [Assigned] (SPARK-26936) insert overwrite local directory can not create temporary path in local staging directory

[ https://issues.apache.org/jira/browse/SPARK-26936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26936: Assignee: (was: Apache Spark) > insert overwrite local directory can not create tempo

[jira] [Assigned] (SPARK-26936) insert overwrite local directory can not create temporary path in local staging directory

[ https://issues.apache.org/jira/browse/SPARK-26936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26936: Assignee: Apache Spark > insert overwrite local directory can not create temporary path i

[jira] [Updated] (SPARK-26759) Arrow optimization in SparkR's interoperability

[ https://issues.apache.org/jira/browse/SPARK-26759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26759: - Description: Arrow 0.12.0 is release and it contains R API. We could optimize Spark DaraFrame <

[jira] [Commented] (SPARK-24295) Purge Structured streaming FileStreamSinkLog metadata compact file data.

[ https://issues.apache.org/jira/browse/SPARK-24295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772588#comment-16772588 ] Jungtaek Lim commented on SPARK-24295: -- While I submitted the PR to reflect the las

[jira] [Comment Edited] (SPARK-26858) Vectorized gapplyCollect, Arrow optimization in native R function execution

[ https://issues.apache.org/jira/browse/SPARK-26858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16766941#comment-16766941 ] Hyukjin Kwon edited comment on SPARK-26858 at 2/20/19 3:27 AM: ---

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xu Jiang updated SPARK-26937: - Target Version/s: 2.4.0 (was: 2.4.1) > Build Spark 2.4 Support Hadoop-3.1 faild > -

[jira] [Resolved] (SPARK-26762) Arrow optimization for conversion from Spark DataFrame to R DataFrame

[ https://issues.apache.org/jira/browse/SPARK-26762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26762. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23760 [https://gi

[jira] [Resolved] (SPARK-26916) Upgrade to Kafka 2.1.1

[ https://issues.apache.org/jira/browse/SPARK-26916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26916. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via https://github.com/apache

[jira] [Commented] (SPARK-20971) Purge the metadata log for FileStreamSource

[ https://issues.apache.org/jira/browse/SPARK-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772632#comment-16772632 ] Jungtaek Lim commented on SPARK-20971: -- Maybe better to clarify what we would like

[jira] [Comment Edited] (SPARK-20971) Purge the metadata log for FileStreamSource

[ https://issues.apache.org/jira/browse/SPARK-20971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772632#comment-16772632 ] Jungtaek Lim edited comment on SPARK-20971 at 2/20/19 5:19 AM: ---

[jira] [Updated] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26929: - Fix Version/s: (was: 2.3.0) > table owner should use user instead of principal when use kerb

[jira] [Updated] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26929: - External issue URL: (was: https://github.com/apache/spark/pull/23837) > table owner should use

[jira] [Updated] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26929: - Description: In kerberos cluster, when use spark-sql or beeline to create table,  the owner wil

[jira] [Commented] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772650#comment-16772650 ] Dongjoon Hyun commented on SPARK-26937: --- Thank you for reporting, [~xujiang]. BTW

[jira] [Updated] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26937: -- Target Version/s: (was: 2.4.0) > Build Spark 2.4 Support Hadoop-3.1 faild >

[jira] [Closed] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26937. - > Build Spark 2.4 Support Hadoop-3.1 faild > > >

[jira] [Resolved] (SPARK-26937) Build Spark 2.4 Support Hadoop-3.1 faild

[ https://issues.apache.org/jira/browse/SPARK-26937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-26937. --- Resolution: Duplicate > Build Spark 2.4 Support Hadoop-3.1 faild > -

[jira] [Updated] (SPARK-26932) Orc compatibility between hive and spark

[ https://issues.apache.org/jira/browse/SPARK-26932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26932: -- Labels: (was: build) > Orc compatibility between hive and spark > --

[jira] [Updated] (SPARK-26931) Hive orcCompatibility

[ https://issues.apache.org/jira/browse/SPARK-26931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26931: -- Target Version/s: (was: 2.4.0) > Hive orcCompatibility > - > >

[jira] [Closed] (SPARK-26931) Hive orcCompatibility

[ https://issues.apache.org/jira/browse/SPARK-26931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-26931. - > Hive orcCompatibility > - > > Key: SPARK-26931 >

[jira] [Updated] (SPARK-26932) Orc compatibility between hive and spark

[ https://issues.apache.org/jira/browse/SPARK-26932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26932: -- Target Version/s: (was: 2.4.0) > Orc compatibility between hive and spark >

[jira] [Updated] (SPARK-26929) table owner should use user instead of principal when use kerberos

[ https://issues.apache.org/jira/browse/SPARK-26929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-26929: -- Description: In kerberos cluster, when use spark-sql or beeline to create table,  the owner w

[jira] [Commented] (SPARK-26932) Orc compatibility between hive and spark

[ https://issues.apache.org/jira/browse/SPARK-26932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772652#comment-16772652 ] Dongjoon Hyun commented on SPARK-26932: --- Hi, [~haiboself]. Could you link the corr

[jira] [Resolved] (SPARK-26925) how to get the statistics when read from or writer to another database by datasourceV2

[ https://issues.apache.org/jira/browse/SPARK-26925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26925. -- Resolution: Invalid > how to get the statistics when read from or writer to another database

[jira] [Commented] (SPARK-26925) how to get the statistics when read from or writer to another database by datasourceV2

[ https://issues.apache.org/jira/browse/SPARK-26925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772666#comment-16772666 ] Hyukjin Kwon commented on SPARK-26925: -- Please ask it to dev mailing list for a que

[jira] [Commented] (SPARK-26926) Disable sparkUI

[ https://issues.apache.org/jira/browse/SPARK-26926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772668#comment-16772668 ] Hyukjin Kwon commented on SPARK-26926: -- {{pyspark}} does not support any applicatio

[jira] [Resolved] (SPARK-26926) Disable sparkUI

[ https://issues.apache.org/jira/browse/SPARK-26926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-26926. -- Resolution: Cannot Reproduce > Disable sparkUI > --- > > Key: SPAR

[jira] [Created] (SPARK-26938) failed task's SparkListenerTaskEnd lost taskMetrics, which is important for problem diagnosis, especially when checking exists of data skew

weiwenda created SPARK-26938: Summary: failed task's SparkListenerTaskEnd lost taskMetrics, which is important for problem diagnosis, especially when checking exists of data skew Key: SPARK-26938 URL: https://issues.

  1   2   >