[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16785277#comment-16785277 ] Jungtaek Lim commented on SPARK-26998: -- [~toopt4] Yeah I tend to agree that hiding more credential

[jira] [Created] (SPARK-27069) Spark(2.3.1) LDA transfomation memory error(java.lang.OutOfMemoryError at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123)

2019-03-05 Thread TAESUK KIM (JIRA)
TAESUK KIM created SPARK-27069: -- Summary: Spark(2.3.1) LDA transfomation memory error(java.lang.OutOfMemoryError at java.io.ByteArrayOutputStream.hugeCapacity(ByteArrayOutputStream.java:123) Key: SPARK-27069 URL:

[jira] [Created] (SPARK-27056) Remove `start-shuffle-service.sh`

2019-03-05 Thread liuxian (JIRA)
liuxian created SPARK-27056: --- Summary: Remove `start-shuffle-service.sh` Key: SPARK-27056 URL: https://issues.apache.org/jira/browse/SPARK-27056 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-23339) Spark UI not loading *.js/*.css files, only raw HTML

2019-03-05 Thread gary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784204#comment-16784204 ] gary commented on SPARK-23339: -- exclude servlet-api-2.5.jar, use servlet-api-3.1.0.jar. It works for me. >

[jira] [Created] (SPARK-27055) Update Structured Streaming documentation because of DSv2 changes

2019-03-05 Thread Gabor Somogyi (JIRA)
Gabor Somogyi created SPARK-27055: - Summary: Update Structured Streaming documentation because of DSv2 changes Key: SPARK-27055 URL: https://issues.apache.org/jira/browse/SPARK-27055 Project: Spark

[jira] [Assigned] (SPARK-27056) Remove `start-shuffle-service.sh`

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27056: Assignee: (was: Apache Spark) > Remove `start-shuffle-service.sh` >

[jira] [Assigned] (SPARK-27056) Remove `start-shuffle-service.sh`

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27056: Assignee: Apache Spark > Remove `start-shuffle-service.sh` >

[jira] [Comment Edited] (SPARK-27019) Spark UI's SQL tab shows inconsistent values

2019-03-05 Thread Shahid K I (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784194#comment-16784194 ] Shahid K I edited comment on SPARK-27019 at 3/5/19 7:59 AM: Please upload

[jira] [Commented] (SPARK-20415) SPARK job hangs while writing DataFrame to HDFS

2019-03-05 Thread Martin Studer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784200#comment-16784200 ] Martin Studer commented on SPARK-20415: --- I'm observing a similar issue where all executor tasks

[jira] [Commented] (SPARK-27019) Spark UI's SQL tab shows inconsistent values

2019-03-05 Thread peay (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784260#comment-16784260 ] peay commented on SPARK-27019: -- Yes, I had edited my message above shortly after posting - cannot reproduce 

[jira] [Resolved] (SPARK-26850) Make EventLoggingListener LOG_FILE_PERMISSIONS configurable

2019-03-05 Thread Sandeep Katta (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sandeep Katta resolved SPARK-26850. --- Resolution: Duplicate > Make EventLoggingListener LOG_FILE_PERMISSIONS configurable >

[jira] [Updated] (SPARK-27056) Remove `start-shuffle-service.sh`

2019-03-05 Thread liuxian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27056?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liuxian updated SPARK-27056: Description: _start-shuffle-service.sh_ was only used by Mesos before _start-mesos-shuffle-service.sh_.

[jira] [Updated] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26602: - Attachment: beforeFixUdf.txt > Insert into table fails after querying the UDF which is loaded

[jira] [Updated] (SPARK-27055) Update Structured Streaming documentation because of DSv2 changes

2019-03-05 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gabor Somogyi updated SPARK-27055: -- Description: Since SPARK-26956 has been merged the Structured Streaming documentation has to

[jira] [Created] (SPARK-27057) Common trait for limit exec operators

2019-03-05 Thread Maxim Gekk (JIRA)
Maxim Gekk created SPARK-27057: -- Summary: Common trait for limit exec operators Key: SPARK-27057 URL: https://issues.apache.org/jira/browse/SPARK-27057 Project: Spark Issue Type: Improvement

[jira] [Created] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Andreas Adamides (JIRA)
Andreas Adamides created SPARK-27059: Summary: spark-submit on kubernetes cluster does not recognise k8s --master property Key: SPARK-27059 URL: https://issues.apache.org/jira/browse/SPARK-27059

[jira] [Comment Edited] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784134#comment-16784134 ] Chakravarthi edited comment on SPARK-26602 at 3/5/19 2:31 PM: -- Hi [~srowen]

[jira] [Commented] (SPARK-27060) DDL Commands are accepting Keywords like create, drop as tableName

2019-03-05 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784503#comment-16784503 ] Sujith Chacko commented on SPARK-27060: --- This looks like a compatibility issue with other engines.

[jira] [Commented] (SPARK-27058) Support mounting host dirs for K8s tests

2019-03-05 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784396#comment-16784396 ] Stavros Kontopoulos commented on SPARK-27058: - [~shaneknapp] added this to keep track of

[jira] [Created] (SPARK-27058) Support mounting host dirs for K8s tests

2019-03-05 Thread Stavros Kontopoulos (JIRA)
Stavros Kontopoulos created SPARK-27058: --- Summary: Support mounting host dirs for K8s tests Key: SPARK-27058 URL: https://issues.apache.org/jira/browse/SPARK-27058 Project: Spark Issue

[jira] [Updated] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Andreas Adamides (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andreas Adamides updated SPARK-27059: - Description: I have successfully installed a Kubernetes cluster and can verify this by:

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread Jungtaek Lim (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784434#comment-16784434 ] Jungtaek Lim commented on SPARK-26998: -- If I understand correctly, the PR would mitigate the issue

[jira] [Commented] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784446#comment-16784446 ] Sean Owen commented on SPARK-26602: --- That sounds like user error. I'd close this as NotAProblem. It

[jira] [Commented] (SPARK-23986) CompileException when using too many avg aggregation after joining

2019-03-05 Thread Pedro Fernandes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784482#comment-16784482 ] Pedro Fernandes commented on SPARK-23986: - Guys, Is there a workaround for the folks that can't 

[jira] [Commented] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784506#comment-16784506 ] Ajith S commented on SPARK-26602: - [~chakravarthi] Hi, thanks for reporting the issue, From your example

[jira] [Updated] (SPARK-26923) Refactor ArrowRRunner and RRunner to share the same base

2019-03-05 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-26923: - Summary: Refactor ArrowRRunner and RRunner to share the same base (was: Refactor ArrowRRunner

[jira] [Comment Edited] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784391#comment-16784391 ] Gabor Somogyi edited comment on SPARK-26998 at 3/5/19 12:51 PM:

[jira] [Assigned] (SPARK-27057) Common trait for limit exec operators

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27057: Assignee: Apache Spark > Common trait for limit exec operators >

[jira] [Assigned] (SPARK-27057) Common trait for limit exec operators

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27057: Assignee: (was: Apache Spark) > Common trait for limit exec operators >

[jira] [Assigned] (SPARK-26923) Refactor ArrowRRunner and RRunner to share the same base

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26923: Assignee: Apache Spark > Refactor ArrowRRunner and RRunner to share the same base >

[jira] [Commented] (SPARK-20415) SPARK job hangs while writing DataFrame to HDFS

2019-03-05 Thread Martin Studer (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784475#comment-16784475 ] Martin Studer commented on SPARK-20415: --- In fact, I believe the underlying issue is actually

[jira] [Comment Edited] (SPARK-27060) DDL Commands are accepting Keywords like create, drop as tableName

2019-03-05 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784503#comment-16784503 ] Sujith Chacko edited comment on SPARK-27060 at 3/5/19 3:11 PM: --- This looks

[jira] [Updated] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Andreas Adamides (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andreas Adamides updated SPARK-27059: - Description: I have successfully installed a Kubernetes cluster and can verify this by:

[jira] [Updated] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Andreas Adamides (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andreas Adamides updated SPARK-27059: - Description: I have successfully installed a Kubernetes cluster and can verify this by:

[jira] [Commented] (SPARK-26972) Issue with CSV import and inferSchema set to true

2019-03-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784453#comment-16784453 ] Sean Owen commented on SPARK-26972: --- I haven't checked when it's case-insensitive, but to be clear:

[jira] [Commented] (SPARK-27060) DDL Commands are accepting Keywords like create, drop as tableName

2019-03-05 Thread Sachin Ramachandra Setty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784560#comment-16784560 ] Sachin Ramachandra Setty commented on SPARK-27060: -- cc [~srowen] > DDL Commands are

[jira] [Assigned] (SPARK-26923) Refactor ArrowRRunner and RRunner to share the same base

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-26923: Assignee: (was: Apache Spark) > Refactor ArrowRRunner and RRunner to share the same

[jira] [Updated] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Andreas Adamides (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andreas Adamides updated SPARK-27059: - Description: I have successfully installed a Kubernetes cluster and can verify this by:

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784391#comment-16784391 ] Gabor Somogyi commented on SPARK-26998: --- [~toopt4] thanks for the info. Are you working on this?

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784406#comment-16784406 ] Gabor Somogyi commented on SPARK-26998: --- {quote} Can be resolved if below PR is merged: [[Github]

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread Gabor Somogyi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784410#comment-16784410 ] Gabor Somogyi commented on SPARK-26998: --- Ahaaa, I see now. 2 problems tried to be solved in one

[jira] [Updated] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Andreas Adamides (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andreas Adamides updated SPARK-27059: - Description: I have successfully installed a Kubernetes cluster and can verify this by:

[jira] [Comment Edited] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784134#comment-16784134 ] Chakravarthi edited comment on SPARK-26602 at 3/5/19 1:52 PM: -- Hi [~srowen]

[jira] [Created] (SPARK-27060) DDL Commands are accepting Keywords like create, drop as tableName

2019-03-05 Thread Sachin Ramachandra Setty (JIRA)
Sachin Ramachandra Setty created SPARK-27060: Summary: DDL Commands are accepting Keywords like create, drop as tableName Key: SPARK-27060 URL: https://issues.apache.org/jira/browse/SPARK-27060

[jira] [Commented] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784526#comment-16784526 ] Chakravarthi commented on SPARK-26602: -- [~srowen] agree,but it should not make any other subsequent

[jira] [Commented] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784528#comment-16784528 ] Sean Owen commented on SPARK-26602: --- If a user adds something to the classpath, it matters to the

[jira] [Commented] (SPARK-27005) Design sketch: Accelerator-aware scheduling

2019-03-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784555#comment-16784555 ] Thomas Graves commented on SPARK-27005: --- so we have both a google design doc and the comment

[jira] [Updated] (SPARK-26602) Subsequent queries are failing after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Chakravarthi (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chakravarthi updated SPARK-26602: - Summary: Subsequent queries are failing after querying the UDF which is loaded with wrong hdfs

[jira] [Updated] (SPARK-23521) SPIP: Standardize SQL logical plans with DataSourceV2

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-23521: -- Attachment: SPIP_ Standardize logical plans.pdf > SPIP: Standardize SQL logical plans with

[jira] [Commented] (SPARK-23521) SPIP: Standardize SQL logical plans with DataSourceV2

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784736#comment-16784736 ] Ryan Blue commented on SPARK-23521: --- I've turned off commenting on the google doc to preserve its

[jira] [Resolved] (SPARK-27067) SPIP: Catalog API for table metadata

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved SPARK-27067. --- Resolution: Fixed I'm resolving this issue because the vote to adopt the proposal passed. I've

[jira] [Resolved] (SPARK-27012) Storage tab shows rdd details even after executor ended

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27012. Resolution: Fixed Assignee: Ajith S Fix Version/s: 3.0.0 > Storage tab

[jira] [Commented] (SPARK-13091) Rewrite/Propagate constraints for Aliases

2019-03-05 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784776#comment-16784776 ] Ajith S commented on SPARK-13091: - can this document be made accessible.? 

[jira] [Commented] (SPARK-26998) spark.ssl.keyStorePassword in plaintext on 'ps -ef' output of executor processes in Standalone mode

2019-03-05 Thread t oo (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784791#comment-16784791 ] t oo commented on SPARK-26998: -- [~gsomogyi] please take it forward. [~kabhwan] truststore password being

[jira] [Comment Edited] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-05 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784714#comment-16784714 ] Stavros Kontopoulos edited comment on SPARK-27063 at 3/5/19 5:52 PM: -

[jira] [Commented] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-05 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784714#comment-16784714 ] Stavros Kontopoulos commented on SPARK-27063: - Yes some other things that I noticed is when

[jira] [Updated] (SPARK-27067) SPIP: Catalog API for table metadata

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-27067: -- Attachment: SPIP_ Spark API for Table Metadata.pdf > SPIP: Catalog API for table metadata >

[jira] [Resolved] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-27059. Resolution: Invalid > spark-submit on kubernetes cluster does not recognise k8s --master

[jira] [Commented] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784762#comment-16784762 ] Marcelo Vanzin commented on SPARK-27059: Sounds like a problem with your system. Maybe your PATH

[jira] [Commented] (SPARK-27059) spark-submit on kubernetes cluster does not recognise k8s --master property

2019-03-05 Thread Andreas Adamides (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784758#comment-16784758 ] Andreas Adamides commented on SPARK-27059: -- Indeed, when in spark 2.4.0 and 2.3.3 running

[jira] [Resolved] (SPARK-26928) Add driver CPU Time to the metrics system

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin resolved SPARK-26928. Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 23838

[jira] [Assigned] (SPARK-26928) Add driver CPU Time to the metrics system

2019-03-05 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reassigned SPARK-26928: -- Assignee: Luca Canali > Add driver CPU Time to the metrics system >

[jira] [Resolved] (SPARK-27043) Nested schema pruning benchmark for ORC

2019-03-05 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27043. --- Resolution: Fixed Assignee: Liang-Chi Hsieh Fix Version/s: 3.0.0 This is

[jira] [Comment Edited] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-05 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784714#comment-16784714 ] Stavros Kontopoulos edited comment on SPARK-27063 at 3/5/19 5:53 PM: -

[jira] [Updated] (SPARK-27066) SPIP: Identifiers for multi-catalog support

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-27066: -- Attachment: SPIP_ Identifiers for multi-catalog Spark.pdf > SPIP: Identifiers for multi-catalog

[jira] [Created] (SPARK-27067) SPIP: Catalog API for table metadata

2019-03-05 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-27067: - Summary: SPIP: Catalog API for table metadata Key: SPARK-27067 URL: https://issues.apache.org/jira/browse/SPARK-27067 Project: Spark Issue Type: Improvement

[jira] [Resolved] (SPARK-27066) SPIP: Identifiers for multi-catalog support

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue resolved SPARK-27066. --- Resolution: Fixed I'm resolving this issue because the vote to adopt the proposal passed. I've

[jira] [Created] (SPARK-27066) SPIP: Identifiers for multi-catalog support

2019-03-05 Thread Ryan Blue (JIRA)
Ryan Blue created SPARK-27066: - Summary: SPIP: Identifiers for multi-catalog support Key: SPARK-27066 URL: https://issues.apache.org/jira/browse/SPARK-27066 Project: Spark Issue Type:

[jira] [Updated] (SPARK-27067) SPIP: Catalog API for table metadata

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-27067: -- Description: Goal: Define a catalog API to create, alter, load, and drop tables > SPIP: Catalog API

[jira] [Updated] (SPARK-27066) SPIP: Identifiers for multi-catalog support

2019-03-05 Thread Ryan Blue (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Blue updated SPARK-27066: -- Description: Goals: * Propose semantics for identifiers and a listing API to support multiple

[jira] [Updated] (SPARK-27062) Refresh Table command register table with table name only

2019-03-05 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27062: - Priority: Minor (was: Major) > Refresh Table command register table with table name only >

[jira] [Updated] (SPARK-27062) CatalogImpl.refreshTable should register query in cache with received tableName

2019-03-05 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27062: - Description: If CatalogImpl.refreshTable() method is invoked against a cached table, this

[jira] [Assigned] (SPARK-27065) avoid more than one active task set managers for a stage

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27065: Assignee: Apache Spark (was: Wenchen Fan) > avoid more than one active task set

[jira] [Assigned] (SPARK-27065) avoid more than one active task set managers for a stage

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27065: Assignee: Wenchen Fan (was: Apache Spark) > avoid more than one active task set

[jira] [Updated] (SPARK-27062) CatalogImpl.refreshTable should register query in cache with received tableName

2019-03-05 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27062: - Description: If _CatalogImpl.refreshTable()_ method is invoked against a cached table, this

[jira] [Commented] (SPARK-27065) avoid more than one active task set managers for a stage

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784654#comment-16784654 ] Apache Spark commented on SPARK-27065: -- User 'cloud-fan' has created a pull request for this issue:

[jira] [Comment Edited] (SPARK-27005) Design sketch: Accelerator-aware scheduling

2019-03-05 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784555#comment-16784555 ] Thomas Graves edited comment on SPARK-27005 at 3/5/19 3:40 PM: --- so we have

[jira] [Comment Edited] (SPARK-27060) DDL Commands are accepting Keywords like create, drop as tableName

2019-03-05 Thread Sachin Ramachandra Setty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784575#comment-16784575 ] Sachin Ramachandra Setty edited comment on SPARK-27060 at 3/5/19 3:40 PM:

[jira] [Comment Edited] (SPARK-27036) Even Broadcast thread is timed out, BroadCast Job is not aborted.

2019-03-05 Thread Sujith Chacko (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16782840#comment-16782840 ] Sujith Chacko edited comment on SPARK-27036 at 3/5/19 3:49 PM: --- It seems

[jira] [Updated] (SPARK-27062) Refresh Table command register table with table name only

2019-03-05 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27062: - Labels: easyfix pull-request-available (was: easyfix) > Refresh Table command register table

[jira] [Assigned] (SPARK-27062) Refresh Table command register table with table name only

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27062: Assignee: (was: Apache Spark) > Refresh Table command register table with table name

[jira] [Assigned] (SPARK-27062) Refresh Table command register table with table name only

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27062: Assignee: Apache Spark > Refresh Table command register table with table name only >

[jira] [Commented] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784611#comment-16784611 ] Ajith S commented on SPARK-26602: - # I have a question about this issue in thrift-server case. If admin

[jira] [Assigned] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27063: Assignee: (was: Apache Spark) > Spark on K8S Integration Tests timeouts are too

[jira] [Assigned] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27063?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27063: Assignee: Apache Spark > Spark on K8S Integration Tests timeouts are too short for some

[jira] [Assigned] (SPARK-27064) create StreamingWrite at the begining of streaming execution

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27064: Assignee: Apache Spark (was: Wenchen Fan) > create StreamingWrite at the begining of

[jira] [Assigned] (SPARK-27064) create StreamingWrite at the begining of streaming execution

2019-03-05 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27064?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27064: Assignee: Wenchen Fan (was: Apache Spark) > create StreamingWrite at the begining of

[jira] [Commented] (SPARK-27060) DDL Commands are accepting Keywords like create, drop as tableName

2019-03-05 Thread Sachin Ramachandra Setty (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784575#comment-16784575 ] Sachin Ramachandra Setty commented on SPARK-27060: -- I verified this issue with 2.3.2

[jira] [Comment Edited] (SPARK-23986) CompileException when using too many avg aggregation after joining

2019-03-05 Thread Pedro Fernandes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784482#comment-16784482 ] Pedro Fernandes edited comment on SPARK-23986 at 3/5/19 3:38 PM: - -Guys,

[jira] [Created] (SPARK-27062) Refresh Table command register table with table name only

2019-03-05 Thread William Wong (JIRA)
William Wong created SPARK-27062: Summary: Refresh Table command register table with table name only Key: SPARK-27062 URL: https://issues.apache.org/jira/browse/SPARK-27062 Project: Spark

[jira] [Created] (SPARK-27064) create StreamingWrite at the begining of streaming execution

2019-03-05 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27064: --- Summary: create StreamingWrite at the begining of streaming execution Key: SPARK-27064 URL: https://issues.apache.org/jira/browse/SPARK-27064 Project: Spark

[jira] [Created] (SPARK-27065) avoid more than one active task set managers for a stage

2019-03-05 Thread Wenchen Fan (JIRA)
Wenchen Fan created SPARK-27065: --- Summary: avoid more than one active task set managers for a stage Key: SPARK-27065 URL: https://issues.apache.org/jira/browse/SPARK-27065 Project: Spark Issue

[jira] [Commented] (SPARK-26727) CREATE OR REPLACE VIEW query fails with TableAlreadyExistsException

2019-03-05 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784666#comment-16784666 ] Ajith S commented on SPARK-26727: - [~rigolaszlo] i see that from stacktrace ThriftHiveMetastore$Client

[jira] [Updated] (SPARK-27060) DDL Commands are accepting Keywords like create, drop as tableName

2019-03-05 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-27060: -- Target Version/s: (was: 2.4.0) Priority: Minor (was: Major) Fix Version/s:

[jira] [Comment Edited] (SPARK-23986) CompileException when using too many avg aggregation after joining

2019-03-05 Thread Pedro Fernandes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784482#comment-16784482 ] Pedro Fernandes edited comment on SPARK-23986 at 3/5/19 3:38 PM: - ~Guys,

[jira] [Created] (SPARK-27061) Expose 4040 port on driver service to access logs using service

2019-03-05 Thread Chandu Kavar (JIRA)
Chandu Kavar created SPARK-27061: Summary: Expose 4040 port on driver service to access logs using service Key: SPARK-27061 URL: https://issues.apache.org/jira/browse/SPARK-27061 Project: Spark

[jira] [Updated] (SPARK-27062) Refresh Table command register table with table name only

2019-03-05 Thread William Wong (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27062?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] William Wong updated SPARK-27062: - Description: If CatalogImpl.refreshTable() method is invoked against a cached table, this

[jira] [Comment Edited] (SPARK-26602) Insert into table fails after querying the UDF which is loaded with wrong hdfs path

2019-03-05 Thread Ajith S (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-26602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784611#comment-16784611 ] Ajith S edited comment on SPARK-26602 at 3/5/19 4:15 PM: - # I have a question

[jira] [Created] (SPARK-27063) Spark on K8S Integration Tests timeouts are too short for some test clusters

2019-03-05 Thread Rob Vesse (JIRA)
Rob Vesse created SPARK-27063: - Summary: Spark on K8S Integration Tests timeouts are too short for some test clusters Key: SPARK-27063 URL: https://issues.apache.org/jira/browse/SPARK-27063 Project:

[jira] [Comment Edited] (SPARK-23986) CompileException when using too many avg aggregation after joining

2019-03-05 Thread Pedro Fernandes (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-23986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16784482#comment-16784482 ] Pedro Fernandes edited comment on SPARK-23986 at 3/5/19 3:39 PM: - -Guys, 

  1   2   >