[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-06-10 Thread Henry Yu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860580#comment-16860580 ] Henry Yu commented on SPARK-27812: -- So, any other idea to fix it? or I will try to make pr with the 

[jira] [Commented] (SPARK-27018) Checkpointed RDD deleted prematurely when using GBTClassifier

2019-06-10 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860579#comment-16860579 ] zhengruifeng commented on SPARK-27018: -- I test on both local env and a cluster env, and your patch

[jira] [Created] (SPARK-27997) kubernetes client token expired

2019-06-10 Thread Henry Yu (JIRA)
Henry Yu created SPARK-27997: Summary: kubernetes client token expired Key: SPARK-27997 URL: https://issues.apache.org/jira/browse/SPARK-27997 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-27499) Support mapping spark.local.dir to hostPath volume

2019-06-10 Thread Junjie Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860519#comment-16860519 ] Junjie Chen edited comment on SPARK-27499 at 6/11/19 5:20 AM: -- Hi,

[jira] [Updated] (SPARK-27845) DataSourceV2: InsertTable

2019-06-10 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27845: --- Summary: DataSourceV2: InsertTable (was: DataSourceV2: Insert into tables in multiple catalogs) >

[jira] [Updated] (SPARK-27322) DataSourceV2 table relation

2019-06-10 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27322: --- Summary: DataSourceV2 table relation (was: DataSourceV2: Select from multiple catalogs) >

[jira] [Updated] (SPARK-27372) Standalone executor process-level isolation to support GPU scheduling

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27372: -- Issue Type: Story (was: Sub-task) Parent: (was: SPARK-27360) > Standalone

[jira] [Updated] (SPARK-27371) Standalone master receives resource info from worker and allocate driver/executor properly

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27371: -- Summary: Standalone master receives resource info from worker and allocate driver/executor

[jira] [Updated] (SPARK-27371) Master receives resource info from worker and allocate driver/executor properly

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27371: -- Summary: Master receives resource info from worker and allocate driver/executor properly

[jira] [Deleted] (SPARK-27370) spark-submit requests GPUs in standalone mode

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng deleted SPARK-27370: -- > spark-submit requests GPUs in standalone mode > -

[jira] [Updated] (SPARK-27369) Standalone worker can load resource conf and discover resources

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27369: -- Summary: Standalone worker can load resource conf and discover resources (was: Standalone

[jira] [Reopened] (SPARK-27499) Support mapping spark.local.dir to hostPath volume

2019-06-10 Thread Junjie Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junjie Chen reopened SPARK-27499: - > Support mapping spark.local.dir to hostPath volume >

[jira] [Commented] (SPARK-27499) Support mapping spark.local.dir to hostPath volume

2019-06-10 Thread Junjie Chen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860519#comment-16860519 ] Junjie Chen commented on SPARK-27499: - Hi, [~dongjoon], I know SPARK_LOCAL_DIRS can be mounted as

[jira] [Updated] (SPARK-27368) Design: Standalone supports GPU scheduling

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27368: -- Description: Design draft: Scenarios: * client-mode, worker might create one or more

[jira] [Updated] (SPARK-27368) Design: Standalone supports GPU scheduling

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27368: -- Description: Design draft: Scenarios: * client-mode, worker might create one or more

[jira] [Updated] (SPARK-27368) Design: Standalone supports GPU scheduling

2019-06-10 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-27368: -- Description: Design draft: Scenarios: * client-mode, worker might create one or more

[jira] [Commented] (SPARK-18112) Spark2.x does not support read data from Hive 2.x metastore

2019-06-10 Thread HonglunChen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860512#comment-16860512 ] HonglunChen commented on SPARK-18112: - Why do my issue still exist,I use the Spark-2.4.3 and

[jira] [Created] (SPARK-27996) Spark UI redirect will be failed behind the https reverse proxy

2019-06-10 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-27996: --- Summary: Spark UI redirect will be failed behind the https reverse proxy Key: SPARK-27996 URL: https://issues.apache.org/jira/browse/SPARK-27996 Project: Spark

[jira] [Assigned] (SPARK-27995) Note the difference between str of Python 2 and 3 at Arrow optimized toPandas

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27995: Assignee: (was: Apache Spark) > Note the difference between str of Python 2 and 3 at

[jira] [Assigned] (SPARK-27995) Note the difference between str of Python 2 and 3 at Arrow optimized toPandas

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27995: Assignee: Apache Spark > Note the difference between str of Python 2 and 3 at Arrow

[jira] [Updated] (SPARK-27995) Note the difference between str of Python 2 and 3 at Arrow optimized toPandas

2019-06-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-27995: - Description: When Arrow optimization is enabled in Python 2.7, {code} import pandas pdf =

[jira] [Created] (SPARK-27995) Note the difference between str of Python 2 and 3 at Arrow optimized toPandas

2019-06-10 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-27995: Summary: Note the difference between str of Python 2 and 3 at Arrow optimized toPandas Key: SPARK-27995 URL: https://issues.apache.org/jira/browse/SPARK-27995

[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-06-10 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860491#comment-16860491 ] Jiatao Tao edited comment on SPARK-27546 at 6/11/19 2:46 AM: - Hi [~dongjoon]

[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-06-10 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860491#comment-16860491 ] Jiatao Tao commented on SPARK-27546: After reading the code,  Although I have set

[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-06-10 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860491#comment-16860491 ] Jiatao Tao edited comment on SPARK-27546 at 6/11/19 2:42 AM: - Hi [~dongjoon]

[jira] [Comment Edited] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-06-10 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860489#comment-16860489 ] Jiatao Tao edited comment on SPARK-27546 at 6/11/19 2:40 AM: - Hi [~dongjoon]

[jira] [Commented] (SPARK-27546) Should repalce DateTimeUtils#defaultTimeZoneuse with sessionLocalTimeZone

2019-06-10 Thread Jiatao Tao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860489#comment-16860489 ] Jiatao Tao commented on SPARK-27546: Hi [~dongjoon]   What I get is "ts", not the "Mon Dec 31

[jira] [Issue Comment Deleted] (SPARK-20894) Error while checkpointing to HDFS

2019-06-10 Thread phan minh duc (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] phan minh duc updated SPARK-20894: -- Comment: was deleted (was: I'm using spark 2.4.0 and facing the same issue when i submit

[jira] [Updated] (SPARK-27825) spark thriftserver session's first username cause the impersonation issue.

2019-06-10 Thread wangxinxin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxinxin updated SPARK-27825: --- Attachment: test.zip > spark thriftserver session's first username cause the impersonation issue.

[jira] [Updated] (SPARK-27825) spark thriftserver session's first username cause the impersonation issue.

2019-06-10 Thread wangxinxin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangxinxin updated SPARK-27825: --- Attachment: (was: test.zip) > spark thriftserver session's first username cause the

[jira] [Resolved] (SPARK-27979) Remove deprecated `--force` option in `build/mvn` and `run-tests.py`

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27979. --- Resolution: Fixed This is resolved back via https://github.com/apache/spark/pull/24833 >

[jira] [Commented] (SPARK-27994) Spark Avro Failed to read logical type decimal backed by bytes

2019-06-10 Thread Nicolas Pascal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860442#comment-16860442 ] Nicolas Pascal commented on SPARK-27994: I'll push a PR with failing test cases to reproduce the

[jira] [Closed] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-25053. - Thank you, [~skonto]. > Allow additional port forwarding on Spark on K8S as needed >

[jira] [Commented] (SPARK-27812) kubernetes client import non-daemon thread which block jvm exit.

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860439#comment-16860439 ] Dongjoon Hyun commented on SPARK-27812: --- Thank you for update, [~Andrew HUALI]. > kubernetes

[jira] [Commented] (SPARK-27697) KubernetesClientApplication alway exit with 0

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860438#comment-16860438 ] Dongjoon Hyun commented on SPARK-27697: --- Please make a PR. Let's see there, [~Andrew HUALI] >

[jira] [Assigned] (SPARK-27993) Port HIVE-12981 to hive-thriftserver

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27993: Assignee: (was: Apache Spark) > Port HIVE-12981 to hive-thriftserver >

[jira] [Assigned] (SPARK-27993) Port HIVE-12981 to hive-thriftserver

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27993: Assignee: Apache Spark > Port HIVE-12981 to hive-thriftserver >

[jira] [Created] (SPARK-27994) Spark Avro Failed to read logical type decimal backed by bytes

2019-06-10 Thread Nicolas Pascal (JIRA)
Nicolas Pascal created SPARK-27994: -- Summary: Spark Avro Failed to read logical type decimal backed by bytes Key: SPARK-27994 URL: https://issues.apache.org/jira/browse/SPARK-27994 Project: Spark

[jira] [Created] (SPARK-27993) Port HIVE-12981 to hive-thriftserver

2019-06-10 Thread Yuming Wang (JIRA)
Yuming Wang created SPARK-27993: --- Summary: Port HIVE-12981 to hive-thriftserver Key: SPARK-27993 URL: https://issues.apache.org/jira/browse/SPARK-27993 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-27217) Nested schema pruning doesn't work for aggregation e.g. `sum`.

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27217: -- Affects Version/s: (was: 2.4.0) 3.0.0 > Nested schema pruning

[jira] [Updated] (SPARK-27217) Nested schema pruning doesn't work for aggregation e.g. `sum`.

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27217: -- Priority: Major (was: Minor) > Nested schema pruning doesn't work for aggregation e.g.

[jira] [Updated] (SPARK-27217) Nested schema pruning doesn't work for aggregation e.g. `sum`.

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27217: -- Description: Since SPARK-4502 is fixed, I would expect queries such as `select sum(b.x)`

[jira] [Updated] (SPARK-27217) Nested schema pruning doesn't work for aggregation e.g. `sum`.

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27217: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-25603 > Nested schema pruning

[jira] [Updated] (SPARK-27290) remove unneed sort under Aggregate

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27290: -- Affects Version/s: (was: 2.4.0) 3.0.0 > remove unneed sort under

[jira] [Updated] (SPARK-19799) Support WITH clause in subqueries

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-19799: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-27764 > Support WITH clause in

[jira] [Updated] (SPARK-27970) Support Hive 3.0 metastore

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27970: -- Description: It seems that some users are using Hive 3.0.0, at least HDP 3.0.0 was: It

[jira] [Resolved] (SPARK-27917) Semantic equals of CaseWhen is failing with case sensitivity of column Names

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27917. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via

[jira] [Assigned] (SPARK-27917) Semantic equals of CaseWhen is failing with case sensitivity of column Names

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27917: - Assignee: Sandeep Katta > Semantic equals of CaseWhen is failing with case sensitivity

[jira] [Assigned] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27992: Assignee: (was: Apache Spark) > PySpark socket server should sync with JVM

[jira] [Assigned] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27992: Assignee: Apache Spark > PySpark socket server should sync with JVM connection thread

[jira] [Updated] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

2019-06-10 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27992: - Affects Version/s: (was: 2.4.3) 3.0.0 > PySpark socket server should

[jira] [Updated] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

2019-06-10 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27992: - Description: Both SPARK-27805 and SPARK-27548 identified an issue that errors in a Spark job

[jira] [Updated] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

2019-06-10 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bryan Cutler updated SPARK-27992: - Environment: (was: Both SPARK-27805 and SPARK-27548 identified an issue that errors in a

[jira] [Created] (SPARK-27992) PySpark socket server should sync with JVM connection thread future

2019-06-10 Thread Bryan Cutler (JIRA)
Bryan Cutler created SPARK-27992: Summary: PySpark socket server should sync with JVM connection thread future Key: SPARK-27992 URL: https://issues.apache.org/jira/browse/SPARK-27992 Project: Spark

[jira] [Updated] (SPARK-27947) Enhance redactOptions to accept any Map type

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27947: -- Priority: Major (was: Minor) > Enhance redactOptions to accept any Map type >

[jira] [Resolved] (SPARK-27947) Enhance redactOptions to accept any Map type

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27947. --- Resolution: Fixed Assignee: John Zhuge Fix Version/s: 3.0.0 This is

[jira] [Updated] (SPARK-27947) Enhance redactOptions to accept any Map type

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27947: -- Summary: Enhance redactOptions to accept any Map type (was: ParsedStatement subclass

[jira] [Updated] (SPARK-27947) Enhance redactOptions to accept any Map type

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27947: -- Issue Type: Improvement (was: Bug) > Enhance redactOptions to accept any Map type >

[jira] [Assigned] (SPARK-27845) DataSourceV2: Insert into tables in multiple catalogs

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27845: Assignee: (was: Apache Spark) > DataSourceV2: Insert into tables in multiple

[jira] [Assigned] (SPARK-27845) DataSourceV2: Insert into tables in multiple catalogs

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27845: Assignee: Apache Spark > DataSourceV2: Insert into tables in multiple catalogs >

[jira] [Commented] (SPARK-27845) DataSourceV2: Insert into tables in multiple catalogs

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860221#comment-16860221 ] Apache Spark commented on SPARK-27845: -- User 'jzhuge' has created a pull request for this issue:

[jira] [Updated] (SPARK-27845) DataSourceV2: Insert into tables in multiple catalogs

2019-06-10 Thread John Zhuge (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Zhuge updated SPARK-27845: --- Summary: DataSourceV2: Insert into tables in multiple catalogs (was: DataSourceV2: InsertInto

[jira] [Created] (SPARK-27991) ShuffleBlockFetcherIterator should take Netty constant-factor overheads into account when limiting number of simultaneous block fetches

2019-06-10 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-27991: -- Summary: ShuffleBlockFetcherIterator should take Netty constant-factor overheads into account when limiting number of simultaneous block fetches Key: SPARK-27991 URL:

[jira] [Resolved] (SPARK-27635) Prevent from splitting too many partitions smaller than row group size in Parquet file format

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27635. --- Resolution: Invalid Please see the discussion on the PR. > Prevent from splitting too many

[jira] [Closed] (SPARK-27635) Prevent from splitting too many partitions smaller than row group size in Parquet file format

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-27635. - > Prevent from splitting too many partitions smaller than row group size in > Parquet file format

[jira] [Resolved] (SPARK-27949) Support SUBSTRING(str FROM n1 [FOR n2]) syntax

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun resolved SPARK-27949. --- Resolution: Fixed Fix Version/s: 3.0.0 This is resolved via

[jira] [Comment Edited] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860014#comment-16860014 ] Stavros Kontopoulos edited comment on SPARK-25053 at 6/10/19 1:59 PM:

[jira] [Comment Edited] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860014#comment-16860014 ] Stavros Kontopoulos edited comment on SPARK-25053 at 6/10/19 1:58 PM:

[jira] [Resolved] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stavros Kontopoulos resolved SPARK-25053. - Resolution: Duplicate > Allow additional port forwarding on Spark on K8S as

[jira] [Comment Edited] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860014#comment-16860014 ] Stavros Kontopoulos edited comment on SPARK-25053 at 6/10/19 1:57 PM:

[jira] [Comment Edited] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860014#comment-16860014 ] Stavros Kontopoulos edited comment on SPARK-25053 at 6/10/19 1:57 PM:

[jira] [Comment Edited] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860014#comment-16860014 ] Stavros Kontopoulos edited comment on SPARK-25053 at 6/10/19 1:56 PM:

[jira] [Comment Edited] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860014#comment-16860014 ] Stavros Kontopoulos edited comment on SPARK-25053 at 6/10/19 1:55 PM:

[jira] [Commented] (SPARK-25053) Allow additional port forwarding on Spark on K8S as needed

2019-06-10 Thread Stavros Kontopoulos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-25053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860014#comment-16860014 ] Stavros Kontopoulos commented on SPARK-25053: - Sure [~dongjoon]. With the pod template

[jira] [Commented] (SPARK-27708) Add documentation for v2 data sources

2019-06-10 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860013#comment-16860013 ] Jacek Laskowski commented on SPARK-27708: - [~rdblue] Mind if I asked you to update the

[jira] [Comment Edited] (SPARK-27930) Add built-in Math Function: RANDOM

2019-06-10 Thread Yuming Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859628#comment-16859628 ] Yuming Wang edited comment on SPARK-27930 at 6/10/19 12:54 PM: ---

[jira] [Resolved] (SPARK-27425) Add count_if functions

2019-06-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-27425. -- Resolution: Fixed Fix Version/s: 3.0.0 Issue resolved by pull request 24335

[jira] [Assigned] (SPARK-27425) Add count_if functions

2019-06-10 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon reassigned SPARK-27425: Assignee: Chaerim Yeo > Add count_if functions > -- > >

[jira] [Commented] (SPARK-27018) Checkpointed RDD deleted prematurely when using GBTClassifier

2019-06-10 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859889#comment-16859889 ] zhengruifeng commented on SPARK-27018: -- [~pkolaczk]  With the codes you provided, I reproduced this

[jira] [Commented] (SPARK-27400) LinearSVC only supports binary classification

2019-06-10 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859878#comment-16859878 ] zhengruifeng commented on SPARK-27400: -- According to current design, LinearSVC only support binary

[jira] [Assigned] (SPARK-27990) Provide a way to recursively load data from datasource

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27990: Assignee: (was: Apache Spark) > Provide a way to recursively load data from

[jira] [Resolved] (SPARK-27867) RegressionEvaluator cache lastest RegressionMetrics to avoid duplicated computation

2019-06-10 Thread zhengruifeng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhengruifeng resolved SPARK-27867. -- Resolution: Not A Problem > RegressionEvaluator cache lastest RegressionMetrics to avoid

[jira] [Assigned] (SPARK-27990) Provide a way to recursively load data from datasource

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27990: Assignee: Apache Spark > Provide a way to recursively load data from datasource >

[jira] [Created] (SPARK-27990) Provide a way to recursively load data from datasource

2019-06-10 Thread Weichen Xu (JIRA)
Weichen Xu created SPARK-27990: -- Summary: Provide a way to recursively load data from datasource Key: SPARK-27990 URL: https://issues.apache.org/jira/browse/SPARK-27990 Project: Spark Issue

[jira] [Commented] (SPARK-27303) PropertyGraph construction (Scala/Java)

2019-06-10 Thread Martin Junghanns (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859853#comment-16859853 ] Martin Junghanns commented on SPARK-27303: -- [~mengxr] I will work on this, you can assign me.

[jira] [Assigned] (SPARK-27989) Add retries on the connection to the driver

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27989: Assignee: (was: Apache Spark) > Add retries on the connection to the driver >

[jira] [Commented] (SPARK-27989) Add retries on the connection to the driver

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859833#comment-16859833 ] Apache Spark commented on SPARK-27989: -- User 'jlpedrosa' has created a pull request for this issue:

[jira] [Assigned] (SPARK-27989) Add retries on the connection to the driver

2019-06-10 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-27989: Assignee: Apache Spark > Add retries on the connection to the driver >

[jira] [Created] (SPARK-27989) Add retries on the connection to the driver

2019-06-10 Thread Jose Luis Pedrosa (JIRA)
Jose Luis Pedrosa created SPARK-27989: - Summary: Add retries on the connection to the driver Key: SPARK-27989 URL: https://issues.apache.org/jira/browse/SPARK-27989 Project: Spark Issue

[jira] [Updated] (SPARK-27949) Support SUBSTRING(str FROM n1 [FOR n2]) syntax

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27949: -- Description: Currently, function `substr/substring`'s usage is like

[jira] [Assigned] (SPARK-27949) Support SUBSTRING(str FROM n1 [FOR n2]) syntax

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun reassigned SPARK-27949: - Assignee: Zhu, Lipeng > Support SUBSTRING(str FROM n1 [FOR n2]) syntax >

[jira] [Updated] (SPARK-27949) Support ANSI SQL grammar `substring(string_expression from n1 [for n2])`

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27949: -- Affects Version/s: (was: 3.1.0) 3.0.0 > Support ANSI SQL grammar

[jira] [Updated] (SPARK-27949) Support SUBSTRING(str FROM n1 [FOR n2]) syntax

2019-06-10 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-27949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun updated SPARK-27949: -- Summary: Support SUBSTRING(str FROM n1 [FOR n2]) syntax (was: Support ANSI SQL grammar