[jira] [Assigned] (SPARK-20090) Add StructType.fieldNames to Python API

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20090: Assignee: Apache Spark > Add StructType.fieldNames to Python API >

[jira] [Commented] (SPARK-20090) Add StructType.fieldNames to Python API

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085224#comment-16085224 ] Apache Spark commented on SPARK-20090: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-20090) Add StructType.fieldNames to Python API

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20090: Assignee: (was: Apache Spark) > Add StructType.fieldNames to Python API >

[jira] [Created] (SPARK-21396) Spark Hive Thriftserver doesn't return UDT field

2017-07-12 Thread Haopu Wang (JIRA)
Haopu Wang created SPARK-21396: -- Summary: Spark Hive Thriftserver doesn't return UDT field Key: SPARK-21396 URL: https://issues.apache.org/jira/browse/SPARK-21396 Project: Spark Issue Type:

[jira] [Comment Edited] (SPARK-20703) Add an operator for writing data out

2017-07-12 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085017#comment-16085017 ] Liang-Chi Hsieh edited comment on SPARK-20703 at 7/13/17 5:28 AM: --

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085160#comment-16085160 ] Apache Spark commented on SPARK-21376: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Assigned] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21376: Assignee: Apache Spark > Token is not renewed in yarn client process in cluster mode >

[jira] [Assigned] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21376: Assignee: (was: Apache Spark) > Token is not renewed in yarn client process in

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085099#comment-16085099 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/13/17 3:42 AM: ---

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085099#comment-16085099 ] Kazuaki Ishizaki commented on SPARK-21391: -- [~hyukjin.kwon] I think that

[jira] [Updated] (SPARK-21395) Spark SQL hive-thriftserver doesn't register operation log before execute sql statement

2017-07-12 Thread Chaozhong Yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaozhong Yang updated SPARK-21395: --- Description: In HiveServer2, TFetchResultsReq has a member which is named as `fetchType`. If

[jira] [Created] (SPARK-21395) Spark SQL hive-thriftserver doesn't register operation log before execute sql statement

2017-07-12 Thread Chaozhong Yang (JIRA)
Chaozhong Yang created SPARK-21395: -- Summary: Spark SQL hive-thriftserver doesn't register operation log before execute sql statement Key: SPARK-21395 URL: https://issues.apache.org/jira/browse/SPARK-21395

[jira] [Updated] (SPARK-21297) Add count in 'JDBC/ODBC Server' page.

2017-07-12 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte updated SPARK-21297: --- Description: 1.Add count about 'Session Statistics' and 'SQL Statistics' in 'JDBC/ODBC

[jira] [Commented] (SPARK-20703) Add an operator for writing data out

2017-07-12 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16085017#comment-16085017 ] Liang-Chi Hsieh commented on SPARK-20703: - Thanks [~ste...@apache.org] for voicing this. For the

[jira] [Updated] (SPARK-21297) Add count in 'JDBC/ODBC Server' page.

2017-07-12 Thread guoxiaolongzte (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] guoxiaolongzte updated SPARK-21297: --- Summary: Add count in 'JDBC/ODBC Server' page. (was: Add State in 'Session Statistics'

[jira] [Resolved] (SPARK-18646) ExecutorClassLoader for spark-shell does not honor spark.executor.userClassPathFirst

2017-07-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-18646. - Resolution: Fixed Assignee: Min Shen Fix Version/s: 2.3.0 > ExecutorClassLoader

[jira] [Updated] (SPARK-21377) Jars specified with --jars or --packages are not added into AM's system classpath

2017-07-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-21377: Summary: Jars specified with --jars or --packages are not added into AM's system classpath (was:

[jira] [Commented] (SPARK-21377) Add a new configuration to extend AM classpath in yarn client mode

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084929#comment-16084929 ] Apache Spark commented on SPARK-21377: -- User 'jerryshao' has created a pull request for this issue:

[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

2017-07-12 Thread Ruslan Dautkhanov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084886#comment-16084886 ] Ruslan Dautkhanov commented on SPARK-13534: --- [~bryanc], thanks for the feedback. We sometimes

[jira] [Comment Edited] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084837#comment-16084837 ] Stuart Reynolds edited comment on SPARK-21392 at 7/12/17 10:27 PM: ---

[jira] [Created] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

2017-07-12 Thread Zahra (JIRA)
Zahra created SPARK-21393: - Summary: spark (pyspark) crashes unpredictably when using show() or toPandas() Key: SPARK-21393 URL: https://issues.apache.org/jira/browse/SPARK-21393 Project: Spark

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084837#comment-16084837 ] Stuart Reynolds commented on SPARK-21392: - I've simplified the example a little more and also

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works {code:none} response = "mi_or_chd_5" sc =

[jira] [Commented] (SPARK-12559) Cluster mode doesn't work with --packages

2017-07-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084776#comment-16084776 ] Marcelo Vanzin commented on SPARK-12559: [~skonto], could you look at whether the same approach

[jira] [Comment Edited] (SPARK-12559) Cluster mode doesn't work with --packages

2017-07-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084776#comment-16084776 ] Marcelo Vanzin edited comment on SPARK-12559 at 7/12/17 9:49 PM: -

[jira] [Comment Edited] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084837#comment-16084837 ] Stuart Reynolds edited comment on SPARK-21392 at 7/12/17 10:27 PM: ---

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading large Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works {code:none} response = "mi_or_chd_5" sc =

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084821#comment-16084821 ] Hyukjin Kwon commented on SPARK-21392: -- I think this is unrelated with that JIRA ^ too. > Unable to

[jira] [Comment Edited] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

2017-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084689#comment-16084689 ] Hyukjin Kwon edited comment on SPARK-21393 at 7/12/17 10:12 PM: Would you

[jira] [Assigned] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21394: Assignee: Apache Spark > Reviving broken callable objects in UDF in PySpark >

[jira] [Commented] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084787#comment-16084787 ] Apache Spark commented on SPARK-21394: -- User 'HyukjinKwon' has created a pull request for this

[jira] [Assigned] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21394: Assignee: (was: Apache Spark) > Reviving broken callable objects in UDF in PySpark >

[jira] [Commented] (SPARK-21374) Reading globbed paths from S3 into DF doesn't work if filesystem caching is disabled

2017-07-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084764#comment-16084764 ] Shixiong Zhu commented on SPARK-21374: -- Yeah, org.apache.spark.deploy.SparkHadoopUtil.globPath uses

[jira] [Comment Edited] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084772#comment-16084772 ] Shixiong Zhu edited comment on SPARK-21378 at 7/12/17 9:49 PM: --- bq. Digging

[jira] [Commented] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084772#comment-16084772 ] Shixiong Zhu commented on SPARK-21378: -- bq. Digging deeper shows that there's an assert statement

[jira] [Reopened] (SPARK-12559) Cluster mode doesn't work with --packages

2017-07-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin reopened SPARK-12559: > Cluster mode doesn't work with --packages > - > >

[jira] [Updated] (SPARK-21378) Spark Poll timeout when specific offsets are passed

2017-07-12 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-21378: - Component/s: (was: Spark Core) DStreams > Spark Poll timeout when specific

[jira] [Resolved] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

2017-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon resolved SPARK-21391. -- Resolution: Cannot Reproduce I can't reproduce this against the current master as described in

[jira] [Created] (SPARK-21394) Reviving broken callable objects in UDF in PySpark

2017-07-12 Thread Hyukjin Kwon (JIRA)
Hyukjin Kwon created SPARK-21394: Summary: Reviving broken callable objects in UDF in PySpark Key: SPARK-21394 URL: https://issues.apache.org/jira/browse/SPARK-21394 Project: Spark Issue

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084697#comment-16084697 ] Hyukjin Kwon commented on SPARK-21392: -- Would you mind running {{outcome.show}} and attaching the

[jira] [Updated] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

2017-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-21393: - Affects Version/s: (was: 2.2.1) > spark (pyspark) crashes unpredictably when using show() or

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084707#comment-16084707 ] Dongjoon Hyun commented on SPARK-21392: --- It seems to be a different issue. SPARK-16975 aims to read

[jira] [Commented] (SPARK-21393) spark (pyspark) crashes unpredictably when using show() or toPandas()

2017-07-12 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084689#comment-16084689 ] Hyukjin Kwon commented on SPARK-21393: -- Would you mind sharing your codes? I want to reproduce this

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084601#comment-16084601 ] Stuart Reynolds commented on SPARK-21392: - Done. I'm simply trying to build a table of two

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works {code:none} response = "mi_or_chd_5"

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works {code:none} response = "mi_or_chd_5"

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works {code:python} response = "mi_or_chd_5"

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works {code:python} response = "mi_or_chd_5"

[jira] [Updated] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stuart Reynolds updated SPARK-21392: Description: The following boring code works {code:python} response = "mi_or_chd_5"

[jira] [Commented] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084583#comment-16084583 ] Sean Owen commented on SPARK-21392: --- Can you format this so it's readable? I don't understand how

[jira] [Issue Comment Deleted] (SPARK-21221) CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest

2017-07-12 Thread Ajay Saini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Saini updated SPARK-21221: --- Comment: was deleted (was: Note: In order for python persistence of OneVsRest inside a

[jira] [Assigned] (SPARK-14280) Update change-version.sh and pom.xml to add Scala 2.12 profiles

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14280: -- Assignee: Josh Rosen > Update change-version.sh and pom.xml to add Scala 2.12 profiles >

[jira] [Assigned] (SPARK-14280) Update change-version.sh and pom.xml to add Scala 2.12 profiles

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14280: -- Assignee: (was: Josh Rosen) > Update change-version.sh and pom.xml to add Scala 2.12

[jira] [Assigned] (SPARK-14650) Compile Spark REPL for Scala 2.12

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14650: -- Assignee: (was: Josh Rosen) > Compile Spark REPL for Scala 2.12 >

[jira] [Resolved] (SPARK-14438) Cross-publish Breeze for Scala 2.12

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-14438. Resolution: Fixed > Cross-publish Breeze for Scala 2.12 > --- > >

[jira] [Assigned] (SPARK-14280) Update change-version.sh and pom.xml to add Scala 2.12 profiles

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen reassigned SPARK-14280: -- Assignee: (was: Josh Rosen) > Update change-version.sh and pom.xml to add Scala 2.12

[jira] [Resolved] (SPARK-14519) Cross-publish Kafka for Scala 2.12

2017-07-12 Thread Josh Rosen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Rosen resolved SPARK-14519. Resolution: Fixed > Cross-publish Kafka for Scala 2.12 > -- > >

[jira] [Comment Edited] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084460#comment-16084460 ] Saisai Shao edited comment on SPARK-21376 at 7/12/17 6:31 PM: -- I'm

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084460#comment-16084460 ] Saisai Shao commented on SPARK-21376: - I'm referrring to o.a.s.deploy.yarn.Client this class, it will

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084458#comment-16084458 ] Thomas Graves commented on SPARK-21376: --- so you are referring to the

[jira] [Updated] (SPARK-21221) CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest

2017-07-12 Thread Ajay Saini (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Saini updated SPARK-21221: --- Summary: CrossValidator and TrainValidationSplit Persist Nested Estimators such as OneVsRest (was:

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-07-12 Thread Ye Zhou (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084451#comment-16084451 ] Ye Zhou commented on SPARK-18085: - I want to add my own testing experience with the codes from the HEAD

[jira] [Created] (SPARK-21392) Unable to infer schema when loading Parquet file

2017-07-12 Thread Stuart Reynolds (JIRA)
Stuart Reynolds created SPARK-21392: --- Summary: Unable to infer schema when loading Parquet file Key: SPARK-21392 URL: https://issues.apache.org/jira/browse/SPARK-21392 Project: Spark Issue

[jira] [Commented] (SPARK-13534) Implement Apache Arrow serializer for Spark DataFrame for use in DataFrame.toPandas

2017-07-12 Thread Bryan Cutler (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084390#comment-16084390 ] Bryan Cutler commented on SPARK-13534: -- Hi [~tagar], the {{ArrowSerializer}} doesn't quite fit as a

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-07-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084369#comment-16084369 ] Marcelo Vanzin commented on SPARK-18085: [~kanjilal] the code is pretty much all written at this

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-07-12 Thread Saikat Kanjilal (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084361#comment-16084361 ] Saikat Kanjilal commented on SPARK-18085: - [~vanzin] I would be interested in helping with this,

[jira] [Commented] (SPARK-20703) Add an operator for writing data out

2017-07-12 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084355#comment-16084355 ] Steve Loughran commented on SPARK-20703: this has just added a whole new stack trace for my

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM: ---

[jira] [Comment Edited] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333 ] Kazuaki Ishizaki edited comment on SPARK-21391 at 7/12/17 5:19 PM: ---

[jira] [Commented] (SPARK-21380) Join with Columns thinks inner join is cross join even when aliased

2017-07-12 Thread Dongjoon Hyun (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084274#comment-16084274 ] Dongjoon Hyun commented on SPARK-21380: --- I see. I agree your point about that warning is misleading

[jira] [Commented] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-07-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084277#comment-16084277 ] Reynold Xin commented on SPARK-18085: - You should email dev@ to notify the list about a new SPIP.

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084270#comment-16084270 ] Saisai Shao commented on SPARK-21376: - Hi [~tgraves], it is the local yarn launcher process which

[jira] [Commented] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084333#comment-16084333 ] Kazuaki Ishizaki commented on SPARK-21391: -- This program works with the master. {code}

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084306#comment-16084306 ] Kazuaki Ishizaki commented on SPARK-21390: -- Another interesting results with Spark-2.2: On IDE

[jira] [Comment Edited] (SPARK-21390) Dataset filter api inconsistency

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084306#comment-16084306 ] Kazuaki Ishizaki edited comment on SPARK-21390 at 7/12/17 5:09 PM: ---

[jira] [Updated] (SPARK-18085) SPIP: Better History Server scalability for many / large applications

2017-07-12 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-18085: Summary: SPIP: Better History Server scalability for many / large applications (was: Better

[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications

2017-07-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084265#comment-16084265 ] Marcelo Vanzin commented on SPARK-18085: Sure, if it's just a matter of adding the label to the

[jira] [Updated] (SPARK-18085) Better History Server scalability for many / large applications

2017-07-12 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marcelo Vanzin updated SPARK-18085: --- Labels: SPIP (was: ) > Better History Server scalability for many / large applications >

[jira] [Commented] (SPARK-21390) Dataset filter api inconsistency

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084266#comment-16084266 ] Kazuaki Ishizaki commented on SPARK-21390: -- Thank you for reporting this. I can reproduce this

[jira] [Created] (SPARK-21391) Cannot convert a Seq of Map whose value type is again a seq, into a dataset

2017-07-12 Thread indraneel rao (JIRA)
indraneel rao created SPARK-21391: - Summary: Cannot convert a Seq of Map whose value type is again a seq, into a dataset Key: SPARK-21391 URL: https://issues.apache.org/jira/browse/SPARK-21391

[jira] [Updated] (SPARK-21390) Dataset filter api inconsistency

2017-07-12 Thread Gheorghe Gheorghe (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gheorghe Gheorghe updated SPARK-21390: -- Description: Hello everybody, I've encountered a strange situation with the

[jira] [Updated] (SPARK-21390) Dataset filter api inconsistency

2017-07-12 Thread Gheorghe Gheorghe (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gheorghe Gheorghe updated SPARK-21390: -- Description: Hello everybody, I've encountered a strange situation with the

[jira] [Created] (SPARK-21390) Dataset filter api inconsistency

2017-07-12 Thread Gheorghe Gheorghe (JIRA)
Gheorghe Gheorghe created SPARK-21390: - Summary: Dataset filter api inconsistency Key: SPARK-21390 URL: https://issues.apache.org/jira/browse/SPARK-21390 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-18646) ExecutorClassLoader for spark-shell does not honor spark.executor.userClassPathFirst

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084193#comment-16084193 ] Apache Spark commented on SPARK-18646: -- User 'jiangxb1987' has created a pull request for this

[jira] [Commented] (SPARK-18838) High latency of event processing for large jobs

2017-07-12 Thread Arun Achuthan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084169#comment-16084169 ] Arun Achuthan commented on SPARK-18838: --- We are facing an issue where randomly some jobs are stuck

[jira] [Created] (SPARK-21389) ALS recommendForAll optimization uses Native BLAS

2017-07-12 Thread Peng Meng (JIRA)
Peng Meng created SPARK-21389: - Summary: ALS recommendForAll optimization uses Native BLAS Key: SPARK-21389 URL: https://issues.apache.org/jira/browse/SPARK-21389 Project: Spark Issue Type:

[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode

2017-07-12 Thread Thomas Graves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084075#comment-16084075 ] Thomas Graves commented on SPARK-21376: --- Can you please clarify the title and description? What do

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-12 Thread Joseph Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084069#comment-16084069 ] Joseph Wang commented on SPARK-20307: - Fantastic function to add. It would be nice to generalize to

[jira] [Commented] (SPARK-20307) SparkR: pass on setHandleInvalid to spark.mllib functions that use StringIndexer

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084058#comment-16084058 ] Apache Spark commented on SPARK-20307: -- User 'yanboliang' has created a pull request for this issue:

[jira] [Resolved] (SPARK-18619) Make QuantileDiscretizer/Bucketizer/StringIndexer inherit from HasHandleInvalid

2017-07-12 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang resolved SPARK-18619. - Resolution: Fixed Assignee: zhengruifeng Fix Version/s: 2.3.0 > Make

[jira] [Commented] (SPARK-21388) GBT inherit from HasStepSize & LInearSVC/Binarizer from HasThreshold

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083974#comment-16083974 ] Apache Spark commented on SPARK-21388: -- User 'zhengruifeng' has created a pull request for this

[jira] [Assigned] (SPARK-21388) GBT inherit from HasStepSize & LInearSVC/Binarizer from HasThreshold

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21388: Assignee: Apache Spark > GBT inherit from HasStepSize & LInearSVC/Binarizer from

[jira] [Assigned] (SPARK-21388) GBT inherit from HasStepSize & LInearSVC/Binarizer from HasThreshold

2017-07-12 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-21388: Assignee: (was: Apache Spark) > GBT inherit from HasStepSize & LInearSVC/Binarizer

[jira] [Created] (SPARK-21388) GBT inherit from HasStepSize & LInearSVC/Binarizer from HasThreshold

2017-07-12 Thread zhengruifeng (JIRA)
zhengruifeng created SPARK-21388: Summary: GBT inherit from HasStepSize & LInearSVC/Binarizer from HasThreshold Key: SPARK-21388 URL: https://issues.apache.org/jira/browse/SPARK-21388 Project: Spark

[jira] [Updated] (SPARK-21374) Reading globbed paths from S3 into DF doesn't work if filesystem caching is disabled

2017-07-12 Thread Andrey Taptunov (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Taptunov updated SPARK-21374: Description: *Motivation:* In my case I want to disable filesystem cache to be able to

[jira] [Resolved] (SPARK-21007) Add SQL function - RIGHT && LEFT

2017-07-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-21007. - Resolution: Fixed Fix Version/s: 2.3.0 Issue resolved by pull request 18228

[jira] [Created] (SPARK-21387) org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM

2017-07-12 Thread Kazuaki Ishizaki (JIRA)
Kazuaki Ishizaki created SPARK-21387: Summary: org.apache.spark.memory.TaskMemoryManager.allocatePage causes OOM Key: SPARK-21387 URL: https://issues.apache.org/jira/browse/SPARK-21387 Project:

[jira] [Assigned] (SPARK-21007) Add SQL function - RIGHT && LEFT

2017-07-12 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-21007: --- Assignee: liuxian > Add SQL function - RIGHT && LEFT > - >

[jira] [Resolved] (SPARK-21078) JobHistory applications synchronized is invalid

2017-07-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-21078. --- Resolution: Duplicate I'm resolving this as 'duplicate' but really this will just go away when the

[jira] [Assigned] (SPARK-21305) The BKM (best known methods) of using native BLAS to improvement ML/MLLIB performance

2017-07-12 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-21305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-21305: - Assignee: Peng Meng Flags: (was: Important) Affects Version/s:

  1   2   >