[jira] [Created] (SPARK-16053) Add `spark_partition_id` in SparkR

Dongjoon Hyun created SPARK-16053: - Summary: Add `spark_partition_id` in SparkR Key: SPARK-16053 URL: https://issues.apache.org/jira/browse/SPARK-16053 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-16037) use by-position resolution when insert into hive table

[ https://issues.apache.org/jira/browse/SPARK-16037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338367#comment-15338367 ] Apache Spark commented on SPARK-16037: -- User 'yhuai' has created a pull request for

[jira] [Commented] (SPARK-16034) Checks the partition columns when calling dataFrame.write.mode("append").saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-16034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338368#comment-15338368 ] Apache Spark commented on SPARK-16034: -- User 'yhuai' has created a pull request for

[jira] [Commented] (SPARK-16036) better error message if the number of columns in SELECT clause doesn't match the table schema

[ https://issues.apache.org/jira/browse/SPARK-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338366#comment-15338366 ] Apache Spark commented on SPARK-16036: -- User 'yhuai' has created a pull request for

[jira] [Updated] (SPARK-15722) Wrong data when CTAS specifies schema

[ https://issues.apache.org/jira/browse/SPARK-15722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rekha Joshi updated SPARK-15722: Description: {code} scala> sql("CREATE TABLE boxes (width INT, length INT, height INT) USING CSV")

[jira] [Assigned] (SPARK-16052) Add CollapseRepartitionBy optimizer

[ https://issues.apache.org/jira/browse/SPARK-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16052: Assignee: (was: Apache Spark) > Add CollapseRepartitionBy optimizer >

[jira] [Assigned] (SPARK-16052) Add CollapseRepartitionBy optimizer

[ https://issues.apache.org/jira/browse/SPARK-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16052: Assignee: Apache Spark > Add CollapseRepartitionBy optimizer > ---

[jira] [Commented] (SPARK-16052) Add CollapseRepartitionBy optimizer

[ https://issues.apache.org/jira/browse/SPARK-16052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338309#comment-15338309 ] Apache Spark commented on SPARK-16052: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Created] (SPARK-16052) Add CollapseRepartitionBy optimizer

Dongjoon Hyun created SPARK-16052: - Summary: Add CollapseRepartitionBy optimizer Key: SPARK-16052 URL: https://issues.apache.org/jira/browse/SPARK-16052 Project: Spark Issue Type: Improvement

[jira] [Assigned] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16024: Assignee: Apache Spark > column comment is ignored for datasource table >

[jira] [Assigned] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16024: Assignee: (was: Apache Spark) > column comment is ignored for datasource table > -

[jira] [Commented] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338286#comment-15338286 ] Apache Spark commented on SPARK-16024: -- User 'gatorsmile' has created a pull request

[jira] [Updated] (SPARK-15973) Fix GroupedData Documentation

[ https://issues.apache.org/jira/browse/SPARK-15973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-15973: Assignee: Josh Howes > Fix GroupedData Documentation > - > >

[jira] [Assigned] (SPARK-16051) Add `read.orc/write.orc` to SparkR

[ https://issues.apache.org/jira/browse/SPARK-16051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16051: Assignee: Apache Spark > Add `read.orc/write.orc` to SparkR >

[jira] [Assigned] (SPARK-16051) Add `read.orc/write.orc` to SparkR

[ https://issues.apache.org/jira/browse/SPARK-16051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16051: Assignee: (was: Apache Spark) > Add `read.orc/write.orc` to SparkR > -

[jira] [Commented] (SPARK-16051) Add `read.orc/write.orc` to SparkR

[ https://issues.apache.org/jira/browse/SPARK-16051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338187#comment-15338187 ] Apache Spark commented on SPARK-16051: -- User 'dongjoon-hyun' has created a pull requ

[jira] [Created] (SPARK-16051) Add `read.orc/write.orc` to SparkR

Dongjoon Hyun created SPARK-16051: - Summary: Add `read.orc/write.orc` to SparkR Key: SPARK-16051 URL: https://issues.apache.org/jira/browse/SPARK-16051 Project: Spark Issue Type: Improvement

[jira] [Commented] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338148#comment-15338148 ] Xiao Li commented on SPARK-16024: - {noformat} test("desc table for parquet data source

[jira] [Commented] (SPARK-6814) Support sorting for any data type in SparkR

[ https://issues.apache.org/jira/browse/SPARK-6814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338147#comment-15338147 ] Dongjoon Hyun commented on SPARK-6814: -- Hi, [~shivaram]. Since SparkR RDD is hiding f

[jira] [Commented] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338128#comment-15338128 ] Xiao Li commented on SPARK-16024: - Thanks! In Spark 2.0, the simplest solution is to put

[jira] [Commented] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338104#comment-15338104 ] Wenchen Fan commented on SPARK-16024: - yea go ahead, thanks! > column comment is ign

[jira] [Commented] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338096#comment-15338096 ] Xiao Li commented on SPARK-16024: - : ) Found a more serious bug in Json when reading the

[jira] [Assigned] (SPARK-14926) OneVsRest labelMetadata uses incorrect name

[ https://issues.apache.org/jira/browse/SPARK-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14926: Assignee: Apache Spark > OneVsRest labelMetadata uses incorrect name > ---

[jira] [Assigned] (SPARK-14926) OneVsRest labelMetadata uses incorrect name

[ https://issues.apache.org/jira/browse/SPARK-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-14926: Assignee: (was: Apache Spark) > OneVsRest labelMetadata uses incorrect name >

[jira] [Commented] (SPARK-14926) OneVsRest labelMetadata uses incorrect name

[ https://issues.apache.org/jira/browse/SPARK-14926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338092#comment-15338092 ] Apache Spark commented on SPARK-14926: -- User 'josh-howes' has created a pull request

[jira] [Created] (SPARK-16050) Flaky Test: Complete aggregation with Console sink

Burak Yavuz created SPARK-16050: --- Summary: Flaky Test: Complete aggregation with Console sink Key: SPARK-16050 URL: https://issues.apache.org/jira/browse/SPARK-16050 Project: Spark Issue Type:

[jira] [Commented] (SPARK-16006) Attemping to write empty DataFrame with no fields throw non-intuitive exception

[ https://issues.apache.org/jira/browse/SPARK-16006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338083#comment-15338083 ] Dongjoon Hyun commented on SPARK-16006: --- Hi, [~tdas]. The PR is updated, could you

[jira] [Resolved] (SPARK-16034) Checks the partition columns when calling dataFrame.write.mode("append").saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-16034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-16034. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13749 [https://github.com/

[jira] [Resolved] (SPARK-16037) use by-position resolution when insert into hive table

[ https://issues.apache.org/jira/browse/SPARK-16037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-16037. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13754 [https://github.com/

[jira] [Resolved] (SPARK-16036) better error message if the number of columns in SELECT clause doesn't match the table schema

[ https://issues.apache.org/jira/browse/SPARK-16036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai resolved SPARK-16036. -- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13754 [https://github.com/

[jira] [Updated] (SPARK-16034) Checks the partition columns when calling dataFrame.write.mode("append").saveAsTable

[ https://issues.apache.org/jira/browse/SPARK-16034?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-16034: - Assignee: Sean Zhong > Checks the partition columns when calling > dataFrame.write.mode("append").saveAs

[jira] [Commented] (SPARK-16032) Audit semantics of various insertion operations related to partitioned tables

[ https://issues.apache.org/jira/browse/SPARK-16032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338066#comment-15338066 ] Yin Huai commented on SPARK-16032: -- We will attach the report to here. > Audit semanti

[jira] [Created] (SPARK-16049) Make InsertIntoTable's expectedColumns support case-insensitive resolution properly

Yin Huai created SPARK-16049: Summary: Make InsertIntoTable's expectedColumns support case-insensitive resolution properly Key: SPARK-16049 URL: https://issues.apache.org/jira/browse/SPARK-16049 Project:

[jira] [Commented] (SPARK-16024) column comment is ignored for datasource table

[ https://issues.apache.org/jira/browse/SPARK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15338024#comment-15338024 ] Xiao Li commented on SPARK-16024: - Let me work on it? > column comment is ignored for da

[jira] [Updated] (SPARK-16047) Sort by status and id fields in Executors table

[ https://issues.apache.org/jira/browse/SPARK-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-16047: Description: With multiple executors with the same ID the default sorting *seems* to be by

[jira] [Updated] (SPARK-16048) spark-shell unresponsive after "FetchFailedException: java.lang.UnsupportedOperationException: Unsupported shuffle manager" with YARN and spark.shuffle.service.enabled

[ https://issues.apache.org/jira/browse/SPARK-16048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-16048: Description: With Spark on YARN with external shuffle service {{java.lang.UnsupportedOpera

[jira] [Created] (SPARK-16048) spark-shell unresponsive after "FetchFailedException: java.lang.UnsupportedOperationException: Unsupported shuffle manager" with YARN and spark.shuffle.service.enabled

Jacek Laskowski created SPARK-16048: --- Summary: spark-shell unresponsive after "FetchFailedException: java.lang.UnsupportedOperationException: Unsupported shuffle manager" with YARN and spark.shuffle.service.enabled Key: SPARK-1

[jira] [Updated] (SPARK-16047) Sort by status and id fields in Executors table

[ https://issues.apache.org/jira/browse/SPARK-16047?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-16047: Attachment: spark-webui-executors.png Current default sorting > Sort by status and id fiel

[jira] [Created] (SPARK-16047) Sort by status and id fields in Executors table

Jacek Laskowski created SPARK-16047: --- Summary: Sort by status and id fields in Executors table Key: SPARK-16047 URL: https://issues.apache.org/jira/browse/SPARK-16047 Project: Spark Issue T

[jira] [Updated] (SPARK-16046) Add Spark SQL Dataset Tutorial

[ https://issues.apache.org/jira/browse/SPARK-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pedro Rodriguez updated SPARK-16046: Description: Issue to update the Spark SQL guide to provide more content around using Data

[jira] [Commented] (SPARK-16046) Add Spark SQL Dataset Tutorial

[ https://issues.apache.org/jira/browse/SPARK-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337869#comment-15337869 ] Pedro Rodriguez commented on SPARK-16046: - I would like to take on this issue and

[jira] [Updated] (SPARK-16046) Add Spark SQL Dataset Tutorial

[ https://issues.apache.org/jira/browse/SPARK-16046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pedro Rodriguez updated SPARK-16046: Component/s: SQL Documentation > Add Spark SQL Dataset Tutorial >

[jira] [Created] (SPARK-16046) Add Spark SQL Dataset Tutorial

Pedro Rodriguez created SPARK-16046: --- Summary: Add Spark SQL Dataset Tutorial Key: SPARK-16046 URL: https://issues.apache.org/jira/browse/SPARK-16046 Project: Spark Issue Type: Documentatio

[jira] [Commented] (SPARK-12197) Kryo's Avro Serializer add support for dynamic schemas using SchemaRepository

[ https://issues.apache.org/jira/browse/SPARK-12197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337838#comment-15337838 ] Apache Spark commented on SPARK-12197: -- User 'RotemShaul' has created a pull request

[jira] [Commented] (SPARK-12947) Spark with Swift throws EOFException when reading parquet file

[ https://issues.apache.org/jira/browse/SPARK-12947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337808#comment-15337808 ] Ovidiu Marcu commented on SPARK-12947: -- Hi, do you filled an issue with Ceph for the

[jira] [Updated] (SPARK-14533) RowMatrix.computeCovariance inaccurate when values are very large

[ https://issues.apache.org/jira/browse/SPARK-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-14533: -- Target Version/s: (was: 2.0.0) > RowMatrix.computeCovariance inaccurate when values are very large >

[jira] [Commented] (SPARK-15909) PySpark classpath uri incorrectly set

[ https://issues.apache.org/jira/browse/SPARK-15909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337774#comment-15337774 ] Liam Fisk commented on SPARK-15909: --- Cluster mode isn't used here, I have a mesos clust

[jira] [Updated] (SPARK-14533) RowMatrix.computeCovariance inaccurate when values are very large

[ https://issues.apache.org/jira/browse/SPARK-14533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-14533: -- Target Version/s: 2.0.0 (was: 1.6.2, 2.0.0) > RowMatrix.computeCovariance inaccurate when values are v

[jira] [Resolved] (SPARK-15893) spark.createDataFrame raises an exception in Spark 2.0 tests on Windows

[ https://issues.apache.org/jira/browse/SPARK-15893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15893. --- Resolution: Duplicate Target Version/s: (was: 2.0.0) Same issue; there's a bit broader di

[jira] [Resolved] (SPARK-6817) DataFrame UDFs in R

[ https://issues.apache.org/jira/browse/SPARK-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-6817. -- Resolution: Done > DataFrame UDFs in R > --- > > Key: SPARK-6817 >

[jira] [Updated] (SPARK-15521) Add high level APIs based on dapply and gapply for easier usage

[ https://issues.apache.org/jira/browse/SPARK-15521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15521: -- Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-6817) > Add high level APIs ba

[jira] [Updated] (SPARK-16012) add gapplyCollect() for SparkDataFrame

[ https://issues.apache.org/jira/browse/SPARK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-16012: -- Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-6817) > add gapplyCollect() fo

[jira] [Commented] (SPARK-6817) DataFrame UDFs in R

[ https://issues.apache.org/jira/browse/SPARK-6817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337728#comment-15337728 ] Sean Owen commented on SPARK-6817: -- No, the best thing is just bulk-changing the issues t

[jira] [Updated] (SPARK-12923) Optimize successive dapply() calls in SparkR

[ https://issues.apache.org/jira/browse/SPARK-12923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-12923: -- Issue Type: Improvement (was: Sub-task) Parent: (was: SPARK-6817) > Optimize successive da

[jira] [Commented] (SPARK-15984) WARN message "o.a.h.y.s.resourcemanager.rmapp.RMAppImpl: The specific max attempts: 0 for application: 8 is invalid" when starting application on YARN

[ https://issues.apache.org/jira/browse/SPARK-15984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337693#comment-15337693 ] Jacek Laskowski commented on SPARK-15984: - The problem is that I am *not* changin

[jira] [Assigned] (SPARK-16012) add gapplyCollect() for SparkDataFrame

[ https://issues.apache.org/jira/browse/SPARK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16012: Assignee: (was: Apache Spark) > add gapplyCollect() for SparkDataFrame > -

[jira] [Assigned] (SPARK-16012) add gapplyCollect() for SparkDataFrame

[ https://issues.apache.org/jira/browse/SPARK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16012: Assignee: Apache Spark > add gapplyCollect() for SparkDataFrame >

[jira] [Commented] (SPARK-16012) add gapplyCollect() for SparkDataFrame

[ https://issues.apache.org/jira/browse/SPARK-16012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337669#comment-15337669 ] Apache Spark commented on SPARK-16012: -- User 'NarineK' has created a pull request fo

[jira] [Assigned] (SPARK-16045) Spark 2.0 ML.feature: doc update for stopwords and binarizer

[ https://issues.apache.org/jira/browse/SPARK-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16045: Assignee: Apache Spark > Spark 2.0 ML.feature: doc update for stopwords and binarizer > --

[jira] [Assigned] (SPARK-16045) Spark 2.0 ML.feature: doc update for stopwords and binarizer

[ https://issues.apache.org/jira/browse/SPARK-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16045: Assignee: (was: Apache Spark) > Spark 2.0 ML.feature: doc update for stopwords and bin

[jira] [Commented] (SPARK-16045) Spark 2.0 ML.feature: doc update for stopwords and binarizer

[ https://issues.apache.org/jira/browse/SPARK-16045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337625#comment-15337625 ] Apache Spark commented on SPARK-16045: -- User 'hhbyyh' has created a pull request for

[jira] [Created] (SPARK-16045) Spark 2.0 ML.feature: doc update for stopwords and binarizer

yuhao yang created SPARK-16045: -- Summary: Spark 2.0 ML.feature: doc update for stopwords and binarizer Key: SPARK-16045 URL: https://issues.apache.org/jira/browse/SPARK-16045 Project: Spark Iss

[jira] [Assigned] (SPARK-16044) input_file_name() returns empty strings in data sources based on NewHadoopRDD.

[ https://issues.apache.org/jira/browse/SPARK-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16044: Assignee: Apache Spark > input_file_name() returns empty strings in data sources based on

[jira] [Assigned] (SPARK-16044) input_file_name() returns empty strings in data sources based on NewHadoopRDD.

[ https://issues.apache.org/jira/browse/SPARK-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-16044: Assignee: (was: Apache Spark) > input_file_name() returns empty strings in data source

[jira] [Commented] (SPARK-16044) input_file_name() returns empty strings in data sources based on NewHadoopRDD.

[ https://issues.apache.org/jira/browse/SPARK-16044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15337624#comment-15337624 ] Apache Spark commented on SPARK-16044: -- User 'HyukjinKwon' has created a pull reques