[jira] [Resolved] (SPARK-15459) Make Range logical and physical explain consistent

[ https://issues.apache.org/jira/browse/SPARK-15459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-15459. - Resolution: Fixed Fix Version/s: 2.0.0 > Make Range logical and physical explain consisten

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295439#comment-15295439 ] Vishnu Prasad commented on SPARK-15467: --- Could this be be related to https://issue

[jira] [Commented] (SPARK-14343) Dataframe operations on a partitioned dataset (using partition discovery) return invalid results

[ https://issues.apache.org/jira/browse/SPARK-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295471#comment-15295471 ] Jurriaan Pruis commented on SPARK-14343: [~davies] This is still an issue on Spar

[jira] [Commented] (SPARK-15393) Writing empty Dataframes doesn't save any _metadata files

[ https://issues.apache.org/jira/browse/SPARK-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295477#comment-15295477 ] Jurriaan Pruis commented on SPARK-15393: [~hyukjin.kwon] I reproduced it again us

[jira] [Created] (SPARK-15472) Add support for the `text` format in FileStreamSink

Liwei Lin created SPARK-15472: - Summary: Add support for the `text` format in FileStreamSink Key: SPARK-15472 URL: https://issues.apache.org/jira/browse/SPARK-15472 Project: Spark Issue Type: Sub

[jira] [Commented] (SPARK-15393) Writing empty Dataframes doesn't save any _metadata files

[ https://issues.apache.org/jira/browse/SPARK-15393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295484#comment-15295484 ] Hyukjin Kwon commented on SPARK-15393: -- Ah.. thank you! I think the examples you wro

[jira] [Resolved] (SPARK-15428) Disable support for multiple streaming aggregations

[ https://issues.apache.org/jira/browse/SPARK-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tathagata Das resolved SPARK-15428. --- Resolution: Fixed Fix Version/s: 2.0.0 > Disable support for multiple streaming aggreg

[jira] [Commented] (SPARK-15472) Add support for partitioned `text` format in FileStreamSink

[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295487#comment-15295487 ] Apache Spark commented on SPARK-15472: -- User 'lw-lin' has created a pull request for

[jira] [Assigned] (SPARK-15472) Add support for partitioned `text` format in FileStreamSink

[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15472: Assignee: (was: Apache Spark) > Add support for partitioned `text` format in FileStrea

[jira] [Updated] (SPARK-15472) Add support for partitioned `text` format in FileStreamSink

[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-15472: -- Summary: Add support for partitioned `text` format in FileStreamSink (was: Add support for the `text`

[jira] [Assigned] (SPARK-15472) Add support for partitioned `text` format in FileStreamSink

[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15472: Assignee: Apache Spark > Add support for partitioned `text` format in FileStreamSink > ---

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295499#comment-15295499 ] Kazuaki Ishizaki commented on SPARK-15467: -- IMHO, this issue is due to Janino's

[jira] [Created] (SPARK-15473) CSV fails to write empty dataframe

Hyukjin Kwon created SPARK-15473: Summary: CSV fails to write empty dataframe Key: SPARK-15473 URL: https://issues.apache.org/jira/browse/SPARK-15473 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-15473) CSV fails to write empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295509#comment-15295509 ] Hyukjin Kwon commented on SPARK-15473: -- I will work on this. > CSV fails to write e

[jira] [Updated] (SPARK-15473) CSV fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hyukjin Kwon updated SPARK-15473: - Description: Currently CSV data source fails to write and read empty data. The code below: {cod

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295512#comment-15295512 ] Don Drake commented on SPARK-15467: --- Vishnu, the 22 field limitation is with Scala 2.10

[jira] [Assigned] (SPARK-15473) CSV fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15473: Assignee: (was: Apache Spark) > CSV fails to write and read back empty dataframe > ---

[jira] [Commented] (SPARK-15473) CSV fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295514#comment-15295514 ] Apache Spark commented on SPARK-15473: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-15473) CSV fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15473: Assignee: Apache Spark > CSV fails to write and read back empty dataframe > --

[jira] [Created] (SPARK-15474) ORC data source fails to write and read back empty dataframe

Hyukjin Kwon created SPARK-15474: Summary: ORC data source fails to write and read back empty dataframe Key: SPARK-15474 URL: https://issues.apache.org/jira/browse/SPARK-15474 Project: Spark

[jira] [Updated] (SPARK-15469) Datanode and nodemanger appear in jps but arent start correctly.

[ https://issues.apache.org/jira/browse/SPARK-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jon updated SPARK-15469: Description: when I execute jps command datanode and nodemanagger appears, but it seems that is not starting corre

[jira] [Created] (SPARK-15475) Add tests for writing and reading back empty data for Parquet, Json and Text data sources

Hyukjin Kwon created SPARK-15475: Summary: Add tests for writing and reading back empty data for Parquet, Json and Text data sources Key: SPARK-15475 URL: https://issues.apache.org/jira/browse/SPARK-15475

[jira] [Assigned] (SPARK-15475) Add tests for writing and reading back empty data for Parquet, Json and Text data sources

[ https://issues.apache.org/jira/browse/SPARK-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15475: Assignee: Apache Spark > Add tests for writing and reading back empty data for Parquet, Js

[jira] [Commented] (SPARK-15475) Add tests for writing and reading back empty data for Parquet, Json and Text data sources

[ https://issues.apache.org/jira/browse/SPARK-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295549#comment-15295549 ] Apache Spark commented on SPARK-15475: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-15475) Add tests for writing and reading back empty data for Parquet, Json and Text data sources

[ https://issues.apache.org/jira/browse/SPARK-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15475: Assignee: (was: Apache Spark) > Add tests for writing and reading back empty data for

[jira] [Created] (SPARK-15476) Support for reading text data source without specifying schema

Hyukjin Kwon created SPARK-15476: Summary: Support for reading text data source without specifying schema Key: SPARK-15476 URL: https://issues.apache.org/jira/browse/SPARK-15476 Project: Spark

[jira] [Resolved] (SPARK-15430) Access ListAccumulator's value could possibly cause java.util.ConcurrentModificationException

[ https://issues.apache.org/jira/browse/SPARK-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15430. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13211 [https://github.co

[jira] [Updated] (SPARK-15430) Access ListAccumulator's value could possibly cause java.util.ConcurrentModificationException

[ https://issues.apache.org/jira/browse/SPARK-15430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15430: -- Assignee: Liang-Chi Hsieh > Access ListAccumulator's value could possibly cause > java.util.Concurrent

[jira] [Resolved] (SPARK-15468) fix some typos while browsing the codes

[ https://issues.apache.org/jira/browse/SPARK-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15468. --- Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13246 [https://github.co

[jira] [Updated] (SPARK-15468) fix some typos while browsing the codes

[ https://issues.apache.org/jira/browse/SPARK-15468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-15468: -- Assignee: Bo Meng Priority: Trivial (was: Minor) > fix some typos while browsing the codes > -

[jira] [Resolved] (SPARK-15469) Datanode and nodemanger appear in jps but arent start correctly.

[ https://issues.apache.org/jira/browse/SPARK-15469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-15469. --- Resolution: Invalid HDFS/YARN question, not Spark > Datanode and nodemanger appear in jps but arent

[jira] [Commented] (SPARK-15475) Add tests for writing and reading back empty data for Parquet, Json and Text data sources

[ https://issues.apache.org/jira/browse/SPARK-15475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295563#comment-15295563 ] Apache Spark commented on SPARK-15475: -- User 'HyukjinKwon' has created a pull reques

[jira] [Assigned] (SPARK-15429) When `spark.streaming.concurrentJobs > 1`, PIDRateEstimator cannot estimate the receiving rate accurately.

[ https://issues.apache.org/jira/browse/SPARK-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15429: Assignee: (was: Apache Spark) > When `spark.streaming.concurrentJobs > 1`, PIDRateEsti

[jira] [Commented] (SPARK-15429) When `spark.streaming.concurrentJobs > 1`, PIDRateEstimator cannot estimate the receiving rate accurately.

[ https://issues.apache.org/jira/browse/SPARK-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295589#comment-15295589 ] Apache Spark commented on SPARK-15429: -- User 'chengqiangboy' has created a pull requ

[jira] [Assigned] (SPARK-15429) When `spark.streaming.concurrentJobs > 1`, PIDRateEstimator cannot estimate the receiving rate accurately.

[ https://issues.apache.org/jira/browse/SPARK-15429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15429: Assignee: Apache Spark > When `spark.streaming.concurrentJobs > 1`, PIDRateEstimator canno

[jira] [Updated] (SPARK-15282) UDF function executed twice when filter on new column created by withColumn

[ https://issues.apache.org/jira/browse/SPARK-15282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Linbo updated SPARK-15282: -- Summary: UDF function executed twice when filter on new column created by withColumn (was: UDF functioin execu

[jira] [Commented] (SPARK-15467) Getting stack overflow when attempting to query a wide Dataset (>200 fields)

[ https://issues.apache.org/jira/browse/SPARK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295668#comment-15295668 ] Reynold Xin commented on SPARK-15467: - [~kiszk] would you like to submit a patch agai

[jira] [Commented] (SPARK-15474) ORC data source fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295750#comment-15295750 ] Apache Spark commented on SPARK-15474: -- User 'sbcd90' has created a pull request for

[jira] [Assigned] (SPARK-15474) ORC data source fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15474: Assignee: (was: Apache Spark) > ORC data source fails to write and read back empty da

[jira] [Assigned] (SPARK-15474) ORC data source fails to write and read back empty dataframe

[ https://issues.apache.org/jira/browse/SPARK-15474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-15474: Assignee: Apache Spark > ORC data source fails to write and read back empty dataframe > -

[jira] [Commented] (SPARK-12013) Add a Hive context method to retrieve the database Names

[ https://issues.apache.org/jira/browse/SPARK-12013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295786#comment-15295786 ] Doug Balog commented on SPARK-12013: In 2.0, you can use `SparkSession.catalog.listD

[jira] [Created] (SPARK-15477) HiveContext is private[hive] and not accessible to users.

Doug Balog created SPARK-15477: -- Summary: HiveContext is private[hive] and not accessible to users. Key: SPARK-15477 URL: https://issues.apache.org/jira/browse/SPARK-15477 Project: Spark Issue

[jira] [Commented] (SPARK-15078) Add all TPCDS 1.4 benchmark queries for SparkSQL

[ https://issues.apache.org/jira/browse/SPARK-15078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295790#comment-15295790 ] Ovidiu Marcu commented on SPARK-15078: -- Running tpcds1.4 queries (using https://gith

[jira] [Commented] (SPARK-15078) Add all TPCDS 1.4 benchmark queries for SparkSQL

[ https://issues.apache.org/jira/browse/SPARK-15078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295794#comment-15295794 ] Ovidiu Marcu commented on SPARK-15078: -- Same exceptions with a second run: val exper

[jira] [Commented] (SPARK-15396) [Spark] [SQL] [DOC] It can't connect hive metastore database

[ https://issues.apache.org/jira/browse/SPARK-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295836#comment-15295836 ] Yi Zhou commented on SPARK-15396: - Hi , even if setting 'spark.sql.warehouse.dir' in spa

[jira] [Comment Edited] (SPARK-15396) [Spark] [SQL] [DOC] It can't connect hive metastore database

[ https://issues.apache.org/jira/browse/SPARK-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295836#comment-15295836 ] Yi Zhou edited comment on SPARK-15396 at 5/23/16 2:06 AM: -- Hi ,

[jira] [Resolved] (SPARK-15379) String to date and timestamp conversion should return null on some invalid date string

[ https://issues.apache.org/jira/browse/SPARK-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-15379. - Resolution: Fixed Fix Version/s: 2.0.0 Issue resolved by pull request 13169 [https://githu

[jira] [Updated] (SPARK-15379) String to date and timestamp conversion should return null on some invalid date string

[ https://issues.apache.org/jira/browse/SPARK-15379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-15379: Assignee: Yang Wang > String to date and timestamp conversion should return null on some invalid >

[jira] [Commented] (SPARK-15396) [Spark] [SQL] [DOC] It can't connect hive metastore database

[ https://issues.apache.org/jira/browse/SPARK-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295874#comment-15295874 ] Yi Zhou commented on SPARK-15396: - Add, 1) Spark SQL can't find existing hive databases i

[jira] [Closed] (SPARK-15466) Make `SparkSession` as the entry point to programming with RDD too

[ https://issues.apache.org/jira/browse/SPARK-15466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dongjoon Hyun closed SPARK-15466. - Resolution: Invalid Spark is trying to avoid returning RDDs in the new apis. This PR seems to cro

[jira] [Commented] (SPARK-1272) Don't fail job if some local directories are buggy

[ https://issues.apache.org/jira/browse/SPARK-1272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295888#comment-15295888 ] He Tianyi commented on SPARK-1272: -- BTW, is there a good approach to skip failed disk if

[jira] [Commented] (SPARK-15396) [Spark] [SQL] [DOC] It can't connect hive metastore database

[ https://issues.apache.org/jira/browse/SPARK-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295899#comment-15295899 ] Xiao Li commented on SPARK-15396: - Uh,,, I know what you are talking about. You need to

[jira] [Updated] (SPARK-12992) Vectorize parquet decoding using ColumnarBatch

[ https://issues.apache.org/jira/browse/SPARK-12992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin updated SPARK-12992: Assignee: Nong Li (was: Apache Spark) > Vectorize parquet decoding using ColumnarBatch > -

[jira] [Commented] (SPARK-15396) [Spark] [SQL] [DOC] It can't connect hive metastore database

[ https://issues.apache.org/jira/browse/SPARK-15396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295920#comment-15295920 ] Yi Zhou commented on SPARK-15396: - these parameters worked well in Spark 1.6.1 but no sur

[jira] [Commented] (SPARK-15449) Wrong Data Format - Documentation Issue

[ https://issues.apache.org/jira/browse/SPARK-15449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295934#comment-15295934 ] Miao Wang commented on SPARK-15449: --- [~sowen] If it is Java example only, let me make i

[jira] [Commented] (SPARK-15472) Add support for partitioned `text` format in FileStreamSink

[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15295953#comment-15295953 ] Apache Spark commented on SPARK-15472: -- User 'lw-lin' has created a pull request for

[jira] [Updated] (SPARK-15472) Add partitioned `csv`, `json`, `text` format support for FileStreamSink

[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-15472: -- Summary: Add partitioned `csv`, `json`, `text` format support for FileStreamSink (was: Add support for

[jira] [Updated] (SPARK-15472) Add partitioned `csv`, `json`, `text` format support for FileStreamSink

[ https://issues.apache.org/jira/browse/SPARK-15472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liwei Lin updated SPARK-15472: -- Description: Support for partitioned `parquet` format in FileStreamSink was added in Spark-14716, now l

[jira] [Resolved] (SPARK-15425) Disallow cartesian joins by default

[ https://issues.apache.org/jira/browse/SPARK-15425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-15425. - Resolution: Fixed Assignee: Sameer Agarwal Fix Version/s: 2.0.0 > Disallow cartes

[jira] [Created] (SPARK-15478) LogisticRegressionModel 's Coefficients always be a empty vector

lichenglin created SPARK-15478: -- Summary: LogisticRegressionModel 's Coefficients always be a empty vector Key: SPARK-15478 URL: https://issues.apache.org/jira/browse/SPARK-15478 Project: Spark

[jira] [Updated] (SPARK-15478) LogisticRegressionModel 's Coefficients always be a empty vector

[ https://issues.apache.org/jira/browse/SPARK-15478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lichenglin updated SPARK-15478: --- Description: I don't know if this is a bug. I'm runing the sample code like this {code} publi

[jira] [Created] (SPARK-15479) Spark job doesn't shut gracefully in yarn mode.

Rakesh created SPARK-15479: -- Summary: Spark job doesn't shut gracefully in yarn mode. Key: SPARK-15479 URL: https://issues.apache.org/jira/browse/SPARK-15479 Project: Spark Issue Type: Bug