[jira] [Created] (SPARK-20371) R wrappers for collect_list and collect_set

2017-04-18 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-20371: -- Summary: R wrappers for collect_list and collect_set Key: SPARK-20371 URL: https://issues.apache.org/jira/browse/SPARK-20371 Project: Spark Issue

[jira] [Commented] (SPARK-15816) SQL server based on Postgres protocol

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-15816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972971#comment-15972971 ] Takeshi Yamamuro commented on SPARK-15816: -- I put the prototype a little forward

[jira] [Created] (SPARK-20370) create external table on read only location fails

2017-04-18 Thread Gaurav Shah (JIRA)
Gaurav Shah created SPARK-20370: --- Summary: create external table on read only location fails Key: SPARK-20370 URL: https://issues.apache.org/jira/browse/SPARK-20370 Project: Spark Issue Type: B

[jira] [Commented] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972914#comment-15972914 ] Apache Spark commented on SPARK-20368: -- User 'kxepal' has created a pull request for

[jira] [Assigned] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20368: Assignee: (was: Apache Spark) > Support Sentry on PySpark workers > --

[jira] [Assigned] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20368: Assignee: Apache Spark > Support Sentry on PySpark workers > -

[jira] [Created] (SPARK-20369) pyspark: Dynamic configuration with SparkConf does not work

2017-04-18 Thread Matthew McClain (JIRA)
Matthew McClain created SPARK-20369: --- Summary: pyspark: Dynamic configuration with SparkConf does not work Key: SPARK-20369 URL: https://issues.apache.org/jira/browse/SPARK-20369 Project: Spark

[jira] [Created] (SPARK-20368) Support Sentry on PySpark workers

2017-04-18 Thread Alexander Shorin (JIRA)
Alexander Shorin created SPARK-20368: Summary: Support Sentry on PySpark workers Key: SPARK-20368 URL: https://issues.apache.org/jira/browse/SPARK-20368 Project: Spark Issue Type: New Fea

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972871#comment-15972871 ] Hyukjin Kwon commented on SPARK-20367: -- Doh. I rushed reading ... > Spark silently

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Juliusz Sompolski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972867#comment-15972867 ] Juliusz Sompolski commented on SPARK-20367: --- Hi [~hyukjin.kwon]. I tested also

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972865#comment-15972865 ] Hyukjin Kwon commented on SPARK-20367: -- Actually, I did while trying to reproduce th

[jira] [Commented] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972850#comment-15972850 ] Hyukjin Kwon commented on SPARK-20367: -- I guess probably this is not a CSV datasourc

[jira] [Commented] (SPARK-20343) SBT master build for Hadoop 2.6 in Jenkins fails due to Avro version resolution

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972842#comment-15972842 ] Hyukjin Kwon commented on SPARK-20343: -- Please let me know if anyone is able to repr

[jira] [Commented] (SPARK-6509) MDLP discretizer

2017-04-18 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972806#comment-15972806 ] Sergio Ramírez commented on SPARK-6509: --- Thanks again Barry for your support. I hope

[jira] [Created] (SPARK-20367) Spark silently escapes partition column names

2017-04-18 Thread Juliusz Sompolski (JIRA)
Juliusz Sompolski created SPARK-20367: - Summary: Spark silently escapes partition column names Key: SPARK-20367 URL: https://issues.apache.org/jira/browse/SPARK-20367 Project: Spark Issue

[jira] [Assigned] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20281: Assignee: Apache Spark > Table-valued function range in SQL should use the same number of

[jira] [Assigned] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20281: Assignee: (was: Apache Spark) > Table-valued function range in SQL should use the same

[jira] [Commented] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972758#comment-15972758 ] Apache Spark commented on SPARK-20281: -- User 'maropu' has created a pull request for

[jira] [Commented] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972753#comment-15972753 ] Herman van Hovell commented on SPARK-20356: --- Here is a reproduction in scala: {

[jira] [Commented] (SPARK-20281) Table-valued function range in SQL should use the same number of partitions as spark.range

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972756#comment-15972756 ] Takeshi Yamamuro commented on SPARK-20281: -- IIUC they internally use the same va

[jira] [Commented] (SPARK-6509) MDLP discretizer

2017-04-18 Thread Barry Becker (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972744#comment-15972744 ] Barry Becker commented on SPARK-6509: - As further proof of relevance, I will be giving

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20366: --- Assignee: Zhenhua Wang > Fix recursive join reordering: inside joins are not reordered > ---

[jira] [Resolved] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20366. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17668 [https://githu

[jira] [Commented] (SPARK-20356) Spark sql group by returns incorrect results after join + distinct transformations

2017-04-18 Thread Ed Lee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972544#comment-15972544 ] Ed Lee commented on SPARK-20356: really quite dangerous bug > Spark sql group by returns

[jira] [Commented] (SPARK-20343) SBT master build for Hadoop 2.6 in Jenkins fails due to Avro version resolution

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972490#comment-15972490 ] Apache Spark commented on SPARK-20343: -- User 'HyukjinKwon' has created a pull reques

[jira] [Commented] (SPARK-1548) Add Partial Random Forest algorithm to MLlib

2017-04-18 Thread Mohamed Baddar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-1548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972474#comment-15972474 ] Mohamed Baddar commented on SPARK-1548: --- [~srowen] [~josephkb] any updates on the po

[jira] [Assigned] (SPARK-20344) Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20344: - Assignee: Robert Stupp > Duplicate call in FairSchedulableBuilder.addTaskSetManager > --

[jira] [Resolved] (SPARK-20344) Duplicate call in FairSchedulableBuilder.addTaskSetManager

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20344. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17647 [https://github.co

[jira] [Resolved] (SPARK-20361) JVM locale affects SQL type names

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20361. --- Resolution: Duplicate > JVM locale affects SQL type names > -- > >

[jira] [Reopened] (SPARK-20361) JVM locale affects SQL type names

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-20361: --- > JVM locale affects SQL type names > -- > > Key: SPARK-

[jira] [Commented] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972400#comment-15972400 ] Robert Kruszewski commented on SPARK-20364: --- Looks like parquet doesn't differe

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972401#comment-15972401 ] Takeshi Yamamuro commented on SPARK-20169: -- I also could reproduce this on /bin/

[jira] [Comment Edited] (SPARK-20364) Parquet predicate pushdown on columns with dots return empty results

2017-04-18 Thread Robert Kruszewski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972400#comment-15972400 ] Robert Kruszewski edited comment on SPARK-20364 at 4/18/17 9:36 AM: ---

[jira] [Resolved] (SPARK-20363) sessionstate.get is get the same object in hive project, when I use spark-beeline

2017-04-18 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20363. --- Resolution: Invalid This is very unclear. Add a comment if you can significantly clarify what this i

[jira] [Commented] (SPARK-19995) Using real user to connect HiveMetastore in HiveClientImpl

2017-04-18 Thread meiyoula (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972373#comment-15972373 ] meiyoula commented on SPARK-19995: -- Will the token be expired? > Using real user to con

[jira] [Comment Edited] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972359#comment-15972359 ] Takeshi Yamamuro edited comment on SPARK-20174 at 4/18/17 9:09 AM:

[jira] [Commented] (SPARK-20174) Analyzer gives mysterious AnalysisException when posexplode used in withColumn

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972359#comment-15972359 ] Takeshi Yamamuro commented on SPARK-20174: -- You could fix this like https://git

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Takeshi Yamamuro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972300#comment-15972300 ] Takeshi Yamamuro commented on SPARK-20169: -- oh, ... good work. > Groupby Bug wi

[jira] [Comment Edited] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972270#comment-15972270 ] Hyukjin Kwon edited comment on SPARK-20169 at 4/18/17 7:50 AM:

[jira] [Commented] (SPARK-20169) Groupby Bug with Sparksql

2017-04-18 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972270#comment-15972270 ] Hyukjin Kwon commented on SPARK-20169: -- Yea, I was confused too when I tried to repr

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: {noformat} Join

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-18 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972253#comment-15972253 ] Miguel Pérez commented on SPARK-20286: -- Thank you! I'll check it again and close the

[jira] [Commented] (SPARK-20320) AnalysisException: Columns of grouping_id (count(value#17L)) does not match grouping columns (count(value#17L))

2017-04-18 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972249#comment-15972249 ] Jacek Laskowski commented on SPARK-20320: - I'm playing with Spark SQL and multi-d

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: Join / \

[jira] [Updated] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhenhua Wang updated SPARK-20366: - Description: If a plan has multi-level successive joins, e.g.: ``` Join / \

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20366: Assignee: (was: Apache Spark) > Fix recursive join reordering: inside joins are not re

[jira] [Commented] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972243#comment-15972243 ] Apache Spark commented on SPARK-20366: -- User 'wzhfy' has created a pull request for

[jira] [Assigned] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20366: Assignee: Apache Spark > Fix recursive join reordering: inside joins are not reordered > -

[jira] [Created] (SPARK-20366) Fix recursive join reordering: inside joins are not reordered

2017-04-18 Thread Zhenhua Wang (JIRA)
Zhenhua Wang created SPARK-20366: Summary: Fix recursive join reordering: inside joins are not reordered Key: SPARK-20366 URL: https://issues.apache.org/jira/browse/SPARK-20366 Project: Spark

[jira] [Updated] (SPARK-20365) Not so accurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saisai Shao updated SPARK-20365: Summary: Not so accurate classpath format for AM and Containers (was: Inaccurate classpath format

[jira] [Commented] (SPARK-20286) dynamicAllocation.executorIdleTimeout is ignored after unpersist

2017-04-18 Thread Umesh Chaudhary (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15972231#comment-15972231 ] Umesh Chaudhary commented on SPARK-20286: - Yep, +1 to the UI changes. However, I

[jira] [Created] (SPARK-20365) Inaccurate classpath format for AM and Containers

2017-04-18 Thread Saisai Shao (JIRA)
Saisai Shao created SPARK-20365: --- Summary: Inaccurate classpath format for AM and Containers Key: SPARK-20365 URL: https://issues.apache.org/jira/browse/SPARK-20365 Project: Spark Issue Type: B

<    1   2