[jira] [Commented] (SPARK-20334) Return a better error message when correlated predicates contain aggregate expression that has mixture of outer and local references

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968650#comment-15968650 ] Apache Spark commented on SPARK-20334: -- User 'dilipbiswal' has created a pull request for this

[jira] [Assigned] (SPARK-20334) Return a better error message when correlated predicates contain aggregate expression that has mixture of outer and local references

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20334: Assignee: Apache Spark > Return a better error message when correlated predicates contain

[jira] [Assigned] (SPARK-20334) Return a better error message when correlated predicates contain aggregate expression that has mixture of outer and local references

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20334: Assignee: (was: Apache Spark) > Return a better error message when correlated

[jira] [Assigned] (SPARK-20335) Children expressions of Hive UDF impacts the determinism of Hive UDF

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20335: Assignee: Apache Spark (was: Xiao Li) > Children expressions of Hive UDF impacts the

[jira] [Assigned] (SPARK-20335) Children expressions of Hive UDF impacts the determinism of Hive UDF

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20335: Assignee: Xiao Li (was: Apache Spark) > Children expressions of Hive UDF impacts the

[jira] [Commented] (SPARK-20335) Children expressions of Hive UDF impacts the determinism of Hive UDF

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968639#comment-15968639 ] Apache Spark commented on SPARK-20335: -- User 'gatorsmile' has created a pull request for this issue:

[jira] [Created] (SPARK-20335) Children expressions of Hive UDF impacts the determinism of Hive UDF

2017-04-13 Thread Xiao Li (JIRA)
Xiao Li created SPARK-20335: --- Summary: Children expressions of Hive UDF impacts the determinism of Hive UDF Key: SPARK-20335 URL: https://issues.apache.org/jira/browse/SPARK-20335 Project: Spark

[jira] [Created] (SPARK-20334) Return a better error message when correlated predicates contain aggregate expression that has mixture of outer and local references

2017-04-13 Thread Dilip Biswal (JIRA)
Dilip Biswal created SPARK-20334: Summary: Return a better error message when correlated predicates contain aggregate expression that has mixture of outer and local references Key: SPARK-20334 URL:

[jira] [Updated] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

2017-04-13 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20333: - Description: In test "don't submit stage until its dependencies map outputs are registered

[jira] [Assigned] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20333: Assignee: Apache Spark > Fix HashPartitioner in DAGSchedulerSuite >

[jira] [Assigned] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20333: Assignee: (was: Apache Spark) > Fix HashPartitioner in DAGSchedulerSuite >

[jira] [Commented] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968576#comment-15968576 ] Apache Spark commented on SPARK-20333: -- User 'jinxing64' has created a pull request for this issue:

[jira] [Updated] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

2017-04-13 Thread jin xing (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jin xing updated SPARK-20333: - Description: In test "don't submit stage until its dependencies map outputs are registered

[jira] [Created] (SPARK-20333) Fix HashPartitioner in DAGSchedulerSuite

2017-04-13 Thread jin xing (JIRA)
jin xing created SPARK-20333: Summary: Fix HashPartitioner in DAGSchedulerSuite Key: SPARK-20333 URL: https://issues.apache.org/jira/browse/SPARK-20333 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-16742) Kerberos support for Spark on Mesos

2017-04-13 Thread Saisai Shao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968545#comment-15968545 ] Saisai Shao commented on SPARK-16742: - [~mgummelt], do you have a design doc of the kerberos support

[jira] [Created] (SPARK-20332) Avro/Parquet GenericFixed decimal is not read into Spark correctly

2017-04-13 Thread Justin Pihony (JIRA)
Justin Pihony created SPARK-20332: - Summary: Avro/Parquet GenericFixed decimal is not read into Spark correctly Key: SPARK-20332 URL: https://issues.apache.org/jira/browse/SPARK-20332 Project: Spark

[jira] [Resolved] (SPARK-20293) In the page of 'jobs' or 'stages' of history server web ui,,click the 'Go' button, query paging data, the page error

2017-04-13 Thread Alex Bozarth (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alex Bozarth resolved SPARK-20293. -- Resolution: Duplicate Already fixed in master and branch-2.1 > In the page of 'jobs' or

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968474#comment-15968474 ] Marcelo Vanzin commented on SPARK-20328: bq. Since the driver is authenticated, it can request

[jira] [Commented] (SPARK-20293) In the page of 'jobs' or 'stages' of history server web ui,,click the 'Go' button, query paging data, the page error

2017-04-13 Thread Hyukjin Kwon (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968473#comment-15968473 ] Hyukjin Kwon commented on SPARK-20293: -- I can't reproduce in the current master. See -

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968469#comment-15968469 ] Michael Gummelt commented on SPARK-20328: - bq. I have no idea what that means. I'm pretty sure a

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968440#comment-15968440 ] Marcelo Vanzin commented on SPARK-20328: bq. that the driver is already logged in via the

[jira] [Assigned] (SPARK-20331) Broaden support for Hive partition pruning predicate pushdown

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20331: Assignee: Apache Spark > Broaden support for Hive partition pruning predicate pushdown >

[jira] [Assigned] (SPARK-20331) Broaden support for Hive partition pruning predicate pushdown

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20331: Assignee: (was: Apache Spark) > Broaden support for Hive partition pruning predicate

[jira] [Commented] (SPARK-20331) Broaden support for Hive partition pruning predicate pushdown

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968439#comment-15968439 ] Apache Spark commented on SPARK-20331: -- User 'mallman' has created a pull request for this issue:

[jira] [Updated] (SPARK-20331) Broaden support for Hive partition pruning predicate pushdown

2017-04-13 Thread Michael Allman (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Allman updated SPARK-20331: --- Description: Spark 2.1 introduced scalable support for Hive tables with huge numbers of

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968432#comment-15968432 ] Michael Gummelt commented on SPARK-20328: - bq. It depends. e.g. on YARN, when you submit in

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968423#comment-15968423 ] Marcelo Vanzin commented on SPARK-20328: bq. But it shouldn't need delegation tokens at all,

[jira] [Comment Edited] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968416#comment-15968416 ] Michael Gummelt edited comment on SPARK-20328 at 4/14/17 12:02 AM: --- bq.

[jira] [Comment Edited] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968411#comment-15968411 ] Michael Gummelt edited comment on SPARK-20328 at 4/13/17 11:59 PM: --- bq.

[jira] [Comment Edited] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968416#comment-15968416 ] Michael Gummelt edited comment on SPARK-20328 at 4/13/17 11:59 PM: --- bq.

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968416#comment-15968416 ] Michael Gummelt commented on SPARK-20328: - > It shouldn't need to do it not for the reasons you

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968411#comment-15968411 ] Michael Gummelt commented on SPARK-20328: - > The Mesos backend (I mean the code in Spark, not the

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968396#comment-15968396 ] Marcelo Vanzin commented on SPARK-20328: bq. The problem can't be solved in the Mesos backend I

[jira] [Comment Edited] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968388#comment-15968388 ] Michael Gummelt edited comment on SPARK-20328 at 4/13/17 11:27 PM: --- Hey

[jira] [Comment Edited] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968388#comment-15968388 ] Michael Gummelt edited comment on SPARK-20328 at 4/13/17 11:27 PM: --- Hey

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968388#comment-15968388 ] Michael Gummelt commented on SPARK-20328: - Hey [~vanzin], thanks for the response. Everything

[jira] [Closed] (SPARK-20330) CLONE - SparkContext.localProperties leaked

2017-04-13 Thread Oleg White (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Oleg White closed SPARK-20330. -- Resolution: Duplicate > CLONE - SparkContext.localProperties leaked >

[jira] [Created] (SPARK-20331) Broaden support for Hive partition pruning predicate pushdown

2017-04-13 Thread Michael Allman (JIRA)
Michael Allman created SPARK-20331: -- Summary: Broaden support for Hive partition pruning predicate pushdown Key: SPARK-20331 URL: https://issues.apache.org/jira/browse/SPARK-20331 Project: Spark

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968343#comment-15968343 ] Marcelo Vanzin commented on SPARK-20328: Hmm... that seems related to delegation token support.

[jira] [Updated] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt updated SPARK-20328: Description: In order to obtain {{InputSplit}} information, {{HadoopRDD}} creates a

[jira] [Created] (SPARK-20330) CLONE - SparkContext.localProperties leaked

2017-04-13 Thread Oleg White (JIRA)
Oleg White created SPARK-20330: -- Summary: CLONE - SparkContext.localProperties leaked Key: SPARK-20330 URL: https://issues.apache.org/jira/browse/SPARK-20330 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-20329) Resolution error when HAVING clause uses GROUP BY expression that involves implicit type coercion

2017-04-13 Thread Josh Rosen (JIRA)
Josh Rosen created SPARK-20329: -- Summary: Resolution error when HAVING clause uses GROUP BY expression that involves implicit type coercion Key: SPARK-20329 URL: https://issues.apache.org/jira/browse/SPARK-20329

[jira] [Updated] (SPARK-16900) Complete-mode output for file sinks

2017-04-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-16900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-16900: - Component/s: (was: DStreams) Structured Streaming > Complete-mode output

[jira] [Updated] (SPARK-20312) query optimizer calls udf with null values when it doesn't expect them

2017-04-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu updated SPARK-20312: - Component/s: (was: Spark Core) SQL > query optimizer calls udf with null

[jira] [Assigned] (SPARK-18127) Add hooks and extension points to Spark

2017-04-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-18127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell reassigned SPARK-18127: - Assignee: Sameer Agarwal (was: Herman van Hovell) > Add hooks and extension

[jira] [Commented] (SPARK-20321) Spark UI cannot be shutdown in spark streaming app

2017-04-13 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968290#comment-15968290 ] Shixiong Zhu commented on SPARK-20321: -- You cannot stop a StreamingContext in foreachRDD. For

[jira] [Updated] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Gummelt updated SPARK-20328: Description: In order to obtain {{InputSplit}} information, {{HadoopRDD}} creates a

[jira] [Commented] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968232#comment-15968232 ] Michael Gummelt commented on SPARK-20328: - cc [~colorant] [~hfeng] [~vanzin] > HadoopRDDs create

[jira] [Created] (SPARK-20328) HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs

2017-04-13 Thread Michael Gummelt (JIRA)
Michael Gummelt created SPARK-20328: --- Summary: HadoopRDDs create a MapReduce JobConf, but are not MapReduce jobs Key: SPARK-20328 URL: https://issues.apache.org/jira/browse/SPARK-20328 Project:

[jira] [Assigned] (SPARK-20038) FileFormatWriter.ExecuteWriteTask.releaseResources() implementations to be re-entrant

2017-04-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid reassigned SPARK-20038: Assignee: Steve Loughran > FileFormatWriter.ExecuteWriteTask.releaseResources()

[jira] [Resolved] (SPARK-20038) FileFormatWriter.ExecuteWriteTask.releaseResources() implementations to be re-entrant

2017-04-13 Thread Imran Rashid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Imran Rashid resolved SPARK-20038. -- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17364

[jira] [Commented] (SPARK-20327) Add CLI support for YARN-3926

2017-04-13 Thread Mark Grover (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968149#comment-15968149 ] Mark Grover commented on SPARK-20327: - Daniel, we don't assign JIRAs in Spark. Folks issue a PR and

[jira] [Assigned] (SPARK-20232) Better combineByKey documentation: clarify memory allocation, better example

2017-04-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk reassigned SPARK-20232: --- Assignee: David Gingrich > Better combineByKey documentation: clarify memory allocation, better

[jira] [Resolved] (SPARK-20232) Better combineByKey documentation: clarify memory allocation, better example

2017-04-13 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] holdenk resolved SPARK-20232. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17545

[jira] [Commented] (SPARK-19909) Batches will fail in case that temporary checkpoint dir is on local file system while metadata dir is on HDFS

2017-04-13 Thread Rob (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15968092#comment-15968092 ] Rob commented on SPARK-19909: - Is there an easy way to avoid this issue while waiting for it to be resolved?

[jira] [Updated] (SPARK-19946) DebugFilesystem.assertNoOpenStreams should report the open streams to help debugging

2017-04-13 Thread Herman van Hovell (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Herman van Hovell updated SPARK-19946: -- Fix Version/s: 2.1.1 > DebugFilesystem.assertNoOpenStreams should report the open

[jira] [Commented] (SPARK-20327) Add CLI support for YARN-3926

2017-04-13 Thread Daniel Templeton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967913#comment-15967913 ] Daniel Templeton commented on SPARK-20327: -- YARN-3926 won't go in until Hadoop 3.0.0, so this

[jira] [Comment Edited] (SPARK-14519) Cross-publish Kafka for Scala 2.12

2017-04-13 Thread Fabio Pinheiro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967897#comment-15967897 ] Fabio Pinheiro edited comment on SPARK-14519 at 4/13/17 5:12 PM: - Kafka

[jira] [Commented] (SPARK-14519) Cross-publish Kafka for Scala 2.12

2017-04-13 Thread Fabio Pinheiro (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-14519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967897#comment-15967897 ] Fabio Pinheiro commented on SPARK-14519: Kafka official supports Scala 2.12 since '0.10.1.1' and

[jira] [Commented] (SPARK-20327) Add CLI support for YARN-3926

2017-04-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967886#comment-15967886 ] Sean Owen commented on SPARK-20327: --- What version of YARN would this require? Spark is still supporting

[jira] [Updated] (SPARK-20327) Add CLI support for YARN-3926

2017-04-13 Thread Daniel Templeton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated SPARK-20327: - Labels: newbie (was: ) > Add CLI support for YARN-3926 > -

[jira] [Commented] (SPARK-20327) Add CLI support for YARN-3926

2017-04-13 Thread Daniel Templeton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967855#comment-15967855 ] Daniel Templeton commented on SPARK-20327: -- [~mgrover], would you mind assigning this one to me?

[jira] [Updated] (SPARK-20327) Add CLI support for YARN-3926

2017-04-13 Thread Daniel Templeton (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Templeton updated SPARK-20327: - Description: YARN-3926 adds the ability for administrators to configure custom

[jira] [Created] (SPARK-20327) Add CLI support for YARN-3926

2017-04-13 Thread Daniel Templeton (JIRA)
Daniel Templeton created SPARK-20327: Summary: Add CLI support for YARN-3926 Key: SPARK-20327 URL: https://issues.apache.org/jira/browse/SPARK-20327 Project: Spark Issue Type:

[jira] [Resolved] (SPARK-20326) Run Spark in local mode from IntelliJ (or other IDE)

2017-04-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20326. --- Resolution: Invalid Questions belong on the mailing list u...@spark.apache.org > Run Spark in local

[jira] [Created] (SPARK-20326) Run Spark in local mode from IntelliJ (or other IDE)

2017-04-13 Thread Marcello Oliva (JIRA)
Marcello Oliva created SPARK-20326: -- Summary: Run Spark in local mode from IntelliJ (or other IDE) Key: SPARK-20326 URL: https://issues.apache.org/jira/browse/SPARK-20326 Project: Spark

[jira] [Commented] (SPARK-8586) SQL add jar command does not work well with Scala REPL

2017-04-13 Thread wasif masood (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-8586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967689#comment-15967689 ] wasif masood commented on SPARK-8586: - I am using spark in Zeppelin and facing the same error. I am

[jira] [Updated] (SPARK-20287) Kafka Consumer should be able to subscribe to more than one topic partition

2017-04-13 Thread Cody Koeninger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cody Koeninger updated SPARK-20287: --- What you're describing is closer to the receiver-based implementation, which had a number of

[jira] [Assigned] (SPARK-20233) Apply star-join filter heuristics to dynamic programming join enumeration

2017-04-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan reassigned SPARK-20233: --- Assignee: Ioana Delaney > Apply star-join filter heuristics to dynamic programming join

[jira] [Resolved] (SPARK-20233) Apply star-join filter heuristics to dynamic programming join enumeration

2017-04-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan resolved SPARK-20233. - Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17546

[jira] [Commented] (SPARK-19946) DebugFilesystem.assertNoOpenStreams should report the open streams to help debugging

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967599#comment-15967599 ] Apache Spark commented on SPARK-19946: -- User 'bogdanrdc' has created a pull request for this issue:

[jira] [Commented] (SPARK-20199) GradientBoostedTreesModel doesn't have Column Sampling Rate Paramenter

2017-04-13 Thread pralabhkumar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967574#comment-15967574 ] pralabhkumar commented on SPARK-20199: -- Shouldn't there be a parameter in GBMParameters() val

[jira] [Created] (SPARK-20325) Spark Structured Streaming documentation Update: checkpoint configuration

2017-04-13 Thread Kate Eri (JIRA)
Kate Eri created SPARK-20325: Summary: Spark Structured Streaming documentation Update: checkpoint configuration Key: SPARK-20325 URL: https://issues.apache.org/jira/browse/SPARK-20325 Project: Spark

[jira] [Commented] (SPARK-650) Add a "setup hook" API for running initialization code on each executor

2017-04-13 Thread Mathieu Boespflug (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967554#comment-15967554 ] Mathieu Boespflug commented on SPARK-650: - [~Skamandros] how did you manage to hook

[jira] [Commented] (SPARK-20323) Calling stop in a transform stage causes the app to hang

2017-04-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967535#comment-15967535 ] Sean Owen commented on SPARK-20323: --- Yeah I don't think it would do anything because that's not even

[jira] [Commented] (SPARK-20323) Calling stop in a transform stage causes the app to hang

2017-04-13 Thread Andrei Taleanu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967516#comment-15967516 ] Andrei Taleanu commented on SPARK-20323: All right, so the behaviour is undefined in that case?

[jira] [Commented] (SPARK-20321) Spark UI cannot be shutdown in spark streaming app

2017-04-13 Thread Andrei Taleanu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967514#comment-15967514 ] Andrei Taleanu commented on SPARK-20321: Not sure what it would help. I could provide more logs,

[jira] [Commented] (SPARK-20321) Spark UI cannot be shutdown in spark streaming app

2017-04-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967502#comment-15967502 ] Sean Owen commented on SPARK-20321: --- I don't recall any other reports like this. Yeah, it's hard to

[jira] [Commented] (SPARK-20323) Calling stop in a transform stage causes the app to hang

2017-04-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967500#comment-15967500 ] Sean Owen commented on SPARK-20323: --- You can't manipulate the context in a transform. > Calling stop

[jira] [Updated] (SPARK-20324) Control itemSets length in PrefixSpan

2017-04-13 Thread Cyril de Vogelaere (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyril de Vogelaere updated SPARK-20324: --- Description: The idea behind this improvement would be to allow better control over

[jira] [Created] (SPARK-20324) Control itemSets length in PrefixSpan

2017-04-13 Thread Cyril de Vogelaere (JIRA)
Cyril de Vogelaere created SPARK-20324: -- Summary: Control itemSets length in PrefixSpan Key: SPARK-20324 URL: https://issues.apache.org/jira/browse/SPARK-20324 Project: Spark Issue

[jira] [Updated] (SPARK-20321) Spark UI cannot be shutdown in spark streaming app

2017-04-13 Thread Andrei Taleanu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrei Taleanu updated SPARK-20321: --- Description: When an exception thrown the transform stage is handled in foreachRDD and the

[jira] [Created] (SPARK-20323) Calling stop in a transform stage causes the app to hang

2017-04-13 Thread Andrei Taleanu (JIRA)
Andrei Taleanu created SPARK-20323: -- Summary: Calling stop in a transform stage causes the app to hang Key: SPARK-20323 URL: https://issues.apache.org/jira/browse/SPARK-20323 Project: Spark

[jira] [Created] (SPARK-20322) MinPattern length in PrefixSpan

2017-04-13 Thread Cyril de Vogelaere (JIRA)
Cyril de Vogelaere created SPARK-20322: -- Summary: MinPattern length in PrefixSpan Key: SPARK-20322 URL: https://issues.apache.org/jira/browse/SPARK-20322 Project: Spark Issue Type:

[jira] [Updated] (SPARK-20322) MinPattern length in PrefixSpan

2017-04-13 Thread Cyril de Vogelaere (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cyril de Vogelaere updated SPARK-20322: --- Description: The current implementation already diposes of a maxPatternLength

[jira] [Created] (SPARK-20321) Spark UI cannot be shutdown in spark streaming app

2017-04-13 Thread Andrei Taleanu (JIRA)
Andrei Taleanu created SPARK-20321: -- Summary: Spark UI cannot be shutdown in spark streaming app Key: SPARK-20321 URL: https://issues.apache.org/jira/browse/SPARK-20321 Project: Spark Issue

[jira] [Updated] (SPARK-19924) Handle InvocationTargetException for all Hive Shim

2017-04-13 Thread Wenchen Fan (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-19924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wenchen Fan updated SPARK-19924: Fix Version/s: 2.1.1 > Handle InvocationTargetException for all Hive Shim >

[jira] [Commented] (SPARK-20081) RandomForestClassifier doesn't seem to support more than 100 labels

2017-04-13 Thread Christian Reiniger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967403#comment-15967403 ] Christian Reiniger commented on SPARK-20081: Thanks for the information, but as you already

[jira] [Updated] (SPARK-20320) AnalysisException: Columns of grouping_id (count(value#17L)) does not match grouping columns (count(value#17L))

2017-04-13 Thread Jacek Laskowski (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jacek Laskowski updated SPARK-20320: Shepherd: Davies Liu (was: Herman van Hovell) Description: I'm not questioning the

[jira] [Created] (SPARK-20320) AnalysisException: Columns of grouping_id (count(value#17L)) does not match grouping columns (count(value#17L))

2017-04-13 Thread Jacek Laskowski (JIRA)
Jacek Laskowski created SPARK-20320: --- Summary: AnalysisException: Columns of grouping_id (count(value#17L)) does not match grouping columns (count(value#17L)) Key: SPARK-20320 URL:

[jira] [Comment Edited] (SPARK-20081) RandomForestClassifier doesn't seem to support more than 100 labels

2017-04-13 Thread 颜发才
[ https://issues.apache.org/jira/browse/SPARK-20081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967347#comment-15967347 ] Yan Facai (颜发才) edited comment on SPARK-20081 at 4/13/17 9:42 AM: -- Yes,

[jira] [Comment Edited] (SPARK-20081) RandomForestClassifier doesn't seem to support more than 100 labels

2017-04-13 Thread 颜发才
[ https://issues.apache.org/jira/browse/SPARK-20081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967347#comment-15967347 ] Yan Facai (颜发才) edited comment on SPARK-20081 at 4/13/17 9:40 AM: -- Yes,

[jira] [Comment Edited] (SPARK-20081) RandomForestClassifier doesn't seem to support more than 100 labels

2017-04-13 Thread 颜发才
[ https://issues.apache.org/jira/browse/SPARK-20081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967347#comment-15967347 ] Yan Facai (颜发才) edited comment on SPARK-20081 at 4/13/17 9:40 AM: -- Yes,

[jira] [Commented] (SPARK-20081) RandomForestClassifier doesn't seem to support more than 100 labels

2017-04-13 Thread 颜发才
[ https://issues.apache.org/jira/browse/SPARK-20081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967347#comment-15967347 ] Yan Facai (颜发才) commented on SPARK-20081: - Yes, you should use `builder.putLong("num_vals",

[jira] [Commented] (SPARK-20081) RandomForestClassifier doesn't seem to support more than 100 labels

2017-04-13 Thread Christian Reiniger (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967328#comment-15967328 ] Christian Reiniger commented on SPARK-20081: Thanks for the feedback. We had to implement

[jira] [Assigned] (SPARK-20319) Already quoted identifiers are getting wrapped with additional quotes

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20319: Assignee: Apache Spark > Already quoted identifiers are getting wrapped with additional

[jira] [Commented] (SPARK-20319) Already quoted identifiers are getting wrapped with additional quotes

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15967318#comment-15967318 ] Apache Spark commented on SPARK-20319: -- User 'umesh9794' has created a pull request for this issue:

[jira] [Assigned] (SPARK-20319) Already quoted identifiers are getting wrapped with additional quotes

2017-04-13 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-20319: Assignee: (was: Apache Spark) > Already quoted identifiers are getting wrapped with

[jira] [Resolved] (SPARK-20284) Make SerializationStream and DeserializationStream extend Closeable

2017-04-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-20284. --- Resolution: Fixed Fix Version/s: 2.2.0 Issue resolved by pull request 17598

[jira] [Assigned] (SPARK-20284) Make SerializationStream and DeserializationStream extend Closeable

2017-04-13 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-20284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reassigned SPARK-20284: - Assignee: Sergei Lebedev > Make SerializationStream and DeserializationStream extend Closeable

  1   2   >