[jira] [Updated] (SPARK-13185) Improve the performance of DateTimeUtils.StringToDate by reusing Calendar objects

2016-02-04 Thread Carson Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carson Wang updated SPARK-13185: Summary: Improve the performance of DateTimeUtils.StringToDate by reusing Calendar objects (was: I

[jira] [Updated] (SPARK-13185) Improve the performance of DateTimeUtils by reusing Calendar objects

2016-02-04 Thread Carson Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carson Wang updated SPARK-13185: Description: It is expensive to create java Calendar objects in each method of DateTimeUtils. We ca

[jira] [Commented] (SPARK-13185) Improve the performance of DateTimeUtils by reusing TimeZone and Calendar objects

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133771#comment-15133771 ] Apache Spark commented on SPARK-13185: -- User 'carsonwang' has created a pull request

[jira] [Updated] (SPARK-13185) Improve the performance of DateTimeUtils by reusing Calendar objects

2016-02-04 Thread Carson Wang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carson Wang updated SPARK-13185: Summary: Improve the performance of DateTimeUtils by reusing Calendar objects (was: Improve the pe

[jira] [Commented] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-02-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133764#comment-15133764 ] Xiao Li commented on SPARK-12720: - Almost done. Need to add more test cases to ensure it

[jira] [Created] (SPARK-13210) NPE in Sort

2016-02-04 Thread Davies Liu (JIRA)
Davies Liu created SPARK-13210: -- Summary: NPE in Sort Key: SPARK-13210 URL: https://issues.apache.org/jira/browse/SPARK-13210 Project: Spark Issue Type: Bug Components: SQL Affects

[jira] [Resolved] (SPARK-13187) Add boolean/long/double type option functions in DataFrameReader/Writer

2016-02-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-13187. - Resolution: Fixed Fix Version/s: 2.0.0 > Add boolean/long/double type option functions in

[jira] [Resolved] (SPARK-13208) Replace Pair with tuples

2016-02-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-13208. - Resolution: Fixed Assignee: Jakob Odersky Fix Version/s: 2.0.0 > Replace Pair wit

[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input

2016-02-04 Thread Mario Briggs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133643#comment-15133643 ] Mario Briggs commented on SPARK-13027: -- [~rameshaaditya117] sure, more than happy to

[jira] [Updated] (SPARK-13209) transitive closure on a dataframe

2016-02-04 Thread Placek (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Placek updated SPARK-13209: --- Description: When I run the following loop the join gets slower and slower regardless of caching. If I chang

[jira] [Created] (SPARK-13209) transitive closure on a dataframe

2016-02-04 Thread Placek (JIRA)
Placek created SPARK-13209: -- Summary: transitive closure on a dataframe Key: SPARK-13209 URL: https://issues.apache.org/jira/browse/SPARK-13209 Project: Spark Issue Type: Bug Components: S

[jira] [Commented] (SPARK-13208) Replace Pair with tuples

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133574#comment-15133574 ] Apache Spark commented on SPARK-13208: -- User 'jodersky' has created a pull request f

[jira] [Assigned] (SPARK-13208) Replace Pair with tuples

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13208: Assignee: (was: Apache Spark) > Replace Pair with tuples > >

[jira] [Assigned] (SPARK-13208) Replace Pair with tuples

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13208: Assignee: Apache Spark > Replace Pair with tuples > > >

[jira] [Updated] (SPARK-13208) Replace Pair with tuples

2016-02-04 Thread Jakob Odersky (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jakob Odersky updated SPARK-13208: -- Priority: Trivial (was: Major) > Replace Pair with tuples > > >

[jira] [Created] (SPARK-13208) Replace Pair with tuples

2016-02-04 Thread Jakob Odersky (JIRA)
Jakob Odersky created SPARK-13208: - Summary: Replace Pair with tuples Key: SPARK-13208 URL: https://issues.apache.org/jira/browse/SPARK-13208 Project: Spark Issue Type: Sub-task R

[jira] [Commented] (SPARK-13196) Optimize the option and flatten in Word2Vec to reduce the max memory consumption

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133547#comment-15133547 ] Apache Spark commented on SPARK-13196: -- User 'hhbyyh' has created a pull request for

[jira] [Assigned] (SPARK-13196) Optimize the option and flatten in Word2Vec to reduce the max memory consumption

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13196: Assignee: Apache Spark > Optimize the option and flatten in Word2Vec to reduce the max mem

[jira] [Commented] (SPARK-12957) Derive and propagate data constrains in logical plan

2016-02-04 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133546#comment-15133546 ] Yin Huai commented on SPARK-12957: -- Update: https://github.com/apache/spark/pull/10844 i

[jira] [Commented] (SPARK-13090) Add initial support for constraint propagation in SparkSQL

2016-02-04 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133549#comment-15133549 ] Yin Huai commented on SPARK-13090: -- Issue resolved by pull request 10844 https://github.

[jira] [Issue Comment Deleted] (SPARK-13196) Optimize the option and flatten in Word2Vec to reduce the max memory consumption

2016-02-04 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuhao yang updated SPARK-13196: --- Comment: was deleted (was: https://github.com/apache/spark/pull/11078) > Optimize the option and fla

[jira] [Updated] (SPARK-12957) Derive and propagate data constrains in logical plan

2016-02-04 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-12957: - Fix Version/s: (was: 2.0.0) > Derive and propagate data constrains in logical plan > ---

[jira] [Assigned] (SPARK-13196) Optimize the option and flatten in Word2Vec to reduce the max memory consumption

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13196: Assignee: (was: Apache Spark) > Optimize the option and flatten in Word2Vec to reduce

[jira] [Commented] (SPARK-13196) Optimize the option and flatten in Word2Vec to reduce the max memory consumption

2016-02-04 Thread yuhao yang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133548#comment-15133548 ] yuhao yang commented on SPARK-13196: https://github.com/apache/spark/pull/11078 > Op

[jira] [Commented] (SPARK-13207) _SUCCESS should not break partition discovery

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133540#comment-15133540 ] Apache Spark commented on SPARK-13207: -- User 'yhuai' has created a pull request for

[jira] [Assigned] (SPARK-13207) _SUCCESS should not break partition discovery

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13207: Assignee: Yin Huai (was: Apache Spark) > _SUCCESS should not break partition discovery >

[jira] [Assigned] (SPARK-13207) _SUCCESS should not break partition discovery

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13207: Assignee: Apache Spark (was: Yin Huai) > _SUCCESS should not break partition discovery >

[jira] [Commented] (SPARK-13179) pyspark row name collision 'count'

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133538#comment-15133538 ] Apache Spark commented on SPARK-13179: -- User 'shoelsch' has created a pull request f

[jira] [Assigned] (SPARK-13179) pyspark row name collision 'count'

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13179: Assignee: (was: Apache Spark) > pyspark row name collision 'count' > -

[jira] [Assigned] (SPARK-13179) pyspark row name collision 'count'

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13179: Assignee: Apache Spark > pyspark row name collision 'count' >

[jira] [Resolved] (SPARK-12850) Support bucket pruning (predicate pushdown for bucketed tables)

2016-02-04 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-12850. - Resolution: Fixed Assignee: Xiao Li Fix Version/s: 2.0.0 > Support bucket pruning

[jira] [Assigned] (SPARK-13145) checkAnswer in SQL query suites should tolerate small float number error

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13145: Assignee: (was: Apache Spark) > checkAnswer in SQL query suites should tolerate small

[jira] [Commented] (SPARK-13145) checkAnswer in SQL query suites should tolerate small float number error

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133531#comment-15133531 ] Apache Spark commented on SPARK-13145: -- User 'bomeng' has created a pull request for

[jira] [Assigned] (SPARK-13145) checkAnswer in SQL query suites should tolerate small float number error

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13145: Assignee: Apache Spark > checkAnswer in SQL query suites should tolerate small float numbe

[jira] [Commented] (SPARK-13207) _SUCCESS should not break partition discovery

2016-02-04 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133529#comment-15133529 ] Yin Huai commented on SPARK-13207: -- It will be better to let partitioning discovery igno

[jira] [Closed] (SPARK-13206) Subquery Alias in Hive Parser

2016-02-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li closed SPARK-13206. --- Resolution: Cannot Reproduce > Subquery Alias in Hive Parser > - > >

[jira] [Commented] (SPARK-13206) Subquery Alias in Hive Parser

2016-02-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133528#comment-15133528 ] Xiao Li commented on SPARK-13206: - True... I am unable to reproduce it. Thank you! Let me

[jira] [Updated] (SPARK-13207) _SUCCESS should not break partition discovery

2016-02-04 Thread Yin Huai (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yin Huai updated SPARK-13207: - Description: Partitioning discovery will fail with the following case {code} test("_SUCCESS should not br

[jira] [Created] (SPARK-13207) _SUCCESS should not break partition discovery

2016-02-04 Thread Yin Huai (JIRA)
Yin Huai created SPARK-13207: Summary: _SUCCESS should not break partition discovery Key: SPARK-13207 URL: https://issues.apache.org/jira/browse/SPARK-13207 Project: Spark Issue Type: Bug

[jira] [Comment Edited] (SPARK-13206) Subquery Alias in Hive Parser

2016-02-04 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133520#comment-15133520 ] Liang-Chi Hsieh edited comment on SPARK-13206 at 2/5/16 2:14 AM: --

[jira] [Commented] (SPARK-13206) Subquery Alias in Hive Parser

2016-02-04 Thread Liang-Chi Hsieh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133520#comment-15133520 ] Liang-Chi Hsieh commented on SPARK-13206: - I tried your query. Looks like it can

[jira] [Commented] (SPARK-13171) Update promise & future to Promise and Future as the old ones are deprecated

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133498#comment-15133498 ] Apache Spark commented on SPARK-13171: -- User 'jodersky' has created a pull request f

[jira] [Assigned] (SPARK-13171) Update promise & future to Promise and Future as the old ones are deprecated

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13171: Assignee: (was: Apache Spark) > Update promise & future to Promise and Future as the o

[jira] [Assigned] (SPARK-13171) Update promise & future to Promise and Future as the old ones are deprecated

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13171?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13171: Assignee: Apache Spark > Update promise & future to Promise and Future as the old ones are

[jira] [Updated] (SPARK-13206) Subquery Alias in Hive Parser

2016-02-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Li updated SPARK-13206: Summary: Subquery Alias in Hive Parser (was: Subquery Alias in Join in Hive Parser) > Subquery Alias in H

[jira] [Commented] (SPARK-12720) SQL generation support for cube, rollup, and grouping set

2016-02-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133459#comment-15133459 ] Xiao Li commented on SPARK-12720: - Since Subquery Alias does not work well, my PR has to

[jira] [Updated] (SPARK-13157) ADD JAR command cannot handle path with @ character

2016-02-04 Thread Cheng Lian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cheng Lian updated SPARK-13157: --- Assignee: Herman van Hovell > ADD JAR command cannot handle path with @ character > -

[jira] [Commented] (SPARK-13206) Subquery Alias in Join in Hive Parser

2016-02-04 Thread Xiao Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133452#comment-15133452 ] Xiao Li commented on SPARK-13206: - [~viirya] This is the one I mentioned just now. Could

[jira] [Created] (SPARK-13206) Subquery Alias in Join in Hive Parser

2016-02-04 Thread Xiao Li (JIRA)
Xiao Li created SPARK-13206: --- Summary: Subquery Alias in Join in Hive Parser Key: SPARK-13206 URL: https://issues.apache.org/jira/browse/SPARK-13206 Project: Spark Issue Type: Sub-task Co

[jira] [Updated] (SPARK-13052) waitingApps metric doesn't show the number of apps currently in the WAITING state

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13052: -- Assignee: Raafat Akkad > waitingApps metric doesn't show the number of apps currently in the WAITING >

[jira] [Resolved] (SPARK-13052) waitingApps metric doesn't show the number of apps currently in the WAITING state

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13052. --- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0 > waitingApps metric

[jira] [Commented] (SPARK-13178) RRDD faces with concurrency issue in case of rdd.zip(rdd).count()

2016-02-04 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133363#comment-15133363 ] Xusen Yin commented on SPARK-13178: --- Yes, it works, we can use read.json to load a Data

[jira] [Comment Edited] (SPARK-13155) add runtime null check when convert catalyst array to external array

2016-02-04 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133324#comment-15133324 ] Huaxin Gao edited comment on SPARK-13155 at 2/4/16 11:54 PM: -

[jira] [Comment Edited] (SPARK-13155) add runtime null check when convert catalyst array to external array

2016-02-04 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133324#comment-15133324 ] Huaxin Gao edited comment on SPARK-13155 at 2/4/16 11:43 PM: -

[jira] [Commented] (SPARK-13155) add runtime null check when convert catalyst array to external array

2016-02-04 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133324#comment-15133324 ] Huaxin Gao commented on SPARK-13155: https://issues.apache.org/jira/secure/ViewProfil

[jira] [Assigned] (SPARK-13205) SQL generation support for self join

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13205: Assignee: Apache Spark > SQL generation support for self join > --

[jira] [Commented] (SPARK-13205) SQL generation support for self join

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133289#comment-15133289 ] Apache Spark commented on SPARK-13205: -- User 'gatorsmile' has created a pull request

[jira] [Assigned] (SPARK-13205) SQL generation support for self join

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13205: Assignee: (was: Apache Spark) > SQL generation support for self join > ---

[jira] [Commented] (SPARK-13068) Extend pyspark ml paramtype conversion to support lists

2016-02-04 Thread Seth Hendrickson (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133285#comment-15133285 ] Seth Hendrickson commented on SPARK-13068: -- Upon closer examination, I think the

[jira] [Created] (SPARK-13205) SQL generation support for self join

2016-02-04 Thread Xiao Li (JIRA)
Xiao Li created SPARK-13205: --- Summary: SQL generation support for self join Key: SPARK-13205 URL: https://issues.apache.org/jira/browse/SPARK-13205 Project: Spark Issue Type: Sub-task Com

[jira] [Created] (SPARK-13204) Replace use of mutable.SynchronizedMap with ConcurrentHashMap

2016-02-04 Thread Ted Yu (JIRA)
Ted Yu created SPARK-13204: -- Summary: Replace use of mutable.SynchronizedMap with ConcurrentHashMap Key: SPARK-13204 URL: https://issues.apache.org/jira/browse/SPARK-13204 Project: Spark Issue Type

[jira] [Commented] (SPARK-13202) Jars specified with --jars do not exist on the worker classpath.

2016-02-04 Thread Michael Schmitz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133142#comment-15133142 ] Michael Schmitz commented on SPARK-13202: - This issue sounds very similar to what

[jira] [Assigned] (SPARK-13136) Data exchange (shuffle, broadcast) should only be handled by the exchange operator

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13136: Assignee: Apache Spark > Data exchange (shuffle, broadcast) should only be handled by the

[jira] [Assigned] (SPARK-13136) Data exchange (shuffle, broadcast) should only be handled by the exchange operator

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13136: Assignee: (was: Apache Spark) > Data exchange (shuffle, broadcast) should only be hand

[jira] [Commented] (SPARK-13136) Data exchange (shuffle, broadcast) should only be handled by the exchange operator

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133104#comment-15133104 ] Apache Spark commented on SPARK-13136: -- User 'hvanhovell' has created a pull request

[jira] [Commented] (SPARK-13186) Migrate away from SynchronizedMap which is derpecated

2016-02-04 Thread Huaxin Gao (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133101#comment-15133101 ] Huaxin Gao commented on SPARK-13186: Hi Holden, I will work on this one. Thanks! >

[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input

2016-02-04 Thread Aaditya Ramesh (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133087#comment-15133087 ] Aaditya Ramesh commented on SPARK-13027: [~zsxwing] Sorry for the late response,

[jira] [Assigned] (SPARK-13203) Add scalastyle rule banning use of mutable.SynchronizedBuffer

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13203: Assignee: (was: Apache Spark) > Add scalastyle rule banning use of mutable.Synchronize

[jira] [Created] (SPARK-13203) Add scalastyle rule banning use of mutable.SynchronizedBuffer

2016-02-04 Thread Ted Yu (JIRA)
Ted Yu created SPARK-13203: -- Summary: Add scalastyle rule banning use of mutable.SynchronizedBuffer Key: SPARK-13203 URL: https://issues.apache.org/jira/browse/SPARK-13203 Project: Spark Issue Typ

[jira] [Commented] (SPARK-13203) Add scalastyle rule banning use of mutable.SynchronizedBuffer

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133080#comment-15133080 ] Apache Spark commented on SPARK-13203: -- User 'ted-yu' has created a pull request for

[jira] [Assigned] (SPARK-13203) Add scalastyle rule banning use of mutable.SynchronizedBuffer

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13203: Assignee: Apache Spark > Add scalastyle rule banning use of mutable.SynchronizedBuffer >

[jira] [Commented] (SPARK-13116) TungstenAggregate though it is supposedly capable of all processing unsafe & safe rows, fails if the input is safe rows

2016-02-04 Thread Asif Hussain Shahid (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133061#comment-15133061 ] Asif Hussain Shahid commented on SPARK-13116: - It seems our code of using rel

[jira] [Commented] (SPARK-12141) Use Jackson to serialize all events when writing event log

2016-02-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133048#comment-15133048 ] Marcelo Vanzin commented on SPARK-12141: https://github.com/FasterXML/jackson-mod

[jira] [Commented] (SPARK-12807) Spark External Shuffle not working in Hadoop clusters with Jackson 2.2.3

2016-02-04 Thread Marcelo Vanzin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15133045#comment-15133045 ] Marcelo Vanzin commented on SPARK-12807: bq. I don't think we can park at 2.2.x.

[jira] [Updated] (SPARK-13202) Jars specified with --jars do not exist on the worker classpath.

2016-02-04 Thread Michael Schmitz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13202?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Schmitz updated SPARK-13202: Description: I have a Spark Scala 2.11 application. To deploy it to the cluster, I create

[jira] [Resolved] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

2016-02-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu resolved SPARK-13195. -- Resolution: Fixed Fix Version/s: 2.0.0 1.6.1 > PairDStreamFunctions.m

[jira] [Created] (SPARK-13202) Jars specified with --jars do not exist on the worker classpath.

2016-02-04 Thread Michael Schmitz (JIRA)
Michael Schmitz created SPARK-13202: --- Summary: Jars specified with --jars do not exist on the worker classpath. Key: SPARK-13202 URL: https://issues.apache.org/jira/browse/SPARK-13202 Project: Spark

[jira] [Created] (SPARK-13201) Make a private non-deprecated version of setRuns

2016-02-04 Thread holdenk (JIRA)
holdenk created SPARK-13201: --- Summary: Make a private non-deprecated version of setRuns Key: SPARK-13201 URL: https://issues.apache.org/jira/browse/SPARK-13201 Project: Spark Issue Type: Improvemen

[jira] [Created] (SPARK-13200) Investigate math.round on integer number in MFDataGenerator.scala:109

2016-02-04 Thread holdenk (JIRA)
holdenk created SPARK-13200: --- Summary: Investigate math.round on integer number in MFDataGenerator.scala:109 Key: SPARK-13200 URL: https://issues.apache.org/jira/browse/SPARK-13200 Project: Spark

[jira] [Resolved] (SPARK-13168) Collapse adjacent Repartition operations

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13168. --- Resolution: Fixed Fix Version/s: 2.0.0 > Collapse adjacent Repartition operations > --

[jira] [Assigned] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

2016-02-04 Thread Shixiong Zhu (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shixiong Zhu reassigned SPARK-13195: Assignee: Shixiong Zhu > PairDStreamFunctions.mapWithState fails in case timeout is set wi

[jira] [Commented] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132797#comment-15132797 ] Apache Spark commented on SPARK-13195: -- User 'zsxwing' has created a pull request fo

[jira] [Assigned] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13195: Assignee: Apache Spark > PairDStreamFunctions.mapWithState fails in case timeout is set wi

[jira] [Assigned] (SPARK-13195) PairDStreamFunctions.mapWithState fails in case timeout is set without updating State[S]

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-13195: Assignee: (was: Apache Spark) > PairDStreamFunctions.mapWithState fails in case timeou

[jira] [Commented] (SPARK-12430) Temporary folders do not get deleted after Task completes causing problems with disk space.

2016-02-04 Thread Fede Bar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132759#comment-15132759 ] Fede Bar commented on SPARK-12430: -- Yes I confirm that I am using version 1.6. Since tha

[jira] [Resolved] (SPARK-13053) Rectify ignored tests in InternalAccumulatorSuite

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13053. --- Resolution: Fixed Fix Version/s: 2.0.0 > Rectify ignored tests in InternalAccumulatorSuite > -

[jira] [Resolved] (SPARK-13162) Standalone mode does not respect `spark.dynamicAllocation.initialExecutors`

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13162. --- Resolution: Fixed Fix Version/s: 2.0.0 > Standalone mode does not respect `spark.dynamicAlloca

[jira] [Updated] (SPARK-13151) Investigate replacing SynchronizedBuffer as it is deprecated/unreliable

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or updated SPARK-13151: -- Target Version/s: 2.0.0 > Investigate replacing SynchronizedBuffer as it is deprecated/unreliable > ---

[jira] [Resolved] (SPARK-13164) Replace deprecated synchronizedBuffer in core

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-13164. --- Resolution: Fixed Assignee: holdenk Fix Version/s: 2.0.0 Target Version/s

[jira] [Resolved] (SPARK-12330) Mesos coarse executor does not cleanup blockmgr properly on termination if data is stored on disk

2016-02-04 Thread Andrew Or (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Or resolved SPARK-12330. --- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0 > Mesos coarse execut

[jira] [Commented] (SPARK-13178) RRDD faces with concurrency issue in case of rdd.zip(rdd).count()

2016-02-04 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132718#comment-15132718 ] Xusen Yin commented on SPARK-13178: --- Thanks! I'll try it. > RRDD faces with concurrenc

[jira] [Commented] (SPARK-13184) Support minPartitions parameter for JSON and CSV datasources as options

2016-02-04 Thread koert kuipers (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13184?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132711#comment-15132711 ] koert kuipers commented on SPARK-13184: --- thanks for creating this! i think the issu

[jira] [Commented] (SPARK-13176) Ignore deprecation warning for ProcessBuilder lines_!

2016-02-04 Thread holdenk (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132710#comment-15132710 ] holdenk commented on SPARK-13176: - cc [~srowen] any thoughts? > Ignore deprecation warni

[jira] [Commented] (SPARK-13079) Provide an in-memory implementation of the catalog API

2016-02-04 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132701#comment-15132701 ] Apache Spark commented on SPARK-13079: -- User 'andrewor14' has created a pull request

[jira] [Commented] (SPARK-13027) Add API for updateStateByKey to provide batch time as input

2016-02-04 Thread Mario Briggs (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132641#comment-15132641 ] Mario Briggs commented on SPARK-13027: -- [~rameshaaditya117] if you are tied up on so

[jira] [Commented] (SPARK-12430) Temporary folders do not get deleted after Task completes causing problems with disk space.

2016-02-04 Thread Iulian Dragos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132640#comment-15132640 ] Iulian Dragos commented on SPARK-12430: --- Are you using 1.6? In that case, the block

[jira] [Closed] (SPARK-11759) Spark task on mesos with docker fails with sh: 1: /opt/spark/bin/spark-class: not found

2016-02-04 Thread Luis Alves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luis Alves closed SPARK-11759. -- Resolution: Later > Spark task on mesos with docker fails with sh: 1: /opt/spark/bin/spark-class: > no

[jira] [Closed] (SPARK-13039) Spark Streaming with Mesos shutdown without any reason on logs

2016-02-04 Thread Luis Alves (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-13039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Luis Alves closed SPARK-13039. -- Resolution: Fixed > Spark Streaming with Mesos shutdown without any reason on logs > --

[jira] [Commented] (SPARK-12430) Temporary folders do not get deleted after Task completes causing problems with disk space.

2016-02-04 Thread Fede Bar (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-12430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15132603#comment-15132603 ] Fede Bar commented on SPARK-12430: -- Hi Jean-Baptiste and Iulian, I took a look at [thi

[jira] [Created] (SPARK-13199) Upgrade apache httpclient version to the latest 4.5 for security

2016-02-04 Thread Ted Yu (JIRA)
Ted Yu created SPARK-13199: -- Summary: Upgrade apache httpclient version to the latest 4.5 for security Key: SPARK-13199 URL: https://issues.apache.org/jira/browse/SPARK-13199 Project: Spark Issue T

  1   2   >