[jira] [Resolved] (SPARK-11273) ArrayData and MapData shouldn't be public in types package

2015-10-23 Thread Reynold Xin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reynold Xin resolved SPARK-11273. - Resolution: Fixed Fix Version/s: 1.6.0 > ArrayData and MapData shouldn't be public in type

[jira] [Created] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Sem Mulder (JIRA)
Sem Mulder created SPARK-11276: -- Summary: SizeEstimator prevents class unloading Key: SPARK-11276 URL: https://issues.apache.org/jira/browse/SPARK-11276 Project: Spark Issue Type: Bug

[jira] [Updated] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Sem Mulder (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sem Mulder updated SPARK-11276: --- Description: The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys

[jira] [Updated] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Sem Mulder (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sem Mulder updated SPARK-11276: --- Description: The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys

[jira] [Updated] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Sem Mulder (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sem Mulder updated SPARK-11276: --- Description: The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys

[jira] [Updated] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Sem Mulder (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sem Mulder updated SPARK-11276: --- Description: The SizeEstimator keeps a cache of ClassInfos but this cache uses Class objects as keys

[jira] [Commented] (SPARK-11234) What's cooking classification

2015-10-23 Thread Xusen Yin (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970596#comment-14970596 ] Xusen Yin commented on SPARK-11234: --- [~mengxr] I add the cooking classification code he

[jira] [Assigned] (SPARK-11271) MapStatus too large for driver

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11271: Assignee: Apache Spark > MapStatus too large for driver > -- >

[jira] [Assigned] (SPARK-11271) MapStatus too large for driver

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11271: Assignee: (was: Apache Spark) > MapStatus too large for driver > -

[jira] [Commented] (SPARK-11271) MapStatus too large for driver

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970626#comment-14970626 ] Apache Spark commented on SPARK-11271: -- User 'viirya' has created a pull request for

[jira] [Commented] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970629#comment-14970629 ] Apache Spark commented on SPARK-11276: -- User 'SemMulder' has created a pull request

[jira] [Assigned] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11276: Assignee: (was: Apache Spark) > SizeEstimator prevents class unloading > -

[jira] [Assigned] (SPARK-11276) SizeEstimator prevents class unloading

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11276: Assignee: Apache Spark > SizeEstimator prevents class unloading >

[jira] [Assigned] (SPARK-10947) With schema inference from JSON into a Dataframe, add option to infer all primitive object types as strings

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10947: Assignee: Apache Spark > With schema inference from JSON into a Dataframe, add option to i

[jira] [Commented] (SPARK-10947) With schema inference from JSON into a Dataframe, add option to infer all primitive object types as strings

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970631#comment-14970631 ] Apache Spark commented on SPARK-10947: -- User 'stephend-realitymine' has created a pu

[jira] [Assigned] (SPARK-10947) With schema inference from JSON into a Dataframe, add option to infer all primitive object types as strings

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-10947: Assignee: (was: Apache Spark) > With schema inference from JSON into a Dataframe, add

[jira] [Created] (SPARK-11277) sort_array throws exception scala.MatchError

2015-10-23 Thread Jia Li (JIRA)
Jia Li created SPARK-11277: -- Summary: sort_array throws exception scala.MatchError Key: SPARK-11277 URL: https://issues.apache.org/jira/browse/SPARK-11277 Project: Spark Issue Type: Bug Co

[jira] [Updated] (SPARK-11277) sort_array throws exception scala.MatchError

2015-10-23 Thread Jia Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jia Li updated SPARK-11277: --- Description: I was trying out the sort_array function then hit this exception. I looked into the spark sour

[jira] [Commented] (SPARK-11277) sort_array throws exception scala.MatchError

2015-10-23 Thread Jia Li (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970654#comment-14970654 ] Jia Li commented on SPARK-11277: I already have a fix for this issue and I'm going to cre

[jira] [Commented] (SPARK-5210) Support log rolling in EventLogger

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-5210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970670#comment-14970670 ] Apache Spark commented on SPARK-5210: - User 'XuTingjun' has created a pull request for

[jira] [Created] (SPARK-11278) PageRank fails with unified memory manager

2015-10-23 Thread Nishkam Ravi (JIRA)
Nishkam Ravi created SPARK-11278: Summary: PageRank fails with unified memory manager Key: SPARK-11278 URL: https://issues.apache.org/jira/browse/SPARK-11278 Project: Spark Issue Type: Bug

[jira] [Created] (SPARK-11279) Add DataFrame#toDF in PySpark

2015-10-23 Thread Jeff Zhang (JIRA)
Jeff Zhang created SPARK-11279: -- Summary: Add DataFrame#toDF in PySpark Key: SPARK-11279 URL: https://issues.apache.org/jira/browse/SPARK-11279 Project: Spark Issue Type: Improvement C

[jira] [Assigned] (SPARK-11277) sort_array throws exception scala.MatchError

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11277: Assignee: Apache Spark > sort_array throws exception scala.MatchError > --

[jira] [Commented] (SPARK-11277) sort_array throws exception scala.MatchError

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970694#comment-14970694 ] Apache Spark commented on SPARK-11277: -- User 'jliwork' has created a pull request fo

[jira] [Assigned] (SPARK-11277) sort_array throws exception scala.MatchError

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11277: Assignee: (was: Apache Spark) > sort_array throws exception scala.MatchError > ---

[jira] [Assigned] (SPARK-11279) Add DataFrame#toDF in PySpark

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11279: Assignee: (was: Apache Spark) > Add DataFrame#toDF in PySpark > --

[jira] [Assigned] (SPARK-11279) Add DataFrame#toDF in PySpark

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11279: Assignee: Apache Spark > Add DataFrame#toDF in PySpark > - > >

[jira] [Commented] (SPARK-11279) Add DataFrame#toDF in PySpark

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970705#comment-14970705 ] Apache Spark commented on SPARK-11279: -- User 'zjffdu' has created a pull request for

[jira] [Created] (SPARK-11280) Mesos cluster deployment using only one node

2015-10-23 Thread Iulian Dragos (JIRA)
Iulian Dragos created SPARK-11280: - Summary: Mesos cluster deployment using only one node Key: SPARK-11280 URL: https://issues.apache.org/jira/browse/SPARK-11280 Project: Spark Issue Type: Bu

[jira] [Updated] (SPARK-11280) Mesos cluster deployment using only one node

2015-10-23 Thread Iulian Dragos (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Iulian Dragos updated SPARK-11280: -- Attachment: Screen Shot 2015-10-23 at 11.37.43.png > Mesos cluster deployment using only one no

[jira] [Updated] (SPARK-11259) Params.validateParams() should be called automatically

2015-10-23 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yanbo Liang updated SPARK-11259: Description: Params.validateParams() can not be called automatically currently. Such as the follow

[jira] [Reopened] (SPARK-11229) NPE in JoinedRow.isNullAt when spark.shuffle.memoryFraction=0

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen reopened SPARK-11229: --- "Fixed" implies there was a change attached to this JIRA that resolved the issue, and we don't have that

[jira] [Resolved] (SPARK-11229) NPE in JoinedRow.isNullAt when spark.shuffle.memoryFraction=0

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-11229. --- Resolution: Cannot Reproduce Fix Version/s: (was: 1.6.0) > NPE in JoinedRow.isNullAt when

[jira] [Updated] (SPARK-7021) JUnit output for Python tests

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-7021: - Assignee: Gabor Liptak > JUnit output for Python tests > - > >

[jira] [Updated] (SPARK-11262) Unit test for gradient, loss layers, memory management for multilayer perceptron

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11262: -- Target Version/s: (was: 1.6.0) Fix Version/s: (was: 1.5.1) [~avulanov] don't set Fix/Targe

[jira] [Updated] (SPARK-11270) Add improved equality testing for TopicAndPartition from the Kafka Streaming API

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11270: -- Target Version/s: (was: 1.5.1) Fix Version/s: (was: 1.5.1) [~manygrams] have a look at ht

[jira] [Updated] (SPARK-11267) NettyRpcEnv and sparkDriver services report the same port in the logs

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen updated SPARK-11267: -- Component/s: Spark Core > NettyRpcEnv and sparkDriver services report the same port in the logs > -

[jira] [Commented] (SPARK-10947) With schema inference from JSON into a Dataframe, add option to infer all primitive object types as strings

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970777#comment-14970777 ] Apache Spark commented on SPARK-10947: -- User 'stephend-realitymine' has created a pu

[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970783#comment-14970783 ] Sean Owen commented on SPARK-11016: --- NB: the resolution here may be to simply remove us

[jira] [Commented] (SPARK-11087) spark.sql.orc.filterPushdown does not work, No ORC pushdown predicate

2015-10-23 Thread patcharee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970786#comment-14970786 ] patcharee commented on SPARK-11087: --- [~zzhan] I found the predicate generated in the ex

[jira] [Closed] (SPARK-11087) spark.sql.orc.filterPushdown does not work, No ORC pushdown predicate

2015-10-23 Thread patcharee (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] patcharee closed SPARK-11087. - Resolution: Not A Problem The predicate is indeed generated and can be found in the executor log > spark

[jira] [Commented] (SPARK-9265) Dataframe.limit joined with another dataframe can be non-deterministic

2015-10-23 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970801#comment-14970801 ] Yanbo Liang commented on SPARK-9265: I'm working on it. > Dataframe.limit joined with

[jira] [Commented] (SPARK-10857) SQL injection bug in JdbcDialect.getTableExistsQuery()

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970836#comment-14970836 ] Sean Owen commented on SPARK-10857: --- Rick you're saying that this code path only comes

[jira] [Commented] (SPARK-11167) Incorrect type resolution on heterogeneous data structures

2015-10-23 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970839#comment-14970839 ] Maciej Szymkiewicz commented on SPARK-11167: spark-csv has a much simpler job

[jira] [Commented] (SPARK-7008) An implementation of Factorization Machine (LibFM)

2015-10-23 Thread Nick Pentreath (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970870#comment-14970870 ] Nick Pentreath commented on SPARK-7008: --- Is this now going in 1.6 (as per SPARK-1032

[jira] [Commented] (SPARK-6724) Model import/export for FPGrowth

2015-10-23 Thread Yanbo Liang (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970887#comment-14970887 ] Yanbo Liang commented on SPARK-6724: [~MeethuMathew] I will take over this task and se

[jira] [Commented] (SPARK-6270) Standalone Master hangs when streaming job completes and event logging is enabled

2015-10-23 Thread Jim Haughwout (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970915#comment-14970915 ] Jim Haughwout commented on SPARK-6270: -- [~tdas]: Can the team update this issue to re

[jira] [Commented] (SPARK-9325) Support `collect` on DataFrame columns

2015-10-23 Thread Russell Pierce (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970917#comment-14970917 ] Russell Pierce commented on SPARK-9325: --- You're right, Spark had been producing an e

[jira] [Comment Edited] (SPARK-9325) Support `collect` on DataFrame columns

2015-10-23 Thread Russell Pierce (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-9325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970917#comment-14970917 ] Russell Pierce edited comment on SPARK-9325 at 10/23/15 12:53 PM: --

[jira] [Created] (SPARK-11281) Issue with creating and collecting DataFrame using environments

2015-10-23 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-11281: -- Summary: Issue with creating and collecting DataFrame using environments Key: SPARK-11281 URL: https://issues.apache.org/jira/browse/SPARK-11281 Project:

[jira] [Created] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
Maciej Bryński created SPARK-11282: -- Summary: Very strange broadcast join behaviour Key: SPARK-11282 URL: https://issues.apache.org/jira/browse/SPARK-11282 Project: Spark Issue Type: Bug

[jira] [Commented] (SPARK-11167) Incorrect type resolution on heterogeneous data structures

2015-10-23 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970922#comment-14970922 ] Maciej Szymkiewicz commented on SPARK-11167: Related problem: https://issues.

[jira] [Updated] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-11282: --- Description: Hi, I found very strange broadcast join behaviour. According to this Jira https

[jira] [Updated] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-11282: --- Attachment: SPARK-11282.py > Very strange broadcast join behaviour >

[jira] [Updated] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-11282: --- Description: Hi, I found very strange broadcast join behaviour. According to this Jira https

[jira] [Resolved] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-11282. --- Resolution: Duplicate [~maver1ck] this could use a better title, and there is no code attached. I al

[jira] [Updated] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Bryński updated SPARK-11282: --- Description: Hi, I found very strange broadcast join behaviour. According to this Jira https

[jira] [Issue Comment Deleted] (SPARK-11167) Incorrect type resolution on heterogeneous data structures

2015-10-23 Thread Maciej Szymkiewicz (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciej Szymkiewicz updated SPARK-11167: --- Comment: was deleted (was: Related problem: https://issues.apache.org/jira/browse/SPA

[jira] [Commented] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970931#comment-14970931 ] Maciej Bryński commented on SPARK-11282: We had race condition here. I was attach

[jira] [Commented] (SPARK-11265) YarnClient can't get tokens to talk to Hive in a secure cluster

2015-10-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970930#comment-14970930 ] Steve Loughran commented on SPARK-11265: I can trigger a failure in a unit test n

[jira] [Comment Edited] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970931#comment-14970931 ] Maciej Bryński edited comment on SPARK-11282 at 10/23/15 1:07 PM: -

[jira] [Comment Edited] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970931#comment-14970931 ] Maciej Bryński edited comment on SPARK-11282 at 10/23/15 1:07 PM: -

[jira] [Commented] (SPARK-11282) Very strange broadcast join behaviour

2015-10-23 Thread JIRA
[ https://issues.apache.org/jira/browse/SPARK-11282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970952#comment-14970952 ] Maciej Bryński commented on SPARK-11282: UPDATE: Looks like: -XX:-UseCompressedO

[jira] [Updated] (SPARK-6270) Standalone Master hangs when streaming job completes and event logging is enabled

2015-10-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steve Loughran updated SPARK-6270: -- Affects Version/s: 1.5.1 > Standalone Master hangs when streaming job completes and event loggin

[jira] [Created] (SPARK-11283) List column gets additional level of nesting when converted to Spark DataFrame

2015-10-23 Thread Maciej Szymkiewicz (JIRA)
Maciej Szymkiewicz created SPARK-11283: -- Summary: List column gets additional level of nesting when converted to Spark DataFrame Key: SPARK-11283 URL: https://issues.apache.org/jira/browse/SPARK-11283

[jira] [Commented] (SPARK-6847) Stack overflow on updateStateByKey which followed by a dstream with checkpoint set

2015-10-23 Thread Glyton Camilleri (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971004#comment-14971004 ] Glyton Camilleri commented on SPARK-6847: - Hi, we managed to actually get rid of t

[jira] [Created] (SPARK-11284) ALS produces predictions as floats and should be double

2015-10-23 Thread Dominik Dahlem (JIRA)
Dominik Dahlem created SPARK-11284: -- Summary: ALS produces predictions as floats and should be double Key: SPARK-11284 URL: https://issues.apache.org/jira/browse/SPARK-11284 Project: Spark I

[jira] [Commented] (SPARK-11258) Remove quadratic runtime complexity for converting a Spark DataFrame into an R data.frame

2015-10-23 Thread Frank Rosner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971025#comment-14971025 ] Frank Rosner commented on SPARK-11258: -- Actually I am pretty confused now. Thinking

[jira] [Assigned] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11016: Assignee: Apache Spark > Spark fails when running with a task that requires a more recent

[jira] [Commented] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971027#comment-14971027 ] Apache Spark commented on SPARK-11016: -- User 'viirya' has created a pull request for

[jira] [Assigned] (SPARK-11016) Spark fails when running with a task that requires a more recent version of RoaringBitmaps

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11016: Assignee: (was: Apache Spark) > Spark fails when running with a task that requires a m

[jira] [Commented] (SPARK-10562) .partitionBy() creates the metastore partition columns in all lowercase, but persists the data path as MixedCase resulting in an error when the data is later attempted

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971042#comment-14971042 ] Apache Spark commented on SPARK-10562: -- User 'cloud-fan' has created a pull request

[jira] [Assigned] (SPARK-11284) ALS produces predictions as floats and should be double

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11284: Assignee: (was: Apache Spark) > ALS produces predictions as floats and should be doubl

[jira] [Assigned] (SPARK-11284) ALS produces predictions as floats and should be double

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11284: Assignee: Apache Spark > ALS produces predictions as floats and should be double > ---

[jira] [Resolved] (SPARK-11261) Provide a more flexible alternative to Jdbc RDD

2015-10-23 Thread Sean Owen (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sean Owen resolved SPARK-11261. --- Resolution: Won't Fix > Provide a more flexible alternative to Jdbc RDD > ---

[jira] [Commented] (SPARK-11284) ALS produces predictions as floats and should be double

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971055#comment-14971055 ] Apache Spark commented on SPARK-11284: -- User 'dahlem' has created a pull request for

[jira] [Commented] (SPARK-10857) SQL injection bug in JdbcDialect.getTableExistsQuery()

2015-10-23 Thread Rick Hillegas (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971101#comment-14971101 ] Rick Hillegas commented on SPARK-10857: --- Hi Sean, That is my understanding after e

[jira] [Created] (SPARK-11285) Infinite TaskCommitDenied loop

2015-10-23 Thread Ryan Williams (JIRA)
Ryan Williams created SPARK-11285: - Summary: Infinite TaskCommitDenied loop Key: SPARK-11285 URL: https://issues.apache.org/jira/browse/SPARK-11285 Project: Spark Issue Type: Bug Co

[jira] [Updated] (SPARK-11258) Converting a Spark DataFrame into an R data.frame is slow / requires a lot of memory

2015-10-23 Thread Frank Rosner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Rosner updated SPARK-11258: - Summary: Converting a Spark DataFrame into an R data.frame is slow / requires a lot of memory (w

[jira] [Updated] (SPARK-11258) Converting a Spark DataFrame into an R data.frame is slow / requires a lot of memory

2015-10-23 Thread Frank Rosner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Frank Rosner updated SPARK-11258: - Description: h4. Problem We tried to collect a DataFrame with > 1 million rows and a few hundred

[jira] [Commented] (SPARK-11258) Converting a Spark DataFrame into an R data.frame is slow / requires a lot of memory

2015-10-23 Thread Frank Rosner (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971155#comment-14971155 ] Frank Rosner commented on SPARK-11258: -- I adjusted the description to be more genera

[jira] [Updated] (SPARK-11285) Infinite TaskCommitDenied loop

2015-10-23 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ryan Williams updated SPARK-11285: -- Description: I've seen several apps enter this failing state in the last couple of days. I've

[jira] [Commented] (SPARK-11285) Infinite TaskCommitDenied loop

2015-10-23 Thread Ryan Williams (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971161#comment-14971161 ] Ryan Williams commented on SPARK-11285: --- A quick note on the structure of the linke

[jira] [Resolved] (SPARK-10382) Make example code in user guide testable

2015-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10382?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10382. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 9109 [https://gi

[jira] [Assigned] (SPARK-11265) YarnClient can't get tokens to talk to Hive in a secure cluster

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11265: Assignee: Apache Spark > YarnClient can't get tokens to talk to Hive in a secure cluster >

[jira] [Commented] (SPARK-11265) YarnClient can't get tokens to talk to Hive in a secure cluster

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971190#comment-14971190 ] Apache Spark commented on SPARK-11265: -- User 'steveloughran' has created a pull requ

[jira] [Assigned] (SPARK-11265) YarnClient can't get tokens to talk to Hive in a secure cluster

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-11265: Assignee: (was: Apache Spark) > YarnClient can't get tokens to talk to Hive in a secur

[jira] [Commented] (SPARK-11265) YarnClient can't get tokens to talk to Hive in a secure cluster

2015-10-23 Thread Steve Loughran (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-11265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971188#comment-14971188 ] Steve Loughran commented on SPARK-11265: Pull request is : https://github.com/apa

[jira] [Resolved] (SPARK-10277) Add @since annotation to pyspark.mllib.regression

2015-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-10277. --- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 8684 [https://gi

[jira] [Updated] (SPARK-6723) Model import/export for ChiSqSelector

2015-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng updated SPARK-6723: - Assignee: Jayant Shekhar > Model import/export for ChiSqSelector > ---

[jira] [Resolved] (SPARK-6723) Model import/export for ChiSqSelector

2015-10-23 Thread Xiangrui Meng (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiangrui Meng resolved SPARK-6723. -- Resolution: Fixed Fix Version/s: 1.6.0 Issue resolved by pull request 6785 [https://githu

[jira] [Updated] (SPARK-10610) Using AppName instead of AppId in the name of all metrics

2015-10-23 Thread Yi Tian (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Tian updated SPARK-10610: Summary: Using AppName instead of AppId in the name of all metrics (was: Using AppName instead AppId in th

[jira] [Commented] (SPARK-7970) Optimize code for SQL queries fired on Union of RDDs (closure cleaner)

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971217#comment-14971217 ] Apache Spark commented on SPARK-7970: - User 'nitin2goyal' has created a pull request f

[jira] [Assigned] (SPARK-7970) Optimize code for SQL queries fired on Union of RDDs (closure cleaner)

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-7970: --- Assignee: (was: Apache Spark) > Optimize code for SQL queries fired on Union of RDDs (clo

[jira] [Assigned] (SPARK-7970) Optimize code for SQL queries fired on Union of RDDs (closure cleaner)

2015-10-23 Thread Apache Spark (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Apache Spark reassigned SPARK-7970: --- Assignee: Apache Spark > Optimize code for SQL queries fired on Union of RDDs (closure cleane

[jira] [Commented] (SPARK-4940) Support more evenly distributing cores for Mesos mode

2015-10-23 Thread Jerry Lam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971228#comment-14971228 ] Jerry Lam commented on SPARK-4940: -- I just want to weight in the importance of this issue

[jira] [Comment Edited] (SPARK-4940) Support more evenly distributing cores for Mesos mode

2015-10-23 Thread Jerry Lam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971228#comment-14971228 ] Jerry Lam edited comment on SPARK-4940 at 10/23/15 4:01 PM: I

[jira] [Comment Edited] (SPARK-4940) Support more evenly distributing cores for Mesos mode

2015-10-23 Thread Jerry Lam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971228#comment-14971228 ] Jerry Lam edited comment on SPARK-4940 at 10/23/15 4:02 PM: I

[jira] [Commented] (SPARK-10975) Shuffle files left behind on Mesos without dynamic allocation

2015-10-23 Thread Chris Bannister (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-10975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971265#comment-14971265 ] Chris Bannister commented on SPARK-10975: - Spark will use MESOS_DIRECTORY sandbox

[jira] [Comment Edited] (SPARK-4940) Support more evenly distributing cores for Mesos mode

2015-10-23 Thread Jerry Lam (JIRA)
[ https://issues.apache.org/jira/browse/SPARK-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971228#comment-14971228 ] Jerry Lam edited comment on SPARK-4940 at 10/23/15 4:15 PM: I

  1   2   >