[
https://issues.apache.org/jira/browse/MAHOUT-1603?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14089745#comment-14089745
]
ASF GitHub Bot commented on MAHOUT-1603:
----------------------------------------
Github user dlyubimov commented on the pull request:
https://github.com/apache/mahout/pull/40#issuecomment-51525329
@pferrel: So one problem with those tests is that they are creating 2 spark
sessions. 1 session is created by tests and another session is created by
driver.
Spark is very strict with this:
(1) Spark is not reentrant w.r.t. session creation are non-reentrant (not
just thread-unsafe) -- meaning you can only safely have at most 1 session
at a time in a jvm.
(2) Spark session itself is reentrant -- meaning multiple threads may
invoke asynchronous computational actions on the same session.
This may not always manifest, but in the end it always will (ask me how i
know :)
so the problem with those tests is that they probably must not be featuring
DistributedSparkSuite but rather just MahoutSuite. Or alternatively you may
pass an already existing mahoutContext to the driver code for reuse. But
you must ensure the above constraint. The effects will range dramatically
if not (from mislabeled rdd partitions in the block manager to lockups
and internal race conditions)
On Thu, Aug 7, 2014 at 12:59 PM, Dmitriy Lyubimov <[email protected]> wrote:
> ok i guess like you said tests are still failing
>
>
> On Thu, Aug 7, 2014 at 12:18 PM, Dmitriy Lyubimov <[email protected]>
> wrote:
>
>> assuming you are on your local branch named spark-1.0.x with your commit
>> on top of mine current head, please execute
>> push [email protected]:dlyubimov/mahout spark-1.0.x
>>
>> this should go thru
>>
>>
>> On Thu, Aug 7, 2014 at 12:13 PM, Pat Ferrel <[email protected]>
>> wrote:
>>
>>> I can't push them back to you so they are here
>>> https://github.com/pferrel/mahout/tree/spark-1.0.x
>>>
>>> —
>>> Reply to this email directly or view it on GitHub
>>> <https://github.com/apache/mahout/pull/40#issuecomment-51517989>.
>>>
>>
>>
>
> Tweaks for Spark 1.0.x
> -----------------------
>
> Key: MAHOUT-1603
> URL: https://issues.apache.org/jira/browse/MAHOUT-1603
> Project: Mahout
> Issue Type: Task
> Affects Versions: 0.9
> Reporter: Dmitriy Lyubimov
> Assignee: Dmitriy Lyubimov
> Fix For: 1.0
>
>
> Tweaks necessary current codebase on top of spark 1.0.x
--
This message was sent by Atlassian JIRA
(v6.2#6252)
