Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-148861222
Sorry, this kind of fell through the cracks. I had intended to do a PR
against Kryo to have their fix backported. I'll reopen this PR when I pick it
up
Github user coolfrood closed the pull request at:
https://github.com/apache/spark/pull/6361
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-116741038
I think the serialization format should be the same with a minor version
bump.
How hard would it be to get Chill updated to use a newer version (2.24) of
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-110419312
@JoshRosen I narrowed down the Kryo version problem a bit. Here's a gist
that demonstrates the problem:
https://gist.github.com/coolfrood/1ff8c65a2b92fe9
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-109321223
This class is deserialized using Kryo, so it is likely that my trying out a
different version of Kryo tickled some bug. I investigated this briefly but I
could not
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-108470313
The Hive tests fail in an unexplained manner, and I couldn't learn much by
looking at the Hive source code for this failure:
```
10:03:48.669
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-107640644
The Hive tests failed again; I guess this is due to the change of Kryo
version. I'll dig into it.
---
If your project is set up for it, you can reply to this
Github user coolfrood commented on a diff in the pull request:
https://github.com/apache/spark/pull/6361#discussion_r31445014
--- Diff:
core/src/main/scala/org/apache/spark/util/SerializableBuffer.scala ---
@@ -21,12 +21,16 @@ import java.io.{EOFException, IOException
Github user coolfrood commented on a diff in the pull request:
https://github.com/apache/spark/pull/6361#discussion_r31439444
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -126,6 +129,21 @@ private[spark] abstract class Task[T](val stageId:
Int, var
Github user coolfrood commented on a diff in the pull request:
https://github.com/apache/spark/pull/6361#discussion_r31436871
--- Diff:
core/src/main/scala/org/apache/spark/util/SerializableBuffer.scala ---
@@ -21,12 +21,16 @@ import java.io.{EOFException, IOException
Github user coolfrood commented on a diff in the pull request:
https://github.com/apache/spark/pull/6361#discussion_r31436725
--- Diff: core/src/main/scala/org/apache/spark/scheduler/Task.scala ---
@@ -126,6 +129,21 @@ private[spark] abstract class Task[T](val stageId:
Int, var
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-107584210
The test failures are all Hive related, so it seems unlikely, but I'll test
them again in my environment to see if they are related.
---
If your project is s
Github user coolfrood commented on a diff in the pull request:
https://github.com/apache/spark/pull/6361#discussion_r31401728
--- Diff:
core/src/test/scala/org/apache/spark/serializer/KryoClosureSerializerSuite.scala
---
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-107315504
@JoshRosen I have rebased and updated the PR based on your suggestions and
the discussion on SPARK-7708. I switched to Kryo version 2.24.0 by excluding it
from `chill
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6361#issuecomment-104786749
@JoshRosen, thanks for your comments. I'll work on these. Even without
the tests, I have identified two places where Kryo closure serialization
doesn't
GitHub user coolfrood opened a pull request:
https://github.com/apache/spark/pull/6361
[SPARK-7708] [Core] [WIP] Fixes for Kryo closure serialization
This PR partially fixes the use of Kryo serialization for closures. It is
not complete, but I would like to discuss if this is the
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6323#issuecomment-104375418
@JoshRosen: it looks like launchTasks() is only called from makeOffers(),
which in turn is only called from the Actor's receive, so it should be safe to
mov
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/6323#issuecomment-104366893
Argh. Sorry. I'll create a JIRA for this and update the PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
GitHub user coolfrood opened a pull request:
https://github.com/apache/spark/pull/6323
Speed up task scheduling in standalone mode by reusing serializer
My experiments with scheduling very short tasks in standalone cluster mode
indicated that a significant amount of time was being
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/3368#issuecomment-63736110
Ah, OK. I'll keep that in mind for future PRs.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/3368#issuecomment-63735202
@andrewor14: I did that initially, but the Jenkins build still failed for
the same problem.
---
If your project is set up for it, you can reply to this email and have
GitHub user coolfrood opened a pull request:
https://github.com/apache/spark/pull/3373
[SPARK-4478] Keep totalRegisteredExecutors up-to-date
This rebases PR 3368.
This commit fixes totalRegisteredExecutors update [SPARK-4478], so that we
can correctly keep track of number
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/3368#issuecomment-63729410
Closing this PR, so that I can create a new one after the rebase.
---
If your project is set up for it, you can reply to this email and have your
reply appear on
Github user coolfrood closed the pull request at:
https://github.com/apache/spark/pull/3368
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/3368#issuecomment-63726857
@andrewor14: Done, I think. I'm not very sure about rebasing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitH
Github user coolfrood commented on the pull request:
https://github.com/apache/spark/pull/3368#issuecomment-63713138
@andrewor14: I agree. This field isn't actually used for anything right
now; I grepped through the code. My guess would be that the original author
intended it
GitHub user coolfrood opened a pull request:
https://github.com/apache/spark/pull/3368
[SPARK-4478] Keep totalRegisteredExecutors up-to-date
totalRegisteredExecutors keeps track of the number of executors
currently registered with the scheduler. This commit fixes SPARK-4478
27 matches
Mail list logo
