Hi all,
In the process of trying to resolve SPARK-3166 (inability to ship custom
serialisers in application jars)
https://issues.apache.org/jira/browse/SPARK-3166 I've discovered that
there's a bit of duplicated code for building the command for launching
Executors across
Hi,
Unless you manually patched Spark, if you have Reynold’s patch for SPARK-2878,
you also have the patch for SPARK-2893 which makes the underlying cause much
more obvious and explicit. So the below is unlikely to be related to
SPARK-2878.
Graham
On 26 Aug 2014, at 4:13 am, npanj
serializers ?
Thanks.
Deb
On Wed, Aug 13, 2014 at 2:47 PM, Graham Dennis graham.den...@gmail.com
wrote:
I now have a complete pull request for this issue that I'd like to get
reviewed and committed. The PR is available here:
https://github.com/apache/spark/pull/1890 and includes a testcase
, 2014 at 11:59 PM, Graham Dennis graham.den...@gmail.com
wrote:
Hi Deb,
The only alternative serialiser is the JavaSerialiser (the default).
Theoretically Spark supports custom serialisers, but due to a related
issue, custom serialisers currently can't live in application jars and must
behaviour.
On 14 August 2014 17:57, Debasish Das debasish.da...@gmail.com wrote:
By the way I have seen this same problem while deploying 1.1.0-SNAPSHOT on
YARN as well...
So it is a common problem in both standalone and YARN mode deployment...
On Thu, Aug 14, 2014 at 12:53 AM, Graham Dennis
) that causes exceptions raised
while attempting to run the custom kryo registrator not to be swallowed.
Thanks,
Graham
On 12 August 2014 18:44, Graham Dennis graham.den...@gmail.com wrote:
I've submitted a work-in-progress pull request for this issue that I'd
like feedback on. See https
to the Executor process at launch time, this won't be an
issue.
On 7 August 2014 12:01, Graham Dennis graham.den...@gmail.com wrote:
See my comment on https://issues.apache.org/jira/browse/SPARK-2878 for
the full stacktrace, but it's in the BlockManager/BlockManagerWorker where
it's trying
Hi Spark devs,
I’ve posted an issue on JIRA (
https://issues.apache.org/jira/browse/SPARK-2878) which occurs when using
Kryo serialisation with a custom Kryo registrator to register custom
classes with Kryo. This is an insidious issue that non-deterministically
causes Kryo to have different ID
.
On Wed, Aug 6, 2014 at 6:47 PM, Graham Dennis graham.den...@gmail.com
wrote:
Hi Spark devs,
I’ve posted an issue on JIRA (
https://issues.apache.org/jira/browse/SPARK-2878) which occurs when using
Kryo serialisation with a custom Kryo registrator to register custom
classes with Kryo
