Hi all,
In the process of trying to resolve SPARK-3166 (inability to ship custom
serialisers in application jars)
https://issues.apache.org/jira/browse/SPARK-3166 I've discovered that
there's a bit of duplicated code for building the command for launching
Executors across SparkDeploySchedulerBacke
Hi,
Unless you manually patched Spark, if you have Reynold’s patch for SPARK-2878,
you also have the patch for SPARK-2893 which makes the underlying cause much
more obvious and explicit. So the below is unlikely to be related to
SPARK-2878.
Graham
On 26 Aug 2014, at 4:13 am, npanj wrote:
>
Then all deser calls should be using the executor's class loader.
>
>
>
>
> On Thu, Aug 14, 2014 at 12:53 AM, Graham Dennis
> wrote:
>
>> Hi Reynold,
>>
>> That would solve this specific issue, but you'd need to be careful that
>> you n
YARN behaviour.
On 14 August 2014 17:57, Debasish Das wrote:
> By the way I have seen this same problem while deploying 1.1.0-SNAPSHOT on
> YARN as well...
>
> So it is a common problem in both standalone and YARN mode deployment...
>
>
> On Thu, Aug 14, 2014 at 12:53 AM, Graha
solve this problem?
>
>
>
>
> On Wed, Aug 13, 2014 at 11:59 PM, Graham Dennis
> wrote:
>
>> Hi Deb,
>>
>> The only alternative serialiser is the JavaSerialiser (the default).
>> Theoretically Spark supports custom serialisers, but due to a related
the other
> recommended efficient serializers ?
>
> Thanks.
> Deb
>
>
> On Wed, Aug 13, 2014 at 2:47 PM, Graham Dennis
> wrote:
>
>> I now have a complete pull request for this issue that I'd like to get
>> reviewed and committed. The PR is available here
1827) that causes exceptions raised
while attempting to run the custom kryo registrator not to be swallowed.
Thanks,
Graham
On 12 August 2014 18:44, Graham Dennis wrote:
> I've submitted a work-in-progress pull request for this issue that I'd
> like feedback on. See https://gi
d. As this PR
adds user jars to the Executor process at launch time, this won't be an
issue.
On 7 August 2014 12:01, Graham Dennis wrote:
> See my comment on https://issues.apache.org/jira/browse/SPARK-2878 for
> the full stacktrace, but it's in the BlockManager/BlockManagerWor
> to stay away from that right now.
>
>
>
>
>
> On Wed, Aug 6, 2014 at 6:47 PM, Graham Dennis
> wrote:
>
>> Hi Spark devs,
>>
>> I’ve posted an issue on JIRA (
>> https://issues.apache.org/jira/browse/SPARK-2878) which occurs when using
>> Kryo
Hi Spark devs,
I’ve posted an issue on JIRA (
https://issues.apache.org/jira/browse/SPARK-2878) which occurs when using
Kryo serialisation with a custom Kryo registrator to register custom
classes with Kryo. This is an insidious issue that non-deterministically
causes Kryo to have different ID nu
10 matches
Mail list logo
