[
https://issues.apache.org/jira/browse/TOREE-372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854877#comment-15854877
]
Jakob Odersky commented on TOREE-372:
-------------------------------------
Spark requires its driver app (SparkContext) and all executors to run on
machines with the same endianess. In the described scenario, Toree acts as the
driver app and therefore cannot communicate with the Spark executors running in
a different-endian environment.
There is currently no fix available, however you may be able to work around the
issue by running toree and jupyter in the same environment as your spark
cluster, and proxying connections with something like
[nb2kg|https://github.com/jupyter/kernel_gateway_demos/tree/master/nb2kg].
> stream corruption cased by big-endian and little-endian
> -------------------------------------------------------
>
> Key: TOREE-372
> URL: https://issues.apache.org/jira/browse/TOREE-372
> Project: TOREE
> Issue Type: Bug
> Reporter: Wang enzhong
> Priority: Critical
>
> We currently run spark on z/OS system which is a big-endian platform and
> jupyter+toree on x86 platform which is small-endian platform. The output
> from spark is unreadable caused by the different byte order.
> If we use spark on z/OS and jupyter+toree on another big-endian platform,
> there is no such error.
> I've done some investigation and it seems toree leverages the bytestring of
> Akka which has endian process, don't know why toree does not work in our
> case.
> Please help to look into the problem. Due to the tight project schedule, it
> will be much appreciated if you can give us some advice on how to fix or
> avoid the problem if it will take some time to change the code. Many thanks
> in advance.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
