Shadowed with
object MyObject {
def mymethod(param: MyParam) = actual_function(param)
}
class MyObject {
import MyObject._
session.map { ... =>
mymethod(...)
}
}
does the job.
Thanks for the advice!
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Friday, November 30, 2018 9:26 AM, <ch...@cmartinit.co.uk> wrote:
> If it’s just a couple of classes and they are actually suitable for
> serializing and you have the source code then you can shadow them in your own
> project with the serializable interface added. Your shadowed classes should
> be on the classpath before the library’s versions which should lead to spark
> being able to use the serializable versions.
>
> That’s very much a last resort though!
>
> Chris
>
> On 30 Nov 2018, at 05:08, Koert Kuipers <ko...@tresata.com> wrote:
>
>> if you only use it in the executors sometimes using lazy works
>>
>> On Thu, Nov 29, 2018 at 9:45 AM James Starks
>> <suse...@protonmail.com.invalid> wrote:
>>
>>> This is not problem directly caused by Spark, but it's related; thus asking
>>> here. I use spark to read data from parquet and processing some http call
>>> with sttp (https://github.com/softwaremill/sttp). However, spark throws
>>>
>>> Caused by: java.io.NotSerializableException:
>>> com.softwaremill.sttp.FollowRedirectsBackend
>>>
>>> It's understood why such exception is thrown because
>>> FollowRedirectsBackend is not seralizable. So I would like know in such
>>> case - are there any ways to get around this problem without modifying,
>>> recompiling original code?
>>>
>>> Thanks
