Optimizing Spark interpreter startup

2023-05-03 Thread Vladimir Prus 
Hi,

I was profiling the startup time of Spark Interpreter in our environment,
and it looks like
a total of 5 seconds is spent at this line in
SparkScala212Interpreter.scala:

sparkILoop.initializeSynchronous()

That line, eventually, calls nsc.Global constructor, which spends 5 seconds
creating mirrors
of every class on the classpath. Obviously, most users will never care
about most of those
classes.

Any ideas on how this can be sped up, maybe by only looking at key spark
classes?

[image: image.png]

-- 
Vladimir Prus
http://vladimirprus.com

Re: Optimizing Spark interpreter startup

2023-05-03 Thread Jeff Zhang 
Hi Vladimir,

Have you compared it with spark shell? I think it is similar as spark shell

On Wed, May 3, 2023 at 10:12 PM Vladimir Prus 
wrote:

> Hi,
>
> I was profiling the startup time of Spark Interpreter in our environment,
> and it looks like
> a total of 5 seconds is spent at this line in
> SparkScala212Interpreter.scala:
>
> sparkILoop.initializeSynchronous()
>
> That line, eventually, calls nsc.Global constructor, which spends 5
> seconds creating mirrors
> of every class on the classpath. Obviously, most users will never care
> about most of those
> classes.
>
> Any ideas on how this can be sped up, maybe by only looking at key spark
> classes?
>
> [image: image.png]
>
> --
> Vladimir Prus
> http://vladimirprus.com
>


-- 
Best Regards

Jeff Zhang