Can this assembly get faster if we do not need the Spark SQL or some other
component in spark ? such as we only need the core of spark.
On Wed, Nov 26, 2014 at 3:37 PM, lihu lihu...@gmail.com wrote:
Matei, sorry for my last typo error. And the tip can improve about 30s in
my computer.
On
Hi,
The spark assembly is time costly. If I only need
the spark-assembly-1.1.0-hadoop2.3.0.jar, do not need
the spark-examples-1.1.0-hadoop2.3.0.jar. How to configure the spark to
avoid assemble the example jar. I know *export
SPARK_PREPEND_CLASSES=**true* method
can reduce the assembly, but
You can do sbt/sbt assembly/assembly to assemble only the main package.
Matei
On Nov 25, 2014, at 7:50 PM, lihu lihu...@gmail.com wrote:
Hi,
The spark assembly is time costly. If I only need the
spark-assembly-1.1.0-hadoop2.3.0.jar, do not need the
BTW as another tip, it helps to keep the SBT console open as you make source
changes (by just running sbt/sbt with no args). It's a lot faster the second
time it builds something.
Matei
On Nov 25, 2014, at 8:31 PM, Matei Zaharia matei.zaha...@gmail.com wrote:
You can do sbt/sbt
Mater, thank you very much!
After take your advice, the time for assembly from about 20min down to 6min
in my computer. that's a very big improvement.
On Wed, Nov 26, 2014 at 12:32 PM, Matei Zaharia matei.zaha...@gmail.com
wrote:
BTW as another tip, it helps to keep the SBT console open as you
Matei, sorry for my last typo error. And the tip can improve about 30s in
my computer.
On Wed, Nov 26, 2014 at 3:34 PM, lihu lihu...@gmail.com wrote:
Mater, thank you very much!
After take your advice, the time for assembly from about 20min down to
6min in my computer. that's a very big
