Thank you, I would expect it to work as you write, but I am probably
experiencing it working other way. But now it seems that Spark is generally
trying to fit everything to RAM. I run Spark on YARN and I have wraped this to
another question:
Now I am getting to problems using:
distData = sc.textFile(sys.argv[2]).coalesce(10)
The problem is that it seems that Spark is trying to put all the data to RAM
first and then perform coalesce. Do you know if there is something that would
do coalesce on fly with for example fixed size of
coalesce() is a streaming operation if used without the second parameter,
it does not put all the data in RAM. If used with the second parameter
(shuffle = true), then it performs a shuffle, but still does not put all
the data in RAM.
On Sat, Nov 1, 2014 at 12:09 PM, jan.zi...@centrum.cz wrote:
Thank you very much lot of very small json files was exactly the speed
performance problem, using coalesce makes my Spark program to run on single
node only twice slower (even with starting Spark) than single node Python
program, which is acceptable.
Jan
Hi,
I have program that I have for single computer (in Python) exection and also
implemented the same for Spark. This program basically only reads .json from
which it takes one field and saves it back. Using Spark my program runs
aproximately 100 times slower on 1 master and 1 slave. So I
How many files do you have and how big is each JSON object?
Spark works better with a few big files vs many smaller ones. So you could try
cat'ing your files together and rerunning the same experiment.
- Evan
On Oct 18, 2014, at 12:07 PM, jan.zi...@centrum.cz jan.zi...@centrum.cz
wrote:
How many CPUs on the slave?
Because the overhead between JVM and Python, single task will be
slower than your local Python scripts, but it's very easy to scale to
many CPUs.
Even one CPUs, it's not common that PySpark was 100 times slower. You
have many small files, each file will be processed
