Your Spark is trying to load a hadoop library "winutils.exe", which you
don't have in your Windows:
14/12/16 12:48:28 ERROR Shell: Failed to locate the winutils binary in the
hadoop binary path
java.io.IOException: Could not locate executable null\bin\winutils.exe in
the Hadoop binaries.
at org.apache.hadoop.util.Shell.
getQualifiedBinPath(Shell.java:318)
...
It's a known bug: https://issues.apache.org/jira/browse/SPARK-2356
That issue references this email thread:
http://qnalist.com/questions/4994960/run-spark-unit-test-on-windows-7
And that email thread references this blog post:
https://social.msdn.microsoft.com/forums/azure/en-US/28a57efb-082b-424b-8d9e-731b1fe135de/please-read-if-experiencing-job-failures?forum=hdinsight
I had the same problem before, you may temporarily solve it by using the
distribution from the summit:
https://databricks-training.s3.amazonaws.com/getting-started.html
Or you may want to try that other solution.
...but in my case, I ended up running Spark from a Linux machine in a VM
after I got other errors.
I have the impression that development for Windows is not currently a big
priority, since the bug is from version 1.0...
I hope that helps.
Best,
*Sebastián Ramírez*
Diseñador de Algoritmos
<http://www.senseta.com>
________________
Tel: (+571) 795 7950 ext: 1012
Cel: (+57) 300 370 77 10
Calle 73 No 7 - 06 Piso 4
Linkedin: co.linkedin.com/in/tiangolo/
Twitter: @tiangolo <https://twitter.com/tiangolo>
Email: sebastian.rami...@senseta.com
www.senseta.com
On Tue, Dec 16, 2014 at 8:04 AM, mj <jone...@gmail.com> wrote:
>
> I've got a simple pyspark program that generates two CSV files and then
> carries out a leftOuterJoin (a fact RDD joined to a dimension RDD). The
> program works fine for smaller volumes of records, but when it goes beyond
> 3
> million records for the fact dataset, I get the error below. I'm running
> PySpark via PyCharm and the information for my environment is:
>
> OS: Windows 7
> Python version: 2.7.9
> Spark version: 1.1.1
> Java version: 1.8
>
> I've also included the py file I am using. I'd appreciate any help you can
> give me,
>
> MJ.
>
>
> ----------------------------ERROR MESSAGE----------------------------
> C:\Python27\python.exe "C:/Users/Mark
> Jones/PycharmProjects/spark_test/spark_error_sample.py"
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 14/12/16 12:48:26 INFO SecurityManager: Changing view acls to: Mark Jones,
> 14/12/16 12:48:26 INFO SecurityManager: Changing modify acls to: Mark
> Jones,
> 14/12/16 12:48:26 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(Mark Jones, );
> users with modify permissions: Set(Mark Jones, )
> 14/12/16 12:48:26 INFO Slf4jLogger: Slf4jLogger started
> 14/12/16 12:48:27 INFO Remoting: Starting remoting
> 14/12/16 12:48:27 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkDriver@192.168.19.83:51387]
> 14/12/16 12:48:27 INFO Remoting: Remoting now listens on addresses:
> [akka.tcp://sparkDriver@192.168.19.83:51387]
> 14/12/16 12:48:27 INFO Utils: Successfully started service 'sparkDriver' on
> port 51387.
> 14/12/16 12:48:27 INFO SparkEnv: Registering MapOutputTracker
> 14/12/16 12:48:27 INFO SparkEnv: Registering BlockManagerMaster
> 14/12/16 12:48:27 INFO DiskBlockManager: Created local directory at
> C:\Users\MARKJO~1\AppData\Local\Temp\spark-local-20141216124827-11ef
> 14/12/16 12:48:27 INFO Utils: Successfully started service 'Connection
> manager for block manager' on port 51390.
> 14/12/16 12:48:27 INFO ConnectionManager: Bound socket to port 51390 with
> id
> = ConnectionManagerId(192.168.19.83,51390)
> 14/12/16 12:48:27 INFO MemoryStore: MemoryStore started with capacity 265.1
> MB
> 14/12/16 12:48:27 INFO BlockManagerMaster: Trying to register BlockManager
> 14/12/16 12:48:27 INFO BlockManagerMasterActor: Registering block manager
> 192.168.19.83:51390 with 265.1 MB RAM
> 14/12/16 12:48:27 INFO BlockManagerMaster: Registered BlockManager
> 14/12/16 12:48:27 INFO HttpFileServer: HTTP File server directory is
>
> C:\Users\MARKJO~1\AppData\Local\Temp\spark-3b772ca1-dbf7-4eaa-b62c-be5e73036f5d
> 14/12/16 12:48:27 INFO HttpServer: Starting HTTP Server
> 14/12/16 12:48:27 INFO Utils: Successfully started service 'HTTP file
> server' on port 51391.
> 14/12/16 12:48:27 INFO Utils: Successfully started service 'SparkUI' on
> port
> 4040.
> 14/12/16 12:48:27 INFO SparkUI: Started SparkUI at
> http://192.168.19.83:4040
> 14/12/16 12:48:27 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 14/12/16 12:48:28 ERROR Shell: Failed to locate the winutils binary in the
> hadoop binary path
> java.io.IOException: Could not locate executable null\bin\winutils.exe in
> the Hadoop binaries.
> at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:318)
> at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:333)
> at org.apache.hadoop.util.Shell.<clinit>(Shell.java:326)
> at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:76)
> at
> org.apache.hadoop.security.Groups.parseStaticMapping(Groups.java:93)
> at org.apache.hadoop.security.Groups.<init>(Groups.java:77)
> at
>
> org.apache.hadoop.security.Groups.getUserToGroupsMappingService(Groups.java:240)
> at
>
> org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:255)
> at
>
> org.apache.hadoop.security.UserGroupInformation.setConfiguration(UserGroupInformation.java:283)
> at
> org.apache.spark.deploy.SparkHadoopUtil.<init>(SparkHadoopUtil.scala:36)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.<init>(SparkHadoopUtil.scala:109)
> at
> org.apache.spark.deploy.SparkHadoopUtil$.<clinit>(SparkHadoopUtil.scala)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:228)
> at
>
> org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:53)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
> at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:234)
> at
> py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
> at py4j.Gateway.invoke(Gateway.java:214)
> at
>
> py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:79)
> at
> py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:68)
> at py4j.GatewayConnection.run(GatewayConnection.java:207)
> at java.lang.Thread.run(Thread.java:745)
> 14/12/16 12:48:28 INFO AkkaUtils: Connecting to HeartbeatReceiver:
> akka.tcp://sparkDriver@192.168.19.83:51387/user/HeartbeatReceiver
> 14/12/16 12:48:28 INFO MemoryStore: ensureFreeSpace(159118) called with
> curMem=0, maxMem=278019440
> 14/12/16 12:48:28 INFO MemoryStore: Block broadcast_0 stored as values in
> memory (estimated size 155.4 KB, free 265.0 MB)
> 14/12/16 12:48:28 INFO MemoryStore: ensureFreeSpace(159118) called with
> curMem=159118, maxMem=278019440
> 14/12/16 12:48:28 INFO MemoryStore: Block broadcast_1 stored as values in
> memory (estimated size 155.4 KB, free 264.8 MB)
> 14/12/16 12:48:28 INFO FileInputFormat: Total input paths to process : 1
> 14/12/16 12:48:28 INFO FileInputFormat: Total input paths to process : 1
> 14/12/16 12:48:28 INFO SparkContext: Starting job: count at C:/Users/Mark
> Jones/PycharmProjects/spark_test/spark_error_sample.py:79
> 14/12/16 12:48:28 INFO DAGScheduler: Registering RDD 8 (RDD at
> PythonRDD.scala:261)
> 14/12/16 12:48:28 INFO DAGScheduler: Got job 0 (count at C:/Users/Mark
> Jones/PycharmProjects/spark_test/spark_error_sample.py:79) with 9 output
> partitions (allowLocal=false)
> 14/12/16 12:48:28 INFO DAGScheduler: Final stage: Stage 0(count at
> C:/Users/Mark Jones/PycharmProjects/spark_test/spark_error_sample.py:79)
> 14/12/16 12:48:28 INFO DAGScheduler: Parents of final stage: List(Stage 1)
> 14/12/16 12:48:28 INFO DAGScheduler: Missing parents: List(Stage 1)
> 14/12/16 12:48:28 INFO DAGScheduler: Submitting Stage 1 (PairwiseRDD[8] at
> RDD at PythonRDD.scala:261), which has no missing parents
> 14/12/16 12:48:28 INFO MemoryStore: ensureFreeSpace(11672) called with
> curMem=318236, maxMem=278019440
> 14/12/16 12:48:28 INFO MemoryStore: Block broadcast_2 stored as values in
> memory (estimated size 11.4 KB, free 264.8 MB)
> 14/12/16 12:48:28 INFO DAGScheduler: Submitting 9 missing tasks from Stage
> 1
> (PairwiseRDD[8] at RDD at PythonRDD.scala:261)
> 14/12/16 12:48:28 INFO TaskSchedulerImpl: Adding task set 1.0 with 9 tasks
> 14/12/16 12:48:28 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID
> 0, localhost, PROCESS_LOCAL, 1275 bytes)
> 14/12/16 12:48:28 INFO Executor: Running task 0.0 in stage 1.0 (TID 0)
> 14/12/16 12:48:29 INFO HadoopRDD: Input split:
> file:/c:/tmp/fact.csv:0+33554432
> 14/12/16 12:48:29 INFO deprecation: mapred.tip.id is deprecated. Instead,
> use mapreduce.task.id
> 14/12/16 12:48:29 INFO deprecation: mapred.task.id is deprecated. Instead,
> use mapreduce.task.attempt.id
> 14/12/16 12:48:29 INFO deprecation: mapred.task.is.map is deprecated.
> Instead, use mapreduce.task.ismap
> 14/12/16 12:48:29 INFO deprecation: mapred.task.partition is deprecated.
> Instead, use mapreduce.task.partition
> 14/12/16 12:48:29 INFO deprecation: mapred.job.id is deprecated. Instead,
> use mapreduce.job.id
> 14/12/16 12:48:51 INFO PythonRDD: Times: total = 22350, boot = 271, init =
> 59, finish = 22020
> 14/12/16 12:48:52 ERROR Executor: Exception in task 0.0 in stage 1.0 (TID
> 0)
> org.apache.spark.api.python.PythonException: Traceback (most recent call
> last):
> File "C:\apps\spark\python\pyspark\worker.py", line 79, in main
> serializer.dump_stream(func(split_index, iterator), outfile)
> File "C:\apps\spark\python\pyspark\serializers.py", line 127, in
> dump_stream
> for obj in iterator:
> File "C:\apps\spark\python\pyspark\rdd.py", line 1495, in add_shuffle_key
> for (k, v) in iterator:
> File "C:\apps\spark\python\pyspark\shuffle.py", line 371, in
> _external_items
> False)
> File "C:\apps\spark\python\pyspark\shuffle.py", line 281, in
> mergeCombiners
> for k, v in iterator:
> File "C:\apps\spark\python\pyspark\serializers.py", line 133, in
> load_stream
> yield self._read_with_length(stream)
> File "C:\apps\spark\python\pyspark\serializers.py", line 146, in
> _read_with_length
> length = read_int(stream)
> File "C:\apps\spark\python\pyspark\serializers.py", line 465, in read_int
> return struct.unpack("!i", length)[0]
> error: unpack requires a string argument of length 4
>
> at
> org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:124)
> at
> org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:154)
> at
> org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:87)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at
> org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:265)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> at org.apache.spark.scheduler.Task.run(Task.scala:54)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 14/12/16 12:48:52 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID
> 1, localhost, PROCESS_LOCAL, 1275 bytes)
> 14/12/16 12:48:52 INFO Executor: Running task 1.0 in stage 1.0 (TID 1)
> 14/12/16 12:48:52 WARN TaskSetManager: Lost task 0.0 in stage 1.0 (TID 0,
> localhost): org.apache.spark.api.python.PythonException: Traceback (most
> recent call last):
> File "C:\apps\spark\python\pyspark\worker.py", line 79, in main
> serializer.dump_stream(func(split_index, iterator), outfile)
> File "C:\apps\spark\python\pyspark\serializers.py", line 127, in
> dump_stream
> for obj in iterator:
> File "C:\apps\spark\python\pyspark\rdd.py", line 1495, in add_shuffle_key
> for (k, v) in iterator:
> File "C:\apps\spark\python\pyspark\shuffle.py", line 371, in
> _external_items
> False)
> File "C:\apps\spark\python\pyspark\shuffle.py", line 281, in
> mergeCombiners
> for k, v in iterator:
> File "C:\apps\spark\python\pyspark\serializers.py", line 133, in
> load_stream
> yield self._read_with_length(stream)
> File "C:\apps\spark\python\pyspark\serializers.py", line 146, in
> _read_with_length
> length = read_int(stream)
> File "C:\apps\spark\python\pyspark\serializers.py", line 465, in read_int
> return struct.unpack("!i", length)[0]
> error: unpack requires a string argument of length 4
>
>
> org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:124)
>
> org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:154)
> org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:87)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>
> org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:265)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> org.apache.spark.scheduler.Task.run(Task.scala:54)
>
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
>
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> 14/12/16 12:48:52 ERROR TaskSetManager: Task 0 in stage 1.0 failed 1 times;
> aborting job
> 14/12/16 12:48:52 INFO TaskSchedulerImpl: Cancelling stage 1
> 14/12/16 12:48:52 INFO TaskSchedulerImpl: Stage 1 was cancelled
> 14/12/16 12:48:52 INFO Executor: Executor is trying to kill task 1.0 in
> stage 1.0 (TID 1)
> 14/12/16 12:48:52 INFO DAGScheduler: Failed to run count at C:/Users/Mark
> Jones/PycharmProjects/spark_test/spark_error_sample.py:79
> Traceback (most recent call last):
> File "C:/Users/Mark
> Jones/PycharmProjects/spark_test/spark_error_sample.py", line 79, in
> <module>
> print("Joined fact count: %d" % joined_facts.count())
> File "C:\apps\spark\python\pyspark\rdd.py", line 847, in count
> return self.mapPartitions(lambda i: [sum(1 for _ in i)]).sum()
> File "C:\apps\spark\python\pyspark\rdd.py", line 838, in sum
> return self.mapPartitions(lambda x: [sum(x)]).reduce(operator.add)
> File "C:\apps\spark\python\pyspark\rdd.py", line 759, in reduce
> vals = self.mapPartitions(func).collect()
> File "C:\apps\spark\python\pyspark\rdd.py", line 723, in collect
> bytesInJava = self._jrdd.collect().iterator()
> File "C:\apps\spark\python\build\py4j\java_gateway.py", line 538, in
> __call__
> self.target_id, self.name)
> File "C:\apps\spark\python\build\py4j\protocol.py", line 300, in
> get_return_value
> format(target_id, '.', name), value)
> py4j.protocol.Py4JJavaError: An error occurred while calling o55.collect.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0
> in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage
> 1.0
> (TID 0, localhost): org.apache.spark.api.python.PythonException: Traceback
> (most recent call last):
> File "C:\apps\spark\python\pyspark\worker.py", line 79, in main
> serializer.dump_stream(func(split_index, iterator), outfile)
> File "C:\apps\spark\python\pyspark\serializers.py", line 127, in
> dump_stream
> for obj in iterator:
> File "C:\apps\spark\python\pyspark\rdd.py", line 1495, in add_shuffle_key
> for (k, v) in iterator:
> File "C:\apps\spark\python\pyspark\shuffle.py", line 371, in
> _external_items
> False)
> File "C:\apps\spark\python\pyspark\shuffle.py", line 281, in
> mergeCombiners
> for k, v in iterator:
> File "C:\apps\spark\python\pyspark\serializers.py", line 133, in
> load_stream
> yield self._read_with_length(stream)
> File "C:\apps\spark\python\pyspark\serializers.py", line 146, in
> _read_with_length
> length = read_int(stream)
> File "C:\apps\spark\python\pyspark\serializers.py", line 465, in read_int
> return struct.unpack("!i", length)[0]
> error: unpack requires a string argument of length 4
>
>
> org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:124)
>
> org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:154)
> org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:87)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>
> org.apache.spark.api.python.PairwiseRDD.compute(PythonRDD.scala:265)
> org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:68)
>
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
> org.apache.spark.scheduler.Task.run(Task.scala:54)
>
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:177)
>
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> java.lang.Thread.run(Thread.java:745)
> Driver stacktrace:
> at
> org.apache.spark.scheduler.DAGScheduler.org
> $apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1185)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1174)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1173)
> at
>
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1173)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at
>
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:688)
> at scala.Option.foreach(Option.scala:236)
> at
>
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:688)
> at
>
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1391)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
> at akka.dispatch.Mailbox.run(Mailbox.scala:219)
> at
>
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
> at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
>
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
>
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
> Process finished with exit code 1
>
>
>
> ----------------------------PYTHON FILE----------------------------
> import random
> from pyspark import SparkContext, SparkConf, StorageLevel
>
> DIM_FILE_NAME="c:/tmp/dim.csv"
> FACT_FILE_NAME="c:/tmp/fact.csv"
> MIN_PARTITIONS=4
>
> class DataGenerator():
>
> def __init__(self):
> pass
>
> def generate_dimension_file(self, file_name, num_items=100):
>
> with open(file_name, 'w') as dim_file:
>
> for i in range(0, num_items):
> id = i
> name = "Dimension Item #%d" % id
>
> dim_file.write("%d,%s\n" %(id, name))
>
> dim_file.flush()
>
> def generate_fact_file(self, file_name, num_items=100):
>
> with open(file_name, 'w') as fact_file:
>
> for i in range(0, num_items):
> fact_id = i
> name = "Fact Item #%d" % fact_id
> dim_id=random.randint(0,99)
>
> fact_file.write("%d,%d,%s\n" %(fact_id, dim_id, name))
>
> fact_file.flush()
>
> class CsvConverter():
>
> def __init__(self):
> pass
>
> def read_dimension(self, line):
>
> items = line.split(',')
>
> return (int(items[0]), items)
>
> def read_fact(self, line):
>
> items = line.split(',')
>
> return (int(items[1]), items)
>
> if __name__ == "__main__":
>
> # Generate data
> generator = DataGenerator()
> generator.generate_dimension_file(file_name=DIM_FILE_NAME,
> num_items=100)
> generator.generate_fact_file(file_name=FACT_FILE_NAME,
> num_items=5000000)
>
>
> # Load files into spark
> appName = "spark_error_sample"
> master = "local"
> conf =
>
> SparkConf().setAppName(appName).setMaster(master).set('spark.executor.memory',
> '4g')
> sc = SparkContext(conf=conf)
>
> dimension_lines=sc.textFile(DIM_FILE_NAME,
> minPartitions=MIN_PARTITIONS)
> fact_lines=sc.textFile(FACT_FILE_NAME, minPartitions=MIN_PARTITIONS)
>
> converter = CsvConverter()
>
> dimensions = dimension_lines.map(converter.read_dimension)
> facts = fact_lines.map(converter.read_fact)
>
> joined_facts = facts.leftOuterJoin(dimensions)
>
> print("Joined fact count: %d" % joined_facts.count())
> print(joined_facts.first())
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Pyspark-1-1-1-error-with-large-number-of-records-serializer-dump-stream-func-split-index-iterator-ou-tp20713.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
--
*----------------------------------------------------*
*This e-mail transmission, including any attachments, is intended only for
the named recipient(s) and may contain information that is privileged,
confidential and/or exempt from disclosure under applicable law. If you
have received this transmission in error, or are not the named
recipient(s), please notify Senseta immediately by return e-mail and
permanently delete this transmission, including any attachments.*
