Could you modify your function so that it streams through the files record by record and outputs them to hdfs, then read them all in as RDDs and take the union? That would only use bounded memory.
On 1 December 2014 at 17:19, Keith Simmons <ke...@pulse.io> wrote: > Actually, I'm working with a binary format. The api allows reading out a > single record at a time, but I'm not sure how to get those records into > spark (without reading everything into memory from a single file at once). > > > > On Mon, Dec 1, 2014 at 5:07 PM, Andy Twigg <andy.tw...@gmail.com> wrote: > >> file => tranform file into a bunch of records >> >> >> What does this function do exactly? Does it load the file locally? >> Spark supports RDDs exceeding global RAM (cf the terasort example), but >> if your example just loads each file locally, then this may cause problems. >> Instead, you should load each file into an rdd with context.textFile(), >> flatmap that and union these rdds. >> >> also see >> >> http://stackoverflow.com/questions/23397907/spark-context-textfile-load-multiple-files >> >> >> On 1 December 2014 at 16:50, Keith Simmons <ke...@pulse.io> wrote: >> >>> This is a long shot, but... >>> >>> I'm trying to load a bunch of files spread out over hdfs into an RDD, >>> and in most cases it works well, but for a few very large files, I exceed >>> available memory. My current workflow basically works like this: >>> >>> context.parallelize(fileNames).flatMap { file => >>> tranform file into a bunch of records >>> } >>> >>> I'm wondering if there are any APIs to somehow "flush" the records of a >>> big dataset so I don't have to load them all into memory at once. I know >>> this doesn't exist, but conceptually: >>> >>> context.parallelize(fileNames).streamMap { (file, stream) => >>> for every 10K records write records to stream and flush >>> } >>> >>> Keith >>> >> >> >
