Yes. With the image & vm I'm having trouble with I get an array with 9,942
elements in it. So its works as you'd expect.
While processing the CSV file the image stays at about 60MB in RAM.
Sven Van Caekenberghe-2 wrote
> Can you successfully run my example code ?
>
>> On 14 Nov 2014, at 22:03, Paul DeBruicker <
> pdebruic@
> > wrote:
>>
>> Hi Sven,
>>
>> Thanks for taking a look and testing the NeoCSVReader portion for me.
>> You're right of course that there's something I'm doing that's slow.
>> But.
>> There is something I can't figure out yet.
>>
>> To provide a little more detail:
>>
>> When the 'csv reading' process completes successfully profiling shows
>> that
>> most of the time is spent in NeoCSVReader>>#peekChar and using
>> NeoCSVReader>>##addField: to convert a string to a DateAndTime. Dropping
>> the DateAndTime conversion speeds things up but doesn't stop it from
>> running
>> out of memory.
>>
>> I start the image with
>>
>> ./pharo-ui --memory 1000m myimage.image
>>
>> Splitting the CSV file helps:
>> ~1.5MB 5,000 lines = 1.2 seconds.
>> ~15MB 50,000 lines = 8 seconds.
>> ~30MB 100,000 lines = 16 seconds.
>> ~60MB 200,000 lines = 45 seconds.
>>
>>
>> It seems that when the CSV file crosses ~70MB in size things start going
>> haywire with performance, and leads to the out of memory condition. The
>> processing never ends. Sending "kill -SIGUSR1" prints a stack primarily
>> composed of:
>>
>> 0xbffc5d08 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n)
>> OutOfMemory class
>> 0xbffc5d20 M OutOfMemory class(Behavior)>basicNew 0x1f7ac060: a(n)
>> OutOfMemory class
>> 0xbffc5d38 M OutOfMemory class(Behavior)>new 0x1f7ac060: a(n) OutOfMemory
>> class
>> 0xbffc5d50 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n)
>> OutOfMemory class
>> 0xbffc5d68 M OutOfMemory class(Behavior)>basicNew 0x1f7ac060: a(n)
>> OutOfMemory class
>> 0xbffc5d80 M OutOfMemory class(Behavior)>new 0x1f7ac060: a(n) OutOfMemory
>> class
>> 0xbffc5d98 M OutOfMemory class(Exception class)>signal 0x1f7ac060: a(n)
>> OutOfMemory class
>>
>> So it seems like its trying to signal that its out of memory after its
>> out
>> of memory which triggers another OutOfMemory error. So that's why
>> progress
>> stops.
>>
>>
>> ** Aside - OutOfMemory should probably be refactored to be able to signal
>> itself without taking up more memory, triggering itself infinitely.
>> Maybe
>> it & its signalling morph infrastructure would be good as a singleton **
>>
>>
>>
>> I'm confused about why it runs out of memory. According to htop the
>> image
>> only takes up about 520-540 MB of RAM when it reaches the 'OutOfMemory'
>> condition. This Macbook Air laptop has 4GB, and has plenty of room for
>> the
>> image to grow. Also I've specified a 1,000MB image size when starting.
>> So
>> it should have plenty of room. Is there something I should check or a
>> flag
>> somewhere that prevents it from growing on a Mac? This is the latest
>> Pharo30 VM.
>>
>>
>> Thanks for helping me get to the bottom of this
>>
>> Paul
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Sven Van Caekenberghe-2 wrote
>>> Hi Paul,
>>>
>>> I think you must be doing something wrong with your class, the #do: is
>>> implemented as streaming over the record one by one, never holding more
>>> than one in memory.
>>>
>>> This is what I tried:
>>>
>>> 'paul.csv' asFileReference writeStreamDo: [ :file|
>>> ZnBufferedWriteStream on: file do: [ :out |
>>> (NeoCSVWriter on: out) in: [ :writer |
>>> writer writeHeader: { #Number. #Color. #Integer. #Boolean}.
>>> 1 to: 1e7 do: [ :each |
>>> writer nextPut: { each. #(Red Green Blue) atRandom. 1e6 atRandom.
>>> #(true false) atRandom } ] ] ] ].
>>>
>>> This results in a 300Mb file:
>>>
>>> $ ls -lah paul.csv
>>> -rw-r--r--@ 1 sven staff 327M Nov 14 20:45 paul.csv
>>> $ wc paul.csv
>>> 10000001 10000001 342781577 paul.csv
>>>
>>> This is a selective read and collect (loads about 10K records):
>>>
>>> Array streamContents: [ :out |
>>> 'paul.csv' asFileReference readStreamDo: [ :in |
>>> (NeoCSVReader on: (ZnBufferedReadStream on: in)) in: [ :reader |
>>> reader skipHeader; addIntegerField; addSymbolField;
>>> addIntegerField;
>>> addFieldConverter: [ :x | x = #true ].
>>> reader do: [ :each | each third < 1000 ifTrue: [ out nextPut: each
>>> ]
>>> ] ] ] ].
>>>
>>> This worked fine on my MacBook Air, no memory problems. It takes a while
>>> to parse that much data, of course.
>>>
>>> Sven
>>>
>>>> On 14 Nov 2014, at 19:08, Paul DeBruicker <
>>
>>> pdebruic@
>>
>>> > wrote:
>>>>
>>>> Hi -
>>>>
>>>> I'm processing a 9 GBs of CSV files (the biggest file is 220MB or so).
>>>> I'm not sure if its because of the size of the files or the code I've
>>>> written to keep track of the domain objects I'm interested in, but I'm
>>>> getting out of memory errors & crashes in Pharo 3 on Mac with the
>>>> latest
>>>> VM. I haven't checked other vms.
>>>>
>>>> I'm going to profile my own code and attempt to split the files
>>>> manually
>>>> for now to see what else it could be.
>>>>
>>>>
>>>> Right now I'm doing something similar to
>>>>
>>>> |file reader|
>>>> file:= '/path/to/file/myfile.csv' asFileReference readStream.
>>>> reader: NeoCSVReader on: file
>>>>
>>>> reader
>>>> recordClass: MyClass;
>>>> skipHeader;
>>>> addField: #myField:;
>>>> ....
>>>>
>>>>
>>>> reader do:[:eachRecord | self seeIfRecordIsInterestingAndIfSoKeepIt:
>>>> eachRecord].
>>>> file close.
>>>>
>>>>
>>>>
>>>> Is there a facility in NeoCSVReader to read a file in batches (e.g.
>>>> 1000
>>>> lines at a time) or an easy way to do that ?
>>>>
>>>>
>>>>
>>>>
>>>> Thanks
>>>>
>>>> Paul
>>
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-file-with-NeoCSVReader-tips-tp4790264p4790319.html
>> Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
--
View this message in context:
http://forum.world.st/running-out-of-memory-while-processing-a-220MB-csv-file-with-NeoCSVReader-tips-tp4790264p4790328.html
Sent from the Pharo Smalltalk Users mailing list archive at Nabble.com.
