nd are in the same order. I want to
>>> create a new data frame that contains the 3rd column from each data file.
>>> My pyspark script runs correctly when I test on small number of files how
>>> ever I get an OOM when I run on all 16000.
>>>
>>&g
pt runs correctly when I test on small number of files how
>> ever I get an OOM when I run on all 16000.
>>
>>
>>
>> To try and debug I ran a small test and set warning level to INFO. I
>> found the following
>>
>>
>>
>>
all test and set warning level to INFO. I found
> the following
>
>
>
> 2021-12-21 00:47:04 INFO CreateViewCommand:57 - Try to uncache
> `rawCounts` before replacing.
>
>
>
> for i in range( 1, len( self.sampleNamesList ) ):
>
> sampleName = s
on small number of files how ever I get an OOM when
I run on all 16000.
To try and debug I ran a small test and set warning level to INFO. I found the
following
2021-12-21 00:47:04 INFO CreateViewCommand:57 - Try to uncache `rawCounts`
before replacing.
for i in range( 1, len
