I sometimes have to work with vectors/matrices with > 2^31 - 1
elements. I have found the bigmemory package to be of great help. My
lab is also going to learn sqldf package for getting bits of big data
into/out of R. Learning both of those packages should help you work
with large datasets in R.
Th
On 03/08/2010 2:28 PM, Dimitri Liakhovitski wrote:
And once one above the limit that Jim indicated - is there anything one can do?
Yes, there are several packages for handling datasets that are too big
to fit in memory: biglm, ff, etc. You need to change your code to work
with them, so i
And once one above the limit that Jim indicated - is there anything one can do?
Thank you!
Dimitri
On Tue, Aug 3, 2010 at 2:12 PM, Dimitri Liakhovitski
wrote:
> Thanks a lot, it's very helpful!
> Dimitri
>
> On Tue, Aug 3, 2010 at 1:53 PM, Duncan Murdoch
> wrote:
>> On 03/08/2010 1:10 PM, Dimi
Thanks a lot, it's very helpful!
Dimitri
On Tue, Aug 3, 2010 at 1:53 PM, Duncan Murdoch wrote:
> On 03/08/2010 1:10 PM, Dimitri Liakhovitski wrote:
>>
>> I understand the question I am about to ask is rather vague and
>> depends on the task and my PC memory. However, I'll give it a try:
>>
>> Let
On 03/08/2010 1:10 PM, Dimitri Liakhovitski wrote:
I understand the question I am about to ask is rather vague and
depends on the task and my PC memory. However, I'll give it a try:
Let's assume the goal is just to read in the data frame into R and
then do some simple analyses with it (e.g., mul
You probably don't want an object that is larger than about 25% of the
physical memory so that copies can be made during some processing. If
you are running on a 32-bit system which will limit you to at most 3GB
of memory, then your largest object should not be greater than 800MB.
If you want to h
I understand the question I am about to ask is rather vague and
depends on the task and my PC memory. However, I'll give it a try:
Let's assume the goal is just to read in the data frame into R and
then do some simple analyses with it (e.g., multiple regression of
some variables onto some - just a
7 matches
Mail list logo
