Re: [R] Coarsening the Resolution of a Dataset

2008-08-01 Thread Steve Murray
Hi Jim, Thanks for your advice. The problem is that I can't lose any of the data - it's a global dataset, where the left-most column = 180 degrees west, and the right-most is 180 degrees east. The top row is the North Pole and the bottom row is the South Pole. I've got 512MB RAM on the

Re: [R] Coarsening the Resolution of a Dataset

2008-08-01 Thread jim holtman
If you can reduce the size of your data by averaging, then you could read in a subset of the rows, average the 6x6 matrices and then write them out for a second phase of processing. The 2160x4230 object would take up 75MB if numeric, which is probably 50% of your available memory if you are

Re: [R] Coarsening the Resolution of a Dataset

2008-08-01 Thread Steve Murray
Ok thanks Jim - I'll give it a go! I'm new to R, so I'm not sure how I'd go about performing averages in subsets... I'll have a look into it, but any subsequent pointers would be gratefully received as ever! I'll also try playing with it in Access, and maybe even Excel 2007 might be able to

Re: [R] Coarsening the Resolution of a Dataset

2008-07-31 Thread Steve Murray
Please find below my command inputs, subsequent outputs and errors that I've been receiving. crops - read.table(crop2000AD.asc, colClasses = numeric, na=-) str(crops[1:10]) 'data.frame': 2160 obs. of 10 variables: $ V1 : num NA NA NA NA NA NA NA NA NA NA ... $ V2 : num NA NA NA NA

Re: [R] Coarsening the Resolution of a Dataset

2008-07-31 Thread jim holtman
In the first case, 'crops' is an object that is 142MB in size, and that alone will take up almost 50% of your memory. You are also probably generating another object of like size in doing the operation. Can you cut down the size of the objects that you need. What operating system do you have and

Re: [R] Coarsening the Resolution of a Dataset

2008-07-30 Thread Steve Murray
: Tue, 29 Jul 2008 15:34:31 -0400 From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Re: [R] Coarsening the Resolution of a Dataset CC: r-help@r-project.org I assume that you are doing this on one column of the matrix which should only have 2160 entries in it. can you send the actual code

Re: [R] Coarsening the Resolution of a Dataset

2008-07-30 Thread Steve Murray
: Tue, 29 Jul 2008 15:34:31 -0400 From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Re: [R] Coarsening the Resolution of a Dataset CC: r-help@r-project.org I assume that you are doing this on one column of the matrix which should only have 2160 entries in it. can you send the actual code

Re: [R] Coarsening the Resolution of a Dataset

2008-07-30 Thread jim holtman
to read in the data (using read.table) and then the code that you supplied. I could send you the actual dataset if you don't mind a file ~50MB?! Thanks again, Steve Date: Tue, 29 Jul 2008 15:34:31 -0400 From: [EMAIL PROTECTED] To: [EMAIL PROTECTED] Subject: Re: [R] Coarsening

Re: [R] Coarsening the Resolution of a Dataset

2008-07-29 Thread Steve Murray
Unfortunately, when I get to the 'myCuts' line, I receive the following error: Error: evaluation nested too deeply: infinite recursion / options(expressions=)? ...and I also receive warnings about memory allocation being reached (even though I've already used memory.limit() to maximise the

Re: [R] Coarsening the Resolution of a Dataset

2008-07-29 Thread jim holtman
I assume that you are doing this on one column of the matrix which should only have 2160 entries in it. can you send the actual code you are using. I tried it with 10,000 samples and it works fine. So I need to understand the data structure you are using. Also the infinite recursion sounds

[R] Coarsening the Resolution of a Dataset

2008-07-26 Thread Steve Murray
Dear all, I have gridded data at 5' (minutes) resolution, which I intend to coarsen to 0.5 degrees. How would I go about doing this in R? I've had a search online and haven't found anything obvious, so any help would be gratefully received. I'm assuming that there will be several 'coarsening'

Re: [R] Coarsening the Resolution of a Dataset

2008-07-26 Thread jim holtman
Here is one approach for using the average of each interval: # generate some test data; assume 1.0 is one degree min.5 - 5 / 60 # portion of degree for 5 minutes n - 100 # number of data points myData - cbind(interval=seq(0, by=min.5, length=n), value=runif(n)) # create breaks every 0.5