Hi Matt,
On Thu, Jun 24, 2010 at 4:49 PM, Matthew Kenworthy <[email protected]> wrote: > Puneet, > > Ultimately, will your 2D piddle be completely populated, or will it be > sparse? Ultimately the 2D piddle will be completely populated, with 100% coverage. It will be a perfectly rectangular piddle with no holes. > My first thought is to stack up your 1D piddles separately and > keep an index piddle that holds your 'serial numbers', then do a > 'qsorti()' on that index piddle. That way you don't need to > preallocate a massive 2D piddle, as you really have all the > information you need already there in your serial number. > So, in a manner of speaking, that is how I have the information already. The problem with keeping the patches separately and unstitched is that selection of elements for a boundary that overlaps several piddles becomes tedious. By stitching (gluing) the patches into one fabric, I can select from anywhere in that patch in one shot. It also makes management easier (fewer data files). That is why I want to move away from lots of small files to one large file. In my tests, the large 2D piddle (of 1D piddles) using fake but correctly sized data results in a file about 150 MB large. It is very fast to write and to read, and provides me 100% coverage of my study area. > Matt > > On Thu, Jun 24, 2010 at 1:59 PM, P Kishor <[email protected]> wrote: >> Imagine, I want to end up with a large, 2D piddle $p, whose every >> element is a 1D piddle $q, so, really, a 3D piddle. But, it is easier >> to think of it as a 2D piddle, a rectangular grid of piddles. >> >> This rectangular piddle is (i x j), say, (2000 x 1500). Each element >> in this (i x j) piddle has a serial number, starting at coordinates >> (0,0) at the top left, which is 1, and increasing to the right most >> edge, then down one row and left, then to the right most edge, and so >> on. So, in my example 2D (i x j) piddle, the bottom-most, right-most >> element's coordinates are (1999 x 1499) and its serial number is 3e6. >> >> I can get the content of any element in the 2D piddle with $p->at(x, >> y) where (x, y) is the coordinate pair. Also, thanks to David Mertens, >> if I know the serial number of an element, I can find its content with >> $p->flat->at(n), where n is its serial number between 1 and 3e6. >> >> Ok. Here's the rub. I don't have all the data. I get the data >> incrementally. That is, my 2k x 1.5k 2D piddle is really made up like >> a patchwork quilt, and I get the patches one at a time. Every patch is >> a series of 1D piddles ($q from my para 1 above) with a unique serial >> number between 1 and 3e6, so I know, for any set of 1D piddles, which >> patch they will go to. By the way, is there a PDL method to find the >> indexes (coordinate pair) of an element in my 2D piddle, given its >> serial number? I could write one in Perl, but PDL might have one >> already. >> >> So, I want to glue(), which, btw, is really clever method, my patches >> to each other, one by one, till they end up as the 2000x1500 2D >> piddle. >> >> How do I do the above? >> >> >> -- >> Puneet Kishor http://www.punkish.org >> Carbon Model http://carbonmodel.org >> Charter Member, Open Source Geospatial Foundation http://www.osgeo.org >> Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor >> Nelson Institute, UW-Madison http://www.nelson.wisc.edu >> ----------------------------------------------------------------------- >> Assertions are politics; backing up assertions with evidence is science >> ======================================================================= >> >> _______________________________________________ >> Perldl mailing list >> [email protected] >> http://mailman.jach.hawaii.edu/mailman/listinfo/perldl >> >> > > > > -- > Matthew Kenworthy / Assistant Professor / Leiden Observatory > Niels Bohrweg 2 (#463) / P.O. Box 9513 / 2300 RA Leiden / NL > -- Puneet Kishor http://www.punkish.org Carbon Model http://carbonmodel.org Charter Member, Open Source Geospatial Foundation http://www.osgeo.org Science Commons Fellow, http://sciencecommons.org/about/whoweare/kishor Nelson Institute, UW-Madison http://www.nelson.wisc.edu ----------------------------------------------------------------------- Assertions are politics; backing up assertions with evidence is science ======================================================================= _______________________________________________ Perldl mailing list [email protected] http://mailman.jach.hawaii.edu/mailman/listinfo/perldl
