Sebastian,

it's a XEON. Your question comes at the right time, we are in the process of compiling performance data this summer. I will soon post detail info here.

Loading 100k x 150k pixels is no problem. You load piecemeal, and the system will align with its internal tiling + maintain the image pyramid.

10 years back we have loaded the whole ortho image mosaic of Bavaria into the database (server was a desktop PC of about 1000 EUR), that was 600 GB compressed (losslessly). At that time, import of an 230 MB image took about 2min (array placement, re-tiling of input, index maintenance, compression, and this for all pyramid levels). Read access, OTOH, was about 150ms. A few years later the Thuringia Mapping Agency, again with a desktop PC as a server, verified that access times are stable under a load of about 15 concurrent users challenging the system with WMS type interactive browsing.

For another mapping agency we have set up a database containing ortho image, 30+ thematic raster layers, DEM. Access was via WMS, the rasdaman system delivered a single image for display (Javascript was not what it is today...). This included on-the-fly classification of the DEM and styling the thematic layers.

So we have already done complete raster servers for several German States.

BTW, standards: We have established the OGC Web Coverage Processing Service (WCPS) standard which lifts the rasdaman concept of a raster querfy language to an OGC standard. Being editor of currently 9 geo raster service specs in OGC we also pursue reference implementations on the basis of rasdaman. See www.earthlook.org for demos.

-Peter

PS: Should data really exceed disk capacity (hard to believe today) then we have a PhD thesis work which has implemented transparent tape cabinetaccess.



On 07/21/2010 09:37 AM, Sebastian E. Ovide wrote:


On Tue, Jul 20, 2010 at 5:08 PM, Peter Baumann <p.baum...@jacobs-university.de <mailto:p.baum...@jacobs-university.de>> wrote:

    On 07/20/2010 05:47 PM, Pierre Racine wrote:

    Since we have this bug with big rasters,


    proven, any-size, ... ;-)


what do you mean ? were you able to load a raster of 100kx150k ?


    Who is proven? Well, running a dozen-TB seamless mosaic on
    PostgreSQL for many years, having online-demos available since
    years, etc... you decide, in comparison.


Peter, how big is your cluster to support dozen of TB ? and how fast are the queries ? (for example a simple lookup)

    -Peter



    I would suggest you split your raster with something like GDAL
    gdal_retile.py and then import your tiles into a single WKT
    Raster table following the instruction provided in our tutorial.
    Each tile will be stored in a column cell of type RASTER similar
    to the PostGIS GEOMETRY type.

    As I said previously you can then query the raster in SQL like this:

    SELECT ST_Value(rast, ST_Geomfromtext('Point(-78.1 58.1)', 4326))

    FROM srtm_tiled_100x100

    WHERE ST_Intersects(rast::geometry, ST_Geomfromtext('Point(-78.1
    58.1)', 4326)) AND whatever you want.

    Basically WKT Raster is the first true SQL interface with which
    is is simple to do such things… It's proven, any-size,
    cloud-scalable, and open source J Maybe not that prooven. But who is?

    There is also rasdaman but I don’t think you can use SQL. It
    would be nice if you could compare both… I can’t find time for
    this. Jorge has started comparing PostGIS WKT Raster with Oracle
    Georaster though.

    Pierre

    *From:* postgis-users-boun...@postgis.refractions.net
    <mailto:postgis-users-boun...@postgis.refractions.net>
    [mailto:postgis-users-boun...@postgis.refractions.net] *On Behalf
    Of *Sebastian E. Ovide
    *Sent:* 20 juillet 2010 10:46
    *To:* PostGIS Users Discussion
    *Subject:* Re: [postgis-users] WKTRaster : gdal2wktraster.py
    cannot read AIG/Arc/Info Binary Grid

    yes... with png worked... but it was a different png (a smaller
    one)...

    so I've converted the ESRI into a png and tried to import it...
    and it didn't work neither...

    so this is the situation:
    A have huge rusters (from 150kx150k).. In Oracle I would just
    load it (the huge raster) in a single row of a GeoRaster table
    and then Oracle GeoRaster would split it in small tiles and store
    one tile per line of another table (Raster Data Table)... then I
    can run a query similar to this one: SELECT
    getcellvalue(t.rastercolumn,x,y) from GeoRasterTable t where t.id
    <http://t.id>=1; and GeoRaster will query automatically the
    spatial indexes and the Raster Data Table and it will find the
    right tile etc...

    I do not know how WKTRaster works.... If I cannot import a such
    big image, of course I can split it in smaller georeferenced
    tiles... (how?)... but.... My main question is: after that, how
    will the table look like ? how can I do the same query (where
    id=1 or where name="UK" etc...) ?

    On Tue, Jul 20, 2010 at 2:31 PM, Pierre Racine
    <pierre.rac...@sbf.ulaval.ca
    <mailto:pierre.rac...@sbf.ulaval.ca>> wrote:

    Wait. You first said the png was working. Now it’s not? Did you
    try gdal_translate with the ESRI grid? For sure I haven’t test
    yet with such big rasters. Is this the result of a merge or all
    your original raster are all this size? The point is that with
    WKT Raster you don’t have to merge your raster first into a
    gigantic raster in order to get it store in a unique table like
    with Oracle Spatial.

    Pierre

    *From:* postgis-users-boun...@postgis.refractions.net
    <mailto:postgis-users-boun...@postgis.refractions.net>
    [mailto:postgis-users-boun...@postgis.refractions.net
    <mailto:postgis-users-boun...@postgis.refractions.net>] *On
    Behalf Of *Sebastian E. Ovide
    *Sent:* 20 juillet 2010 05:51
    *To:* PostGIS Users Discussion
    *Subject:* Re: [postgis-users] WKTRaster : gdal2wktraster.py
    cannot read AIG/Arc/Info Binary Grid

    Hi Pierre,

    Does gdal2wktraster.py have any limitation on the maximum number
    of columnsxrows ?

    in my case, my raster is  107759 x 168633...

    gdal works well:

    se...@seanspc:~/rasters$ gdal_translate -of PNG raster/ test.png
    Input file size is 107759, 168633
    0...10...20...30...40...50...60...70...80...90...100 - done.


    se...@seanspc:~/rasters$ python gdal2wktraster.py -r raster/ -t
    sebastable -o ok.sql
    gdal2wktraster.py:695: DeprecationWarning: 'H' format requires 0
    <= number <= 65535
      hexwkb += wkblify('H', xsize)
    gdal2wktraster.py:696: DeprecationWarning: 'H' format requires 0
    <= number <= 65535
      hexwkb += wkblify('H', ysize)
    gdal2wktraster.py:727: DeprecationWarning: integer argument
    expected, got float
      hexwkb += wkblify(pt2fmt(pixtype), nodata)
    Traceback (most recent call last):
      File "gdal2wktraster.py", line 1013, in <module>
        main()
      File "gdal2wktraster.py", line 976, in main
        wkblify_raster(opts, filename, i)
      File "gdal2wktraster.py", line 921, in wkblify_raster
        summary = wkblify_raster_level(options, ds,
    options.overview_level, band_range, infile, i)
      File "gdal2wktraster.py", line 888, in wkblify_raster_level
        hexwkb += wkblify_band(options, band, level, xoff, yoff,
    read_block_size, block_size, infile, b)
      File "gdal2wktraster.py", line 777, in wkblify_band
        target_block_size[0], target_block_size[1])
      File "/usr/lib/python2.6/dist-packages/osgeo/gdal.py", line
    895, in ReadAsArray
        buf_xsize, buf_ysize, buf_obj )
      File "/usr/lib/python2.6/dist-packages/osgeo/gdal_array.py",
    line 228, in BandReadAsArray
        ar = numpy.empty([buf_ysize,buf_xsize], dtype = typecode)
    MemoryError


    se...@seanspc:~/rasters$ python gdal2wktraster.py -r test.png  -t
    sebastable -o ok.sql
    gdal2wktraster.py:695: DeprecationWarning: 'H' format requires 0
    <= number <= 65535
      hexwkb += wkblify('H', xsize)
    gdal2wktraster.py:696: DeprecationWarning: 'H' format requires 0
    <= number <= 65535
      hexwkb += wkblify('H', ysize)
    gdal2wktraster.py:727: DeprecationWarning: integer argument
    expected, got float
      hexwkb += wkblify(pt2fmt(pixtype), nodata)
    Traceback (most recent call last):
      File "gdal2wktraster.py", line 1013, in <module>
        main()
      File "gdal2wktraster.py", line 976, in main
        wkblify_raster(opts, filename, i)
      File "gdal2wktraster.py", line 921, in wkblify_raster
        summary = wkblify_raster_level(options, ds,
    options.overview_level, band_range, infile, i)
      File "gdal2wktraster.py", line 888, in wkblify_raster_level
        hexwkb += wkblify_band(options, band, level, xoff, yoff,
    read_block_size, block_size, infile, b)
      File "gdal2wktraster.py", line 777, in wkblify_band
        target_block_size[0], target_block_size[1])
      File "/usr/lib/python2.6/dist-packages/osgeo/gdal.py", line
    895, in ReadAsArray
        buf_xsize, buf_ysize, buf_obj )
      File "/usr/lib/python2.6/dist-packages/osgeo/gdal_array.py",
    line 228, in BandReadAsArray
        ar = numpy.empty([buf_ysize,buf_xsize], dtype = typecode)
    MemoryError



    On Mon, Jul 19, 2010 at 5:56 PM, Pierre Racine
    <pierre.rac...@sbf.ulaval.ca
    <mailto:pierre.rac...@sbf.ulaval.ca>> wrote:

    Hi Sebastian,

    I can convert ESRI Grid file to .sql without problem using
    gdal2wktraster.py and the same parameters as you. I can do both
    integer and floating point rasters.

    Maybe this is a GDAL problem. Try to convert it using
    gdal_translate (to tiff for example). This would be a better test
    than just gdalinfo.

    Could you provide us with a file sample?

    Pierre

    *From:* postgis-users-boun...@postgis.refractions.net
    <mailto:postgis-users-boun...@postgis.refractions.net>
    [mailto:postgis-users-boun...@postgis.refractions.net
    <mailto:postgis-users-boun...@postgis.refractions.net>] *On
    Behalf Of *Sebastian E. Ovide
    *Sent:* 19 juillet 2010 12:28
    *To:* PostGIS Users Discussion
    *Subject:* [postgis-users] WKTRaster : gdal2wktraster.py cannot
    read AIG/Arc/Info Binary Grid

    Hi All,

    trying to create a SQL with gdal2wktraster.py. It works on PNG
    but it doesn't on AIG files...

    Note: As Gdal works fine.

    C:\Program Files\PostgreSQL\8.4\bin>gdalinfo c:\tmp\raster
    Driver: AIG/Arc/Info Binary Grid
    Files: c:\tmp\raster
           c:\tmp\raster\dblbnd.adf
           c:\tmp\raster\hdr.adf
           c:\tmp\raster\metadata.xml
           c:\tmp\raster\prj.adf
           c:\tmp\raster\sta.adf
           c:\tmp\raster\vat.adf
           c:\tmp\raster\w001000.adf
           c:\tmp\raster\w001000x.adf
           c:\tmp\raster\w001001.adf
           c:\tmp\raster\w001001x.adf
           c:\tmp\raster\z001001.adf
           c:\tmp\raster\z001001x.adf
           c:\tmp\raster\z001002.adf
           c:\tmp\raster\z001002x.adf
           c:\tmp\raster\z001003.adf
           c:\tmp\raster\z001003x.adf
           c:\tmp\raster\z001004.adf
           c:\tmp\raster\z001004x.adf
           c:\tmp\raster\z001005.adf
           c:\tmp\raster\z001005x.adf
           c:\tmp\raster\z001006.adf
           c:\tmp\raster\z001006x.adf
           c:\tmp\raster\z001007.adf
           c:\tmp\raster\z001007x.adf
           c:\tmp\raster\z001008.adf
           c:\tmp\raster\z001008x.adf
           c:\tmp\raster\z001009.adf
           c:\tmp\raster\z001009x.adf
           c:\tmp\raster\z001010.adf
           c:\tmp\raster\z001010x.adf
           c:\tmp\raster\z001011.adf
           c:\tmp\raster\z001011x.adf
           c:\tmp\raster\z001012.adf
           c:\tmp\raster\z001012x.adf
           c:\tmp\raster\z001013.adf
           c:\tmp\raster\z001013x.adf
           c:\tmp\raster\z001014.adf
           c:\tmp\raster\z001014x.adf
           c:\tmp\raster\z001015.adf
           c:\tmp\raster\z001015x.adf
    Size is 107759, 168633
    Coordinate System is:
    PROJCS["unnamed",
        GEOGCS["Unknown datum based upon the Airy 1830 ellipsoid",
            DATUM["Not_specified_based_on_Airy_1830_ellipsoid",
                SPHEROID["Airy 1830",6377563.396,299.3249646,
                    AUTHORITY["EPSG","7001"]],
                AUTHORITY["EPSG","6001"]],
            PRIMEM["Greenwich",0,
                AUTHORITY["EPSG","8901"]],
            UNIT["degree",0.01745329251994328,
                AUTHORITY["EPSG","9122"]],
            AUTHORITY["EPSG","4001"]],
        PROJECTION["Transverse_Mercator"],
        PARAMETER["latitude_of_origin",49],
        PARAMETER["central_meridian",-2],
        PARAMETER["scale_factor",0.9996012717],
        PARAMETER["false_easting",400000],
        PARAMETER["false_northing",-100000],
        UNIT["METERS",1]]
    Origin = (128110.000000000000000,813270.000000000000000)
    Pixel Size = (5.000000000000000,-5.000000000000000)
    Corner Coordinates:
    Upper Left  (  128110.000,  813270.000) (  6d29'37.32"W, 57d
    7'47.53"N)
    Lower Left  (  128110.000,  -29895.000) (  5d45'40.00"W,
    49d34'10.24"N)
    Upper Right (  666905.000,  813270.000) (  2d24'41.72"E, 57d
    7'58.04"N)
    Lower Right (  666905.000,  -29895.000) (  1d41'32.29"E,
    49d34'18.23"N)
    Center      (  397507.500,  391687.500) (  2d 2'15.04"W,
    53d25'18.19"N)
    Band 1 Block=256x4 Type=Byte, ColorInterp=Undefined
      Min=1.000 Max=4.000
      NoData Value=255


    C:\Program Files\PostgreSQL\8.4\bin>python gdal2wktraster.py -r
    c:\tmp\raster -t sebastable -o c:\tmp\sebas.sql
    gdal2wktraster.py:644: DeprecationWarning: 'H' format requires 0
    <= number <= 65535
      hexstr = binascii.hexlify(struct.pack(fmt_little, data)).upper()
    gdal2wktraster.py:644: DeprecationWarning: integer argument
    expected, got float
      hexstr = binascii.hexlify(struct.pack(fmt_little, data)).upper()
    ERROR 2: Multiplication overflow : 107759 * 168633 * 1
    Traceback (most recent call last):
      File "gdal2wktraster.py", line 1013, in <module>
        main()
      File "gdal2wktraster.py", line 976, in main
        wkblify_raster(opts, filename, i)
      File "gdal2wktraster.py", line 921, in wkblify_raster
        summary = wkblify_raster_level(options, ds,
    options.overview_level, band_range, infile, i)
      File "gdal2wktraster.py", line 888, in wkblify_raster_level
        hexwkb += wkblify_band(options, band, level, xoff, yoff,
    read_block_size, block_size, infile, b)
      File "gdal2wktraster.py", line 777, in wkblify_band
        target_block_size[0], target_block_size[1])
      File "C:\OSGeo4W\apps\gdal-16\pymod\osgeo\gdal.py", line 835,
    in ReadAsArray
        buf_xsize, buf_ysize, buf_obj )
      File "C:\OSGeo4W\apps\gdal-16\pymod\osgeo\gdal_array.py", line
    140, in BandReadAsArray
        ar = numpy.reshape(ar, [buf_ysize,buf_xsize])
      File
    "C:\OSGeo4W\apps\Python25\lib\site-packages\numpy\core\fromnumeric.py",
    line 116, in reshape
        return reshape(newshape, order=order)
    ValueError: total size of new array must be unchanged

    Any ideas ?
-- Sebastian E. Ovide


    _______________________________________________
    postgis-users mailing list
    postgis-users@postgis.refractions.net
    <mailto:postgis-users@postgis.refractions.net>
    http://postgis.refractions.net/mailman/listinfo/postgis-users




-- Sebastian E. Ovide

    skype: sebastian.ovide

    +353 (0) 87 6340149


    _______________________________________________
    postgis-users mailing list
    postgis-users@postgis.refractions.net
    <mailto:postgis-users@postgis.refractions.net>
    http://postgis.refractions.net/mailman/listinfo/postgis-users




-- Sebastian E. Ovide




-- Dr. Peter Baumann
      - Professor of Computer Science, Jacobs University Bremen
        www.faculty.jacobs-university.de/pbaumann  
<http://www.faculty.jacobs-university.de/pbaumann>
        mail:p.baum...@jacobs-university.de  
<mailto:p.baum...@jacobs-university.de>
        tel: +49-421-200-3178, fax: +49-421-200-493178
      - Executive Director, rasdaman GmbH Bremen (HRB 147737)
        www.rasdaman.com  <http://www.rasdaman.com>, mail:baum...@rasdaman.com  
<mailto:baum...@rasdaman.com>
        tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
    "Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis 
dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat 
quisquam non sibi parata." (mail disclaimer, AD 10xx)



    _______________________________________________
    postgis-users mailing list
    postgis-users@postgis.refractions.net
    <mailto:postgis-users@postgis.refractions.net>
    http://postgis.refractions.net/mailman/listinfo/postgis-users




--
Sebastian E. Ovide




--
Dr. Peter Baumann
 - Professor of Computer Science, Jacobs University Bremen
   www.faculty.jacobs-university.de/pbaumann
   mail: p.baum...@jacobs-university.de
   tel: +49-421-200-3178, fax: +49-421-200-493178
 - Executive Director, rasdaman GmbH Bremen (HRB 147737)
   www.rasdaman.com, mail: baum...@rasdaman.com
   tel: 0800-rasdaman, fax: 0800-rasdafax, mobile: +49-173-5837882
"Si forte in alienas manus oberraverit hec peregrina epistola incertis ventis 
dimissa, sed Deo commendata, precamur ut ei reddatur cui soli destinata, nec preripiat 
quisquam non sibi parata." (mail disclaimer, AD 10xx)



_______________________________________________
postgis-users mailing list
postgis-users@postgis.refractions.net
http://postgis.refractions.net/mailman/listinfo/postgis-users

Reply via email to