Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Rahkonen Jukka]

Hi,

If you reproject images one by one to any common coordinate system you will get 
black collars to most of them because images will be rotated. In order to make 
them transparent by using nodata you must keep images uncompressed. However, 
you may find it teasing to use the jpeg-in-tiff compression because that way 
you can save 90 of disk space and your users will not notice any difference in 
the quality and the speed can still be pretty good. For reprojecting it would 
be best to reproject into a regular South-North and West-East oriented grid in 
the target SRID. Then you could safely compress the reprojected tiles. That is 
a bit more complicated but still possible to do with scripts. If you want to 
keep your coverage up-to-date and you should slip in new images from here and 
there it will make a bit more trouble because usually the target tiles are 
mosaiced from many originals. This is what I am doing right now with a couple 
of terabytes of Finnish aerial photos.

With your data I would do what Frank W. already suggested. Convert your jpeg 
files into tiled tiffs with jpeg compression and use creation option 
PHOTOMETRIC=YCBCR. Create also overviews which are compressed in the same way. 
Make one layer for each UTM zone by combining them with tileindex and group all 
the zones together with a layer group. You can also start from the original 
images by creating tileindexes from them and convert them into more speedy 
tiled jpeg-in-tiff images once you have time for that. 

-Jukka Rahkonen-



 Evans, James wrote:

> For some reason I didn't see the reply from Robert Sanson.  Anyway, we have 
> no requirement to stay with NAD83.  I have global mapper, and I think it will 
> do a bulk reprojection, from my local hard drive to a network drive.  Maybe I 
> will take a look at that tomorrow.  Is there a better tool for that?  Some 
> GDAL utility?  Anyway, thanks for all the great suggestions.
James


-Original Message-
From: mapserver-users-boun...@lists.osgeo.org on behalf of Stephen Woodbridge
Sent: Mon 6/10/2013 6:36 PM
To: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?

On 6/10/2013 7:07 PM, Robert Sanson wrote:
> Sounds all very complicated. I would advise that you choose a single
> projection that will work across your entire are and then re-project
> your imagery first, and then build your image datastore around that.

I would agree with this that if htere are not other contraints on the
problem this is often the best way to go. But that will not owrk if his
users HAVE to have the data in UTM projection for some reason.

-Steve

 "Evans, James R Civ USAF ACC 84 RADES/SCZE"
  11/06/2013 10:48 a.m. >>>
> So, I'm guessing there's no easy way to automate this?  Even looking
> at the states, some of the states are in two zones, and Texas is
> across 3 zones. At least the naming convention of the files indicate
> the UTM zone.  For instance:
> m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far as I
> can tell, all the files in a particular directory are all in the
> same UTM zone.  I could create a layer for each UTM zone across
> CONUS, but that's not going to be particularly useful to my users.
> I'm thinking of making a layer for each state.  For the stats that
> cross zones, there will probably be two layers.  For Texas, there
> would be Texas_east, Texas_middle, and Texas_west.  I will probably
> limit visibility until zoomed in sufficiently to see the whole state
> on the screen anyway, since the continental view of this data is
> pretty crappy anyway.  So now it seems like it will be a lot of grunt
> work just copying these directories up to the server, and going
> through and creating a shape file index for each state.  For states
> in more than one UTM, there would be more than one shape.  Then I'll
> have to add a layer for to my mapfile for each shapefile, using the
> correct projection. Is there an easier way?  I'm starting with
> Oklahoma, which is also in three UTM zones.  I'll get that working
> before moving on.  Any suggestions on making this pretty would be
> welcomed.  :-)
>
>
>
>
> -Original Message- From: Stephen Woodbridge
> [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34
> PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc:
> mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best
> way to import 4.5TB of imagery?
>
> On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE
> wrote:
>> Hi Stephen, Thanks, for the reply.  I previously got 4 sample
>> images from the USDA, and was able to get them to work just fine.
>> There was no processing
> required.
>> The sample images I got were all from Utah, and they are NAD83, UTM
>> zone
> 12.
>> I added the 4 sample images to a shape file using gdaltindex.   I
>> used
> UPSG
>> 26912 and mapserver served them up very quickly for such large
>> files.
>>
>> Now I 

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Stephen Woodbridge]

I have used gdalwarp to do this in the past and find or a perl script. I 
have not used global mapper for anything so I can't compare.


-Steve

On 6/10/2013 10:00 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:


For some reason I didn't see the reply from Robert Sanson.  Anyway,
we have no requirement to stay with NAD83.  I have global mapper, and
I think it will do a bulk reprojection, from my local hard drive to a
network drive.  Maybe I will take a look at that tomorrow.  Is there
a better tool for that?  Some GDAL utility?  Anyway, thanks for all
the great suggestions. James


-Original Message- From:
mapserver-users-boun...@lists.osgeo.org on behalf of Stephen
Woodbridge Sent: Mon 6/10/2013 6:36 PM To:
mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best
way to import 4.5TB of imagery?

On 6/10/2013 7:07 PM, Robert Sanson wrote:

Sounds all very complicated. I would advise that you choose a
single projection that will work across your entire are and then
re-project your imagery first, and then build your image datastore
around that.


I would agree with this that if htere are not other contraints on
the problem this is often the best way to go. But that will not owrk
if his users HAVE to have the data in UTM projection for some
reason.

-Steve


"Evans, James R Civ USAF ACC 84 RADES/SCZE"
 11/06/2013 10:48 a.m. >>>

So, I'm guessing there's no easy way to automate this?  Even
looking at the states, some of the states are in two zones, and
Texas is across 3 zones. At least the naming convention of the
files indicate the UTM zone.  For instance:
m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far as
I can tell, all the files in a particular directory are all in the
same UTM zone.  I could create a layer for each UTM zone across
CONUS, but that's not going to be particularly useful to my users.
I'm thinking of making a layer for each state.  For the stats that
cross zones, there will probably be two layers.  For Texas, there
would be Texas_east, Texas_middle, and Texas_west.  I will
probably limit visibility until zoomed in sufficiently to see the
whole state on the screen anyway, since the continental view of
this data is pretty crappy anyway.  So now it seems like it will be
a lot of grunt work just copying these directories up to the
server, and going through and creating a shape file index for each
state.  For states in more than one UTM, there would be more than
one shape.  Then I'll have to add a layer for to my mapfile for
each shapefile, using the correct projection. Is there an easier
way?  I'm starting with Oklahoma, which is also in three UTM zones.
I'll get that working before moving on.  Any suggestions on making
this pretty would be welcomed.  :-)




-Original Message- From: Stephen Woodbridge
[mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34
PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc:
mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users]
Best way to import 4.5TB of imagery?

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE
wrote:

Hi Stephen, Thanks, for the reply.  I previously got 4 sample
images from the USDA, and was able to get them to work just
fine. There was no processing

required.

The sample images I got were all from Utah, and they are NAD83,
UTM zone

12.

I added the 4 sample images to a shape file using gdaltindex.
I used

UPSG

26912 and mapserver served them up very quickly for such large
files.

Now I have this entire data set, and it stretches from UTM zone
10, to UTM zone 19.  The data is divided into directories by two
letter state abbreviations, and under that into subdirectories.
I'm just wondering how to add this to my mapfile.  Do I need a
different entry for each UTM

Zone?

How is it possible to get a single layer entry that includes
multiple projections?  This is looking like a huge job and I
just want to know the best strategy for getting this done.


So now you have a problem. You data is in UTM spread over 10
different projections. What do you plane to do when have your
image is zone 10 and half is in zone 11 or if you zoom out and you
images has 3-4 zones displayed?

All data in an image must be rendered in the same projection. While
I don't doubt that your test with 4 images worked fine, did you
you test this a multiple zoom levels and at some point you will
probably want to create a super overlay image so you do not have to
open multiple files to just pull a tiny overlay out of each one.

Your use cases will determine how you want to deal with the data.
For example does it HAVE to be in a UTM projection, or can you
work with a Spherical Mercator or geographic projection? The end
solution will be much easier if you can work with one common
projection over your whole data set. Otherwise, you will have to
deal with the transitions from one zone to the next or maybe set up
10 separate servers that only serve one zone.

Having pushed larges amounts (4-25TB) of

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Evans, James R Civ USAF ACC 84 RADES/SCZE]

For some reason I didn't see the reply from Robert Sanson.  Anyway, we have no 
requirement to stay with NAD83.  I have global mapper, and I think it will do a 
bulk reprojection, from my local hard drive to a network drive.  Maybe I will 
take a look at that tomorrow.  Is there a better tool for that?  Some GDAL 
utility?  Anyway, thanks for all the great suggestions.  
James


-Original Message-
From: mapserver-users-boun...@lists.osgeo.org on behalf of Stephen Woodbridge
Sent: Mon 6/10/2013 6:36 PM
To: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?
 
On 6/10/2013 7:07 PM, Robert Sanson wrote:
> Sounds all very complicated. I would advise that you choose a single
> projection that will work across your entire are and then re-project
> your imagery first, and then build your image datastore around that.

I would agree with this that if htere are not other contraints on the 
problem this is often the best way to go. But that will not owrk if his 
users HAVE to have the data in UTM projection for some reason.

-Steve

 "Evans, James R Civ USAF ACC 84 RADES/SCZE"
  11/06/2013 10:48 a.m. >>>
> So, I'm guessing there's no easy way to automate this?  Even looking
> at the states, some of the states are in two zones, and Texas is
> across 3 zones. At least the naming convention of the files indicate
> the UTM zone.  For instance:
> m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far as I
> can tell, all the files in a particular directory are all in the
> same UTM zone.  I could create a layer for each UTM zone across
> CONUS, but that's not going to be particularly useful to my users.
> I'm thinking of making a layer for each state.  For the stats that
> cross zones, there will probably be two layers.  For Texas, there
> would be Texas_east, Texas_middle, and Texas_west.  I will probably
> limit visibility until zoomed in sufficiently to see the whole state
> on the screen anyway, since the continental view of this data is
> pretty crappy anyway.  So now it seems like it will be a lot of grunt
> work just copying these directories up to the server, and going
> through and creating a shape file index for each state.  For states
> in more than one UTM, there would be more than one shape.  Then I'll
> have to add a layer for to my mapfile for each shapefile, using the
> correct projection. Is there an easier way?  I'm starting with
> Oklahoma, which is also in three UTM zones.  I'll get that working
> before moving on.  Any suggestions on making this pretty would be
> welcomed.  :-)
>
>
>
>
> -Original Message- From: Stephen Woodbridge
> [mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34
> PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc:
> mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best
> way to import 4.5TB of imagery?
>
> On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE
> wrote:
>> Hi Stephen, Thanks, for the reply.  I previously got 4 sample
>> images from the USDA, and was able to get them to work just fine.
>> There was no processing
> required.
>> The sample images I got were all from Utah, and they are NAD83, UTM
>> zone
> 12.
>> I added the 4 sample images to a shape file using gdaltindex.   I
>> used
> UPSG
>> 26912 and mapserver served them up very quickly for such large
>> files.
>>
>> Now I have this entire data set, and it stretches from UTM zone 10,
>> to UTM zone 19.  The data is divided into directories by two letter
>> state abbreviations, and under that into subdirectories.  I'm just
>> wondering how to add this to my mapfile.  Do I need a different
>> entry for each UTM
> Zone?
>> How is it possible to get a single layer entry that includes
>> multiple projections?  This is looking like a huge job and I just
>> want to know the best strategy for getting this done.
>
> So now you have a problem. You data is in UTM spread over 10
> different projections. What do you plane to do when have your image
> is zone 10 and half is in zone 11 or if you zoom out and you images
> has 3-4 zones displayed?
>
> All data in an image must be rendered in the same projection. While I
> don't doubt that your test with 4 images worked fine, did you you
> test this a multiple zoom levels and at some point you will probably
> want to create a super overlay image so you do not have to open
> multiple files to just pull a tiny overlay out of each one.
>
> Your use cases will determine how you want to deal with the data.
> For example does it HAVE to be in a UTM projection, or can you work
> with a Spherical Mercator or geographic projection? The end solution
> will be much easier if you can work with one common projection over
> your whole data set. Otherwise, you will have to deal with the
> transitions from one zone to the next or maybe set up 10 separate
> servers that only serve one zone.
>
> Having pushed larges amounts (4-25TB) of imagery data more than once
> it is impo

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Stephen Woodbridge]

On 6/10/2013 7:07 PM, Robert Sanson wrote:

Sounds all very complicated. I would advise that you choose a single
projection that will work across your entire are and then re-project
your imagery first, and then build your image datastore around that.


I would agree with this that if htere are not other contraints on the 
problem this is often the best way to go. But that will not owrk if his 
users HAVE to have the data in UTM projection for some reason.


-Steve


"Evans, James R Civ USAF ACC 84 RADES/SCZE"
 11/06/2013 10:48 a.m. >>>

So, I'm guessing there's no easy way to automate this?  Even looking
at the states, some of the states are in two zones, and Texas is
across 3 zones. At least the naming convention of the files indicate
the UTM zone.  For instance:
m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far as I
can tell, all the files in a particular directory are all in the
same UTM zone.  I could create a layer for each UTM zone across
CONUS, but that's not going to be particularly useful to my users.
I'm thinking of making a layer for each state.  For the stats that
cross zones, there will probably be two layers.  For Texas, there
would be Texas_east, Texas_middle, and Texas_west.  I will probably
limit visibility until zoomed in sufficiently to see the whole state
on the screen anyway, since the continental view of this data is
pretty crappy anyway.  So now it seems like it will be a lot of grunt
work just copying these directories up to the server, and going
through and creating a shape file index for each state.  For states
in more than one UTM, there would be more than one shape.  Then I'll
have to add a layer for to my mapfile for each shapefile, using the
correct projection. Is there an easier way?  I'm starting with
Oklahoma, which is also in three UTM zones.  I'll get that working
before moving on.  Any suggestions on making this pretty would be
welcomed.  :-)




-Original Message- From: Stephen Woodbridge
[mailto:wood...@swoodbridge.com] Sent: Monday, June 10, 2013 12:34
PM To: Evans, James R Civ USAF ACC 84 RADES/SCZE Cc:
mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best
way to import 4.5TB of imagery?

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE
wrote:

Hi Stephen, Thanks, for the reply.  I previously got 4 sample
images from the USDA, and was able to get them to work just fine.
There was no processing

required.

The sample images I got were all from Utah, and they are NAD83, UTM
zone

12.

I added the 4 sample images to a shape file using gdaltindex.   I
used

UPSG

26912 and mapserver served them up very quickly for such large
files.

Now I have this entire data set, and it stretches from UTM zone 10,
to UTM zone 19.  The data is divided into directories by two letter
state abbreviations, and under that into subdirectories.  I'm just
wondering how to add this to my mapfile.  Do I need a different
entry for each UTM

Zone?

How is it possible to get a single layer entry that includes
multiple projections?  This is looking like a huge job and I just
want to know the best strategy for getting this done.


So now you have a problem. You data is in UTM spread over 10
different projections. What do you plane to do when have your image
is zone 10 and half is in zone 11 or if you zoom out and you images
has 3-4 zones displayed?

All data in an image must be rendered in the same projection. While I
don't doubt that your test with 4 images worked fine, did you you
test this a multiple zoom levels and at some point you will probably
want to create a super overlay image so you do not have to open
multiple files to just pull a tiny overlay out of each one.

Your use cases will determine how you want to deal with the data.
For example does it HAVE to be in a UTM projection, or can you work
with a Spherical Mercator or geographic projection? The end solution
will be much easier if you can work with one common projection over
your whole data set. Otherwise, you will have to deal with the
transitions from one zone to the next or maybe set up 10 separate
servers that only serve one zone.

Having pushed larges amounts (4-25TB) of imagery data more than once
it is important to make these decisions up front and and prototype up
something like a 4-10 degree square across a UTM boundary and make
sure that the results are going to be what you expect before you
process all the data.

-Steve


Thanks, James





-Original Message- From:
mapserver-users-boun...@lists.osgeo.org
[mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of
Stephen Woodbridge Sent: Friday, June 07, 2013 8:41 AM To:
mapserver-users@lists.osgeo.org Subject: Re: [mapserver-users] Best
way to import 4.5TB of imagery?

On 6/7/2013 10:31 AM, James_in_Utah wrote:

Hi, We just got 3 hard drive, loaded with 4.5TB of NAIP imagery
for all of CONUS.  I think there's a total of about 400,000 jpgs.
The data is in directories, by states.  Under each state, there
are subfol

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Robert Sanson]

Sounds all very complicated. I would advise that you choose a single projection 
that will work across your entire are and then re-project your imagery first, 
and then build your image datastore around that. 

>>> "Evans, James R Civ USAF ACC 84 RADES/SCZE"  
>>> 11/06/2013 10:48 a.m. >>>
So, I'm guessing there's no easy way to automate this?  Even looking at the
states, some of the states are in two zones, and Texas is across 3 zones.
At least the naming convention of the files indicate the UTM zone.  For
instance:  m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far
as I can tell, all the files in a particular directory are all in the same
UTM zone.  I could create a layer for each UTM zone across CONUS, but that's
not going to be particularly useful to my users.  I'm thinking of making a
layer for each state.  For the stats that cross zones, there will probably
be two layers.  For Texas, there would be Texas_east, Texas_middle, and
Texas_west.  I will probably limit visibility until zoomed in sufficiently
to see the whole state on the screen anyway, since the continental view of
this data is pretty crappy anyway.  So now it seems like it will be a lot of
grunt work just copying these directories up to the server, and going
through and creating a shape file index for each state.  For states in more
than one UTM, there would be more than one shape.  Then I'll have to add a
layer for to my mapfile for each shapefile, using the correct projection.
Is there an easier way?  I'm starting with Oklahoma, which is also in three
UTM zones.  I'll get that working before moving on.  Any suggestions on
making this pretty would be welcomed.  :-)




-Original Message-
From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] 
Sent: Monday, June 10, 2013 12:34 PM
To: Evans, James R Civ USAF ACC 84 RADES/SCZE
Cc: mapserver-users@lists.osgeo.org 
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:
> Hi Stephen,
> Thanks, for the reply.  I previously got 4 sample images from the 
> USDA, and was able to get them to work just fine.  There was no processing
required.
> The sample images I got were all from Utah, and they are NAD83, UTM zone
12.
> I added the 4 sample images to a shape file using gdaltindex.   I used
UPSG
> 26912 and mapserver served them up very quickly for such large files.
>
> Now I have this entire data set, and it stretches from UTM zone 10, to 
> UTM zone 19.  The data is divided into directories by two letter state 
> abbreviations, and under that into subdirectories.  I'm just wondering 
> how to add this to my mapfile.  Do I need a different entry for each UTM
Zone?
> How is it possible to get a single layer entry that includes multiple 
> projections?  This is looking like a huge job and I just want to know 
> the best strategy for getting this done.

So now you have a problem. You data is in UTM spread over 10 different
projections. What do you plane to do when have your image is zone 10 and
half is in zone 11 or if you zoom out and you images has 3-4 zones
displayed?

All data in an image must be rendered in the same projection. While I don't
doubt that your test with 4 images worked fine, did you you test this a
multiple zoom levels and at some point you will probably want to create a
super overlay image so you do not have to open multiple files to just pull a
tiny overlay out of each one.

Your use cases will determine how you want to deal with the data. For
example does it HAVE to be in a UTM projection, or can you work with a
Spherical Mercator or geographic projection? The end solution will be much
easier if you can work with one common projection over your whole data set.
Otherwise, you will have to deal with the transitions from one zone to the
next or maybe set up 10 separate servers that only serve one zone.

Having pushed larges amounts (4-25TB) of imagery data more than once it is
important to make these decisions up front and and prototype up something
like a 4-10 degree square across a UTM boundary and make sure that the
results are going to be what you expect before you process all the data.

-Steve

> Thanks,
> James
>
>
>
>
>
> -Original Message-
> From: mapserver-users-boun...@lists.osgeo.org 
> [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen 
> Woodbridge
> Sent: Friday, June 07, 2013 8:41 AM
> To: mapserver-users@lists.osgeo.org 
> Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?
>
> On 6/7/2013 10:31 AM, James_in_Utah wrote:
>> Hi,
>> We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all 
>> of CONUS.  I think there's a total of about 400,000 jpgs.  The data 
>> is in directories, by states.  Under each state, there are 
>> subfolders, probably reference by longitude.  Other than going 
>> through folder by folder, adding each image to a shape file using 
>> gdaltindex, what's the best strategy for loading

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Stephen Woodbridge]

On 6/10/2013 6:48 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:

So, I'm guessing there's no easy way to automate this?  Even looking at the
states, some of the states are in two zones, and Texas is across 3 zones.
At least the naming convention of the files indicate the UTM zone.  For
instance:  m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far
as I can tell, all the files in a particular directory are all in the same
UTM zone.  I could create a layer for each UTM zone across CONUS, but that's
not going to be particularly useful to my users.  I'm thinking of making a
layer for each state.  For the stats that cross zones, there will probably
be two layers.  For Texas, there would be Texas_east, Texas_middle, and
Texas_west.  I will probably limit visibility until zoomed in sufficiently
to see the whole state on the screen anyway, since the continental view of
this data is pretty crappy anyway.  So now it seems like it will be a lot of
grunt work just copying these directories up to the server, and going
through and creating a shape file index for each state.  For states in more
than one UTM, there would be more than one shape.  Then I'll have to add a
layer for to my mapfile for each shapefile, using the correct projection.
Is there an easier way?  I'm starting with Oklahoma, which is also in three
UTM zones.  I'll get that working before moving on.  Any suggestions on
making this pretty would be welcomed.  :-)


If you are using something like OpenLayers for your client you can code 
smarts into that to change mapfiles based on the zone. I think I would 
start with a Perl script that takes every image file and creates a 
record in a postgresql/postgis database that represents the extents of 
the image.


path/to/image/file.jpg | wgs84_geom | utm_geom | zone

or something like that. Then you can create mapfiles that displays these 
rectangles in wgs84, or work out your logic for display the polygons in 
shifting zones. This is quick and easy to setup and lets you proceed 
with the disgn of that with waiting on imagery and it wil be faster.


Beyond that, pushing around 4.5 TB of imagery is not going to be fast.

Good luck,
  -Steve W


-Original Message-
From: Stephen Woodbridge [mailto:wood...@swoodbridge.com]
Sent: Monday, June 10, 2013 12:34 PM
To: Evans, James R Civ USAF ACC 84 RADES/SCZE
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:

Hi Stephen,
Thanks, for the reply.  I previously got 4 sample images from the
USDA, and was able to get them to work just fine.  There was no processing

required.

The sample images I got were all from Utah, and they are NAD83, UTM zone

12.

I added the 4 sample images to a shape file using gdaltindex.   I used

UPSG

26912 and mapserver served them up very quickly for such large files.

Now I have this entire data set, and it stretches from UTM zone 10, to
UTM zone 19.  The data is divided into directories by two letter state
abbreviations, and under that into subdirectories.  I'm just wondering
how to add this to my mapfile.  Do I need a different entry for each UTM

Zone?

How is it possible to get a single layer entry that includes multiple
projections?  This is looking like a huge job and I just want to know
the best strategy for getting this done.


So now you have a problem. You data is in UTM spread over 10 different
projections. What do you plane to do when have your image is zone 10 and
half is in zone 11 or if you zoom out and you images has 3-4 zones
displayed?

All data in an image must be rendered in the same projection. While I don't
doubt that your test with 4 images worked fine, did you you test this a
multiple zoom levels and at some point you will probably want to create a
super overlay image so you do not have to open multiple files to just pull a
tiny overlay out of each one.

Your use cases will determine how you want to deal with the data. For
example does it HAVE to be in a UTM projection, or can you work with a
Spherical Mercator or geographic projection? The end solution will be much
easier if you can work with one common projection over your whole data set.
Otherwise, you will have to deal with the transitions from one zone to the
next or maybe set up 10 separate servers that only serve one zone.

Having pushed larges amounts (4-25TB) of imagery data more than once it is
important to make these decisions up front and and prototype up something
like a 4-10 degree square across a UTM boundary and make sure that the
results are going to be what you expect before you process all the data.

-Steve


Thanks,
James





-Original Message-
From: mapserver-users-boun...@lists.osgeo.org
[mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen
Woodbridge
Sent: Friday, June 07, 2013 8:41 AM
To: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Frank Warmerdam]

On Mon, Jun 10, 2013 at 3:48 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE <
james.ev...@hill.af.mil> wrote:

> So, I'm guessing there's no easy way to automate this?  Even looking at the
> states, some of the states are in two zones, and Texas is across 3 zones.
> At least the naming convention of the files indicate the UTM zone.


James,

Note it is pretty easy to write a script (ie. in Python) that would walk
the directory tree and sort the images into distinct (per utm zone)
collections. So, yes, the processing of this data is very automatable.


> For
> instance:  m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far
> as I can tell, all the files in a particular directory are all in the same
> UTM zone.  I could create a layer for each UTM zone across CONUS, but
> that's
> not going to be particularly useful to my users.  I'm thinking of making a
> layer for each state.  For the stats that cross zones, there will probably
> be two layers.  For Texas, there would be Texas_east, Texas_middle, and
> Texas_west.  I will probably limit visibility until zoomed in sufficiently
>

Is it important to you to distinguish things by state?  If not, why not one
layer per utm zone, and then join them in a layer group 'UTM NAIP' that the
users would either turn on or off?


> to see the whole state on the screen anyway, since the continental view of
> this data is pretty crappy anyway.


Limiting visibility to reasonable resolutions should be fine and would help
you avoiding needing additional preprocessing to create an overview of the
whole collection.


>  So now it seems like it will be a lot of
> grunt work just copying these directories up to the server, and going
> through and creating a shape file index for each state.  For states in more
> than one UTM, there would be more than one shape.  Then I'll have to add a
> layer for to my mapfile for each shapefile, using the correct projection.
> Is there an easier way?  I'm starting with Oklahoma, which is also in three
> UTM zones.  I'll get that working before moving on.  Any suggestions on
> making this pretty would be welcomed.  :-)
>

Well, I still think you should take states out of the equation unless that
is important to your users.

Good luck,

Best regards,

-- 
---+--
I set the clouds in motion - turn up   | Frank Warmerdam,
warmer...@pobox.com
light and sound - activate the windows | http://pobox.com/~warmerdam
and watch the world go round - Rush| Geospatial Software Developer
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Evans, James R Civ USAF ACC 84 RADES/SCZE]

So, I'm guessing there's no easy way to automate this?  Even looking at the
states, some of the states are in two zones, and Texas is across 3 zones.
At least the naming convention of the files indicate the UTM zone.  For
instance:  m_2408002_ne_17_1_20100422_201001123.jp2, is in zone 17.  As far
as I can tell, all the files in a particular directory are all in the same
UTM zone.  I could create a layer for each UTM zone across CONUS, but that's
not going to be particularly useful to my users.  I'm thinking of making a
layer for each state.  For the stats that cross zones, there will probably
be two layers.  For Texas, there would be Texas_east, Texas_middle, and
Texas_west.  I will probably limit visibility until zoomed in sufficiently
to see the whole state on the screen anyway, since the continental view of
this data is pretty crappy anyway.  So now it seems like it will be a lot of
grunt work just copying these directories up to the server, and going
through and creating a shape file index for each state.  For states in more
than one UTM, there would be more than one shape.  Then I'll have to add a
layer for to my mapfile for each shapefile, using the correct projection.
Is there an easier way?  I'm starting with Oklahoma, which is also in three
UTM zones.  I'll get that working before moving on.  Any suggestions on
making this pretty would be welcomed.  :-)




-Original Message-
From: Stephen Woodbridge [mailto:wood...@swoodbridge.com] 
Sent: Monday, June 10, 2013 12:34 PM
To: Evans, James R Civ USAF ACC 84 RADES/SCZE
Cc: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:
> Hi Stephen,
> Thanks, for the reply.  I previously got 4 sample images from the 
> USDA, and was able to get them to work just fine.  There was no processing
required.
> The sample images I got were all from Utah, and they are NAD83, UTM zone
12.
> I added the 4 sample images to a shape file using gdaltindex.   I used
UPSG
> 26912 and mapserver served them up very quickly for such large files.
>
> Now I have this entire data set, and it stretches from UTM zone 10, to 
> UTM zone 19.  The data is divided into directories by two letter state 
> abbreviations, and under that into subdirectories.  I'm just wondering 
> how to add this to my mapfile.  Do I need a different entry for each UTM
Zone?
> How is it possible to get a single layer entry that includes multiple 
> projections?  This is looking like a huge job and I just want to know 
> the best strategy for getting this done.

So now you have a problem. You data is in UTM spread over 10 different
projections. What do you plane to do when have your image is zone 10 and
half is in zone 11 or if you zoom out and you images has 3-4 zones
displayed?

All data in an image must be rendered in the same projection. While I don't
doubt that your test with 4 images worked fine, did you you test this a
multiple zoom levels and at some point you will probably want to create a
super overlay image so you do not have to open multiple files to just pull a
tiny overlay out of each one.

Your use cases will determine how you want to deal with the data. For
example does it HAVE to be in a UTM projection, or can you work with a
Spherical Mercator or geographic projection? The end solution will be much
easier if you can work with one common projection over your whole data set.
Otherwise, you will have to deal with the transitions from one zone to the
next or maybe set up 10 separate servers that only serve one zone.

Having pushed larges amounts (4-25TB) of imagery data more than once it is
important to make these decisions up front and and prototype up something
like a 4-10 degree square across a UTM boundary and make sure that the
results are going to be what you expect before you process all the data.

-Steve

> Thanks,
> James
>
>
>
>
>
> -Original Message-
> From: mapserver-users-boun...@lists.osgeo.org
> [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen 
> Woodbridge
> Sent: Friday, June 07, 2013 8:41 AM
> To: mapserver-users@lists.osgeo.org
> Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?
>
> On 6/7/2013 10:31 AM, James_in_Utah wrote:
>> Hi,
>> We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all 
>> of CONUS.  I think there's a total of about 400,000 jpgs.  The data 
>> is in directories, by states.  Under each state, there are 
>> subfolders, probably reference by longitude.  Other than going 
>> through folder by folder, adding each image to a shape file using 
>> gdaltindex, what's the best strategy for loading a couple of hundred 
>> thousand files up to our server and making the imagery available via 
>> our mapserver?  Should I maintain the current directory structure 
>> when I copy the imagery to the server, or just dump all of it into a 
>> single directory?  Do I want to stay with 1 shape fil

Re: [mapserver-users] MapServer 6.2.1 and ArcSDE 10

[Weisbender, Eric]

Hello,

I recompiled MapServer / SDE setting CFLAGS='-DSDE64'.  I am now getting the 
following error when I try to draw the SDE Layer.

msDrawMap(): Image handling error. Failed to draw layer named 'sde' 
sdeCopyShape(): SDE error. Unsupported SDE shape type (140733193388036).

This is my SDE Layer Definition.  Any help would be greatly appreciated.

LAYER
  NAME "sde"
  TYPE LINE
  STATUS OFF
  CONNECTION "JLVGISGATEM,5353,sde,cso_editor,c1rcu1t#"
  CONNECTIONTYPE SDE
  DATA "RMR.MNT_SPANS,SHAPE,SDE.DEFAULT"
  TEMPLATE "/data/gis/www/html/ms_common/tlines.html"
  CLASS
 STYLE
  COLOR 200 200 0
  WIDTH 10
 END
  END
END

Thanks,

Eric Weisbender
Western Area Power Administration
Corporate Service Office
U.S. Dept. of Energy
office: (720) 962-7280
email: weisb...@wapa.gov

From: Lime, Steve D (MNIT) [mailto:steve.l...@state.mn.us]
Sent: June 07, 2013 1:46 PM
To: Weisbender, Eric; mapserver-users@lists.osgeo.org
Subject: RE: MapServer 6.2.1 and ArcSDE 10

Ah, I ran into this to as well, see ticket 
https://github.com/mapserver/mapserver/issues/4521. I've attached a diff that 
made things work well for me. Once a fix is put in place for 6.4 (git master) 
we can backport to 6.2.

Steve

From: 
mapserver-users-boun...@lists.osgeo.org
 [mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Weisbender, Eric
Sent: Friday, June 07, 2013 2:26 PM
To: mapserver-users@lists.osgeo.org
Subject: [mapserver-users] MapServer 6.2.1 and ArcSDE 10

Hello,

We have successfully compiled MapServer 6.2.1 with ArcSDE10 and can see the SDE 
server ok.  When I try to draw a line layer I get the following error.

msDrawMap(): Image handling error. Failed to draw layer named 'sde' 
msSDELayerWhichShapes(): SDE error. getSDEQueryInfo(): Success. (0 
getSDEQueryInfo(): SDE error. SE_queryinfo_set_columns(): Invalid parameter 
value passed to function. (4294967230)

The layer definition looks like this and all of the parameters look good.

LAYER
  NAME "sde"
  TYPE LINE
  STATUS OFF
  CONNECTIONTYPE SDE
  CONNECTION "JLVGISGATEM,5353,sde,cso_editor,c1rcu1t#"
  DATA "RMR.MNT_SPANS,SHAPE,SDE.DEFAULT"
  TEMPLATE "/data/gis/www/html/ms_common/tlines.html"
  CLASS
 STYLE
  COLOR 200 200 0
  WIDTH 10
 END
  END
END

If anyone has an idea of what I am missing or doing wrong any help is greatly 
appreciated.


Thanks,

Eric Weisbender

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Stephen Woodbridge]

On 6/10/2013 12:57 PM, Evans, James R Civ USAF ACC 84 RADES/SCZE wrote:

Hi Stephen,
Thanks, for the reply.  I previously got 4 sample images from the USDA, and
was able to get them to work just fine.  There was no processing required.
The sample images I got were all from Utah, and they are NAD83, UTM zone 12.
I added the 4 sample images to a shape file using gdaltindex.   I used UPSG
26912 and mapserver served them up very quickly for such large files.

Now I have this entire data set, and it stretches from UTM zone 10, to UTM
zone 19.  The data is divided into directories by two letter state
abbreviations, and under that into subdirectories.  I'm just wondering how
to add this to my mapfile.  Do I need a different entry for each UTM Zone?
How is it possible to get a single layer entry that includes multiple
projections?  This is looking like a huge job and I just want to know the
best strategy for getting this done.


So now you have a problem. You data is in UTM spread over 10 different 
projections. What do you plane to do when have your image is zone 10 and 
half is in zone 11 or if you zoom out and you images has 3-4 zones 
displayed?


All data in an image must be rendered in the same projection. While I 
don't doubt that your test with 4 images worked fine, did you you test 
this a multiple zoom levels and at some point you will probably want to 
create a super overlay image so you do not have to open multiple files 
to just pull a tiny overlay out of each one.


Your use cases will determine how you want to deal with the data. For 
example does it HAVE to be in a UTM projection, or can you work with a 
Spherical Mercator or geographic projection? The end solution will be 
much easier if you can work with one common projection over your whole 
data set. Otherwise, you will have to deal with the transitions from one 
zone to the next or maybe set up 10 separate servers that only serve one 
zone.


Having pushed larges amounts (4-25TB) of imagery data more than once it 
is important to make these decisions up front and and prototype up 
something like a 4-10 degree square across a UTM boundary and make sure 
that the results are going to be what you expect before you process all 
the data.


-Steve


Thanks,
James





-Original Message-
From: mapserver-users-boun...@lists.osgeo.org
[mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen
Woodbridge
Sent: Friday, June 07, 2013 8:41 AM
To: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?

On 6/7/2013 10:31 AM, James_in_Utah wrote:

Hi,
We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of
CONUS.  I think there's a total of about 400,000 jpgs.  The data is in
directories, by states.  Under each state, there are subfolders,
probably reference by longitude.  Other than going through folder by
folder, adding each image to a shape file using gdaltindex, what's the
best strategy for loading a couple of hundred thousand files up to our
server and making the imagery available via our mapserver?  Should I
maintain the current directory structure when I copy the imagery to
the server, or just dump all of it into a single directory?  Do I want
to stay with 1 shape file, or break it up by state?  We eventually
want a contiguous layer for all of CONUS to be served up to our users.


James,

Since imagery data is served via gdal, you might want to also ask this
question on the gdal list.

There are issues with jpg related to the fact that if you only want a small
part of the image you still have to uncompress the whole image. So part of
the answer might be that you need to pre-process all the imagery into
something like a jpg compress tiled geotif or something else.

You also need to consider what projection your imagery is in and what
projection you want to display it in. Because if you need to preprocess the
data, that would also be a good time to reproject it.

Anyway the gdal list can probably ask additional questions to help sort all
that out.

-Steve W

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users



___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Templated for related data

[Johan Forsman]

All:

I have a case where I want to show data in a one-to-many relationship, but the 
related data are more than one table "away", or perhaps expressed as 
one-to-many-to-many.
For example, I have three tables: [sampling_point], [samples], 
[analytical_results].

[samples] is linked to [sampling_point] and
[analytical_results] is linked to [samples].
[samples] contains sample id, sampling date, etc., and
[analytical_results] contains the analytical names and results for each sample.

I want to map each sampling point once, then I want to display the related 
analytical data as a table so that the data are "grouped" first by sampling 
point, then by sample id + sampling date, approximately like so:

[sampling_point_id1]
[samples_id1] + [samples_date1]
[analyte_name1] + [analyte_result1]
[analyte_name2] + [analyte_result2]
[analyte_name3] + [analyte_result3]
[samples_id2] + [samples_date2]
[analyte_name1] + [analyte_result1]
[analyte_name2] + [analyte_result2]
[analyte_name3] + [analyte_result3]
[sampling_point_id2]
[samples_id1] + [samples_date1]
[analyte_name1] + [analyte_result1]

Etc.

I can successfully display the data at one level "shallower" by using the LAYER 
+ JOIN structure and the associated templates, however I have only been able to 
accomplish this between one layer and one table, thus either losing the 
grouping of unique sample_id and associated sample_date, or including those 
fields with the analytical results and creating long displays of repeated id 
and date values for each analytical result like so:

[sampling_point_id1]
[samples_id1] + [samples_date1] + [analyte_name1] + [analyte_result1]
[samples_id1] + [samples_date1] + [analyte_name2] + [analyte_result2]
[samples_id1] + [samples_date1] + [analyte_name3] + [analyte_result3]
[samples_id2] + [samples_date2] + [analyte_name1] + [analyte_result1]
[samples_id2] + [samples_date2] + [analyte_name2] + [analyte_result2]
[samples_id2] + [samples_date2] + [analyte_name3] + [analyte_result3]
[sampling_point_id2]
Etc.

Please advise how I may accomplish this or redirect my thinking, whichever is 
necessary.

Thank you,
Johan

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

Re: [mapserver-users] Best way to import 4.5TB of imagery?

[Evans, James R Civ USAF ACC 84 RADES/SCZE]

Hi Stephen,
Thanks, for the reply.  I previously got 4 sample images from the USDA, and
was able to get them to work just fine.  There was no processing required.
The sample images I got were all from Utah, and they are NAD83, UTM zone 12.
I added the 4 sample images to a shape file using gdaltindex.   I used UPSG
26912 and mapserver served them up very quickly for such large files.  

Now I have this entire data set, and it stretches from UTM zone 10, to UTM
zone 19.  The data is divided into directories by two letter state
abbreviations, and under that into subdirectories.  I'm just wondering how
to add this to my mapfile.  Do I need a different entry for each UTM Zone?
How is it possible to get a single layer entry that includes multiple
projections?  This is looking like a huge job and I just want to know the
best strategy for getting this done.
Thanks,
James




 
-Original Message-
From: mapserver-users-boun...@lists.osgeo.org
[mailto:mapserver-users-boun...@lists.osgeo.org] On Behalf Of Stephen
Woodbridge
Sent: Friday, June 07, 2013 8:41 AM
To: mapserver-users@lists.osgeo.org
Subject: Re: [mapserver-users] Best way to import 4.5TB of imagery?

On 6/7/2013 10:31 AM, James_in_Utah wrote:
> Hi,
> We just got 3 hard drive, loaded with 4.5TB of NAIP imagery for all of 
> CONUS.  I think there's a total of about 400,000 jpgs.  The data is in 
> directories, by states.  Under each state, there are subfolders, 
> probably reference by longitude.  Other than going through folder by 
> folder, adding each image to a shape file using gdaltindex, what's the 
> best strategy for loading a couple of hundred thousand files up to our 
> server and making the imagery available via our mapserver?  Should I 
> maintain the current directory structure when I copy the imagery to 
> the server, or just dump all of it into a single directory?  Do I want 
> to stay with 1 shape file, or break it up by state?  We eventually 
> want a contiguous layer for all of CONUS to be served up to our users.

James,

Since imagery data is served via gdal, you might want to also ask this
question on the gdal list.

There are issues with jpg related to the fact that if you only want a small
part of the image you still have to uncompress the whole image. So part of
the answer might be that you need to pre-process all the imagery into
something like a jpg compress tiled geotif or something else.

You also need to consider what projection your imagery is in and what
projection you want to display it in. Because if you need to preprocess the
data, that would also be a good time to reproject it.

Anyway the gdal list can probably ask additional questions to help sort all
that out.

-Steve W

___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users


smime.p7s
Description: S/MIME cryptographic signature
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users

[mapserver-users] Installing mod_mapcache on Apache 2.4.4

[Rahkonen Jukka]

Hi,

In  case this could help someone, I found that for some reason the instructions 
for installing Mapcache on Apache version >2.4 did not work for me.
The document http://www.mapserver.org/trunk/mapcache/install.html suggests to 
use the new access control syntax which is described in a similar way also in 
this Apache document http://www.mapserver.org/trunk/mapcache/install.html

The syntax to use nowadays is like

   
  Require all granted
   
   MapCacheAlias /mapcache "/path/to/directory/mapcache.xml"
 "

However, with that configuration my Apache v. 2.4.4 (Unix) on CentOS told that 
there is no permission to /mapcache. After changing back to the Apache <2.4 
syntax I can access  Mapcache


  Order Allow,Deny
  Allow from all


-Jukka Rahkonen-
___
mapserver-users mailing list
mapserver-users@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/mapserver-users