Hi, I can also have a look if you can put original, gray scale creater by Kakadu and the one created with GM somewhere for downloading. Private message is OK, I can mail my findings back to the list.
-Jukka Rahkonen- EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote: > > Hi Jukka, > This has certainly been entertaining. I grabbed kakadu, and it could convert > the > images from jp2000 to jp2000 in about 5 seconds. The speed was impressive. > Unfortunately, probably due to my limited jp2000 experience, kakadu sees the > image as gray scale only. When I export the image to another jp2000, my > mapserver can process it just fine, but it's only in gray scale. I then > tried Global > Mapper scripting. It exports the jp2 image to jp2 in about 30 seconds, but > retains the color. Plus, the file size is reduce about a third. > Fortunately, I can > run about 3 of these scripts at once on a machine, and still write them out > ever > 30 seconds or so, so I'm averaging about 1 file ever 10 seconds. I've got 3 > machines working on this so it should take about a week or two to convert the > whole set. If I try to open one of the new jp2 files produced by GM in > kakadu, it > complains that there is something illegal in the header, but then does > display the > image and it looks just fine. I'm not sure what encoder the USDA is using to > produce these files, but hopefully they get an update before we have to get > more files from them. > Thanks, > James > > > > ________________________________________ > From: Rahkonen Jukka (Tike) [jukka.rahko...@mmmtike.fi] > Sent: Friday, March 21, 2014 7:53 AM > To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; 'mapserver- > us...@lists.osgeo.org' > Subject: Re: [mapserver-users] Processing JP2000 files > > Hi James, > > It has been a while since I have done any experiments with JPEG2000 and > stopwatch but I happened to have a rather recent versions of Kakadu demo > executables on my machine (version 7.3.2, the very latest is 7.3.3). We do > not > have a license for the Kakadu speed pack, but even without it the conversion > from lossless JPEG2000 into lossless GeoTIFF feels quite speedy. Here comes > the > facts: > > Hardware is a Windows 7 laptop with Intel i7, 8 cores and 8 GB RAM. > Test image is 12000 x 12000 pixel 3-band aerial image, 443 MB as > uncompressed, 170 MB as lossless JPEG2000 Here is the kdu_expand command > and conversion log: > > Kakadu_732\bin_x64\kdu_expand -i test_in.JP2 -o test_out.tif -cpu 0 -mem > Copying Geo box info, size = 389 > End-to-end CPU time = 6.317000 seconds (0.014623 us/sample) > > Consumed 8 tile-part(s) from a total of 1 tile(s). > Consumed 174,371,002 codestream bytes (excluding any file format) = 9.687278 > bits/pel. Processed using the multi-threaded environment, with 8 parallel > threads of execution > > Native Kakadu application used only 6.3 seconds CPU time for this task. I > also > made a comparison with GDAL 64-bit with driver > JP2ECW (rov): ERDAS JPEG2000 (SDK 5.0) > End-to-end time including disk write was 21 second. Kadudu timing was 6.3 > seconds but I took timing also to include disk write and it was still under 7 > seconds. It may be that file system was still doing some delayed writing but > the > scale should be about right. > My test image was compressed as one tile and JP2OpenJPEG driver could not > convert it. > > Final comparison: gdal_translate from uncompressed tif to uncompressed tif: 3 > seconds. > > TIFF to TIFF with GDAL: 3 seconds > JP2 to TIFF with native Kakadu: 7 seconds > JP2 to TIFF with GDAL ECWJP2 (SDK 5.0): 21 seconds > > Your Global Mapper timing with much smaller image was 3 minutes. You have > 400000 files... > > -Jukka Rahkonen- > > > > > > -----Alkuperäinen viesti----- > > Lähettäjä: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE > > [mailto:james.evan...@us.af.mil] > > Lähetetty: 21. maaliskuuta 2014 14:22 > > Vastaanottaja: Rahkonen Jukka (Tike); 'mapserver-users@lists.osgeo.org' > > Aihe: RE: [mapserver-users] Processing JP2000 files > > > > Hi Jukka, > > I have Global Mapper, and used it to do a batch convert to geotiff on > > a test set of images. The converted files work fine with the OpenJPEG > > drive, but the size almost doubles and on my 10 year old pc at home it > > takes about 3 minutes per file to convert. I will look at all the options > > you > mentioned below. > > Thanks! > > James > > > > > > > > ________________________________________ > > From: Rahkonen Jukka (Tike) [jukka.rahko...@mmmtike.fi] > > Sent: Friday, March 21, 2014 3:33 AM > > To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; 'mapserver- > > us...@lists.osgeo.org' > > Subject: Re: [mapserver-users] Processing JP2000 files > > > > Hi, > > > > From your other mail I concluded that your images are converted into > > JPEG2000 with some Lizardtech product so it is not a surprise that > > MrSID/MrSIG JPEG2000 driver is happy with the SOT markers. I do not > > know if it is a real error in file structure or only a different > > JPEG2000 dialect. JPEG2000 is very feature rich and complicated. > > > > It seems that OpenJPEG driver is not most happy with the JPEG2000 > > structure in your images. It could be happier if the data streams > > inside JP2 file were organized in some other way. There are tools (for > > example Kakadu > > kdu_transcode) for re-organizing JPEG2000 files without recompression > > and data loss but that would still mean a rewrite for all your images. > > OpenJPEG is as its best is rather fast and almost usable for > > production but it depends on how much load you wait for your server. > > However, myself I would not follow that route. I would consider two > alternatives: > > > > - Acquire Mapserver + GDAL with some fast JPEG2000 driver: JPKAK, > > JP2ECW, or JP2MrSID. Test with your images and take care of a license. > > Those SDKs are not free and you can't use them on server without paying. > > - Convert your images into GeoTIFF format (tiled, jpeg compressed, > > with compressed overviews). You need a fast JPEG200O driver also for > > that but for example JP2ECW license allows desktop use without payed license > (check it!). > > Conversion will take some time because there are 1440 minutes in a day > > and you said you have 400000 images. With one computer it would take > > several months so I suggest to gather lots of computers and lots of > > USB drives so that you can read from one drive and write converted > > images into another. There is lots of computing power around nowadays > > and from any medium scaled office you can find 100 laptops which are > > idle for all nights and weekends. They could handle your data within a > > week or two. GeoTIFFs will take a bit more disk space mainly because > > of overviews (JPEG2000 streams are progressive and overviews are > > unnecessary) but they will be faster with your Mapserver and other > > GDAL based software like QGIS and with Geoserver too. > > > > -Jukka Rahkonen- > > > > EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE wrote: > > > > > Hi Jukka, > > > > > > The MS4W version says it is using Driver: JP2MrSID/MrSID JPEG2000, > > > and the GISInternals version is using Driver: JP2OpenJPEG/JPEG-2000 > > > driver based on OpenJPEG library. I tried to do a gdal_translate on > > > the same file, using the each of the drivers. They both translated > > > the file into a 512x512 jpg, but the JP2OpenJPEG driver listed over > > > a 1000 of these warnings before it stopped listing > > > warnings: > > > Warning 1: Empty SOT marker detected: Psot=12. > > > > > > So, I have 400,000 of these JP2000 images, equally 4.3TBs of data. > > > Should I consider a massive batch conversion to JPG or Geotiff, or > > > is there some way to get the JP2OpenJPEG driver to play nice with these > files? > > > > > > Thanks, > > > James > > > ________________________________________ > > > From: Rahkonen Jukka (Tike) [jukka.rahko...@mmmtike.fi] > > > Sent: Thursday, March 20, 2014 7:44 AM > > > To: EVANS, JAMES R GS-13 USAF ACC 84 RADES/SCZE; mapserver- > > > us...@lists.osgeo.org > > > Subject: VS: [mapserver-users] Processing JP2000 files > > > > > > Hi, > > > > > > Are your JPEG2000 images the same? SOT marker is JPEG2000 stuff and > > > it means "start of tilepart". If images are the same then the > > > message means that something has changed in GDAL part. GDAL has > > > several > > > JPEG200 drivers and they all behave in a bit different way. Check > > > which driver you have in your old system and which one you have now > > > with > > "gdalinfo --formats". > > > I warn that from the free alternatives only OpenJPEG driver is > > > something that can be considered for real use. > > > > > > -Jukka Rahkonen- > > > > > > > > > > > > James_in_Utah wrote: > > > > > > > > There must have been something wrong with my shapefile, because I > > > > tried this at home and I was able to get the same data to work. > > > > Now the problem is that the image I see on my client is all washed > > > > out, not a variant RGB image but mostly brownish with weak greens. > > > > None of the reds or blues are making it through. Plus, where the > > > > tiles overlap I can see the edges are blended from the overlapping > > > > tiles. > > > > This wasn't the case with the older GDAL from MS4W. Has something > > > changed? Is there some directive in the layer that I'm missing? > > > > Thanks, > > > > James > > > > > > > > > > > > > > > > > > > > -- > > > > View this message in context: http://osgeo- > > > > org.1560.x6.nabble.com/Processing-JP2000-files-tp5129997p5130136.h > > > > tm l Sent from the Mapserver - User mailing list archive at > > > > Nabble.com. > > > > _______________________________________________ > > > > mapserver-users mailing list > > > > mapserver-users@lists.osgeo.org > > > > http://lists.osgeo.org/mailman/listinfo/mapserver-users _______________________________________________ mapserver-users mailing list mapserver-users@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/mapserver-users