RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms
Landon, >It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Hmm, interesting angle, to expand on your idea a bit more, what about a processing suite (or set of suites) that process data for different types of uses, visual display, DEM analysis, etc. Each "processor" stack would/could have it's own rules associated with data resolution vs files sizes, etc. bobb >>> "Landon Blake" wrote: Bobb wrote: “Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard.” I agree with some of the points you are making in your argument Bobb. There is certainly a practical limit to how much you data you should put in a single file. That is why we have lumber cut to 8 foot lengths. You don’t need a flatbed semi to carry it to your house. :] When you refer to a standard for splitting data up front, what do you mean? It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn’t been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Landon From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 7:33 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms All, Ok, I'm probably going to get someone irritated, but here goes . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb >>> "Fawcett, David" wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some[mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms > Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the "itch" sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation
RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms
Bobb wrote: "Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard." I agree with some of the points you are making in your argument Bobb. There is certainly a practical limit to how much you data you should put in a single file. That is why we have lumber cut to 8 foot lengths. You don't need a flatbed semi to carry it to your house. :] When you refer to a standard for splitting data up front, what do you mean? It would be interesting to come up with a standard structure on a computer file system that could be used to accessed tiled raster data, if this hasn't been done already. One the file system structure was defined, it would be fairly easy to write open source software that accessed this structure and provided individual tiles as a service to desktop GIS applications. Landon From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bob Basques Sent: Monday, August 24, 2009 7:33 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms All, Ok, I'm probably going to get someone irritated, but here goes . . . Why not approach this from the other end of the spectrum and work at making the original files smaller. Work with the providers to make the images smaller in the first place, or at least come up with a maximum practical size to work with, I mean if this is the only (or biggest reason) for implementing JP2, then getting folks to make the smaller deliverables seems like a better long term approach. Here's my reasoning, we're never (ever?) going to hit the top end on how big files ever get, resolution just keeps going up and up, so there is always going to be some upper limit that will need to be breached somehow. Working out a proper method for segregating the data up front (dare I say it), as some sort of standard (which can be adjusted as time passes) will make everything work nicely, then all will work with available tools when they are available, if tools to handle larger datasets become available, and the community feels there is a reason/need that these new larger files need to be handled, then they get to change the standard. bobb >>> "Fawcett, David" wrote: I realize that there are likely not a large number of people who have the expertise and experience to write this kind of code. Is this a project that should be shopped around for funding? Google Summer of Code? A grant from our ~benevolent overlord Google? Some other foundation or org interested in open data formats? David. -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael P. Gerlek Sent: Thursday, August 20, 2009 4:36 PM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms > Do you know why there hasn't been a broader adoption of JP2? Not through lack of trying on my part :-) I think the two biggest reasons are: (1) The algorithms for handling large images in memory really are rocket science, and no one in the FOSS community has gotten the "itch" sufficiently bad enough to go and do the work needed inside the existing open source packages. Hopefully someday someone will. ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss ___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss Warning: Information provided via electronic media is not guaranteed against defects including translation and transmission errors. If the reader is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this information in error, please notify the sender immediately.___ Discuss mailing list Discuss@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/discuss
RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED]
"Tiling" essentially means you can take a large file and compress pieces of it independently. This avoids having to deal with the large memory footprint issues, but it can also lead to seam-line artifacts under certain conditions. Ideally, one would prefer to have the option of compressing large images without resorting to using tiles. Note too that, in addition to the large image issue, many of the JP2 implementations out there are either not fully compliant or are not tuned for performance. A viable solution would need both of these. -mpg -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Chris Puttick Sent: Friday, August 21, 2009 9:37 AM To: OSGeo Discussions Subject: Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides "complete JP2 support at the decoding side" - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - "Christopher Schmidt" wrote: > On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: > > Thanks for the information Michael. I am downloading Opticks right > now. > > :] > > > > I also found this Java library for JP2, thought I'm not sure how > > complete/up-to-date it is: > > > > http://jj2000.epfl.ch/ > > > > Maybe we need a JPEG 2000 page on the OSGeo wiki. > > Note that "JPEG 2000 support" is different from "JPEG 2000 support > which > works on geo-sized images." The tiling (or 'paging'? as Michael calls > it) support that's supposed to be provided by OpenJPEG2000 has been > coming 'real soon now' for about 18 months now from my uneducated > observations, and until it's there, most tools using OpenJPEG for JP2s > are > going to suffering under much the same limitations. > > -- Chris > > > Landon > > Office Phone Number: (209) 946-0268 > > Cell Phone Number: (209) 992-0658 > > > > > > > > -----Original Message- > > From: discuss-boun...@lists.osgeo.org > > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, > Michael > > Sent: Friday, August 21, 2009 8:09 AM > > To: OSGeo Discussions > > Subject: RE: [OSGeo-Discuss] Open File Formats and > > ProprietaryAlgorithms[SEC=UNCLASSIFIED] > > > > All, > > > > Opticks is an open source remote sensing application and > development > > framework. We recently started the process to add JPEG 2000 support > to > > our framework. We picked OpenJpeg to add JPEG 2000 support to our > > application. They are also open source. We currently support > importing > > JPEG 2000 files but we are currently limited to the 4GB memory size > > after decoding. > > > > Our plan is to continue development and to upgrade to OpenJpeg 2.0 > once > > they have a stable release. That will allow Opticks to use a pager > to > > display and support much larger files. > > > > Michael Considine > > > > -Original Message- > > From: discuss-boun...@lists.osgeo.org > > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce > Bannerman > > Sent: Thursday, August 20, 2009 8:15 PM > > To: 'OSGeo Discussions' > > Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary > > Algorithms[SEC=UNCLASSIFIED] > > > > > > IMO: > > > > > > Just another thought on this issue (though we do seem to be > recycling > > arguments over the years...): > > > > > > Assuming that I have a very large archive of spatial data, be it > imagery > > or any other spatial format and that I store my data in a variety > of > > proprietary formats: > > > > > > In ten years from now, can I be sure that: > > > > - the company that created, understands, and holds the IP in the > > data format will still be around? > > > > - there will still be software that runs on the then current > > operating environment, that can read and 'fully exploit' the data > > in the proprietary standard? > > > > - that this future software will work seamlessly with my then > current > > spatial environment? > > > > - if all of the above risks prove to eventuate, can I be sure that > I'll > > be able to salvage my data into another format, retaining its > complete > > > > semantic context? > > > > > > IMO, it is a hig
Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED]
Well, according to this page: http://jpeg2000.epfl.ch/ v.5.1, courtesy in part Eastman Kodak, provides "complete JP2 support at the decoding side" - not sure whether that covers the tiling or other geo needs, but doesn't it sound worth investigating? Chris - "Christopher Schmidt" wrote: > On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: > > Thanks for the information Michael. I am downloading Opticks right > now. > > :] > > > > I also found this Java library for JP2, thought I'm not sure how > > complete/up-to-date it is: > > > > http://jj2000.epfl.ch/ > > > > Maybe we need a JPEG 2000 page on the OSGeo wiki. > > Note that "JPEG 2000 support" is different from "JPEG 2000 support > which > works on geo-sized images." The tiling (or 'paging'? as Michael calls > it) support that's supposed to be provided by OpenJPEG2000 has been > coming 'real soon now' for about 18 months now from my uneducated > observations, and until it's there, most tools using OpenJPEG for JP2s > are > going to suffering under much the same limitations. > > -- Chris > > > Landon > > Office Phone Number: (209) 946-0268 > > Cell Phone Number: (209) 992-0658 > > > > > > > > -Original Message- > > From: discuss-boun...@lists.osgeo.org > > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, > Michael > > Sent: Friday, August 21, 2009 8:09 AM > > To: OSGeo Discussions > > Subject: RE: [OSGeo-Discuss] Open File Formats and > > ProprietaryAlgorithms[SEC=UNCLASSIFIED] > > > > All, > > > > Opticks is an open source remote sensing application and > development > > framework. We recently started the process to add JPEG 2000 support > to > > our framework. We picked OpenJpeg to add JPEG 2000 support to our > > application. They are also open source. We currently support > importing > > JPEG 2000 files but we are currently limited to the 4GB memory size > > after decoding. > > > > Our plan is to continue development and to upgrade to OpenJpeg 2.0 > once > > they have a stable release. That will allow Opticks to use a pager > to > > display and support much larger files. > > > > Michael Considine > > > > -Original Message- > > From: discuss-boun...@lists.osgeo.org > > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce > Bannerman > > Sent: Thursday, August 20, 2009 8:15 PM > > To: 'OSGeo Discussions' > > Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary > > Algorithms[SEC=UNCLASSIFIED] > > > > > > IMO: > > > > > > Just another thought on this issue (though we do seem to be > recycling > > arguments over the years...): > > > > > > Assuming that I have a very large archive of spatial data, be it > imagery > > or any other spatial format and that I store my data in a variety > of > > proprietary formats: > > > > > > In ten years from now, can I be sure that: > > > > - the company that created, understands, and holds the IP in the > > data format will still be around? > > > > - there will still be software that runs on the then current > > operating environment, that can read and 'fully exploit' the data > > in the proprietary standard? > > > > - that this future software will work seamlessly with my then > current > > spatial environment? > > > > - if all of the above risks prove to eventuate, can I be sure that > I'll > > be able to salvage my data into another format, retaining its > complete > > > > semantic context? > > > > > > IMO, it is a high risk proposition to lock public (or private) > archives > > away in proprietary data formats. It makes more sense to use open > > standards and formats that are publically available. > > > > > > > > Bruce Bannerman > > > > > > > > > > > > > -Original Message- > > > From: discuss-boun...@lists.osgeo.org > > > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael > > > P. Gerlek > > > Sent: Friday, 21 August 2009 6:55 AM > > > To: OSGeo Discussions > > > Subject: RE: [OSGeo-Discuss] Open File Formats and > > > Proprietary Algorithms > > > > > > Some clarifications: > > > > > > > > > > > > - MrSID has both lossy and lossless m
Re: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED]
On Fri, Aug 21, 2009 at 08:27:13AM -0700, Landon Blake wrote: > Thanks for the information Michael. I am downloading Opticks right now. > :] > > I also found this Java library for JP2, thought I'm not sure how > complete/up-to-date it is: > > http://jj2000.epfl.ch/ > > Maybe we need a JPEG 2000 page on the OSGeo wiki. Note that "JPEG 2000 support" is different from "JPEG 2000 support which works on geo-sized images." The tiling (or 'paging'? as Michael calls it) support that's supposed to be provided by OpenJPEG2000 has been coming 'real soon now' for about 18 months now from my uneducated observations, and until it's there, most tools using OpenJPEG for JP2s are going to suffering under much the same limitations. -- Chris > Landon > Office Phone Number: (209) 946-0268 > Cell Phone Number: (209) 992-0658 > > > > -Original Message- > From: discuss-boun...@lists.osgeo.org > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, Michael > Sent: Friday, August 21, 2009 8:09 AM > To: OSGeo Discussions > Subject: RE: [OSGeo-Discuss] Open File Formats and > ProprietaryAlgorithms[SEC=UNCLASSIFIED] > > All, > > Opticks is an open source remote sensing application and development > framework. We recently started the process to add JPEG 2000 support to > our framework. We picked OpenJpeg to add JPEG 2000 support to our > application. They are also open source. We currently support importing > JPEG 2000 files but we are currently limited to the 4GB memory size > after decoding. > > Our plan is to continue development and to upgrade to OpenJpeg 2.0 once > they have a stable release. That will allow Opticks to use a pager to > display and support much larger files. > > Michael Considine > > -Original Message- > From: discuss-boun...@lists.osgeo.org > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce Bannerman > Sent: Thursday, August 20, 2009 8:15 PM > To: 'OSGeo Discussions' > Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary > Algorithms[SEC=UNCLASSIFIED] > > > IMO: > > > Just another thought on this issue (though we do seem to be recycling > arguments over the years...): > > > Assuming that I have a very large archive of spatial data, be it imagery > or any other spatial format and that I store my data in a variety of > proprietary formats: > > > In ten years from now, can I be sure that: > > - the company that created, understands, and holds the IP in the > data format will still be around? > > - there will still be software that runs on the then current > operating environment, that can read and 'fully exploit' the data > in the proprietary standard? > > - that this future software will work seamlessly with my then current > spatial environment? > > - if all of the above risks prove to eventuate, can I be sure that I'll > be able to salvage my data into another format, retaining its complete > > semantic context? > > > IMO, it is a high risk proposition to lock public (or private) archives > away in proprietary data formats. It makes more sense to use open > standards and formats that are publically available. > > > > Bruce Bannerman > > > > > > > -Original Message- > > From: discuss-boun...@lists.osgeo.org > > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael > > P. Gerlek > > Sent: Friday, 21 August 2009 6:55 AM > > To: OSGeo Discussions > > Subject: RE: [OSGeo-Discuss] Open File Formats and > > Proprietary Algorithms > > > > Some clarifications: > > > > > > > > - MrSID has both lossy and lossless modes > > > > - MrSID is not fractal based; it uses wavelets (and > > arithmetic encoding) > > > > - you can't copyright algorithms; the MrSID source code > > certainly is, however > > > > - MrSID relies on a number of patents, not all of which are > > owned by LizardTech > > > > - reading MrSID does not require any fees; we have libraries > > you can download, although they are not open source > > > > > > > > That said, some editorial comments (although I'm now wishing > > I hadn't been so quick to rise to Landon's bait :-) > > > > > > > > - Some of you know the history of trying to open source > > MrSID; I won't go into that here, except to say that > > LizardTech doesn't own all of the required IP needed to make > > that happen. > > > > - If we are speaking o
RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED]
Thanks for the information Michael. I am downloading Opticks right now. :] I also found this Java library for JP2, thought I'm not sure how complete/up-to-date it is: http://jj2000.epfl.ch/ Maybe we need a JPEG 2000 page on the OSGeo wiki. Landon Office Phone Number: (209) 946-0268 Cell Phone Number: (209) 992-0658 -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Considine, Michael Sent: Friday, August 21, 2009 8:09 AM To: OSGeo Discussions Subject: RE: [OSGeo-Discuss] Open File Formats and ProprietaryAlgorithms[SEC=UNCLASSIFIED] All, Opticks is an open source remote sensing application and development framework. We recently started the process to add JPEG 2000 support to our framework. We picked OpenJpeg to add JPEG 2000 support to our application. They are also open source. We currently support importing JPEG 2000 files but we are currently limited to the 4GB memory size after decoding. Our plan is to continue development and to upgrade to OpenJpeg 2.0 once they have a stable release. That will allow Opticks to use a pager to display and support much larger files. Michael Considine -Original Message- From: discuss-boun...@lists.osgeo.org [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Bruce Bannerman Sent: Thursday, August 20, 2009 8:15 PM To: 'OSGeo Discussions' Subject: RE: [OSGeo-Discuss] Open File Formats and Proprietary Algorithms[SEC=UNCLASSIFIED] IMO: Just another thought on this issue (though we do seem to be recycling arguments over the years...): Assuming that I have a very large archive of spatial data, be it imagery or any other spatial format and that I store my data in a variety of proprietary formats: In ten years from now, can I be sure that: - the company that created, understands, and holds the IP in the data format will still be around? - there will still be software that runs on the then current operating environment, that can read and 'fully exploit' the data in the proprietary standard? - that this future software will work seamlessly with my then current spatial environment? - if all of the above risks prove to eventuate, can I be sure that I'll be able to salvage my data into another format, retaining its complete semantic context? IMO, it is a high risk proposition to lock public (or private) archives away in proprietary data formats. It makes more sense to use open standards and formats that are publically available. Bruce Bannerman > -Original Message- > From: discuss-boun...@lists.osgeo.org > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Michael > P. Gerlek > Sent: Friday, 21 August 2009 6:55 AM > To: OSGeo Discussions > Subject: RE: [OSGeo-Discuss] Open File Formats and > Proprietary Algorithms > > Some clarifications: > > > > - MrSID has both lossy and lossless modes > > - MrSID is not fractal based; it uses wavelets (and > arithmetic encoding) > > - you can't copyright algorithms; the MrSID source code > certainly is, however > > - MrSID relies on a number of patents, not all of which are > owned by LizardTech > > - reading MrSID does not require any fees; we have libraries > you can download, although they are not open source > > > > That said, some editorial comments (although I'm now wishing > I hadn't been so quick to rise to Landon's bait :-) > > > > - Some of you know the history of trying to open source > MrSID; I won't go into that here, except to say that > LizardTech doesn't own all of the required IP needed to make > that happen. > > - If we are speaking of the NAIP data, then no, it is not > exclusively available in MrSID format; it is also shipped as GeoTIFFs. > > - JPEG 2000 is a very robust open standard alternative to > MrSID, and a number of players already support it (including > LizardTech), but not enough to make it viable for certain > domains like NAIP. > > - some of you also know the history on open JP2 support: > there is today no open source implementation of JP2 that is > suitable for geo work. Alas. > > > > -mpg > > > > > > From: discuss-boun...@lists.osgeo.org > [mailto:discuss-boun...@lists.osgeo.org] On Behalf Of Eric Wolf > Sent: Thursday, August 20, 2009 2:15 PM > To: OSGeo Discussions > Subject: Re: [OSGeo-Discuss] Open File Formats and > Proprietary Algorithms > > > > The MRSID format is a very special case - and perhaps an > opportunity for a new FOSS file format. MRSID is a lossless, > fractal-based, multi-scale raster compression format. > LizardTech has the algorithms to encode and decode MRSID > locked up in copyrights, and I believe, patents. Even &
