Re: [gdal-dev] FITS driver update?

[Chiara Marmo]

Hi Simon,

> It's been well over a decade since I last looked at this code, and I
> originally wrote it for a non-astronomy use-case (we just needed a format
> that could handle multi-spectral imagery), so it's entirely possible it
> doesn't quite align with normal astronomy conventions... My main concern
> though is that making these changes would likely break existing code that
> uses this library. Maybe you could add a param to enable a "flip" mode that
> flips the y-axis while keeping the current default behavior?

I did a (quick, maybe incomplete) research on which software uses gdal to read 
FITS and the only one I have found, beeing largely used , is QGIS. I do not 
know a lot of peaple (excepting me :) ) using QGIS to display FITS... that's 
why I'm working on that. 

> - I tend to think of image origin as a display issue. It's not clear to me
> that GDAL implies any particular origin for its coordinate system, so it
> seems simpler to me to have the first row of data in the FITS file
> correspond to the first row of data from GDAL's POV as well. For that
> matter, it's not clear to me that FITS implies a specific origin -
> displaying the first pixel in the lower left is more of a convention? It's
> true that converting to a TIFF and then displaying the TIFF in the usual way
> flips the image, but I'd probably tackle that by simply flipping the TIFF
> image vertically after conversion rather than messing around with the data
> ordering. Just my 2c.

Indeed, the standard description of FITS says that for raster the origin is 
bottom left pixel, and its center is 1,1: those two definitions are important 
when georeferencing a raster.
I'm working at the inteface between planetary mappers , using GIS software and 
GDAL, and astronomers ,using traditional FITS visualization software (like 
Aladin or ds9): FITS images are sometimes raw data, without geographical 
information, the results is that GDAL based tools display flipped FITS with 
respect to astronomical software. Each time I have to explain that there is a 
different convention in FITS and generic raster description in computer 
sciences and direction doesn't matter... but users do forget fast...

I need homogeneity in visualization for the two communities could talk 
together... :)   

> - Implementing the BSCALE and BZERO via GetOffset() and GetScale() sounds
> like a good idea, in which case, it's fine to exclude those keywords from
> the metadata.

Yep, thanks Even for suggestion.

Chiara

-- 
Chiara Marmo
Ingénieur de Recherche GEOPS - Paris Sud-11
Bât 509
Tel: +33 (0)1 69 15 49 03 
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

[gdal-dev] Fwd: Re: FITS driver update?

[Even Rouault]

Forwarded on behalf of Simon

--  Message transmis  --

Sujet : Re: [gdal-dev] FITS driver update?
Date : vendredi 25 mars 2016, 17:29:27
De : Simon Perkins 
À : Even Rouault , gdal-dev@lists.osgeo.org
CC : Chiara Marmo 

It's been well over a decade since I last looked at this code, and I
originally wrote it for a non-astronomy use-case (we just needed a format
that could handle multi-spectral imagery), so it's entirely possible it
doesn't quite align with normal astronomy conventions... My main concern
though is that making these changes would likely break existing code that
uses this library. Maybe you could add a param to enable a "flip" mode that
flips the y-axis while keeping the current default behavior?

Some random comments:

- I tend to think of image origin as a display issue. It's not clear to me
that GDAL implies any particular origin for its coordinate system, so it
seems simpler to me to have the first row of data in the FITS file
correspond to the first row of data from GDAL's POV as well. For that
matter, it's not clear to me that FITS implies a specific origin -
displaying the first pixel in the lower left is more of a convention? It's
true that converting to a TIFF and then displaying the TIFF in the usual
way flips the image, but I'd probably tackle that by simply flipping the
TIFF image vertically after conversion rather than messing around with the
data ordering. Just my 2c.

- Implementing the BSCALE and BZERO via GetOffset() and GetScale() sounds
like a good idea, in which case, it's fine to exclude those keywords from
the metadata.

- Other changes seem fine since they just add functionality.

Sy

On Fri, Mar 25, 2016 at 3:09 AM Even Rouault 
wrote:

> Bonjour Chiara,
>
> Not being a specialist of the format, I've no in-depth opinion on the
> changes.
> I'm mostly surprised that the issue with the raster origin didn't surface
> before. Are there references for that ? I've CC'ed Simon Perkins (hoping
> his
> email is still OK) who is the original author, in case he can comment.
>
> Regarding scaling (hoping that we talk about the same concepts), there are
> 2
> choices: either you apply the offset/scale automatically in IReadBlock() /
> IRasterIO() (or the underlying lib does it automatically), or you return
> the
> raw values and implement the GetOffset() and GetScale() method of
> GDALRasterBand (in which case, applying the scale is typically done
> afterwards
> with gdal_translate -unscale )
>
> Best regards,
>
> Even
>
> > Hi gdal-list,
> >
> > I work in a planetary/geology/astronomy lab, and I'm interested in having
> > standard and full implementation of FITS format in GDAL. Geographical
> > metadata are not yet well defined in FITS, then in the first time I'm
> just
> > verifying that GDAL works as expected (by astronomers :) ) with FITS
> > format.
> >
> > I have forked the OsGeo project in https://github.com/epn-vespa/gdal/
> and
> > addressed three points: 1) FITS origin is defined on the left bottom
> > corner of a raster, this is the expected behaviour of FITS processing
> > software: I have modified the read and write block direction. 2) Unsigned
> > INT 16 and 32 bit are supported by FITS via specific BZERO value
> > (https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node23.html
> ):
> > I have added USHORT and ULONG support. 3) BZERO BSCALE are 'reserved
> > keywords' and copying them in the metadata produce inconsistency in
> > reverse conversion: a FITS containing BZERO BSCALE is converted to tiff,
> > tiff reconverted to FITS, BZERO BSCALE become strings and FITS
> > visualization ended up in a 0 raster. I have excluded BZERO and BSCALE
> > from metadata keywords.
> >
> > I haven't figured out yet how to manage scaled values... this will take
> > some more time, as geographical conversion.
> >
> > I'm wondering if those small changes could be tested by the community and
> > maybe integrated in the code at some time.
> >
> > All comments are welcome!
> > Thanks for listening,
> > Chiara
>
> --
> Spatialys - Geospatial professional services
> http://www.spatialys.com
>

---
-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GSOC 2016

[sarthak agarwal]

I have submitted the proposal, please check it once and provide your
feedback.

Sarthak

On Fri, Mar 25, 2016 at 2:01 PM, Dmitry Baryshnikov 
wrote:

> Hi Sarthak,
>
> Thank you for you note, but I already wrote:
>
> >Don't wait for anybody with proposal. The new GSoC site is right
> place to discuss proposals.
>
> So I expected to see and comment, if needed, your proposal on this site.
> Let me remind you the site - https://summerofcode.withgoogle.com/
>
> Best regards,
> Dmitry
>
> 25.03.2016 10:17, sarthak agarwal пишет:
>
> The deadline is today.
>
> Sarthak
>
> On Thu, Mar 24, 2016 at 1:52 AM, sarthak agarwal < 
> sarthak0...@gmail.com> wrote:
>
>> Hello Dmitry,
>>
>> I fixed the bug (I guess).
>> Now coming to my proposal for GSoC, So I was thinking of working on
>> project #4 *Auto-detection of EPSG codes from incomplete WKT.*
>>
>> What I understood from the project is that we need to predict the EPSG
>> code of certain files on the basis of some attributes which are available
>> in the file.
>>
>> The attributes can be extracted from the file for which I read this
>> .
>>
>> Now to solve this problem I thought a lot of methods but I think the best
>> way to solve it will be using machine learning.
>>
>> The way ML will handle this problem is as follows-
>>
>>1. We need to find the EPSG code for a file (testing data)
>>2. We have a file with some attributes (projections,datum,etc ).
>>3. We need to the guess the best suitable class for that file(EPSG)
>>4. Also, we have many files for which we know the attributes and the
>>corresponding class (training data).
>>
>> This problem is now translated into an ML problem which can be solved
>> using the following models-
>>
>> 1. Bayesian Stastics
>> 
>>
>> where,
>> posteriror probability = probability of this file have EPSG code 'a'.
>> prior probability = probability of occurence of EPSG code 'a'.
>>
>> likelihood probablity = cases where we saw such attributes when the EPSG
>> code is 'a'.
>>
>>
>> 2. or we can use a simple knn where k is the number of possible EPSG code
>> and the dimension of the feature vector is the number of possible
>> attributes. we need to the find a valid and promising weight function).
>>
>>
>> 3. We can use multi-class SVM.
>>
>> 4. any other suggestion from the community regarding the possible choice
>> of the algo.
>>
>> I am thinking of actually implementing all these algo(may add algo in
>> future depending upon the suggestion) and select the algo which gives the
>> best performance among all of them.
>>
>> Please provide me feedback on my proposal and suggestion if I can
>> add/change anything.
>> And since very less time is left in the deadline, I would like to convert
>> it into proposal ASAP with your help.
>>
>> Regards,
>> Sarthak
>> ​
>>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] FITS driver update?

[Even Rouault]

Bonjour Chiara,

Not being a specialist of the format, I've no in-depth opinion on the changes. 
I'm mostly surprised that the issue with the raster origin didn't surface 
before. Are there references for that ? I've CC'ed Simon Perkins (hoping his 
email is still OK) who is the original author, in case he can comment.

Regarding scaling (hoping that we talk about the same concepts), there are 2 
choices: either you apply the offset/scale automatically in IReadBlock() / 
IRasterIO() (or the underlying lib does it automatically), or you return the 
raw values and implement the GetOffset() and GetScale() method of 
GDALRasterBand (in which case, applying the scale is typically done afterwards 
with gdal_translate -unscale )

Best regards,

Even

> Hi gdal-list,
> 
> I work in a planetary/geology/astronomy lab, and I'm interested in having
> standard and full implementation of FITS format in GDAL. Geographical
> metadata are not yet well defined in FITS, then in the first time I'm just
> verifying that GDAL works as expected (by astronomers :) ) with FITS
> format.
> 
> I have forked the OsGeo project in https://github.com/epn-vespa/gdal/ and
> addressed three points: 1) FITS origin is defined on the left bottom
> corner of a raster, this is the expected behaviour of FITS processing
> software: I have modified the read and write block direction. 2) Unsigned
> INT 16 and 32 bit are supported by FITS via specific BZERO value
> (https://heasarc.gsfc.nasa.gov/docs/software/fitsio/c/c_user/node23.html):
> I have added USHORT and ULONG support. 3) BZERO BSCALE are 'reserved
> keywords' and copying them in the metadata produce inconsistency in
> reverse conversion: a FITS containing BZERO BSCALE is converted to tiff,
> tiff reconverted to FITS, BZERO BSCALE become strings and FITS
> visualization ended up in a 0 raster. I have excluded BZERO and BSCALE
> from metadata keywords.
> 
> I haven't figured out yet how to manage scaled values... this will take
> some more time, as geographical conversion.
> 
> I'm wondering if those small changes could be tested by the community and
> maybe integrated in the code at some time.
> 
> All comments are welcome!
> Thanks for listening,
> Chiara

-- 
Spatialys - Geospatial professional services
http://www.spatialys.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GSOC 2016

[Dmitry Baryshnikov]

Hi Sarthak,

Thank you for you note, but I already wrote:

>Don't wait for anybody with proposal. The new GSoC site is right 
place to discuss proposals.


So I expected to see and comment, if needed, your proposal on this site. 
Let me remind you the site - https://summerofcode.withgoogle.com/


Best regards,
Dmitry

25.03.2016 10:17, sarthak agarwal пишет:

The deadline is today.

Sarthak

On Thu, Mar 24, 2016 at 1:52 AM, sarthak agarwal 
> wrote:


Hello Dmitry,

I fixed the bug (I guess).
Now coming to my proposal for GSoC, So I was thinking of working
on project #4 *Auto-detection of EPSG codes from incomplete WKT.*

What I understood from the project is that we need to predict the
EPSG code of certain files on the basis of some attributes which
are available in the file.

The attributes can be extracted from the file for which I read
this
.

Now to solve this problem I thought a lot of methods but I think
the best way to solve it will be using machine learning.

The way ML will handle this problem is as follows-

 1. We need to find the EPSG code for a file (testing data)
 2. We have a file with some attributes (projections,datum,etc ).
 3. We need to the guess the best suitable class for that file(EPSG)
 4. Also, we have many files for which we know the attributes and
the corresponding class (training data).

This problem is now translated into an ML problem which can be
solved using the following models-

1. Bayesian Stastics


where,
posteriror probability = probability of this file have EPSG
code 'a'.
prior probability = probability of occurence of EPSG code 'a'.

likelihood probablity = cases where we saw such attributes
when the EPSG code is 'a'.


2. or we can use a simple knn where k is the number of possible
EPSG code and the dimension of the feature vector is the number of
possible attributes. we need to the find a valid and promising
weight function).


3. We can use multi-class SVM.

4. any other suggestion from the community regarding the possible
choice of the algo.

I am thinking of actually implementing all these algo(may add algo
in future depending upon the suggestion) and select the algo which
gives the best performance among all of them.

Please provide me feedback on my proposal and suggestion if I can
add/change anything.
And since very less time is left in the deadline, I would like to
convert it into proposal ASAP with your help.

Regards,
Sarthak

​

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
http://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] GSOC 2016

[sarthak agarwal]

The deadline is today.

Sarthak

On Thu, Mar 24, 2016 at 1:52 AM, sarthak agarwal 
wrote:

> Hello Dmitry,
>
> I fixed the bug (I guess).
> Now coming to my proposal for GSoC, So I was thinking of working on
> project #4 *Auto-detection of EPSG codes from incomplete WKT.*
>
> What I understood from the project is that we need to predict the EPSG
> code of certain files on the basis of some attributes which are available
> in the file.
>
> The attributes can be extracted from the file for which I read this
> .
>
> Now to solve this problem I thought a lot of methods but I think the best
> way to solve it will be using machine learning.
>
> The way ML will handle this problem is as follows-
>
>1. We need to find the EPSG code for a file (testing data)
>2. We have a file with some attributes (projections,datum,etc ).
>3. We need to the guess the best suitable class for that file(EPSG)
>4. Also, we have many files for which we know the attributes and the
>corresponding class (training data).
>
> This problem is now translated into an ML problem which can be solved
> using the following models-
>
> 1. Bayesian Stastics 
>
> where,
> posteriror probability = probability of this file have EPSG code 'a'.
> prior probability = probability of occurence of EPSG code 'a'.
>
> likelihood probablity = cases where we saw such attributes when the EPSG
> code is 'a'.
>
>
> 2. or we can use a simple knn where k is the number of possible EPSG code
> and the dimension of the feature vector is the number of possible
> attributes. we need to the find a valid and promising weight function).
>
>
> 3. We can use multi-class SVM.
>
> 4. any other suggestion from the community regarding the possible choice
> of the algo.
>
> I am thinking of actually implementing all these algo(may add algo in
> future depending upon the suggestion) and select the algo which gives the
> best performance among all of them.
>
> Please provide me feedback on my proposal and suggestion if I can
> add/change anything.
> And since very less time is left in the deadline, I would like to convert
> it into proposal ASAP with your help.
>
> Regards,
> Sarthak
> ​
>
> On Tue, Mar 22, 2016 at 8:09 PM, sarthak agarwal 
> wrote:
>
>> Hello Dmitry,
>>
>> I have made all the necessary changes and now the code is working as
>> expected ie If we provide no dbname, it will take the username by default.
>>
>> Here is the
>> https://github.com/OSGeo/gdal/commit/417f4ed2642c56729f93fdb959e2bf1b9f1fdfb1
>>  to
>> the fix.
>>
>> Regards,
>> Sarthak
>> ​
>>
>> On Tue, Mar 22, 2016 at 1:49 AM, Dmitry Baryshnikov > > wrote:
>>
>>> Hi Sarthak,
>>>
>>> The problem is here
>>> https://github.com/sarthak-0415/gdal/blob/trunk/gdal/frmts/postgisraster/postgisrasterdriver.cpp#L78
>>>
>>> Your code produce situation there pszDbnameIn can be NULL. Before your
>>> fixes, that cannot be.
>>> So the line "CPLString osKey = pszDbnameIn;" produces crash.
>>>
>>> Don't wait for anybody with proposal. The new GSoC site is right place
>>> to discuss proposals.
>>>
>>> Best regards,
>>> Dmitry
>>>
>>> 20.03.2016 19:41, sarthak agarwal пишет:
>>>
>>> Hello to all,
>>>
>>> Sorry for taking too long (exams and travelling).
>>>
>>> After running few tests -
>>>
>>>- In my opinion, in Both the codes the error is not in
>>>GetConnectionInfo function. if you replace return true with return
>>>false at the last of the function it won’t fail in both cases.
>>>- If you run this
>>>
>>> 
>>>and this
>>>
>>> 
>>>code, it won’t give you the error, the value passed here are correct 
>>> (NULL
>>>if that’s the case with *ppszDbname, the error is somewhere else and
>>>I am not able to debug it.)
>>>- When I am trying to configure the gdal with enable-debug options
>>>following error comes
>>>
>>>make[1]: *** [gdalserver] Error 1
>>>make[1]: *** Waiting for unfinished jobs
>>>/home/sarthak/gsoc2016/repos/gdal/gdal/.libs/libgdal.so: undefined 
>>> reference to `CPLMutexHolder::CPLMutexHolder(_CPLMutex**, double, char 
>>> const*, int, int)'
>>>collect2: error: ld returned 1 exit status
>>>
>>>
>>> My current config options are
>>> ./configure --prefix=/home/sarthak/gsoc2016/repos/gdal/install/
>>> --with-python=yes -enabl-debug=yes
>>>
>>> the following error is for gdalServer gdalInfo gdal_translate and
>>> gdaladdo. All are having the same problem with undefined reference to
>>> `CPLMutexHolder::CPLMutexHolder(_CPLMutex**, double, char const*, int, int)'
>>>
>>> Please review the code and send me the feedback.
>>> Also, I would like to continue with the bug along