Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Even Rouault]

Greg,


an I wonder if this file (in some VFS?) has an associated mode?  I would
sort of expect openssl to error if the private key file were readable by
group or other, and I don't understand this filefromstring big.  (I get
the concept; I just have no idea how it works.)


The file is indeed created in a VFS specific to GDAL, but GDAL then 
reads that file, extracts the private key from it and passes it as a 
string to the openssl API. What I notice now is that the test just 
before in that file, test_eedai_3, uses the same key, with a slightly 
different way of specifying credentials but ultimately this goes to the 
same function that invokes openssl API. One difference is that in 
test_eedai_GOOGLE_APPLICATION_CREDENTIALS there are explicit \n at the 
end of each Base64 encoded line.


Could you try removing those explicit \n (I'm really talking about the 2 
characters \ and n, not the line ending)


and/or apply that patch

diff --git a/gdal/port/cpl_google_cloud.cpp b/gdal/port/cpl_google_cloud.cpp
index 7cbe945ffe..6043f363b2 100644
--- a/gdal/port/cpl_google_cloud.cpp
+++ b/gdal/port/cpl_google_cloud.cpp
@@ -476,7 +476,7 @@ bool 
VSIGSHandleHelper::GetConfiguration(CSLConstList papszOptions,

 return false;
 }
 CPLString osPrivateKey = oDoc.GetRoot().GetString("private_key");
-    osPrivateKey.replaceAll("\\n", "\n");
+    osPrivateKey.replaceAll("\\n", "\n").replaceAll("\n\n", "\n");
 CPLString osClientEmail = 
oDoc.GetRoot().GetString("client_email");

 const char* pszScope =
 CSLFetchNameValueDef( papszOptions, "GS_OAUTH2_SCOPE",
@@ -518,7 +518,7 @@ bool 
VSIGSHandleHelper::GetConfiguration(CSLConstList papszOptions,

 CPLFree(pabyBuffer);
 }
 }
-    osPrivateKey.replaceAll("\\n", "\n");
+    osPrivateKey.replaceAll("\\n", "\n").replaceAll("\n\n", "\n");

 CPLString osClientEmail = CSLFetchNameValueDef(papszOptions,
 "GS_OAUTH2_CLIENT_EMAIL",


My suspicion is that some openssl versions might not like 2 consecutive 
line endings.


Hum, I've actually tried that on our CI MacOSX instance and that doesn't 
help, so I'm not optimistic above my above suggestion.


Even


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Greg Troxel]

Even Rouault  writes:

>> FAILED gdrivers/eedai.py::test_eedai_GOOGLE_APPLICATION_CREDENTIALS - assert 
>> ...
> Not sure what happens here. Probably some harden configuration of
> openssl that rejects the private key ? Apparently it fails on Mac too
> from
> https://github.com/OSGeo/gdal/commit/7816b68130cc7a79e5477a47dc6eb0227f2ae47d 

This is pure speculation, but I see in the code:

   gdal.FileFromMemBuffer('/vsimem/my.json', """{
  "private_key":"-BEGIN PRIVATE KEY-

an I wonder if this file (in some VFS?) has an associated mode?  I would
sort of expect openssl to error if the private key file were readable by
group or other, and I don't understand this filefromstring big.  (I get
the concept; I just have no idea how it works.)

This may or may not be a more detailed run log:


= test session starts ==
platform netbsd9 -- Python 3.8.11, pytest-6.2.4, py-1.10.0, pluggy-0.13.1
rootdir: /home/n0/gdt/SOFTWARE/GEO/GDAL/gdal/autotest, configfile: pytest.ini
plugins: sugar-0.9.4, env-0.6.2
collected 9 items

gdrivers/eedai.py ...Fs...s  [100%]

=== FAILURES ===
__ test_eedai_GOOGLE_APPLICATION_CREDENTIALS ___

def test_eedai_GOOGLE_APPLICATION_CREDENTIALS():

if gdaltest.eedai_drv is None:
pytest.skip()

gdal.FileFromMemBuffer('/vsimem/my.json', """{
"private_key":"-BEGIN PRIVATE KEY-
MIICeAIBADANBgkqhkiG9w0BAQEFAASCAmIwggJeAgEAAoGBAOlwJQLLDG1HeLrk\n
VNcFR5Qptto/rJE5emRuy0YmkVINT4uHb1be7OOo44C2Ev8QPVtNHHS2XwCY5gTm\n
i2RfIBLv+VDMoVQPqqE0LHb0WeqGmM5V1tHbmVnIkCcKMn3HpK30grccuBc472LQ\n
DVkkGqIiGu0qLAQ89JP/r0LWWySRAgMBAAECgYAWjsS00WRBByAOh1P/dz4kfidy\n
TabiXbiLDf3MqJtwX2Lpa8wBjAc+NKrPXEjXpv0W3ou6Z4kkqKHJpXGg4GRb4N5I\n
2FA+7T1lA0FCXa7dT2jvgJLgpBepJu5b//tqFqORb4A4gMZw0CiPN3sUsWsSw5Hd\n
DrRXwp6sarzG77kvZQJBAPgysAmmXIIp9j1hrFSkctk4GPkOzZ3bxKt2Nl4GFrb+\n
bpKSon6OIhP1edrxTz1SMD1k5FiAAVUrMDKSarbh5osCQQDwxq4Tvf/HiYz79JBg\n
Wz5D51ySkbg01dOVgFW3eaYAdB6ta/o4vpHhnbrfl6VO9oUb3QR4hcrruwnDHsw3\n
4mDTAkEA9FPZjbZSTOSH/cbgAXbdhE4/7zWOXj7Q7UVyob52r+/p46osAk9i5qj5\n
Kvnv2lrFGDrwutpP9YqNaMtP9/aLnwJBALLWf9n+GAv3qRZD0zEe1KLPKD1dqvrj\n
j+LNjd1Xp+tSVK7vMs4PDoAMDg+hrZF3HetSQM3cYpqxNFEPgRRJOy0CQQDQlZHI\n
yzpSgEiyx8O3EK1iTidvnLXbtWabvjZFfIE/0OhfBmN225MtKG3YLV2HoUvpajLq\n
gwE6fxOLyJDxuWRf\n
-END PRIVATE KEY-",
"client_email":"m...@email.com"
}""")

gdal.SetConfigOption('EEDA_URL', '/vsimem/ee/')
gdal.SetConfigOption('GOOGLE_APPLICATION_CREDENTIALS', 
'/vsimem/my.json')
gdal.SetConfigOption('EEDA_PRIVATE_KEY', None)
gdal.SetConfigOption('EEDA_CLIENT_EMAIL', None)
gdal.SetConfigOption('GO2A_AUD', '/vsimem/oauth2/v4/token')
gdal.SetConfigOption('GOA2_NOW', '123456')

gdal.FileFromMemBuffer('/vsimem/oauth2/v4/token=grant_type=urn%3Aietf%3Aparams%3Aoauth%3Agrant-type%3Ajwt-bearer=eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiAibXlAZW1haWwuY29tIiwgInNjb3BlIjogImh0dHBzOi8vd3d3Lmdvb2dsZWFwaXMuY29tL2F1dGgvZWFydGhlbmdpbmUucmVhZG9ubHkiLCAiYXVkIjogIi92c2ltZW0vb2F1dGgyL3Y0L3Rva2VuIiwgImlhdCI6IDEyMzQ1NiwgImV4cCI6IDEyNzA1Nn0%3D.1W564xcQESVsqZmBEIMzj4rr0RuGa4RiUPZp5H%2FNENN9V9oPSTdacw%2BMiu3pcFf9AJv8wj0ajUeRsgTmvSicAftER49xeCQYUrs6uV122FGVsxml26kMFacNsCgRad%2Fy7xCAhMPfRJsqxS2%2BB392ssBeEzTGCSI6W3AsJg64OfA%3D',
   '{ "access_token": "my_token", "token_type": 
"Bearer", "expires_in": 3600 }')

ds = gdal.Open('EEDAI:image')

gdal.Unlink('/vsimem/my.json')

gdal.SetConfigOption('EEDA_URL', None)
gdal.SetConfigOption('GOOGLE_APPLICATION_CREDENTIALS', None)
gdal.SetConfigOption('EEDA_PRIVATE_KEY', None)
gdal.SetConfigOption('EEDA_CLIENT_EMAIL', None)

if gdal.GetLastErrorMsg().find('CPLRSASHA256Sign() not implemented') >= 
0:
pytest.skip()

if ds is None and gdaltest.is_github_workflow_mac():
print(gdal.GetLastErrorMsg())
pytest.xfail('Failure. See 
https://github.com/rouault/gdal/runs/1329425333?check_suite_focus=true')

>   assert ds is not None
E   assert None is not None

/home/n0/gdt/SOFTWARE/GEO/GDAL/gdal/autotest/gdrivers/eedai.py:439: 
AssertionError
- Captured stderr call -
ERROR 1: PEM_read_bio_PrivateKey() failed
=== short test summary info 
FAILED gdrivers/eedai.py::test_eedai_GOOGLE_APPLICATION_CREDENTIALS - assert ...
 1 failed, 6 passed, 2 skipped in 0.59s 


signature.asc
Description: PGP signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Even Rouault]

Greg,

FAILED ogr/ogr_gmt.py::test_ogr_gmt_2 - AssertionError: assert 1 == 0

Hopefully https://github.com/OSGeo/gdal/pull/4424 might help

FAILED gcore/cog.py::test_cog_small_world_to_web_mercator - assert [26228, 22...
Was fixed in master 3.4.0 dev per 
https://github.com/OSGeo/gdal/commit/7816b68130cc7a79e5477a47dc6eb0227f2ae47d

FAILED gdrivers/aigrid.py::test_aigrid_broken - FileExistsError: [Errno 17] F...
Probably result of a previous failure in that test. Hopefully 
https://github.com/OSGeo/gdal/pull/4424 might help

FAILED gdrivers/eedai.py::test_eedai_GOOGLE_APPLICATION_CREDENTIALS - assert ...
Not sure what happens here. Probably some harden configuration of 
openssl that rejects the private key ? Apparently it fails on Mac too 
from 
https://github.com/OSGeo/gdal/commit/7816b68130cc7a79e5477a47dc6eb0227f2ae47d 


FAILED alg/warp.py::test_warp_53[cubic-generalCase-Byte] - assert 3965 in [39...
FAILED alg/warp.py::test_warp_53[cubic-generalCase-UInt16] - assert 3965 in [...
FAILED alg/warp.py::test_warp_53[cubic-generalCase-Int16] - assert 3965 in [3...
FAILED alg/warp.py::test_warp_53[cubic-default-Byte] - assert 3965 in [3942, ...
FAILED alg/warp.py::test_warp_53[cubic-default-UInt16] - assert 3965 in [3942...
FAILED alg/warp.py::test_warp_53[cubic-default-Int16] - assert 3965 in [3942,...
There's NAD27 to WGS 84 involved. Actually I see I get your results when 
the HPGN grids are installed (compared to having only 
proj-datumgrid-1.8). I've rewritten the tests to be PROJ independent in 
https://github.com/OSGeo/gdal/pull/4424

FAILED utilities/test_gdalwarp.py::test_gdalwarp_15 - AssertionError: Bad che...
= 12 failed, 7249 passed, 1479 skipped, 2 xfailed, 2 warnings in 713.86s 
(0:11:53) =


Same issue with HPGN grids. PR 4424 updated to remove that test as 
redundant with another one




FAILED osr/osr_metacrs.py::TestMetaCRS::test_metacrs[WGS84 Geogrpahic 2D to 
CSPC Z3 USFT NAD83]
Also I can see this is still related to the HPGN grids. I've completely 
removed that test in the PR as it tests more PROJ behaviour than GDAL one


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Greg Troxel]

Thanks for all the hints so far about tests.  With the fopen/fread fix
(locally applied to 3.3.2rc3), and rtree added in to sqlite3 (which I'm
calling a bug in my world), I am down to:

FAILED ogr/ogr_gmt.py::test_ogr_gmt_2 - AssertionError: assert 1 == 0
FAILED gcore/cog.py::test_cog_small_world_to_web_mercator - assert [26228, 22...
FAILED gdrivers/aigrid.py::test_aigrid_broken - FileExistsError: [Errno 17] F...
FAILED gdrivers/eedai.py::test_eedai_GOOGLE_APPLICATION_CREDENTIALS - assert ...
FAILED osr/osr_metacrs.py::TestMetaCRS::test_metacrs[WGS84 Geogrpahic 2D to 
CSPC Z3 USFT NAD83]
FAILED alg/warp.py::test_warp_53[cubic-generalCase-Byte] - assert 3965 in [39...
FAILED alg/warp.py::test_warp_53[cubic-generalCase-UInt16] - assert 3965 in [...
FAILED alg/warp.py::test_warp_53[cubic-generalCase-Int16] - assert 3965 in [3...
FAILED alg/warp.py::test_warp_53[cubic-default-Byte] - assert 3965 in [3942, ...
FAILED alg/warp.py::test_warp_53[cubic-default-UInt16] - assert 3965 in [3942...
FAILED alg/warp.py::test_warp_53[cubic-default-Int16] - assert 3965 in [3942,...
FAILED utilities/test_gdalwarp.py::test_gdalwarp_15 - AssertionError: Bad che...
= 12 failed, 7249 passed, 1479 skipped, 2 xfailed, 2 warnings in 713.86s 
(0:11:53) =

And now, the ogr_mitab test passes without crashing.  So there's
probably some error checking missing someplace and I'm no longer hitting
the case that triggers it.

Full log (minus ogr_mitab):

  http://www.lexort.com/GDAL/gdal-tests.no-mitab.txt

if anybody wants to look a it.

I will start to dig through these, but it smells like the warp ones are
the same problem, and this is now down to a non-scary number.  A few may
be due to proj 6.3.2 (with grids installed), but I haven't figured that
out yet.



signature.asc
Description: PGP signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Greg Troxel]

Even Rouault  writes:

>> What's wrong with just always doing the stat?
> Nothing, but just one extra system call in a performance critical path.

OK - but do you really think that in an entire sequence of
fopen/fread/prcoess-data that one more stat (which will be cached w.r.t
the inode read in open) is going to even be measurable?  (No, I haven't
benchmarked it either.)   But I'm not the one maintaining the more
complicated code...



signature.asc
Description: PGP signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Even Rouault]

What's wrong with just always doing the stat?

Nothing, but just one extra system call in a performance critical path.

Or, if it's really important to avoid the stat, put the code that makes
beyond-POSIX assumptions under #ifdef linux.   Then we'd have code that
is in general correct, and avoids a stat on systems known to not return
any bytes on a directory.

I basically whitelisted OSs that are tested by our CI and for which 
fread() on a directory FILE* returns 0 and where we can save the stat() 
when it is a non-empty file, which is the nominal case for most GDAL usage.


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Greg Troxel]

Even Rouault  writes:

>> As I read the spec, it is a violation to return NULL when the first
>> argument is a directory and the second is r or rb.  A test program
>> succeeds in calling fopen on . with rb, on both NetBSD and macOS 10.13.
>
> It is not clear for me. There's a mention that opening a directory in
> update mode should fail.

Directories are not writable as files, even if they have the w bit.
That's been true since at least Sixth Edition.  To me, that text
prohibiting opening directories for write is clear evidence of intent
not to prohibit opening them for reading.

> But I can't see the point of succeeding in read-only mode. You can't
> read anything.

Directories have names and inode numbers, in some data structure.  There
is something to read -- it's just something that usually one does not
want to do.

I agree that in the modern world trying to read a directory is not
really a sensible thing to do under almost all circumstances.  But it's
a huge leap from that observation to assuming that attempts must fail.

> But actually I do see that on my current Linux system too
> fopen(somedir, "rb") works, but later in that function, we try to read
> some bytes from it. If we don't manage to read a single byte, then we
> fallback to a stat() to check the nature of the file. So I suspect
> that on your OS and FreeBSD, the fread() does return some bytes,
> whereas the other OS on which we regularly test, return 0.

Interesting that this is even more subtle than I thought.  Still, I
don't think POSIX says that fread on a directory can't return bytes.  In
fact I can't find anything in fread or fgetc that allows it to fail on
directories.  I suppose though that a system can define a directory to
have zero bytes of content when viewed as a file.

On NetBSD, bytes are returned, and I see filenames.  The output from
fread matches "od -c .".

> I guess we might want to change the current test as #if
> !(defined(_WIN32) || defined(__linux__) || defined(__ANDROID__)) so as
> to match all the BSDs, and do the stat() before trying to fopen() the
> file.
>
> Trying that in https://github.com/OSGeo/gdal/pull/4411

I don't understand why there should be an explicit list of systems when
the basic issue is code relying on behavior that is not specified by
POSIX.  From a traditional UNIX perspective, I find fread refusing to
read a directory to be strange -- but at the same time it's a reasonable
choice from an abstraction viewpoint.

It seems far safer to just not make assumptions about what fopen/fread
will do, when one can't say a system that doesn't meet the assumptions
is defective.

What's wrong with just always doing the stat?

Or, if it's really important to avoid the stat, put the code that makes
beyond-POSIX assumptions under #ifdef linux.   Then we'd have code that
is in general correct, and avoids a stat on systems known to not return
any bytes on a directory.

> This will be post 3.3.2 material as it is not a new issue.

Sure, that's fine.


signature.asc
Description: PGP signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Even Rouault]

Le 01/09/2021 à 21:22, Greg Troxel a écrit :

Looking into the avc failure, I find

$ ogrinfo ogr/data/avc/testavc/testavc/
INFO: Open of `ogr/data/avc/testavc/testavc/'
using driver `AVCBin' successful.
1: ARC (Line String)
2: LAB (Point)

But if I leave off the trailing / I get a failure to find a driver.  A
trailing slash on a directrory name seems odd to me, and usually the
result of completion.

Hum, I suspect you might hit a similar issue as the one for FreeBSD in
https://github.com/OSGeo/gdal/blob/a95e796f65b26379b0e5c699bacef29f7684f79f/gdal/gcore/gdalopeninfo.cpp#L216
where fopen("/some/dir", "rb") succeeds.

Can you test changing that with whatever define is appropriate to test
for your OS and submit the resulting patch ?

The comment about FreeBSD being odd seems strange to me, as POSIX has no
notion that fopen must fail when opening a directory for reading.

   https://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html
   https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html

As I read the spec, it is a violation to return NULL when the first
argument is a directory and the second is r or rb.  A test program
succeeds in calling fopen on . with rb, on both NetBSD and macOS 10.13.

Perhaps I am misreading the POSIX spec.


It is not clear for me. There's a mention that opening a directory in 
update mode should fail. But I can't see the point of succeeding in 
read-only mode. You can't read anything.


But actually I do see that on my current Linux system too fopen(somedir, 
"rb") works, but later in that function, we try to read some bytes from 
it. If we don't manage to read a single byte, then we fallback


to a stat() to check the nature of the file. So I suspect that on your 
OS and FreeBSD, the fread() does return some bytes, whereas the other OS 
on which we regularly test, return 0.


I guess we might want to change the current test as #if 
!(defined(_WIN32) || defined(__linux__) || defined(__ANDROID__)) so as 
to match all the BSDs, and do the stat() before trying to fopen() the file.


Trying that in https://github.com/OSGeo/gdal/pull/4411

This will be post 3.3.2 material as it is not a new issue.


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Even Rouault]

Looks like a lot of this the rtree module in sqlite3, which seems to
think telling you to go back and rebuild something with different
non-default options is a reasonable approach :-(


sqlite3 with rtree support is a requirement for GeoPackage and Spatialite.


I had not previously grasped that rtree was required, and configure
seems not to test for it.
No rtree isn't tested by GDAL's configure. Looking at sqlite3 own 
configure, I see rtree is supposed to be built-in by default


 ./configure --help | grep rtree
  --enable-rtree  include rtree support [default=yes]

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Greg Troxel]

>> Looking into the avc failure, I find
>>
>> $ ogrinfo ogr/data/avc/testavc/testavc/
>> INFO: Open of `ogr/data/avc/testavc/testavc/'
>>using driver `AVCBin' successful.
>> 1: ARC (Line String)
>> 2: LAB (Point)
>>
>> But if I leave off the trailing / I get a failure to find a driver.  A
>> trailing slash on a directrory name seems odd to me, and usually the
>> result of completion.
>
> Hum, I suspect you might hit a similar issue as the one for FreeBSD in
> https://github.com/OSGeo/gdal/blob/a95e796f65b26379b0e5c699bacef29f7684f79f/gdal/gcore/gdalopeninfo.cpp#L216
> where fopen("/some/dir", "rb") succeeds.
>
> Can you test changing that with whatever define is appropriate to test
> for your OS and submit the resulting patch ?

The comment about FreeBSD being odd seems strange to me, as POSIX has no
notion that fopen must fail when opening a directory for reading.

  https://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html
  https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html

As I read the spec, it is a violation to return NULL when the first
argument is a directory and the second is r or rb.  A test program
succeeds in calling fopen on . with rb, on both NetBSD and macOS 10.13.

Perhaps I am misreading the POSIX spec.

I applied a patch to always do the directory check, and now ogrinfo
works with or without the trailing /.   Patch looks more complicated
than it is

--- gcore/gdalopeninfo.cpp.orig 2021-09-01 09:48:45.0 +
+++ gcore/gdalopeninfo.cpp
@@ -213,12 +213,8 @@ retry:  // TODO(schwehr): Stop using got
 
 #endif  // HAVE_READLINK
 
-#ifdef __FreeBSD__
-/* FreeBSD 8 oddity: fopen(a_directory, "rb") returns non NULL */
+/* fopen(dir, "r") is required to succeed, per POSIX */
 bool bPotentialDirectory = (eAccess == GA_ReadOnly);
-#else
-bool bPotentialDirectory = false;
-#endif  // __FreeBDS__
 
 /* Check if the filename might be a directory of a special virtual file 
system */
 if( STARTS_WITH(pszFilename, "/vsizip/") ||


A test run with this fix and GDAL_DATA unset got to only 20 failures.

/*
 * https://pubs.opengroup.org/onlinepubs/9699919799/functions/fopen.html
 */

#include 
#include 

int
main()
{
  FILE *fp;

  errno = 0;

  fp = fopen(".", "rb");

  printf("fopen returns %p errno %d\n", fp, errno);

  return 0;
}


signature.asc
Description: PGP signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Greg Troxel]

Even Rouault  writes:

>> and now have a test run that shows:
>>
>>= 312 failed, 6939 passed, 1489 skipped, 2 xfailed, 2 warnings in 746.96s 
>> (0:12:26) =
>>
>> which seems quite good overall.
>
> For a reasonably well setup environment, you should not get more than
> a handful of failures. 312 definitely indicates something
> wrong. Looking a bit at the logs you provide, they are explicit and

I meant that for a system where test issues were not yet debugged it
didn't seem too bad.  But we are rapidly reducing the count: now down to
20.

> implicit complaints about GDAL_DATA not being set. I'm not sure if
> you're running agains an installed GDAL lib (in which case, there's
> something wrong in your build if it can't find /usr/share/gdal/ or
> whatever similar dir in which GDAL data is installed), or if it a
> in-build GDAL (in which case GDAL_DATA env var must be explicitly set
> as done in setdevenv.sh)

I am running against installed gdal, which was built with configure
fairly normally.  There's a lot of stuff in /usr/pkg/share/gdal.  But,
from trying to look at the dev env setup, I had a bad GDAL_DATA
variable, and that was the problem.

> A number of PROJ related failures can be caused by some grids being
> missing. The 'conus' grid from
> http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz (there might
> be others from that package. at least this is the one used for our CI)
> must in particular be installed in /usr/share/proj (or
> 'us_noaa_conus.tif' in PROJ 7 or later)

OK - I have the grids intalled, but it is interesting to note that proj
tests require them not to be installed while gdal tests require them to
be installed.

> It is possible to get a few PROJ related failures from time to time
> given which version you use. Our CI covers a number of versions, but
> not all obviously.

I would hope 6.3.2 is still on the ok list.  Lots of systems are a bit
behind.

> Hum, I suspect you might hit a similar issue as the one for FreeBSD in
> https://github.com/OSGeo/gdal/blob/a95e796f65b26379b0e5c699bacef29f7684f79f/gdal/gcore/gdalopeninfo.cpp#L216
> where fopen("/some/dir", "rb") succeeds.

Separate reply coming about this.  Thanks very much for the pointer,
which saved me a really long time looking into this.



New test log at

http://www.lexort.com/GDAL/

Looks like a lot of this the rtree module in sqlite3, which seems to
think telling you to go back and rebuild something with different
non-default options is a reasonable approach :-(  

I had not previously grasped that rtree was required, and configure
seems not to test for it.   Or perhaps it's not required and the tests
should skip?

But it looks like adjusting sqlite to have rtree will resolve more than
half the remaining errors.

(Thus, the test suite is being highly useful.)


signature.asc
Description: PGP signature
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] test failures on pkgsrc build of 3.3.2rc3

[Even Rouault]

and now have a test run that shows:

   = 312 failed, 6939 passed, 1489 skipped, 2 xfailed, 2 warnings in 746.96s 
(0:12:26) =

which seems quite good overall.


For a reasonably well setup environment, you should not get more than a 
handful of failures. 312 definitely indicates something wrong. Looking a 
bit at the logs you provide, they are explicit and implicit complaints 
about GDAL_DATA not being set. I'm not sure if you're running agains an 
installed GDAL lib (in which case, there's something wrong in your build 
if it can't find /usr/share/gdal/ or whatever similar dir in which GDAL 
data is installed), or if it a in-build GDAL (in which case GDAL_DATA 
env var must be explicitly set as done in setdevenv.sh)


A number of PROJ related failures can be caused by some grids being 
missing. The 'conus' grid from 
http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz (there might be 
others from that package. at least this is the one used for our CI) must 
in particular be installed in /usr/share/proj (or 'us_noaa_conus.tif' in 
PROJ 7 or later)




Note that this my gdal was built with proj 6.3.2, partly old to avoid a
version with removed interfaces that other software needs, and partly
because I haven't gotten to dealing (in packaging) with the new proj
data layout.  As I understand it, though, that's a supported version for
gdal, and the gdal build succeeds.  I suspect if this does matter, it's
only for a few tests.
It is possible to get a few PROJ related failures from time to time 
given which version you use. Our CI covers a number of versions, but not 
all obviously.


pkgsrc build log (showing configue output), driver list (from ogrinfo
failure), and test output at:

   http://www.lexort.com/GDAL/

Looking into the avc failure, I find

$ ogrinfo ogr/data/avc/testavc/testavc/
INFO: Open of `ogr/data/avc/testavc/testavc/'
   using driver `AVCBin' successful.
1: ARC (Line String)
2: LAB (Point)

But if I leave off the trailing / I get a failure to find a driver.  A
trailing slash on a directrory name seems odd to me, and usually the
result of completion.


Hum, I suspect you might hit a similar issue as the one for FreeBSD in 
https://github.com/OSGeo/gdal/blob/a95e796f65b26379b0e5c699bacef29f7684f79f/gdal/gcore/gdalopeninfo.cpp#L216 
where fopen("/some/dir", "rb") succeeds.


Can you test changing that with whatever define is appropriate to test 
for your OS and submit the resulting patch ?


Even

--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test Failures

[Even Rouault]

Do you have any idea why the differences exist? Is it worth investigating?


Quite likely subtle differences in behavior of the JPEG compression 
library. Mine on ubuntu 20.04 is libjpeg-turbo 2.0.3 with IJG libjpeg 8 
ABI. I presume if you built if from source and linked it against GDAL, 
you'd get the same checksums.


We should try to minimize tests using JPEG for that reason, but in that 
instance, this test is really meant to check JPEG specific code paths in 
the COG driver. Or use some kind of perceptualdiff test for such test, 
but the "perceptualdiff" utility does consider "your" and "mine" image 
to be different, so that wouldn't help. We'd probably need something 
clever that tolerates pixel values differences that can be quite 
important (the max here is a difference of 19 units) but detects there's 
no spatial shift.




On Wed, Apr 28, 2021 at 8:35 AM Even Rouault 
mailto:even.roua...@spatialys.com>> wrote:


Andew,

visually the difference between your image and the one generated
on my PC are almost indistinguishable (no spatial shift at least).
Please submit a PR adding your checksums as alternate accepted
ones. You may just remove the "if sys.platform == 'darwin'" case.
Hopefully adding your checksums should fix the OSX CI too


--
Andrew Bell
andrew.bell...@gmail.com 


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test Failures

[Andrew Bell]

Even,

Do you have any idea why the differences exist? Is it worth investigating?

On Wed, Apr 28, 2021 at 8:35 AM Even Rouault 
wrote:

> Andew,
>
> visually the difference between your image and the one generated on my PC
> are almost indistinguishable (no spatial shift at least). Please submit a
> PR adding your checksums as alternate accepted ones. You may just remove
> the "if sys.platform == 'darwin'" case. Hopefully adding your checksums
> should fix the OSX CI too
>

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test Failures

[Even Rouault]

Andew,

visually the difference between your image and the one generated on my 
PC are almost indistinguishable (no spatial shift at least). Please 
submit a PR adding your checksums as alternate accepted ones. You may 
just remove the "if sys.platform == 'darwin'" case. Hopefully adding 
your checksums should fix the OSX CI too


Even

Le 28/04/2021 à 14:20, Andrew Bell a écrit :
Here is the output tif file along with the command and its terminal 
output.


On Tue, Apr 27, 2021 at 3:02 PM Even Rouault 
mailto:even.roua...@spatialys.com>> wrote:



Le 27/04/2021 à 20:49, Andrew Bell a écrit :



On Tue, Apr 27, 2021 at 2:27 PM Even Rouault
mailto:even.roua...@spatialys.com>>
wrote:

Andrew,

This could be due to the PROJ version, but as I'm running
with 8.0 too (and one of the CI config runs PROJ master), I
suspect you just need to install the "conus" grid for NAD27
to WGS84 transformations into {proj_prefix}/share/proj .
Unpack
http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz


I have PROJ_NETWORK set to ON. I was under the impression that
this would cause PROJ to find whatever grids shift files it
needed at runtime. Is there more configuration that's necessary
to get this to happen or do I misunderstand the PROJ_NETWORK setting?


That should be fine. Actually I was wrong. This test case doesn't
involve any datum shift / grids as it is EPSG:4326 to EPSG:3857
reprojection.

That might then just be a subtle different rounding issue in
computations in GDAL and/or PROJ, or more likely, due to using a
different JPEG library as this test case involves JPEG
compression. If that was the case, you might add your checksums as
an alternative expected output. We do that in a number of places.

Looking a few lines before the failure, I actually see that the CI
got that issue and we worked around it with:

    if sys.platform == 'darwin' and gdal.GetConfigOption('TRAVIS',
None) is not None:
    assert got_cs != [0, 0, 0]
    else:

Could you run

gdal_translate ../autotest/gdrivers/data/small_world.tif out.tif
-of COG co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=JPEG

and attach the resulting file ?



Thanks,
-- 
Andrew Bell

andrew.bell...@gmail.com 


-- 
http://www.spatialys.com  

My software is free, but my time generally not.



--
Andrew Bell
andrew.bell...@gmail.com 


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test Failures

[Andrew Bell]

Here is the output tif file along with the command and its terminal output.

On Tue, Apr 27, 2021 at 3:02 PM Even Rouault 
wrote:

>
> Le 27/04/2021 à 20:49, Andrew Bell a écrit :
>
>
>
> On Tue, Apr 27, 2021 at 2:27 PM Even Rouault 
> wrote:
>
>> Andrew,
>>
>> This could be due to the PROJ version, but as I'm running with 8.0 too
>> (and one of the CI config runs PROJ master), I suspect you just need to
>> install the "conus" grid for NAD27 to WGS84 transformations into
>> {proj_prefix}/share/proj . Unpack
>> http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz
>>
> I have PROJ_NETWORK set to ON. I was under the impression that this would
> cause PROJ to find whatever grids shift files it needed at runtime. Is
> there more configuration that's necessary to get this to happen or do I
> misunderstand the PROJ_NETWORK setting?
>
> That should be fine. Actually I was wrong. This test case doesn't involve
> any datum shift / grids as it is EPSG:4326 to EPSG:3857 reprojection.
>
> That might then just be a subtle different rounding issue in computations
> in GDAL and/or PROJ, or more likely, due to using a different JPEG library
> as this test case involves JPEG compression. If that was the case, you
> might add your checksums as an alternative expected output. We do that in a
> number of places.
>
> Looking a few lines before the failure, I actually see that the CI got
> that issue and we worked around it with:
>
> if sys.platform == 'darwin' and gdal.GetConfigOption('TRAVIS', None)
> is not None:
> assert got_cs != [0, 0, 0]
> else:
>
> Could you run
>
> gdal_translate ../autotest/gdrivers/data/small_world.tif out.tif -of COG
> co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=JPEG
>
> and attach the resulting file ?
>
>
> Thanks,
>
> --
> Andrew Bell
> andrew.bell...@gmail.com
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>

-- 
Andrew Bell
andrew.bell...@gmail.com


gdal_translate.tar.gz
Description: GNU Zip compressed data
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test Failures

[Even Rouault]

Le 27/04/2021 à 20:49, Andrew Bell a écrit :



On Tue, Apr 27, 2021 at 2:27 PM Even Rouault 
mailto:even.roua...@spatialys.com>> wrote:


Andrew,

This could be due to the PROJ version, but as I'm running with 8.0
too (and one of the CI config runs PROJ master), I suspect you
just need to install the "conus" grid for NAD27 to WGS84
transformations into {proj_prefix}/share/proj . Unpack
http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz


I have PROJ_NETWORK set to ON. I was under the impression that this 
would cause PROJ to find whatever grids shift files it needed at 
runtime. Is there more configuration that's necessary to get this to 
happen or do I misunderstand the PROJ_NETWORK setting?


That should be fine. Actually I was wrong. This test case doesn't 
involve any datum shift / grids as it is EPSG:4326 to EPSG:3857 
reprojection.


That might then just be a subtle different rounding issue in 
computations in GDAL and/or PROJ, or more likely, due to using a 
different JPEG library as this test case involves JPEG compression. If 
that was the case, you might add your checksums as an alternative 
expected output. We do that in a number of places.


Looking a few lines before the failure, I actually see that the CI got 
that issue and we worked around it with:


    if sys.platform == 'darwin' and gdal.GetConfigOption('TRAVIS', 
None) is not None:

    assert got_cs != [0, 0, 0]
    else:

Could you run

gdal_translate ../autotest/gdrivers/data/small_world.tif out.tif -of COG 
co TILING_SCHEME=GoogleMapsCompatible -co COMPRESS=JPEG


and attach the resulting file ?



Thanks,
--
Andrew Bell
andrew.bell...@gmail.com 


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test Failures

[Andrew Bell]

On Tue, Apr 27, 2021 at 2:27 PM Even Rouault 
wrote:

> Andrew,
>
> This could be due to the PROJ version, but as I'm running with 8.0 too
> (and one of the CI config runs PROJ master), I suspect you just need to
> install the "conus" grid for NAD27 to WGS84 transformations into
> {proj_prefix}/share/proj . Unpack
> http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz
>
I have PROJ_NETWORK set to ON. I was under the impression that this would
cause PROJ to find whatever grids shift files it needed at runtime. Is
there more configuration that's necessary to get this to happen or do I
misunderstand the PROJ_NETWORK setting?

Thanks,

-- 
Andrew Bell
andrew.bell...@gmail.com
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev

Re: [gdal-dev] Test Failures

[Even Rouault]

Andrew,

This could be due to the PROJ version, but as I'm running with 8.0 too 
(and one of the CI config runs PROJ master), I suspect you just need to 
install the "conus" grid for NAD27 to WGS84 transformations into 
{proj_prefix}/share/proj . Unpack 
http://download.osgeo.org/proj/proj-datumgrid-1.8.tar.gz


Even

Le 27/04/2021 à 20:17, Andrew Bell a écrit :

Hi,

I'm having a few test failures with gdal master on OSX. They seem 
related to transformations. I notice that the CI build is using PROJ 
7.1. I'm linking PROJ 8.0. An example failure follows. Can this be 
explained by PROJ versions or are there other issues that I might need 
to investigate?


==

>           assert got_cs == [26293, 23439, 14955]
E           assert [26228, 22085, 12992] == [26293, 23439, 14955]
E             At index 0 diff: 26228 != 26293
E             Use -v to get the full diff

/Users/acbell/gdal/autotest/gcore/cog.py:392: AssertionError
--- Captured stderr call 
---

ERROR 1: PROJ: webmerc: Invalid latitude
ERROR 1: PROJ: webmerc: Invalid latitude
ERROR 1: PROJ: webmerc: Invalid latitude

--
Andrew Bell
andrew.bell...@gmail.com 

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev


--
http://www.spatialys.com
My software is free, but my time generally not.

___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev