Dear Roger, Let me know if I can help you in any way.
Best Gilberto > On 28 Nov 2020, at 18:08, Roger Bivand <roger.biv...@nhh.no> wrote: > > For completeness, I've put up a skeleton package on > https://github.com/rsbivand/checkGDALDrivers > <https://github.com/rsbivand/checkGDALDrivers>, with Github Actions to check > the package on macOS-latest as well as ubuntu-20.04 (latest and devel) and > Windows-latest. The MacOS platform opens a GTiff file, but does not open a > JP2 file (using rgdal::GDAL.open()): > > https://github.com/rsbivand/checkGDALDrivers/runs/1467699832 > <https://github.com/rsbivand/checkGDALDrivers/runs/1467699832> > > I've replicated this on R-hub for MacOS using CRAN and brew (the GDAL from > brew in that workflow also misses openjpeg it seems): > > CRAN: > https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-64d505798f4b251d1c801ea5dbe6aa33/ > > <https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-64d505798f4b251d1c801ea5dbe6aa33/> > brew > https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-5dfcf72469d1effc742710d53d167caf/ > > <https://artifacts.r-hub.io/checkGDALDrivers_0.0-1.tar.gz-5dfcf72469d1effc742710d53d167caf/> > > On Winbuilder the GA results are replicated (success with both GTiff and JP2 > on all platforms - there is no guarantee that Debian or Fedora packaged GDAL > will include this or any other optional driver). > > The skeleton package might easily be forked, and could also simply check the > driver listings rather than actually trying to read a GTiff and a JP2 file > (PRs very welcome). > > So we have now a two-threaded way to check for JP2 absence, using this > package's GA, and through R-hub. > > Next step to work out how to get openjpeg into the recipes listings so that > GDAL there is built using it. > > On Sat, 28 Nov 2020, Gilberto Camara wrote: > >> Dear Roger >> >> Many thanks for your patience and your guidance. >> >> I installed the rgdal binary from CRAN and it gives errors in both commands, >> as follows: >> >>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)] >> character(0) >> > > Sorry, should have been: > > rgdal::rgdal_extSoftVersion() > > Best wishes, > > Roger > > >>> rgdal::extSoftVersion() >> >> Error: 'extSoftVersion' is not an exported object from 'namespace:rgdal’ >> >> After installing rgdal from source the first command works: >> >>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)] >> “JP2OpenJPEG" >> >> but the second does not: >> >>> rgdal::extSoftVersion() >> >> Error: 'extSoftVersion' is not an exported object from 'namespace:rgdal’ >> >> I am using R version 4.0.3 Patched (2020-11-25 r79505), the latest one >> available on CRAN. The rgdal version is 1.5-18 (SVN revision 1082). >> >> More details on rgdal: >> >>> library(rgdal) >> Loading required package: sp >> rgdal: version: 1.5-18, (SVN revision 1082) >> Geospatial Data Abstraction Library extensions to R successfully loaded >> Loaded GDAL runtime: GDAL 3.2.0, released 2020/10/26 >> Path to GDAL shared files: >> /Users/gilbertocamara/Library/R/4.0/library/sf/gdal >> GDAL binary built with GEOS: TRUE >> Loaded PROJ runtime: Rel. 7.1.1, September 1st, 2020, [PJ_VERSION: 711] >> Path to PROJ shared files: /Users/gilbertocamara/Library/Application >> Support/proj:/usr/local/share/proj:/usr/local/share/proj >> PROJ CDN enabled: FALSE >> Linking to sp version:1.4-4 >> >> As for sf, it also has to be installed from source. The binary version of sf >> also fails to recognise the JP2 driver, while the one built from source does. >> >>> sf::sf_extSoftVersion() works and gives: >> GEOS GDAL proj.4 GDAL_with_GEOS USE_PROJ_H >> "3.8.1" "3.2.0" "7.1.1" "true" "true” >> >> So, it appears that the automated recipes system missed the openjpeg library. >> >> Best regards >> Gilberto >> =========================== >> Prof Dr Gilberto Camara >> Secretariat Director >> GEO - Group on Earth Observations >> 7 bis, Avenue de La Paix >> CH-1211 Geneva - Switzerland >> Tel: +41227308480 >> Web: www.earthobservations.org >> >> >>> On 27 Nov 2020, at 22:23, Roger Bivand <roger.biv...@nhh.no> wrote: >>> >>> Dear Gilberto, >>> >>> The easiest way to deploy the JP2OpenJPEG driver is to check whether it is >>> in the CRAN MacOS rgdal or sf binary packages, as I described before. If it >>> is not there, then we need to post an issue or PR on >>> https://github.com/R-macos/recipes <https://github.com/R-macos/recipes> >>> <https://github.com/R-macos/recipes <https://github.com/R-macos/recipes>>, >>> I think. >>> >>> Could any MacOS user with rgdal installed binary from CRAN please run: >>> >>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)] >>> rgdal::extSoftVersion() >>> >>> and the R and rgdal versions? The equivalent for sf is: >>> >>> drvs <- sf::st_drivers("raster")$name; drvs[grep("JP2", drvs)] >>> sf::sf_extSoftVersion() >>> >>> This is just to check whether the recent change to the automated recipes >>> system perhaps missed the openjpeg library which is used automatically if >>> present. >>> >>> If it went missing, we should be able to reinstate it (or if never >>> available - provide it - the CRAN Windows binary does have this driver), >>> then all users of CRAM MacOS binary packages built against GDAL will be >>> able to use the driver without having to build from source. >>> >>> Best wishes, >>> >>> Roger >>> >>> On Fri, 27 Nov 2020, Gilberto Camara wrote: >>> >>>> Dear Roger >>>> >>>> Many thanks for your support. I built GDAL from scratch and now it >>>> supports the JPEG-2000 driver using the “openjpeg” library. The reported >>>> problems are now solved. >>>> >>>> The whole exercise gave me some lessons on RGDAL and GDAL for users of >>>> MacOS BigSur: >>>> >>>> 1. The usual ways of installing GDAL in macOS (using Homebrew or the >>>> Frameworks provided by KingChaos) do not provide support for >>>> JPEG-2000. So, one has to install GDAL and RGDAL from source. >>>> >>>> 2. One has to use Apple’s C/C++ clang compiler version 12.0, following >>>> the advice given in Stack Overflow: >>>> >>>> https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target >>>> >>>> <https://stackoverflow.com/questions/63972113/big-sur-clang-invalid-version-error-due-to-macosx-deployment-target> >>>> >>>> 3. For GDAL to compile properly, it is necessary to install first >>>> “pkg-config”, and then the libraries “proj”, “geo”, “tiff”, “jpeg”, >>>> “openjpeg”, and “zstd”. >>>> >>>> Thus, if other users of “gdal” find problems with BigSur, I can volunteer >>>> to support them. >>>> >>>> Best regards >>>> Gilberto >>>> >>>> =========================== >>>> Prof Dr Gilberto Camara >>>> Secretariat Director >>>> GEO - Group on Earth Observations >>>> 7 bis, Avenue de La Paix >>>> CH-1211 Geneva - Switzerland >>>> Tel: +41227308480 >>>> Web: www.earthobservations.org >>>> >>>> >>>>> On 26 Nov 2020, at 15:40, Roger Bivand <roger.biv...@nhh.no> wrote: >>>>> >>>>> Dear Gilberto, >>>>> >>>>> On Thu, 26 Nov 2020, GilbertoCamara wrote: >>>>> >>>>>> Dear R-SIG-GEO and Roger >>>>>> >>>>>> I am developing an R package (https://github.com/e-sensing/sits >>>>>> <https://github.com/e-sensing/sits>) to analyse satellite image time >>>>>> series (SITS) which is heavily dependent on “rgdal”. >>>>>> >>>>>> I would like to report an issue with rgdal in the new macOS BigSur. >>>>>> While I agree with Roger that it is not wise to install a new OS until >>>>>> it is stable, I hope the information below will be useful. >>>>>> >>>>>> For starters, let me praise Roger, Edzer and all those involved in >>>>>> “rgdal”. I did some debugging on the source code and was humbled by the >>>>>> effort involved in making the code an operational software many of us >>>>>> rely on. Chapeau bas! >>>>>> >>>>>> That said, please allow me to report on three issues: >>>>>> >>>>>> 1. Installation >>>>>> >>>>>> I have installed GDAL-3.2.0 using Homebrew on BigSur; it works fine. >>>>>> When installing the binary version of rgdal-1.5-18 (SVN revision 1082) >>>>>> from CRAN, recently released by Simon Urbanek, “rgdal" reports that is >>>>>> is using GDAL-3.1.1, which is not installed. The actual message is: >>>>>> "Loaded GDAL runtime: GDAL 3.1.1, released 2020/06/22”. >>>>>> >>>>>> When installing from source, rgdal recognises GDAL-3.2.0, if the >>>>>> Homebrew libraries are on the path. The message changes to: "Loaded GDAL >>>>>> runtime: GDAL 3.2.0, released 2020/10/26." >>>>>> >>>>>> It is odd that the binary version would report using a runtime that is >>>>>> not there. >>>>> >>>>> On MacOS and Windows, CRAN binary packages are built using static >>>>> linking, for a number of reasons. One is to limit installation >>>>> difficulties, another to make support easier, as all users of a updated >>>>> package/R version will be using the same set of external software >>>>> components (of course, locales will vary, but most variables are fixed). >>>>> >>>>>> >>>>>> 2. Errors in /vsis3 file access >>>>>> >>>>>> In earlier versions of macOS, rgdal had no problems with accessing files >>>>>> in AWS with commands such as: >>>>>> >>>>>>> s3_file <- >>>>>>> "/vsis3/sentinel-s2-l2a/tiles/20/L/KP/2018/8/17/0/R20m/B02.jp2” >>>>>>> rgdal::GDALinfo(s3_file) >>>>>> >>>>>> In BigSur, it produces an error. Debugging the rgdal code, I found the >>>>>> error to arise from line 723 of the “gdal-bindings.cpp” file. This is >>>>>> one of the places where rgdal calls the GDAL function “GDALOpenEx”, as >>>>>> follows: >>>>> >>>>> Please first check on running systems with pre-Big Sur and Big Sur that >>>>> the returned lists of drivers match. Crucially, jp2 (Jpeg 2000) needs >>>>> GDAL built against openjpeg. If one of the systems has GDAL built with >>>>> the driver, and the other without, this outcome would be expected. I >>>>> suspect that the Big Sur GDAL (or the CRAN MacOS rgdal binary) has >>>>> missing/defective drivers. >>>>> >>>>> For my Fedora 33 system with GDAL installed from source, I have: >>>>> >>>>> drvs <- rgdal::gdalDrivers()$name; drvs[grep("JP2", drvs)] >>>>> ## [1] "JP2OpenJPEG" >>>>> >>>>> There are other JP2* drivers, but they need closed source libraries when >>>>> GDAL itself is built, and cannot be redistributed from CRAN. >>>>> >>>>>> >>>>>> GDALDataset *pDataset = (GDALDataset *) GDALOpenEx(fn, RWFlag, >>>>>> papszAllowedDrivers, papszOpenOptions, NULL); >>>>>> >>>>>> The function call returns a NULL pointer. Since calling gdalinfo >>>>>> externally or using gdalUtils::gdalinfo works, it seems there is a >>>>>> problem with the return parameters which are provided to rgdal by GDAL. >>>>>> >>>>>> To reproduce the issue, one needs AWS credentials. If Roger and/or Edzer >>>>>> wish to have a go at this issue, I can provide a temporary credential >>>>>> for doing so. >>>>> >>>>> First, we need to be sure that the driver is present, then present and >>>>> working. >>>>> >>>>>> >>>>>> 3. Errors in TIFF Decoding >>>>>> >>>>>> Running devtools::check() produces a strange error on TIFF decoding on a >>>>>> very small file. The error is odd: >>>>>> >>>>>> "TIFFReadDirectory: Warning, Unknown field with tag 42112 (0xa480) >>>>>> encountered." >>>>>> >>>>>> The call stack is as follows: >>>>>> >>>>>> 1: .Call("RGDAL_OpenDataset", .normalize_if_path(filename, mustWork = >>>>>> NA), TRUE, silent, allowedDrivers, options, PACKAGE = "rgdal") >>>>>> 2: .local(.Object, ...) >>>>>> 3: initialize(value, ...) >>>>>> 4: initialize(value, ...) >>>>>> 5: new("GDALReadOnlyDataset", filename, silent = silent, allowedDrivers >>>>>> = allowedDrivers, options = options) >>>>>> 6: GDAL.open(fname, silent = silent, allowedDrivers = allowedDrivers, >>>>>> options = options) >>>>>> 7: rgdal::GDALinfo(file, silent = FALSE) >>>>>> >>>>>> The error appears to be in the same place as the one reported above. >>>>>> However, it is hard to reproduce, because when running on the console, >>>>>> rgdal works. >>>>> >>>>> This feels again like a driver mismatch. Please try with the CRAN MacOS >>>>> binary package, for which >>>>> https://cran.r-project.org/web/checks/check_results_rgdal.html looks >>>>> clean. The examples for rgdal::readGDAL() include reading TIFF files, so >>>>> are run in CRAN daily checks. >>>>> >>>>> I have no access to any MacOS platform, so cannot do more than arm's >>>>> length debug. My clear suspicion is that the GDAL you are using has >>>>> drivers missing/defective. A small JP2 raster could be added to the >>>>> examples to trap driver malfunction. >>>>> >>>>> Best wishes, >>>>> >>>>> Roger >>>>> >>>>>> >>>>>>> ndvi_file <- >>>>>>> c(system.file("extdata/raster/mod13q1/sinop-ndvi-2014.tif", package = >>>>>>> "sits”)) >>>>>>> rgdal::GDALinfo(ndvi_file, silent = FALSE) >>>>>> >>>>>> However, given that the place on the “rgdal” code where the error occurs >>>>>> is the same, it might be worthwhile to look into the RGDAL-GDAL >>>>>> interface in more detail. >>>>>> >>>>>> I know it’s my responsibility to have moved to macOS BigSur too soon. >>>>>> Thus, I do not expect that you address the issue. That said, any help >>>>>> would be much appreciated. >>>>>> >>>>>> Best regards >>>>>> Gilberto >>>>>> =========================== >>>>>> Prof Dr Gilberto Camara >>>>>> Secretariat Director >>>>>> GEO - Group on Earth Observations >>>>>> 7 bis, Avenue de La Paix >>>>>> CH-1211 Geneva - Switzerland >>>>>> Tel: +41227308480 >>>>>> Web: www.earthobservations.org <http://www.earthobservations.org/> >>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> -- >>>>> Roger Bivand >>>>> Department of Economics, Norwegian School of Economics, >>>>> Helleveien 30, N-5045 Bergen, Norway. >>>>> voice: +47 55 95 93 55; e-mail: roger.biv...@nhh.no >>>>> https://orcid.org/0000-0003-2392-6140 >>>>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en >>>> >>>> >>> >>> -- >>> Roger Bivand >>> Department of Economics, Norwegian School of Economics, >>> Helleveien 30, N-5045 Bergen, Norway. >>> voice: +47 55 95 93 55; e-mail: roger.biv...@nhh.no >>> <mailto:roger.biv...@nhh.no> <mailto:roger.biv...@nhh.no >>> <mailto:roger.biv...@nhh.no>> >>> https://orcid.org/0000-0003-2392-6140 >>> <https://orcid.org/0000-0003-2392-6140> >>> <https://orcid.org/0000-0003-2392-6140 >>> <https://orcid.org/0000-0003-2392-6140>> >>> https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ >>> >>> <https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________><https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ >>> >>> <https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________>> >>> R-sig-Geo mailing list >>> R-sig-Geo@r-project.org <mailto:R-sig-Geo@r-project.org> >>> <mailto:R-sig-Geo@r-project.org <mailto:R-sig-Geo@r-project.org>> >>> https://stat.ethz.ch/mailman/listinfo/r-sig-geo >>> <https://stat.ethz.ch/mailman/listinfo/r-sig-geo> >>> <https://stat.ethz.ch/mailman/listinfo/r-sig-geo >>> <https://stat.ethz.ch/mailman/listinfo/r-sig-geo>> >> > > -- > Roger Bivand > Department of Economics, Norwegian School of Economics, > Helleveien 30, N-5045 Bergen, Norway. > voice: +47 55 95 93 55; e-mail: roger.biv...@nhh.no > https://orcid.org/0000-0003-2392-6140 > https://scholar.google.no/citations?user=AWeghB0AAAAJ&hl=en_______________________________________________ > R-sig-Geo mailing list > R-sig-Geo@r-project.org > https://stat.ethz.ch/mailman/listinfo/r-sig-geo [[alternative HTML version deleted]] _______________________________________________ R-sig-Geo mailing list R-sig-Geo@r-project.org https://stat.ethz.ch/mailman/listinfo/r-sig-geo
