Re: [gdal-dev] Building GDAL documentation with ReadTheDocs
> From what I see, robots.txt is automatically configured for the default > version: > https://docs.readthedocs.io/en/stable/reference/robots.html I read this as saying that the robots.txt published by the default branch will be served from the top level of the domain (https://proj.org/robots.txt) and that a robots.txt associated with any other branch will be ignored. If no robots.txt is provided, ReadTheDocs uses a default which allows indexing of all non-hidden branches. PROJ does not appear to have any branches that are hidden with RTD, so all versions >= 5.0 would be indexed. User-agent: * Disallow: # Allow everything Sitemap: https://proj.org/sitemap.xml Dan ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Building GDAL documentation with ReadTheDocs
Thanks, Rob. I've added a robots.txt to the PR (using /en/latest/, since /en/stable/ is not yet published) Dan On Wed, Jul 24, 2024 at 7:44 AM Robert Coup via gdal-dev wrote: > > On Wed, 24 Jul 2024 at 12:27, Even Rouault wrote: > > > > > > Le 24/07/2024 à 12:33, Javier Jimenez Shaw via gdal-dev a écrit : > > > >> I see the point, and I agree... but I don't know if it will work > >> RTD redirects to whatever we configure. In proj.org it is going directly > >> to 9.4 (current latest release). > > > > Seems like a good idea and something that can be easily be tuned by editing > > https://readthedocs.org/dashboard/osgeo-proj/edit/ . But when looking at > > https://proj.org/en/stable/about.html for example, the only thing that is > > missing to me is a hint of what "stable" alias to. So you have on the top > > left, under the PROJ logo, a "stable" text. It would be nice if that would > > be "stable (9.4)". Not sure if that can be tuned. > > > > So every 6 months it points to something different. Telling in robots.txt > > that the full page has a different URL is not good for seo and history (if > > I understand correctly how it works, that I am not sure). > > I think using `/en/stable/` (or /latest) as the default URL redirect is the > thing — as a new release gets promoted, google/etc will keep using/crawling > that, and that link doesn't change. > > I think the robots thing would need something a bit like: > > allow: /en/stable/ > disallow: /en/ > > Though in principle, no-index would be preferable to "do not crawl". Not sure > what levers RTD has in that regard, if /stable/ and /9.4/ are actually > different sets of html files then adding in an HTML tag for noindex > might be simple enough for !=stable ? > > Rob :) > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Building GDAL documentation with ReadTheDocs
> Could a random user easily check
> out the docs repo and (with 100% open source tooling) generate the
> docs?
Yes, we are currently and would continue to use Sphinx for the
documentation build.
Dan
On Tue, Jul 23, 2024 at 1:18 PM Greg Troxel via gdal-dev
wrote:
>
> Daniel Baston via gdal-dev writes:
>
> > Hello,
> >
> > I've put together a configuration to build GDAL's documentation using
> > ReadTheDocs. As far as I know, this is the simplest way of publishing
> > version-specific documentation and rending documentation for pull
> > requests. A current build of the documentation can be viewed at
> > https://gdal.readthedocs.io/en/latest/. A pull request with the
> > proposed configuration is available at
> > https://github.com/OSGeo/gdal/pull/10434.
> >
> > If anyone has feedback about using ReadTheDocs for this project, or
> > the specifics of the proposed configuration, please share it here or
> > as a comment on the pull request.
>
> I agree that version-specific doc publication and docs on PRs is nice.
>
> With things that feel like hosted services, I always want to ask:
>
> 1) Is the toolchain that we will be using 100% open source?
>
> 2) Could we, if readthedocs.io shut down, or if we felt like it, easily
> switch to doing this all on an osgeo website? Could we easily have
> downloads of browseable tarballs? Could a random user easily check
> out the docs repo and (with 100% open source tooling) generate the
> docs?
>
> 3) The website loads ads from "ethicalads.io":
>
> https://media.ethicalads.io/media/client/beta/ethicalads.min.js
>
> which is hard to read. It loads something else that uBlock Origin
> objects to:
>
>
> https://readthedocs.org/api/v2/sustainability/data/?callback=jQuery360005151577479217162_1721754548069=jsonp=gdal&_=1721754548070
>
> which results in
>
>
> jQuery360005151577479217162_1721754548069({"ad_free":false,"campaign_types":["community","house","paid"],"keywords":["only
>
> words","readthedocs-project-1031375","readthedocs-project-gdal"],"publisher":"readthedocs"});
>
> and also loads self-hosted analytics:
>
>
> https://gdal.readthedocs.io/_/api/v2/analytics/?project=gdal=latest_uri=https%3A%2F%2Fgdal.readthedocs.io%2Fen%2Flatest%2F
> https://gdal.readthedocs.io/_/static/javascript/readthedocs-analytics.js
>
> I do not find a doc site privacy policy in a footer. There is no GDPR
> cookie consent -- those can be dreadful, I know -- but I doubt "no
> cookies are set" is really how it is. I have no fundamental problem
> with cookies -- I'm thinking about tracking.
>
> I would say that if 1 or 2 is not ok, then I see this as problematic,
> but I'm hoping they are both fine.
>
> For 3, not sure what the community wants to do, but I'm mildly
> uncomfortable. It seems less bad than using github, but that doesn't
> automatically make it ok :-)
>
> In my view, docs should be part of a package build and installed so they
> are available on the computer with the software. If it's just sphinx,
> that can just be part of the gdal build and installed to
> $prefix/share/doc/gdal/html, optionally off, or buildable separately to
> make the gdal-doc package. This could be the same toolchain as RTD of
> course.
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Building GDAL documentation with ReadTheDocs
Hello, I've put together a configuration to build GDAL's documentation using ReadTheDocs. As far as I know, this is the simplest way of publishing version-specific documentation and rending documentation for pull requests. A current build of the documentation can be viewed at https://gdal.readthedocs.io/en/latest/. A pull request with the proposed configuration is available at https://github.com/OSGeo/gdal/pull/10434. If anyone has feedback about using ReadTheDocs for this project, or the specifics of the proposed configuration, please share it here or as a comment on the pull request. Thanks, Dan ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] OGR SQL CAST geometry example
If a single MULTIPOINT record is OK, then you can do ogrinfo test.shp -q -dialect SQLITE -sql 'SELECT ST_DissolvePoints(GEOMETRY) FROM test' Dan On Thu, May 30, 2024 at 8:49 PM Dan Jacobson via gdal-dev wrote: > > $ ogrinfo 0.lines.kml -q -dialect SQLITE -sql \ > 'SELECT ST_PointN(GEOMETRY, generate_series(1, ST_NPoints(GEOMETRY))) FROM > "0"' > ERROR 1: In ExecuteSQL(): sqlite3_prepare_v2( > no such function: generate_series > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Doxygen Arguments
It looks like Doxygen is using argument names from the function prototype instead of the implementation: https://github.com/OSGeo/gdal/blob/master/gcore/gdal_priv.h#L753 I did some searching but didn't come across a way to change this behavior. It looks like many other functions include the argument names in the prototype, so maybe RasterIO should be updated to include them too? Dan On Thu, May 9, 2024 at 9:20 AM Andrew Bell via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > I'm looking at the documentation for GDALRasterBand::RasterIO and it's > surprising that the names of the arguments aren't included in the function > prototype despite them being in the doxygen function definition. The > arguments are described well below the function prototype, making it hard > to match things up and you have to assume that the order matches. > > Does some doxygen expert know how the argument names could somehow be > included in the function prototype? > > Web page: > https://gdal.org/api/gdalrasterband_cpp.html#classGDALRasterBand_1a75d4af97b3436a4e79d9759eedf89af4 > > Source: https://fossies.org/linux/gdal/gcore/gdalrasterband.cpp (line 110) > > -- > Andrew Bell > andrew.bell...@gmail.com > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] About -oo -lco -dsco in docs
Perhaps a Sphinx substitution could be made with the relevant text and added to https://github.com/OSGeo/gdal/blob/master/doc/source/substitutions.rst Dan On Wed, Mar 27, 2024 at 5:01 PM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Salut Thomas, > > There's a tension between being clear to users, and also aiming at being > concise / not repeating us too much (we have already one thousand pages > of doc to maintain), and if we need to repeat, find the technical way of > doing it without actually having to copy the same text. > > Probably a solution could be to have under each "Open options" > paragraph, and introduction sentence like > > "Consult :ref:`how to specify open options ` with command > line utilities and the API." that links to a page explaining the concept > and how to use it. > > I was wondering if we'd just want to copy that sentence at all > relevant places, or create a custom Sphinx extension for a directive > that would do that. The later is probably overkill. Possibly a one-time > simple Python script could automatically insert that at relevant places > > Even > > > Le 26/03/2024 à 21:25, Thomas Gratier via gdal-dev a écrit : > > Hi, > > > > There are implicit rules when you read the OGR/GDAL docs in fact, in > > particular when you use `ogr2ogr` or `ogrinfo` > > > > `-oo =` like "Open Options" > > `-lco =` for "Layer Creation Option" > > `-dsco =` for "DataSet Creation Option" > > > > So, when you read for instance > > https://gdal.org/drivers/vector/csv.html, you will see there is a > > section "Open options" so you will know that you can use `-oo`. Also, > > you will see a "Layer Creation options" block so again, you will know > > it mean use `-lco`. > > > > It's documented in "ogr2ogr" but you need to know the rule to make the > > link in your mind when reading the doc. Any idea to clarify the doc > > about the "tip". > > > > An idea could be about adding in drivers docs "Open options" block a > > sentence like "When using the command line, use the flag `-oo > > =`" and do the same for "Layer creation options" block > > "When using the command line, use the flag `-lco =`" > > > > Thoughts about the suggestion? Better idea? Not worth? > > > > Regards, > > > > Thomas Gratier > > > > ___ > > gdal-dev mailing list > > gdal-dev@lists.osgeo.org > > https://lists.osgeo.org/mailman/listinfo/gdal-dev > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Minimum python version?
The minimum version (3.6) is documented at https://gdal.org/development/dev_environment.html#build-requirements The CI still has configurations using Ubuntu 18.04, which is using 3.6.9: docker run --rm ghcr.io/osgeo/gdal-deps:ubuntu_18.04-master python3 --version Dan On Tue, Jun 13, 2023 at 1:22 PM Kurt Schwehr wrote: > Hi all, > > What is the minimum python version for the GDAL python bindings? This is > the only thing I found easily: > > find . -type f | grep -v git | xargs grep 3 | grep python_version > > cat autotest/requirements.txt > pytest>=6.0.0 > pytest-sugar<=0.9.6; python_version < '3.7' > pytest-sugar; python_version >= '3.7' > pytest-env > lxml > jsonschema > filelock > > I think the real answer is that it's 3.8 from what I see here: > > swig/python/osgeo/__init__.py > > # The following is a Sequence of tuples in the form of (gdal_version, > python_version). > # Each line represents the minimum supported Python version of a given > GDAL version. > # Introducing a new line for the next GDAL version will trigger a > deprecation warning > # when importing osgeo from a Python version which will not be > # supported in the next version of GDAL. > gdal_version_and_min_supported_python_version = ( > ((0, 0), (0, 0)), > ((1, 0), (2, 0)), > ((2, 0), (2, 7)), > ((3, 3), (3, 6)), > # ((3, 4), (3, 7)), > # ((3, 5), (3, 8)), > ) > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] configuration option GDAL_RASTERIO_RESAMPLING
Hi Mike, The content on the page you linked is for the most part a simple transfer of what was available on the wiki, plus a few options that were previously defined on driver or other pages. It is not complete. There are also about 500 configuration options that are used in the code via CPLGetConfigOption that do not appear in the documentation, GDAL_RASTERIO_RESAMPLING being one of them. Some of these are clearly not intended to be part of the GDAL interface, but there's still quite a bit that can be added to the documentation. I can create an issue and post the script I used to scan for these if others are interested in filling some of the gaps. Dan On Sat, May 27, 2023 at 8:18 PM Michael Sumner wrote: > Hi, I'm very happy to see the refactor of documentation for configuration > and environment options, primarily by Dan Baston in recent weeks - thanks! > > https://gdal.org/user/configoptions.html > > Does the GDAL_RASTERIO_RESAMPLING option belong in this list? I'm > exploring its use versus setting resampling with the API, and I gather also > that there is not an analogous config option for the warper. > > Thank you, Mike > > -- > Michael Sumner > Software and Database Engineer > Australian Antarctic Division > Hobart, Australia > e-mail: mdsum...@gmail.com > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Report on GEOS maintenance grant
Hi All, As February comes to a close, I’m winding down the GEOS maintenance work that the GDAL PSC funded in 2022 [1]. I appreciate the opportunity to do this work and am especially thankful to those who reported issues, reviewed pull requests, or participated in discussions about this work. For anyone who’s interested, I wanted to share a few of the highlights from my perspective. New functionality: GEOS now supports reading, writing, and overlay of geometries with an M dimension. I added a number of C API utility functions requested by client library developers: GEOSLineSubstring, GEOSEqualsIdentical, and thread/context-specific interrupt handlers. I also brought in clustering algorithms such as DBSCAN that were previously only available in PostGIS. Performance enhancements: I found some nice performance wins in key GEOS algorithms such as geometry intersection testing (GEOSIntersects and GEOSPreparedIntersects), distance (GEOSPreparedDistance and GEOSPreparedDistanceWithin) and buffering complex geometries. There are also some gains in core internal classes such as CoordinateSequence and STRtree that deliver across-the-board gains. Project maintenance. I was able to simplify our CI setup by removing the use of Azure and AppVeyor services and bringing the associated environments into GitHub Actions, write a harness for performance testing and visualization, and perform code cleanup such as migrating from raw to managed pointers in key classes. I resolved a number of long-standing bug reports with WKT input/output, geometry simplification, and crashes/incorrect results from multiple algorithms in edge cases such as empty geometry inputs. Not everything panned out, or at least not yet. Something we’ve wanted in GEOS for a long time is the ability to create a GEOS geometry directly from an external memory buffer, such as a PostGIS LWGEOM or a GDAL OGRGeometry (“zero-copy”). I added experimental support for this, but so far I haven’t found a case where it offers much performance benefit. Dan [1] https://lists.osgeo.org/pipermail/gdal-dev/2022-February/055433.html ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Call for discussion on RFC 92 text: WKB Only geometries
| And only construct an OGRGeometry when it's asked for? Such as when GetGeometryRef is called? I'm wondering about a more broad application of this. Would it be helpful to have the ability to lazy-initialize an OGRGeometry from multiple source types such as WKB and GEOS, initially storing only a reference to the external data in WKB/GEOS/etc and actually materializing the geometry when required? Then methods such as OGRGeometry::exportToWkb and OGRGeometry::exportToGEOS could check the external data type and use it directly if it is compatible, avoiding materialization. This would avoid multiple conversions to/from GEOS in cases where operations are chained, as well as allowing WKB to pass directly between input and output drivers that support it. Relatedly, this ability could be used to cache external-format data when it is generated for an OGRGeometry, avoiding inefficiencies such as two conversions to GEOS when checking to see if two geometries intersect before calculating their intersection. Dan On Sat, Feb 4, 2023 at 1:55 PM Even Rouault wrote: > Hi Sean, > > but wouldn't it be possible for all OGRFeatures to carry WKB data by > default and add a method to provide it to callers? > > My understanding of what you propose would involve massive code rewrites > in all drivers and wouldn't be desirable from a performance point of view, > because most drivers can't generate WKB easily (PostGIS and GPKG are the > exceptions rather the norm). So either all other drivers should be modified > to compose WKB at hand (massive coding effort. Probably several weeks of > effort and significant risk of regressions). Or get it from the > ExportToWkb() method of the OGRGeometry instance they currently build, but > then you pay the price in memory and CPU time to generate WKB that might > not be consumed by users. > > | And only construct an OGRGeometry when it's asked for? Such as when > GetGeometryRef is called? > > Good point, we could both make GetGeometryRef() and GetGeomFieldRef() > virtual methods whose default implementation would be the same as > currently, ie. return the value of the corresponding member variable in the > base OGRFeature class stored with > SetGeometry[Directly]()/SetGeomField[Directly]() > > And add a new virtual method: > > virtual GByte* OGRFeature::GetWKBGeometry(int iGeomField, size_t* > pnOutSize) const > > whose default implementation would just use > GetGeomFieldRef(iGeomField)->ExportToWkb(). > > The few drivers that can provide a more efficient implementation (GPKG > typically) would create a derived class OGRFeatureGPKG with a specific > implementation of those new virtual methods to avoid systematic OGRGeometry > instantiation. The only drawback I see is that making GetGeometryRef() and > GetGeomFieldRef() virtual would have a slight performance impact, but > probably small enough. > > > But fundamentally I'm wondering if RFC 92 hasn't been made mostly out > fashioned now that we have RFC 86. RFC 86 generally leads to 2x speed-up or > more on real-world datasets compared to OGRFeature iteration (as measured > by the bench_ogr_c_api vs bench_ogr_batch utilities) on drivers that have > implemented it (currently Arrow, Parquet, FlatGeoBuf, GPKG), whereas RFC 92 > only applies to GPKG & PostGIS and in the best - artificial - case only > lead to 30% speed-up. > > Of course, adopting RFC 86 requires significant effort from GDAL users, > but the benefit is really measurable whereas with RFC 92 it would be > marginal in most scenarios. As far as I can tell, the performance boost of > RFC 86 comes mostly from saving creation & destruction of millions of > OGRFeature instances, its array members, string attributes, geometries > objects, more than the columnar organization of the ArrowArray data > structures. In the GeoPackage driver, I've also shown that it makes it > possible for efficient multi-threading pre-fetching, totally transparent > for the user. > > But to avoid selling false hopes, the benefit of RFC 86 in end-to-end > scenarios would probably drop significantly (at least if looking at > performance gain in percentage. The absolute performance savings on the > GDAL side would remain) if you need to recreate individual features (QGIS' > QgsFeature or MapServer' msShape objects) from the content of ArrowArray. > So this is likely a complete shift of concepts that would be required. > > Even > > > > On Tue, Jan 31, 2023 at 4:27 AM Even Rouault > wrote: > >> Hi, >> >> Please find for review "RFC 92 text: WKB Only geometries" at >> https://github.com/OSGeo/gdal/pull/7149 >> >> This RFC provides shortcuts to avoid instantiation of full OGRGeometry >> instances >> in scenarios where only the WKB representation of geometries is needed. >> The >> hope is to save CPU time. >> >> This is something I wanted to at least experiment. I've mixed feelings >> if it's something we actually want to adopt. >> >> Even >> >> -- >> http://www.spatialys.com >> My software is free, but my time
Re: [gdal-dev] [geos-devel] GEOS Maintenance Grant
Thank you, all. I'm honored by your support. Please don't hesitate to open issues on the GEOS GitHub page to share any ideas you have about how GEOS can better support GDAL and friends. Dan On Thu, Feb 17, 2022 at 11:13 AM Howard Butler wrote: > Declaring this motion passed with +1's from > > DanielM, EvenR, NormanB, TamasS, HowardB, MateuszL, FrankW, KurtS, and > SeanG > > The GDAL PSC is fully delegating these resources to the GEOS PSC to use as > it sees fit. Paul Ramsey has agreed to be the administrative contact for > the GEOS PSC, and will coordinate the effort in regards to payment > sign-off, work task development, rate negotiation, and reporting to the > GDAL PSC. > > Howard > > PS: As a member of both PSCs, I will abstain from any funding or resource > direction votes in the GEOS PSC to avoid any potential conflict. > > > On Feb 15, 2022, at 9:37 AM, Howard Butler wrote: > > > > GDAL PSC, > > > > When we wrote the GDAL RFCs on sponsorship, we provided an escape clause > to allow us to direct resources to other projects upon which GDAL depends. > Our sponsorship numbers are still increasing, which provides us an > opportunity to directly support some of those projects, and one of them is > obviously GEOS. GEOS provides all of the geometry algebra support for > GDAL/OGR and many other open source geospatial softwares including Shapely, > PostGIS, GeoPandas, MapServer, and more. > > > > Dan Baston of the GEOS PSC has been identified as the developer with > capacity and interest in the next year to take on GEOS development on APIs > and performance, which he has a long history of doing for the project. This > support should allow him to work longer, multi-release upgrades that will > provide strong performance and convenience benefits for the project. > > > > I motion to provide the GEOS PSC with a $50,000 USD grant to address > performance, API, and other work that does not attract directed funding in > GEOS. The GEOS PSC will be responsible for coordinating work tasks, rates, > and development timelines. Howard Butler or Even Rouault of the GDAL > NumFocus liaison team will coordinate dispersement as directed by the GEOS > PSC and NumFocus rules. > > > > Thank you again to the GDAL Sponsors > https://gdal.org/sponsors/index.html who have made this kind of grant > possible. A better GEOS makes for a better GDAL. > > > > Howard > > > > ___ > geos-devel mailing list > geos-de...@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/geos-devel > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
