Re: [gdal-dev] Motion: Annual Contracts for Maintainers
For the record: Norman and Daniel also +1'd on October 11. On 2023-10-16 15:59, Howard Butler via gdal-dev wrote: On Oct 11, 2023, at 11:13 AM, Howard Butler wrote: PSC, I'm a little late but I would like to make the following motions in regards to GDAL maintainers: * Authorize Alessandro Pasotti for up to 30 hrs/month with a rate increase of 10 euros/hr until 30 NOV 2024. * Authorize Even Rouault for up to 90 hrs/month with a rate increase of 10 $/hr until 30 JUL 2024 Howard Declaring this passed with +1s from Jukka, Frank, Kurt, Sean, and Javier. Howard ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev -- Daniel Morissette Mapgears Inc T: +1 418-696-5056 #201 ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Motion: Annual Contracts for Maintainers
> On Oct 11, 2023, at 11:13 AM, Howard Butler wrote: > > PSC, > > I'm a little late but I would like to make the following motions in regards > to GDAL maintainers: > > * Authorize Alessandro Pasotti for up to 30 hrs/month with a rate increase of > 10 euros/hr until 30 NOV 2024. > * Authorize Even Rouault for up to 90 hrs/month with a rate increase of 10 > $/hr until 30 JUL 2024 > > Howard Declaring this passed with +1s from Jukka, Frank, Kurt, Sean, and Javier. Howard ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression
> In short: multithreading is hard So true! With the introduction of tsan things are a little less bad, but tsan is still a tool with plenty of false positives and false negatives. And that assumes that a particular issue is covered by the tests being run under tsan. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression
Thanks for the announcement, Even! I wonder if we should track such issues in a list? Or maybe give them a unique GitHub label? We don't plan to release a 3.6.5, correct? I'm going to make a Rasterio post release that patches 3.6.4 by tomorrow: https://github.com/rasterio/rasterio/issues/2943. On Mon, Oct 16, 2023 at 9:26 AM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > > Le 16/10/2023 à 17:14, Kurt Schwehr a écrit : > > Thanks for the heads up! > > Can you share that SHAs of the fix and cause? > > master: d083af1ec9c38e79bfb0532885570de6e5b8a410 > > 3.7 branch (should apply to 3.6 as well): > b5858ed5bc5004c97f7cd6000674015bdc70b33b > > cause: a worker thread of the multithreaded decoding could, in a situation > where the block cache is full, cause a "dirty" (ie modified but not yet > serialized to disk) block to be written, resulting in either a deadlock > between the lock of the multithreaded decoder and the lock of the job queue > mechanism, or other decoding threads could see their file handle being > seek() by another thread. In short: multithreading is hard. > > > On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev < > gdal-dev@lists.osgeo.org> wrote: > >> Hi, >> >> For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff >> compression+decompression, in particular within the same file, as for >> example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow >> can lead to deadlocks (process stalled forever) and/or data corruption >> (sometimes without errors at generation). Cf >> https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is >> in https://github.com/OSGeo/gdal/pull/8561 >> >> The issue is particularly visible on Windows, or more generally any >> operating system (or file system where the output file is located) which >> has no VSIVirtualHandle::PRead() implementation, but it can also be >> occasionally reproduced on Linux (at least as a deadlock). >> >> Even >> >> -- Sean Gillies ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression
Le 16/10/2023 à 17:14, Kurt Schwehr a écrit : Thanks for the heads up! Can you share that SHAs of the fix and cause? master: d083af1ec9c38e79bfb0532885570de6e5b8a410 3.7 branch (should apply to 3.6 as well): b5858ed5bc5004c97f7cd6000674015bdc70b33b cause: a worker thread of the multithreaded decoding could, in a situation where the block cache is full, cause a "dirty" (ie modified but not yet serialized to disk) block to be written, resulting in either a deadlock between the lock of the multithreaded decoder and the lock of the job queue mechanism, or other decoding threads could see their file handle being seek() by another thread. In short: multithreading is hard. On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev wrote: Hi, For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff compression+decompression, in particular within the same file, as for example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow can lead to deadlocks (process stalled forever) and/or data corruption (sometimes without errors at generation). Cf https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is in https://github.com/OSGeo/gdal/pull/8561 The issue is particularly visible on Windows, or more generally any operating system (or file system where the output file is located) which has no VSIVirtualHandle::PRead() implementation, but it can also be occasionally reproduced on Linux (at least as a deadlock). Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression
Thanks for the heads up! Can you share that SHAs of the fix and cause? On Mon, Oct 16, 2023, 7:44 AM Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff > compression+decompression, in particular within the same file, as for > example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow > can lead to deadlocks (process stalled forever) and/or data corruption > (sometimes without errors at generation). Cf > https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is > in https://github.com/OSGeo/gdal/pull/8561 > > The issue is particularly visible on Windows, or more generally any > operating system (or file system where the output file is located) which > has no VSIVirtualHandle::PRead() implementation, but it can also be > occasionally reproduced on Linux (at least as a deadlock). > > Even > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression
Thanks Even for notifying this. I see the fix will be included in the next releases 3.7.3 and 3.8.0, both planned for November 1st (just in a couple of weeks) https://github.com/OSGeo/gdal/milestones On Mon, 16 Oct 2023 at 16:42, Even Rouault via gdal-dev < gdal-dev@lists.osgeo.org> wrote: > Hi, > > For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff > compression+decompression, in particular within the same file, as for > example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow > can lead to deadlocks (process stalled forever) and/or data corruption > (sometimes without errors at generation). Cf > https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is > in https://github.com/OSGeo/gdal/pull/8561 > > The issue is particularly visible on Windows, or more generally any > operating system (or file system where the output file is located) which > has no VSIVirtualHandle::PRead() implementation, but it can also be > occasionally reproduced on Linux (at least as a deadlock). > > Even > > -- > http://www.spatialys.com > My software is free, but my time generally not. > > ___ > gdal-dev mailing list > gdal-dev@lists.osgeo.org > https://lists.osgeo.org/mailman/listinfo/gdal-dev > ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Notice: issue about multi-threaded GTiff compression+decompression
Hi, For GDAL 3.6.0 to 3.7.2, use of multi-threaded GTiff compression+decompression, in particular within the same file, as for example in a "gdalwarp -co COMPRESS=... -co NUM_THREADS=..." workflow can lead to deadlocks (process stalled forever) and/or data corruption (sometimes without errors at generation). Cf https://github.com/OSGeo/gdal/issues/8470 for a reproducer. The fix is in https://github.com/OSGeo/gdal/pull/8561 The issue is particularly visible on Windows, or more generally any operating system (or file system where the output file is located) which has no VSIVirtualHandle::PRead() implementation, but it can also be occasionally reproduced on Linux (at least as a deadlock). Even -- http://www.spatialys.com My software is free, but my time generally not. ___ gdal-dev mailing list gdal-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Reading a shapefile containing multi-parts Polyline from OGR (C++ API)
Thanks a lot Javier. I'll test this and come back to you.
Here is a download link to the shapefile causing this issue for me:
https://filesender.renater.fr/?s=download=da2a7ff6-9755-4086-b1ed-71c6ebc72a0f
Le lun. 16 oct. 2023 à 15:21, Even Rouault a
écrit :
>
> Le 16/10/2023 à 15:15, Javier Jimenez Shaw via gdal-dev a écrit :
>
> Do you mean a MultiLineString?
>
> This piece of code is working for me (I hope without any bug).
>
> if (wkbFlatten(poGeometry->getGeometryType()) == wkbMultiLineString)
> multiLineStringGeometry(poGeometry);
> ...
> multiLineStringGeometry(OGRGeometry* poGeometry)
> {
> OGRMultiLineString* multiLine =
> dynamic_cast(poGeometry);
> int numGeometries = multiLine->getNumGeometries();
> for (int i = 0; i < numGeometries; i++)
> {
> OGRGeometry* eachGeometry =
> multiLine->getGeometryRef(i);
> OGRLineString* line =
> dynamic_cast(eachGeometry);
> // do whatever with that LineString
> }
> }
>
> Can further be simplified to something like:
> multiLineStringGeometry(const OGRGeometry* poGeometry)
> {
> for (const OGRLineString* line:
> poGeometry->toMultiLineString())
> {
> // do whatever with that LineString
> }
> }
>
>
> On Mon, 16 Oct 2023 at 12:14, Roland Baviere via gdal-dev <
> gdal-dev@lists.osgeo.org> wrote:
>
>> Hi all,
>>
>> I hope this e-mel finds you well.
>>
>> I am trying to read a shapefile layer from OGR from a C++ application
>> following the tutorial found here:
>> https://gdal.org/tutorials/vector_api_tut.html#reading-from-ogr
>>
>> We have difficulties with a layer containing PolyLines with several parts.
>>
>> The pre-existing code works fine for polylines having a unique part.
>>
>> Do you have any example or advice to help me implement a version that
>> works for polylines containing several parts. My intention is to display
>> each shape on a map, so I need to retrieve the coordinates of points and
>> draw segment between points when relevant.
>>
>> Thanks a lot for your help.
>>
>> Kind regards,
>>
>> Roland
>>
>>
>>
>> CODE THAT WORKS for "simple" polylines but fails when dealiing with
>> a polyline containing several parts
>>
>> int o = 0;
>> for(auto y : poGeometry->toLineString()) {
>> if (o == 0) {
>> line.mInCoord = QGeoCoordinate(y.getX(),
>> y.getY());
>> } else if (o ==
>> (poGeometry->toLineString()->getNumPoints() - 1)) {
>> line.mOutCoord = QGeoCoordinate(y.getX(),
>> y.getY());
>> } else {
>>
>> line.mPath.addCoordinate(QGeoCoordinate(y.getX(), y.getY()));
>> }
>> qDebug() << o;
>> o++;
>> }
>>
>> ___
>> gdal-dev mailing list
>> gdal-dev@lists.osgeo.org
>> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>>
>
> ___
> gdal-dev mailing
> listgdal-dev@lists.osgeo.orghttps://lists.osgeo.org/mailman/listinfo/gdal-dev
>
> -- http://www.spatialys.com
> My software is free, but my time generally not.
>
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Reading a shapefile containing multi-parts Polyline from OGR (C++ API)
Le 16/10/2023 à 15:15, Javier Jimenez Shaw via gdal-dev a écrit :
Do you mean a MultiLineString?
This piece of code is working for me (I hope without any bug).
if (wkbFlatten(poGeometry->getGeometryType()) == wkbMultiLineString)
multiLineStringGeometry(poGeometry);
...
multiLineStringGeometry(OGRGeometry* poGeometry)
{
OGRMultiLineString* multiLine =
dynamic_cast(poGeometry);
int numGeometries = multiLine->getNumGeometries();
for (int i = 0; i < numGeometries; i++)
{
OGRGeometry* eachGeometry =
multiLine->getGeometryRef(i);
OGRLineString* line =
dynamic_cast(eachGeometry);
// do whatever with that LineString
}
}
Can further be simplified to something like:
multiLineStringGeometry(const OGRGeometry* poGeometry)
{
for (const OGRLineString* line:
poGeometry->toMultiLineString())
{
// do whatever with that LineString
}
}
On Mon, 16 Oct 2023 at 12:14, Roland Baviere via gdal-dev
wrote:
Hi all,
I hope this e-mel finds you well.
I am trying to read a shapefile layer from OGR from a C++
application following the tutorial found here:
https://gdal.org/tutorials/vector_api_tut.html#reading-from-ogr
We have difficulties with a layer containing PolyLines with
several parts.
The pre-existing code works fine for polylines having a unique part.
Do you have any example or advice to help me implement a version
that works for polylines containing several parts. My intention is
to display each shape on a map, so I need to retrieve the
coordinates of points and draw segment between points when relevant.
Thanks a lot for your help.
Kind regards,
Roland
CODE THAT WORKS for "simple" polylines but fails when
dealiing with a polyline containing several parts
int o = 0;
for(auto y : poGeometry->toLineString()) {
if (o == 0) {
line.mInCoord =
QGeoCoordinate(y.getX(), y.getY());
} else if (o ==
(poGeometry->toLineString()->getNumPoints() - 1)) {
line.mOutCoord =
QGeoCoordinate(y.getX(), y.getY());
} else {
line.mPath.addCoordinate(QGeoCoordinate(y.getX(), y.getY()));
}
qDebug() << o;
o++;
}
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
--
http://www.spatialys.com
My software is free, but my time generally not.
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Reading a shapefile containing multi-parts Polyline from OGR (C++ API)
Do you mean a MultiLineString?
This piece of code is working for me (I hope without any bug).
if (wkbFlatten(poGeometry->getGeometryType()) == wkbMultiLineString)
multiLineStringGeometry(poGeometry);
...
multiLineStringGeometry(OGRGeometry* poGeometry)
{
OGRMultiLineString* multiLine =
dynamic_cast(poGeometry);
int numGeometries = multiLine->getNumGeometries();
for (int i = 0; i < numGeometries; i++)
{
OGRGeometry* eachGeometry =
multiLine->getGeometryRef(i);
OGRLineString* line =
dynamic_cast(eachGeometry);
// do whatever with that LineString
}
}
On Mon, 16 Oct 2023 at 12:14, Roland Baviere via gdal-dev <
gdal-dev@lists.osgeo.org> wrote:
> Hi all,
>
> I hope this e-mel finds you well.
>
> I am trying to read a shapefile layer from OGR from a C++ application
> following the tutorial found here:
> https://gdal.org/tutorials/vector_api_tut.html#reading-from-ogr
>
> We have difficulties with a layer containing PolyLines with several parts.
>
> The pre-existing code works fine for polylines having a unique part.
>
> Do you have any example or advice to help me implement a version that
> works for polylines containing several parts. My intention is to display
> each shape on a map, so I need to retrieve the coordinates of points and
> draw segment between points when relevant.
>
> Thanks a lot for your help.
>
> Kind regards,
>
> Roland
>
>
>
> CODE THAT WORKS for "simple" polylines but fails when dealiing with a
> polyline containing several parts
>
> int o = 0;
> for(auto y : poGeometry->toLineString()) {
> if (o == 0) {
> line.mInCoord = QGeoCoordinate(y.getX(),
> y.getY());
> } else if (o ==
> (poGeometry->toLineString()->getNumPoints() - 1)) {
> line.mOutCoord = QGeoCoordinate(y.getX(),
> y.getY());
> } else {
>
> line.mPath.addCoordinate(QGeoCoordinate(y.getX(), y.getY()));
> }
> qDebug() << o;
> o++;
> }
>
> ___
> gdal-dev mailing list
> gdal-dev@lists.osgeo.org
> https://lists.osgeo.org/mailman/listinfo/gdal-dev
>
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
[gdal-dev] Reading a shapefile containing multi-parts Polyline from OGR (C++ API)
Hi all,
I hope this e-mel finds you well.
I am trying to read a shapefile layer from OGR from a C++ application
following the tutorial found here:
https://gdal.org/tutorials/vector_api_tut.html#reading-from-ogr
We have difficulties with a layer containing PolyLines with several parts.
The pre-existing code works fine for polylines having a unique part.
Do you have any example or advice to help me implement a version that works
for polylines containing several parts. My intention is to display each
shape on a map, so I need to retrieve the coordinates of points and draw
segment between points when relevant.
Thanks a lot for your help.
Kind regards,
Roland
CODE THAT WORKS for "simple" polylines but fails when dealiing with a
polyline containing several parts
int o = 0;
for(auto y : poGeometry->toLineString()) {
if (o == 0) {
line.mInCoord = QGeoCoordinate(y.getX(),
y.getY());
} else if (o ==
(poGeometry->toLineString()->getNumPoints() - 1)) {
line.mOutCoord = QGeoCoordinate(y.getX(),
y.getY());
} else {
line.mPath.addCoordinate(QGeoCoordinate(y.getX(), y.getY()));
}
qDebug() << o;
o++;
}
___
gdal-dev mailing list
gdal-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/gdal-dev
