Re: [GRASS-dev] A load of updates in OSGeo4W

[Edouard Choinière via grass-dev]

Thanks a lot for the heads up!
So we’ll now be more strict with Python 3.12 syntax warnings and others. 
Assuring Python 3.12 compatibility was already started, but never officialized. 
(No responses on https://github.com/OSGeo/grass/issues/3285)

However, I launched a CI build for a PR a couple minutes ago, and usually 
installing the OSGeo4W packages takes about 2 minutes, now took 5min15. Is it 
just the server that is slower now? I was tracing with xperf this week the 
build process on windows CI, getting ETW files (1.6 Gb once extracted). I was 
trying to find out what was slow on our side, but noticed that the 2 minutes 
for OSGeo4W package installation wasn’t optimal: not CPU limited, not I/O 
limited, and not memory limited (compared to steps before and after, like 
msys2). I didn’t search more closely though.


That new build also fails, since it looks for Python 3.9 specifically, was 
there a patch that was needed for the official grass osgeo4w build?


Edouard Choinière

Le 14 avr. 2024 à 19:02, Jürgen E. Fischer via grass-dev 
 a écrit :

Hi there,

I just uploaded an update to OSGeo4W.   It is basically a rebuild of everything
with a newer compiler (Visual C++ 2022) and updates of almost all versions.
The update should work cleanly and cause no visual changes.

The main change is the move from Python 3.9.18 to 3.12.3.  But there's also an
update to Qt 5.15.13, OpenSSL 3.   Qt6 was also added enabling an experimental
build of QGIS master with Qt6. Based on that there is also an experimental
version of QField.

GDAL and PROJ also were updated to the latest version.

The legacy GRASS 7 was removed because it doesn't support Python 3.12.  GRASS 8
was already available and QGIS switch to it long before this update.

Please test and report.


Jürgen


More details in the commit message:

A load of updates (fixes #788, #810, #816, #819, #820, #823, closes 
jef-n/OSGeo4W#21, refs qgis/QGIS#54491, qgis/QGIS#56499)

Highlights:
 Switched to Visual Studio 2022
 Qt6 6.6.3, PyQt6 6.6.1
 Qt5 5.15.13, PyQt5 5.15.10
 Python 3.12.3
 OpenSSL 3.0.13
 PROJ 9.4.0
 GDAL 3.8.5
 qgis-qt6-dev based on Qt6 next to qgis-dev based on Qt5
 experimental QField based on Qt6 / qgis-qt6-dev

Details:
 Removed:
   libjpeg (already replaced with libjpeg-turbo earlier)
   python3-clcache (replaced by ccache)
   python3-pyuv (dependency of python3-clcache)

 Updated:
   apache 2.4.52 -> 2.4.58
   arrow-cpp 7.0.0 -> 15.0.2
   boost 1.74.0 -> 1.84.0
   brotli 1.0.9 -> 1.1.0
   curl 8.4.0 -> 8.6.0
   draco 1.5.6 -> 1.5.7
   exiv2 0.27.3 -> 0.28.2
   expat 2.2.10 -> 2.6.2
   ffmpeg 5.1 -> 6.1.1
   freetype 2.10.2 -> 2.13.2
   gdal 3.8.4 -> 3.8.5
   gpsbabel 1.8.0 -> 1.9.0
   grass 7.8.8 -> 8.3.2
   grass8 8.3.2 -> 99 (transitional; depends on grass)
   gsl 2.6 -> 2.7.1+
   hdf4 4.2.16 -> 4.3.0
   hdf5 1.14.0 -> 1.14.3
   kealib 1.4.14 -> 1.5.3
   lerc 3.0 -> 4.0.0
   libharu 2.3.0 -> 2.4.4
   libiconv 1.16 -> 1.17
   libjpeg-turbo 2.0.7-esr -> 3.0.2
   libjxl 0.8.1 -> 0.10.2
   libmysql 8.0.21 -> 8.2.0
   libosmium-devel 2.18.0 -> 2.20.0
   libpng 1.6.37 -> 1.6.43
   libtiff 4.5.1 -> 4.6.0
   libxml2 2.9.10 -> 2.12.5
   libxslt 1.1.34 -> 1.1.39
   libzip 1.7.3 -> 1.10.1
   lua 5.4.4 -> 5.4.6
   lz4 1.9.3 -> 1.9.4
   minizip-ng-devel 3.0.2 -> 4.0.4
   node 16.14.0 -> 20.11.1
   oci 19.11 -> 21.13
   ogdi 4.1.0 -> 4.1.1
   opencl 2.0.10 -> 2023.12.14
   openfyba-devel 20150103 -> 20240408
   openjpeg 2.4.0 -> 2.5.2
   openssl 1.1.1w -> 3.0.13
   osm2pgsql 1.8.1 -> 1.11.0
   osmium 1.15.0 -> 1.16.0
   pdal 2.6.0 -> 2.6.3
   poppler 23.07.0 -> 24.04.0
   proj 9.3.1 -> 9.4.0
   proj-data 1.16 -> 1.17
   python3 3.9.18 -> 3.12.3
   protobuf-devel 3.13.0 -> 25.3
   qca 2.3.1 -> 2.3.8
   qscintilla 2.13.4 -> 2.14.1
   qt5 5.15.3 -> 5.15.13
   qtkeychain 0.13.2 -> 0.14.2
   qwc2 20220311-671a6e7 -> 20240408-3d95409
   qwt 6.1.6 -> 6.2.0
   saga 7.8.2 -> 9.3.1
   saga9 9.2.0 -> 99 (transitional; depends on saga)
   snappy-devel 1.1.9 -> 1.1.10
   spdlog-devel 1.10.0 -> 1.13.0
   sqlite3 3.41.1 -> 3.45.1
   swig 4.0.2 -> 4.2.1
   thrift 0.16.0 -> 0.20.0
   transifex-cli 1.6.5 -> 1.6.10
   utf8proc 2.7.0 -> 2.9.0
   wxwidgets 3.2.1 -> 3.2.4
   xerces-c 3.2.3 -> 3.2.5
   xz 5.2.5 -> 5.4.5
   yarnpkg 1.22.17 -> 1.22.21
   zlib 1.2.12 -> 1.3.1
   zstd 1.4.5 -> 1.5.5

 Updated Python extensions:
   python3-access 1.1.1 -> 1.1.9
   python3-affine 2.3.0 -> 2.4.0
   python3-alabaster 0.7.12 -> 0.7.16
   python3-argon2-cffi 20.1.0 -> 23.1.0
   python3-atomicwrites 1.4.0 -> 1.4.1
   python3-attrdict 2.0.1 -> python3-attrdict3 2.0.2
   python3-attrs 20.2.0 -> 23.2.0
   python3-autopep8 2.0.1 -> 2.1.0
   python3-babel 2.8.0 -> 2.14.0
   python3-backports.entry-points-selectable 1.1.0 -> 1.3.0
   python3-beautifulsoup4 4.9.3 -> 4.12.3
   python3-bleach 3.2.1 -> 6.1.0
   python3-certifi 2020.6.20 -> 2024.2.2
   python3-cffi 1.14.3 -> 1.16.0
   python3-cftime 1.2.1 -> 1.6.3
   python3-chardet 3.0.4 -> 5.2.0
   python3-click 7.1.2 -> 8.1.7
   python3-cligj 

Re: [GRASS-dev] GRASS GIS: backport of CI speed updates to G83?

[Edouard Choinière via grass-dev]

But as one of the members with the maintain role (higher than triage and 
write), he has access to bypass permissions. If you go look back in the PR 
where I explained the branch rules, I showed the checkbox that allows to bypass 
for example in PRs. Here is the public combined view of the rules that are 
active for releasebranch_8_3 
https://github.com/OSGeo/grass/rules/270686?ref=refs%2Fheads%2Freleasebranch_8_3
 and https://github.com/OSGeo/grass/rules?ref=refs%2Fheads%2Freleasebranch_8_3

Note how they are different than 
https://github.com/OSGeo/grass/rules?ref=refs%2Fheads%2Fmain

It was made such as changing workflow requirements for main don’t restrict the 
older branches to be merged, or that changes in the CI infrastructure (outside 
of our repo) make it so they don’t work anymore. So they are kind of run on a 
best-effort basis, instead of religiously.


Edouard Choinière

Le 20 févr. 2024 à 06:45, Nicklas Larsson  a écrit :


On 20 Feb 2024, at 10:14, Markus Neteler via grass-dev 
mailto:grass-dev@lists.osgeo.org>> wrote:


In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs ... which is a pain).


I agree this is a case where we have limited ourself too much, with all those
required 1.5 hrs tests, approval, etc. (not even [skip ci] would work) . What 
you
would need is a (ideally) direct commit access or at least  “Rebase and merge”
option to merge (thus enable a number of commits to be merged at one time,
as opposed to "Squash and merge”).

We must find a solution to improve this situation for preparing releases, I
wouldn’t mind temporary lifting necessary constraints.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] GRASS GIS: backport of CI speed updates to G83?

[Edouard Choinière via grass-dev]

Releasebranches don’t require workflows to pass now so he cannot use auto-merge.

But once tests of the OSGeo4W are started, since they don’t fail the CI, you 
could just not wait, it won’t change anything.

Edouard Choinière

Le 20 févr. 2024 à 06:45, Nicklas Larsson  a écrit :


On 20 Feb 2024, at 10:14, Markus Neteler via grass-dev 
mailto:grass-dev@lists.osgeo.org>> wrote:


In fact the slowest CI run determines how much time I have to wait
with each release step (i.e., editing VERSION file, wait 1:30hs, do
some steps, wait 1:30hs, create tarball, wait 1:30hs, reset VERSION
file, wait 1:30hs ... which is a pain).


I agree this is a case where we have limited ourself too much, with all those
required 1.5 hrs tests, approval, etc. (not even [skip ci] would work) . What 
you
would need is a (ideally) direct commit access or at least  “Rebase and merge”
option to merge (thus enable a number of commits to be merged at one time,
as opposed to "Squash and merge”).

We must find a solution to improve this situation for preparing releases, I
wouldn’t mind temporary lifting necessary constraints.
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] GRASS GIS: backport of CI speed updates to G83?

[Edouard Choinière via grass-dev]

I don’t know a lot about backporting, but if you’re talking about splitting of 
the Ubuntu workflow’s gunittest tests, I don’t see a reason why the content of 
the changes couldn’t be copied to a new commit for that branch too. Since they 
are ran less often, the tests could be split in three or even 4 jobs, where the 
%of time spent compiling the same code would be a bit higher, but the time for 
all tests to run could be smaller. The tests in the temporal folder still take 
half the total test time though.

If you are talking about the macOS arm runner, it might require some other 
changes to port. My opinion would be to leave that alone in the 8.3 as pre-arm 
code. But the same splitting could easily be done for 8.3, and split as wished.

In main, for macOS workflows, with only 20-25 mins spent on tests, with 3-4 min 
spent on compiling, I don’t see the need to split it.

Windows workflows I don’t think it’s advantageous, as more than 20 mins is 
spent before starting tests.


Edouard Choinière

> Le 19 févr. 2024 à 19:00, Markus Neteler  a écrit :
> 
> Hi Edouard,
> 
> Do you see a chance to backport the CI speed updates to G83? That would make 
> my release manager life much easier as the ever lasting waiting would 
> probably halfen ... :)
> 
> I could also cherry-pick that in case.
> 
> Best
> Markus
___
grass-dev mailing list
grass-dev@lists.osgeo.org
https://lists.osgeo.org/mailman/listinfo/grass-dev

Re: [GRASS-dev] GRASS GIS for Apple Silicon Macs

[Edouard Choinière via grass-dev]

Well, it’s not new, it’s been like that since 2020 with Big Sur and the 
introduction of M1 (before that M1 isn’t supported).  Big Sur (version 11), was 
followed by Monterey (version 12), then Ventura (version 13), then Sonoma 
(version 14), which is the beta you’re talking about and the expected General 
availability is on September 26, 2023, in a couple of days.

Even homebrew had problems at that time, so we’re not alone.


Edouard Choinière

Le 22 sept. 2023 à 14:23, Michael Barton via grass-dev 
 a écrit :

 This would be a major change by Apple and a royal PITA. I hope it is only 
something in the current beta and not in the final release (which I have not 
yet installed).

We can probably sign through OSGEO. But when I looked into signing, it seems 
difficult unless you use Apple XCode. That would be a big step back from the 
current ease of compiling Mac binaries with Conda. When I looked into it 
several years ago, I could not find any clear instructions about how to sign 
code without XCode, although it may be possible.

Michael
_

C. Michael Barton
Associate Director, School of Complex Adaptive Systems 
(https://scas.asu.edu)
Professor, School of Human Evolution & Social Change (https://shesc.asu.edu)
Director, Center for Social Dynamics & Complexity (https://complexity.asu.edu)
Arizona State University
Tempe, AZ 85287-2701
USA

Executive Director, Open Modeling Foundation 
(https://openmodelingfoundation.github.io)
Director, Network for Computational Modeling in Social & Ecological Sciences 
(https://comses.net)

personal website: http://www.public.asu.edu/~cmbarton


On Sep 22, 2023, at 10:34 AM, grass-dev-requ...@lists.osgeo.org wrote:

Date: Fri, 22 Sep 2023 17:33:54 +
From: Edouard Choini?re mailto:e@outlook.com>>
To: Nicklas Larsson mailto:n_lars...@yahoo.com>>
Cc: GRASS developers 
mailto:grass-dev@lists.osgeo.org>>
Subject: Re: [GRASS-dev] GRASS GIS for Apple Silicon Macs
Message-ID:
mailto:sa1pr12mb73447ec8cde24093c1f2bd14ef...@sa1pr12mb7344.namprd12.prod.outlook.com>>

Content-Type: text/plain; charset="utf-8"

I think I figured out an explanation. I tried to read about CI for macOS, then 
on why there aren?t a lot of CI for macOS (especially Apple Silicon). I also 
couldn?t look into the build infrastructure used for your grass macOS builds 
since they don?t seem to be available on GitHub. Is it local only?

Ok, so now to a possible explanation on why Rosetta 2 is asked to be installed.
It seems that with Apple Silicon, arm64 code needs to be signed (which is new), 
while x86_64 doesn?t, like before. I think it was mentioned in the thread that 
the app might be unsigned. So I suspect that even if a universal binary 
contains arm64 and x86_64 binaries, if it is unable to use the arm64 binary, it 
will try using the intel ones.



[Apple-Silicon-Rosetta-2-and-the-Challenges-for-Endpoint-Security-7.jpg]
Why Your macOS EDR Solution Shouldn't Be Running Under Rosetta 
2
sentinelone.com

In particular, see the part where it says:


That?s because one of the changes Apple brought in with Big 
Sur that only applies to Apple silicon Macs is that native arm64 code cannot 
execute on an M1 Mac unless it has a valid code signature.

An Apple silicon Mac doesn?t permit native arm64 code execution under any 
conditions unless a valid signature is attached. Translated x86_64 code, 
however, is not subject to this 
restriction: translated x86_64 code is permitted to execute through Rosetta with no 
signature information at all.



There?s also that thread that was linked to from my reading some Reddit threads 
(like