Hi Hugo, On Thu, 27. Apr 2017 at 18:23:35 +0200, Hugo Mercier wrote: > We've setup a compilation / CI infrastructure based on Gitlab CI that > triggers Virtual box virtual machines for the actual compilation and > packaging.
Sounds like something on my "one-fine-day" TODO list for long. > So far, we noticed some issues with the current osgeo4w project and > infrastructure: > - the main repository seems heavily loaded and hard to reach sometimes limited disk space is also a constantly reoccuring issue (although not actually visible from the outside) > - the build of the entire distribution is hard to reproduce Indeed. > - the points listed above also result in a situation where there is a mix of > compiler versions used for packages. Which should be avoided > (http://siomsystems.com/mixing-visual-studio-versions/) That's certainly a fragile issue. We don't track which compilers were used to produce the individual packages - so it's also unclear on which version of the msvcrt they depend on (msvcrt used to have several, although it was meanwhile split up, the dependencies were still not refined everywhere). But I think it's unavoidable sometimes - I wouldn't like to have three GDALs and all their dependencies for QGIS 2 (msvc2010/py2), 3 (msvc2015/py3) and GRASS (MinGW) in OSGeo4W (not to mention the GRASS plugin where all three would meet anyway; also newer version of GDAL will not work with msvc2010 because of c++11). I think it's meant to work and it does, still it should be avoided where possible. Others: * not listing of build dependencies (libraries and tools) * no build recipes for everything * those that exist don't have a form that would allow automatic invocation in a uniform way * source packages don't contain sources nor automatically usable pointers to download them. So there's much to do on that front before all this can work. Also there are some proprietary SDKs (esp. for use with GDAL: ECW, MrSID, FileGDB etc.) that I'm not sure you are allowed to make them public in such a setup - although you are allowed to ship the redistributable runtime parts. > We actually hit a problem probably related to that > (https://trac.osgeo.org/osgeo4w/ticket/529) and that could mean to recompile > gdal and all its dependencies with msvc 2015. I think that is just due to the fact that numpy wasn't installed, when the package was built. That that problem is not already fixed, is just because I'm working on transitioning building gdal with msvc2015 (needed for 2.2). Having all this in place would have certainly helped with this of course. The nightly should already work again - 2.2 not yet. > So here are some propositions: > - find a way to add a mirroring system for the osgeo4w distribution. The idea > is to register a list of mirrors that would receive a copy of files > when they are uploaded to the main repository. AFAIK there is also a OSUSL file server/proxy that could be set in front download.osgeo.org to help with this - bandwidth should be better. Not sure if that extra mirror architecture would still be necessary. > - share access to our gitlab-ci infrastructure and vbox VMs (and allow > it to be copied somewhere else if needed). The idea would be to have a > unique repository with all package "sources" and all the machinery needed > to trigger a build of the whole distribution or of some selected packages > on a known environment. Not sure if that also would be candidates for osgeo hosting - but self hosting is probably easier and more flexible (also a reason why I didn't push this). Perhaps leased hosted servers are even be cheaper than hosting "own" hardware at OSUSL. > Regarding the issue of mixing compiler versions, we apparently still > need to compile python2.7 modules with msvc 2010 (which is the version > used to compile python27 and all wheel binaries). > Apart from that (which may imply to maintain two versions of the same > libraries), do you see problems if all packages get compiled with the > same (last) version of msvc (2015) ? For GRASS this will certainly not work - that requires MinGW - no version of msvc will work for that matter. > Are there any interests for that work? Raise your hand if so, and raise > both hands if you could contribute (time or funds) :) we also still need > to find a way to fund this alternative osgeo4w / CI server. Both hands full ;) - but consider them raised. Although I have somewhat mixed feelings about this. Except for Martin Landa who took over GRASS of course, I have been pretty much on my own on this for a while... Jürgen -- Jürgen E. Fischer norBIT GmbH Tel. +49-4931-918175-31 Dipl.-Inf. (FH) Rheinstraße 13 Fax. +49-4931-918175-50 Software Engineer D-26506 Norden http://www.norbit.de
signature.asc
Description: PGP signature
_______________________________________________ osgeo4w-dev mailing list osgeo4w-dev@lists.osgeo.org https://lists.osgeo.org/mailman/listinfo/osgeo4w-dev