[gdal-dev] Code sprint Korea, reformat sources tree
Hi everybody, During the code sprint in FOSS4G 2015 (Korea, Seoul) I plan to start refactoring Cmake for GDAL (everybody are welcome http://2015.foss4g.org/programme/code-sprint/). This is good starting point to try release an idea to reformat source tree (combine drivers on some principles - raster/vector/raster+vector). I digging the mailing list, but didn't found discussion started by Even about this. Also we have new type of drivers - network. So, how it'll be best to organise sources? This can be not only drivers, but the whole source tree. How should the ideal GDAL source tree looks like? Also I plan: 1. Move all internal libraries (zlib, libtiff, libjpeg, etc.) to separate github repos to use CMake ExternalProject feature. 2. Remove any other building systems 3. Try CTest for testing As for me the ideal structure should looks like: + apps + autotests + bindings + core + port + ogr + gcore + cmake + data + docs + doxygen + readme + drivers + raster + vector + network + combined + CMakeLists.txt + LICENSE So, at the root of sources tree we will have only 8 folders and 2 files. -- Best regards, Dmitry ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Code sprint Korea, reformat sources tree
Dmitry, > > During the code sprint in FOSS4G 2015 (Korea, Seoul) I plan to start > refactoring Cmake for GDAL (everybody are welcome > http://2015.foss4g.org/programme/code-sprint/). This is good starting > point to try release an idea to reformat source tree (combine drivers on > some principles - raster/vector/raster+vector). I digging the mailing > list, but didn't found discussion started by Even about this. Regarding unified drivers, it was a bit mentionned in https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification . Basically the PCIDSK drivers have been merged in frmts/pcidsk, the PDF ones in frmts/pdf. And the raster side of GPKG has been added to the existing ogr/ogrsf_frmts/geopackage Potential changes on the tree structure were left out in the "Potential changes that are *NOT* included in this RFC" paragraph. > Also we have > new type of drivers - network. So, how it'll be best to organise sources? > This can be not only drivers, but the whole source tree. How should the > ideal GDAL source tree looks like? > > Also I plan: > 1. Move all internal libraries (zlib, libtiff, libjpeg, etc.) to > separate github repos to use CMake ExternalProject feature. Just to give some context: the point for the internal libraries was to have a no-brainer way of building GDAL without any prerequisite. - internal zlib is identical to its upstream v1.2.3 AFAIK - internal libtiff: most files are identical to libtiff CVS, but a few ones (tiffconf.h, tif_config.h) have been modified for integration with GDAL CPL, and tif_vsi.c is GDAL specific (I/O implementation) + a build time hack for TIFF JPEG 12 bit support - internal libjpeg is mostly upstream libjpeg v6b + one patch. There's also the build hack for libjpeg12 > 2. Remove any other building systems That sounds ambitious. Given the complexity and maturity of our current build systems, I guess this would take some time to have CMake catch up. > 3. Try CTest for testing What do you think it will bring w.r.t our current testing system ? Do we want to be dependant of a particular build system for our tests ? Regarding testing, I've somehow understood Kurt had mentionned plans for a "gdalautotest2" Regarding all the above, I assume you mean in a fork of yours ? > > As for me the ideal structure should looks like: > + apps > + autotests > + bindings > + core >+ port >+ ogr >+ gcore gnm core would go here too ? > + cmake > + data > + docs >+ doxygen >+ readme > + drivers >+ raster >+ vector >+ network >+ combined > + CMakeLists.txt > + LICENSE > > So, at the root of sources tree we will have only 8 folders and 2 files. Is the churn in the tree structure worth the effort ? Be aware that there are a number of interdependencies between drivers, so this might require fixing a number of source files. What advantages do you see in a new structure ? I'm afraid that if you want to change multiple things at a time (build system, testing mechanisms, tree structure), you will never manage to get a working result. Incremental approaches when feasible are less risky (but admitedly involve potentially a larger cumulated effort). Even -- Spatialys - Geospatial professional services http://www.spatialys.com ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
Re: [gdal-dev] Code sprint Korea, reformat sources tree
Hi Even, 05.09.2015 17:10, Even Rouault пишет: Dmitry, During the code sprint in FOSS4G 2015 (Korea, Seoul) I plan to start refactoring Cmake for GDAL (everybody are welcome http://2015.foss4g.org/programme/code-sprint/). This is good starting point to try release an idea to reformat source tree (combine drivers on some principles - raster/vector/raster+vector). I digging the mailing list, but didn't found discussion started by Even about this. Regarding unified drivers, it was a bit mentionned in https://trac.osgeo.org/gdal/wiki/rfc46_gdal_ogr_unification . Basically the PCIDSK drivers have been merged in frmts/pcidsk, the PDF ones in frmts/pdf. And the raster side of GPKG has been added to the existing ogr/ogrsf_frmts/geopackage Potential changes on the tree structure were left out in the "Potential changes that are *NOT* included in this RFC" paragraph. I plan to experiment with this and if I get good results, RFC will be written. Also we have new type of drivers - network. So, how it'll be best to organise sources? This can be not only drivers, but the whole source tree. How should the ideal GDAL source tree looks like? Also I plan: 1. Move all internal libraries (zlib, libtiff, libjpeg, etc.) to separate github repos to use CMake ExternalProject feature. Just to give some context: the point for the internal libraries was to have a no-brainer way of building GDAL without any prerequisite. - internal zlib is identical to its upstream v1.2.3 AFAIK - internal libtiff: most files are identical to libtiff CVS, but a few ones (tiffconf.h, tif_config.h) have been modified for integration with GDAL CPL, and tif_vsi.c is GDAL specific (I/O implementation) + a build time hack for TIFF JPEG 12 bit support - internal libjpeg is mostly upstream libjpeg v6b + one patch. There's also the build hack for libjpeg12 I only plan to move this internal libraries in separate repos, not to link official ones. So this is only give more structured sources tree. 2. Remove any other building systems That sounds ambitious. Given the complexity and maturity of our current build systems, I guess this would take some time to have CMake catch up. Yes, certainly. But anyhow current CMake branch not fully consistent for current build system. So this have to be done. 3. Try CTest for testing What do you think it will bring w.r.t our current testing system ? Do we want to be dependant of a particular build system for our tests ? Regarding testing, I've somehow understood Kurt had mentionned plans for a "gdalautotest2" This is only subject of experiments. Let's try CTest and see if it fits. Regarding all the above, I assume you mean in a fork of yours ? Yes. All experiments will be on forked GDAL in separate branch. As for me the ideal structure should looks like: + apps + autotests + bindings + core + port + ogr + gcore gnm core would go here too ? Yes + cmake + data + docs + doxygen + readme + drivers + raster + vector + network + combined + CMakeLists.txt + LICENSE So, at the root of sources tree we will have only 8 folders and 2 files. Is the churn in the tree structure worth the effort ? Be aware that there are a number of interdependencies between drivers, so this might require fixing a number of source files. What advantages do you see in a new structure ? 1. More ease to understand sources tree for novice. 2. More useful for CMake macro. In current release there are a lot of hardcoded things. Macro give more flexibility. 3. More ease to add some new check needed by separate drivers. 4. More configurable (ease selected depended libraries installed in OS, or should be loaded via ExternalProject), more dependence checks. 5. May be CPack using in future to create distros. I'm afraid that if you want to change multiple things at a time (build system, testing mechanisms, tree structure), you will never manage to get a working result. Incremental approaches when feasible are less risky (but admitedly involve potentially a larger cumulated effort). Yes, you may be right. But it seems to me that current Cmake version is too complicated than it can be. If Ican improve it it'll solve lot of problems, if not - ok this will be only an unsuccessful experiment. Even I do not insist, maybe it's a crazy idea. But as was the discussion of unification, it seemed to me that this worth trying during improvements Cmake build system. Best regards, Dmitry ___ gdal-dev mailing list gdal-dev@lists.osgeo.org http://lists.osgeo.org/mailman/listinfo/gdal-dev
