On 24/04/2019 19:00, Jeremy Huntwork wrote: > On Wed, Apr 24, 2019 at 4:07 AM Pierre Labastie <[email protected]> > wrote: >> >> On 23/04/2019 16:49, Jeremy Huntwork wrote: >>> On Tue, Apr 23, 2019 at 9:47 AM Pierre Labastie <[email protected]> >>> wrote: >>>> >>>> Forgotten point above: >>>> - Create an organization on github with name (tentative) automate-lfs: >>>> it will contain the jhalfs repo, of course, but could also host >>>> experimental versions of the books, with xml added (or modified) in >>>> order to make the scriptlet generation easier (LFS is already good in >>>> this respect, BLFS is not, CLFS is intermediate...). All this will >>>> depend on manpower (hmm, don't want to exclude women, of course, maybe >>>> should use "humanpower"?). Note that the original idea of ALFS (creating >>>> a DTD for xml based GNU-linux distribution builds) is not what I have in >>>> mind, hence the proposal for a different name. >>> >>> Sounds great! >>> >>> Your description brings up a few thoughts that I have been >>> contemplating for a little bit: >>> >>> 1. Do you have in mind supporting more advanced/alternative build methods? >>> >>> For example, in Mere Linux, I've replaced all chroot actions with >>> containers. Having a generic automated system that can help support or >>> drive ideas like that would be interesting. >>> >>> 2. Moving away from shell scripts? >>> >>> Shell scripts are great, and I admit I enjoy writing them. However, >>> they are hard to test. Specifically, they're really hard to unit test >>> in a meaningful way. And, of course, as the requirements and >>> complexity of the tool grows, the nicer it is to work with a more >>> modern and flexible language. >>> >>> So I've started exploring the idea of gradually swapping out the shell >>> bits with Python and doing it in a Test Driven Development style. I >>> have a little boilerplate setup that looks promising. Specifically, >>> there's a few places that I think a move to Python might really help >>> with: >>> >>> * Using a library like https://github.com/ulfalizer/Kconfiglib to >>> avoid in-source menu code. All you would need is the Config.in file. >>> * Using either built-in or third-party XML libraries, I believe we >>> could make the command extraction easier and avoid XSL stylesheets >>> entirely. >>> * Better download management, caching, verification of sources. e.g., >>> https://2.python-requests.org//en/v0.10.6/user/advanced/#asynchronous-requests >>> * Packaging, versioning and distribution becomes easier via pip and >>> setuptools. Some of the host pre-requisites would be a little less, or >>> easier to manage because pip will track and install the required >>> dependencies of the tool. >>> * Testing, of course. The boilerplate I have uses flake8 >>> (http://flake8.pycqa.org), pytest (https://docs.pytest.org) and >>> coverage (https://coverage.readthedocs.io) all driven by tox >>> (https://tox.readthedocs.io). >>> >>> 3. CI/CD? >>> >>> The move to Github opens up some doors around Continuous Integration >>> and Continuous Delivery that may be really interesting to explore. >>> Have you thought about this at all, even if in a limited way? For >>> example, having pull requests trigger automated syntax/linting checks >>> or even unit tests if we get that far. Or, periodically triggering >>> some automated builds for testing/reporting purposes. >>> >>> I could write more, I tend to have a lot of ideas. That can be both a >>> good and bad thing. :) But I'll leave it there for now. >> >> Wow, what a program :) >> >> Maybe I should just try to make it clearer what I think this organization >> should aim at. Nothing really incompatible with what you say, but maybe not >> going as far into some directions: >> >> - The main aim of the tools developed in this organization, is to take >> existing {,C,B,anything}LFS books (referred to as "the books" in the >> following), and generate scriptlets from them, with a way to run those >> scriptlets in the right order. They can serve to build a GNU linux system, >> but >> also to test the books' instructions. The scriptlets may add instructions to >> use a package manager. It may also be possible to develop a dedicated package >> management system in this organization, if it is found that the existing ones >> do not fit all the needs. >> >> - The aim is not to write the books themselves. But people in this >> organization may ask the authors of the books to add some visible or >> invisible >> (xml tags such as sect1info, or attributes such as remap or role) text that >> can be used by the tools in this organization to ease the conversion. Also, >> there is nothing forbidding people in this organization to be authors of some >> books. >> >> - Presently the tools in this organization take books written in DocBook-4.5 >> xml as input. This is not necessary: tools to accept books written in other >> markup languages may be developed. There are two conditions for that: (i) >> somebody wants to write the tools, (ii) there exist viable books written in >> this other language. Viable means there is good chance they get maintained >> for >> a long time. When a book (think HLFS) stops being maintained, all the effort >> done to write a tool to use it is just lost. >> >> Note: the above is written in a somewhat definitive manner, because it is >> easier for me than using should, would, etc everywhere. But this is only a >> proposition. Please discuss additions or removals. But do not forget that, at >> the moment, two persons, Jeremy and I, seem to want to be involved in this >> project. And none of those two persons are able to say how much time they >> will >> be able to spend on the project... > > Thanks, I appreciate hearing your thoughts on the scope. > > It seems like it answers my first question with both a 'yes' and a > 'no'. 'No', in that, by itself the project shouldn't be trying to add > alternative build techniques. However, 'yes' if it is > automating/supporting some other published and maintained > instructions, which in the future may or may not be XML source. > > Seems perfectly reasonable. > > The second two questions were more about advancing quality, which, > after reviewing the code, I think may still deserve some > consideration. For example, I couldn't get a CLFS build to work out of > the box. First, because the upstream git url is broken, but then after > pointing to a valid url, I hit a logical bug in the way it checks out > the git source. This is one example, there are a few others I've hit > as well, mostly minor and most out of the path of the typical LFS > builds. > > This isn't meant as a criticism in any way. Much of it can probably be > attributed to bit rot. (There's a pretty good list of reasons for why > this happens here: https://en.wikipedia.org/wiki/Software_rot) > > So apologies if the previous email was a little overwhelming with > ideas. The intent of the my questions around Python and CI/CD, though, > was to help identify ways to improve quality. So in that spirit, if I > may make just a few other more generic suggestions that might be > easier to tackle... > > 1. Consider reducing the current features. It seems like there's a lot > of untested/unused books or features. Or at least, ones that haven't > been tested or used for a while. And as you point out, less man power > at the moment. Energy is probably better spent on the ones that are > most used and aim to improve the usability/quality of those. Find the > 'common' baseline and make that solid. And to make that easier, it > might help to actually remove uncommonly used features. People wanting > those can always refer to previous versions. And they could always be > eventually added in again as needs/demands arise. > > 2. Consider making some useful, modular refactors. I say this because > the core code has many conditionals spread all throughout it in order > to support different designs of the books. If you modularize more and > isolate definitions of a book, it's easier to maintain quality core > code. For example, instead of a 'get_book()' function that has > conditionals for all different cases, make a 'get_svn_book($url)' and > 'get_git_book($url)' and then call the appropriate function with the > appropriate arguments after collecting in one spot the needs of a > book. Do this for all the cases of a generic feature that a book may > want. > > 3. Unit test. Similarly, find ways to independently test small > sections or 'units' of your code. As per the above example, each > function is generic enough that it could work with any upstream > source, and you can have tests that prove that. When it come time to > actually use in real life with a given book, there is confidence it > works as advertised. This is part of what prompted the suggestion for > Python. It's difficult, though maybe not impossible, to do this with > shell. > > I'd be happy to help walk you through some examples if you like, and I > think I might be able to keep contributing some code here and there. > Or if this is all too much to take in and I'm just being annoying, I'm > also fine just continuing to lurk. :) >
You're all but annoying! If refactoring is what you are interested in, please go ahead. I've maintained the code by adding what I needed at a given time. But the code is certainly in big need of some more modular approach. I've never taken the time to do refactoring, except when modifying big chunks of code (mainly in the BLFS part). Also, I've stopped testing CLFS when I realized that the book was not updated often enough to be buildable on more recent platforms. So I would have spent more time trying to fix the book than on the jhalfs code. CLFS has recently moved to github. I'm not sure the development has restarted though. Do you want to do the update for the CLFS git urls? I'm rather busy with the move: need to figure out how to send notifications to the linuxfromscratch lists, and how to couple with trac on the wiki... Pierre -- http://lists.linuxfromscratch.org/listinfo/alfs-discuss FAQ: http://www.linuxfromscratch.org/faq/ Unsubscribe: See the above information page
