2017-02-17 1:18 GMT+01:00 Cirilo Bernardo <[email protected]>: > Hi Oliver, > > I agree with pretty much everything, especially removing the 3D models > from the tree. I have argued for that a number of times in the past but I > think as library dictator you may simply have to enforce it at some stage > even if angry hordes complain later. > > On Fri, Feb 17, 2017 at 10:19 AM, Oliver Walters > <[email protected]> wrote: >> Hi everyone, >> >> I am looking for some input on some thoughts I have regarding the future of >> the Kicad "standard libraries" on GitHub. I think there are some serious >> maintainability issues with the way the libraries are currently handled, and >> this will only get worse once the new eeschema / .sweet changes are rolled >> in. >> >> 1. Number of repositories >> >> There are currently 96 .pretty repos at https://github.com/KiCad . This is a >> huge number of repos to maintain especially considering that for each pull >> request, the user has to fork / branch / PR yet another repository (and then >> the librarians have to do the same to check it ). And then, you must pull >> the latest changes from (up to) 96 repos to keep all the libs up to date. >> >> There are currently 90 .lib files (symbol libraries) at >> https://github.com/KiCad/kicad-library . If the .sweet format follows the >> .pretty format we will be asking users (at least the ones to want to make >> use of the GitHub library feature) to manage over 180 separate GitHub >> repositories. >> >> That's a whole lotta repos. >> > > I would suggest putting a commit freeze on all current repos and merging > them into one footprint repo. Contributors can pull the new repo and > continue their work from there with little hassle. Individual (frozen) repos > should remain until we're happy that everything is working and any helper > scripts have been updated to work with the new structure. > > Changes to the schematic repos should be made when we get to the point > of converting existing symbols - at that point a new schematic repo should > be created and the 3D models moved. > > Some thought needs to go into how we associate data between the 3 > disjoint repos (symbols, footprints, 3D). One simple solution is to require > the repos to be cloned to specific directories at the same directory level, > for example: > > kicad_libs > +--symbols > +--footprints > +--3D > > With such an arrangement we can always define a ${KICAD_DATA_ROOT} > and make modifications to the filename search algorithms; stock symbols > and footprints can then reference a 3D model via ${KICAD_DATA_ROOT}/3D. > > Some thought needs to go into the github plugin and how it can continue > to be useful to people who do use it. I don't believe there will be any > problems encountered in merging the footprint libs which would not be > encountered with the libs separated as they currently are. Pegging specific > files to specific commits will always be a nightmare to manage and simply > isn't a problem for the kicad team to solve. One thing which may displease > users is that they will not see a "Group_A" vs "Group_B" of footprints - > it will appear that all footprints are stuffed into one single group, but this > sort of thing can all be managed. > >> 2. Library Management Tools >> >> Currently we are investigating using some CI tools to help manage pull >> requests. If we go ahead with this we need to roll out these changes to 96 >> repositories. And then maintain those CI tools. >>
I don't see how having many repos dictate that we need so many "CI tools" individually. We could potentially have the CI to generate previews or somthing similar for each pull request. > > Yup, that's just crazy. Aim for the 3, but put on your flameproof suit. > >> 3. GitHub "plugin" drawbacks >> >> AFAIK (from discussing this issue on IRC) the Kicad git plugin does not >> actually make use of git, but rather makes use of the download-as-zip >> functionality that github provides. This, to me, presents the following >> issues: >> >> i) GitHub may break this feature at any time - even by changing the format >> of the download URL! >> >> ii) Incompatible with non-github repositories - No facility to link in with >> company repositories, for e.g., that are not served on GitHub >> >> iii) Excessive bandwidth - Each time a .pretty lib is loaded, it downloads a >> zip of the entire library? If we were making use of git functionality it >> would be simple to just update the library (and store a local cache) and >> thus only download the diff rather than re-download the entire thing. IIRC there is a session cache, so it will only re-download it when you restart kicad or pcbnew. >> >> iv) Increased github load - As a follow-on from c) if Kicad attracts >> significant user base then constant download of github libs will impact >> significantly on github bandwidth. Other repositories have been shut or >> throttled for high bandwidth use. >> > > Definitely a problem. Personally I'd prefer a custom tool running over git > to help users manage selection of footprints and their revisions etc. - the > question is who has time to do that. Personally I'd never used the github > plugin as it just doesn't sit well with the way I do things. Other people > love it though. Some serious thought needs to go into the concept of > remote access of footprints etc. Actually using git rather than pulling > zip files with no history is much more inviting - we can then use commit > hashes of the files to help manage revision control of those files within > a project. (Do I update this footprint, can I roll it back, is there a new > revision available, etc.) > >> 4. User Ease-Of-Use >> >> Many users (newbies, especially) are confused by the library management. >> Other EDA packages provide links to packaged library downloads, whereas the >> initial library setup for Kicad is a lot more obtuse. >> >> Suggestions: >> >> I suggest that this problem be tackled as follows: >> >> A) Combine .pretty libs. The entire size of the pretty libraries is >> currently ~50MB which is not huge. These should be combined into a single >> repository on github which would make library management infinitely easier. >> > > Definitely. > >> B) Single repo for .sweet libs (future) - As per A) >> > > Definitely - as I said above, this should be done when we've converted and > tested the .lib files to the new format. I don't think it is best to combine the pretty libs. I think it is nice to store the data that way. We could start using a tool like "repo" (from google, used to manage android sources) which is a tool to manage many git repos. Then all that is needed when a new lib is added is to add one line in a manifest file, and it is possible to point to speficic versions in each repo making it easy to fetch a specific release. > >> C) Single repo for .3dshapes libs - Currently there is no real reason that >> the 3D shapes have to be with the symbols. In fact, they are really bloating >> the symbols library! >> > > Definitely - I've always argued that they have no business being in there and > simply waste bandwidth for people who have no interest whatsoever in VRML > models. The ability to use STEP models in the dev branch will make matters > much worse. > >> D) Versioned downloads on the website - The website should offer a libraries >> page under ./Downloads which has versioned releases of the libraries. Users >> can choose to download the entire set of footprints/symbols/3dmodels, or >> download subdirectories individually. This presents an easy introduction to >> the libraries and also means that users can be assured that unless they >> specifically download a newer version, their libraries are not going to >> change underneath them. >> > > I prefer to tag a git repository and provide instructions for retrieval. This > is > easily done with a single repository for each of symbols, footprints, and 3D; > it is an absolute nuisance with the current huge number of footprint repos > (that tagging script had better work just right ...). Github in particular > has a > "Releases" scheme in its API. Maybe that could be of some help. > >> E) As an extension of D) users who wish to contribute or be at the >> bleeding-edge of the libraries can fork the repos (3, not 186) and keep >> these up to date very easily. The 3D model lib will be large but a one-time >> download is fine as git will keep the changes up to date with very little >> bandwidth. >> >> F) Improve the github plugin - This should be turned into a proper GIT >> plugin (not a github plugin). This will have the following advantages >> i) Not reliant on github (users can manage their own git repos for e.g.) >> ii) Potential to partially check out repos (e.g. only pull a single >> subdirectory) >> >> OR >> >> G) Remove the github plugin - Library versioning might be better performed >> by the user external to Kicad. I do not rely on Kicad to manage the >> libraries for me, I do that in git externally. I know that many other users >> do this too. From an architectural point of view, I do not think that Kicad >> should "force" the user to manage their libraries in this way. Should we add >> hard-coded support for other online services too? Personally I think the >> github plugin makes life harder than any problems it solves. >> We are not really forcing users to manage their libraries in this way. It is just the default. > > I'm all for (G), but see my notes above. I don't find the github plugin useful > for me, but we have to be mindful of people who do use it. > > - Cirilo > >> TL;DR - If / when the .sweet format change goes ahead, if we add another >> ~100 github repositories, the librarians are going to go postal. >> >> Cheers, >> Oliver >> >> _______________________________________________ >> Mailing list: https://launchpad.net/~kicad-developers >> Post to : [email protected] >> Unsubscribe : https://launchpad.net/~kicad-developers >> More help : https://help.launchpad.net/ListHelp >> > > _______________________________________________ > Mailing list: https://launchpad.net/~kicad-developers > Post to : [email protected] > Unsubscribe : https://launchpad.net/~kicad-developers > More help : https://help.launchpad.net/ListHelp _______________________________________________ Mailing list: https://launchpad.net/~kicad-developers Post to : [email protected] Unsubscribe : https://launchpad.net/~kicad-developers More help : https://help.launchpad.net/ListHelp
