Hi Oliver, I agree with pretty much everything, especially removing the 3D models from the tree. I have argued for that a number of times in the past but I think as library dictator you may simply have to enforce it at some stage even if angry hordes complain later.
On Fri, Feb 17, 2017 at 10:19 AM, Oliver Walters <[email protected]> wrote: > Hi everyone, > > I am looking for some input on some thoughts I have regarding the future of > the Kicad "standard libraries" on GitHub. I think there are some serious > maintainability issues with the way the libraries are currently handled, and > this will only get worse once the new eeschema / .sweet changes are rolled > in. > > 1. Number of repositories > > There are currently 96 .pretty repos at https://github.com/KiCad . This is a > huge number of repos to maintain especially considering that for each pull > request, the user has to fork / branch / PR yet another repository (and then > the librarians have to do the same to check it ). And then, you must pull > the latest changes from (up to) 96 repos to keep all the libs up to date. > > There are currently 90 .lib files (symbol libraries) at > https://github.com/KiCad/kicad-library . If the .sweet format follows the > .pretty format we will be asking users (at least the ones to want to make > use of the GitHub library feature) to manage over 180 separate GitHub > repositories. > > That's a whole lotta repos. > I would suggest putting a commit freeze on all current repos and merging them into one footprint repo. Contributors can pull the new repo and continue their work from there with little hassle. Individual (frozen) repos should remain until we're happy that everything is working and any helper scripts have been updated to work with the new structure. Changes to the schematic repos should be made when we get to the point of converting existing symbols - at that point a new schematic repo should be created and the 3D models moved. Some thought needs to go into how we associate data between the 3 disjoint repos (symbols, footprints, 3D). One simple solution is to require the repos to be cloned to specific directories at the same directory level, for example: kicad_libs +--symbols +--footprints +--3D With such an arrangement we can always define a ${KICAD_DATA_ROOT} and make modifications to the filename search algorithms; stock symbols and footprints can then reference a 3D model via ${KICAD_DATA_ROOT}/3D. Some thought needs to go into the github plugin and how it can continue to be useful to people who do use it. I don't believe there will be any problems encountered in merging the footprint libs which would not be encountered with the libs separated as they currently are. Pegging specific files to specific commits will always be a nightmare to manage and simply isn't a problem for the kicad team to solve. One thing which may displease users is that they will not see a "Group_A" vs "Group_B" of footprints - it will appear that all footprints are stuffed into one single group, but this sort of thing can all be managed. > 2. Library Management Tools > > Currently we are investigating using some CI tools to help manage pull > requests. If we go ahead with this we need to roll out these changes to 96 > repositories. And then maintain those CI tools. > Yup, that's just crazy. Aim for the 3, but put on your flameproof suit. > 3. GitHub "plugin" drawbacks > > AFAIK (from discussing this issue on IRC) the Kicad git plugin does not > actually make use of git, but rather makes use of the download-as-zip > functionality that github provides. This, to me, presents the following > issues: > > i) GitHub may break this feature at any time - even by changing the format > of the download URL! > > ii) Incompatible with non-github repositories - No facility to link in with > company repositories, for e.g., that are not served on GitHub > > iii) Excessive bandwidth - Each time a .pretty lib is loaded, it downloads a > zip of the entire library? If we were making use of git functionality it > would be simple to just update the library (and store a local cache) and > thus only download the diff rather than re-download the entire thing. > > iv) Increased github load - As a follow-on from c) if Kicad attracts > significant user base then constant download of github libs will impact > significantly on github bandwidth. Other repositories have been shut or > throttled for high bandwidth use. > Definitely a problem. Personally I'd prefer a custom tool running over git to help users manage selection of footprints and their revisions etc. - the question is who has time to do that. Personally I'd never used the github plugin as it just doesn't sit well with the way I do things. Other people love it though. Some serious thought needs to go into the concept of remote access of footprints etc. Actually using git rather than pulling zip files with no history is much more inviting - we can then use commit hashes of the files to help manage revision control of those files within a project. (Do I update this footprint, can I roll it back, is there a new revision available, etc.) > 4. User Ease-Of-Use > > Many users (newbies, especially) are confused by the library management. > Other EDA packages provide links to packaged library downloads, whereas the > initial library setup for Kicad is a lot more obtuse. > > Suggestions: > > I suggest that this problem be tackled as follows: > > A) Combine .pretty libs. The entire size of the pretty libraries is > currently ~50MB which is not huge. These should be combined into a single > repository on github which would make library management infinitely easier. > Definitely. > B) Single repo for .sweet libs (future) - As per A) > Definitely - as I said above, this should be done when we've converted and tested the .lib files to the new format. > C) Single repo for .3dshapes libs - Currently there is no real reason that > the 3D shapes have to be with the symbols. In fact, they are really bloating > the symbols library! > Definitely - I've always argued that they have no business being in there and simply waste bandwidth for people who have no interest whatsoever in VRML models. The ability to use STEP models in the dev branch will make matters much worse. > D) Versioned downloads on the website - The website should offer a libraries > page under ./Downloads which has versioned releases of the libraries. Users > can choose to download the entire set of footprints/symbols/3dmodels, or > download subdirectories individually. This presents an easy introduction to > the libraries and also means that users can be assured that unless they > specifically download a newer version, their libraries are not going to > change underneath them. > I prefer to tag a git repository and provide instructions for retrieval. This is easily done with a single repository for each of symbols, footprints, and 3D; it is an absolute nuisance with the current huge number of footprint repos (that tagging script had better work just right ...). Github in particular has a "Releases" scheme in its API. Maybe that could be of some help. > E) As an extension of D) users who wish to contribute or be at the > bleeding-edge of the libraries can fork the repos (3, not 186) and keep > these up to date very easily. The 3D model lib will be large but a one-time > download is fine as git will keep the changes up to date with very little > bandwidth. > > F) Improve the github plugin - This should be turned into a proper GIT > plugin (not a github plugin). This will have the following advantages > i) Not reliant on github (users can manage their own git repos for e.g.) > ii) Potential to partially check out repos (e.g. only pull a single > subdirectory) > > OR > > G) Remove the github plugin - Library versioning might be better performed > by the user external to Kicad. I do not rely on Kicad to manage the > libraries for me, I do that in git externally. I know that many other users > do this too. From an architectural point of view, I do not think that Kicad > should "force" the user to manage their libraries in this way. Should we add > hard-coded support for other online services too? Personally I think the > github plugin makes life harder than any problems it solves. > I'm all for (G), but see my notes above. I don't find the github plugin useful for me, but we have to be mindful of people who do use it. - Cirilo > TL;DR - If / when the .sweet format change goes ahead, if we add another > ~100 github repositories, the librarians are going to go postal. > > Cheers, > Oliver > > _______________________________________________ > Mailing list: https://launchpad.net/~kicad-developers > Post to : [email protected] > Unsubscribe : https://launchpad.net/~kicad-developers > More help : https://help.launchpad.net/ListHelp > _______________________________________________ Mailing list: https://launchpad.net/~kicad-developers Post to : [email protected] Unsubscribe : https://launchpad.net/~kicad-developers More help : https://help.launchpad.net/ListHelp
