Hoi Stijn, first, best wishes for the new year!
On 05 Jan, 2013, at 05:18, Stijn De Weirdt wrote: > disclaimer: i was never a big fan of the auto-download/source_urls option in > EB for exact this reason, ie sometimes EB will appear to be failing, due to > external reasons (however, the augmented userfriendliness far outweighs this) A common case "for the impatient user" like me is, that you parallelize builds and you end up overwriting the downloads (eg. GCC-Cloog fighting with plain GCC). I have done it a few times already and it takes a bit to understand what the issue is. Whatever solution we may pick, it will likely address nicely this case, too. > > For instance, if you try to build DOLFIN from zero, you will quickly realize >> that that is doomed to fail, since source for MTL4/4.0.8878 is no longer >> avail. >> There are work-arounds that someone can do (eg. I now symlink the newer >> version, >> faking it to be older) but, these are all inferior to having a copy of the >> tarball. > please contribute the newer working .eb files so others don't run into this > same issue Done, along with its deps: easyconfigs/#76 >> Now, I understand that not everything will be possible but, I would really >> like we had a mirroring solution, at least for the open source software >> codes. > i'm not sure what you mean with "open source", but afaik it is mainly the > software license that defnies who can distribute what. Yeah, better clarify: normally OSS codes, as per OSI definition, allow infinite redistribution; I hereby liberally put under OSS label all such codes but further clarification may be needed. For now, let's focus on the codes which present no problem to build a combo repo from. (whatever that definition means) >> a) Have you heard of any other kind of "open source" registry project, >> perhaps something that we could ride on registering specific tarballs? > most p2p tools offer something along these lines. (but i'm still waiting for > a fuse interface ;) FUSE and any filesystem will likely give you one more interface with potential new failure modes; but, we can shop from the idea and export the data via AFS, your p2p-FUSE code and so on ;-) ie. it could be yet another interface to export the bunch of files, and that's easily doable. >> b) What technology do you think should be deployed? What are your >> preferences? >> (http, ftp, git, rsync, zsync ... whatever you think should be offered) >> > most have master/slave models, so that might not be what you are looking for > (granted, you can write scripts to keep things in sync). > a better solution would be distributed filesystem with one (or more) replica > per site. > i'm aware of a project called REDDnet/L-store that aims for something similar. > i'm not sure if accessing data from a WAN shared filesystem counts as > distributing, so that might be a way out. Do people prefer to have an exact local complete copy or rather a "subset cache"? your stance? One aspect I like about zsync or git, is that hashes make the copy mechanism irrelevant (if there is an issue, you are going to catch it). With that property on, anything goes! ie. why limit ourselves? every HPC site could pick/introduce the technology it prefers. But I agree there are important design criteria to fulfill and the picture is not 100% clear. We will likely start with something and fix as we go. >> c) Is your preference to integrate this to easybuild or, perhaps, keep it >> orthogonal? >> (eg. someone could bootstrap .local/easybuild/source with zsync & then >> let it go) > i would keep it out of EB. integration should be provided, but it is a data > management issue, EB has enough problems to deal with as is. (but i'll be > more then happy to test/contribute ;) Good. Orthogonality has the advantage of being future-proof. >> ps3. >> Ubuntu's "LTS" lineage is a good example of a commendable vendor retention >> policy. >> Somehow, not everybody around understands the universal need for LTS style >> solutions... > what do you mean? LTS is only 5 years, hardly the lifetime of any of long > running experiment ;) IMHO, long running experiments have eventually to face the fate of their obsolete hardware :-( Here is a description of good design aspects of LTS: https://wiki.ubuntu.com/LTS (I am not promoting LTS, RHEL has even longer life cycle; I just don't have its commitments). Anyway, the point above is that typically software providers are not as serious as OS vendors for backward compatibility, at the expense of user time. eg. Intel, with a 3-year window, may pull the plug of past compilers/MPI stack etc. If you know some contradicting story, let it be known... thanks, Fotis
