You could probably embed a custom sort sub into a tool like efsdeploy, but for runtime dependencies in things like EFS::Perl::Depends, the algorithm needs to be supported internally, so that we can keep the contents of the metadata.conf files minimal. Remember this runs for EVERY invocation of our EFS-based perl builds, unless you use -f.
I'm just adding these on a case by case basis for now, as the use cases arise. On Fri, Jan 13, 2012 at 2:24 PM, Gay, Jerry <[email protected]> wrote: > based on the detail earlier in the thread, this seems to make sense. the > question i'm left with is what if there's a custom sort order for this > project? maybe something that uses unicode characters, or right-to-left sort > order, or .... > > i don't have a concrete example (it would really make thinking about this > easier) but i wonder if there should be a way to specify a custom sort sub > that can be run as a callback. it doesn't make sense to put that in each > release, though, since this is really a project-level concept. > > so, is there a place to put a custom sort sub per project, or will it have to > be baked in to deploy-config? > ~jerry > > -----Original Message----- > From: [email protected] > [mailto:[email protected]] On Behalf Of Phillip Moore > Sent: Thursday, 12 January, 2012 16:31 > To: EFS core development list > Subject: Re: [EFS-dev] Project sorting algorithms > > Better idea -- write this into the metadata.conf file for each release. > > [config] > sort = gnu | perl | text | whatever > > It's in the efsdeploy config rules, so just write it out when the releases > are installed. We're already parsing the metadata.conf files > in EPD, so this information will be readily available. This also > means I can get this implemented very simply, and I should have a solution my > the middle of next week. > > On Thu, Jan 12, 2012 at 5:45 PM, Phillip Moore <[email protected]> > wrote: >> OK, I've made some progress here, but the results are not very good. >> >> I tested out a patch to efs-perl that treats versions as v-strings, >> but that's just a trade off. That algorithm can sort HTML-Templates >> versions, but then it breaks on Module-Build and DateTime. There >> doesn't not exist a single algorithm that correctly implement version >> sorting automatically. I am convinced we have no choice here but to >> manage this with project-specific metadata. >> >> Now, as noted below, it's easy to manage this in efsdeploy's config >> data, however that information is intended to be a BUILD-time >> dependency ONLY. I do not, under any circumstances, want that data >> to have RUNTIME-dependencies. >> >> The only realistic solution I can see is to have efs-perl read this >> information from something like: >> >> /efs/dist/perl5/EFS-Perl-Config/current >> >> IOW, we'll still use an autorelease (the EFS 3 replacement for incr >> releases -- they kick ass), but it will be compiled from the >> deploy-config data, and distributed separately. In reality, we will >> only need to specify a special sort order as an exception in the perl5 >> metaproj, so I expect this data to change fairly infrequently. The >> deploy-config stuff, OTOH, will change a LOT, and this will allow us >> to update the deploy-config stuff as often as we want, with no risk to >> your production systems. >> >> This seems reasonable for several reasons. First, remember the edge >> condition this applies to. When EPD is expanding a dependency tree, >> and there are two DIFFERENT releases specified at the same level of >> the dependency tree, then and only then does this code kick in. >> Honestly, we're splitting hairs here, but it's important to get this >> right. >> >> I'm working through a rebuild of my perl5/core release, to introduce >> the perl5/EFS-Perl-Indirect project, and this is a good time to code >> something up for EFS-Perl-Config, which just be a couple of hooks in >> some efsdeploy rules. It should be straight forward to have efs-perl >> query this information. >> >> On Mon, Jan 9, 2012 at 12:03 PM, Phillip Moore >> <[email protected]> wrote: >>> I'm in the middle of totally reworking how dependencies are managed >>> in the efsdeploy code, and I've finally run into a problem I've been >>> avoiding for a while. How do we manage the metadata that specifies >>> how a project's releases are sorted? >>> >>> This is a LOT harder than you might think. First of all, the main >>> problem is that we need this chunk of metadata in several places. >>> The first time this problem reared it's head was EFS::Perl::Depends, >>> which was really the first time we had to think about this. The >>> first major release of EPD got it totally wrong, and used >>> Sort::Versions to sort everything, which is really only correct for a >>> subset of the CPAN distros. In V2 I fixed this by switching to >>> version objects, which is the correct default, but then you discover >>> that there are distros like HTML-Template that use versions which do >>> NOT sort correctly with version.pm. Those, you have to use Sort::Versions >>> for. >>> >>> This defines an important requirement for managing this information: >>> it has to be available to EPD, and be CHEAP. EFS database >>> attributes would be a nice place to manage this, but we can't have >>> sitecustomize.pl querying a database for every perl invocation. >>> Right now, EPD is dependent only on the metadata.conf text files >>> found in each release. Wherever we end up managing this information, >>> it will have to be written out into these metadata.conf files by >>> efsdeploy (easy). >>> >>> RIght now, I see 3 distinct sorting algorithms. Here's the keywords >>> I'll be using in the code to describe them: >>> >>> [*] gnu (Sort::Versions) >>> >>> Every single project on gnu.org uses a versioning scheme that sorts >>> correctly with Sort::Versions, and this will probably work for most >>> open source projects that use a major.minor.subminor release >>> convention. >>> >>> [*] perl (version objects) >>> >>> MOST of CPAN can be sorted using version.pm objects, with a few >>> exceptions we'll have to configure, as described above. >>> >>> [*] text (natural perl "sort" order) >>> >>> This is how you would sort, say, EFS autoreleases (YYYYMMDD-HHMMSS), >>> for example, or anything that uses a convention that can be sorted as >>> plain text. Autoreleases are a stupid example, though -- you always >>> access them (BY DESIGN) through the current releasealias. >>> >>> So, where do we manage this data? We want to be able to specify it >>> in ONE place, and then have it propogate wherever else it is needed >>> as a by-product of the automated efsdeploy workflow. Seems obvious >>> to me this project-specific information that should be in the >>> deploy-config rules, therefore in globals.conf: >>> >>> [config] >>> sort = gnu | perl | text >>> >>> This will allow us to specify system defaults, so the gnu system >>> would obviously default "gnu", perl5 would default to "perl". >>> >>> This would make things easy for efsdeploy, which is doing the >>> expansions per-project. However, for EFS::Perl::Depend (the only >>> other consumer of this information right now), it will be a little >>> trickier, since it currently just assumes "perl" and falls back on >>> "gnu" (i.e. using the Sort::Versions algorithm) only when the >>> releases can't be converted into version objects. >>> >>> There are two approaches I could take with EPD. First, since we now >>> (as of last week) have all the project-specific efsdeploy rules >>> available in the various */deploy-config-* projects, it might be >>> possible for EPD to query that data. The second approach would be >>> to only worry about this when we have to. For example, if a project >>> only has one release, then there's nothing to sort. Only if a >>> project has two or more releases do we need to query the sort >>> algorithm. In this case, we could read the metadata.conf for all >>> the releases, make sure they are the same and then sort. if the >>> releases support different algorithms, abort, and require the >>> dependency to be specified manually. >>> >>> Now, with either solution, we're looking at more filesystem overhead. >>> I'm leaning towards this algorithm: >>> >>> If project-specific efsdeploy rules exist, then use that sort >>> algorithm If not, use the current algorithm (try version.pm, fall >>> back on S::V) >>> >>> I think the IO overhead, as long as we memoize the values for each >>> project, should prove minimal for EPD. >>> >>> If anyone's watching, you'll see some commits over the next few days >>> that implement all of this. > _______________________________________________ > EFS-dev mailing list > [email protected] > http://mailman.openefs.org/mailman/listinfo/efs-dev > > ---------------------------------------------------------------------- > This message w/attachments (message) is intended solely for the use of the > intended recipient(s) and may contain information that is privileged, > confidential or proprietary. If you are not an intended recipient, please > notify the sender, and then please delete and destroy all copies and > attachments, and be advised that any review or dissemination of, or the > taking of any action in reliance on, the information contained in or attached > to this message is prohibited. > Unless specifically indicated, this message is not an offer to sell or a > solicitation of any investment products or other financial product or > service, an official confirmation of any transaction, or an official > statement of Sender. Subject to applicable law, Sender may intercept, > monitor, review and retain e-communications (EC) traveling through its > networks/systems and may produce any such EC to regulators, law enforcement, > in litigation and as required by law. > The laws of the country of each sender/recipient may impact the handling of > EC, and EC may be archived, supervised and produced in countries other than > the country in which you are located. This message cannot be guaranteed to be > secure or free of errors or viruses. > > References to "Sender" are references to any subsidiary of Bank of America > Corporation. Securities and Insurance Products: * Are Not FDIC Insured * Are > Not Bank Guaranteed * May Lose Value * Are Not a Bank Deposit * Are Not a > Condition to Any Banking Service or Activity * Are Not Insured by Any Federal > Government Agency. Attachments that are part of this EC may have additional > important disclosures and disclaimers, which you should read. This message is > subject to terms available at the following link: > http://www.bankofamerica.com/emaildisclaimer. By messaging with Sender you > consent to the foregoing. > _______________________________________________ > EFS-dev mailing list > [email protected] > http://mailman.openefs.org/mailman/listinfo/efs-dev _______________________________________________ EFS-dev mailing list [email protected] http://mailman.openefs.org/mailman/listinfo/efs-dev
