chris you have *exactly* mirrored my feelings as a longtime zope2 user/application developer getting in touch with plone3/five. now after a year digging in it I still am not free of it
thanks a lot robert Chris McDonough schrieb: > I realize now that I've neglected to give sufficient reasoning for why less > granular packaging would be a good thing. > > I've noticed that there's a common theme in Zope development, software, and > packages that I can only describe as "power law development" or "turtles all > the > way down". It's a bit of an antipattern, unfortunately. > > I'll provide an example by way of Zope-3-the-appserver. In an application > that > uses Zope-3-the-appserver, many individual subframeworks will be used. For > example, there is a traversal subframework, a security subframework, a > cataloging subframework, and so on. > > Each of these subframeworks acts as a logical unit, and through the magic of > the > component architecture, each can be replaced wholesale by registering some > adapter. However, each of these subframeworks tend to also have settings > that > can be configured. For example, individual traversal steps for certain types > of > objects can be overridden by registering an adapter that *configures* the > subframework. In the case of Zope 3, we have a traversal situation where the > larger traversal subframework can either be replaced wholesale via an adapter > registration or extended piecemeal via other adapter registrations. > > The problem is that the mechanism to *replace* the subframework is the same > as > the mechanism to *configure* it (both are done via adapter registration, > sometimes even in the same file). This is theoretically fine. But in > reality, > it's tremendously hard for someone just walking up to a complex system like > Zope > 3 to discern adapter registrations that replace subsystems from those which > merely configure subsystems. An inability to discern the difference leads to > situations where people just "don't get the joke" and try to wiggle wires to > configure-to-death a existing subsystem that's clearly suboptimal for their > use > case instead of just replacing it wholesale with a much simpler custom > policy. > They just don't know it was engineered to be replaced. So they keep adding > more > configuration code to the existing subframework to handle various 1% edge > cases. > Often this code makes the subframework tremendously complex, and the > subframework grows inappropriate dependencies along the way. *Sometimes* the > situation gets so confusing for a new user, they just quit and go use > something > else. > > This is a pattern that happens over and over again in Zope development. In > my > personal opinion, the original error was trying to make the subframework > configurable at all. Instead, the subframework should be replaceable easily, > but it should itself be relatively rigid. At very least, for subframeworks > that > really do require extra configuration (should be very few), this > configuration > should be done via highly specialized ZCML directives (or grokkers), as > opposed > to some very general adapter registration that can't be easily discerned from > other adapter registrations by a newbie. > > If the subframeworks were more rigid (but replaceable), the *intent* of the > subframework author could be more easily discerned, and fewer people would > fall > into the trap of adding more configuration code to a subframework instead of > just replacing it entirely. And fewer people would just walk away in > frustration. > > What does this have to do with packaging? Well, currently, there's a > dizzying > number of packages that make up "the ZTK" (nee "Zope 3"). Each of these > packages is a pure peer of all others in a PyPI listing with no real way to > get > a sense of their relative importance other than performing a linear audit. > Even > if a user *does* do a linear search of all of them, it's still awful hard to > discern for some new user which ones are "important", and which ones just > happen > to exist by some inequity of history without trying to install it. The user > needs to gain some holistic knowledge of the system in order to discern the > important bits from these historical inequities. > > Most new users understandably just walk away from *all* Zope packages before > they gain this knowledge; it's just too hard for them to tell the difference > between the truly important and reusable bits and the stuff that just happens > to > be packaged up and released but which is useless outside of some highly > specific > context. In effect, we just don't communicate *intent* very effectively in > our > current packaging structure. > > In my opinion, this is why a lot of Python developers who are otherwise very > smart have given up on trying to use Zope packages. The time required to > figure > out which ones are useful and which ones aren't is just too high. It's way > "easier" for them to write them all off wholesale and just write what they > need > from scratch or use somebody else's software. Good developers tend to like > small, useful libraries and frameworks. > > We can ameliorate the situation in a few ways: > > - We can reduce the number of distributions. > > - We can make each current Zope package distribution independently useful. > > My suggestion is that we do the former first in order to communicate *current > intent* (the "state of reality right now"). We can do the latter after this > in > pieces, hopefully aided by some new developers we've picked up by making it > easier to find useful stuff. > > Once we deflate our current set of packages down to a reasonable number, the > packages listed in PyPI will immediately start to reflect "the state of > reality > right now". As a result, we'll hopefully be able to get some new blood in > the > form of new developers that use the smaller bits outside Zope to help us > tease > the truly independent pieces out of the larger pile. If we do this, at no > time > after the deflation will PyPI listings ever as badly advertise "the state of > reality" as it is advertised right now, and the community will hopefully > again > start to grow. > > - C > > > On 5/11/09 11:11 AM, Martijn Faassen wrote: >> Hey, >> >> Chris McDonough wrote: >> >> > Instead, I have argued for promoting packages that have some life of >> > their own (independent of the rest of the pile) into subprojects that >> > have their own release cycles. >> >> > Then outside projects such as Plone and Grok could depend on >> > independent versions of such packages, giving them slightly more >> > flexibility than requiring a "version of the ZTK". >> >> We already have that flexibility today. To me, the utility of a release >> of version numbers in the ZTK does not at all exclude the potential to >> evolve the packages to more independent sub-projects. >> >>> Given that this suggestion has been met with skepticism, let me try another >>> tact. >> I think that's an inaccurate description of the response you got. I'm >> quite positive about trying to give as many packages as possible a life >> of their own. I don't think you got anyone else arguing against this >> point of view. >> >> I'm also quite positive that some packages are: >> >> * useful as independently distributed packages >> >> * only make sense in a Zope 3 or a Grok or a Zope 2 context, i.e. they >> depend on a significant set of Zope packages. >> >> I'd like to get out of this paradigm: >> >> * the Zope packages are independent sub-projects. >> >> * therefore we cannot distribute a list of versions that work best together. >> >> And this one: >> >> * if we distribute a KGS of anything >> >> * packages in that anything aren't independently reusable automatically >> and should be merged into a ball. >> >> I'd also like to get out of the following paradigm: >> >> * the Zope packages are not independently reusable yet >> >> * therefore we should distribute them all together >> >> We're in a grey area. Some package are here, some packages are there, >> some are in between. Some packages build on other packages, but have >> clear dependency structures. Some don't have clear dependency >> structures. Some have better documentation and better focus than others. >> >> If there is to be a merging of code together, then I propose we continue >> the project where the ZMI code is merged into one or a few packages. We >> can also investigate merging 2 or 3 packages together where it seems to >> make sense, or simply moving code between packages (some code needs to >> go down the dependency list, some up). >> >>> Instead of thinking about it this way, could we think about it as >>> *deflating* the current set of zope.* packages that do not currently have a >>> meaningful life of their own into a single setuptools distribution named >>> "ZTK". >>> This package would include most everything in zope.app*, plus things like >>> zope.server, zope.publisher, and other things that just aren't useful >>> outside of >>> Zope-the-appserver, or which currently just depend on "too much". >> -1 >> >> This would make it a lot harder to: >> >> * clean up dependency relationships with the goal of creating more >> reusable code. We'd all hide them in a sumo ball again. >> >> * get rid of bits we *don't want anymore*. If I need anything in a sumo >> package I'd need *all* of it. >> >> * override individual packages with newer versions >> >> * we've done a lot of refactoring recently trying to separate the UI >> from packages. This is done by creating a *new* package, leaving the old >> package behind. We can do this, test this and release this >> package-by-package now. >> >>> Over time, we'd tease out the dependencies of packages that live in the ZTK >>> distribution, making them useful outside without any dependency on the ZTK. >>> The >>> names of these packages could be arbitrary (they wouldn't need to retain >>> their >>> old "zope.*" names). Some backwards dependency shims would be added to the >>> ZTK >>> to prevent old imports from breaking, and the ZTK distribution would then >>> just >>> have a dependency on the thing that got broken out. >> I don't like the attempt to redefine what the ZTK means to a giant ball >> of Python packages. That's implying that, say, zope.component is *not* >> in the ZTK. That's wrong. >> >> Why generate a whole lot of work for ourselves getting from where we are >> now to here? We've learned how to work with the current situation in >> 2007 already. >> >>> I'm thinking that this would simplify things greatly for people who want to >>> be >>> consumers of "truly independent" Zope packages. There'd be exactly N of >>> them >>> available for download (where N is much less than 100, more like 20), plus >>> the >>> ZTK, which would have the rest of the pile in it over time. >> I don't see why a big package would "simplify things greatly" for you or >> anyone else. >> >>> If someone wanted >>> to use a forked version of a package that lived in the ZTK distribution, >>> they'd >>> either do so by teasing out the dependencies and making it "truly >>> independent" >>> or they'd just reroll a custom version of the entire ZTK distribution. >> And that's easier than the current situation how? Are you really >> proposing we destroy the dependency information we've already teased out >> and then make people do the work again? >> >>> Does this make any sense? >> Not a lot in my book. >> >> I think an important reason why there's so much awareness of dependency >> issues in the Zope world now (and effort spent on it) is precisely >> because we released our separate packages and can see the dependency >> information clearly. >> >> Regards, >> >> Martijn >> >> _______________________________________________ >> Zope-Dev maillist - Zope-Dev@zope.org >> http://mail.zope.org/mailman/listinfo/zope-dev >> ** No cross posts or HTML encoding! ** >> (Related lists - >> http://mail.zope.org/mailman/listinfo/zope-announce >> http://mail.zope.org/mailman/listinfo/zope ) >> > > _______________________________________________ > Zope-Dev maillist - Zope-Dev@zope.org > http://mail.zope.org/mailman/listinfo/zope-dev > ** No cross posts or HTML encoding! ** > (Related lists - > http://mail.zope.org/mailman/listinfo/zope-announce > http://mail.zope.org/mailman/listinfo/zope ) > > _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )