Re: ApacheCon BOF about Module Repository
Sander Temme wrote: One aspect of TCAHMAN that I hadn't covered in my original discussion is how to add modules from the repository to an existing Apache install. This would require a program, installed with the server, that can fetch the module code and run the build/install. We have tentatively named this program apxs++ since it's a logical extension of what apxs does today. For maximum compatibility, this tool would have to be written in C. Currently, apxs is a Perl program but you can't always count on the availability of Perl on the system, especially on Windows. The apxs++ tool would be available only when mod_so is available. FYI - please don't worry about perl on win32. Between Randy's distro and the ActiveState offering, perl is fairly ubiquitous (at least more common than python). If we need a common denominator, I really have never had an issue with perl. The bigger problem is scripting for the lowest common denominator, at least perl 5.005, since there are so many flavors commonly distributed. Digging into C implies a serf-like http: client library, which is certainly less prevalant than perl itself. At least libwww is trivial to obtain, if not already bundled. Bill
Re: ApacheCon BOF about Module Repository
On Wed, Sep 07, 2005 at 03:34:19PM -0500, William A. Rowe, Jr. wrote: Digging into C implies a serf-like http: client library, which is certainly less prevalant than perl itself. At least libwww is trivial to obtain, if not already bundled. Don't we need something like that in C anyway? iirc mod cache requestor is currently stuck with curl? I wouldn't mind seeing serf or similar functionality in apr - there seems to be plenty of uses for it. vh Mads Toftum -- `Darn it, who spiked my coffee with water?!' - lwall
Re: ApacheCon BOF about Module Repository
Mads Toftum wrote: On Wed, Sep 07, 2005 at 03:34:19PM -0500, William A. Rowe, Jr. wrote: Digging into C implies a serf-like http: client library, which is certainly less prevalant than perl itself. At least libwww is trivial to obtain, if not already bundled. Don't we need something like that in C anyway? iirc mod cache requestor is currently stuck with curl? I wouldn't mind seeing serf or similar functionality in apr - there seems to be plenty of uses for it. No, mod_proxy_http has it's own http requestor; and yes - making this code generic would likely be a very good thing. Integrating it with serf would be even more useful ;-) Bill
Re: ApacheCon BOF about Module Repository
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Folks, I have been meaning to follow up on this ever since that BOF, and am deeply annoyed that it has taken me this long. As a matter of fact, my memory of that BOF session has now faded to a considerable extent and I don't feel comfortable even giving a list of attendees because I would leave people out. I have pinned to my office wall the flip-over sheet with notes I took during the session and will now transcribe those. If anyone present at the event notices I'm leaving something out, please speak up. Building on my original message below, we discussed what should be implemented and how. One aspect of TCAHMAN that I hadn't covered in my original discussion is how to add modules from the repository to an existing Apache install. This would require a program, installed with the server, that can fetch the module code and run the build/install. We have tentatively named this program apxs++ since it's a logical extension of what apxs does today. For maximum compatibility, this tool would have to be written in C. Currently, apxs is a Perl program but you can't always count on the availability of Perl on the system, especially on Windows. The apxs++ tool would be available only when mod_so is available. If you're linking modules statically, you're compiling your own httpd and should be able to fetch the source code for the desired module(s) before you start compiling. I have no notes about the scenarios described below, fetching the module code from configure or pointing configure at a module source tarball, but I seem to recall the group was not overly enthusiastic about configure downloading and injecting source code into the build. All this goodness, if and when it happens, would be run from a newly created httpd-modules subproject. We discussed the proposed nature and structure of this subproject (which itself has not been proposed yet) and the general idea seemed to be that we creat a flat sandbox where module developers can commit to everything. Every httpd committer automatically gets httpd-modules, and the subproject could be a breeding ground for new httpd committers. If and when a module develops its own community, it can get its own subproject (example: mod_python) or even go top level (example: mod_perl, mod_tcl). The httpd-modules subproject would also own the repository code. The TCAHMAN system would be targeted at: a) builders (who build their own Apache) b) enhancers (what did I mean by this? Perhaps folks who want to hang additional modules into an existing Apache?) c) packagers (TCAHMAN could register installed modules with the various package registries out there, giving httpd packagers a powerful way to manage the installed core and modules) d) Testers (perl-framework) (Not sure what I meant by this) We would initially populate the repository with modules that were formerly in the core, and eventually open it up to third-party module developers. Having easy access to modules through TCAHMAN will allow us (httpd) to lighten the distribution Once we open the repository up to third-party developers, we may have to do a 'click through' (or key through) acknowledgement that we (ASF) are not responsible for code that is not ours. IANAL, so I don't know what is required/comfy. The TCAHMAN repository would utilize our existing mirror infrastructure, and would be a great service to offer third-party developers. We discussed CPAN, from which a lot of people blindly and trustingly download module upon module, as root. How did this get so trusted? Who is responsible for the code? We hear that nobody owns CPAN, and there is no identifiable target for any legal action anyone might want to bring. This obviously wouldn't fly for the ASF. The designated front for TCAHMAN would be modules.apache.org, which is currently run by Covalent. We would run TCAHMAN on our own infrastructure, so we'd need to get the vhost back from them. While this is technically really easy (we own the DNS for apache.org, after all), it would be a good thing to arrange a smooth transition. Every module uploaded to the network would come with metadata, including (but not limited to): * License * Versioning (compatible with (1.3, 2.0, 2.1, ...), not before, not after (MMN?)) * Documentation URL * Author info * Build options Tasks: * Write apxs++ * Define module metadata * Write the backend * Take back modules.apache.org That's all I have right now. Remarks? Comments? Additions? Flames? Please discuss: we need to find out whether there is community support for the idea. Thanks, Sander On Jul 21, 2005, at 3:14 AM, Sander Temme wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All who are at ApacheCon or are otherwise interested, I snatched a BOF slot tonight (Thursday the 21st) at 20:30 to
Re: ApacheCon BOF about Module Repository
Sander Temme wrote: For maximum compatibility, this tool would have to be written in C. Currently, apxs is a Perl program but you can't always count on the availability of Perl on the system, especially on Windows. The apxs++ tool would be available only when mod_so is available. Sorry for jumping in... been roughly following. As far as I can see, apxs is only written in perl because of the REGEX to find wether were are in a Location or other container before adding the LoadModule line. Could we not use PCRE and APR to rewrite this in C as well? Maybe 2.3 ? And this comming from a mod_perl commiter :) -- END What doesn't kill us can only make us stronger. Nothing is impossible. Philip M. Gollucci ([EMAIL PROTECTED]) 301.254.5198 Consultant / http://p6m7g8.net/Resume/ Senior Developer / Liquidity Services, Inc. http://www.liquidityservicesinc.com http://www.liquidation.com http://www.uksurplus.com http://www.govliquidation.com http://www.gowholesale.com
Re: ApacheCon BOF about Module Repository
Sander Temme wrote: One aspect of TCAHMAN that I hadn't covered in my original discussion is how to add modules from the repository to an existing Apache install. This would require a program, installed with the server, that can fetch the module code and run the build/install. We have tentatively named this program apxs++ since it's a logical extension of what apxs does today. For maximum compatibility, this tool would have to be written in C. Currently, apxs is a Perl program but you can't always count on the availability of Perl on the system, especially on Windows. The apxs++ tool would be available only when mod_so is available. I think we did discuss this a little, though my memory is faded too. I don't see the problem with apxs being perl. Sure, it is not quite universally available, but then neither is a C compiler, which is another prerequisite for anything APXS-like. That implies at least the option of the tool downloading a binary (and its dependencies where applicable), and it no longer really looks like apxs. So why not preserve apxs as-is - perhaps enhanced to use the new archive for those who do have the prerequisites - and make a clean new start with the new tool? If you're linking modules statically, you're compiling your own httpd and should be able to fetch the source code for the desired module(s) before you start compiling. I don't think we should worry too much about supporting static compiles. Rather concentrate on making sure everyone has a dynamic build unless they consciously override that, for their own reasons. All this goodness, if and when it happens, would be run from a newly created httpd-modules subproject. We discussed the proposed nature and structure of this subproject (which itself has not been proposed yet) and the general idea seemed to be that we creat a flat sandbox where module developers can commit to everything. Every httpd committer automatically gets httpd-modules, and the subproject could be a breeding ground for new httpd committers. If and when a module develops its own community, it can get its own subproject (example: mod_python) or even go top level (example: mod_perl, mod_tcl). The httpd-modules subproject would also own the repository code. This blurs the distinction between bundled and addon modules. That may be no bad thing, but needs to be thought through. Do we, for example, officially accept bug reports for third-party modules in our bugzilla? 'Cos if not, that's a potentially nasty limbo state for the end-users. The TCAHMAN system would be targeted at: a) builders (who build their own Apache) b) enhancers (what did I mean by this? Perhaps folks who want to hang additional modules into an existing Apache?) c) packagers (TCAHMAN could register installed modules with the various package registries out there, giving httpd packagers a powerful way to manage the installed core and modules) d) Testers (perl-framework) (Not sure what I meant by this) e) Any server admin We would initially populate the repository with modules that were formerly in the core, and eventually open it up to third-party module developers. Having easy access to modules through TCAHMAN will allow us (httpd) to lighten the distribution Are you thinking of even the very-core modules? I think that could make sense, provided the default install always includes them unless the builder makes a conscious decision and overrides a warning about some core functions may not work. Once we open the repository up to third-party developers, we may have to do a 'click through' (or key through) acknowledgement that we (ASF) are not responsible for code that is not ours. IANAL, so I don't know what is required/comfy. Indeed. The TCAHMAN repository would utilize our existing mirror infrastructure, and would be a great service to offer third-party developers. We discussed CPAN, from which a lot of people blindly and trustingly download module upon module, as root. How did this get so trusted? Who is responsible for the code? We hear that nobody owns CPAN, and there is no identifiable target for any legal action anyone might want to bring. This obviously wouldn't fly for the ASF. The designated front for TCAHMAN would be modules.apache.org, which is currently run by Covalent. We would run TCAHMAN on our own infrastructure, so we'd need to get the vhost back from them. While this is technically really easy (we own the DNS for apache.org, after all), it would be a good thing to arrange a smooth transition. Every module uploaded to the network would come with metadata, including (but not limited to): * License * Versioning (compatible with (1.3, 2.0, 2.1, ...), not before, not after (MMN?)) * Documentation URL * Author info * Build options * Dependencies (e.g. external libs) * Exports (e.g. modules that provide or consume an API) *
Re: ApacheCon BOF about Module Repository
On Tue, 6 Sep 2005, Sander Temme wrote: Folks, I have been meaning to follow up on this ever since that BOF, and am deeply annoyed that it has taken me this long. As a matter of fact, my memory of that BOF session has now faded to a considerable extent and I don't feel comfortable even giving a list of attendees because I would leave people out. I have pinned to my office wall the flip-over sheet with notes I took during the session and will now transcribe those. If anyone present at the event notices I'm leaving something out, please speak up. Building on my original message below, we discussed what should be implemented and how. [ ... ] We discussed CPAN, from which a lot of people blindly and trustingly download module upon module, as root. How did this get so trusted? Who is responsible for the code? We hear that nobody owns CPAN, and there is no identifiable target for any legal action anyone might want to bring. This obviously wouldn't fly for the ASF. Jarkko Hietaniemi, the CPAN Master Librarian, wrote up some thoughts on CPAN-like archives at http://www.cpan.org/misc/ZCAN.html Most (but not all) activity in CPAN is connected with Perl modules; for this, PAUSE (http://pause.perl.org/) is the means by which authors can upload and manage submissions. The code for PAUSE is available: ftp://pause.perl.org/pub/PAUSE/PAUSE-code/ Every module uploaded to the network would come with metadata, including (but not limited to): [ ... ] In the Perl world, metadata now comes in the form of a META.yml file: http://module-build.sourceforge.net/META-spec.html which is based on YAML: http://www.yaml.org/ This is one area that the CPAN/PAUSE maintainers emphasize as being crucial in being able to index and search effectively. -- best regards, randy kobes
ApacheCon BOF about Module Repository
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 All who are at ApacheCon or are otherwise interested, I snatched a BOF slot tonight (Thursday the 21st) at 20:30 to discuss ideas for dealing with modules inside and outside the httpd distribution. This is so far just an idea... I named it TCAHMAN (pronounced Tikkaman) for The Comprehensive Apache Httpd Module Archive Network. The basic premise is to run: $ ./configure (...) --with-tcahman-shared=funkymod (...) and configure will contact the tcahman server (a.k.a. modules.apache.org), download the source code for funkymod and compile it into the server as an so. Or, it could access a locally downloaded module tarball in case the build box can't see the net: $ ./configure (...) --with-tcahman=/path/to/funkymod.tar.gz will find the tarball in the file system and compile it (statically, in this case) into the server. In a similar fashion, an installed httpd could come with a script that can download, build and install a module on the existing server. Perhaps an enhancement to APXS? For instance: $ apxs build --with-tcahman=funkymod On the server side of TCAHMAN, the main issue is Organization. I would like to model this after CPAN, but I have no idea how CPAN works in any case, what we'd need is a standard for what module code and its meta-data looks like: proper autoconf language to get it built, name and description for the search engine, which Apache version(s) the tarball works with, etc. The other side of the organization aspect is who gets to upload modules to this archive. Do we just open it up? Or do we impose any regulations on code quality or evilness? Who gets to enforce this (major time sink danger here)? What language would we use to make sure people don't attribute uploads of third-party code to us? Will modules.apache.org have a feedback engine where users can tell module developers their shit is broken? This or a similar construction would provide people who build httpd easier access to third-party modules. It would also provide a way out for modules we might not want in the core distribution anymore, but would still like to make available. It gives module authors visibility to users, to get their code in front of people who might want to run it. Let's bat this around tonight and see if this is something we want to do, and how we would go about it. Is there any beer left over from the Hackathon? S. - -- [EMAIL PROTECTED] http://www.temme.net/sander/ PGP FP: 51B4 8727 466A 0BC3 69F4 B7B8 B2BE BC40 1529 24AF -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFC33WCnjkrwtLH+RIRAl2eAJwITvvpbf297JUUVg2e1kPsWPiLpQCfTbCU 2y4cd6SZ+Rj83JO2IEY2y2E= =RwhU -END PGP SIGNATURE-
