Re: [Mailman-Developers] Adding keys in the configuration file
Hi Mailmaners, Here's a version of the external configuration files patch. As Barry suggested, it adds a configure entry in the archiver's section, which can point to an ini-compatible configuration file (ConfigParser is used) for additional archiver-specific configuration. I've migrated the existing archiver config to separate files too. The stock files are shipped in the config directory, and can be overrided by a configuration file of the same name in $ext_dir. I've tried to keep the ini-file loading method generic, in case we want the same feature somewhere else. Please tell me what you think of it, if and where it needs some more work. Aurélien mailman3-externalconfig.patch Description: Binary data ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
On Sep 17, 2012, at 05:06 PM, Aurelien Bompard wrote: Please tell me what you think of it, if and where it needs some more work. Is there any chance you could push this to Launchpad as a branch? Patches in email threads are easily missed. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
Is there any chance you could push this to Launchpad as a branch? Sure. I've pushed it to Launchpad and filed a merge proposal. Cheers, Aurélien ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
Yeah, sorry I'm not familiar enough with HyperKitty to know how you get messages *into* it. ;) Don't be sorry for not knowing *everything* ;-) Aurélien ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
Why do you think that the configuration of HK should be a part of the MM core? Like the Postorius configuration UI, HK is supposed to be a separable part of the overall system which is possibly running on an entirely separate host machine. As such, its daemon would be configured and operated as a distinct service. The linkage between the two systems should be in the form of a single reference. On Sep 13, 2012, at 2:19 AM, Aurelien Bompard wrote: Hi people, I'm working on HyperKitty, and I've understood that I need to add this section to mailman.cfg to enable it : [archiver.hyperkitty] class: hyperkitty.lib.archiver.Archiver enable: yes base_url: http://hyperkitty.example.com That works well. However, I need another key to specify the location to HK's configuration file (Django's settings.py). I've tried simply adding the key, but it's not picked up by the configuration class, probably because of the schema validation. Is there a way I could add this configuration key ? Maybe via another section, or another file ? Thanks ! Aurélien ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/richard%40nfsnet.org Security Policy: http://wiki.list.org/x/QIA9 ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
Why do you think that the configuration of HK should be a part of the MM core? I didn't intend to put all of HK's config in mailman.cfg of course. But I'm supposed to reference a class in the archiver.hyperkitty section, which will be imported on startup, so it must be on the same host. Since there are already archiver-specific configuration keys there, like a command to run or an email address to forward to, I supposed it would be OK to add a conffile key that I could use. So what's the best way ? Add a conf_file key to schema.cfg ? Something else, maybe more extendable ? Thanks. Aurélien ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
Sorry I messed up with a list-reply, so you people missed Richard's reply. What concerns me is that MM core should not depend on Django in order to run. The Archive interface needs to accept the message and place it in the store. MM won't depend on Django, only if the HK section in mailman.cfg is enabled. I can't just send the message to the store however, since the IArchiver interface also has methods to get the list and message's final archive URL, and that's dependent on the frontend. The store does not know how the frontend is installed (at the root URL, on /hyperkitty, on /archives, etc.). Those methods are run before the message is stored in order for their result to be added to the message headers, so I can't even get it from the archive_message call. Aurélien ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
You had mentioned a reference to the Django settings for HK. Yes, my idea was to have something like that in the mailman.cfg file : [archiver.hyperkitty] enabled: yes class: hyperkitty.archiver.Archiver base_url: http://archives.example.com conf_file: /path/to/django/app/settings.py Only the conf_file key is invalid at the moment. It references the path to a Django settings file, but that's all. The Archiver class will then use this value to import the settings and go on with its job. That's the only place where there will be an import from the django namespace. Even if HK is enabled, django should not get imported into the MM space. I'm not sure I understand that. You mean in the same process memory ? Then I'm limited to some kind of client-server architecture, which I fear will have performance issues since we're passing every messages around. Or am I missing something ? The proper way to handle the issue of URLs would be to have the MM core query the UI for them. Well, I thought that's what it did currently. It imports a HK-specific class, which loads Django's settings to build the proper URL fragment when the appropriate method is called. What kind of query do you mean ? However, I have yet to convince Barry that the present model (where, at present, MM is the only part allowed to be authoritative over ANY information that it uses) is inappropriate. So, for the present, we need to have a proxy for the UI and a section of the configuration that specifies the values for the parameters which it supplies. IMHO, we should actually treat it as multiple proxies so that we can handle the more generic configuration in a consistent manner. I'm kinda lost here. I sort of understand that you're in favor of making the configuration file more extendable to fit the extension's needs, which would solve my problem. Mailman can't know in advance what kind of configuration values an extension needs, so in my opinion the conf file should either allow extensions to put arbitrary keys there (in sub-sub-sections for example), or let the admin specify the path to another configuration file, which would be extension-specific. Anything I'm missing ? Barry, any comments ? Thanks for your input ! Aurélien ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
On Sep 13, 2012, at 04:53 PM, Aurelien Bompard wrote: MM won't depend on Django, only if the HK section in mailman.cfg is enabled. I can't just send the message to the store however, since the IArchiver interface also has methods to get the list and message's final archive URL, and that's dependent on the frontend. The store does not know how the frontend is installed (at the root URL, on /hyperkitty, on /archives, etc.). Those methods are run before the message is stored in order for their result to be added to the message headers, so I can't even get it from the archive_message call. The configuration options for IArchiver.list_url() and IArchive.permalink() are fairly rough, mostly because we haven't had many examples we can generalize from. If you look at the way mhonarc.py is implemented, it expects that the base_url specified in the config file would have up to three substitution variables: listname, hostname, and fqdn_listname. These get filled in with the appropriate information from the mailing list. It doesn't really know anything about private archives, and can't currently make that distinction. Further, the mhonarc implementation expects that you should be able to append the X-Message-ID-Hash to the base_url to get the final url that the message will be archived at. The HyperKitty IArchiver implementation doesn't live in the trunk yet, though I think it would be fine if it did. But the question is, how would the core know what the base url for the archive is, and how would it calculate the url for a specific message in the archive. Note that ideally, it would not have to contact the archiver to figure out this latter; that's what the whole idea between our proposed extension to RFC 5064 is all about: http://wiki.list.org/display/DEV/Stable+URLs I can't answer this question for HyperKitty because I haven't looked at it closely enough, but I can think of a few ways this might be implemented. If the base url is static, perhaps with some placeholders for list-specific information, then the same approach used for mhonarc could be used for hyperkitty. If the base url for a specific list were available in hyperkitty, and say, the core could issue an HTTP call (e.g. REST+JSON) to get it, then we could implement this in much the same way that notification templates are implemented (i.e. we do the query and cache the results for a little while). You wouldn't want to do this for every message being archived, because I think it would be too slow, but if you could make archive_message() and permalink() agree to their answers, then there's no technical reason why you couldn't implement it that way. In that case, [archiver.hyperkitty]base_url might be a bit of a misnomer, but so be it. Ultimately, the HyperKitty implementation of IArchiver could do whatever is necessary to provide accurate answers to that interface's methods. -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
On Sep 13, 2012, at 10:37 AM, Richard Wackerbarth wrote: The proper way to handle the issue of URLs would be to have the MM core query the UI for them. However, I have yet to convince Barry that the present model (where, at present, MM is the only part allowed to be authoritative over ANY information that it uses) is inappropriate. So, for the present, we need to have a proxy for the UI and a section of the configuration that specifies the values for the parameters which it supplies. I wouldn't necessarily categorize that way, because in some cases, the core already is not the authoritative source of information. Take welcome messages for example. Unlike in MM2, MM3 does not store the welcome message template in its database. The MM2 limitation to this approach was that we could only support welcome messages in one language. Now, what MM3 does is store a welcome_message_uri which may contain placeholders for the fqdn_listname, and a language code. So when the t...@example.com list needs to send a welcome message in fr, it essentially does this: * expand the list's welcome_message_uri by interpolation of $listname and $language placeholders * issue an HTTP GET on the resulting url * cache the results for some period of time * use the resulting text as the welcome message template, into which it interpolates a bunch of other placeholders, both list specific and user specific. I think this is a powerful approach that could be extended to other pieces of information stored in external systems. So essentially, MM3 is a consumer of other web resources for which the authoritative source can be delegated. This is only moderately generalized in the core right now, but I see no reason why this approach can't be used in more places. Certainly a HyperKitty IArchiver implementation could easily do something like this today. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
On Sep 13, 2012, at 06:13 PM, Aurelien Bompard wrote: Yes, my idea was to have something like that in the mailman.cfg file : [archiver.hyperkitty] enabled: yes class: hyperkitty.archiver.Archiver base_url: http://archives.example.com conf_file: /path/to/django/app/settings.py Only the conf_file key is invalid at the moment. It references the path to a Django settings file, but that's all. The Archiver class will then use this value to import the settings and go on with its job. That's the only place where there will be an import from the django namespace. You could do it this way, but it would require HyperKitty to be colocated on the same machine as the core, and that its settings.py file be importable by the core. Better IMHO would be for the core to issue an HTTP/REST+JSON style GET call on some configurable URL to ask the Django server for the information. Then HyperKitty could reside anywhere. Even if HK is enabled, django should not get imported into the MM space. I'm not sure I understand that. You mean in the same process memory ? Then I'm limited to some kind of client-server architecture, which I fear will have performance issues since we're passing every messages around. Or am I missing something ? If you did it as a REST response, you'd want the HyperKitty IArchiver implementation to cache the result for a while. More savings would be achieved if MM3 could calculate the archive url from a base url, a la our extension to RFC 5064 I mentioned previously. I'm kinda lost here. I sort of understand that you're in favor of making the configuration file more extendable to fit the extension's needs, which would solve my problem. Mailman can't know in advance what kind of configuration values an extension needs, so in my opinion the conf file should either allow extensions to put arbitrary keys there (in sub-sub-sections for example), or let the admin specify the path to another configuration file, which would be extension-specific. We do something like this already to implement password hashing via the third party passlib package. If you look at the [passwords]path configuration variable, it names a url to an external passlib.cfg configuration file. This can use standard file: or http: urls, or as is shown in the default, a special python: url scheme. The semantics of python: url schemes is that the value names a file in Python dotted-path name syntax for a file locatable on sys.path (it doesn't need to be importable, but the parent must be). I can certainly see something like this work for archiver configuration. Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
On Sep 13, 2012, at 04:07 PM, Aurelien Bompard wrote: I didn't intend to put all of HK's config in mailman.cfg of course. But I'm supposed to reference a class in the archiver.hyperkitty section, which will be imported on startup, so it must be on the same host. Since there are already archiver-specific configuration keys there, like a command to run or an email address to forward to, I supposed it would be OK to add a conffile key that I could use. Remember too that the HyperKitty implementation of IArchiver can kind of do whatever it wants to do. It is a limitation of lazr.config that you can't set arbitrary variables in config file sections, i.e. ones that are not defined in the schema.cfg. That's unfortunate and maybe something we'll have to fix, workaround, or find a substitute for lazr.config (which we may have to do anyway if/when MM3 moves to Python 3). So what's the best way ? Add a conf_file key to schema.cfg ? Something else, maybe more extendable ? If you wanted to implement this without code changes to the core, I'd reuse the `base_url` variable, and inside the HyperKitty IArchiver implementation, I'd interpret that as the url to a configuration file. Kind of icky I know, but it's probably the best you can do without changing schema.cfg. OTOH, if you wanted to propose a patch/branch to MM3, then I'd probably suggest a more general approach of adding a `configure` variable to [archiver.master] and giving this semantics similar to [passwords]path as described previously. This [archiver.*]configure would name the path to a ini style file that doesn't have the lazr.config schema limitation. Then inside there, you could add anything that HyperKitty would need. This would be nice because we could probably even remove recipient and command from the general [archiver.master] section, and move them to the specific mail-archive.com and mhonarc .ini files (which would have to be added, but that's easy). Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
This is very interesting, thanks a lot for your suggestions. If the base url is static, perhaps with some placeholders for list-specific information, then the same approach used for mhonarc could be used for hyperkitty. That would work nicely for list_url() and permalink() indeed, even if it means that the URL mappings must be kept in sync. However, I need other pieces of information from the settings.py file, for example the store's database URL. You could do it this way, but it would require HyperKitty to be colocated on the same machine as the core, and that its settings.py file be importable by the core. Better IMHO would be for the core to issue an HTTP/REST+JSON style GET call on some configurable URL to ask the Django server for the information. Then HyperKitty could reside anywhere. True, but I still need to send the message to the store somehow. However, this method allows a decoupling between the web archive configuration and the storage action, which is interesting. OTOH, if you wanted to propose a patch/branch to MM3, then I'd probably suggest a more general approach of adding a `configure` variable to [archiver.master] and giving this semantics similar to [passwords]path as described previously. This [archiver.*]configure would name the path to a ini style file that doesn't have the lazr.config schema limitation. Then inside there, you could add anything that HyperKitty would need. This would be nice because we could probably even remove recipient and command from the general [archiver.master] section, and move them to the specific mail-archive.com and mhonarc .ini files (which would have to be added, but that's easy). This looks like an excellent solution to me, I'll go ahead and do that. Afterwards I'll look into making it possible to have HK on a separate host (and maybe only keep the storing library with Mailman). Thanks a lot for your ideas ! Aurélien -- http://aurelien.bompard.org~~jabber: aurel...@bompard.org This mail is displayed with recycled electrons ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
Re: [Mailman-Developers] Adding keys in the configuration file
On Sep 13, 2012, at 11:42 PM, Aurélien Bompard wrote: True, but I still need to send the message to the store somehow. Yeah, sorry I'm not familiar enough with HyperKitty to know how you get messages *into* it. ;) Cheers, -Barry ___ Mailman-Developers mailing list Mailman-Developers@python.org http://mail.python.org/mailman/listinfo/mailman-developers Mailman FAQ: http://wiki.list.org/x/AgA3 Searchable Archives: http://www.mail-archive.com/mailman-developers%40python.org/ Unsubscribe: http://mail.python.org/mailman/options/mailman-developers/archive%40jab.org Security Policy: http://wiki.list.org/x/QIA9
