Re: Swigged python package name...
On 07/10/2013 12:25 PM, Gordon Sim wrote: I have been working to get (at least some of) the python tests running over the swigged implementation. My motivation here was initially to add some new tests for the AMQP 1.0 functionality using python. However it would also be desirable to be able to run some of the existing tests over 1.0 as well as continuing to be able to run over 0-10. I hope to have a patch up for review before too long, For those interested: https://reviews.apache.org/r/12515/ but by adding another module (I called it qpid_messaging.py) that tries to import from the swigged module, then falls back to importing qpid.messaging, I was able to have one import statement in tests of interest and then run them over either swigged or pure implementations based on whether the former was on the path or not. In the end I pushed this redirection into a utility module in the tests package. - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: Swigged python package name...
On 07/10/2013 10:47 PM, Darryl L. Pierce wrote: I think your previous suggestion to have a compatibility import might be the better way to go. How about something like this: 1. move the pure Python bindings to the qpid.pure package I think we should leave the existing pure python client as it is (at the very least for the time being). I see no convincing case to change what that namespace refers to. 2. leave the Swig bindings in cqpid for the time being I do think it might be worth considering a renaming of the swigged client. This of course would need to be discussed on the user list first to assess impact and opinion of those most effected. 3. create a new module that lives in qpid.messaging and which will import either qpid.pure or cqpid depending on a setting from the application importing them? As above I think we should keep qpid.messaging to mean what it always has. I'd suggest this third module be called qpid_messaging. (I'd suggest the swigged version then be qpidc_messaging). I started off thinking some sort of application setting might be good. In the end I think the simplest solution is to use the path - prefer one then fallback to the other. If the application code has to set a change a variable then you sort of negate the point of the exercise (i.e. not wanting to edit code). Where there is an existing mechanism for runtime options (e.g. command line options or a config file) then it is easy enough for users to build the ability to switch more flexibly if it is important. - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: Swigged python package name...
On 07/09/2013 08:34 PM, Darryl L. Pierce wrote: One of the issues that needs to be fixed for this is the package name for the code. Currently it's cqpid, which isn't close to what we have in other languages (which all use a variable on qpid.messaging). However, we can't use that since it's what our pure Python code uses. Or can we? I don't think we should. They should each have their own name to clearly identify them (the name refers to the implementation, not just the API as it does in say c++). I agree that cqpid is not a good name for the swigged module; it is too ambiguous. I would suggest something like qpidc_messaging or qpid_cmessaging instead. However changes such as those need to be discussed on the user list. That's the question: how do we package up the code? I've experimented with the examples we have and they seem to work just fine with cqpid substituted for qpid.messaging for the imports. I have been working to get (at least some of) the python tests running over the swigged implementation. My motivation here was initially to add some new tests for the AMQP 1.0 functionality using python. However it would also be desirable to be able to run some of the existing tests over 1.0 as well as continuing to be able to run over 0-10. I hope to have a patch up for review before too long, but by adding another module (I called it qpid_messaging.py) that tries to import from the swigged module, then falls back to importing qpid.messaging, I was able to have one import statement in tests of interest and then run them over either swigged or pure implementations based on whether the former was on the path or not. So in this scheme you have qpid.messaging for the exiting, pure python implementation (only supports 0-10 at present), cqpid (which I agree is a poor name, but see above) for the swigged implementation that supports 0-10 and 1.0, and qpid_messaging that can be used in contexts where switching between the two without editing files is desirable (e.g. tests and perhaps examples). - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: Swigged python package name...
On Tue, Jul 09, 2013 at 04:01:02PM -0400, Ted Ross wrote: I would think that downstream repos would package one or the other but not both. It would be nice if downstream, it was called qpid.messagingregardless of the implementation used. Agreed. I think it ought to be a goal to have the Swigged bindings be a drop-in replacement for the pure Python bindings. The users shouldn't care which implementation they use. Is there a way to parameterize it such that the namespace could be overridden when built and packaged? Well, the name space is defined by the directory structure when the modules are installed. So wherever the .py files are installed defines what the package name will be. So I did a small experiment (see below) just to verify that we can use different package names without having to munge the code. For example, call the wrapped client cqpid.messaging but make it easy for a distro to build and package it as qpid.messaging. We can accomplish this by just having a qpid/messaging directory tree under the site packages location and in __init__.py there do: from cqpid import * This would put the entire cqpid.* set into the qpid/messaging namespace. Regarding the module name coming out of Swig, I don't think we can use a segmented name. Looking at the documentation [1] it appears that's solely used to name the shared library and how that name is interpreted is dynamic language dependant. Unless you were suggesting using cqpid.messaging similar to what I described above. WRT the name space, I'm still wanting to keep it as qpid.messaging since, in the long run, that would require developers to not change their imports and usage. If, down the road, a group decided to maintain the pure Python bindings separately they could use a different namespace, like we saw with the evolution of qpid::client to qpid::messaging in the C++ code itself? [1] http://www.swig.org/Doc2.0/Modules.html -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpxTqiP0Y2Ix.pgp Description: PGP signature
Re: Swigged python package name...
On 07/10/2013 01:45 PM, Darryl L. Pierce wrote: On Tue, Jul 09, 2013 at 04:01:02PM -0400, Ted Ross wrote: I would think that downstream repos would package one or the other but not both. It would be nice if downstream, it was called qpid.messagingregardless of the implementation used. Agreed. I disagree. I can see cases where it would be beneficial to have both installed. E.g. you have existing applications that you don't want to switch (if it ain't broke etc), but you also want to try 1.0 (or just the c++ impl) for some new use cases. I think it ought to be a goal to have the Swigged bindings be a drop-in replacement for the pure Python bindings. The users shouldn't care which implementation they use. Some won't, some will. Though the APIs are getting very close, there are still - and I believe will always be - behavioural differences. Even apart from those, we have had users using eventlet and monkey-patching and they are going to care which implementation they have. [...] WRT the name space, I'm still wanting to keep it as qpid.messaging since, in the long run, that would require developers to not change their imports and usage. I want to keep qpid.messaging for the existing python client, have the swigged implementation available through another name and optionally have a third that simply tries one and falls back on the other based on what is available (for those use cases that want to be able to switch between the two). - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: Swigged python package name...
On Wed, Jul 10, 2013 at 03:54:44PM +0100, Gordon Sim wrote: On 07/10/2013 01:45 PM, Darryl L. Pierce wrote: On Tue, Jul 09, 2013 at 04:01:02PM -0400, Ted Ross wrote: I would think that downstream repos would package one or the other but not both. It would be nice if downstream, it was called qpid.messagingregardless of the implementation used. Agreed. I disagree. I can see cases where it would be beneficial to have both installed. E.g. you have existing applications that you don't want to switch (if it ain't broke etc), but you also want to try 1.0 (or just the c++ impl) for some new use cases. I can see someone wanting to try out 1.0, but they'll still eventually have to make a decision, right? If they go with the swigged APIs then they should be able to drop in the updated APIs with as little impact on their code as possible, IMO. Regarding not wanting to refactor existing apps that work, etc., that can be handled by the administrators by putting the old Python bindings in a separate location from the newer bindings by naming the eggs differently; i.e., python-qpid is the old API set while python-qpid_messaging is the newer set. Using pkg_resources the app developer can have both APIs living on the same machine, using the same packages, without collision. I think it ought to be a goal to have the Swigged bindings be a drop-in replacement for the pure Python bindings. The users shouldn't care which implementation they use. Some won't, some will. Though the APIs are getting very close, there are still - and I believe will always be - behavioural differences. Even apart from those, we have had users using eventlet and monkey-patching and they are going to care which implementation they have. I'm not sure how to respond to that. The behavioral differences will result in large differences in how the APIs are expected to work? For the latter case, I see what you're saying. It'll be an issue, and I'd love to get some of them involved in discussions regarding this. WRT the name space, I'm still wanting to keep it as qpid.messaging since, in the long run, that would require developers to not change their imports and usage. I want to keep qpid.messaging for the existing python client, have the swigged implementation available through another name and optionally have a third that simply tries one and falls back on the other based on what is available (for those use cases that want to be able to switch between the two). My main issue is looking at the pure Python bindings and limiting what feels, to me at least, the package name for the swigged bindings due to that. I think we can overcome that without having extra packages on top of them. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgp1xqkryeOUz.pgp Description: PGP signature
Re: Swigged python package name...
On 07/10/2013 04:51 PM, Darryl L. Pierce wrote: On Wed, Jul 10, 2013 at 03:54:44PM +0100, Gordon Sim wrote: On 07/10/2013 01:45 PM, Darryl L. Pierce wrote: On Tue, Jul 09, 2013 at 04:01:02PM -0400, Ted Ross wrote: I would think that downstream repos would package one or the other but not both. It would be nice if downstream, it was called qpid.messagingregardless of the implementation used. Agreed. I disagree. I can see cases where it would be beneficial to have both installed. E.g. you have existing applications that you don't want to switch (if it ain't broke etc), but you also want to try 1.0 (or just the c++ impl) for some new use cases. I can see someone wanting to try out 1.0, but they'll still eventually have to make a decision, right? The point is that the decision taken for one use case shouldn't necessarily force the same choice across the board. [...] Regarding not wanting to refactor existing apps that work, etc., that can be handled by the administrators by putting the old Python bindings in a separate location from the newer bindings by naming the eggs differently; i.e., python-qpid is the old API set while python-qpid_messaging is the newer set. Using pkg_resources the app developer can have both APIs living on the same machine, using the same packages, without collision. I think its important to keep the different notions of package separate. On the one hand there is the python namespace. On the other hand there is the platform specific installation approach in use (e.g. rpms or whatever). In the case of the latter, I think it is decidedly odd to e.g. yum install python-qpid, yum install python-qpid_messaging and have the installation of the second package overwrite artefacts of the first one (even assuming this doesn't totally break things). I think it ought to be a goal to have the Swigged bindings be a drop-in replacement for the pure Python bindings. The users shouldn't care which implementation they use. Some won't, some will. Though the APIs are getting very close, there are still - and I believe will always be - behavioural differences. Even apart from those, we have had users using eventlet and monkey-patching and they are going to care which implementation they have. I'm not sure how to respond to that. The behavioral differences will result in large differences in how the APIs are expected to work? There are differences, and those differences could mean that an application that works with one does not work with the other. To state the obvious, the swigged implementation is a different library offering (broadly) the same API. It is not an evolution of the existing implementation. It is clearer to give these two different implementations two different names. I can certainly also see cases where users don't want to explicitly choose one or the other and want their code to run against either without change (I have that need myself as explained in an earlier mail). However I think that can be done by a simple (5 line) module with a third name. That way users can pick one, or the other, or explicitly decide they don't care and want to pick whichever is available based on path. [...] My main issue is looking at the pure Python bindings and limiting what feels, to me at least, the package name for the swigged bindings due to that. I think we can overcome that without having extra packages on top of them. I don't really understand what you are saying here. Are you saying you want the name qpid.messaging for the swigged implementation because it's a nicer name than alternatives? - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
Re: Swigged python package name...
On Wed, Jul 10, 2013 at 05:59:43PM +0100, Gordon Sim wrote: On 07/10/2013 04:51 PM, Darryl L. Pierce wrote: On Wed, Jul 10, 2013 at 03:54:44PM +0100, Gordon Sim wrote: On 07/10/2013 01:45 PM, Darryl L. Pierce wrote: On Tue, Jul 09, 2013 at 04:01:02PM -0400, Ted Ross wrote: I would think that downstream repos would package one or the other but not both. It would be nice if downstream, it was called qpid.messagingregardless of the implementation used. Agreed. I disagree. I can see cases where it would be beneficial to have both installed. E.g. you have existing applications that you don't want to switch (if it ain't broke etc), but you also want to try 1.0 (or just the c++ impl) for some new use cases. I can see someone wanting to try out 1.0, but they'll still eventually have to make a decision, right? The point is that the decision taken for one use case shouldn't necessarily force the same choice across the board. +1 Regarding not wanting to refactor existing apps that work, etc., that can be handled by the administrators by putting the old Python bindings in a separate location from the newer bindings by naming the eggs differently; i.e., python-qpid is the old API set while python-qpid_messaging is the newer set. Using pkg_resources the app developer can have both APIs living on the same machine, using the same packages, without collision. I think its important to keep the different notions of package separate. On the one hand there is the python namespace. On the other hand there is the platform specific installation approach in use (e.g. rpms or whatever). In the case of the latter, I think it is decidedly odd to e.g. yum install python-qpid, yum install python-qpid_messaging and have the installation of the second package overwrite artefacts of the first one (even assuming this doesn't totally break things). That wouldn't be the case given that the two can be installed using separate names, both for the RPM itself also the underlying egg file shared via other Python methods. The was the collision avoidance I was talking about above; i.e., installing python-qpid (which ships the qpid egg's contents) would live in directories named python-qpid-VEV, while installing python-qpid_messaging (whose foundation is a different egg named qpid_messaging) would live in directories named python-qpid_messaging-VER. Both very sheltered from each other. The app developer can then use something similar to your suggestion about conditional imports and, instead of catching a raised exception to pick a different package, could use: pkg_resource.require(qpid==VER) or: pkg_resource.require(qpid_messaging=VER) to import the desired implementation of qpid.messaging to get the APIs they want. I think it ought to be a goal to have the Swigged bindings be a drop-in replacement for the pure Python bindings. The users shouldn't care which implementation they use. Some won't, some will. Though the APIs are getting very close, there are still - and I believe will always be - behavioural differences. Even apart from those, we have had users using eventlet and monkey-patching and they are going to care which implementation they have. I'm not sure how to respond to that. The behavioral differences will result in large differences in how the APIs are expected to work? There are differences, and those differences could mean that an application that works with one does not work with the other. To state the obvious, the swigged implementation is a different library offering (broadly) the same API. It is not an evolution of the existing implementation. It is clearer to give these two different implementations two different names. I can certainly also see cases where users don't want to explicitly choose one or the other and want their code to run against either without change (I have that need myself as explained in an earlier mail). However I think that can be done by a simple (5 line) module with a third name. That way users can pick one, or the other, or explicitly decide they don't care and want to pick whichever is available based on path. Understood. See what I wrote above as another way of approaching this. My main issue is looking at the pure Python bindings and limiting what feels, to me at least, the package name for the swigged bindings due to that. I think we can overcome that without having extra packages on top of them. I don't really understand what you are saying here. Are you saying you want the name qpid.messaging for the swigged implementation because it's a nicer name than alternatives? Not so much nicer as more consistent across the supported set of languages. In Ruby we have Qpid::Messaging, in Perl it's qpid::messaging, in C++ it's qpid::messaging. It would be nice to have Python also be qpid.messaging and not something different, to keep it as intuitive as possible. And it _is_ a nicer name. Just not primarily
Re: Swigged python package name...
On 07/10/2013 06:24 PM, Darryl L. Pierce wrote: On Wed, Jul 10, 2013 at 05:59:43PM +0100, Gordon Sim wrote: On 07/10/2013 04:51 PM, Darryl L. Pierce wrote: On Wed, Jul 10, 2013 at 03:54:44PM +0100, Gordon Sim wrote: On 07/10/2013 01:45 PM, Darryl L. Pierce wrote: On Tue, Jul 09, 2013 at 04:01:02PM -0400, Ted Ross wrote: I would think that downstream repos would package one or the other but not both. It would be nice if downstream, it was called qpid.messagingregardless of the implementation used. Agreed. I disagree. I can see cases where it would be beneficial to have both installed. E.g. you have existing applications that you don't want to switch (if it ain't broke etc), but you also want to try 1.0 (or just the c++ impl) for some new use cases. I can see someone wanting to try out 1.0, but they'll still eventually have to make a decision, right? The point is that the decision taken for one use case shouldn't necessarily force the same choice across the board. +1 Regarding not wanting to refactor existing apps that work, etc., that can be handled by the administrators by putting the old Python bindings in a separate location from the newer bindings by naming the eggs differently; i.e., python-qpid is the old API set while python-qpid_messaging is the newer set. Using pkg_resources the app developer can have both APIs living on the same machine, using the same packages, without collision. I think its important to keep the different notions of package separate. On the one hand there is the python namespace. On the other hand there is the platform specific installation approach in use (e.g. rpms or whatever). In the case of the latter, I think it is decidedly odd to e.g. yum install python-qpid, yum install python-qpid_messaging and have the installation of the second package overwrite artefacts of the first one (even assuming this doesn't totally break things). That wouldn't be the case given that the two can be installed using separate names, both for the RPM itself also the underlying egg file shared via other Python methods. The was the collision avoidance I was talking about above; i.e., installing python-qpid (which ships the qpid egg's contents) would live in directories named python-qpid-VEV, while installing python-qpid_messaging (whose foundation is a different egg named qpid_messaging) would live in directories named python-qpid_messaging-VER. Both very sheltered from each other. Right now the python-qpid fedora package, to use a concrete example, installs into sitepackages (qpid/messaging.py etc). Are you proposing to change that? The app developer can then use something similar to your suggestion about conditional imports and, instead of catching a raised exception to pick a different package, could use: pkg_resource.require(qpid==VER) or: pkg_resource.require(qpid_messaging=VER) to import the desired implementation of qpid.messaging to get the APIs they want. Those statements go in the application code? If so how is that better than simply importing from one or the other namespace? Is package_resource not a distribution/deployment tool rather than something applications would directly invoke in their code? I think it ought to be a goal to have the Swigged bindings be a drop-in replacement for the pure Python bindings. The users shouldn't care which implementation they use. Some won't, some will. Though the APIs are getting very close, there are still - and I believe will always be - behavioural differences. Even apart from those, we have had users using eventlet and monkey-patching and they are going to care which implementation they have. I'm not sure how to respond to that. The behavioral differences will result in large differences in how the APIs are expected to work? There are differences, and those differences could mean that an application that works with one does not work with the other. To state the obvious, the swigged implementation is a different library offering (broadly) the same API. It is not an evolution of the existing implementation. It is clearer to give these two different implementations two different names. I can certainly also see cases where users don't want to explicitly choose one or the other and want their code to run against either without change (I have that need myself as explained in an earlier mail). However I think that can be done by a simple (5 line) module with a third name. That way users can pick one, or the other, or explicitly decide they don't care and want to pick whichever is available based on path. Understood. See what I wrote above as another way of approaching this. I don't really understand what you wrote above. My main issue is looking at the pure Python bindings and limiting what feels, to me at least, the package name for the swigged bindings due to that. I think we can overcome that without having extra packages on top of them. I don't really understand what you are saying
Re: Swigged python package name...
On Wed, Jul 10, 2013 at 09:23:01PM +0100, Gordon Sim wrote: Right now the python-qpid fedora package, to use a concrete example, installs into sitepackages (qpid/messaging.py etc). Are you proposing to change that? No, I think I've combined how Ruby and Python work in this case. With Ruby gems you can have multiple versions of a gem installed and require only the version you need in your app. When skimming over the Python docs on eggs I mistook some of what it said in there as being the equivalent to what Ruby can do. Not so much nicer as more consistent across the supported set of languages. In Ruby we have Qpid::Messaging, in Perl it's qpid::messaging, in C++ it's qpid::messaging. It would be nice to have Python also be qpid.messaging It is! And has been long before the Ruby or Perl equivalents even existed. The issue is that we now have two *different* messaging libraries for python... and not something different, to keep it as intuitive as possible. ... I think trying to force the two into the same namespace is far from intuitive. I think your previous suggestion to have a compatibility import might be the better way to go. How about something like this: 1. move the pure Python bindings to the qpid.pure package 2. leave the Swig bindings in cqpid for the time being 3. create a new module that lives in qpid.messaging and which will import either qpid.pure or cqpid depending on a setting from the application importing them? This, I think, comes close to what you were proposing. -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgpzF9vh4ovj8.pgp Description: PGP signature
Swigged python package name...
One of the issues that needs to be fixed for this is the package name for the code. Currently it's cqpid, which isn't close to what we have in other languages (which all use a variable on qpid.messaging). However, we can't use that since it's what our pure Python code uses. Or can we? That's the question: how do we package up the code? I've experimented with the examples we have and they seem to work just fine with cqpid substituted for qpid.messaging for the imports. If our goal is to eventually replace the pure Python with the swigged version, then we could use the same package with a note to users that they need to pick one or the other. This would give us a clear migration path. Ideas? -- Darryl L. Pierce, Sr. Software Engineer @ Red Hat, Inc. Delivering value year after year. Red Hat ranks #1 in value among software vendors. http://www.redhat.com/promo/vendor/ pgplsFloXVJRA.pgp Description: PGP signature
Re: Swigged python package name...
I would think that downstream repos would package one or the other but not both. It would be nice if downstream, it was called qpid.messagingregardless of the implementation used. Is there a way to parameterize it such that the namespace could be overridden when built and packaged? For example, call the wrapped client cqpid.messaging but make it easy for a distro to build and package it as qpid.messaging. -Ted On 07/09/2013 03:34 PM, Darryl L. Pierce wrote: One of the issues that needs to be fixed for this is the package name for the code. Currently it's cqpid, which isn't close to what we have in other languages (which all use a variable on qpid.messaging). However, we can't use that since it's what our pure Python code uses. Or can we? That's the question: how do we package up the code? I've experimented with the examples we have and they seem to work just fine with cqpid substituted for qpid.messaging for the imports. If our goal is to eventually replace the pure Python with the swigged version, then we could use the same package with a note to users that they need to pick one or the other. This would give us a clear migration path. Ideas? - To unsubscribe, e-mail: dev-unsubscr...@qpid.apache.org For additional commands, e-mail: dev-h...@qpid.apache.org
