Re: Sharing code between different projects?
> On Monday, August 13, 2012 at 7:53:32 PM UTC+3, andrea crotti wrote: >> I am in the situation where I am working on different projects that >> might potentially share a lot of code. >> >> I started to work on project A, then switched completely to project B >> and in the transiction I copied over a lot of code with the >> corresponding tests, and I started to modify it. >> >> Now it's time to work again on project A, but I don't want to copy >> things over again. I use so called "namespace packages" for this. Those are "virtual" packages "containing" a set of related "real" packages - individually manageable on PyPI. The "real" packages describe in their dependencies on which other packages they depend, among them (potentially) "friend" packages. An example is "dm.zope.rpc" and "dm.zope.rpc.wsdl_suds". In this case, "dm.zope.rpc" provides general rpc infrastructure for the web application framework Zope and implementations for some elementary rpc protocols and "dm.zipe.rpc.wsdl_suds" uses this infrastructure for the implementation of a WSDL based rpc protocol by means of "suds". -- https://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
On Monday, August 13, 2012 at 7:53:32 PM UTC+3, andrea crotti wrote:
> I am in the situation where I am working on different projects that
> might potentially share a lot of code.
>
> I started to work on project A, then switched completely to project B
> and in the transiction I copied over a lot of code with the
> corresponding tests, and I started to modify it.
>
> Now it's time to work again on project A, but I don't want to copy
> things over again.
>
> I would like to design a simple and nice way to share between projects,
> where the things I want to share are simple but useful things as for
> example:
>
> class TempDirectory:
> """Create a temporary directory and cd to it on enter, cd back to
> the original position and remove it on exit
> """
> def __init__(self):
> self.oldcwd = getcwd()
> self.temp_dir = mkdtemp()
>
> def __enter__(self):
> logger.debug("create and move to temp directory %s" % self.temp_dir)
> return self.temp_dir
>
> def __exit__(self, type, value, traceback):
> # I first have to move out
> chdir(self.oldcwd)
> logger.debug("removing the temporary directory and go back to
> the original position %s" % self.temp_dir)
> rmtree(self.temp_dir)
>
>
> The problem is that there are functions/classes from many domains, so it
> would not make much sense to create a real project, and the only name I
> could give might be "utils or utilities"..
>
> In plus the moment the code is shared I must take care of versioning and
> how to link different pieces together (we use perforce by the way).
>
> If then someone else except me will want to use these functions then of
> course I'll have to be extra careful, designing really good API's and so
> on, so I'm wondering where I should set the trade-off between ability to
> share and burden to maintain..
>
> Anyone has suggestions/real world experiences about this?
I would try bit:
https://github.com/teambit/bit
Hope it helps.
--
https://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
andrea crotti wrote: 2012/8/14 Cameron Simpson c...@zip.com.au: Having just skimmed this thread, one thing I haven't quite seen suggested is this: Really do make a third utilities project, and treat the project and deploy as separate notions. So to actually run/deploy project A's code you'd have a short script that copied project A and the utilities project code into a tree and ran off that. Or even a simple process/script to update the copy of utilities in project A's area. So you don't share code on an even handed basis but import the utilities library into each project as needed. I do this (one my own very small scale) in one of two ways: - as needed, copy the desired revision of utilities into the project's library space and do perforce's equivalent of Mercurial's addremove on that library tree (comment update utilities to revision X). - keep a perforce work area for the utilities in your project A area, where your working project A can hook into it with a symlink or some deploy/copy procedure as suggested above. With this latter one you can push back into the utilities library from your live project, because you have a real checkout. So: projectAdir projectA-perforce-checkout utilities-perforce-checkout projectBdir projectB-perforce-checkout utilities-perforce-checkout Thanks, is more or less what I was going to do.. But I would not use symlinks and similar things, because then every user should set it up accordingly. Potentially we could instead use the perforce API to change the workspace mappings at run-time, and thus force perforce to checkout the files in the right place.. There is still the problem that people should checkout things from two places all the time instead of one.. SVN allows to define external dependencies, where one repository will actually checkout another one at a specific version. If SVN does it, I guess any decent SCM also provide such feature. Assuming our project is named 'common', and you have 2 projects A and B : A - common@rev1 B - common@rev2 Project A references the lib as A.common, B as B.common. You need to be extra carefull to never reference common as 'common' in any place. JM -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
2012/8/16 Jean-Michel Pichavant jeanmic...@sequans.com: SVN allows to define external dependencies, where one repository will actually checkout another one at a specific version. If SVN does it, I guess any decent SCM also provide such feature. Assuming our project is named 'common', and you have 2 projects A and B : A - common@rev1 B - common@rev2 Project A references the lib as A.common, B as B.common. You need to be extra carefull to never reference common as 'common' in any place. JM Unfortunately I think you guess wrong http://forums.perforce.com/index.php?/topic/553-perforce-svnexternals-equivalent/ Anyway with views and similar things is not that hard to implement the same thing.. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
2012/8/16 andrea crotti andrea.crott...@gmail.com: Unfortunately I think you guess wrong http://forums.perforce.com/index.php?/topic/553-perforce-svnexternals-equivalent/ Anyway with views and similar things is not that hard to implement the same thing.. I'm very happy to say that I finally made it! It took 3 hours to move / merge a few thousand lines around but everything seems to work perfectly now.. At the moment I'm just using symlinks, I'll see later if something smarter is necessary, thanks to everyone for the ideas. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
2012/8/14 Cameron Simpson c...@zip.com.au: Having just skimmed this thread, one thing I haven't quite seen suggested is this: Really do make a third utilities project, and treat the project and deploy as separate notions. So to actually run/deploy project A's code you'd have a short script that copied project A and the utilities project code into a tree and ran off that. Or even a simple process/script to update the copy of utilities in project A's area. So you don't share code on an even handed basis but import the utilities library into each project as needed. I do this (one my own very small scale) in one of two ways: - as needed, copy the desired revision of utilities into the project's library space and do perforce's equivalent of Mercurial's addremove on that library tree (comment update utilities to revision X). - keep a perforce work area for the utilities in your project A area, where your working project A can hook into it with a symlink or some deploy/copy procedure as suggested above. With this latter one you can push back into the utilities library from your live project, because you have a real checkout. So: projectAdir projectA-perforce-checkout utilities-perforce-checkout projectBdir projectB-perforce-checkout utilities-perforce-checkout Thanks, is more or less what I was going to do.. But I would not use symlinks and similar things, because then every user should set it up accordingly. Potentially we could instead use the perforce API to change the workspace mappings at run-time, and thus force perforce to checkout the files in the right place.. There is still the problem that people should checkout things from two places all the time instead of one.. Personally I become more and more resistent to cut/paste even for small things as soon as multiple people use it; you will never get to backport updates to even trivial code to all the copies. Cheers, Well sure, but on the other end as soon as multiple people use it you can't change any of the public functions signatures without being afraid that you'll break something.. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
Also looking at logilab-common I thought that it would be great if we could actually make this common library even open source, and use it as one of the other many external libraries. Since Python code is definitively not the the core business of this company I might even convince them, but the problem is that then all the internal people working on it would not be able to use the standard tools that they use with everything else.. Did anyone manage to convince his company to do something similar? -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
In plus I'm using perforce which doesn't have any svn:externals-like You can probably use views to this (http://www.perforce.com/perforce/r12.1/manuals/cmdref/o.views.html). Second problem is that one of the two projects has a quite insane requirement, which is to be able to re-run itself on a specific version depending on a value fetched from the database. You can probably play with PYTHONPATH to do that. The third problem is that from the moment is not just me using these things, how can I be sure that changing something will not break someone else code? That's always a problem with libraries you distribute, no matter what is the mechanism for distribution. If you go the views/link way. You can link to tags instead of trunk (or the perforce equivalent) and then client can still to know good version. If you go the PyPi route, you can specify package version in setup.py dependencies (foo==0.10.2) -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
2012/8/13 Rob Day robert@merton.oxon.org: I'd just create a module - called shared_utils.py or similar - and import that in both projects. It might be a bit messy if there's no 'unifying theme' to the module - but surely it'd be a lot less messy than your TempDirectory class, and anyone else who knows Python will understand 'import shared_utils' much more easily. I realise you might not want to say, but if you could give some idea what sort of projects these are, and what sorts of code you're trying to share, it might make things a bit clearer. I'm not really sure what your concerns about 'versioning and how to link different pieces together' are - what d you think could go wrong here? It's actually not so simple.. Because the two projects live in different parts of the repository with different people allowed to work on them, and they have to run on different machines.. In plus I'm using perforce which doesn't have any svn:externals-like thing as far as I know.. The thing I should do probably is to set up workspace (which contains *absolute* paths of the machines) with the right setting to make module available in the right position. Second problem is that one of the two projects has a quite insane requirement, which is to be able to re-run itself on a specific version depending on a value fetched from the database. This becomes harder if divide code around, but in theory I can use the changeset number which is like a SVN revision so this should be fine. The third problem is that from the moment is not just me using these things, how can I be sure that changing something will not break someone else code? I have unit tests on both projects plus the tests for the utils, but as soon as I separate them it becomes harder to test everything.. So well everything can have a solution probably, I just hope it's worth the effort.. Another thing which would be quite cool might be a import hook which fetches things from the repository when needed, with a simple bootstrap script for every project to be able to use this feature, but it only makes sense if I need this kind of feature in many projects. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
andrea crotti wrote: I am in the situation where I am working on different projects that might potentially share a lot of code. I started to work on project A, then switched completely to project B and in the transiction I copied over a lot of code with the corresponding tests, and I started to modify it. Now it's time to work again on project A, but I don't want to copy things over again. I would like to design a simple and nice way to share between projects, where the things I want to share are simple but useful things as for example: class TempDirectory: Create a temporary directory and cd to it on enter, cd back to the original position and remove it on exit def __init__(self): self.oldcwd = getcwd() self.temp_dir = mkdtemp() def __enter__(self): logger.debug(create and move to temp directory %s % self.temp_dir) return self.temp_dir def __exit__(self, type, value, traceback): # I first have to move out chdir(self.oldcwd) logger.debug(removing the temporary directory and go back to the original position %s % self.temp_dir) rmtree(self.temp_dir) The problem is that there are functions/classes from many domains, so it would not make much sense to create a real project, and the only name I could give might be utils or utilities.. In plus the moment the code is shared I must take care of versioning and how to link different pieces together (we use perforce by the way). If then someone else except me will want to use these functions then of course I'll have to be extra careful, designing really good API's and so on, so I'm wondering where I should set the trade-off between ability to share and burden to maintain.. Anyone has suggestions/real world experiences about this? I can think of logilab-common (http://www.logilab.org/848/) Having a company-wide python module properly distributed is one to achieve your goal. Without distributing your module to the public, there's a way to have a pypi-like server runnning on your private network : http://pypi.python.org/pypi/pypiserver/ JM Note : looks like pypi.python.org is having some trouble, the above link is broken. Search for recent announcement about pypiserver. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
On Mon, 13 Aug 2012 17:53:32 +0100, andrea crotti wrote: I started to work on project A, then switched completely to project B and in the transiction I copied over a lot of code with the corresponding tests, and I started to modify it. Now it's time to work again on project A, but I don't want to copy things over again. I would like to design a simple and nice way to share between projects, where the things I want to share are simple but useful things as for example: [...] The problem is that there are functions/classes from many domains, so it would not make much sense to create a real project, and the only name I could give might be utils or utilities.. I feel your pain. Copy and paste is perhaps the simplest, most seductive, and *worst* of the developer anti-patterns. I wish I had a solution to share with you, because I'm in the same position, but I don't. The best I have come up with is a combination of: 1) For really small code snippets, just forget about sharing code. If you need a five-line function Foo in ten projects, just re-write them (well, copy and paste them...) each time. Forget about trying to keep them syncronised. 2) For bigger functions/classes, put them in their own module which you can share across multiple projects. 3) Consider having a utilities module to include all the assorted bits and pieces. If your projects are in-house, then a utilities module makes more sense. If you're writing libraries for independent release, not so much. -- Steven -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
2012/8/14 Jean-Michel Pichavant jeanmic...@sequans.com: I can think of logilab-common (http://www.logilab.org/848/) Having a company-wide python module properly distributed is one to achieve your goal. Without distributing your module to the public, there's a way to have a pypi-like server runnning on your private network : http://pypi.python.org/pypi/pypiserver/ JM Note : looks like pypi.python.org is having some trouble, the above link is broken. Search for recent announcement about pypiserver. Thanks, yes we need something like this.. I'll copy the name probably, I prefer common to utils/utilities.. -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
On 13Aug2012 17:53, andrea crotti andrea.crott...@gmail.com wrote: | I am in the situation where I am working on different projects that | might potentially share a lot of code. | | I started to work on project A, then switched completely to project B | and in the transiction I copied over a lot of code with the | corresponding tests, and I started to modify it. | | Now it's time to work again on project A, but I don't want to copy | things over again. [...] | The problem is that there are functions/classes from many domains, so it | would not make much sense to create a real project, and the only name I | could give might be utils or utilities.. | | In plus the moment the code is shared I must take care of versioning and | how to link different pieces together (we use perforce by the way). [...] Having just skimmed this thread, one thing I haven't quite seen suggested is this: Really do make a third utilities project, and treat the project and deploy as separate notions. So to actually run/deploy project A's code you'd have a short script that copied project A and the utilities project code into a tree and ran off that. Or even a simple process/script to update the copy of utilities in project A's area. So you don't share code on an even handed basis but import the utilities library into each project as needed. I do this (one my own very small scale) in one of two ways: - as needed, copy the desired revision of utilities into the project's library space and do perforce's equivalent of Mercurial's addremove on that library tree (comment update utilities to revision X). - keep a perforce work area for the utilities in your project A area, where your working project A can hook into it with a symlink or some deploy/copy procedure as suggested above. With this latter one you can push back into the utilities library from your live project, because you have a real checkout. So: projectAdir projectA-perforce-checkout utilities-perforce-checkout projectBdir projectB-perforce-checkout utilities-perforce-checkout Personally I become more and more resistent to cut/paste even for small things as soon as multiple people use it; you will never get to backport updates to even trivial code to all the copies. Cheers, -- Cameron Simpson c...@zip.com.au The mere existence of a problem is no proof of the existence of a solution. - Yiddish Proverb -- http://mail.python.org/mailman/listinfo/python-list
Sharing code between different projects?
I am in the situation where I am working on different projects that might potentially share a lot of code. I started to work on project A, then switched completely to project B and in the transiction I copied over a lot of code with the corresponding tests, and I started to modify it. Now it's time to work again on project A, but I don't want to copy things over again. I would like to design a simple and nice way to share between projects, where the things I want to share are simple but useful things as for example: class TempDirectory: Create a temporary directory and cd to it on enter, cd back to the original position and remove it on exit def __init__(self): self.oldcwd = getcwd() self.temp_dir = mkdtemp() def __enter__(self): logger.debug(create and move to temp directory %s % self.temp_dir) return self.temp_dir def __exit__(self, type, value, traceback): # I first have to move out chdir(self.oldcwd) logger.debug(removing the temporary directory and go back to the original position %s % self.temp_dir) rmtree(self.temp_dir) The problem is that there are functions/classes from many domains, so it would not make much sense to create a real project, and the only name I could give might be utils or utilities.. In plus the moment the code is shared I must take care of versioning and how to link different pieces together (we use perforce by the way). If then someone else except me will want to use these functions then of course I'll have to be extra careful, designing really good API's and so on, so I'm wondering where I should set the trade-off between ability to share and burden to maintain.. Anyone has suggestions/real world experiences about this? -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
I'd just create a module - called shared_utils.py or similar - and import that in both projects. It might be a bit messy if there's no 'unifying theme' to the module - but surely it'd be a lot less messy than your TempDirectory class, and anyone else who knows Python will understand 'import shared_utils' much more easily. I realise you might not want to say, but if you could give some idea what sort of projects these are, and what sorts of code you're trying to share, it might make things a bit clearer. I'm not really sure what your concerns about 'versioning and how to link different pieces together' are - what d you think could go wrong here? On 13 August 2012 17:53, andrea crotti andrea.crott...@gmail.com wrote: I am in the situation where I am working on different projects that might potentially share a lot of code. I started to work on project A, then switched completely to project B and in the transiction I copied over a lot of code with the corresponding tests, and I started to modify it. Now it's time to work again on project A, but I don't want to copy things over again. I would like to design a simple and nice way to share between projects, where the things I want to share are simple but useful things as for example: class TempDirectory: Create a temporary directory and cd to it on enter, cd back to the original position and remove it on exit def __init__(self): self.oldcwd = getcwd() self.temp_dir = mkdtemp() def __enter__(self): logger.debug(create and move to temp directory %s % self.temp_dir) return self.temp_dir def __exit__(self, type, value, traceback): # I first have to move out chdir(self.oldcwd) logger.debug(removing the temporary directory and go back to the original position %s % self.temp_dir) rmtree(self.temp_dir) The problem is that there are functions/classes from many domains, so it would not make much sense to create a real project, and the only name I could give might be utils or utilities.. In plus the moment the code is shared I must take care of versioning and how to link different pieces together (we use perforce by the way). If then someone else except me will want to use these functions then of course I'll have to be extra careful, designing really good API's and so on, so I'm wondering where I should set the trade-off between ability to share and burden to maintain.. Anyone has suggestions/real world experiences about this? -- http://mail.python.org/mailman/listinfo/python-list -- Robert K. Day robert@merton.oxon.org -- http://mail.python.org/mailman/listinfo/python-list
Re: Sharing code between different projects?
On Tue, Aug 14, 2012 at 2:53 AM, andrea crotti andrea.crott...@gmail.com wrote: The problem is that there are functions/classes from many domains, so it would not make much sense to create a real project, and the only name I could give might be utils or utilities.. There's actually much merit in a generic utilities module. Keep things nicely segregated (ideally such that you know what things depend on what other, but at very least keep track of where one ends and another begins - that's trivial if everything's one function or one class, but less so when you have a family of related functions), and then you can consider promoting one block of code to stand-alone module. But in the meantime, you have a single module used in two places, even if it doesn't have a very clear definition as yet. ChrisA -- http://mail.python.org/mailman/listinfo/python-list
