Re: [Tutor] any best practice on how to glue tiny tools together
Le Fri, 6 Feb 2009 06:44:11 -0500, Kent Johnson ken...@tds.net a écrit : My first thought was, use shell pipelines and bash. Then I remembered, David Beazley shows how to use generators to implement a processing pipeline in Python: http://www.dabeaz.com/generators-uk/ see also http://en.wikipedia.org/wiki/Python_Pipelines -- la vida e estranya ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] any best practice on how to glue tiny tools together
On Fri, Feb 6, 2009 at 7:21 AM, spir denis.s...@free.fr wrote: see also http://en.wikipedia.org/wiki/Python_Pipelines That looks pretty dead, or at least unpublished. The google code project is almost two years old but it contains no code. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] any best practice on how to glue tiny tools together
spir wrote: Le Fri, 6 Feb 2009 06:44:11 -0500, Kent Johnson ken...@tds.net a écrit : My first thought was, use shell pipelines and bash. Then I remembered, David Beazley shows how to use generators to implement a processing pipeline in Python: http://www.dabeaz.com/generators-uk/ see also http://en.wikipedia.org/wiki/Python_Pipelines Thank you for the plug! The code is currently in flux so not available. I hope to have an alpha version out soon. -- Bob Gailer Chapel Hill NC 919-636-4239 ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] any best practice on how to glue tiny tools together
Kent Johnson wrote: On Fri, Feb 6, 2009 at 7:21 AM, spir denis.s...@free.fr wrote: see also http://en.wikipedia.org/wiki/Python_Pipelines That looks pretty dead, or at least unpublished. The google code project is almost two years old but it contains no code. It is sluggish but not dead. The code is on my PC - not ready to release. I started the google code to help establish the project. Since development is voluntary right now it is slow. Keep tuned... FWIW I was on target for an alpha version when I got sidetracked by writing a universal parser which I am now finalizing. Assistants are always welcome. -- Bob Gailer Chapel Hill NC 919-636-4239 ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] any best practice on how to glue tiny tools together
On Fri, Feb 6, 2009 at 4:11 AM, Daniel daniel.chaow...@gmail.com wrote: Hi Tutors, I want to use python to finish some routine data processing tasks automatically (on Windows). The main task could be split to sub small tasks. Each can be done by executing some small tools like awk or by some other python scripts. One example of such task is conducting a data processing job, including: use tool-A to produce some patterns. feed tool-B with these patterns to mine more related data repeat these tasks circularly until meeting some conditions. The real task includes more tools which run in parallel or sequential manner. I know how to do this with modules like subprocess, but the final python program looks somewhat messy and hard to adapt for changes. Do you have any best practices on this? My first thought was, use shell pipelines and bash. Then I remembered, David Beazley shows how to use generators to implement a processing pipeline in Python: http://www.dabeaz.com/generators-uk/ It's a fascinating read, it might take a couple of times to get it but it might fit your needs quite well. You would write a generator that wraps a subprocess call and use that to access external tools; other pieces and the control logic would be in Python. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
Re: [Tutor] any best practice on how to glue tiny tools together
These are really valuable info. I will try. Thanks a lot. On Fri, Feb 6, 2009 at 7:44 PM, Kent Johnson ken...@tds.net wrote: On Fri, Feb 6, 2009 at 4:11 AM, Daniel daniel.chaow...@gmail.com wrote: Hi Tutors, I want to use python to finish some routine data processing tasks automatically (on Windows). The main task could be split to sub small tasks. Each can be done by executing some small tools like awk or by some other python scripts. One example of such task is conducting a data processing job, including: use tool-A to produce some patterns. feed tool-B with these patterns to mine more related data repeat these tasks circularly until meeting some conditions. The real task includes more tools which run in parallel or sequential manner. I know how to do this with modules like subprocess, but the final python program looks somewhat messy and hard to adapt for changes. Do you have any best practices on this? My first thought was, use shell pipelines and bash. Then I remembered, David Beazley shows how to use generators to implement a processing pipeline in Python: http://www.dabeaz.com/generators-uk/ It's a fascinating read, it might take a couple of times to get it but it might fit your needs quite well. You would write a generator that wraps a subprocess call and use that to access external tools; other pieces and the control logic would be in Python. Kent ___ Tutor maillist - Tutor@python.org http://mail.python.org/mailman/listinfo/tutor
