Re: Suitability of Python for daemon processes
On Thu, Oct 29, 2009 at 12:04:08AM -0400, Ben Boeckel wrote: Hopefully we have 2.4 as the minimum version. I'm pretty sure 2.4 is ubiquitous even on the enterprise-y distributions, but if we need older, we can try for that as well. Just a note. If you're targeting RHEL/CentOS4 as your minimum, you need to target python-2.3 as your minimum python version. -Toshio pgplAY2pBBlt5.pgp Description: PGP signature ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore den...@ausil.us wrote: On Sunday 25 October 2009 06:26:49 pm Jeroen van Meeuwen wrote: And koji.fedoraproject.org, no? koji is a mod_python app. it doesnt run as a daemon at all. but it it all python. There are python daemons in the system though. The builders run kojid, which is a daemon, and the process that triggers repo regeneration (kojira) is a daemon. Of course, these are not public facing -- they really only talk to the hub. The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security. It's entirely possible to create large and complex systems with Python. It's expressiveness and interpreted nature are a big help here. Robustness is probably more a factor of design and good coding than the language itself. That being said, the relative ease of programming in Python, coupled with its exception handling, likely give it an advantage over C in this department (over the same development time at least). That's my opinion; others may disagree (and of course Python is not the only rapid development language with good exception handling). Security threats are everywhere. With Python at least you have to worry much less about buffer overflows, but of course you can introduce security flaws in any language. Obviously, as an interpreted language, system load and speed are where Python suffers. There are optimization tricks, but you'll never get close to execution speed of C. You can rewrite crucial portions in C, though I rarely see that actually done. ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Wed, Oct 28, 2009 at 05:08:03PM -0400, Mike McLean wrote: On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore den...@ausil.us wrote: The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security. nod By and large I agree with you. One thing further to think about is that becoming dependent on a tool written in an interpreted language means that you need to install that language on your system and may become tied to a specific version of that language. In theory, this shouldn't be worse than tying yourself to a specific C library but in practice I've found that parallel installing interpreters and their libraries is a lot less supported than parallel installing a C lib. Using python in Fedora Infrastructure probably isn't too big a deal as we have abundant python programmers here to port things forward if the main developers don't but it is something to consider with other languages or in other environments. -Toshio pgphNJhXYnri1.pgp Description: PGP signature ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Toshio Kuratomi wrote: On Wed, Oct 28, 2009 at 05:08:03PM -0400, Mike McLean wrote: On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore dennis- omedqnbl...@public.gmane.org wrote: The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security. nod By and large I agree with you. One thing further to think about is that becoming dependent on a tool written in an interpreted language means that you need to install that language on your system and may become tied to a specific version of that language. In theory, this shouldn't be worse than tying yourself to a specific C library but in practice I've found that parallel installing interpreters and their libraries is a lot less supported than parallel installing a C lib. Hopefully we have 2.4 as the minimum version. I'm pretty sure 2.4 is ubiquitous even on the enterprise-y distributions, but if we need older, we can try for that as well. Using python in Fedora Infrastructure probably isn't too big a deal as we have abundant python programmers here to port things forward if the main developers don't but it is something to consider with other languages or in other environments. We're aiming to get it to work on RHEL/CentOS 4 (base, not + EPEL) and Debian stable so that current mirrors running systems that old don't need to upgrade the OS to start using our software (I'm pretty sure we have the hardware to run test instances of the software with various setups). If the libraries we end up using just aren't available, I think RHEL 5 will be a suitable minimum. When Python 3 comes looming over any distribution that starts using our software, we should be able to help port things over. I'd expect Fedora to be the first for this to happen to, but I could see Gentoo having howtos for getting 3.0 as the main Python before that (not that servers would be running such a setup anyways). -Toshio - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCAAGBQJK6RQ4AAoJEKaxavVX4C1Xw54P/2z9WxTx03+EQLF9LMjtrzSV gdwCAoV4kmjI0eiWN1uaMVZp8qRWuWDF7WW74wl4wbbwoi5ipc+iekuM8a91Mzzp jwE/MAbRufX6zCFkZL48ToU9Hvga02CeCIC6wmQIsruJ/TT5Y3O3XdHyVXSB1DD/ +okXSLTyL0PpVKBG+v/qy6Gn+qNzNBLTIzJQkRG4bQS7ThbfteumD5gdGDpx51nm RNze8l+e3giY/XQzO+tf9025xv9Wac3e0DzWSTcDYKtiaEDGbeF/asr767gV5Q8f RjbqzKcLUHn15Rp8jAWl4KtGo9OgbL9MQi3nPsFArqbzb836cuO/LTNkG+BnbhPj 49sVele1+GMcOOlwdtT4qvAOg7WH668FgNc5x0zAx6xxP1RGsi5oLVEFtRReUXx1 t4sOFT/qIKTSbJW0BrOvFJ7m2DNaq5v/JE5vYSZ30KccjiW4PAzBzRPNdl2tbuD7 j9o9N9bTTe+HFSI+6VBiwvf193oS8nS0oZxBcUaBsvoR5FlKw1GggclHVEYBU3jK rj3dyBh7OH9QlukX8ueBOp7yIydsz5AkWNMwRS9LtsRmubGBvrO5qE24DE0M7YVv MkeQwPSwkAQ08OUKlK8NanHoMdFNSSkqJeEyHWNgHP0oekpjwGHyBml+j0Vgpiy1 eiTRw28QhGdA94AQOl4V =X7Gb -END PGP SIGNATURE- ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mike McLean wrote: On Mon, Oct 26, 2009 at 12:41 AM, Dennis Gilmore dennis- omedqnbl...@public.gmane.org wrote: On Sunday 25 October 2009 06:26:49 pm Jeroen van Meeuwen wrote: And koji.fedoraproject.org, no? koji is a mod_python app. it doesnt run as a daemon at all. but it it all python. There are python daemons in the system though. The builders run kojid, which is a daemon, and the process that triggers repo regeneration (kojira) is a daemon. Of course, these are not public facing -- they really only talk to the hub. The daemon distinction might be wrong thing to fixate on here. There is nothing in that distinction that should exclude python (or most any language). I think the real factors of concern are: size, complexity, speed, system load, robustness, and security. True. I think the speed at which we can deliver with Python makes it much more attractive, even if not just at first. We could something similar to what git did with bash and replace the Python with C incrementally if necessary, but I think that if we go about it right, it shouldn't be much of an issue. It's entirely possible to create large and complex systems with Python. It's expressiveness and interpreted nature are a big help here. Robustness is probably more a factor of design and good coding than the language itself. That being said, the relative ease of programming in Python, coupled with its exception handling, likely give it an advantage over C in this department (over the same development time at least). That's my opinion; others may disagree (and of course Python is not the only rapid development language with good exception handling). I agree here. This thread is a fact-finding mission and research. We wanted to make sure we weren't headed down the wrong road before we headed out. Security threats are everywhere. With Python at least you have to worry much less about buffer overflows, but of course you can introduce security flaws in any language. Of course. Obviously, as an interpreted language, system load and speed are where Python suffers. There are optimization tricks, but you'll never get close to execution speed of C. You can rewrite crucial portions in C, though I rarely see that actually done. One place where we had concerns were when dealing with the shopping list I referred to elsewhere in the thread. This list is 60MB on one of the Ubuntu mirror directories we have here (our Fedora mirror has been decommissioned while we await more disk space). I imagine Fedora's will be much larger due to the drpm files and secondary architectures put even more into it. We have code that is able to extract information without loading the entire file into our process space. It's very close to what it would be if it were C except for a .split() call on a string (but it's simple enough that a sscanf could do it as well if needed) and the memory/strcmp stuff Python has baked in. Hopefully we will have unit tests and timing on our code this weekend. - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCAAGBQJK6RZ8AAoJEKaxavVX4C1XjsMP/0g54tOwnu0MhsH+uhZDx23N uTHMs0CCOZaG+e4VJR3xocOU/Xul5IfZ2fDodKai9YwsA9S3ORuvLR0riKh0IbLW Js8RJW7e2oBWDHUK9nCXIGbuJoPhQrwD+3x88LctxXIy6I34Sw+d8GvrT7vTxgdp S2keUtAgOOm1JGw+JEAyutTa28yRGnFRLZmopgGAXNtUuFf2nvl77Dm1/R1T/9OE YcFykF9eEt8LeZ/9M895eGswmASH3LFPTeYOhmz1g5xPHauu1y7wHoHBkfeFieVa mhtMWLaA8ZfhLKt8FUBUALc0KcDlv42yqyCq7FeKipaaPpyWoHGf6Ce4ZOKgLGPs 5KcDDfVg7xxYH9xoTJ0kiYfy/7ExDr70WUwFg4oRVAl6sEZhVl6+Iz+64UXGVU0z 3o5QH7OKOkyKI9iQheg0b3sXLuhNGyrGz+b0vaEYj7TtoQWHk9qV5UEnoawsTXw+ f51GiQdeZmRNZS1N4m1reX2tUtw0ol1cHcpfpvdqOZazLaSVDwu7CbTyIfalmy79 fDc/yPWWPdcS+mU9LbgqBcEng4lzo/aQSMWHDDzk+r5uIuTibGzslP+hB/fnuuws ibZvpaDZhLElmqcZbAqMZdHVvS5DCXUjeLcPcx5qJBZ3xrcLnO3/GHapXVec3mIr d9UZxP4emQ7cOl8DMkbJ =Nhy3 -END PGP SIGNATURE- ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote: Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place. Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists. CPU performance can come into play if you are performing hashes or compression of the data to be transferred. I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback. ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Mon, 26 Oct 2009, Chuck Anderson wrote: On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote: Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place. Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists. CPU performance can come into play if you are performing hashes or compression of the data to be transferred. I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback. Very true, if this behaves similarly to rsync. Reading over a large change set to transmit only small changes is very resource intensive everywhere but the network. -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mike McGrath wrote: On Mon, 26 Oct 2009, Chuck Anderson wrote: On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote: Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place. Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists. CPU performance can come into play if you are performing hashes or compression of the data to be transferred. I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback. Very true, if this behaves similarly to rsync. Reading over a large change set to transmit only small changes is very resource intensive everywhere but the network. -Mike The way we are planning to handle updates should keep this small. The network side knows nothing of the on-disk file structure and the tree-creator part knows nothing of the network other than it's where the things I need come from, so updates won't be what's changed in this directory? so much as what do I not have that is on my shopping list? which is then asked for of the network. - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCAAGBQJK5mVtAAoJEKaxavVX4C1XJVQP/RgFr9vXeqFv+fbASL6HWAr4 rUYsltcI7U5+9j7jmsbh5K+8FvOJOmdc2erEaE+hmOowdv7jvCw46ywLjcU/ZbkY VyKMFeDsfmfd8geaujhQZULYbiDjmpFS6+uqX45ONBvIXk1L7f2zrnN9sTciSx7Z XOmT4V9NwKnHEoIKWNaYnb7dt4Q3/8xgJ53ZdF4JNB3tJ23IsRtzCRZOofPvfy4Y NdiAOxz5V0HY/tP4mmMH147W4QCL2lpCBLstLwcT/7rc5z3pDknPR4p8iQN4xvAn 7Q+y/12ewqxcGn8O6hKTdlG0CBt6YVeYegonZYp8V+BJ74ozaqDiR+IKVIXbJBZt VwTzdEhiBQfAb/BVZz+Vg4YW03WxitKAKlMQ7oGKjtHuTC5j/3EVX6Gio1Lo4EHX 4DsW9XRGqDEPwqwwgKsIXMkjfNXRkdKQQzamFKVO6XTzsyJaJ+Hsv1Az0+De4SQ+ lLEgrCdJ5JLrj3+yoWoH0sk7AJDLpyrxGSaYEs0wvz+AepJK5pXVd++OivTuGdWk lvXuSB15HG5tzKzDT45yFOR3vBhY1Bsa2j5uNBKrHUdjOuMZL/n/lfxdDH0e+GxF RE3b8/P/LV6qGDZNyWIEVgZraqpbDohKOI98M3Tdo1kiLZtIY+/ulKtg/EuIUEra hpc7/IiTPVQqe82zvbYR =2gfL -END PGP SIGNATURE- ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Chuck Anderson wrote: On Sun, Oct 25, 2009 at 06:46:05PM -0400, Ben Boeckel wrote: Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place. Speaking from experience, the network isn't always the bottleneck. I/O performance is often a performance problem, especially when walking the directory tree to build filelists. AIUI, the tree will only be walked over during update composition and applying an update to the mirror (updates are done out-of-tree so that there is no catching a server with its pants down per se. CPU performance can come into play if you are performing hashes or compression of the data to be transferred. Yes, we'll have to keep an eye on this. As things come down from the tubes, the hashes should be working so they'll most likely be waiting on the network to get done with its files. I suggest you post your message to the Fedora mirror-list-d where I'm sure you'll get lots of feedback. Will do. Thanks for the tip. - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCAAGBQJK5mYyAAoJEKaxavVX4C1XPAUQAIcSdP2Gy4plniWtP6cIrUyC Fq0DKkZyOZg24Nng6iCe6kyB7yGOK30D6pn77WRedc6eFygdhe9OZWNjc0yZm98a QruyNOzoqfbWg4g77vkFbYTzidiI6G6wgoLMtnVokApIfK8Y8MDbkHIcrlPOm7xp 4+3P+eHH+lZbw9LL2CJTairLWG94sZ9tvHfdlXN8yj7NwI8mpratj6I6jZPU4Y0v 8YS7m47rEZZzvkZQ/MPJ/qVmDttH9ZzryBo0wypZcn9K9pKs37Q345y2gjUOA3ZG ssBvWwEzuuIVmJZKpHUTOh2/06ilN53j2Q/paxwG3kvsUJz6+EARfPClbd4jKpaf tD0GdQH+5hF900bwesu51uLkgatkk/HHOLFn77Y3o8lkk90O643NAHiMN/s6+jlk gjYJjsJVkm4JRilNjn4DV/2cdAERjnQeFYm/USGapnhxtbWkaQ4CMLKKqXIm8KmK khV9+YsVstv4Grj2uqNSDvmeXBiwedDFZFfj8auris9fxPqycFfv71L5UxT8kBxq /n6ETIRipwL29W4KHEJQSmfG0HIiyJ4lzO7REMxOahLSxvBWJyBU+8aDYhnmM7F0 ybbGARc6Deq/i/vsKmtsElR/xR7RZsOvppqy0XpBHgZO/fSaXhNbjAs2bgIBhD+N nZ985xlfyQEHMB0qRGX1 =8OzK -END PGP SIGNATURE- ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Sun, 25 Oct 2009, Ben Boeckel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, I am working on a project designed to mirror large, changing archives of software in a manner that ensures data integrity and atomic updates using a peer-to-peer protocol. My team and I would like to design this software such that it provides a suitable replacement for rsync. As Fedora is among the leading distributions, I would like to solicit your opinions on the implementation. We have designed much of the architecture, but have not implemented anything yet. One issue we wish to address currently is that of interpreted vs. compiled languages. My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. If so, it would greatly simplify the implementation and allow it to be more robust. Python's built-in libraries and facilities provide much of the path and network manipulation that the daemon requires. Using the Python standard libraries allows us to rely upon a well-tested base and focus on higher-level issues. What are your opinions, as system administrators, on using Python for long-running daemon processes when the developers are explicitly mindful of memory considerations? I've generally had better luck with C/C++ based daemons from a systems admin point of view. But we've used plenty of python based daemons that worked just fine. I think the problem is the lower barrier to using python means it's easier for less experienced developers to create python daemons. Really though, if it's done right, python daemons can be just as good as C daemons. And extending Python is very easy as you mentioned with using pythons built in libraries. If you're concerned about performance though it shouldn't take much to do the basics of what you want and compare C to python. Sometimes they're identical, sometimes C wins but I don't think I've seen python win yet (with performance). But it is much easier to work with python :) -Mike ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mike McGrath wrote: On Sun, 25 Oct 2009, Ben Boeckel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, I am working on a project designed to mirror large, changing archives of software in a manner that ensures data integrity and atomic updates using a peer-to-peer protocol. My team and I would like to design this software such that it provides a suitable replacement for rsync. As Fedora is among the leading distributions, I would like to solicit your opinions on the implementation. We have designed much of the architecture, but have not implemented anything yet. One issue we wish to address currently is that of interpreted vs. compiled languages. My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. If so, it would greatly simplify the implementation and allow it to be more robust. Python's built-in libraries and facilities provide much of the path and network manipulation that the daemon requires. Using the Python standard libraries allows us to rely upon a well-tested base and focus on higher-level issues. What are your opinions, as system administrators, on using Python for long-running daemon processes when the developers are explicitly mindful of memory considerations? I've generally had better luck with C/C++ based daemons from a systems admin point of view. But we've used plenty of python based daemons that worked just fine. I think the problem is the lower barrier to using python means it's easier for less experienced developers to create python daemons. Really though, if it's done right, python daemons can be just as good as C daemons. And extending Python is very easy as you mentioned with using pythons built in libraries. If you're concerned about performance though it shouldn't take much to do the basics of what you want and compare C to python. Sometimes they're identical, sometimes C wins but I don't think I've seen python win yet (with performance). But it is much easier to work with python :) -Mike Thanks for your reply. Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place. - --Ben -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) iQIcBAEBCAAGBQJK5NUuAAoJEKaxavVX4C1XwboQANvsgFvNHSuYUFGNT6Vze/4u fCxzohcTnM+UosCEKDbSzknVlPhcoRGq71Lm1bju/Spsr8o1VHy18a5+nlEv7FlE icyo/J1ls7diWDMnA/zkoadQUlzg2nb5qpxV5jZVCrNWf3oDmxGuTLb1f9KlucQV Z2gROXyJw9ALvZJ+nY3HYE5MEBa/w44RkS/HoE0PDFeToMufnv728JXJ6AE3dTVs IEIq9EBhFZ/wqvEShdoz/UqmVTx7Y9CfD/31PN5yat9eGPaomXrN9jaG4LB4l2sV A2TuaV0lC+qy6woScnIC0WMfREc5QHxQLXag9L8DOJN/nnRCV3fKKFxY3zW8xK92 cIDhUqjWWXDBc5kAuxwyId5UpKD3KJmd8ZoyGNwuYGdEff96thiAOurKIYpmKjXK r8eI5BarfyeELOvMo5sb4vi1WEpzxQN+xjsrUGEM3BtvXy7+rsIwkpUkL20/5gYG HYDo3xQG0ya1XX27Tm6+pbORoWW6YjhgJ8JN+acNAiopLP2P04/ErLyC4Jvga4+u G1FSd8hvv9gz3I2uDf98PD/npye2yj/w/gWPwofPI5Ono9mJIPCHPzD89WinFYE8 BARSfJaWhNv1tM6BffpRg51t1w4D82ulVWdPMRs5224HlRSozoaoVLkMYcRWEL+O q+mTgi/7jKZ31pEjwxqV =PF8I -END PGP SIGNATURE- ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Sun, 25 Oct 2009, Ben Boeckel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Mike McGrath wrote: On Sun, 25 Oct 2009, Ben Boeckel wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Hi, I am working on a project designed to mirror large, changing archives of software in a manner that ensures data integrity and atomic updates using a peer-to-peer protocol. My team and I would like to design this software such that it provides a suitable replacement for rsync. As Fedora is among the leading distributions, I would like to solicit your opinions on the implementation. We have designed much of the architecture, but have not implemented anything yet. One issue we wish to address currently is that of interpreted vs. compiled languages. My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. If so, it would greatly simplify the implementation and allow it to be more robust. Python's built-in libraries and facilities provide much of the path and network manipulation that the daemon requires. Using the Python standard libraries allows us to rely upon a well-tested base and focus on higher-level issues. What are your opinions, as system administrators, on using Python for long-running daemon processes when the developers are explicitly mindful of memory considerations? I've generally had better luck with C/C++ based daemons from a systems admin point of view. But we've used plenty of python based daemons that worked just fine. I think the problem is the lower barrier to using python means it's easier for less experienced developers to create python daemons. Really though, if it's done right, python daemons can be just as good as C daemons. And extending Python is very easy as you mentioned with using pythons built in libraries. If you're concerned about performance though it shouldn't take much to do the basics of what you want and compare C to python. Sometimes they're identical, sometimes C wins but I don't think I've seen python win yet (with performance). But it is much easier to work with python :) -Mike Thanks for your reply. Seeing as it is a mirroring daemon, the network is the bottleneck. If it isn't then either you're sitting next door, our implementation is bad, or the hardware shouldn't be a mirror in the first place. With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11arch=i386 -Mike [1] By 'ourselves' I mean Matt Domsch who, quite honestly, has designed the most robust, fault tolerant dynamic content system we run. ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Sun, Oct 25, 2009 at 06:14:14PM -0400, Ben Boeckel wrote: My team and I would like to know whether the community would be accepting of such a project (which includes a daemon) if it were written in Python rather than C or C++. No problems here, MirrorManager includes several daemon processes written in python. The only real problem I have with python is the memory manager. I have to constantly think gee, if I read this file line by line and do something with it, will my RAM needs grow to 10-20x the size of the file? And I love python on x86_64 - it makes people buy twice as much RAM from Dell! * So, go for it. If it turns out that python is inefficient for parts of the problem, you can always split those parts out. I had to split out the mirrorlist_server bit from the mod_wsgi app, to get the benefits of shared pages (fork() on a 100+MB process) w/o killing Apache with duplicate processes but no sharing. * (no kidding: the mirrorlist_server process takes 125MB on x86_32, 250MB on x86_64. ugh.) -- Matt Domsch Technology Strategist, Dell Office of the CTO linux.dell.com www.dell.com/linux ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On 10/25/2009 11:51 PM, Mike McGrath wrote: With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11arch=i386 And koji.fedoraproject.org, no? And translation is going through Django these days? -- Jeroen ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On 10/26/2009 12:37 AM, Mike McGrath wrote: On Mon, 26 Oct 2009, Jeroen van Meeuwen wrote: On 10/25/2009 11:51 PM, Mike McGrath wrote: With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11arch=i386 And koji.fedoraproject.org, no? Yes but I think it's more along the lines of a python script that gets called when a page gets loaded, it doesn't actually run statefully like our tg apps do. But the backend does run python foo, right? And translation is going through Django these days? That's true, django is another one we run. I bet there's a few more sitting around that we haven't thought of yet :) I bet too ;-)) -- Jeroen ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
Re: Suitability of Python for daemon processes
On Sunday 25 October 2009 06:26:49 pm Jeroen van Meeuwen wrote: On 10/25/2009 11:51 PM, Mike McGrath wrote: With all my babbling I forgot to mention we do already run python daemons in Fedora Infrastructure. Func is one, TurboGears (though it's wrapped in mod_wsgi) and one that we wrote ourselves[1] is our mirrorlist server. It's the backend that powers: http://mirrors.fedoraproject.org/mirrorlist?repo=fedora-11arch=i386 And koji.fedoraproject.org, no? koji is a mod_python app. it doesnt run as a daemon at all. but it it all python. signature.asc Description: This is a digitally signed message part. ___ Fedora-infrastructure-list mailing list Fedora-infrastructure-list@redhat.com https://www.redhat.com/mailman/listinfo/fedora-infrastructure-list
