Re: [Python-Dev] python sendmsg()/recvmsg() implementation
Done, it's at: http://bugs.python.org/issue6560 Kalman Gergely Aahz wrote: On Thu, Jul 23, 2009, K?lm?n Gergely wrote: This is the rewritten-from-scratch implementation of the sendmsg()/recvmsg() methods. Any comments / suggestions / flames are very welcome. Currently it supports what I need and I'm only releasing it, because I don't have much time to develop it further in the forseeable future (1-2 months). It is rewritten from scratch, using the python c-api documents. I've tried my best, but I wouldn't bet that it works as it's supposed to. I'd be glad if someone could give me a review on what I've done wrong. Please post this to bugs.python.org ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python sendmsg()/recvmsg() implementation
Hello
This is the rewritten-from-scratch implementation of the
sendmsg()/recvmsg() methods.
Any comments / suggestions / flames are very welcome. Currently it
supports what I need
and I'm only releasing it, because I don't have much time to develop it
further in the
forseeable future (1-2 months). It is rewritten from scratch, using the
python c-api
documents. I've tried my best, but I wouldn't bet that it works as it's
supposed to. I'd be
glad if someone could give me a review on what I've done wrong.
The core parts are implemented correctly (I think), the features that
are missing:
- using scatter/gather
- using it with non-stream oriented sockets (doesn't support addresses
/msg_name/)
These should be very easy to implement though. I will fix the errors
that are present right
now, and if no one takes up the task I will implement the missing
features also. You might
have to wait for it a little though.
Thanks in advance
Cheers,
Kalman Gergely
--- py3k_2/Modules/socketmodule.c 2009-07-23 17:07:55.474581000 +0200
+++ py3k/Modules/socketmodule.c 2009-07-23 17:22:16.880415500 +0200
@@ -2388,6 +2388,7 @@
return n;
}
+
/* s.recvfrom(nbytes [,flags]) method */
static PyObject *
@@ -2440,6 +2441,143 @@
Like recv(buffersize, flags) but also return the sender's address info.);
+/* s.recvmsg(datalen, controllen, flags) method */
+
+static PyObject *
+sock_recvmsg(PySocketSockObject *s, PyObject *args)
+{
+ PyObject *dbuf, *cbuf, *alist, *tmp;
+ ssize_t n = -1;
+ int status, timeout;
+ int rdlen, rclen, flags = 0;
+ struct msghdr mhdr;
+ struct cmsghdr *chdr;
+ struct iovec iov[1];
+
+ if (!PyArg_ParseTuple(args, ii|i:recvmsg, rdlen, rclen, flags))
+ return NULL;
+
+ if (rdlen 0 || rclen 0)
+ {
+ PyErr_SetString(PyExc_ValueError, negative buffersize in recvmsg);
+ return NULL;
+ }
+
+ /* allocate buffers */
+ dbuf = PyBytes_FromStringAndSize((char *) 0, rdlen);
+ if (dbuf == NULL)
+ {
+ return NULL;
+ }
+
+ cbuf = PyBytes_FromStringAndSize((char *) 0, rclen);
+ if (cbuf == NULL)
+ {
+ Py_DECREF(dbuf);
+ return NULL;
+ }
+
+ alist = PyList_New(0);
+ if (alist == NULL)
+ {
+ Py_DECREF(dbuf);
+ Py_DECREF(cbuf);
+ return NULL;
+ }
+
+ /* set up the msghdr struct */
+ memset(mhdr, 0, sizeof(struct msghdr));
+
+ // iov -- we use only one buffer, and don't use scatter-gather possible TODO
+ iov[0].iov_base = PyBytes_AS_STRING(dbuf);
+ iov[0].iov_len = rdlen;
+ memset(iov[0].iov_base, 0, iov[0].iov_len);
+
+ // msghdr
+ mhdr.msg_name = NULL;// TODO make use of this
+ mhdr.msg_namelen = 0;
+ mhdr.msg_iov = iov;
+ mhdr.msg_iovlen = 1;
+ mhdr.msg_control = PyBytes_AS_STRING(cbuf);
+ mhdr.msg_controllen = rclen;
+ mhdr.msg_flags = 0;
+ memset(mhdr.msg_control, 0, mhdr.msg_controllen);
+
+ /* call recvmsg() */
+ Py_BEGIN_ALLOW_THREADS
+ timeout = internal_select(s, 0);
+ if (!timeout)
+ n = recvmsg(s-sock_fd, mhdr, flags);
+ Py_END_ALLOW_THREADS
+
+ if (timeout == 1)
+ {
+ PyErr_SetString(socket_timeout, timed out);
+ goto err;
+ }
+
+ if (n 0)
+ {
+ s-errorhandler();
+ goto err;
+ }
+
+ /* process the ancillary data */
+ for (chdr = CMSG_FIRSTHDR(mhdr); chdr != NULL; chdr = CMSG_NXTHDR(mhdr, chdr))
+ {
+ tmp = Py_BuildValue((iiy#), chdr-cmsg_level, chdr-cmsg_type, (char *)CMSG_DATA(chdr),
+// TODO XXX ugly hack, to compute CMSG_DATA's size
+(int)chdr-cmsg_len - ((int)CMSG_DATA(chdr) - (int)chdr));
+ if (tmp == NULL)
+ goto err;
+
+ status = PyList_Append(alist, tmp);
+ Py_DECREF(tmp);
+
+ if (status == -1)
+ goto err;
+ }
+
+ /* if we received less than we anticipated, resize the buffer */
+ if (n != rdlen)
+ {
+ if (_PyBytes_Resize(dbuf, n) == -1)
+ {
+ Py_DECREF(cbuf);
+ Py_DECREF(alist);
+ return NULL;
+ }
+ }
+
+ /* assemble the final return value, watch out for offsets! */
+ tmp = PyTuple_New(3);
+ if (tmp == NULL)
+ goto err;
+
+ PyTuple_SetItem(tmp, 0, dbuf);
+ PyTuple_SetItem(tmp, 1, alist);
+ PyTuple_SetItem(tmp, 2, Py_BuildValue(i, mhdr.msg_flags));
+
+ /* dbuf and alist are now in tmp, remove reference to cbuf and return */
+ Py_DECREF(cbuf);
+
+ return tmp;
+
+err:
+ Py_DECREF(dbuf);
+ Py_DECREF(cbuf);
+ Py_DECREF(alist);
+ return NULL;
+}
+
+PyDoc_STRVAR(recvmsg_doc,
+recvmsg(datalen, controllen, flags) method - (bytes(), [(msglevel, msgtype, msgdata) ... ], flags)\n\
+\n\
+Returns a tuple with three elements, 0: data bytes, 1: list of tuples with three elements\n\
+containing msg_level, msg_type, msg_data, 2: msg_flags\n\
+Currently it's incapable of using multiple buffers and addresses.);
+
+
/* s.recvfrom_into(buffer[, nbytes [,flags]]) method */
static PyObject *
@@ -2655,6 +2793,140 @@
For IP sockets, the address is a pair (hostaddr, port).);
+/* s.sendmsg(data, [(msglevel, msgtype, msgdata), ...], flags) method */
+
+static PyObject *
+sock_sendmsg(PySocketSockObject *s, PyObject *args)
+{
+ Py_buffer dbuf;
+ PyObject *cbuf;
+ PyObject *tmp;
+ Py_buffer tmpdata;
+ PyObject *control;
+ size_t cbuflen = 0;
+ size_t
Re: [Python-Dev] python sendmsg()/recvmsg() implementation
On Thu, Jul 23, 2009, K?lm?n Gergely wrote: This is the rewritten-from-scratch implementation of the sendmsg()/recvmsg() methods. Any comments / suggestions / flames are very welcome. Currently it supports what I need and I'm only releasing it, because I don't have much time to develop it further in the forseeable future (1-2 months). It is rewritten from scratch, using the python c-api documents. I've tried my best, but I wouldn't bet that it works as it's supposed to. I'd be glad if someone could give me a review on what I've done wrong. Please post this to bugs.python.org -- Aahz (a...@pythoncraft.com) * http://www.pythoncraft.com/ The volume of a pizza of thickness 'a' and radius 'z' is given by pi*z*z*a ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python sendmsg()/recvmsg() implementation
Jean-Paul Calderone wrote:
On Tue, 09 Jun 2009 16:46:54 +0200, Kálmán Gergely
kalman.gerg...@duodecad.hu wrote:
Hello, my name is Greg.
I've just started using python after many years of C programming, and
I'm also new to the list. I wanted to clarify this
first, so that maybe I will get a little less beating for my
stupidity :)
Welcome!
[snip]
Browsing the net I've found a patch to the python core
(http://bugs.python.org/issue1194378), dated 2005.
First of all, I would like to ask you guys, whether you know of any
way of doing this FD passing magic, or that you know
of any 3rd party module / patch / anything that can do this for me.
Aside from the patch in the tracker, there are several implementations of
these APIs as third-party extension modules.
Since I'm fairly familiar with C and (not that much, but I feel the
power) python, I would take the challenge of writing
it, given that the above code is still somewhat usable. If all else
fails I would like to have your help to guide me through
this process.
What would be great is if you could take the patch in the tracker and get
it into shape so that it is suitable for inclusion. This would involve
three things, I think:
1. Write unit tests for the functionality (since the patch itself
provides
none)
2. Update the patch so that it again applies cleanly against trunk
3. Add documentation for the new APIs
Once this is done, you can get a committer to look at it and either
provide
more specific feedback or apply it.
Thanks,
Jean-Paul
___
Python-Dev mailing list
Python-Dev@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe:
http://mail.python.org/mailman/options/python-dev/synapse%40jasmin.hu
Hello again
So, after a little cleanup I've managed to integrate the code into
socketmodule.c/h. It works fine now, though I needed to
add it to Lib/socket.py, otherwise it wouldn't show up in the socket
module (I've searched for recvfrom and added it).
I've also cleaned up the code a little, fixed some codingstyle issues
(which might still exist).
Since I am not a python core developer the patch might still be in a
pretty outdated state. I'd like someone to look it over
and direct me to some documentation (the ones I've found so far were
pretty sketchy), for it to be acceptable for inclusion.
The sanity of the code is what matters to me the most. I've looked it
over though and found it in a sound state, but I guess
you guys might have a different opinion about that ;)
With writing the test cases, some documentation would be nice.
I've attached the patch generated with svn diff for revision 73434, and
the test code that I use to pass a file descriptor
between processes. It works just fine :).
Thanks
Kalman Gergely
Index: Lib/socket.py
===
--- Lib/socket.py (revision 73434)
+++ Lib/socket.py (working copy)
@@ -159,14 +159,14 @@
# All the method names that must be delegated to either the real socket
# object or the _closedsocket object.
_delegate_methods = (recv, recvfrom, recv_into, recvfrom_into,
- send, sendto)
+ recvmsg, send, sendto, sendmsg)
class _closedsocket(object):
__slots__ = []
def _dummy(*args):
raise error(EBADF, 'Bad file descriptor')
# All _delegate_methods must also be initialized here.
-send = recv = recv_into = sendto = recvfrom = recvfrom_into = _dummy
+send = recv = recv_into = sendto = recvfrom = recvfrom_into = recvmsg = sendmsg = _dummy
__getattr__ = _dummy
# Wrapper around platform socket objects. This implements
Index: Modules/socketmodule.c
===
--- Modules/socketmodule.c (revision 73434)
+++ Modules/socketmodule.c (working copy)
@@ -251,6 +251,9 @@
#ifdef HAVE_SYS_TYPES_H
#include sys/types.h
#endif
+#include sys/stat.h
+#include unistd.h
+#include structmember.h
/* Generic socket object definitions and includes */
#define PySocket_BUILDING_SOCKET
@@ -1840,6 +1843,7 @@
int optname;
int res;
PyObject *buf;
+ char *data;
socklen_t buflen = 0;
#ifdef __BEOS__
@@ -1852,6 +1856,36 @@
level, optname, buflen))
return NULL;
+#ifdef SO_PEERCRED
+ if( level == SOL_SOCKET optname == SO_PEERCRED ) {
+ /* Buffer length for struct ucred, ignore parameter. */
+ buflen = sizeof(int)*3;
+
+ /* Allocate ucred structure and data buffer. */
+ if( !( buf = PyType_GenericAlloc((PyTypeObject*)ucred_type,1) ) )
+ return NULL;
+ if( !( data = PyMem_Malloc(buflen) ) ) {
+ Py_DECREF(buf);
+ return NULL;
+ }
+
+ /* Use getsockopt to retrieve data. */
+ if( ( res = getsockopt(s-sock_fd,level,optname,data,buflen) )
+ 0 ) {
+ Py_DECREF(buf);
+ return s-errorhandler();
+ }
+
+ /* Write it out to object. */
+ ((PySocketUcredObject*)buf)-uid =
Re: [Python-Dev] python sendmsg()/recvmsg() implementation
Kálmán Gergely wrote: Since I am not a python core developer the patch might still be in a pretty outdated state. I'd like someone to look it over and direct me to some documentation (the ones I've found so far were pretty sketchy), for it to be acceptable for inclusion. The sanity of the code is what matters to me the most. I've looked it over though and found it in a sound state, but I guess you guys might have a different opinion about that ;) With writing the test cases, some documentation would be nice. Most unit tests these days are written based on either doctest or unittest. When adding new features to an existing module, it is usually best to follow the testing style already used for the rest of that module (in this case, that should be test/test_socket.py). The relevant question in the dev FAQ gives good pointers: http://www.python.org/dev/faq/#how-to-test-a-patch I've attached the patch generated with svn diff for revision 73434, and the test code that I use to pass a file descriptor between processes. It works just fine :). Uploading files to the tracker is generally the best option - patches tend to get lost if they're just sent to the mailing list. Cheers, Nick. -- Nick Coghlan | ncogh...@gmail.com | Brisbane, Australia --- ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
[Python-Dev] python sendmsg()/recvmsg() implementation
Hello, my name is Greg. I've just started using python after many years of C programming, and I'm also new to the list. I wanted to clarify this first, so that maybe I will get a little less beating for my stupidity :) I use python3 and Linux on arch x86 (production will be on x86_64, though this shouldn't matter much). The application that I'm presently working on is a network server. It would use separate processes to accept the connections, and to do the work (much like how apache prefork does). One process accept()s on the original socket and the received socket (client socket) will be read for a request. After the request is received and parsed this process (the controller) will choose one from its' children that is most capable of handling the said request. It would then pass the file descriptor through a socketpair to the appropriate children and go handle the next client. All works fine and smooth, but I realized that I need sendmsg()/recvmsg() to pass the FD. Since these are not implemented in the python socket module, and Linux has no other way to do this, I'm stuck. Fell flat on my face, too :) Browsing the net I've found a patch to the python core (http://bugs.python.org/issue1194378), dated 2005. First of all, I would like to ask you guys, whether you know of any way of doing this FD passing magic, or that you know of any 3rd party module / patch / anything that can do this for me. Since I'm fairly familiar with C and (not that much, but I feel the power) python, I would take the challenge of writing it, given that the above code is still somewhat usable. If all else fails I would like to have your help to guide me through this process. Thanks Kalman Gergely ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python sendmsg()/recvmsg() implementation
On Tue, 09 Jun 2009 16:46:54 +0200, Kálmán Gergely kalman.gerg...@duodecad.hu wrote: Hello, my name is Greg. I've just started using python after many years of C programming, and I'm also new to the list. I wanted to clarify this first, so that maybe I will get a little less beating for my stupidity :) Welcome! [snip] Browsing the net I've found a patch to the python core (http://bugs.python.org/issue1194378), dated 2005. First of all, I would like to ask you guys, whether you know of any way of doing this FD passing magic, or that you know of any 3rd party module / patch / anything that can do this for me. Aside from the patch in the tracker, there are several implementations of these APIs as third-party extension modules. Since I'm fairly familiar with C and (not that much, but I feel the power) python, I would take the challenge of writing it, given that the above code is still somewhat usable. If all else fails I would like to have your help to guide me through this process. What would be great is if you could take the patch in the tracker and get it into shape so that it is suitable for inclusion. This would involve three things, I think: 1. Write unit tests for the functionality (since the patch itself provides none) 2. Update the patch so that it again applies cleanly against trunk 3. Add documentation for the new APIs Once this is done, you can get a committer to look at it and either provide more specific feedback or apply it. Thanks, Jean-Paul ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
Re: [Python-Dev] python sendmsg()/recvmsg() implementation
On Tue, Jun 9, 2009 at 7:46 AM, Kálmán Gergelykalman.gerg...@duodecad.hu wrote: Hello, my name is Greg. I've just started using python after many years of C programming, and I'm also new to the list. I wanted to clarify this first, so that maybe I will get a little less beating for my stupidity :) I use python3 and Linux on arch x86 (production will be on x86_64, though this shouldn't matter much). The application that I'm presently working on is a network server. It would use separate processes to accept the connections, and to do the work (much like how apache prefork does). One process accept()s on the original socket and the received socket (client socket) will be read for a request. After the request is received and parsed this process (the controller) will choose one from its' children that is most capable of handling the said request. It would then pass the file descriptor through a socketpair to the appropriate children and go handle the next client. All works fine and smooth, but I realized that I need sendmsg()/recvmsg() to pass the FD. Since these are not implemented in the python socket module, and Linux has no other way to do this, I'm stuck. Fell flat on my face, too :) Browsing the net I've found a patch to the python core (http://bugs.python.org/issue1194378), dated 2005. First of all, I would like to ask you guys, whether you know of any way of doing this FD passing magic, or that you know of any 3rd party module / patch / anything that can do this for me. IIRC, this is already implemented in the multiprocessing package, which comes standard with Python 2.6 and 3.0 . It looks like the test is disabled in test/test_multiprocessing.py , and re-enabling it (on Windows) produces errors that make me think it's more an issue with the tests than with multiprocessing itself. Dig into it, see if you can get the tests to pass :) - Josiah Since I'm fairly familiar with C and (not that much, but I feel the power) python, I would take the challenge of writing it, given that the above code is still somewhat usable. If all else fails I would like to have your help to guide me through this process. Thanks Kalman Gergely ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/josiah.carlson%40gmail.com ___ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/mailman/options/python-dev/archive%40mail-archive.com
