Hi Fred,
thanks a lot for the debug patch.
It works with trial:
---
# trial test/test_sendmsg.py
twext.python.test.test_sendmsg
SendmsgTestCase
test_roundtrip ... [OK]
test_sendSubProcessFD ... [ERROR]
test_wrongTypeAncillary ... [OK]
===============================================================================
[ERROR]
Traceback (most recent call last):
Failure: twext.python.test.test_sendmsg.ExitedWithStderr:
Traceback (most recent call last):
File
"/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-9.1-RELEASE-amd64.egg/twext/python/test/pullpipe.py",
line 22, in <module>
fd, description = recvfd(int(sys.argv[1]))
File
"/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-9.1-RELEASE-amd64.egg/twext/python/sendfd.py",
line 65, in recvfd
[unpackedFD] = unpack("i", packedFD)
struct.error: unpack requires a string argument of length 4
twext.python.test.test_sendmsg.SendmsgTestCase.test_sendSubProcessFD
-------------------------------------------------------------------------------
Ran 3 tests in 0.122s
FAILED (errors=1, successes=2)
# ls -l /tmp/debug.dat*
total 2
-rw-r--r-- 1 root wheel 31 Feb 13 23:05 debug.dat.48537
-rw-r--r-- 1 root wheel 55 Feb 13 23:05 debug.dat.48538
# cat /tmp/debug.dat.4853*
recvmsg: fd [8] [0] result [d]
recvmsg: fd [8] [0] result [5]
recvmsg: value:
3 0 0 0
# rm /tmp/debug.dat.4853*
---
But produces no output in real life:
---
[caldav-0] --- <exception caught here> ---
[caldav-0] File
"/usr/local/lib/python2.7/site-packages/twisted/internet/kqreactor.py", line
279, in _doWriteOrRead
[caldav-0] why = selectable.doRead()
[caldav-0] File
"/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-9.1-RELEASE-amd64.egg/twext/internet/sendfdport.py",
line 281, in doRead
[caldav-0] fd, description = recvfd(self.fd)
[caldav-0] File
"/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-9.1-RELEASE-amd64.egg/twext/python/sendfd.py",
line 65, in recvfd
[caldav-0] [unpackedFD] = unpack("i", packedFD)
[caldav-0] struct.error: unpack requires a string argument of length 4
...
# ls -l /tmp/debug.dat.485*
-rw-r--r-- 1 root wheel 0 Feb 13 23:07 /tmp/debug.dat.48563
-rw-r--r-- 1 root wheel 0 Feb 13 23:07 /tmp/debug.dat.48577
-rw-r--r-- 1 root wheel 0 Feb 13 23:07 /tmp/debug.dat.48578
-rw-r--r-- 1 root wheel 0 Feb 13 23:07 /tmp/debug.dat.48579
-rw-r--r-- 1 root wheel 0 Feb 13 23:07 /tmp/debug.dat.48580
#
---
I have no explanation for that.
Axel
Am 13.02.2013 um 20:20 schrieb Fredrik Unger:
> Hi,
>
> Ok.
>
> My problems were not with the kqreactor.py so I have not so much experience
> of that does but .
>
> To concentrate on the sendfd.py problem:
>
> [caldav-0] File
> "/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-8.2-RELEASE-p9-amd64.egg/twext/python/sendfd.py",
> line 65, in recvfd
> [caldav-0] [unpackedFD] = unpack("i", packedFD)
> [caldav-0] struct.error: unpack requires a string argument of
> length 4
>
> What I did was to add a debug to file thing in sendmsg.c
> Attached a small test patch (against current trunk).
> Patch it, then you need to compile the sendmsg.c module :
>
> gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall
> -Wstrict-prototypes -g -O2 -fstack-protector --param=ssp-buffer-size=4
> -Wformat -Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC
> -I/usr/include/python2.7 -c sendmsg.c -o sendmsg.o
>
>
> gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro
> -Wl,-z,relro -Wl,-z,relro -g -O2 -fstack-protector --param=ssp-buffer-size=4
> -Wformat -Wformat-security -Werror=format-security -D_FORTIFY_SOURCE=2
> sendmsg.o -o sendmsg.so
>
> or build whole calendar server.
>
> Then run the test
>
> trial test/test_sendmsg.py
>
> In ls /tmp/debug.dat.PID should be the output..
> In my case :
> recvmsg: fd [8] [0] result [d]
>
> Now, you can adapt the things to track your problem and start
> a real calendar server, at the crash there should be several pid files..
> maybe you get some data out of them..
>
> /Fred
>
>
>
> On 02/13/2013 12:05 AM, Axel Rau wrote:
>> I did not make any progress. I tried
>> - in sendmsg_sendmsg to check for the right type arg (with PyErr_Format, got
>> no error)
>> - to print out the args with PySys_WriteStdout (did not get anything)
>> - I'm not even sure, if this error is related to my problem:
>> test_sendSubProcessFD ... [ERROR]
>> - If I try control_message->cmsg_type = 0;, I get Errno 22 in sendfd, as
>> expected
>>
>> I need more pointers...
>>
>> Axel
>>
>> Am 08.02.2013 um 13:30 schrieb Fredrik Unger:
>>
>>> Hi,
>>>
>>> I found some problem with sendfd.c some time ago. [1]
>>>
>>> My problem was with 64 bit Debian and twext/python/sendmsg.c
>>>
>>> Have a look and see if there is anything that is FreeBSD non friendly there.
>>>
>>> You can try with the tests :
>>> cd twext/python
>>> trial test/test_sendmsg.py
>>> That could be easier to debug than the full code.
>>>
>>> On my setup I failed test_sendSubProcessFD now, but I just did a quick
>>> test. Sources are not up to date or anything..
>>>
>>> You can also add some printouts to the C code, and see if you find
>>> something bad.
>>>
>>> It is most probably some flags or so that might not be covered for BSD ?
>>> and sendmsg.c is the interface to the operating system from python..
>>>
>>> Just some suggestions. If you need more pointers let me know, but I have no
>>> experience with FreeBSD..
>>>
>>> /Fred
>>>
>>> [1]
>>> https://lists.macosforge.org/pipermail/calendarserver-users/2012-August/002032.html
>>>
>>> On 02/08/2013 01:04 PM, Axel Rau wrote:
>>>>
>>>> Am 02.02.2013 um 17:39 schrieb Axel Rau:
>>>>
>>>>> [caldav-0] File
>>>>> "/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-8.2-RELEASE-p9-amd64.egg/twext/python/sendfd.py",
>>>>> line 65, in recvfd
>>>>> [caldav-0] [unpackedFD] = unpack("i", packedFD)
>>>>> [caldav-0] struct.error: unpack requires a string argument of
>>>>> length 4
>>>> This is hard to debug. I did not make much progress.
>>>> If I switch to kqueue-reactor, I'm getting another FD problem when
>>>> starting listening:
>>>> ---
>>>> Log opened.
>>>> twistd 12.1.0 (/usr/local/bin/python2.7 2.7.3) starting up.
>>>> reactor class: twext.internet.kqreactor.KQueueReactor.
>>>> ControlSocket starting on '/var/db/caldavd/caldavd.sock'
>>>> Traceback (most recent call last):
>>>> File "/usr/local/bin/twistd", line 14, in<module>
>>>> run()
>>>> File "/usr/local/lib/python2.7/site-packages/twisted/scripts/twistd.py",
>>>> line 27, in run
>>>> app.run(runApp, ServerOptions)
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/application/app.py", line
>>>> 652, in run
>>>> runApp(config)
>>>> File "/usr/local/lib/python2.7/site-packages/twisted/scripts/twistd.py",
>>>> line 23, in runApp
>>>> _SomeApplicationRunner(config).run()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/application/app.py", line
>>>> 390, in run
>>>> self.postApplication()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/scripts/_twistd_unix.py",
>>>> line 230, in postApplication
>>>> self.startApplication(self.application)
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/scripts/_twistd_unix.py",
>>>> line 340, in startApplication
>>>> service.IService(application).privilegedStartService()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/application/service.py",
>>>> line 277, in privilegedStartService
>>>> service.privilegedStartService()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/application/service.py",
>>>> line 277, in privilegedStartService
>>>> service.privilegedStartService()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-9.1-RELEASE-amd64.egg/calendarserver/tap/caldav.py",
>>>> line 487, in privilegedStartService
>>>> super(GroupOwnedUNIXServer, self).privilegedStartService()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/application/internet.py",
>>>> line 105, in privilegedStartService
>>>> self._port = self._getPort()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/application/internet.py",
>>>> line 133, in _getPort
>>>> 'listen%s' % (self.method,))(*self.args, **self.kwargs)
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/internet/posixbase.py",
>>>> line 406, in listenUNIX
>>>> p.startListening()
>>>> File "/usr/local/lib/python2.7/site-packages/twisted/internet/unix.py",
>>>> line 303, in startListening
>>>> self.startReading()
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/twisted/internet/abstract.py",
>>>> line 416, in startReading
>>>> self.reactor.addReader(self)
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-9.1-RELEASE-amd64.egg/twext/internet/kqreactor.py",
>>>> line 84, in addReader
>>>> self._updateRegistration(fd, KQ_FILTER_READ, KQ_EV_ADD|KQ_EV_ENABLE)
>>>> File
>>>> "/usr/local/lib/python2.7/site-packages/Calendar_and_Contacts_Server-4.2-py2.7-freebsd-9.1-RELEASE-amd64.egg/twext/internet/kqreactor.py",
>>>> line 74, in _updateRegistration
>>>> self._kq.control([ev], 0, 0)
>>>> OSError: [Errno 9] Bad file descriptor
>>>> ---
>>>> Any help appreciated,
>>>> Axel
>>>> ---
>>>> PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
>>>>
>>>> _______________________________________________
>>>> calendarserver-dev mailing list
>>>> calendarserver-dev@lists.macosforge.org
>>>> https://lists.macosforge.org/mailman/listinfo/calendarserver-dev
>>>
>>> _______________________________________________
>>> calendarserver-dev mailing list
>>> calendarserver-dev@lists.macosforge.org
>>> https://lists.macosforge.org/mailman/listinfo/calendarserver-dev
>>
>> ---
>> PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
>>
>> _______________________________________________
>> calendarserver-dev mailing list
>> calendarserver-dev@lists.macosforge.org
>> https://lists.macosforge.org/mailman/listinfo/calendarserver-dev
>
> <debug.patch>_______________________________________________
> calendarserver-dev mailing list
> calendarserver-dev@lists.macosforge.org
> https://lists.macosforge.org/mailman/listinfo/calendarserver-dev
---
PGP-Key:29E99DD6 ☀ +49 151 2300 9283 ☀ computing @ chaos claudius
_______________________________________________
calendarserver-dev mailing list
calendarserver-dev@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-dev
