Bug#780530: [calendarserver]

[Rahul Amaram]

On Wednesday 03 February 2016 11:41 PM, Ximin Luo wrote:

On 03/02/16 13:33, Ximin Luo wrote:

As part of the fix to (k) earlier, I pushed another patch to a side branch of 
calendarserver: [1]. You haven't applied it to debian/sid yet, but I think it may be 
necessary. To test, you should try to add/remove events with non-ascii unicode characters 
in them such as "ßßß". If it fails, then try the patch and if it works please 
add it to git as well.

X

[1] 
https://anonscm.debian.org/cgit/calendarserver/calendarserver.git/diff/debian/patches/unicode-fixes.patch?h=debian/_wip_sid=4476a73ff4df39baa297606e00e66241f371178c


I will see how I could test unicode characters. Also, I do not know the impact 
of the change. Should I get it reviewed by upstream?


For me, I could reproduce the bug by adding an event in a client (e.g. icedove+iceowl) 
called "ßßß" or some other thing, then the event wouldn't actually be created, 
and I could see some decode errors in the caldavd error logs.

The change should have minimal impact - it merely allows that function to 
accept more values for self.scheduleTag than it was doing previously (raising 
exception when it was of type 'unicode' containing non-ascii chars). But yes it 
would be good to ask upstream to review it.


My mistake - the "adding-and-removing events test" that I just described, 
triggered the bug that I described in (k) above and was fixed by my patch to pg8000.

OTOH the patch I am talking about here ([1] from above), fixes a bug that 
appeared in my log which was *also* a unicode error. I *guess* (but I'm not 
sure) that it happens the local client syncs with the server, and *old events* 
(that the client and server both already have [note]) *already contains* 
unicode characters. The server trieds to generate an etag for it, and 7.0 fails 
for this. The exception looks like this:

2016-01-23 22:49:26+0100 [-] [caldav-0]  [-] [twext.enterprise.jobs.jobitem#error] 
JobItem: 759, WorkItem: 113481 failed: [Failure instance: Traceback: : 'ascii' codec can't decode byte 0xe2 in position 
335: ordinal not in range(128)
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1274:unwindGenerator
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1128:_inlineCallbacks
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/txdav/caldav/datastore/sql.py:4336:upgradeData
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1274:unwindGenerator
2016-01-23 22:49:26+0100 [-] [caldav-0] ---  ---
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1128:_inlineCallbacks
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/txdav/caldav/datastore/sql.py:4032:updateDatabase
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/txdav/caldav/datastore/sql.py:3906:_generateEtag
2016-01-23 22:49:26+0100 [-] [caldav-0] ]

You *might* need to patch 
/usr/lib/python2.7/dist-packages/twext/enterprise/jobs/jobitem.py to say 
log.error([etc], exc_info=sys.exc_info()) to get this stack trace to show, I 
can't remember exactly what I did.

[note] because of (k) you will either need to create such an event in server 
<7.0 then upgrade to 7.0, or else install our newer patched version of pg8000.

X

I have been unable to reproduce the unicode bug related to logging. I 
created a event with the non-ascii characters you gave in both a fresh 
installation and older installation and in both cases it worked perfectly.


The pg8000 fix seems necessary though and that patch has already been 
included.


Now the only pending action is to upload pg8000, post which 
calendarserver could finally be uploaded. Request you to do at the earliest.


Thanks,
Rahul.

Bug#780530: [calendarserver]

[Ximin Luo]

On 03/02/16 04:47, Rahul Amaram wrote:
> On Wednesday 03 February 2016 12:05 AM, Ximin Luo wrote:
>>> iv. Upload new packages twextpy and pg8000
>>>
>> I've just uploaded these. I fixed a few minor things before uploading, 
>> please check git and review them. In general, I run "lintian -i -I 
>> --pedantic --color auto xxx.changes" to catch things like that, and it's a 
>> good habit to get into.
> I had created two entries in debian/changelog because I had already tagged 
> the first release version and did not want to delete the tags. I have now, 
> however, deleted the tags and recreated them. Like you said, I think it is 
> best to create tags, once the packages are uploaded.
> Also, I always run lintian, but don't use the --pedantic option. Will start 
> using it hereafter.

Ah yes, not deleting/recreating tags is good practice in general. However, for 
Debian at this time, FTP is the primary source for "what is a particular 
version" and the git repos are secondary, so it's OK to go back and fix up 
history.

In the future maybe/hopefully Debian will integrate the FTP and git systems 
better, then if that happens there ought to be automatic enforcement of "can't 
push tag xxx because it's not in FTP yet".

>>
>> As part of the fix to (k) earlier, I pushed another patch to a side branch 
>> of calendarserver: [1]. You haven't applied it to debian/sid yet, but I 
>> think it may be necessary. To test, you should try to add/remove events with 
>> non-ascii unicode characters in them such as "ßßß". If it fails, then try 
>> the patch and if it works please add it to git as well.
>>
>> X
>>
>> [1] 
>> https://anonscm.debian.org/cgit/calendarserver/calendarserver.git/diff/debian/patches/unicode-fixes.patch?h=debian/_wip_sid=4476a73ff4df39baa297606e00e66241f371178c
>>
> I will see how I could test unicode characters. Also, I do not know the 
> impact of the change. Should I get it reviewed by upstream?
> 

For me, I could reproduce the bug by adding an event in a client (e.g. 
icedove+iceowl) called "ßßß" or some other thing, then the event wouldn't 
actually be created, and I could see some decode errors in the caldavd error 
logs.

The change should have minimal impact - it merely allows that function to 
accept more values for self.scheduleTag than it was doing previously (raising 
exception when it was of type 'unicode' containing non-ascii chars). But yes it 
would be good to ask upstream to review it.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Ximin Luo]

On 03/02/16 13:33, Ximin Luo wrote:
>>>
>>> As part of the fix to (k) earlier, I pushed another patch to a side branch 
>>> of calendarserver: [1]. You haven't applied it to debian/sid yet, but I 
>>> think it may be necessary. To test, you should try to add/remove events 
>>> with non-ascii unicode characters in them such as "ßßß". If it fails, then 
>>> try the patch and if it works please add it to git as well.
>>>
>>> X
>>>
>>> [1] 
>>> https://anonscm.debian.org/cgit/calendarserver/calendarserver.git/diff/debian/patches/unicode-fixes.patch?h=debian/_wip_sid=4476a73ff4df39baa297606e00e66241f371178c
>>>
>> I will see how I could test unicode characters. Also, I do not know the 
>> impact of the change. Should I get it reviewed by upstream?
>>
> 
> For me, I could reproduce the bug by adding an event in a client (e.g. 
> icedove+iceowl) called "ßßß" or some other thing, then the event wouldn't 
> actually be created, and I could see some decode errors in the caldavd error 
> logs.
> 
> The change should have minimal impact - it merely allows that function to 
> accept more values for self.scheduleTag than it was doing previously (raising 
> exception when it was of type 'unicode' containing non-ascii chars). But yes 
> it would be good to ask upstream to review it.
> 

My mistake - the "adding-and-removing events test" that I just described, 
triggered the bug that I described in (k) above and was fixed by my patch to 
pg8000.

OTOH the patch I am talking about here ([1] from above), fixes a bug that 
appeared in my log which was *also* a unicode error. I *guess* (but I'm not 
sure) that it happens the local client syncs with the server, and *old events* 
(that the client and server both already have [note]) *already contains* 
unicode characters. The server trieds to generate an etag for it, and 7.0 fails 
for this. The exception looks like this:

2016-01-23 22:49:26+0100 [-] [caldav-0]  [-] 
[twext.enterprise.jobs.jobitem#error] JobItem: 759, WorkItem: 113481 failed: 
[Failure instance: Traceback: : 'ascii' 
codec can't decode byte 0xe2 in position 335: ordinal not in range(128)
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1274:unwindGenerator
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1128:_inlineCallbacks
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/txdav/caldav/datastore/sql.py:4336:upgradeData
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1274:unwindGenerator
2016-01-23 22:49:26+0100 [-] [caldav-0] ---  ---
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1128:_inlineCallbacks
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/txdav/caldav/datastore/sql.py:4032:updateDatabase
2016-01-23 22:49:26+0100 [-] [caldav-0] 
/usr/lib/python2.7/dist-packages/txdav/caldav/datastore/sql.py:3906:_generateEtag
2016-01-23 22:49:26+0100 [-] [caldav-0] ]

You *might* need to patch 
/usr/lib/python2.7/dist-packages/twext/enterprise/jobs/jobitem.py to say 
log.error([etc], exc_info=sys.exc_info()) to get this stack trace to show, I 
can't remember exactly what I did.

[note] because of (k) you will either need to create such an event in server 
<7.0 then upgrade to 7.0, or else install our newer patched version of pg8000.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Ximin Luo]

On 29/01/16 08:42, Rahul Amaram wrote:
> On Friday 29 January 2016 05:34 AM, Ximin Luo wrote:
>> Hey just a quick reply for this (I will answer the other things later) - no 
>> I don't need to wait, I just had other stuff to do in the meantime. :) I 
>> will upload your packages soon, hopefully this weekend when I get some free 
>> time at FOSDEM. X 
> 
> Thanks. The good news is we now have all the issues sorted out. Upgrade 
> directly from 3.x works (yipee!) and I have identified the cause of the tasks 
> not being enabled. I had accidentally removed a config option which gets 
> distributed by default with the package config file. So, in your case, 
> post-upgrade the tasks should work. Could you check on this? Here are the 
> final actions items:
> 

Great! It will take me a while to test tasks as I was not using them myself, 
but I'll try to find some time for it soon.

> i. Get python-xattr version bumped to 0.7.5. Already raised a ticket for this 
> - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813075.
> 
> ii. [fixed]
> 
> iii. If possible try to get rid of these two errors in log files (these do 
> not seem to be blocking functionality as of now).
> 
> 2016-01-29 12:44:18+0530 [-] [twext.enterprise.jobs.queue#error] workCheck: 
> jobqueue is no longer overloaded
> 
> 2016-01-29 12:37:11+0530 [-] [caldav-1] [PooledMemCacheProtocol,client] 
> [twistedcaldav.memcachepool.MemCachePool#error] Memcache error: Invalid type 
> for key: , expecting a string; request: set 
> cacheToken:/calendars/__uids__/F343A113-0C2B-5A69-B049-A4C72FCAEF9B/ 
> 40aed427-a35f-41dd-9f79-fefc56ff8006
> 

I think the first one actually isn't an error, and I did try for a bit myself 
to figure out the second problem, but I gave up because it seemed not too 
important. If we don't figure it out before the remaining issues are sorted, 
I'm happy to upload, and just file another open bug for it (and keep poking 
upstream...)

> 
> iv. Upload new packages twextpy and pg8000
> 

I've just uploaded these. I fixed a few minor things before uploading, please 
check git and review them. In general, I run "lintian -i -I --pedantic --color 
auto xxx.changes" to catch things like that, and it's a good habit to get into.

As part of the fix to (k) earlier, I pushed another patch to a side branch of 
calendarserver: [1]. You haven't applied it to debian/sid yet, but I think it 
may be necessary. To test, you should try to add/remove events with non-ascii 
unicode characters in them such as "ßßß". If it fails, then try the patch and 
if it works please add it to git as well.

X

[1] 
https://anonscm.debian.org/cgit/calendarserver/calendarserver.git/diff/debian/patches/unicode-fixes.patch?h=debian/_wip_sid=4476a73ff4df39baa297606e00e66241f371178c

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

On Wednesday 03 February 2016 12:05 AM, Ximin Luo wrote:

iv. Upload new packages twextpy and pg8000


I've just uploaded these. I fixed a few minor things before uploading, please check git 
and review them. In general, I run "lintian -i -I --pedantic --color auto 
xxx.changes" to catch things like that, and it's a good habit to get into.
I had created two entries in debian/changelog because I had already 
tagged the first release version and did not want to delete the tags. I 
have now, however, deleted the tags and recreated them. Like you said, I 
think it is best to create tags, once the packages are uploaded.
Also, I always run lintian, but don't use the --pedantic option. Will 
start using it hereafter.


As part of the fix to (k) earlier, I pushed another patch to a side branch of 
calendarserver: [1]. You haven't applied it to debian/sid yet, but I think it may be 
necessary. To test, you should try to add/remove events with non-ascii unicode characters 
in them such as "ßßß". If it fails, then try the patch and if it works please 
add it to git as well.

X

[1] 
https://anonscm.debian.org/cgit/calendarserver/calendarserver.git/diff/debian/patches/unicode-fixes.patch?h=debian/_wip_sid=4476a73ff4df39baa297606e00e66241f371178c

I will see how I could test unicode characters. Also, I do not know the 
impact of the change. Should I get it reviewed by upstream?


Thanks,
Rahul.

Bug#780530: [calendarserver]

[Rahul Amaram]

On Friday 29 January 2016 01:12 PM, Rahul Amaram wrote:


ii. Figure out a fix for the bug with python-psutil 3.4.1.


This has been resolved. I have pushed the fix to the repo.

Bug#780530: [calendarserver]

[Rahul Amaram]

On Friday 29 January 2016 05:34 AM, Ximin Luo wrote:
Hey just a quick reply for this (I will answer the other things later) 
- no I don't need to wait, I just had other stuff to do in the 
meantime. :) I will upload your packages soon, hopefully this weekend 
when I get some free time at FOSDEM. X 


Thanks. The good news is we now have all the issues sorted out. Upgrade 
directly from 3.x works (yipee!) and I have identified the cause of the 
tasks not being enabled. I had accidentally removed a config option 
which gets distributed by default with the package config file. So, in 
your case, post-upgrade the tasks should work. Could you check on this? 
Here are the final actions items:


i. Get python-xattr version bumped to 0.7.5. Already raised a ticket for 
this - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=813075.


ii. Figure out a fix for the bug with python-psutil 3.4.1.

iii. If possible try to get rid of these two errors in log files (these 
do not seem to be blocking functionality as of now).


2016-01-29 12:44:18+0530 [-] [twext.enterprise.jobs.queue#error] 
workCheck: jobqueue is no longer overloaded


2016-01-29 12:37:11+0530 [-] [caldav-1] [PooledMemCacheProtocol,client] 
[twistedcaldav.memcachepool.MemCachePool#error] Memcache error: Invalid 
type for key: , expecting a string; request: set 
cacheToken:/calendars/__uids__/F343A113-0C2B-5A69-B049-A4C72FCAEF9B/ 
40aed427-a35f-41dd-9f79-fefc56ff8006



iv. Upload new packages twextpy and pg8000


Because of the super-powers of uploading new packages that has been 
granted to you, and as with great power comes great responsibility, iv. 
is definitely on your plate :p. See if you can help out with ii. and 
iii. I will follow up on action item i from my end.


Thanks,
Rahul.

Bug#780530: [calendarserver]

[Rahul Amaram]

On Wednesday 27 January 2016 03:15 AM, Rahul Amaram wrote:


Ok. I have included this as a patch. I am almost done with everything, 
except for the following bug while ugrading. After upgarding from 
5.2.2 as per the instructions in READE.Debian (which I have updated), 
I am seeing the following errors in error.log whenever I try to 
connect via Icedove:


===

2016-01-27 02:54:58+0530 [BinaryBoxProtocol,1,] Unhandled Error
Traceback (most recent call last):
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 954, 
in _commandReceived

deferred = self.dispatchCommand(box)
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 
1011, in dispatchCommand

return maybeDeferred(responder, box)
  File 
"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 
150, in maybeDeferred

result = f(*args, **kw)
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 
1100, in doit

return maybeDeferred(aCallable, **kw).addCallback(
---  ---
  File 
"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 
150, in maybeDeferred

result = f(*args, **kw)
  File 
"/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", line 
685, in logStats

self.observer.logStats(stats)
  File 
"/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", line 
378, in logStats

self.systemStats = SystemMonitor()
  File 
"/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", line 
587, in __init__
"cpu count" : psutil.NUM_CPUS if psutil is not None 
else -1,
exceptions.AttributeError: 'module' object has no attribute 
'NUM_CPUS'


2016-01-27 02:54:58+0530 [BinaryBoxProtocol,1,] Amp server or network 
failure unhandled by client application.  Dropping connection!  To 
avoid, add errbacks to ALL remote commands!

Traceback (most recent call last):
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 978, 
in ampBoxReceived

self._commandReceived(box)
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 957, 
in _commandReceived

deferred.addCallback(self._safeEmit)
  File 
"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 
317, in addCallback

callbackKeywords=kw)
  File 
"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 
306, in addCallbacks

self._runCallbacks()
---  ---
  File 
"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 
588, in _runCallbacks

current.result = callback(current.result, *args, **kw)
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 989, 
in _safeEmit

aBox._sendTo(self.boxSender)
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 667, 
in _sendTo

proto.transport.loseConnection()
exceptions.AttributeError: 'DispatchingSender' object has no 
attribute 'transport'


2016-01-27 02:54:58+0530 [-] [caldav-1] [BinaryBoxProtocol,client] 
[calendarserver.accesslog#error] Unable to format event {'log_level': 

Bug#780530: [calendarserver]

[Ximin Luo]

On 29/01/16 00:46, Rahul Amaram wrote:
>> Also, you can go ahead and upload the two new packages twext and pg8000. I 
>> don't anticipate any more changes.
> Let me know when the packages are upgraded. Or do you to wait till all 
> calendarserver issues are resolved?
> 

Hey just a quick reply for this (I will answer the other things later) - no I 
don't need to wait, I just had other stuff to do in the meantime. :) I will 
upload your packages soon, hopefully this weekend when I get some free time at 
FOSDEM.

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

On Sunday 24 January 2016 04:13 AM, Ximin Luo wrote:

(k) Another bug that hit me with 7.0: I couldn't save events with unicode characters such 
as "ß" in them. Looking at the error log got me this:

2016-01-23 22:40:03+0100 [-] [caldav-0]  [-] [twistedcaldav.storebridge#error] 
Error while handling (calendar) PUT: 'ascii' codec can't decode byte 0xc3 in 
position 355: ordinal not in range(128)

Adding some "import traceback; traceback.print_exc()" to storebridge.py told me 
that the error was from pg8000/core.py:

2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] Traceback (most recent call last):
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/twistedcaldav/storebridge.py", line 2869, in 
http_PUT
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] response = (yield 
self.storeComponent(component, options=options))
[..]
many many lines omitted, omg
[..]
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py", line 
83, in execute
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] self.realCursor.execute(sql, 
args)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/pg8000/core.py", line 910, in execute
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] self._c.execute(self, 
operation, args)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/pg8000/core.py", line 2031, in execute
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] val = send_func(value)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/pg8000/core.py", line 1353, in text_out
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] return 
v.encode(self._client_encoding)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] UnicodeDecodeError: 'ascii' codec 
can't decode byte 0xc3 in position 322: ordinal not in range(128)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] [twistedcaldav.storebridge#error] 
Error while handling (calendar) PUT: 'ascii' codec can't decode byte 0xc3 in 
position 322: ordinal not in range(128)

Adding some debugging code reveals that text_out() is called with both unicode() and 
str() objects. When I do PUT, the event object is serialised into a str 
"BEGIN:VCALENDER..." and eventually makes its way to text_out(), causing an 
exception.

Experimenting a bit, the errors go away and I can save the event again, if I 
patch pg8000 like this:


diff --git a/pg8000/core.py b/pg8000/core.py
index 246e0b6..14093cc 100644
--- a/pg8000/core.py
+++ b/pg8000/core.py
@@ -1350,6 +1350,7 @@ class Connection(object):
  self.ParameterStatusReceived += self.handle_PARAMETER_STATUS
  
  def text_out(v):

+if not isinstance(v, text_type): return v
  return v.encode(self._client_encoding)
  
  def time_out(v):



I will forward this bug upstream to pg8000. It may *also* be a bug on the 
calendarserver side, but I'm pretty sure pg8000 should be fixed as well - since 
text_out() is an internal function, if pg8000 has any requirements on the input 
it receives from calendarserver, it should raise a better error message 
*before* it gets to text_out().

X



Ok. I have included this as a patch. I am almost done with everything, 
except for the following bug while ugrading. After upgarding from 5.2.2 
as per the instructions in READE.Debian (which I have updated), I am 
seeing the following errors in error.log whenever I try to connect via 
Icedove:


===

2016-01-27 02:54:58+0530 [BinaryBoxProtocol,1,] Unhandled Error
Traceback (most recent call last):
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 954, 
in _commandReceived

deferred = self.dispatchCommand(box)
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 1011, 
in dispatchCommand

return maybeDeferred(responder, box)
  File 
"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 150, 
in maybeDeferred

result = f(*args, **kw)
  File 
"/usr/lib/python2.7/dist-packages/twisted/protocols/amp.py", line 1100, 
in doit

return maybeDeferred(aCallable, **kw).addCallback(
---  ---
  File 
"/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 150, 
in maybeDeferred

result = f(*args, **kw)
  File 
"/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", line 
685, in logStats

self.observer.logStats(stats)
  File 
"/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", line 
378, in logStats

self.systemStats = SystemMonitor()
  File 
"/usr/lib/python2.7/dist-packages/calendarserver/accesslog.py", line 
587, in __init__
"cpu count" : psutil.NUM_CPUS if psutil is not None 
else -1,
exceptions.AttributeError: 'module' 

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 18:48, Rahul Amaram wrote:
> On Wednesday 20 January 2016 10:43 PM, Ximin Luo wrote:
>> On 20/01/16 17:26, Ximin Luo wrote:
>>> On 20/01/16 15:23, Ximin Luo wrote:
 Hey, I just tested 7.0+dfsg from git and it seems to work fine. Thanks for 
 the work, and looking forward to the eventual upload! Just a few notes:

 (a) [..]

 (b) [..]

 (c) [..]

 (d) [..]

>>> (e) [..]
>>>
>>> (f) [..]
>>>
>>> (g) [..]
>>>
>>> (h) [..]
>>
>> (j) [..]
>>

(k) Another bug that hit me with 7.0: I couldn't save events with unicode 
characters such as "ß" in them. Looking at the error log got me this:

2016-01-23 22:40:03+0100 [-] [caldav-0]  [-] [twistedcaldav.storebridge#error] 
Error while handling (calendar) PUT: 'ascii' codec can't decode byte 0xc3 in 
position 355: ordinal not in range(128)

Adding some "import traceback; traceback.print_exc()" to storebridge.py told me 
that the error was from pg8000/core.py:

2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] Traceback (most recent call last):
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/twistedcaldav/storebridge.py", line 2869, in 
http_PUT
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] response = (yield 
self.storeComponent(component, options=options))
[..]
many many lines omitted, omg
[..]
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py", line 
83, in execute
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] self.realCursor.execute(sql, 
args)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/pg8000/core.py", line 910, in execute
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] self._c.execute(self, 
operation, args)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/pg8000/core.py", line 2031, in execute
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] val = send_func(value)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-]   File 
"/usr/lib/python2.7/dist-packages/pg8000/core.py", line 1353, in text_out
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] return 
v.encode(self._client_encoding)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] UnicodeDecodeError: 'ascii' codec 
can't decode byte 0xc3 in position 322: ordinal not in range(128)
2016-01-23 22:46:11+0100 [-] [caldav-1]  [-] [twistedcaldav.storebridge#error] 
Error while handling (calendar) PUT: 'ascii' codec can't decode byte 0xc3 in 
position 322: ordinal not in range(128)

Adding some debugging code reveals that text_out() is called with both 
unicode() and str() objects. When I do PUT, the event object is serialised into 
a str "BEGIN:VCALENDER..." and eventually makes its way to text_out(), causing 
an exception.

Experimenting a bit, the errors go away and I can save the event again, if I 
patch pg8000 like this:


diff --git a/pg8000/core.py b/pg8000/core.py
index 246e0b6..14093cc 100644
--- a/pg8000/core.py
+++ b/pg8000/core.py
@@ -1350,6 +1350,7 @@ class Connection(object):
 self.ParameterStatusReceived += self.handle_PARAMETER_STATUS
 
 def text_out(v):
+if not isinstance(v, text_type): return v
 return v.encode(self._client_encoding)
 
 def time_out(v):


I will forward this bug upstream to pg8000. It may *also* be a bug on the 
calendarserver side, but I'm pretty sure pg8000 should be fixed as well - since 
text_out() is an internal function, if pg8000 has any requirements on the input 
it receives from calendarserver, it should raise a better error message 
*before* it gets to text_out().

X

-- 
GPG: ed25519/56034877E1F87C35
GPG: rsa4096/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

On Saturday 23 January 2016 12:20 AM, Ximin Luo wrote:

On 20/01/16 18:29, Rahul Amaram wrote:

(d) The git history for the debian/wheezy branch of calendarserver is also 
messed up - you have version 3.2 committed on top of 5.2.2. It involves a bit 
of git magic to fix, which I can do, if you want to avoid the hassle yourself 
and you give me access.

I've also asked to be added to the calendarserver group on Alioth, so that I 
can make these changes myself.

Feel free to do the changes. Not sure how it got messed up. Thanks.

OK, I've rewritten the history so things make more sense looking at the logs - all the 
"Imported Upstream version" commits are now in the correct linear sequence, as well as 
the "Imported Debian patch" commits, and debian/wheezy branches away from the main 
history at the correct place.

You'll need to update your own repo to point all your local branches to the new 
rewritten history. If it's too messy, you can just clone it again.

For transparency, I've attached the script I used, which is a combination of 
writing .git/info/grafts and git-filter-branch. You can run it on your own 
local old repo, and in theory you should even get the same commit SHA1 objects 
out. (But be careful! This will drop the gpg signatures on some old tags, so be 
careful not to push these back out.)

For reference, here's what you should expect the script to output, twice, to 
check that I didn't backdoor the source :)


debian/3.2+dfsg-4+deb7u1 fe8398cfe0b68277b4c95e116e8d37e6b88f5061
debian/3.2+dfsg-4+deb7u2 fb51628bb376ea59c7650a213a698b9c913f2f08
debian/3.2+dfsg-4+deb7u3 2254a4d52ff2338cf803e04a68656ae22571a668
debian/5.2.2+dfsg-1 612b68e1fc65c75cff8e23e860cd7e9d4e67db78
debian/5.2.2+dfsg-2 a7c22e20c5ca1d7d117f6130fac65c445b685361
debian/5.2+dfsg-1 9ea1e23624d0f0e8415ee4f248e7328c355c1e44
debian/sid 4e10242c8786ed237654dc3badfcae13742881a4
debian/wheezy 2254a4d52ff2338cf803e04a68656ae22571a668
dfsg/3.2+dfsg 1e54e2ea03c6729ae01f55771d1a62d8583d8d37
dfsg/6.2+dfsg 8792d5025eaf7babc5d7e79aca52d22f16f69acb
dfsg/7.0+dfsg 0b8deb3e9e47d68def2ea19699bb5faf88169e8c
dfsg/sid 0b8deb3e9e47d68def2ea19699bb5faf88169e8c
dfsg/wheezy 1e54e2ea03c6729ae01f55771d1a62d8583d8d37
upstream/5.2.2+dfsg c5f692769be97e5757fe6f8ee6082ef70c2386b0
upstream/5.2+dfsg 45caacfffe16dd80e9052b0797f588014de4008b
upstream/6.2 53592e714f2a8ff81f4bc0a6ac6153f703beec4b
upstream/7.0 d7604402e323a79dd8275c313c5b2377a3b35b7c
upstream/sid d7604402e323a79dd8275c313c5b2377a3b35b7c


X


Thanks. I've just cloned the repo again.

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 18:29, Rahul Amaram wrote:
>> (d) The git history for the debian/wheezy branch of calendarserver is also 
>> messed up - you have version 3.2 committed on top of 5.2.2. It involves a 
>> bit of git magic to fix, which I can do, if you want to avoid the hassle 
>> yourself and you give me access.
>>
>> I've also asked to be added to the calendarserver group on Alioth, so that I 
>> can make these changes myself.
> Feel free to do the changes. Not sure how it got messed up. Thanks.

OK, I've rewritten the history so things make more sense looking at the logs - 
all the "Imported Upstream version" commits are now in the correct linear 
sequence, as well as the "Imported Debian patch" commits, and debian/wheezy 
branches away from the main history at the correct place.

You'll need to update your own repo to point all your local branches to the new 
rewritten history. If it's too messy, you can just clone it again.

For transparency, I've attached the script I used, which is a combination of 
writing .git/info/grafts and git-filter-branch. You can run it on your own 
local old repo, and in theory you should even get the same commit SHA1 objects 
out. (But be careful! This will drop the gpg signatures on some old tags, so be 
careful not to push these back out.)

For reference, here's what you should expect the script to output, twice, to 
check that I didn't backdoor the source :)


debian/3.2+dfsg-4+deb7u1 fe8398cfe0b68277b4c95e116e8d37e6b88f5061
debian/3.2+dfsg-4+deb7u2 fb51628bb376ea59c7650a213a698b9c913f2f08
debian/3.2+dfsg-4+deb7u3 2254a4d52ff2338cf803e04a68656ae22571a668
debian/5.2.2+dfsg-1 612b68e1fc65c75cff8e23e860cd7e9d4e67db78
debian/5.2.2+dfsg-2 a7c22e20c5ca1d7d117f6130fac65c445b685361
debian/5.2+dfsg-1 9ea1e23624d0f0e8415ee4f248e7328c355c1e44
debian/sid 4e10242c8786ed237654dc3badfcae13742881a4
debian/wheezy 2254a4d52ff2338cf803e04a68656ae22571a668
dfsg/3.2+dfsg 1e54e2ea03c6729ae01f55771d1a62d8583d8d37
dfsg/6.2+dfsg 8792d5025eaf7babc5d7e79aca52d22f16f69acb
dfsg/7.0+dfsg 0b8deb3e9e47d68def2ea19699bb5faf88169e8c
dfsg/sid 0b8deb3e9e47d68def2ea19699bb5faf88169e8c
dfsg/wheezy 1e54e2ea03c6729ae01f55771d1a62d8583d8d37
upstream/5.2.2+dfsg c5f692769be97e5757fe6f8ee6082ef70c2386b0
upstream/5.2+dfsg 45caacfffe16dd80e9052b0797f588014de4008b
upstream/6.2 53592e714f2a8ff81f4bc0a6ac6153f703beec4b
upstream/7.0 d7604402e323a79dd8275c313c5b2377a3b35b7c
upstream/sid d7604402e323a79dd8275c313c5b2377a3b35b7c


X

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git


rewrite-calendarserver-history.sh
Description: application/shellscript


signature.asc
Description: OpenPGP digital signature

Bug#780530: [calendarserver]

[Ximin Luo]

Hey, I just tested 7.0+dfsg from git and it seems to work fine. Thanks for the 
work, and looking forward to the eventual upload! Just a few notes:

(a) You should say "python-pycalendar (>= 2.1~svn15020)" in Depends: otherwise 
you get an error at startup.

(b) You can (Closes: #770416) since the new upstream version no longer has 
calendarserver_bootstrap_database. Instead, the instructions in README.Debian 
seem to work fine for me (I tested peer authentication).

(c) You probably want to add a `debian/gbp.conf` to all your repos, like this:

 calendarserver/debian/gbp.conf 
[DEFAULT]
pristine-tar = true
debian-branch = debian/sid
upstream-tag = dfsg/%(version)s

[buildpackage]
dist = DEP14
 [other packages]/debian/gbp.conf 
[DEFAULT]
pristine-tar = true
debian-branch = debian/sid

[buildpackage]
dist = DEP14


Then one can simply use `gbp buildpackage`, with or without `--git-pbuilder 
--git-arch=amd64` as desired.

(d) The git history for the debian/wheezy branch of calendarserver is also 
messed up - you have version 3.2 committed on top of 5.2.2. It involves a bit 
of git magic to fix, which I can do, if you want to avoid the hassle yourself 
and you give me access.

I've also asked to be added to the calendarserver group on Alioth, so that I 
can make these changes myself.

X

On 28/11/15 04:53, Rahul Amaram wrote:
> Hi Ximin,
> 
> I can upload calendarsever myself. There are a few action items left (see 
> above), which needs to be completed before pushing calendarserver 7.0 debian 
> package. It would be great if you can help me with those tasks.
> 
> Thanks,
> Rahul.
> 

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

On Wednesday 20 January 2016 11:31 PM, Ximin Luo wrote:

On 20/01/16 18:48, Rahul Amaram wrote:

I had removed the calenarserver_upgrade instructions from README.Debian because 
it was not working and I was assuming that when I start calendarserver the 
upgrade will happen automatically. I will take care of all the points that you 
have mentioned and also mostly write a script for xml file upgrade while 
installing calendarserver 7.0. But this is odd. I feel, upstream should take 
care of all these upgrades automatically :).


Yes, they ought to have better documentation to help admins during upgrades. :( 
Even doc/Admin/DirectoryService-XML.rst describes the old format... I had to guess 
and also by looking at the output of http://$server:8008/addressbooks/ >:(


Thanks for all the help. I just can't tell you how awesome you have been. I do 
have one final request. I need the packages twextypy and pg8000 to be uploaded 
to Debian. I have reached out to my mentor but have not yet heard. He gets busy 
at times. In case I do not hear from him, would you be willing to upload these 
two packages? I have already raised ITP for them.


Sure, just let me know when you are ready. :) You can either send me a link to 
a git commit, or a link to a .dsc that you upload to mentors.debian.net.
Should I tag the commit so that you can easily build from it? If you 
would be reviewing it, it might be better to do it before I tag it. I 
will anyway, implement all your suggestions in the earlier mails. If you 
have anything apart from that, let me know.


X

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 18:38, Rahul Amaram wrote:
>> (g) I also need to edit 
>> /usr/lib/python2.7/dist-packages/CalendarServer-7.0a1+unknown.egg-info/requires.txt
>>  and remove the line that says "kerberos" (I am not using it anyway). 
>> Probably we need to remove that from setup.py in the source package.
> What does this break?
>>

This breaks calendarserver_upgrade, and I think all of the other 
calendarserver_* utility scripts too. They fail like this:

$ sudo -u caldavd -g caldavd calendarserver_upgrade -D -s
Traceback (most recent call last):
  File "/usr/bin/calendarserver_upgrade", line 5, in 
from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3138, 
in 
@_call_aside
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3124, 
in _call_aside
f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3151, 
in _initialize_master_working_set
working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 661, 
in _build_master
ws.require(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 962, 
in require
needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 849, 
in resolve
raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'kerberos' distribution was not found 
and is required by CalendarServer
1

The error also goes away if you replace "kerberos" with "pykerberos", since 
that is the Name: defined in 
/usr/lib/python2.7/dist-packages/pykerberos-1.1.5.egg-info/PKG-INFO and so 
pkg_resources *can* find it.

However, I am not sure if this is the correct fix. (Perhaps there are two 
python kerberos libraries or something, I haven't checked. We should probably 
consult upstream on this.)

X

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 17:26, Ximin Luo wrote:
> On 20/01/16 15:23, Ximin Luo wrote:
>> Hey, I just tested 7.0+dfsg from git and it seems to work fine. Thanks for 
>> the work, and looking forward to the eventual upload! Just a few notes:
>>
>> (a) [..]
>>
>> (b) [..]
>>
>> (c) [..]
>>
>> (d) [..]
>>
> 
> (e) [..]
> 
> (f) [..]
> 
> (g) [..]
> 
> (h) calendarserver_upgrade is not working for me at the moment. I've migrated 
> accounts.xml to the new format ( ->  and  -> 
> ) and I get this:
> 
> $ sudo calendarserver_upgrade -D -s   
>   
>   
> Converting augments.xml
> Traceback (most recent call last):
> [..]
> 

Oh, this was my fault. It works if you run it as `sudo -u caldavd -g caldavd 
calendarserver_upgrade` (the `-D` is optional). So we should put that in 
README.Debian.

But then you get a lot of twistd errors in /var/log/caldavd/error.log:

2016-01-20 17:31:24+0100 [-] [directoryproxy] Usage: twistd [options]
2016-01-20 17:31:24+0100 [-] [directoryproxy] Options:
[..]
2016-01-20 17:31:24+0100 [-] [directoryproxy] /usr/bin/twistd: Unknown command: 
caldav_directoryproxy

This is due to a bug in calendarserver, which we should forward at some point. 
The easiest way to work around it is to patch:


--- /usr/lib/python2.7/dist-packages/twisted/plugins/caldav.py 2016-01-20 
17:53:23.741080020 +0100
+++ /usr/lib/python2.7/dist-packages/twisted/plugins/caldav.py  2016-01-20 
17:49:19.201450679 +0100
@@ -50,5 +50,9 @@
 return self._serviceMaker.makeService(options)
 
 
-TwistedCalDAV = TAP("calendarserver.tap.caldav.CalDAVServiceMaker")
-DirectoryProxy = TAP("txdav.dps.server.DirectoryProxyServiceMaker")
+class T1(TAP):
+pass
+class T2(TAP):
+pass
+TwistedCalDAV = T1("calendarserver.tap.caldav.CalDAVServiceMaker")
+DirectoryProxy = T2("txdav.dps.server.DirectoryProxyServiceMaker")


after doing all of this, I successfully upgraded my instance of 5.2.2+dfsg-2 to 
7.0+dfsg-1.

(j) You have to upgrade accounts.xml very specifically. Like this:

 old 

  
admin


Anna Mouse
  

 new 

  
 


admin 
Anna Mouse 
  



You need to do this *before* running calendarserver_upgrade, otherwise that 
won't work.

If you edit accounts.xml a different way, calendarserver will appear to work 
but then clients will get authentication/resource-not-found errors. I tested 
the above with tested with DavDroid 0.9.1.3 and IceOwl 4.0.5, synchronisation 
(add/del) events works both ways as before.

And I think that's everything, finally. :)

X

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

On Wednesday 20 January 2016 10:43 PM, Ximin Luo wrote:

On 20/01/16 17:26, Ximin Luo wrote:

On 20/01/16 15:23, Ximin Luo wrote:

Hey, I just tested 7.0+dfsg from git and it seems to work fine. Thanks for the 
work, and looking forward to the eventual upload! Just a few notes:

(a) [..]

(b) [..]

(c) [..]

(d) [..]


(e) [..]

(f) [..]

(g) [..]

(h) calendarserver_upgrade is not working for me at the moment. I've migrated accounts.xml to the new format 
( ->  and  -> ) and I get this:

$ sudo calendarserver_upgrade -D -s
Converting augments.xml
Traceback (most recent call last):
[..]


Oh, this was my fault. It works if you run it as `sudo -u caldavd -g caldavd 
calendarserver_upgrade` (the `-D` is optional). So we should put that in 
README.Debian.

But then you get a lot of twistd errors in /var/log/caldavd/error.log:

2016-01-20 17:31:24+0100 [-] [directoryproxy] Usage: twistd [options]
2016-01-20 17:31:24+0100 [-] [directoryproxy] Options:
[..]
2016-01-20 17:31:24+0100 [-] [directoryproxy] /usr/bin/twistd: Unknown command: 
caldav_directoryproxy

This is due to a bug in calendarserver, which we should forward at some point. 
The easiest way to work around it is to patch:


--- /usr/lib/python2.7/dist-packages/twisted/plugins/caldav.py 2016-01-20 
17:53:23.741080020 +0100
+++ /usr/lib/python2.7/dist-packages/twisted/plugins/caldav.py  2016-01-20 
17:49:19.201450679 +0100
@@ -50,5 +50,9 @@
  return self._serviceMaker.makeService(options)
  
  
-TwistedCalDAV = TAP("calendarserver.tap.caldav.CalDAVServiceMaker")

-DirectoryProxy = TAP("txdav.dps.server.DirectoryProxyServiceMaker")
+class T1(TAP):
+pass
+class T2(TAP):
+pass
+TwistedCalDAV = T1("calendarserver.tap.caldav.CalDAVServiceMaker")
+DirectoryProxy = T2("txdav.dps.server.DirectoryProxyServiceMaker")


after doing all of this, I successfully upgraded my instance of 5.2.2+dfsg-2 to 
7.0+dfsg-1.

(j) You have to upgrade accounts.xml very specifically. Like this:

 old 

   
 admin
 
 
 Anna Mouse
   

 new 

   
  
 
 
 admin 
 Anna Mouse 
   



You need to do this *before* running calendarserver_upgrade, otherwise that 
won't work.

If you edit accounts.xml a different way, calendarserver will appear to work 
but then clients will get authentication/resource-not-found errors. I tested 
the above with tested with DavDroid 0.9.1.3 and IceOwl 4.0.5, synchronisation 
(add/del) events works both ways as before.

And I think that's everything, finally. :)

X

I had removed the calenarserver_upgrade instructions from README.Debian 
because it was not working and I was assuming that when I start 
calendarserver the upgrade will happen automatically. I will take care 
of all the points that you have mentioned and also mostly write a script 
for xml file upgrade while installing calendarserver 7.0. But this is 
odd. I feel, upstream should take care of all these upgrades 
automatically :).


Thanks for all the help. I just can't tell you how awesome you have 
been. I do have one final request. I need the packages twextypy and 
pg8000 to be uploaded to Debian. I have reached out to my mentor but 
have not yet heard. He gets busy at times. In case I do not hear from 
him, would you be willing to upload these two packages? I have already 
raised ITP for them.



Thanks,
Rahul.

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 19:07, Rahul Amaram wrote:
> 
> 
> On Wednesday 20 January 2016 11:31 PM, Ximin Luo wrote:
>> On 20/01/16 18:48, Rahul Amaram wrote:
>>> I had removed the calenarserver_upgrade instructions from README.Debian 
>>> because it was not working and I was assuming that when I start 
>>> calendarserver the upgrade will happen automatically. I will take care of 
>>> all the points that you have mentioned and also mostly write a script for 
>>> xml file upgrade while installing calendarserver 7.0. But this is odd. I 
>>> feel, upstream should take care of all these upgrades automatically :).
>>>
>> Yes, they ought to have better documentation to help admins during upgrades. 
>> :( Even doc/Admin/DirectoryService-XML.rst describes the old format... I had 
>> to guess and also by looking at the output of 
>> http://$server:8008/addressbooks/ >:(
>>
>>> Thanks for all the help. I just can't tell you how awesome you have been. I 
>>> do have one final request. I need the packages twextypy and pg8000 to be 
>>> uploaded to Debian. I have reached out to my mentor but have not yet heard. 
>>> He gets busy at times. In case I do not hear from him, would you be willing 
>>> to upload these two packages? I have already raised ITP for them.
>>>
>> Sure, just let me know when you are ready. :) You can either send me a link 
>> to a git commit, or a link to a .dsc that you upload to mentors.debian.net.
> Should I tag the commit so that you can easily build from it? If you would be 
> reviewing it, it might be better to do it before I tag it. I will anyway, 
> implement all your suggestions in the earlier mails. If you have anything 
> apart from that, let me know.

No, just a commit reference will do, I can find it. I usually only tag things 
after I get the "ACCEPTED" email from ftp-master, and then I run "gbp 
buildpackage --git-tag-only" and push it.

X

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 15:23, Ximin Luo wrote:
> Hey, I just tested 7.0+dfsg from git and it seems to work fine. Thanks for 
> the work, and looking forward to the eventual upload! Just a few notes:
> 
> (a) [..]
> 
> (b) [..]
> 
> (c) [..]
> 
> (d) [..]
> 

Some extra notes, after trying to migrate 5.2.2+dfsg-2 to 7.0+dfsg-1 (commit 
01c5df30):

(e) python-twext should declare

Replaces: calendarserver (<< 7.0+dfsg-1)
Breaks: calendarserver (<< 7.0+dfsg-1)

(see Policy 7.6.1)

(f) We should tell people explicitly in README.Debian to run 
calendarserver_upgrade. At least, I assume this is what I have to do, because 
error.log says this:

2016-01-20 17:11:25+0100 [-] Log opened.
2016-01-20 17:11:25+0100 [-] 
[txdav.common.datastore.upgrade.sql.upgrade.UpgradeDatabaseSchemaStep#warn] 
Beginning database schema check.
2016-01-20 17:11:25+0100 [-] 
[txdav.common.datastore.upgrade.sql.upgrade.UpgradeDatabaseSchemaStep#warn] 
Required database key VERSION: 58.
2016-01-20 17:11:25+0100 [-] 
[txdav.common.datastore.upgrade.sql.upgrade.UpgradeDatabaseSchemaStep#warn] 
Actual database key VERSION: 28.
2016-01-20 17:11:25+0100 [-] [calendarserver.tap.caldav#error] Data store not 
available; shutting down
2016-01-20 17:11:25+0100 [-] [memcached-Default] Signal handled: Terminated.

(g) I also need to edit 
/usr/lib/python2.7/dist-packages/CalendarServer-7.0a1+unknown.egg-info/requires.txt
 and remove the line that says "kerberos" (I am not using it anyway). Probably 
we need to remove that from setup.py in the source package.

(h) calendarserver_upgrade is not working for me at the moment. I've migrated 
accounts.xml to the new format ( ->  and  -> 
) and I get this:

$ sudo calendarserver_upgrade -D -s 

  
Converting augments.xml
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, 
in callWithContext
return func(*args,**kw)
  File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
48, in _run
while self._qpull():
  File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
69, in _qpull
return True
  File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
82, in _oneWorkUnit
self._reactor.callFromThread(deferred.callback, result)
---  ---
  File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
74, in _oneWorkUnit
result = instruction()
  File "/usr/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line 
1158, in initCursor
txn._connection = self.connectionFactory()
  File "/usr/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py", 
line 243, in connect
connection = self.dbModule.connect(*self.connectArgs, **self.connectKw)
  File "/usr/lib/python2.7/dist-packages/pg8000/__init__.py", line 103, in 
connect
user, host, unix_sock, port, database, password, ssl, timeout)
  File "/usr/lib/python2.7/dist-packages/pg8000/core.py", line 1618, in __init__
raise self.error
pg8000.core.InterfaceError: md5 password authentication failed
^CTraceback (most recent call last):
Failure: twext.enterprise.ienterprise.ConnectionError: connection pool shut 
down while txn waiting for database connection.
[previous 2 lines repeated another 8 times]

I also tried setting Enabled for md5 authentication in 
caldavd.plist and the result was the same.

X

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

On Wednesday 20 January 2016 07:53 PM, Ximin Luo wrote:

Hey, I just tested 7.0+dfsg from git and it seems to work fine. Thanks for the 
work, and looking forward to the eventual upload! Just a few notes:

(a) You should say "python-pycalendar (>= 2.1~svn15020)" in Depends: otherwise 
you get an error at startup.

Done.


(b) You can (Closes: #770416) since the new upstream version no longer has 
calendarserver_bootstrap_database. Instead, the instructions in README.Debian 
seem to work fine for me (I tested peer authentication).

Yes. Will take care of this once calenarserver 7.0 is uploaded.


(c) You probably want to add a `debian/gbp.conf` to all your repos, like this:

 calendarserver/debian/gbp.conf 
[DEFAULT]
pristine-tar = true
debian-branch = debian/sid
upstream-tag = dfsg/%(version)s

[buildpackage]
dist = DEP14
 [other packages]/debian/gbp.conf 
[DEFAULT]
pristine-tar = true
debian-branch = debian/sid

[buildpackage]
dist = DEP14


Then one can simply use `gbp buildpackage`, with or without `--git-pbuilder 
--git-arch=amd64` as desired.

Done.


(d) The git history for the debian/wheezy branch of calendarserver is also 
messed up - you have version 3.2 committed on top of 5.2.2. It involves a bit 
of git magic to fix, which I can do, if you want to avoid the hassle yourself 
and you give me access.

I've also asked to be added to the calendarserver group on Alioth, so that I 
can make these changes myself.

Feel free to do the changes. Not sure how it got messed up. Thanks.


X

On 28/11/15 04:53, Rahul Amaram wrote:

Hi Ximin,

I can upload calendarsever myself. There are a few action items left (see 
above), which needs to be completed before pushing calendarserver 7.0 debian 
package. It would be great if you can help me with those tasks.

Thanks,
Rahul.

Bug#780530: [calendarserver]

[Rahul Amaram]

On Wednesday 20 January 2016 09:56 PM, Ximin Luo wrote:

On 20/01/16 15:23, Ximin Luo wrote:

Hey, I just tested 7.0+dfsg from git and it seems to work fine. Thanks for the 
work, and looking forward to the eventual upload! Just a few notes:

(a) [..]

(b) [..]

(c) [..]

(d) [..]


Some extra notes, after trying to migrate 5.2.2+dfsg-2 to 7.0+dfsg-1 (commit 
01c5df30):

(e) python-twext should declare

Replaces: calendarserver (<< 7.0+dfsg-1)
Breaks: calendarserver (<< 7.0+dfsg-1)

Done.


(see Policy 7.6.1)

(f) We should tell people explicitly in README.Debian to run 
calendarserver_upgrade. At least, I assume this is what I have to do, because 
error.log says this:

2016-01-20 17:11:25+0100 [-] Log opened.
2016-01-20 17:11:25+0100 [-] 
[txdav.common.datastore.upgrade.sql.upgrade.UpgradeDatabaseSchemaStep#warn] 
Beginning database schema check.
2016-01-20 17:11:25+0100 [-] 
[txdav.common.datastore.upgrade.sql.upgrade.UpgradeDatabaseSchemaStep#warn] 
Required database key VERSION: 58.
2016-01-20 17:11:25+0100 [-] 
[txdav.common.datastore.upgrade.sql.upgrade.UpgradeDatabaseSchemaStep#warn] 
Actual database key VERSION: 28.
2016-01-20 17:11:25+0100 [-] [calendarserver.tap.caldav#error] Data store not 
available; shutting down
2016-01-20 17:11:25+0100 [-] [memcached-Default] Signal handled: Terminated.

Done.


(g) I also need to edit 
/usr/lib/python2.7/dist-packages/CalendarServer-7.0a1+unknown.egg-info/requires.txt and 
remove the line that says "kerberos" (I am not using it anyway). Probably we 
need to remove that from setup.py in the source package.

What does this break?


(h) calendarserver_upgrade is not working for me at the moment. I've migrated accounts.xml to the new format 
( ->  and  -> ) and I get this:

$ sudo calendarserver_upgrade -D -s
Converting augments.xml
Traceback (most recent call last):
   File "/usr/lib/python2.7/dist-packages/twisted/python/context.py", line 81, 
in callWithContext
 return func(*args,**kw)
   File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
48, in _run
 while self._qpull():
   File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
69, in _qpull
 return True
   File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
82, in _oneWorkUnit
 self._reactor.callFromThread(deferred.callback, result)
---  ---
   File "/usr/lib/python2.7/dist-packages/twext/internet/threadutils.py", line 
74, in _oneWorkUnit
 result = instruction()
   File "/usr/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line 
1158, in initCursor
 txn._connection = self.connectionFactory()
   File "/usr/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py", 
line 243, in connect
 connection = self.dbModule.connect(*self.connectArgs, **self.connectKw)
   File "/usr/lib/python2.7/dist-packages/pg8000/__init__.py", line 103, in 
connect
 user, host, unix_sock, port, database, password, ssl, timeout)
   File "/usr/lib/python2.7/dist-packages/pg8000/core.py", line 1618, in 
__init__
 raise self.error
pg8000.core.InterfaceError: md5 password authentication failed
^CTraceback (most recent call last):
Failure: twext.enterprise.ienterprise.ConnectionError: connection pool shut 
down while txn waiting for database connection.
[previous 2 lines repeated another 8 times]

I also tried setting Enabled for md5 authentication in 
caldavd.plist and the result was the same.


Upgrade failure is something that I am aware of and have started a 
conversation on calendarserver mailing list. However, I do believe that 
you have found a fix for this.

X

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 18:48, Rahul Amaram wrote:
> I had removed the calenarserver_upgrade instructions from README.Debian 
> because it was not working and I was assuming that when I start 
> calendarserver the upgrade will happen automatically. I will take care of all 
> the points that you have mentioned and also mostly write a script for xml 
> file upgrade while installing calendarserver 7.0. But this is odd. I feel, 
> upstream should take care of all these upgrades automatically :).
> 

Yes, they ought to have better documentation to help admins during upgrades. :( 
Even doc/Admin/DirectoryService-XML.rst describes the old format... I had to 
guess and also by looking at the output of http://$server:8008/addressbooks/ >:(

> Thanks for all the help. I just can't tell you how awesome you have been. I 
> do have one final request. I need the packages twextypy and pg8000 to be 
> uploaded to Debian. I have reached out to my mentor but have not yet heard. 
> He gets busy at times. In case I do not hear from him, would you be willing 
> to upload these two packages? I have already raised ITP for them.
> 

Sure, just let me know when you are ready. :) You can either send me a link to 
a git commit, or a link to a .dsc that you upload to mentors.debian.net.

X

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Ximin Luo]

On 20/01/16 15:23, Ximin Luo wrote:
> pristine-tar = true

Whoops my bad, that should be True with a capital T.

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

Hi Ximin,

I can upload calendarsever myself. There are a few action items left 
(see above), which needs to be completed before pushing calendarserver 
7.0 debian package. It would be great if you can help me with those tasks.


Thanks,
Rahul.

On Friday 27 November 2015 03:26 PM, Ximin Luo wrote:

Hi Rahul,

Any progress on this? I'm now a DD and can help sponsor your package, if you 
need it.

X

On Sun, 25 Oct 2015 23:41:00 +0530 Rahul Amaram 
 wrote:

Package: calendarserver

--- Please enter the report below this line. ---
1. has been fixed.
2. can be ignored.

Changes have been pushed to calendarserver git repository.

Will look into the remaining action items over this week.

--- System information. ---
Architecture: amd64
Kernel: Linux 4.2.0-1-amd64

Debian Release: stretch/sid
500 trusty ppa.launchpad.net
500 stable dl.google.com
500 stable deb.opera.com
500 oldstable packages.x2go.org
500 all liveusb.info
1000 testing security.debian.org
1000 testing httpredir.debian.org

--- Package information. ---
Package's Depends field is empty.

Package's Recommends field is empty.

Package's Suggests field is empty.

Bug#780530: [calendarserver]

[Ximin Luo]

Hi Rahul,

Any progress on this? I'm now a DD and can help sponsor your package, if you 
need it.

X

On Sun, 25 Oct 2015 23:41:00 +0530 Rahul Amaram 
 wrote:
> Package: calendarserver
> 
> --- Please enter the report below this line. ---
> 1. has been fixed.
> 2. can be ignored.
> 
> Changes have been pushed to calendarserver git repository.
> 
> Will look into the remaining action items over this week.
> 
> --- System information. ---
> Architecture: amd64
> Kernel: Linux 4.2.0-1-amd64
> 
> Debian Release: stretch/sid
> 500 trusty ppa.launchpad.net
> 500 stable dl.google.com
> 500 stable deb.opera.com
> 500 oldstable packages.x2go.org
> 500 all liveusb.info
> 1000 testing security.debian.org
> 1000 testing httpredir.debian.org
> 
> --- Package information. ---
> Package's Depends field is empty.
> 
> Package's Recommends field is empty.
> 
> Package's Suggests field is empty.
> 

-- 
GPG: 4096R/1318EFAC5FBBDBCE
git://github.com/infinity0/pubkeys.git

Bug#780530: [calendarserver]

[Rahul Amaram]

Package: calendarserver

--- Please enter the report below this line. ---
1. has been fixed.
2. can be ignored.

Changes have been pushed to calendarserver git repository.

Will look into the remaining action items over this week.

--- System information. ---
Architecture: amd64
Kernel: Linux 4.2.0-1-amd64

Debian Release: stretch/sid
500 trusty ppa.launchpad.net
500 stable dl.google.com
500 stable deb.opera.com
500 oldstable packages.x2go.org
500 all liveusb.info
1000 testing security.debian.org
1000 testing httpredir.debian.org

--- Package information. ---
Package's Depends field is empty.

Package's Recommends field is empty.

Package's Suggests field is empty.

Bug#780530: calendarserver: new version

[Shawn Landden]

Package: calendarserver
Version: 6.0+dfsg-1
Severity: normal

Dear Maintainer,

6.0 has been released. I have worked on packaging it a bit, but it is not 
complete:

git pull git://churchofgit.com/shawn/calendarserver


Thanks,

Shawn
---

Debian Release: 8.0
  APT prefers unstable
  APT policy: (500, 'unstable'), (1, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 3.2.0-4-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages calendarserver depends on:
ii  adduser3.113+nmu3
ii  lsb-base   4.1+Debian13+nmu1
ii  memcached  1.4.21-1.1
ii  python 2.7.8-4
ii  python-cffi0.8.6-1
ii  python-crypto  2.6.1-5+b2
ii  python-dateutil2.2-2
ii  python-kerberos1.1.5-0.1
ii  python-ldap2.4.10-1
ii  python-nevow   0.11.1-1
ii  python-openssl 0.14-1
ii  python-psutil  2.1.1-1+b1
ii  python-pyasn1  0.1.7-1
ii  python-pycalendar  2.0~svn13177-2
ii  python-pycparser   2.10+dfsg-3
ii  python-pygresql1:4.0-3.1
ii  python-setproctitle1.1.8-1
ii  python-sqlparse0.1.13-2
ii  python-twisted 14.0.2-3
ii  python-twisted-conch   1:14.0.2-3
ii  python-twisted-core14.0.2-3
ii  python-twisted-mail14.0.2-3
ii  python-twisted-web 14.0.2-3
ii  python-twisted-words   14.0.2-3
ii  python-tz  2012c+dfsg-0.1
ii  python-xattr   0.6.4-3
ii  python-zope.interface  4.1.1-3.1
pn  python:any none
ii  ssl-cert   1.0.35

Versions of packages calendarserver recommends:
ii  python-pam  0.4.2-13.1

Versions of packages calendarserver suggests:
pn  pyflakes none
pn  python-epydocnone
ii  python-pyasn10.1.7-1
pn  python-pydoctor  none

-- Configuration Files:
/etc/caldavd/servertoserver.dtd 6959f93d1fce2acccff5971775cb3e20 [Errno 2] No 
such file or directory: u'/etc/caldavd/servertoserver.dtd 
6959f93d1fce2acccff5971775cb3e20'
/etc/caldavd/sudoers.plist c4e456244e69c8e3f0449219e4cc589b [Errno 2] No such 
file or directory: u'/etc/caldavd/sudoers.plist 
c4e456244e69c8e3f0449219e4cc589b'
/etc/default/calendarserver changed:
start_calendarserver=yes


-- no debconf information


-- 
To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org
with a subject of unsubscribe. Trouble? Contact listmas...@lists.debian.org