> On Feb 5, 2015, at 8:51 AM, Jacques Distler <dist...@golem.ph.utexas.edu>
> wrote:
>
>> There is one known issue in 5.3 regarding how we handle updating calendar
>> event data when the format changes (as it has recently). In 5.3, the
>> approach is to leave the data in the old format until it is requested, and
>> then update it on the fly as it is vended to the client. This fails when
>> there are enough events being 'touched' by the request that the processing
>> time for doing the format upgrades exceeds the client's request timeout
>> threshold. This is fixed in 5.4-dev.
>
> Is there (say) a commandline utility to upgrade the data to the new format?
We don’t have a single command line tool to do this, no...
> Or some other way to address these (rather persistent) timeouts?
You can find out how many resources are in the old format by running this SQL:
select count(*) from calendar_object where dataversion = 0;
If the timeouts are being caused by the data format upgrade taking too long
(because it’s operating in batches that are too large), one way to work around
it is to manually request all the calendar resources one by one, causing each
one to be upgraded individually. I had to work around this problem on a live
server without the benefit of any code fixes, and here’s what I ended up doing.
There are two scripts: make-curl-command-file.sh and old_resources.sql; the
contents of each are shown below, along with output from a run of
make-curl-command-file.sh. To use this, you need to specify a principal in
ReadPrincipals in caldavd.plist - preferably an admin user - to authorize that
user to access all the resources. ReadPrincipals is an array, with string
values specifying principal URLs, like the ones you see on a user’s principal
page (at /principals/users/foo). Use the credentials for this user with the
scripts below by creating a netrc file for curl. The script gives an example if
netrc is missing. You may also need to customize SOCKETPATH, or perhaps switch
to tcp/ip - depending on how your postgres service is deployed.
(and note that although this script worked for me, it comes with no warranty)
# begin make-curl-command-file.sh
#!/bin/bash
# * execute some SQL that emits URLs of all calendar events with an old data
# format.
# * put the results into a new file, trimming the leading and trailing lines
# * emit the curl command required to process this file
# * run that command
SOCKETPATH="/var/run/caldavd/PostgresSocket"
PGUSER="caldav"
SQLFILE="old_resources.sql"
CURLFILE="curl-commands.txt"
NETRC="netrc.txt"
rm ${CURLFILE}
touch ${CURLFILE}
echo "Getting total number of un-upgraded items: "
sudo psql -h ${SOCKETPATH} -U ${PGUSER} -c "select count(*) from
calendar_object where dataversion = 0;"
echo ""
echo "Gathering list of events that need to be upgraded in 1000 item chunks..."
sudo psql -q -h ${SOCKETPATH} -U ${PGUSER} -f ${SQLFILE} | grep "url =" >
${CURLFILE}
echo ""
ITEMS=$(wc -l ${CURLFILE} | awk '{print $1}')
if [ ${ITEMS} -eq 0 ]
then echo "No more work to do!"
exit 0
fi
echo -n "Generated curl command file with this many items: ${ITEMS}"
echo ""
if [ ! -f ${NETRC} ]
then
echo "Put the credentials of a ReadAdmin user into ${NETRC}, e.g. :"
echo "machine example.com <http://solar.burningman.com/> login Administrator
password 12345"
echo ""
fi
echo ""
echo "now running:"
echo "curl --netrc-file ${NETRC} -K ${CURLFILE} > /dev/null"
echo ""
curl --netrc-file ${NETRC} -K ${CURLFILE} > /dev/null
echo "Re-run this script until it has no more work to do.”
# end make-curl-command-file.sh
# begin old_resources.sql
SELECT 'url = "https://example.com:8443/calendars/__uids__/'
<https://example.com:8443/calendars/__uids__/'>
|| owner_uid
|| '/'
|| calendar_resource_name
|| '/'
|| resource_name
|| '"' as "curl command file contents"
FROM calendar_bind
JOIN calendar_home
ON calendar_bind.calendar_home_resource_id = calendar_home.resource_id
JOIN calendar_object
ON calendar_bind.calendar_resource_id =
calendar_object.calendar_resource_id
WHERE calendar_bind.bind_mode = 0
AND calendar_object.dataversion = 0
LIMIT 1000;
# end old_resources.sql
server:bin administrator$ ./make-curl-command-file.sh
Total number of un-upgraded items: count
-------
48842
(1 row)
Gathering list of events that need to be upgraded in 1000 item chunks...
Generated curl command file with this many items: 1000
now running:
curl --netrc-file netrc.txt -K curl-commands.txt > /dev/null
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 1514 100 1514 0 0 7656 0 --:--:-- --:--:-- --:--:-- 7685
100 803 100 803 0 0 5378 0 --:--:-- --:--:-- --:--:-- 784k
100 1140 100 1140 0 0 7580 0 --:--:-- --:--:-- --:--:-- 7580
100 707 100 707 0 0 4813 0 --:--:-- --:--:-- --:--:-- 4813
100 733 100 733 0 0 4982 0 --:--:-- --:--:-- --:--:-- 4982
100 1035 100 1035 0 0 6948 0 --:--:-- --:--:-- --:--:-- 6948
100 691 100 691 0 0 4690 0 --:--:-- --:--:-- --:--:-- 4690
100 1317 100 1317 0 0 8444 0 --:--:-- --:--:-- --:--:-- 1286k
100 748 100 748 0 0 5102 0 --:--:-- --:--:-- --:--:-- 5102
100 368 100 368 0 0 2507 0 --:--:-- --:--:-- --:--:-- 2507
100 725 100 725 0 0 4943 0 --:--:-- --:--:-- --:--:-- 4943
100 717 100 717 0 0 4773 0 --:--:-- --:--:-- --:--:-- 4773
...
> FWIW, I ran calendarserver_verify_data and everything checked out OK.
Good deal.
-dre_______________________________________________
calendarserver-users mailing list
calendarserver-users@lists.macosforge.org
https://lists.macosforge.org/mailman/listinfo/calendarserver-users
