Re: backupd IOERROR reading backup files larger than 2GB
Uf ... Thank you so much Ellie, now it works fine. Right now I was fighting against my rusty gdb knowledge, although I had almost defeated him ;) Cheers, Carlos El 17/06/2019 a las 2:44, ellie timoney escribió: Hi Carlos, Sudden overnight intuition, and I think this will fix the issue -- the problem isn't that the lseek is failing at >2GB; the problem is that its off_t return value is being truncated to an int before being checked to see if it's negative for the error case (so any "I succeeded, the offset is > 2GB" response looks like "negative, it's an error"). Doh! I think this will fix the immediate issue: https://github.com/cyrusimap/cyrus-imapd/commit/63f8c09fa2076c0e2fa55436e071e3e341fe48b6 But then I found and fixed some similar ones: https://github.com/cyrusimap/cyrus-imapd/commit/0930d3af4ed9bd44d328db8cfa4d9f2e2be7bada Remains to be seen whether more similar issues pop up -- since no-one's tripped over this previously, I guess you're the first person to try this with a >2GB backup file :( Cheers, ellie On Fri, Jun 14, 2019, at 3:50 PM, ellie timoney wrote: Hi Carlos, This is quite weird, I'm not sure why a 64bit platform would have any trouble around the 2GB mark?? What does the Cyrus ./configure report for your system's integer sizes? e.g. mine shows: checking size of int... 4 checking size of long... 8 checking size of size_t... 8 checking size of off_t... 8 checking size of time_t... 8 checking size of long long int... 8 checking size of unsigned long long int... 8 checking whether byte ordering is bigendian... no What's your level of comfort with C debugging? It'd be very helpful to see a core file+binary from the time that lseek error occurs? Cheers, ellie On Fri, Jun 7, 2019, at 1:58 AM, Carlos Larrañaga wrote: Hi Ellie, Thanks for answering. We use latest 64bit Oracle Linux (not CentOs like I said before, sorry) and zlib is also 64bit version: # uname -a Linux xxx 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 17:35:45 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux # yum list installed zlib Loaded plugins: langpacks, ulninfo Installed Packages zlib.x86_64 1.2.7-18.el7 @ol7_latest # lsof -p 17005 |grep lib |grep -i z backupd 17005 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7 # file /usr/lib64/libz.so.1.2.7 /usr/lib64/libz.so.1.2.7: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b9d5f73428bd6ad68c96986b57bea3b7cedb9745, stripped # rpm -qf /usr/lib64/libz.so.1.2.7 zlib-1.2.7-18.el7.x86_64 I have summarized here some information about the backupd error reading the file descriptor 15, which is a backup larger than 2GB. The error is logged 1755 times. Instead, there is no error for fd 12, which is a backup of less than 2 GB: *#-- CLIENT SIDE* *---* # time sync_client -v -o -n backup -A Thu Jun 6 17:08:02 CEST 2019 USER aaa QUOTA user.aaa USER ccc Error from do_user(ccc): bailing out! real 30m16.223s user 0m0.006s sys 0m0.006s *#-- BACKUP SERVER SIDE * # LOGFILE # tail -f /var/log/imapd.log Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory *# lsof of the backupd PROCESS *# lsof -P -p 2584 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME backupd 2584 cyrus cwd DIR 253,0 4096 128 / backupd 2584 cyrus rtd DIR 253,0 4096 128 / backupd 2584 cyrus txt REG 253,0 768632 939542214 /usr/local/cyrus/libexec/backupd backupd 2584 cyrus mem REG 253,0 37208 135746334 /usr/lib64/libnss_sss.so.2 backupd 2584 cyrus mem REG 253,0 31416 134406707 /usr/lib64/libnss_dns-2.17.so backupd 2584 cyrus mem REG 253,0 61632 134406709 /usr/lib64/libnss_files-2.17.so backupd 2584 cyrus mem REG 253,0 43336 68814977 /usr/lib64/sasl2/webmail.so.0.0.0 backupd 2584 cyrus mem REG 253,0 57960 68856627 /usr/lib64/sasl2/libdigestmd5.so.3.0.0 backupd 2584 cyrus mem REG 253,0 24232 68797220 /usr/lib64/sasl2/libcrammd5.so.3.0.0 backupd 2584 cyrus mem REG 253,0 20088 68797044 /usr/lib64/sasl2/libplain.so.3.0.0 backupd 2584 cyrus mem REG 253,0 20056
Re: backupd IOERROR reading backup files larger than 2GB
Hi Carlos, Sudden overnight intuition, and I think this will fix the issue -- the problem isn't that the lseek is failing at >2GB; the problem is that its off_t return value is being truncated to an int before being checked to see if it's negative for the error case (so any "I succeeded, the offset is > 2GB" response looks like "negative, it's an error"). Doh! I think this will fix the immediate issue: https://github.com/cyrusimap/cyrus-imapd/commit/63f8c09fa2076c0e2fa55436e071e3e341fe48b6 But then I found and fixed some similar ones: https://github.com/cyrusimap/cyrus-imapd/commit/0930d3af4ed9bd44d328db8cfa4d9f2e2be7bada Remains to be seen whether more similar issues pop up -- since no-one's tripped over this previously, I guess you're the first person to try this with a >2GB backup file :( Cheers, ellie On Fri, Jun 14, 2019, at 3:50 PM, ellie timoney wrote: > Hi Carlos, > > This is quite weird, I'm not sure why a 64bit platform would have any trouble > around the 2GB mark?? > > What does the Cyrus ./configure report for your system's integer sizes? e.g. > mine shows: > >> checking size of int... 4 >> checking size of long... 8 >> checking size of size_t... 8 >> checking size of off_t... 8 >> checking size of time_t... 8 >> checking size of long long int... 8 >> checking size of unsigned long long int... 8 >> checking whether byte ordering is bigendian... no > > What's your level of comfort with C debugging? It'd be very helpful to see a > core file+binary from the time that lseek error occurs? > > Cheers, > > ellie > > On Fri, Jun 7, 2019, at 1:58 AM, Carlos Larrañaga wrote: >> Hi Ellie, >> >> Thanks for answering. We use latest 64bit Oracle Linux (not CentOs like I >> said before, sorry) and zlib is also 64bit version: >>> # uname -a >>> Linux xxx 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 17:35:45 PDT 2019 >>> x86_64 x86_64 x86_64 GNU/Linux >>> >>> # yum list installed zlib >>> Loaded plugins: langpacks, ulninfo >>> Installed Packages >>> zlib.x86_64 1.2.7-18.el7 @ol7_latest >>> >>> # lsof -p 17005 |grep lib |grep -i z >>> backupd 17005 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7 >>> >>> # file /usr/lib64/libz.so.1.2.7 >>> /usr/lib64/libz.so.1.2.7: ELF 64-bit LSB shared object, x86-64, version 1 >>> (SYSV), dynamically linked, >>> BuildID[sha1]=b9d5f73428bd6ad68c96986b57bea3b7cedb9745, stripped >>> >>> # rpm -qf /usr/lib64/libz.so.1.2.7 >>> zlib-1.2.7-18.el7.x86_64 >> I have summarized here some information about the backupd error reading the >> file descriptor 15, which is a backup larger than 2GB. The error is logged >> 1755 times. Instead, there is no error for fd 12, which is a backup of less >> than 2 GB: >>> *#-- CLIENT SIDE* >>> *---* >>> >>> >>> # time sync_client -v -o -n backup -A >>> Thu Jun 6 17:08:02 CEST 2019 >>> USER aaa >>> QUOTA user.aaa >>> USER ccc >>> Error from do_user(ccc): bailing out! >>> >>> real 30m16.223s >>> user 0m0.006s >>> sys 0m0.006s >>> ** >>> >>> *#-- BACKUP SERVER SIDE * >>> # LOGFILE >>> >>> # tail -f /var/log/imapd.log >>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >>> such file or directory >>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >>> such file or directory >>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >>> such file or directory >>> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >>> such file or directory >>> >>> >>> *# lsof of the backupd PROCESS *# lsof -P -p 2584 >>> >>> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >>> backupd 2584 cyrus cwd DIR 253,0 4096 128 / >>> backupd 2584 cyrus rtd DIR 253,0 4096 128 / >>> backupd 2584 cyrus txt REG 253,0 768632 939542214 >>> /usr/local/cyrus/libexec/backupd >>> backupd 2584 cyrus mem REG 253,0 37208 135746334 /usr/lib64/libnss_sss.so.2 >>> backupd 2584 cyrus mem REG 253,0 31416 134406707 >>> /usr/lib64/libnss_dns-2.17.so >>> backupd 2584 cyrus mem REG 253,0 61632 134406709 >>> /usr/lib64/libnss_files-2.17.so >>> backupd 2584 cyrus mem REG 253,0 43336 68814977 >>> /usr/lib64/sasl2/webmail.so.0.0.0 >>> backupd 2584 cyrus mem REG 253,0 57960 68856627 >>> /usr/lib64/sasl2/libdigestmd5.so.3.0.0 >>> backupd 2584 cyrus mem REG 253,0 24232 68797220 >>> /usr/lib64/sasl2/libcrammd5.so.3.0.0 >>> backupd 2584 cyrus mem REG 253,0 20088 68797044 >>> /usr/lib64/sasl2/libplain.so.3.0.0 >>> backupd 2584 cyrus mem REG 253,0 20056 68797036 >>> /usr/lib64/sasl2/liblogin.so.3.0.0 >>> backupd 2584 cyrus mem REG 253,0 1845816 135873512 /usr/lib64/libdb-5.3.so >>> backupd 2584 cyrus mem REG 253,0 28272 67113426 >>> /usr/lib64/sasl2/libsasldb.so.3.0.0 >>> backupd 2584 cyrus mem REG 253,0 20064 67113423 >>> /usr/lib64/sasl2/libanonymous.so.3.0.0 >>> backupd 2584 cyrus mem REG 253,0 11448 135872733 /usr/lib64/libfreebl3.so >>> backupd 2584
Re: backupd IOERROR reading backup files larger than 2GB
Hi Carlos, This is quite weird, I'm not sure why a 64bit platform would have any trouble around the 2GB mark?? What does the Cyrus ./configure report for your system's integer sizes? e.g. mine shows: > checking size of int... 4 > checking size of long... 8 > checking size of size_t... 8 > checking size of off_t... 8 > checking size of time_t... 8 > checking size of long long int... 8 > checking size of unsigned long long int... 8 > checking whether byte ordering is bigendian... no What's your level of comfort with C debugging? It'd be very helpful to see a core file+binary from the time that lseek error occurs? Cheers, ellie On Fri, Jun 7, 2019, at 1:58 AM, Carlos Larrañaga wrote: > Hi Ellie, > > Thanks for answering. We use latest 64bit Oracle Linux (not CentOs like I > said before, sorry) and zlib is also 64bit version: >> # uname -a >> Linux xxx 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 17:35:45 PDT 2019 >> x86_64 x86_64 x86_64 GNU/Linux >> >> # yum list installed zlib >> Loaded plugins: langpacks, ulninfo >> Installed Packages >> zlib.x86_64 1.2.7-18.el7 @ol7_latest >> >> # lsof -p 17005 |grep lib |grep -i z >> backupd 17005 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7 >> >> # file /usr/lib64/libz.so.1.2.7 >> /usr/lib64/libz.so.1.2.7: ELF 64-bit LSB shared object, x86-64, version 1 >> (SYSV), dynamically linked, >> BuildID[sha1]=b9d5f73428bd6ad68c96986b57bea3b7cedb9745, stripped >> >> # rpm -qf /usr/lib64/libz.so.1.2.7 >> zlib-1.2.7-18.el7.x86_64 > I have summarized here some information about the backupd error reading the > file descriptor 15, which is a backup larger than 2GB. The error is logged > 1755 times. Instead, there is no error for fd 12, which is a backup of less > than 2 GB: >> *#-- CLIENT SIDE* >> *---* >> >> >> # time sync_client -v -o -n backup -A >> Thu Jun 6 17:08:02 CEST 2019 >> USER aaa >> QUOTA user.aaa >> USER ccc >> Error from do_user(ccc): bailing out! >> >> real 30m16.223s >> user 0m0.006s >> sys 0m0.006s >> ** >> >> *#-- BACKUP SERVER SIDE * >> # LOGFILE >> >> # tail -f /var/log/imapd.log >> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >> such file or directory >> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >> such file or directory >> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >> such file or directory >> Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No >> such file or directory >> >> >> *# lsof of the backupd PROCESS *# lsof -P -p 2584 >> >> COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME >> backupd 2584 cyrus cwd DIR 253,0 4096 128 / >> backupd 2584 cyrus rtd DIR 253,0 4096 128 / >> backupd 2584 cyrus txt REG 253,0 768632 939542214 >> /usr/local/cyrus/libexec/backupd >> backupd 2584 cyrus mem REG 253,0 37208 135746334 /usr/lib64/libnss_sss.so.2 >> backupd 2584 cyrus mem REG 253,0 31416 134406707 >> /usr/lib64/libnss_dns-2.17.so >> backupd 2584 cyrus mem REG 253,0 61632 134406709 >> /usr/lib64/libnss_files-2.17.so >> backupd 2584 cyrus mem REG 253,0 43336 68814977 >> /usr/lib64/sasl2/webmail.so.0.0.0 >> backupd 2584 cyrus mem REG 253,0 57960 68856627 >> /usr/lib64/sasl2/libdigestmd5.so.3.0.0 >> backupd 2584 cyrus mem REG 253,0 24232 68797220 >> /usr/lib64/sasl2/libcrammd5.so.3.0.0 >> backupd 2584 cyrus mem REG 253,0 20088 68797044 >> /usr/lib64/sasl2/libplain.so.3.0.0 >> backupd 2584 cyrus mem REG 253,0 20056 68797036 >> /usr/lib64/sasl2/liblogin.so.3.0.0 >> backupd 2584 cyrus mem REG 253,0 1845816 135873512 /usr/lib64/libdb-5.3.so >> backupd 2584 cyrus mem REG 253,0 28272 67113426 >> /usr/lib64/sasl2/libsasldb.so.3.0.0 >> backupd 2584 cyrus mem REG 253,0 20064 67113423 >> /usr/lib64/sasl2/libanonymous.so.3.0.0 >> backupd 2584 cyrus mem REG 253,0 11448 135872733 /usr/lib64/libfreebl3.so >> backupd 2584 cyrus mem REG 253,0 155784 134400716 /usr/lib64/libselinux.so.1 >> backupd 2584 cyrus mem REG 253,0 15464 134401447 >> /usr/lib64/libkeyutils.so.1.5 >> backupd 2584 cyrus mem REG 253,0 40672 134399432 /usr/lib64/libcrypt-2.17.so >> backupd 2584 cyrus mem REG 253,0 105824 134781819 >> /usr/lib64/libresolv-2.17.so >> backupd 2584 cyrus mem REG 253,0 53944 134406785 >> /usr/lib64/libjansson.so.4.10.0 >> backupd 2584 cyrus mem REG 253,0 88776 135352942 >> /usr/lib64/libgcc_s-4.8.5-20150702.so.1 >> backupd 2584 cyrus mem REG 253,0 1137032 134400337 /usr/lib64/libm-2.17.so >> backupd 2584 cyrus mem REG 253,0 991616 135352396 >> /usr/lib64/libstdc++.so.6.0.19 >> backupd 2584 cyrus mem REG 253,0 19296 134399434 /usr/lib64/libdl-2.17.so >> backupd 2584 cyrus mem REG 253,0 142008 134722750 >> /usr/lib64/libpthread-2.17.so >> backupd 2584 cyrus mem REG 253,0 2151704 134365659 /usr/lib64/libc-2.17.so >> backupd 2584 cyrus mem REG 253,0 753232 134401161 >> /usr/lib64/libsqlite3.so.0.8.6 >>
Re: backupd IOERROR reading backup files larger than 2GB
Hi Ellie, Thanks for answering. We use latest 64bit Oracle Linux (not CentOs like I said before, sorry) and zlib is also 64bit version: # uname -a Linux xxx 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 17:35:45 PDT 2019 x86_64 x86_64 x86_64 GNU/Linux # yum list installed zlib Loaded plugins: langpacks, ulninfo Installed Packages zlib.x86_64 1.2.7-18.el7 @ol7_latest # lsof -p 17005 |grep lib |grep -i z backupd 17005 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7 # file /usr/lib64/libz.so.1.2.7 /usr/lib64/libz.so.1.2.7: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=b9d5f73428bd6ad68c96986b57bea3b7cedb9745, stripped # rpm -qf /usr/lib64/libz.so.1.2.7 zlib-1.2.7-18.el7.x86_64 I have summarized here some information about the backupd error reading the file descriptor 15, which is a backup larger than 2GB. The error is logged 1755 times. Instead, there is no error for fd 12, which is a backup of less than 2 GB: *#-- CLIENT SIDE ---*** # time sync_client -v -o -n backup -A Thu Jun 6 17:08:02 CEST 2019 USER aaa QUOTA user.aaa USER ccc Error from do_user(ccc): bailing out! real 30m16.223s user 0m0.006s sys 0m0.006s * * **#-- BACKUP SERVER SIDE *# LOGFILE *# tail -f /var/log/imapd.log Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory Jun 6 17:09:11 bcrux cyrus/backupd[2584]: IOERROR: gzuc_read: lseek 15: No such file or directory *# lsof of the backupd PROCESS *# lsof -P -p 2584 COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME backupd 2584 cyrus cwd DIR 253,0 4096 128 / backupd 2584 cyrus rtd DIR 253,0 4096 128 / backupd 2584 cyrus txt REG 253,0 768632 939542214 /usr/local/cyrus/libexec/backupd backupd 2584 cyrus mem REG 253,0 37208 135746334 /usr/lib64/libnss_sss.so.2 backupd 2584 cyrus mem REG 253,0 31416 134406707 /usr/lib64/libnss_dns-2.17.so backupd 2584 cyrus mem REG 253,0 61632 134406709 /usr/lib64/libnss_files-2.17.so backupd 2584 cyrus mem REG 253,0 43336 68814977 /usr/lib64/sasl2/webmail.so.0.0.0 backupd 2584 cyrus mem REG 253,0 57960 68856627 /usr/lib64/sasl2/libdigestmd5.so.3.0.0 backupd 2584 cyrus mem REG 253,0 24232 68797220 /usr/lib64/sasl2/libcrammd5.so.3.0.0 backupd 2584 cyrus mem REG 253,0 20088 68797044 /usr/lib64/sasl2/libplain.so.3.0.0 backupd 2584 cyrus mem REG 253,0 20056 68797036 /usr/lib64/sasl2/liblogin.so.3.0.0 backupd 2584 cyrus mem REG 253,0 1845816 135873512 /usr/lib64/libdb-5.3.so backupd 2584 cyrus mem REG 253,0 28272 67113426 /usr/lib64/sasl2/libsasldb.so.3.0.0 backupd 2584 cyrus mem REG 253,0 20064 67113423 /usr/lib64/sasl2/libanonymous.so.3.0.0 backupd 2584 cyrus mem REG 253,0 11448 135872733 /usr/lib64/libfreebl3.so backupd 2584 cyrus mem REG 253,0 155784 134400716 /usr/lib64/libselinux.so.1 backupd 2584 cyrus mem REG 253,0 15464 134401447 /usr/lib64/libkeyutils.so.1.5 backupd 2584 cyrus mem REG 253,0 40672 134399432 /usr/lib64/libcrypt-2.17.so backupd 2584 cyrus mem REG 253,0 105824 134781819 /usr/lib64/libresolv-2.17.so backupd 2584 cyrus mem REG 253,0 53944 134406785 /usr/lib64/libjansson.so.4.10.0 backupd 2584 cyrus mem REG 253,0 88776 135352942 /usr/lib64/libgcc_s-4.8.5-20150702.so.1 backupd 2584 cyrus mem REG 253,0 1137032 134400337 /usr/lib64/libm-2.17.so backupd 2584 cyrus mem REG 253,0 991616 135352396 /usr/lib64/libstdc++.so.6.0.19 backupd 2584 cyrus mem REG 253,0 19296 134399434 /usr/lib64/libdl-2.17.so backupd 2584 cyrus mem REG 253,0 142008 134722750 /usr/lib64/libpthread-2.17.so backupd 2584 cyrus mem REG 253,0 2151704 134365659 /usr/lib64/libc-2.17.so backupd 2584 cyrus mem REG 253,0 753232 134401161 /usr/lib64/libsqlite3.so.0.8.6 *backupd 2584 cyrus mem REG 253,0 90248 134400930 /usr/lib64/libz.so.1.2.7* backupd 2584 cyrus mem REG 253,0
Re: backupd IOERROR reading backup files larger than 2GB
It kinda sounds like your platform might be 32bit? Or your zlib is compiled to use 32bit integer sizes? On Mon, Jun 3, 2019, at 10:07 PM, Carlos Larrañaga wrote: > Hi, > > We're testing backup feature un cyrus-imapd 3.0.10. There's no problem when > backup is created first time, but when the backup already exists and is > larger than 2GB, we get the following error from the backupd: >> cyrus/backupd[]: IOERROR: gzuc_read: lseek 12: No such file or directory > As said, it happens only when then backup file already exists and is larger > than 2GB. The backupd process keeps reading de compressed backup file til > sync_client exit with "Error from sync_do_user(xxx): bailing out!". > > We use xfs with latest CentOs. Seems like backupd is unable to read the > compressed backup file. > > Anyone else with this problem? Any idea how to fix it? > Thanks in advance for your help. > > Best regards, > Carlos. > > Cyrus Home Page: http://www.cyrusimap.org/ > List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ > To Unsubscribe: > https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus > > *Attachments:* > * smime.p7s Cyrus Home Page: http://www.cyrusimap.org/ List Archives/Info: http://lists.andrew.cmu.edu/pipermail/info-cyrus/ To Unsubscribe: https://lists.andrew.cmu.edu/mailman/listinfo/info-cyrus
