Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function?
Jennifer, If you are not doing so, while the bug is still around, I suggest you use Item Status to edit so that you can edit a number of items at one time using the Actions for Catalogers menu. You can import a file of barcodes that need to be edited (you can create the file from a report. I would chop it into workable chunks), then highlight 20 or so at a time, edit volumes, then edit item attributes (or vice versa). If you try to edit to many at a time, it will take a long time for the system to respond, so you may want to experiment to get the optimum number. You can run through a list pretty quickly. If the system is adding empty volumes with the new call number, rather than deleting, you should transfer the item from the old call number to the new one. Or delete all the empty vols (I think there may be a script for that) and edit the old call numbers/items in Item status. I think the later would be easier since you can do so several at a time where transferring the items has to be done one at a time. For those that have already been deleted, I think you may need to undelete the volumes under the hood. I don't know how to do that but I am sure others on the list can tell you. Elaine J. Elaine Hardy PINES Collaborative Projects Manager Georgia Public Library Service 1800 Century Place, Ste 150 Atlanta, Ga. 30345-4304 404.235.7128 404.235.7201, fax eha...@georgialibraries.org www.georgialibraries.org www.georgialibraries.org/pines From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Kathy Lussier Sent: Thursday, July 23, 2015 3:16 PM To: open-ils-general@list.georgialibraries.org Subject: Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function? Hi Jennifer, There is an existing bug on this issue in Launchpad. https://bugs.launchpad.net/evergreen/+bug/1040686 Kathy On 07/23/2015 03:14 PM, Walz, Jennifer wrote: All - We are experiencing what we think is a bug, but maybe we are just mis-understanding something. When we edit an item with the edit items / Volumes per bib function, it ADDS a second, empty call number record. (We can see it via the Holdings Maintenance screen). All we want to do is to CHANGE a call number on a record. We DON'T want to create a second call number record with no items attached. It would be nice to also change the copy location at the same time (what is this combo function for, if not that??).But all we get is a second empty record. Not an actual update of the current items / volumes. What is going on here? Is this a bug or something? Why is this occurring? Is something wrong with this combo edit function? However, when we edit the item with the Edit Volume option only, it does just what we expect - it changes only the call number and updates it. It does not create a second record.Then we have to go into the Edit Item Attributes option to fix the copy location. Now all is good. On top of that, since we have discovered this problem, we have been DELETING volumes and REPLACING them with the same barcode from the original, in holdings maintenance.So, now the system thinks we have deleted that volume / copy. When I run a report, it has marked those as deleted with the indicator, but they are ACUTALLY STILL THERE! Oh now what? How do I remove the deleted indicator from those records?They are not actually deleted!! Please tell me how to fix all of this! It is a nightmare. All we are trying to do is to move our Juvenile items from one copy location to another and change all the call numbers. What gives!? Thanks! Jennifer -- Jennifer Walz, MLS - Head of ILS Crazy Kinlaw Library - Asbury University One Macklem Drive, Wilmore, KY 40390 859-858-3511 ext. 2269 jlw...@asbury.edu -- Kathy Lussier Project Coordinator Massachusetts Library Network Cooperative (508) 343-0128 kluss...@masslnc.org Twitter: http://www.twitter.com/kmlussier
Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function?
All - Here is where I don't understand the current construct and wouldn't it make more sense to have the call number and the barcode be at the item level for each record? Like this: Title blah blah blah etc, author and owning library and so on. - 345.0998 B58a 1908987293 - 345.0998 B58a 1908987294 - 345.0998 B58a 1908987294 Why do the call numbers need to have their own level called volume? What does it add to the mix? In other words, what does this particular construct enable libraries to do specifically? If you had the call number at the same level of the barcode, you could STILL update either and not affect the owner or copy location (unless you wanted to). Let's say an owning library had 5 copies of a title, but wanted to put them in five different locations - each with a different call number. You could if you wanted, without creating and fiddling with volume level data. Why can't that level just be eliminated altogether? Just saying. I'm just not seeing the benefit of having the call number / volume level. Seems to complicate matters unnecessarily. If anyone can give me ANY help to fix about 300 records that have gotten deleted and then mysteriously not, I would be most grateful! Where is that pesky deleted indicator anyway?? I want to turn it off for these records. (my other pet peeve! Items should be GONE from the system entirely and not in a phantom zone!) Thanks! Jennifer -- Jennifer Walz, MLS - Head of Research Distance Services Kinlaw Library - Asbury University One Macklem Drive, Wilmore, KY 40390 859-858-3511 ext. 2269 jlw...@asbury.edu -Original Message- From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Kathy Lussier Sent: Thursday, July 23, 2015 4:29 PM To: open-ils-general@list.georgialibraries.org Subject: Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function? Hi Jason, Yes, I understand the mindset behind the current behavior. If I were to look at tackling this bug/wishlist request, I think I would look at adding a prompt that appears when the user is updating a volume from the unified editor if there are other copies attached to the volume that aren't being edited at the time the update is being made. In many cases, I think the answer to the question is Yes, but I can see why you wouldn't want to change the call number label for all six copies if the intent was just to update the label for one. Kathy On 07/23/2015 04:22 PM, Jason Etheridge wrote: Should we expect for all copies on a volume to inherit a call number tweak if just a single copy was being edited as the entry point? An answer of No went into the mindset that built the current behavior. -- Kathy Lussier Project Coordinator Massachusetts Library Network Cooperative (508) 343-0128 kluss...@masslnc.org Twitter: http://www.twitter.com/kmlussier
Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function?
Should we expect for all copies on a volume to inherit a call number tweak if just a single copy was being edited as the entry point? An answer of No went into the mindset that built the current behavior. -- Jason Etheridge | Community and Migration Manager | Equinox Software, Inc. / The Open Source Experts | phone: 1-877-OPEN-ILS (673-6457) | email: ja...@esilibrary.com | web: http://www.esilibrary.com
Re: [OPEN-ILS-GENERAL] Apache leaking sockets/FD
Josh, When you see this happen again, please try `lsof -n -P -p pid` (note the -n and -P) instead. That will give the IP addrs and port numbers without attempting to convert host or service names and should help you identify the offending connections. Regards, -- Mike Rylander | President | Equinox Software, Inc. / The Open Source Experts | phone: 1-877-OPEN-ILS (673-6457) | email: mi...@esilibrary.com | web: http://www.esilibrary.com On Wed, Jul 22, 2015 at 9:14 PM, Josh Stompro stomp...@exchange.larl.org wrote: Greetings, I’ve been trying to figure out why my two front end Evergreen application servers keep hitting some resource limits having to do with tcp sockets (numtcpsock openvz beancounters). I’m running EG 2.8.2, OpenSRF 2.4.1, Debian Jessie in an Openvz container on Proxmox VE 3.4 Nothing looks out of the ordinary when I look at the output of ‘ss –s’ or ‘netstat –a’, but the numtcpsock counter keeps going up, until I have 5000+ reported open tcp socket connections. I think I’ve narrowed it down to apache, since restarting apache resets the numtcpsock numbers back in line with what is reported by ‘ss –s’ If I take a look at all the open fd’s of an apache process, I see a bunch of the following. So I think some socket connections are being opened but not closed properly. (lsof –p pid) /usr/sbin 11821 opensrf 171u sock0,6 0t0 61135031 can't identify protocol /usr/sbin 11821 opensrf 172u sock0,6 0t0 61135034 can't identify protocol /usr/sbin 11821 opensrf 173u sock0,6 0t0 61135037 can't identify protocol /usr/sbin 11821 opensrf 174u sock0,6 0t0 61321969 can't identify protocol /usr/sbin 11821 opensrf 175u sock0,6 0t0 61321972 can't identify protocol /usr/sbin 11821 opensrf 176u sock0,6 0t0 61321975 can't identify protocol /usr/sbin 11821 opensrf 177u sock0,6 0t0 61321978 can't identify protocol /usr/sbin 11821 opensrf 178u sock0,6 0t0 61321981 can't identify protocol /usr/sbin 11821 opensrf 179u sock0,6 0t0 61458539 can't identify protocol /usr/sbin 11821 opensrf 180u sock0,6 0t0 61458542 can't identify protocol /usr/sbin 11821 opensrf 181u sock0,6 0t0 61458545 can't identify protocol /usr/sbin 11821 opensrf 182u sock0,6 0t0 61458548 can't identify protocol /usr/sbin 11821 opensrf 183u sock0,6 0t0 61458551 can't identify protocol /usr/sbin 11821 opensrf 184u sock0,6 0t0 62085495 can't identify protocol /usr/sbin 11821 opensrf 185u sock0,6 0t0 62085498 can't identify protocol /usr/sbin 11821 opensrf 186u sock0,6 0t0 62085501 can't identify protocol /usr/sbin 11821 opensrf 187u sock0,6 0t0 62085504 can't identify protocol /usr/sbin 11821 opensrf 188u sock0,6 0t0 62085507 can't identify protocol /usr/sbin 11821 opensrf 189u sock0,6 0t0 63801157 can't identify protocol /usr/sbin 11821 opensrf 190u sock0,6 0t0 63801160 can't identify protocol /usr/sbin 11821 opensrf 191u sock0,6 0t0 63801163 can't identify protocol /usr/sbin 11821 opensrf 192u sock0,6 0t0 63801166 can't identify protocol /usr/sbin 11821 opensrf 193u sock0,6 0t0 63801169 can't identify protocol /usr/sbin 11821 opensrf 194u sock0,6 0t0 63961716 can't identify protocol /usr/sbin 11821 opensrf 195u sock0,6 0t0 63961719 can't identify protocol /usr/sbin 11821 opensrf 196u sock0,6 0t0 63961722 can't identify protocol /usr/sbin 11821 opensrf 197u sock0,6 0t0 63961725 can't identify protocol /usr/sbin 11821 opensrf 198u sock0,6 0t0 63961728 can't identify protocol /usr/sbin 11821 opensrf 199u sock0,6 0t0 64808966 can't identify protocol /usr/sbin 11821 opensrf 200u sock0,6 0t0 64808971 can't identify protocol /usr/sbin 11821 opensrf 201u sock0,6 0t0 64808974 can't identify protocol /usr/sbin 11821 opensrf 202u sock0,6 0t0 64808977 can't identify protocol /usr/sbin 11821 opensrf 203u sock0,6 0t0 64808980 can't identify protocol I’m not sure how to track down the problem, I’ll try using strace to see what connections are being created, but I’m not quite sure what to look for. If anyone has run into this before, please let me know. Josh
Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
Adding a close seems to have fixed the problem for me. To try it out I edited
/usr/local/share/perl/5.20.2/OpenILS/WWW/EGCatLoader/Record.pm and changed line
577 to
576 # To avoid a lot of hanging connections.
577 if ($content-{request}) {
578 $content-{request}-shutdown(2);
579 $content-{request}-close();
580 }
Now when I load a bib detail record the number of orphaned sock connections
doesn’t keep climbing. I’ll test some more and open a bug if it continues to
look good.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 2:27 PM
To: Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
I just took a look at a test system running Debian Wheezy with EG 2.8.2 and
Opensrf 2.4.1, same issue, each page load of a record detail page leaks 5 file
descriptors, that show up when doing a “lsof | grep “\sock\” | wc –l” before
and after the request.
So the steps to test it are.
1. Run “lsof |grep \sock\ | wc –l” to see how many orphan FD there
currently are.
2. Load a record detail page to trigger the added_content connections
back to the local host. http://egcatalog/eg/opac/record/10
3. Run “lsof |grep \sock\ | wc –l” to see if the number increased.
I don’t have a non openvz based system to test on right now. I would love to
hear if anyone else sees this.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 2:06 PM
To: Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
I found this page that seems to say that a close is always needed after a
shutdown of a socket to free the FD.
http://www.perlmonks.org/?node=108244
I’ll look at my other test systems and see if I see the same issue, but haven’t
noticed it because of the low number of requests.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 1:31 PM
To: Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
This is what strace shos me.
[pid 14793] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP unfinished ...
[pid 14793] ... socket resumed ) = 83
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] fcntl(83, F_SETFD, FD_CLOEXEC unfinished ...
[pid 14793] ... fcntl resumed ) = 0
[pid 14793] connect(83, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(192.168.46.32)}, 16 unfinished ...
[pid 14793] ... connect resumed ) = 0
[pid 14793] write(83, HEAD /opac/extras/ac/summary/html/r/1001
HTTP/1.1\r\nConnection: close\r\nHost: virt-egapp2.larl.org\r\n\r\n, 100
unfinished ...
[pid 14793] ... write resumed ) = 100
[pid 14793] read(83, unfinished ...
[pid 14793] ... read resumed HTTP/1.1 404 Not Found\r\nDate: Thu, 23 Jul
2015 03:16:49 GMT\r\nServer: Apache/2.4.10 (Debian)\r\nConnection:
close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n, 1024) = 159
[pid 14793] shutdown(83, SHUT_RDWR unfinished ...
[pid 14793] ... shutdown resumed )= 0
After this point FD 83 never shows up again in the strace log, but it does show
up in the lsof –p pid display as shown before. I’m wondering if that is
because no close for FD 83 is called? I’ve read that sometimes the shutdown()
implementation includes the close, and sometimes it does not. I’m trying to
figure out if the shutdown at the end of EGCatLoader/Record.pm includes a
close.. or if the implementation changed with the versions of the perl libs
that Jessie has.
I’m also wondering if I’m way off base or not.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 10:03 AM
To: Evergreen Development Discussion List; Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
Hello Mike,
Lsof –n –P –p pid doesn’t give any new info about those connections.
ot@virt-egapp2:/openils/var/templates# lsof -n -P -p 5684
COMMANDPIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
/usr/sbin 5684 opensrf cwdDIR 0,45 4096 34071974 /
/usr/sbin 5684 opensrf
Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function?
Hi Jason, Yes, I understand the mindset behind the current behavior. If I were to look at tackling this bug/wishlist request, I think I would look at adding a prompt that appears when the user is updating a volume from the unified editor if there are other copies attached to the volume that aren't being edited at the time the update is being made. In many cases, I think the answer to the question is Yes, but I can see why you wouldn't want to change the call number label for all six copies if the intent was just to update the label for one. Kathy On 07/23/2015 04:22 PM, Jason Etheridge wrote: Should we expect for all copies on a volume to inherit a call number tweak if just a single copy was being edited as the entry point? An answer of No went into the mindset that built the current behavior. -- Kathy Lussier Project Coordinator Massachusetts Library Network Cooperative (508) 343-0128 kluss...@masslnc.org Twitter: http://www.twitter.com/kmlussier
Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function?
So is the edit items/vols, then a way to change the call number on one copy attached to the volume, thereby creating a new volume and transferring that item to it? That is not what I would expect from something called that. I would expect to be able to edit the item attributes and call number at the same time, as Jennifer did. I'm not sure what I would call it, given space limitations but I can definitely see the confusion. As you and Kathy are describing it, the functionality is great since otherwise you would have multiple steps to create a new vol, then transfer the items. Elaine J. Elaine Hardy PINES Collaborative Projects Manager Georgia Public Library Service 1800 Century Place, Ste 150 Atlanta, Ga. 30345-4304 404.235.7128 404.235.7201, fax eha...@georgialibraries.org www.georgialibraries.org www.georgialibraries.org/pines -Original Message- From: Open-ils-general [mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Jason Etheridge Sent: Thursday, July 23, 2015 4:22 PM To: Evergreen Discussion Group Subject: Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function? Should we expect for all copies on a volume to inherit a call number tweak if just a single copy was being edited as the entry point? An answer of No went into the mindset that built the current behavior. -- Jason Etheridge | Community and Migration Manager | Equinox Software, Inc. / The Open Source Experts | phone: 1-877-OPEN-ILS (673-6457) | email: ja...@esilibrary.com | web: http://www.esilibrary.com
Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
Hello Mike, Lsof –n –P –p pid doesn’t give any new info about those connections. ot@virt-egapp2:/openils/var/templates# lsof -n -P -p 5684 COMMANDPIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME /usr/sbin 5684 opensrf cwdDIR 0,45 4096 34071974 / /usr/sbin 5684 opensrf rtdDIR 0,45 4096 34071974 / /usr/sbin 5684 opensrf txtREG 0,45 654136 34947316 /usr/sbin/apache2 /usr/sbin 5684 opensrf memREG 253,2 35374581 /lib/x86_64-linux-gnu/libnss_dns-2.19.so (path dev=0,45) /usr/sbin 5684 opensrf memREG 253,2 39379859 /usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Hash/Util/Util.so (path dev=0,45) /usr/sbin 5684 opensrf memREG 0,50 66964636 (deleted)/dev/zero (stat: No such file or directory) SNIP /usr/sbin 5684 opensrf 35u sock0,6 0t0 67405034 can't identify protocol /usr/sbin 5684 opensrf 36u sock0,6 0t0 67405037 can't identify protocol /usr/sbin 5684 opensrf 37u sock0,6 0t0 67405040 can't identify protocol /usr/sbin 5684 opensrf 38u sock0,6 0t0 67405043 can't identify protocol /usr/sbin 5684 opensrf 39u sock0,6 0t0 67405046 can't identify protocol /usr/sbin 5684 opensrf 40u sock0,6 0t0 67689829 can't identify protocol /usr/sbin 5684 opensrf 41u sock0,6 0t0 67689832 can't identify protocol /usr/sbin 5684 opensrf 42u sock0,6 0t0 67689835 can't identify protocol /usr/sbin 5684 opensrf 43u sock0,6 0t0 67689838 can't identify protocol /usr/sbin 5684 opensrf 44u sock0,6 0t0 67689841 can't identify protocol From using strace it looks like the problem connections are from apache trying to load the various added content types, the connections get shutdown but the FD for the socket never gets closed. https://github.com/evergreen-library-system/Evergreen/blob/6bb8ea5599d39d41d623d1891b3c509c4e439178/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader/Record.pm#L577 I’ll post more info when I get a chance, time to take the kids to the park before we all go stir crazy ;-) Josh From: Open-ils-dev [mailto:open-ils-dev-boun...@list.georgialibraries.org] On Behalf Of Mike Rylander Sent: Thursday, July 23, 2015 8:02 AM To: Evergreen Discussion Group Cc: open-ils-...@list.georgialibraries.org Subject: Re: [OPEN-ILS-DEV] [OPEN-ILS-GENERAL] Apache leaking sockets/FD Josh, When you see this happen again, please try `lsof -n -P -p pid` (note the -n and -P) instead. That will give the IP addrs and port numbers without attempting to convert host or service names and should help you identify the offending connections. Regards, -- Mike Rylander | President | Equinox Software, Inc. / The Open Source Experts | phone: 1-877-OPEN-ILS (673-6457) | email: mi...@esilibrary.commailto:mi...@esilibrary.com | web: http://www.esilibrary.com On Wed, Jul 22, 2015 at 9:14 PM, Josh Stompro stomp...@exchange.larl.orgmailto:stomp...@exchange.larl.org wrote: Greetings, I’ve been trying to figure out why my two front end Evergreen application servers keep hitting some resource limits having to do with tcp sockets (numtcpsock openvz beancounters). I’m running EG 2.8.2, OpenSRF 2.4.1, Debian Jessie in an Openvz container on Proxmox VE 3.4 Nothing looks out of the ordinary when I look at the output of ‘ss –s’ or ‘netstat –a’, but the numtcpsock counter keeps going up, until I have 5000+ reported open tcp socket connections. I think I’ve narrowed it down to apache, since restarting apache resets the numtcpsock numbers back in line with what is reported by ‘ss –s’ If I take a look at all the open fd’s of an apache process, I see a bunch of the following. So I think some socket connections are being opened but not closed properly. (lsof –p pid) /usr/sbin 11821 opensrf 171u sock0,6 0t0 61135031 can't identify protocol /usr/sbin 11821 opensrf 172u sock0,6 0t0 61135034 can't identify protocol /usr/sbin 11821 opensrf 173u sock0,6 0t0 61135037 can't identify protocol /usr/sbin 11821 opensrf 174u sock0,6 0t0 61321969 can't identify protocol /usr/sbin 11821 opensrf 175u sock0,6 0t0 61321972 can't identify protocol /usr/sbin 11821 opensrf 176u sock0,6 0t0 61321975 can't identify protocol /usr/sbin 11821 opensrf 177u sock0,6 0t0 61321978 can't identify protocol /usr/sbin 11821 opensrf 178u sock0,6 0t0 61321981 can't identify protocol /usr/sbin 11821 opensrf 179u sock0,6 0t0 61458539 can't identify protocol /usr/sbin 11821 opensrf 180u sock0,6 0t0 61458542 can't identify protocol
Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
I found this page that seems to say that a close is always needed after a
shutdown of a socket to free the FD.
http://www.perlmonks.org/?node=108244
I’ll look at my other test systems and see if I see the same issue, but haven’t
noticed it because of the low number of requests.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 1:31 PM
To: Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
This is what strace shows me.
[pid 14793] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP unfinished ...
[pid 14793] ... socket resumed ) = 83
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] fcntl(83, F_SETFD, FD_CLOEXEC unfinished ...
[pid 14793] ... fcntl resumed ) = 0
[pid 14793] connect(83, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(192.168.46.32)}, 16 unfinished ...
[pid 14793] ... connect resumed ) = 0
[pid 14793] write(83, HEAD /opac/extras/ac/summary/html/r/1001
HTTP/1.1\r\nConnection: close\r\nHost: virt-egapp2.larl.org\r\n\r\n, 100
unfinished ...
[pid 14793] ... write resumed ) = 100
[pid 14793] read(83, unfinished ...
[pid 14793] ... read resumed HTTP/1.1 404 Not Found\r\nDate: Thu, 23 Jul
2015 03:16:49 GMT\r\nServer: Apache/2.4.10 (Debian)\r\nConnection:
close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n, 1024) = 159
[pid 14793] shutdown(83, SHUT_RDWR unfinished ...
[pid 14793] ... shutdown resumed )= 0
After this point FD 83 never shows up again in the strace log, but it does show
up in the lsof –p pid display as shown before. I’m wondering if that is
because no close for FD 83 is called? I’ve read that sometimes the shutdown()
implementation includes the close, and sometimes it does not. I’m trying to
figure out if the shutdown at the end of EGCatLoader/Record.pm includes a
close.. or if the implementation changed with the versions of the perl libs
that Jessie has.
I’m also wondering if I’m way off base or not.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 10:03 AM
To: Evergreen Development Discussion List; Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
Hello Mike,
Lsof –n –P –p pid doesn’t give any new info about those connections.
ot@virt-egapp2:/openils/var/templates# lsof -n -P -p 5684
COMMANDPIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
/usr/sbin 5684 opensrf cwdDIR 0,45 4096 34071974 /
/usr/sbin 5684 opensrf rtdDIR 0,45 4096 34071974 /
/usr/sbin 5684 opensrf txtREG 0,45 654136 34947316
/usr/sbin/apache2
/usr/sbin 5684 opensrf memREG 253,2 35374581
/lib/x86_64-linux-gnu/libnss_dns-2.19.so (path dev=0,45)
/usr/sbin 5684 opensrf memREG 253,2 39379859
/usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Hash/Util/Util.so (path dev=0,45)
/usr/sbin 5684 opensrf memREG 0,50 66964636
(deleted)/dev/zero (stat: No such file or directory)
SNIP
/usr/sbin 5684 opensrf 35u sock0,6 0t0 67405034 can't
identify protocol
/usr/sbin 5684 opensrf 36u sock0,6 0t0 67405037 can't
identify protocol
/usr/sbin 5684 opensrf 37u sock0,6 0t0 67405040 can't
identify protocol
/usr/sbin 5684 opensrf 38u sock0,6 0t0 67405043 can't
identify protocol
/usr/sbin 5684 opensrf 39u sock0,6 0t0 67405046 can't
identify protocol
/usr/sbin 5684 opensrf 40u sock0,6 0t0 67689829 can't
identify protocol
/usr/sbin 5684 opensrf 41u sock0,6 0t0 67689832 can't
identify protocol
/usr/sbin 5684 opensrf 42u sock0,6 0t0 67689835 can't
identify protocol
/usr/sbin 5684 opensrf 43u sock0,6 0t0 67689838 can't
identify protocol
/usr/sbin 5684 opensrf 44u sock0,6 0t0 67689841 can't
identify protocol
From using strace it looks like the problem connections are from apache trying
to load the various added content types, the connections get shutdown but the
FD for the socket never gets closed.
[OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function?
All - We are experiencing what we think is a bug, but maybe we are just mis-understanding something. When we edit an item with the edit items / Volumes per bib function, it ADDS a second, empty call number record. (We can see it via the Holdings Maintenance screen). All we want to do is to CHANGE a call number on a record. We DON'T want to create a second call number record with no items attached. It would be nice to also change the copy location at the same time (what is this combo function for, if not that??).But all we get is a second empty record. Not an actual update of the current items / volumes. What is going on here? Is this a bug or something? Why is this occurring? Is something wrong with this combo edit function? However, when we edit the item with the Edit Volume option only, it does just what we expect - it changes only the call number and updates it. It does not create a second record.Then we have to go into the Edit Item Attributes option to fix the copy location. Now all is good. On top of that, since we have discovered this problem, we have been DELETING volumes and REPLACING them with the same barcode from the original, in holdings maintenance.So, now the system thinks we have deleted that volume / copy. When I run a report, it has marked those as deleted with the indicator, but they are ACUTALLY STILL THERE! Oh now what? How do I remove the deleted indicator from those records?They are not actually deleted!! Please tell me how to fix all of this! It is a nightmare. All we are trying to do is to move our Juvenile items from one copy location to another and change all the call numbers. What gives!? Thanks! Jennifer -- Jennifer Walz, MLS - Head of ILS Crazy Kinlaw Library - Asbury University One Macklem Drive, Wilmore, KY 40390 859-858-3511 ext. 2269 jlw...@asbury.edu
Re: [OPEN-ILS-GENERAL] Problem with Edit Items / Volumes per Bib function?
Hi Jennifer, There is an existing bug on this issue in Launchpad. https://bugs.launchpad.net/evergreen/+bug/1040686 Kathy On 07/23/2015 03:14 PM, Walz, Jennifer wrote: All – We are experiencing what we think is a bug, but maybe we are just mis-understanding something. When we edit an item with the “edit items / Volumes per bib” function, it ADDS a second, empty call number record. (We can see it via the Holdings Maintenance screen). All we want to do is to CHANGE a call number on a record. We DON’T want to create a second call number record with no items attached. It would be nice to also change the copy location at the same time (what is this combo function for, if not that??).But all we get is a second empty record. Not an actual update of the current items / volumes. What is going on here? Is this a bug or something? Why is this occurring? Is something wrong with this combo edit function? However, when we edit the item with the “Edit Volume” option only, it does just what we expect – it changes only the call number and updates it. It does not create a second record.Then we have to go into the “Edit Item Attributes” option to fix the copy location. Now all is good. On top of that, since we have discovered this problem, we have been DELETING volumes and REPLACING them with the same barcode from the original, in “holdings maintenance.”So, now the system thinks we have deleted that volume / copy. When I run a report, it has marked those as “deleted” with the indicator, but they are ACUTALLY STILL THERE! Oh now what? How do I remove the deleted indicator from those records?They are not actually deleted!! Please tell me how to fix all of this! It is a nightmare. All we are trying to do is to move our Juvenile items from one copy location to another and change all the call numbers. What gives!? Thanks! Jennifer -- Jennifer Walz, MLS - Head of ILS Crazy Kinlaw Library - *Asbury University* One Macklem Drive, Wilmore, KY 40390 859-858-3511 ext. 2269 jlw...@asbury.edu -- Kathy Lussier Project Coordinator Massachusetts Library Network Cooperative (508) 343-0128 kluss...@masslnc.org Twitter: http://www.twitter.com/kmlussier
Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
I just took a look at a test system running Debian Wheezy with EG 2.8.2 and
Opensrf 2.4.1, same issue, each page load of a record detail page leaks 5 file
descriptors, that show up when doing a “lsof | grep “\sock\” | wc –l” before
and after the request.
So the steps to test it are.
1. Run “lsof |grep \sock\ | wc –l” to see how many orphan FD there
currently are.
2. Load a record detail page to trigger the added_content connections
back to the local host. http://egcatalog/eg/opac/record/10
3. Run “lsof |grep \sock\ | wc –l” to see if the number increased.
I don’t have a non openvz based system to test on right now. I would love to
hear if anyone else sees this.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 2:06 PM
To: Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
I found this page that seems to say that a close is always needed after a
shutdown of a socket to free the FD.
http://www.perlmonks.org/?node=108244
I’ll look at my other test systems and see if I see the same issue, but haven’t
noticed it because of the low number of requests.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 1:31 PM
To: Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
This is what strace shos me.
[pid 14793] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP unfinished ...
[pid 14793] ... socket resumed ) = 83
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] fcntl(83, F_SETFD, FD_CLOEXEC unfinished ...
[pid 14793] ... fcntl resumed ) = 0
[pid 14793] connect(83, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(192.168.46.32)}, 16 unfinished ...
[pid 14793] ... connect resumed ) = 0
[pid 14793] write(83, HEAD /opac/extras/ac/summary/html/r/1001
HTTP/1.1\r\nConnection: close\r\nHost: virt-egapp2.larl.org\r\n\r\n, 100
unfinished ...
[pid 14793] ... write resumed ) = 100
[pid 14793] read(83, unfinished ...
[pid 14793] ... read resumed HTTP/1.1 404 Not Found\r\nDate: Thu, 23 Jul
2015 03:16:49 GMT\r\nServer: Apache/2.4.10 (Debian)\r\nConnection:
close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n, 1024) = 159
[pid 14793] shutdown(83, SHUT_RDWR unfinished ...
[pid 14793] ... shutdown resumed )= 0
After this point FD 83 never shows up again in the strace log, but it does show
up in the lsof –p pid display as shown before. I’m wondering if that is
because no close for FD 83 is called? I’ve read that sometimes the shutdown()
implementation includes the close, and sometimes it does not. I’m trying to
figure out if the shutdown at the end of EGCatLoader/Record.pm includes a
close.. or if the implementation changed with the versions of the perl libs
that Jessie has.
I’m also wondering if I’m way off base or not.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 10:03 AM
To: Evergreen Development Discussion List; Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
Hello Mike,
Lsof –n –P –p pid doesn’t give any new info about those connections.
ot@virt-egapp2:/openils/var/templates# lsof -n -P -p 5684
COMMANDPIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
/usr/sbin 5684 opensrf cwdDIR 0,45 4096 34071974 /
/usr/sbin 5684 opensrf rtdDIR 0,45 4096 34071974 /
/usr/sbin 5684 opensrf txtREG 0,45 654136 34947316
/usr/sbin/apache2
/usr/sbin 5684 opensrf memREG 253,2 35374581
/lib/x86_64-linux-gnu/libnss_dns-2.19.so (path dev=0,45)
/usr/sbin 5684 opensrf memREG 253,2 39379859
/usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Hash/Util/Util.so (path dev=0,45)
/usr/sbin 5684 opensrf memREG 0,50 66964636
(deleted)/dev/zero (stat: No such file or directory)
SNIP
/usr/sbin 5684 opensrf 35u sock0,6 0t0 67405034 can't
identify protocol
/usr/sbin 5684 opensrf 36u sock0,6 0t0 67405037 can't
identify protocol
/usr/sbin 5684 opensrf 37u sock0,6 0t0
Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
This is what strace shows me.
[pid 14793] socket(PF_INET, SOCK_STREAM, IPPROTO_TCP unfinished ...
[pid 14793] ... socket resumed ) = 83
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] ioctl(83, SNDCTL_TMR_TIMEBASE or SNDRV_TIMER_IOCTL_NEXT_DEVICE or
TCGETS unfinished ...
[pid 14793] ... ioctl resumed , 0x7fffb83df850) = -1 EINVAL (Invalid argument)
[pid 14793] lseek(83, 0, SEEK_CUR unfinished ...
[pid 14793] ... lseek resumed ) = -1 ESPIPE (Illegal seek)
[pid 14793] fcntl(83, F_SETFD, FD_CLOEXEC unfinished ...
[pid 14793] ... fcntl resumed ) = 0
[pid 14793] connect(83, {sa_family=AF_INET, sin_port=htons(80),
sin_addr=inet_addr(192.168.46.32)}, 16 unfinished ...
[pid 14793] ... connect resumed ) = 0
[pid 14793] write(83, HEAD /opac/extras/ac/summary/html/r/1001
HTTP/1.1\r\nConnection: close\r\nHost: virt-egapp2.larl.org\r\n\r\n, 100
unfinished ...
[pid 14793] ... write resumed ) = 100
[pid 14793] read(83, unfinished ...
[pid 14793] ... read resumed HTTP/1.1 404 Not Found\r\nDate: Thu, 23 Jul
2015 03:16:49 GMT\r\nServer: Apache/2.4.10 (Debian)\r\nConnection:
close\r\nContent-Type: text/html; charset=iso-8859-1\r\n\r\n, 1024) = 159
[pid 14793] shutdown(83, SHUT_RDWR unfinished ...
[pid 14793] ... shutdown resumed )= 0
After this point FD 83 never shows up again in the strace log, but it does show
up in the lsof –p pid display as shown before. I’m wondering if that is
because no close for FD 83 is called? I’ve read that sometimes the shutdown()
implementation includes the close, and sometimes it does not. I’m trying to
figure out if the shutdown at the end of EGCatLoader/Record.pm includes a
close.. or if the implementation changed with the versions of the perl libs
that Jessie has.
I’m also wondering if I’m way off base or not.
Josh
From: Open-ils-general
[mailto:open-ils-general-boun...@list.georgialibraries.org] On Behalf Of Josh
Stompro
Sent: Thursday, July 23, 2015 10:03 AM
To: Evergreen Development Discussion List; Evergreen Discussion Group
Subject: Re: [OPEN-ILS-GENERAL] [OPEN-ILS-DEV] Apache leaking sockets/FD
Hello Mike,
Lsof –n –P –p pid doesn’t give any new info about those connections.
ot@virt-egapp2:/openils/var/templates# lsof -n -P -p 5684
COMMANDPIDUSER FD TYPE DEVICE SIZE/OFF NODE NAME
/usr/sbin 5684 opensrf cwdDIR 0,45 4096 34071974 /
/usr/sbin 5684 opensrf rtdDIR 0,45 4096 34071974 /
/usr/sbin 5684 opensrf txtREG 0,45 654136 34947316
/usr/sbin/apache2
/usr/sbin 5684 opensrf memREG 253,2 35374581
/lib/x86_64-linux-gnu/libnss_dns-2.19.so (path dev=0,45)
/usr/sbin 5684 opensrf memREG 253,2 39379859
/usr/lib/x86_64-linux-gnu/perl/5.20.2/auto/Hash/Util/Util.so (path dev=0,45)
/usr/sbin 5684 opensrf memREG 0,50 66964636
(deleted)/dev/zero (stat: No such file or directory)
SNIP
/usr/sbin 5684 opensrf 35u sock0,6 0t0 67405034 can't
identify protocol
/usr/sbin 5684 opensrf 36u sock0,6 0t0 67405037 can't
identify protocol
/usr/sbin 5684 opensrf 37u sock0,6 0t0 67405040 can't
identify protocol
/usr/sbin 5684 opensrf 38u sock0,6 0t0 67405043 can't
identify protocol
/usr/sbin 5684 opensrf 39u sock0,6 0t0 67405046 can't
identify protocol
/usr/sbin 5684 opensrf 40u sock0,6 0t0 67689829 can't
identify protocol
/usr/sbin 5684 opensrf 41u sock0,6 0t0 67689832 can't
identify protocol
/usr/sbin 5684 opensrf 42u sock0,6 0t0 67689835 can't
identify protocol
/usr/sbin 5684 opensrf 43u sock0,6 0t0 67689838 can't
identify protocol
/usr/sbin 5684 opensrf 44u sock0,6 0t0 67689841 can't
identify protocol
From using strace it looks like the problem connections are from apache trying
to load the various added content types, the connections get shutdown but the
FD for the socket never gets closed.
https://github.com/evergreen-library-system/Evergreen/blob/6bb8ea5599d39d41d623d1891b3c509c4e439178/Open-ILS/src/perlmods/lib/OpenILS/WWW/EGCatLoader/Record.pm#L577
I’ll post more info when I get a chance, time to take the kids to the park
before we all go stir crazy ;-)
Josh
From: Open-ils-dev [mailto:open-ils-dev-boun...@list.georgialibraries.org] On
Behalf Of Mike Rylander
Sent: Thursday, July 23, 2015 8:02 AM
To: Evergreen Discussion Group
Cc:
open-ils-...@list.georgialibraries.orgmailto:open-ils-...@list.georgialibraries.org
Subject: Re: [OPEN-ILS-DEV] [OPEN-ILS-GENERAL] Apache
