Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests?
Thanks Josh! I will have to take a look at this when I get a chance. It sounds like you are doing some good stuff here. Hopefully those that have chosen to integrate the carousel can also benefit. I will merge the branch when I have the time to devote to it. -Blake- Conducting Magic MOBIUS On 10/16/2018 9:57 AM, Josh Stompro wrote: Blake, I found that duplicate records in the title bucket will break the carousel with my changes. I think that may be because open-ils.search.biblio.record.mods_slim.retrieve must not return duplicates. Or maybe it is because I create a hash based off the bibid to store the ordering? That isn’t a deal breaker for us, since I didn’t intend to have duplicates in the lists, they were just there by mistake. But it probably shouldn’t break because of that. So I’ll look into that. I also was happy to find that there are options to allow wrapping of the carousel, so once you get to the end it starts again at the beginning, along with autoscrolling to new items get shown ever x seconds. Adding the following to the .jcarousel config enables those wrap: ‘both’, auto: 9, The next step to make it load even faster would be to cache the open-ils.search.biblio.record.mods_slim.retrieve results somehow. For our new arrival list, it takes our system 3219ms to generate that for 200 titles. Josh Stompro - LARL IT Director *From:*Open-ils-dev *On Behalf Of *Josh Stompro *Sent:* Saturday, October 13, 2018 4:50 PM *To:* Evergreen Development Discussion List *Subject:* Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests? Blake, I sent you a pull request with the changes. https://github.com/mcoia/mobius_evergreen/pull/3/commits/9e6de7dd80f182488012b64fa8311162ed433e6b I did get the ordering to work, but I’m not sure I did it in the most efficient manner. Josh *From:*Open-ils-dev <mailto:open-ils-dev-boun...@list.georgialibraries.org>> *On Behalf Of *Blake Henderson *Sent:* Saturday, October 13, 2018 4:11 PM *To:* open-ils-dev@list.georgialibraries.org <mailto:open-ils-dev@list.georgialibraries.org> *Subject:* Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests? Josh, Great detective work! The culprit is likely the bbag.js code as you have mentioned. It sounds like you have made a major improvement. Really, the whole thing needs to be moved into action_trigger and library settings and eventually baked into Evergreen. Share your improvements as you make them! I could imagine a path forward coding a custom JS carousel (instead of using jcarousel) using some of the great Angular things. Coupled with a server side component in the form of an additional gateway-v1 listener to provide the JSON data as needed. Right now, the bbag.js code downloads the entire bookbag and injects the ID's into a hidden DOM element. Perhaps it's time to open that can of worms again? Is there a LP bug for this? -Blake- Conducting Magic MOBIUS On 10/13/2018 10:05 AM, Josh Stompro wrote: Hello, I’m moving this over to the dev list. Using the batch method to flesh the titles from the bookbags does seem to offer significant performance improvements for us. (Sending all bib id’s in one array to open-ils.search.biblio.record.mods_slim.retrieve) Measured in chrome 69.0.3497.100 Before: (hot cache) 19 second page load, 480 requests, 757kb transferred. After: (hot cache) 6 second page load, 76 requests, 434kb transferred. Difference: 68% reduction in page load time, 84% reduction in requests, 42% reduction in transfer amount. I haven’t dealt with getting the correct order yet, open-ils.search.biblio.record.mods_slim.retrieve returns results in bibid order. Josh *From:* Open-ils-general <mailto:open-ils-general-boun...@list.georgialibraries.org> *On Behalf Of *Josh Stompro *Sent:* Friday, October 12, 2018 2:31 PM *To:* Evergreen Discussion Group <mailto:open-ils-gene...@list.georgialibraries.org> *Subject:* [OPEN-ILS-GENERAL] Mobius Carousel - reducing number of requests? Hello, We have been using the carousel from Mobius for a while now. And I just noticed how resource intensive it seems to be. Here are the slides from the presentation that Bake created about it. http://slides.mobiusconsortium.org/blake/bookcarousel/#/1 And the code https://github.com/mcoia/mobius_evergreen/tree/master/bookbag_update We have ~ 30 catalog machines that are set to reset every 5 minutes of idle time (since people don’t always log themselves out.) When I watch our catalog page load which currently has 3 sliders, I see 483 requests made from the browser. ~450 of those requests are from the carousel, since it request each book bag, and then request info for each bookbag item. And then request images for the visible items. So our 30 catalog machines generate
Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests?
Blake, I found that duplicate records in the title bucket will break the carousel with my changes. I think that may be because open-ils.search.biblio.record.mods_slim.retrieve must not return duplicates. Or maybe it is because I create a hash based off the bibid to store the ordering? That isn't a deal breaker for us, since I didn't intend to have duplicates in the lists, they were just there by mistake. But it probably shouldn't break because of that. So I'll look into that. I also was happy to find that there are options to allow wrapping of the carousel, so once you get to the end it starts again at the beginning, along with autoscrolling to new items get shown ever x seconds. Adding the following to the .jcarousel config enables those wrap: 'both', auto: 9, The next step to make it load even faster would be to cache the open-ils.search.biblio.record.mods_slim.retrieve results somehow. For our new arrival list, it takes our system 3219ms to generate that for 200 titles. Josh Stompro - LARL IT Director From: Open-ils-dev On Behalf Of Josh Stompro Sent: Saturday, October 13, 2018 4:50 PM To: Evergreen Development Discussion List Subject: Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests? Blake, I sent you a pull request with the changes. https://github.com/mcoia/mobius_evergreen/pull/3/commits/9e6de7dd80f182488012b64fa8311162ed433e6b I did get the ordering to work, but I'm not sure I did it in the most efficient manner. Josh From: Open-ils-dev mailto:open-ils-dev-boun...@list.georgialibraries.org>> On Behalf Of Blake Henderson Sent: Saturday, October 13, 2018 4:11 PM To: open-ils-dev@list.georgialibraries.org<mailto:open-ils-dev@list.georgialibraries.org> Subject: Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests? Josh, Great detective work! The culprit is likely the bbag.js code as you have mentioned. It sounds like you have made a major improvement. Really, the whole thing needs to be moved into action_trigger and library settings and eventually baked into Evergreen. Share your improvements as you make them! I could imagine a path forward coding a custom JS carousel (instead of using jcarousel) using some of the great Angular things. Coupled with a server side component in the form of an additional gateway-v1 listener to provide the JSON data as needed. Right now, the bbag.js code downloads the entire bookbag and injects the ID's into a hidden DOM element. Perhaps it's time to open that can of worms again? Is there a LP bug for this? -Blake- Conducting Magic MOBIUS On 10/13/2018 10:05 AM, Josh Stompro wrote: Hello, I'm moving this over to the dev list. Using the batch method to flesh the titles from the bookbags does seem to offer significant performance improvements for us. (Sending all bib id's in one array to open-ils.search.biblio.record.mods_slim.retrieve) Measured in chrome 69.0.3497.100 Before: (hot cache) 19 second page load, 480 requests, 757kb transferred. After: (hot cache) 6 second page load, 76 requests, 434kb transferred. Difference: 68% reduction in page load time, 84% reduction in requests, 42% reduction in transfer amount. I haven't dealt with getting the correct order yet, open-ils.search.biblio.record.mods_slim.retrieve returns results in bibid order. Josh From: Open-ils-general <mailto:open-ils-general-boun...@list.georgialibraries.org> On Behalf Of Josh Stompro Sent: Friday, October 12, 2018 2:31 PM To: Evergreen Discussion Group <mailto:open-ils-gene...@list.georgialibraries.org> Subject: [OPEN-ILS-GENERAL] Mobius Carousel - reducing number of requests? Hello, We have been using the carousel from Mobius for a while now. And I just noticed how resource intensive it seems to be. Here are the slides from the presentation that Bake created about it. http://slides.mobiusconsortium.org/blake/bookcarousel/#/1 And the code https://github.com/mcoia/mobius_evergreen/tree/master/bookbag_update We have ~ 30 catalog machines that are set to reset every 5 minutes of idle time (since people don't always log themselves out.) When I watch our catalog page load which currently has 3 sliders, I see 483 requests made from the browser. ~450 of those requests are from the carousel, since it request each book bag, and then request info for each bookbag item. And then request images for the visible items. So our 30 catalog machines generate 162000 requests an hour. I'm wondering if anyone has tried pre-generating the bookbag json data on a schedule so the web page can do just one request for the titles? It also seems like the only info that is used is the title, while 1-2kb of data gets returned about each item. So a lot of unused data gets moved around. The script that generates the static data file could do things like randomize the order of the items, so a static bookbag list would show different titles whenever it is re-genera
Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests?
Blake, I sent you a pull request with the changes. https://github.com/mcoia/mobius_evergreen/pull/3/commits/9e6de7dd80f182488012b64fa8311162ed433e6b I did get the ordering to work, but I'm not sure I did it in the most efficient manner. Josh From: Open-ils-dev On Behalf Of Blake Henderson Sent: Saturday, October 13, 2018 4:11 PM To: open-ils-dev@list.georgialibraries.org Subject: Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests? Josh, Great detective work! The culprit is likely the bbag.js code as you have mentioned. It sounds like you have made a major improvement. Really, the whole thing needs to be moved into action_trigger and library settings and eventually baked into Evergreen. Share your improvements as you make them! I could imagine a path forward coding a custom JS carousel (instead of using jcarousel) using some of the great Angular things. Coupled with a server side component in the form of an additional gateway-v1 listener to provide the JSON data as needed. Right now, the bbag.js code downloads the entire bookbag and injects the ID's into a hidden DOM element. Perhaps it's time to open that can of worms again? Is there a LP bug for this? -Blake- Conducting Magic MOBIUS On 10/13/2018 10:05 AM, Josh Stompro wrote: Hello, I'm moving this over to the dev list. Using the batch method to flesh the titles from the bookbags does seem to offer significant performance improvements for us. (Sending all bib id's in one array to open-ils.search.biblio.record.mods_slim.retrieve) Measured in chrome 69.0.3497.100 Before: (hot cache) 19 second page load, 480 requests, 757kb transferred. After: (hot cache) 6 second page load, 76 requests, 434kb transferred. Difference: 68% reduction in page load time, 84% reduction in requests, 42% reduction in transfer amount. I haven't dealt with getting the correct order yet, open-ils.search.biblio.record.mods_slim.retrieve returns results in bibid order. Josh From: Open-ils-general <mailto:open-ils-general-boun...@list.georgialibraries.org> On Behalf Of Josh Stompro Sent: Friday, October 12, 2018 2:31 PM To: Evergreen Discussion Group <mailto:open-ils-gene...@list.georgialibraries.org> Subject: [OPEN-ILS-GENERAL] Mobius Carousel - reducing number of requests? Hello, We have been using the carousel from Mobius for a while now. And I just noticed how resource intensive it seems to be. Here are the slides from the presentation that Bake created about it. http://slides.mobiusconsortium.org/blake/bookcarousel/#/1 And the code https://github.com/mcoia/mobius_evergreen/tree/master/bookbag_update We have ~ 30 catalog machines that are set to reset every 5 minutes of idle time (since people don't always log themselves out.) When I watch our catalog page load which currently has 3 sliders, I see 483 requests made from the browser. ~450 of those requests are from the carousel, since it request each book bag, and then request info for each bookbag item. And then request images for the visible items. So our 30 catalog machines generate 162000 requests an hour. I'm wondering if anyone has tried pre-generating the bookbag json data on a schedule so the web page can do just one request for the titles? It also seems like the only info that is used is the title, while 1-2kb of data gets returned about each item. So a lot of unused data gets moved around. The script that generates the static data file could do things like randomize the order of the items, so a static bookbag list would show different titles whenever it is re-generated. It looks like open-ils.search.biblio.record.mods_slim.retrieve can accept an array of ID's, so that seems like it would reduce the number of requests, so that may be another way to reduce the number of requests. Thanks Josh Lake Agassiz Regional Library - Moorhead MN larl.org Josh Stompro | Office 218.233.3757 EXT-139 LARL IT Director | Cell 218.790.2110
Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests?
Josh, Great detective work! The culprit is likely the bbag.js code as you have mentioned. It sounds like you have made a major improvement. Really, the whole thing needs to be moved into action_trigger and library settings and eventually baked into Evergreen. Share your improvements as you make them! I could imagine a path forward coding a custom JS carousel (instead of using jcarousel) using some of the great Angular things. Coupled with a server side component in the form of an additional gateway-v1 listener to provide the JSON data as needed. Right now, the bbag.js code downloads the entire bookbag and injects the ID's into a hidden DOM element. Perhaps it's time to open that can of worms again? Is there a LP bug for this? -Blake- Conducting Magic MOBIUS On 10/13/2018 10:05 AM, Josh Stompro wrote: Hello, I’m moving this over to the dev list. Using the batch method to flesh the titles from the bookbags does seem to offer significant performance improvements for us. (Sending all bib id’s in one array to open-ils.search.biblio.record.mods_slim.retrieve) Measured in chrome 69.0.3497.100 Before: (hot cache) 19 second page load, 480 requests, 757kb transferred. After: (hot cache) 6 second page load, 76 requests, 434kb transferred. Difference: 68% reduction in page load time, 84% reduction in requests, 42% reduction in transfer amount. I haven’t dealt with getting the correct order yet, open-ils.search.biblio.record.mods_slim.retrieve returns results in bibid order. Josh *From:* Open-ils-general *On Behalf Of *Josh Stompro *Sent:* Friday, October 12, 2018 2:31 PM *To:* Evergreen Discussion Group *Subject:* [OPEN-ILS-GENERAL] Mobius Carousel - reducing number of requests? Hello, We have been using the carousel from Mobius for a while now. And I just noticed how resource intensive it seems to be. Here are the slides from the presentation that Bake created about it. http://slides.mobiusconsortium.org/blake/bookcarousel/#/1 And the code https://github.com/mcoia/mobius_evergreen/tree/master/bookbag_update We have ~ 30 catalog machines that are set to reset every 5 minutes of idle time (since people don’t always log themselves out.) When I watch our catalog page load which currently has 3 sliders, I see 483 requests made from the browser. ~450 of those requests are from the carousel, since it request each book bag, and then request info for each bookbag item. And then request images for the visible items. So our 30 catalog machines generate 162000 requests an hour. I’m wondering if anyone has tried pre-generating the bookbag json data on a schedule so the web page can do just one request for the titles? It also seems like the only info that is used is the title, while 1-2kb of data gets returned about each item. So a lot of unused data gets moved around. The script that generates the static data file could do things like randomize the order of the items, so a static bookbag list would show different titles whenever it is re-generated. It looks like open-ils.search.biblio.record.mods_slim.retrieve can accept an array of ID’s, so that seems like it would reduce the number of requests, so that may be another way to reduce the number of requests. Thanks Josh Lake Agassiz Regional Library - Moorhead MN larl.org Josh Stompro | Office 218.233.3757 EXT-139 LARL IT Director | Cell 218.790.2110
Re: [OPEN-ILS-DEV] Mobius Carousel - reducing number of requests?
Hello, I'm moving this over to the dev list. Using the batch method to flesh the titles from the bookbags does seem to offer significant performance improvements for us. (Sending all bib id's in one array to open-ils.search.biblio.record.mods_slim.retrieve) Measured in chrome 69.0.3497.100 Before: (hot cache) 19 second page load, 480 requests, 757kb transferred. After: (hot cache) 6 second page load, 76 requests, 434kb transferred. Difference: 68% reduction in page load time, 84% reduction in requests, 42% reduction in transfer amount. I haven't dealt with getting the correct order yet, open-ils.search.biblio.record.mods_slim.retrieve returns results in bibid order. Josh From: Open-ils-general On Behalf Of Josh Stompro Sent: Friday, October 12, 2018 2:31 PM To: Evergreen Discussion Group Subject: [OPEN-ILS-GENERAL] Mobius Carousel - reducing number of requests? Hello, We have been using the carousel from Mobius for a while now. And I just noticed how resource intensive it seems to be. Here are the slides from the presentation that Bake created about it. http://slides.mobiusconsortium.org/blake/bookcarousel/#/1 And the code https://github.com/mcoia/mobius_evergreen/tree/master/bookbag_update We have ~ 30 catalog machines that are set to reset every 5 minutes of idle time (since people don't always log themselves out.) When I watch our catalog page load which currently has 3 sliders, I see 483 requests made from the browser. ~450 of those requests are from the carousel, since it request each book bag, and then request info for each bookbag item. And then request images for the visible items. So our 30 catalog machines generate 162000 requests an hour. I'm wondering if anyone has tried pre-generating the bookbag json data on a schedule so the web page can do just one request for the titles? It also seems like the only info that is used is the title, while 1-2kb of data gets returned about each item. So a lot of unused data gets moved around. The script that generates the static data file could do things like randomize the order of the items, so a static bookbag list would show different titles whenever it is re-generated. It looks like open-ils.search.biblio.record.mods_slim.retrieve can accept an array of ID's, so that seems like it would reduce the number of requests, so that may be another way to reduce the number of requests. Thanks Josh Lake Agassiz Regional Library - Moorhead MN larl.org Josh Stompro | Office 218.233.3757 EXT-139 LARL IT Director | Cell 218.790.2110
