[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Victor Grousset/tuxayo changed: What|Removed |Added Resolution|--- |FIXED Status|Pushed to oldstable |RESOLVED CC||vic...@tuxayo.net --- Comment #54 from Victor Grousset/tuxayo --- Not backported to oldoldstable (19.05.x). Feel free to ask if it's needed. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Aleisha Amohia changed: What|Removed |Added Version(s)|20.11.00, 20.05.06 |20.11.00, 20.05.06, released in||19.11.12 Status|Pushed to stable|Pushed to oldstable --- Comment #53 from Aleisha Amohia --- backported to 19.11.x for 19.11.12 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Aleisha Amohia changed: What|Removed |Added Status|Pushed to master|Pushed to stable CC||alei...@catalyst.net.nz -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #52 from David Cook --- Using CGI::Emulate::PSGI::Streaming (version 1.0.1 since 1.0.0 is buggy) and a custom Plack::Component I created based on Plack::App::WrapCGI, I was able to stream 65MB of bib records from /cgi-bin/koha/tools/export.pl in 13 minutes using koha-testing-docker (and an external database on a different server). I could explore a solution using that tech, although I think that I'm still more inclined to moving forward with Bug 26791, as replacing export.pl provides a good example for replacing CGI scripts with Mojolicious controllers. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Lucas Gass changed: What|Removed |Added CC||lu...@bywatersolutions.com Version(s)|20.11.00|20.11.00, 20.05.06 released in|| --- Comment #51 from Lucas Gass --- backported to 20.05.x for 20.05.06 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #50 from David Cook --- Hurray! I won't have work time to spend on Bug 26791, but sometime soon I could share the work I did to get it working in Mojolicious. It'll need a fair bit more work to be production ready, but it could be the path forward for fixing all these proxy exceptions. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #49 from Jonathan Druart --- Pushed to master for 20.11, thanks to everybody involved! -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Jonathan Druart changed: What|Removed |Added Version(s)||20.11.00 released in|| Status|Passed QA |Pushed to master -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Katrin Fischer changed: What|Removed |Added Attachment #112224|0 |1 is obsolete|| --- Comment #48 from Katrin Fischer --- Created attachment 112448 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112448&action=edit Bug 8437: Exclude export.pl from plack When attempting to download large files from Koha plack can timeout Excluding the script from plack is a simple fix until we have a more permanent fix for this issue. To test: 1 - Try to export your entire DB from Tools->Export 2 - If big enough, it fails 3 - Apply patch, copy changes to /etc/koha/apache-shared-intranet-plack.conf 4 - Restart all the things 5 - Repeat export, it succeeds Signed-off-by: David Cook Signed-off-by: Katrin Fischer -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Katrin Fischer changed: What|Removed |Added Patch complexity|--- |Trivial patch Status|Signed Off |Passed QA -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Katrin Fischer changed: What|Removed |Added QA Contact||katrin.fisc...@bsz-bw.de -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Nick Clemens changed: What|Removed |Added Assignee|gmcha...@gmail.com |n...@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #47 from David Cook --- Side note: It's harder to make this work in Mojolicious than I thought it would, but only because Mojolicious is non-blocking by design. I think I should be able to create something that should work quite well in the end... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 David Cook changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=26791 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 David Cook changed: What|Removed |Added Attachment #109632|0 |1 is obsolete|| --- Comment #46 from David Cook --- Created attachment 112224 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=112224&action=edit Bug 8437: Exclude export.pl from plack When attempting to download large files from Koha plack can timeout Excluding the script from plack is a simple fix until we have a more permanent fix for this issue. To test: 1 - Try to export your entire DB from Tools->Export 2 - If big enough, it fails 3 - Apply patch, copy changes to /etc/koha/apache-shared-intranet-plack.conf 4 - Restart all the things 5 - Repeat export, it succeeds Signed-off-by: David Cook -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 David Cook changed: What|Removed |Added Status|Needs Signoff |Signed Off -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #45 from David Cook --- (In reply to David Cook from comment #44) > (In reply to Nick Clemens from comment #30) > > Created attachment 109632 [details] [review] [review] > > Bug 8437: Exclude export.pl from plack > > > > When attempting to download large files from Koha plack can timeout > > > > Excluding the script from plack is a simple fix until we have a more > > permanent fix for this > > issue. > > > > To test: > > 1 - Try to export your entire DB from Tools->Export > > 2 - If big enough, it fails > > 3 - Apply patch, copy changes to /etc/koha/apache-shared-intranet-plack.conf > > 4 - Restart all the things > > 5 - Repeat export, it succeeds > > My test plan: > 0. Using koha-testing-docker > 1. cp debian/templates/apache-shared-intranet-plack.conf > /etc/koha/apache-shared-intranet-plack.conf > 2. enable "backup_db_via_tools" in /etc/koha/sites/kohadev/koha-conf.xml > 3. echo 'flush_all' | nc -q 1 memcached 11211 > 4. service apache2 restart > 5. koha-plack --restart kohadev > 6. Go to http://localhost:8081/cgi-bin/koha/tools/export.pl > 7. Note "Unfortunately, no backups are available." > > So that's not going to work... but I have another idea. My test plan: 0. Using koha-testing-docker 1. cp debian/templates/apache-shared-intranet-plack.conf /etc/koha/apache-shared-intranet-plack.conf 2. Point koha-testing-docker at a database with a large number of biblios (e.g. 40,000 biblios) 3. echo 'flush_all' | nc -q 1 memcached 11211 4. restart_all 5. Go to http://localhost:8081/cgi-bin/koha/tools/export.pl 6. Click "Don't export items" 7. Click "Export bibliographic records" 8. Note after about 5 seconds that a download for koha.mrc starts. Perfect. Cheers, Nick. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #44 from David Cook --- (In reply to Nick Clemens from comment #30) > Created attachment 109632 [details] [review] > Bug 8437: Exclude export.pl from plack > > When attempting to download large files from Koha plack can timeout > > Excluding the script from plack is a simple fix until we have a more > permanent fix for this > issue. > > To test: > 1 - Try to export your entire DB from Tools->Export > 2 - If big enough, it fails > 3 - Apply patch, copy changes to /etc/koha/apache-shared-intranet-plack.conf > 4 - Restart all the things > 5 - Repeat export, it succeeds My test plan: 0. Using koha-testing-docker 1. cp debian/templates/apache-shared-intranet-plack.conf /etc/koha/apache-shared-intranet-plack.conf 2. enable "backup_db_via_tools" in /etc/koha/sites/kohadev/koha-conf.xml 3. echo 'flush_all' | nc -q 1 memcached 11211 4. service apache2 restart 5. koha-plack --restart kohadev 6. Go to http://localhost:8081/cgi-bin/koha/tools/export.pl 7. Note "Unfortunately, no backups are available." So that's not going to work... but I have another idea. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 David Cook changed: What|Removed |Added Attachment #11177|0 |1 is obsolete|| --- Comment #43 from David Cook --- Comment on attachment 11177 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=11177 Bug 8437: variable scoping for tools/export.pl I think that this one is obsolete at this point. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #42 from David Cook --- In any case, I think we do need to solve the immediate problem ASAP, so I'm going to go ahead with Nick's patch. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #41 from David Cook --- I do have another idea... based on what Dobrica said at https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437#c7 When using Plack, we could re-direct away from export.pl to a URL where we have mounted a PSGI application (e.g. we could use Mojo::Serve::PSGI like we do with the REST API). This would also give us a path forward to transition away from CGI to Plack. The tools/export.pl file is actually pretty short, and the majority of the logic is in Perl modules (the "model" in MVC terms). The .pl files are basically just overloaded controllers (in MVC terms). I don't have time today, but maybe I would be willing to work on a solution for this. We could keep the existing template file (or "view" in MVC terms), and just provide an alternative controller. We could deprecate tools/export.pl and focus in favour on a PSGI controller. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #40 from David Cook --- I can't figure out why the problem with CGI::Emulate::PSGI::Streaming is happening, but I've reported the bug. It's unusable until it's fixed though. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #39 from David Cook --- That truncated response is 32489 bytes long... I'm guessing the issue is with CGI::Parse::PSGI::Streaming... According to the tools/export.pl, I'm writing a body of 33441 characters... Looking at CGI::Parse::PSGI::Streaming, I'm only seeing 32768 characters (which includes the HTTP headers and the HTTP body), and 32768 bytes is the same as 32KB... CGI::Parse::PSGI::Streaming::Handle shows the full response in the buffer and says it is 33750 characters long... Ah, I think it's a bug in CGI::Parse::PSGI::Streaming::Handle. For some reason, the author takes the data printed to STDOUT, creates an in-memory file handle stored in a scalar variable, and prints the contents of that "STDOUT" to the in-memory file handle, and that's when it is truncated from 33750 characters to 32768 characters... This is the line in CGI::Parse::PSGI::Streaming::Handle which seemingly breaks things: print {$self->{fh}} substr($buf, $offset, $len); The weird thing is that I'm trying to reproduce the problem in a more stripped back program and I can't do it. It just works... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #38 from David Cook --- For some reason, my HTML response is being cut off here:
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #37 from David Cook --- (In reply to David Cook from comment #36) > Hmm the difference seems to be in the HTML... Ahh my custom handler isn't sending a complete HTML response for some reason. I think a Perl error. I think I must not be setting up the CGI env quite right. But so close... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #36 from David Cook --- Hmm the difference seems to be in the HTML... -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #35 from David Cook --- One alternative could be to use https://metacpan.org/pod/Plack::App::CGIBin::Streaming. Another would be a custom handler using CGI::Emulate::PSGI::Streaming: cpan2deb CGI::Emulate::PSGI::Streaming apt install ./libcgi-emulate-psgi-streaming-perl_1.0.0-1_all.deb I'm exploring the second option now (via a custom Koha::Plack::App::StreamCGI module)... It seems like the custom handler is returning an empty koha.mrc file after about 10 seconds, and I've only processed about 6000 out of 4 records... Of course, as soon as I say that, suddenly it seems to start working without me doing anything different. Although I notice that usually my custom handler doesn't load some Javascript or CSS quite right... and I hit the "Export bibliographic records" button from a different window... But still... progress... I've streamed 10,000 bib records (10MB) out of Starman via Apache. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #34 from David Cook --- (In reply to David Cook from comment #33) > As Dobrica mentioned, it does appear that Starman is buffering the whole > response before sending it. I am guessing this is due to our usage of > Plack::App::CGIBin... but I'll investigate further. If I understand correctly, this will never work for us, so long as we're using Plack::App::CGIBin to serve export.pl. At https://metacpan.org/release/Plack/source/lib/Plack/App/CGIBin.pm#L47, Plack::App::CGIBin uses Plack::App::WrapCGI. Plack::App::WrapCGI uses CGI::Emulate::PSGI at https://metacpan.org/source/Plack::App::WrapCGI#L87. CGI::Emulate::PSGI writes the response to a temporary file before returning the response (as per https://metacpan.org/release/CGI-Emulate-PSGI/source/lib/CGI/Emulate/PSGI.pm#L19). If you look at CGI::Emulate::PSGI::Streaming (https://metacpan.org/release/CGI-Emulate-PSGI-Streaming/source/lib/CGI/Emulate/PSGI/Streaming.pm), you can see how that handler returns a closure rather than a tuple (as noted at http://www.catalystframework.org/calendar/2013/10). In order to do a streaming response, you need to leverage that $responder coderef in the closure. Of course, this should be doable. I have an idea germinating. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #33 from David Cook --- Ok, so I'm running a large bib export using export.pl, and I'm monitoring the Apache with "LogLevel Debug". I see a few things that might be interesting, but the key interesting thing is this: [Thu Oct 22 00:26:23.676618 2020] [reqtimeout:info] [pid 60205] [client 172.21.0.1:55460] AH01382: Request header read timeout As Dobrica mentioned, it does appear that Starman is buffering the whole response before sending it. I am guessing this is due to our usage of Plack::App::CGIBin... but I'll investigate further. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #32 from David Cook --- I've bumped into this one again, so I'm going to have a go at solving it for real. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #31 from David Cook --- Based on my analysis on Bug 26128, I'm wondering what kind of timeouts we're talking about here. In the bug I note above, the timeout used by CGI was effectively 2 times as long as the timeout used when proxying to Plack/Starman. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 David Cook changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=22417 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #30 from Nick Clemens --- Created attachment 109632 --> https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=109632&action=edit Bug 8437: Exclude export.pl from plack When attempting to download large files from Koha plack can timeout Excluding the script from plack is a simple fix until we have a more permanent fix for this issue. To test: 1 - Try to export your entire DB from Tools->Export 2 - If big enough, it fails 3 - Apply patch, copy changes to /etc/koha/apache-shared-intranet-plack.conf 4 - Restart all the things 5 - Repeat export, it succeeds -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Nick Clemens changed: What|Removed |Added Status|In Discussion |Needs Signoff -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Martin Renvoize changed: What|Removed |Added CC||martin.renvoize@ptfs-europe ||.com Severity|critical|normal -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org https://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #29 from David Cook --- (In reply to Kyle M Hall from comment #24) > Avoiding plack will 'fix' the script, but most likely cause everyone to stop > looking for an actual solution. That's my worry. That being said, it's > probably better to get it working in the near-term. I'm looking at export.pl and thinking how export.pl's job should be to authenticate the user, process user input about what to download, and then direct the user somewhere else better suited to serving files... This is kind of interesting although perhaps not relevant in our hybrid CGI/PSGI way of doing things, and it's old so doesn't take into account that X-Sendfile seems deprecated: http://www.catalystframework.org/calendar/2014/5 But it does make the point that trying to serve a large file from Perl is going to be slow. It might be worth looking at frameworks using Plack to see how they do it. For example send_file and _send_file in Dancer: https://metacpan.org/release/Dancer/source/lib/Dancer.pm. Or we could look at https://metacpan.org/release/Plack/source/lib/Plack/App/File.pm, although that looks like it uses PSGI magic to send the file too, which would be tricky in our hybrid environment. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #28 from David Cook --- (In reply to Liz Rea from comment #26) > What if we change dump strategy for large database, and use mydumper and > myloader? > > https://dotlayer.com/extremely-fast-mysql-backup-restore-using- > mydumpermyloader/ I like the sound of that, but I'm not sure that it's relevant to this issue, since we're just sending an already existing backup? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #27 from Liz Rea --- OPTIONALLY, of course. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Liz Rea changed: What|Removed |Added CC||wizzy...@gmail.com --- Comment #26 from Liz Rea --- What if we change dump strategy for large database, and use mydumper and myloader? https://dotlayer.com/extremely-fast-mysql-backup-restore-using-mydumpermyloader/ -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Jonathan Druart changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=22481 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Fridolin SOMERS changed: What|Removed |Added CC||fridolin.som...@biblibre.co ||m See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=17240 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #25 from David Cook --- (In reply to Kyle M Hall from comment #24) > Avoiding plack will 'fix' the script, but most likely cause everyone to stop > looking for an actual solution. That's my worry. That being said, it's > probably better to get it working in the near-term. This is my worry as well but I didn't want to say it and seem like a naysayer. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #24 from Kyle M Hall --- (In reply to Barton Chittenden from comment #23) > (In reply to Magnus Enger from comment #22) > > > Would it make sense to add export.pl until someone comes up with a proper > > solution? Or are there so few people downloading huge files that it is not > > worth it? > > I think it's probably worth it -- it's a one-line change, and it's easy to > back out when we finally fix it. Nick, Kyle, Larry, care to weigh in? That's up to the RM :) Avoiding plack will 'fix' the script, but most likely cause everyone to stop looking for an actual solution. That's my worry. That being said, it's probably better to get it working in the near-term. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 David Cook changed: What|Removed |Added CC||dc...@prosentient.com.au -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #23 from Barton Chittenden --- (In reply to Magnus Enger from comment #22) > Would it make sense to add export.pl until someone comes up with a proper > solution? Or are there so few people downloading huge files that it is not > worth it? I think it's probably worth it -- it's a one-line change, and it's easy to back out when we finally fix it. Nick, Kyle, Larry, care to weigh in? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Barton Chittenden changed: What|Removed |Added CC||k...@bywatersolutions.com, ||la...@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #22 from Magnus Enger --- (In reply to Barton Chittenden from comment #21) > ... the "!" means "Don't run this under plack". Thanks! There are already a number of pages listed here: http://git.koha-community.org/gitweb/?p=koha.git;a=blob;f=debian/templates/apache-shared-intranet-plack.conf Would it make sense to add export.pl until someone comes up with a proper solution? Or are there so few people downloading huge files that it is not worth it? -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #21 from Barton Chittenden --- (In reply to Magnus Enger from comment #19) > (In reply to Nick Clemens from comment #18) > > Hi Nancy, We have also worked around this by simply excluding that page from > > plack, it avoids the issue and allows exports to complete. > > Anyone got an example of how to do that? Is it something we should do in the > strandard config, until someone comes up with a better solution? > > I think the second patch from Dobrica should be a separate bug. Magnus, take a look at /etc/koha/apache-shared-intranet-plack.conf, you'll find lines that look like this: ProxyPass "/cgi-bin/koha/offline_circ/process_koc.pl" "!" ... the "!" means "Don't run this under plack". -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Magnus Enger changed: What|Removed |Added Status|NEW |In Discussion --- Comment #20 from Magnus Enger --- I have taken the liberty of moving the second patch from Dobrica to a dedicated bug: Bug 22143. I also signed it off. Obsoleting the patch here to avoid confusion. Also setting this bug to "In discussion", since there does not seem to be a clear plan for solving the main problem. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Magnus Enger changed: What|Removed |Added Attachment #11355|0 |1 is obsolete|| -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Magnus Enger changed: What|Removed |Added See Also||https://bugs.koha-community ||.org/bugzilla3/show_bug.cgi ||?id=22143 -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Magnus Enger changed: What|Removed |Added CC||mag...@libriotech.no --- Comment #19 from Magnus Enger --- (In reply to Nick Clemens from comment #18) > Hi Nancy, We have also worked around this by simply excluding that page from > plack, it avoids the issue and allows exports to complete. Anyone got an example of how to do that? Is it something we should do in the strandard config, until someone comes up with a better solution? I think the second patch from Dobrica should be a separate bug. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Josef Moravec changed: What|Removed |Added CC||josef.mora...@gmail.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #18 from Nick Clemens --- Hi Nancy, We have also worked around this by simply excluding that page from plack, it avoids the issue and allows exports to complete. -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #17 from Jonathan Druart --- (In reply to NancyK. from comment #16) > This bug is going to cause us a really big problem. We are doing weekly > exports and have plans for more. Please fix this soon. > Thanks > > NancyK This will not be fixed soon as nobody as a solution. You should consider using the CLI script misc/export_records.pl for your weekly export (with a cronjob even). -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 NancyK. changed: What|Removed |Added CC||nkee...@washoecounty.us --- Comment #16 from NancyK. --- This bug is going to cause us a really big problem. We are doing weekly exports and have plans for more. Please fix this soon. Thanks NancyK -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 --- Comment #15 from Barton Chittenden --- (In reply to Jonathan Druart from comment #14) > (In reply to Dobrica Pavlinusic from comment #7) > > Correct way to fix this is to issue redirect under plack in download_backup > > to some plack handled URL so we don't read whole file in memory. > > The solution seems to be here, but I don't understand what Dobrica meant. Dobrica, Can you clarify and/or submit a further patch? This issue is really causing issues for large libraries. Thanks, --Barton -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/
[Koha-bugs] [Bug 8437] Large database backups and large exports from export.pl fail under plack
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8437 Barton Chittenden changed: What|Removed |Added Summary|Plack and database backup |Large database backups and ||large exports from ||export.pl fail under plack CC||bar...@bywatersolutions.com -- You are receiving this mail because: You are watching all bug changes. ___ Koha-bugs mailing list Koha-bugs@lists.koha-community.org http://lists.koha-community.org/cgi-bin/mailman/listinfo/koha-bugs website : http://www.koha-community.org/ git : http://git.koha-community.org/ bugs : http://bugs.koha-community.org/