Bug#436059: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
> I tried with multiple numbers - 100, 50, 10. > No luck, none of them helped. I killed apt-listbugs with a signal on i386/kfreebsd, and it died in Parsing Found/Fixed information... 0% Exception `SignalException' at /usr/lib/ruby/1.8/debian.rb:78 - SIGTERM Terminated This is dpkg-ruby trying to fork dpkg, and dpkg never coming back home. def compare_versions(a, rel, b) return system(DPKG, '--compare-versions', a, rel, b) end regards, junichi -- [EMAIL PROTECTED],netfort.gr.jp} Debian Project -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#436059: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
Hi, > I tried with multiple numbers - 100, 50, 10. > No luck, none of them helped. strange. > As I had mentioned in my earlier posts, this issue is only seen when the > number of packages passed is too high. In both the cases for me, the meta > package being installed was xserver-xorg which led to installation of 41 > packages. > If I lower down the number of packages, thing will work. > > So maybe you could try to look into ways to cut down one shot execution of > Parsing all packages. If the number of packages passed is too high (say > packages > 15), process them in order. > > I think this should be the cause because lower number of packages passed > doesn't create trouble for apt-listbug on other platforms. This will be absurdly slow. Can you actually look at the memory consumption to verify that it's really a problem? It could be ruby Garbage Collector being broken on those arches. > # process each bug number and get contents, in bach of ParseStep > BTW, I couldn't make out the meaning of this comment. Are you talking > about "batch" or "back" ? batch. regards, junichi -- [EMAIL PROTECTED],netfort.gr.jp} Debian Project -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#436059: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
On Tuesday 07 August 2007, Junichi Uekawa wrote:
> in lib/debian/bts.rb there is a constant
>
> ParseStep = 200
>
> This is the value which determines how many bugs to process at a time.
> Reducing this number will make things slow, and it should really be
> determined dynamically based on available memory, but it's currently
> defined static.
>
> If you can tweak this number and find out if this improves the
> situation for you, it will be a source of motivation for me to make
> this variable dynamic (or configurable).
Hi Junichi,
I tried with multiple numbers - 100, 50, 10.
No luck, none of them helped.
FYI, this issue is also reproducible on Debian GNU/kFreeBSD running under a
separate VM with 256M of RAM.
So the swapping issue should stay valid or not is a difficult thing to say, at
least for BSDs.
I looked into bts.rb
ParseStep = 200
def parse(pkgs, severities = ["critical", "grave"])
require 'debian/btssoap'
soap = Debian::BTSSOAP::Soap.new(@host, @port)
sa = Debian::BTSSOAP::StringArray.new
bugs = Debian::Bugs.new
reqbugs = soap.get_bugs('severity', severities, 'package', pkgs)
# the total number of bugs
max = reqbugs.length
# progressed amount
offset_i = 0
# process each bug number and get contents, in bach of ParseStep
reqbugs.each { |bug|
sa << bug
if sa.length > ParseStep
soap.get_status(sa).each { |b| bugs << b }
offset_i += sa.length
sa = Debian::BTSSOAP::StringArray.new
yield "#{((offset_i).to_f*100/max.to_f).to_i}%"
As I had mentioned in my earlier posts, this issue is only seen when the
number of packages passed is too high. In both the cases for me, the meta
package being installed was xserver-xorg which led to installation of 41
packages.
If I lower down the number of packages, thing will work.
So maybe you could try to look into ways to cut down one shot execution of
Parsing all packages. If the number of packages passed is too high (say
packages > 15), process them in order.
I think this should be the cause because lower number of packages passed
doesn't create trouble for apt-listbug on other platforms.
# process each bug number and get contents, in bach of ParseStep
BTW, I couldn't make out the meaning of this comment. Are you talking
about "batch" or "back" ?
Ritesh
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
signature.asc
Description: This is a digitally signed message part.
Bug#436059: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
Hi, > I might agree with you on that because the Hurd OS is running under KVM > with 128M of RAM allocated for it. > Although KVM is fast I've felt that Hurd is not _very_ good at memory > management. > This could be true because when I lower down the number of packages, the > installation proceeds. Yes, that sounds like it. 128MB is definitely not enough, and I think the process is starting to swap. The last time I used Hurd (which is like 5 years ago) swap management was done by Mach, and it wasn't that good. > BTW, if this holds true what do you think would be a possibility for a > workaround (excluding the idea of > Lowering down the number of packages ;-)) ? > > Is there a possibility to parse reports one-by-one and maybe write to a > temporary file and then read it ? > Just let's avoid reading the entire thing into RAM. What do you say ? in lib/debian/bts.rb there is a constant ParseStep = 200 This is the value which determines how many bugs to process at a time. Reducing this number will make things slow, and it should really be determined dynamically based on available memory, but it's currently defined static. If you can tweak this number and find out if this improves the situation for you, it will be a source of motivation for me to make this variable dynamic (or configurable). regards, junichi -- [EMAIL PROTECTED],netfort.gr.jp} Debian Project -- To UNSUBSCRIBE, email to [EMAIL PROTECTED] with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#436059: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
I might agree with you on that because the Hurd OS is running under KVM
with 128M of RAM allocated for it.
Although KVM is fast I've felt that Hurd is not _very_ good at memory
management.
This could be true because when I lower down the number of packages, the
installation proceeds.
BTW, if this holds true what do you think would be a possibility for a
workaround (excluding the idea of
Lowering down the number of packages ;-)) ?
Is there a possibility to parse reports one-by-one and maybe write to a
temporary file and then read it ?
Just let's avoid reading the entire thing into RAM. What do you say ?
Ritesh
-Original Message-
From: Junichi Uekawa [mailto:[EMAIL PROTECTED]
Sent: Tuesday, August 07, 2007 4:39 AM
To: Sarraf, Ritesh
Cc: [EMAIL PROTECTED]; Junichi Uekawa
Subject: Re: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
Hi,
At Mon, 6 Aug 2007 20:23:14 +0530,
Ritesh Raj Sarraf wrote:
>
> Hi Junichi,
>
> Please ignore my previous mail because that is not containing the
> correct information. It is timing out at the network level only. It
> was a network issue.
>
> Here's the proper log that you were wanting:
> {}forwarded="" {}severity="serious" {}owner=""
> {}log_modified=1185310445 {}location="db-h" {}subject="lacks upstream
URL in the copyright file"
> {}originator="Josip Rodin <[EMAIL PROTECTED]>"
{}pending="done"
> {}tags="" {}fixed_date=[] {}package="xfonts-encodings"
> {}found_date=[]> Retrieving bug reports... Done Parsing Found/Fixed
> information... 0%
A wild guess, how much memory do you have?
It occurred to me that could be memory swap management issue with Hurd.
Looking at memory usage running top might tell you it's using up memory.
>From my experience, it may require around 200MB of memory at this point.
Of course, I could be barking at the wrong tree, but one possibility.
regards,
junichi
--
[EMAIL PROTECTED],netfort.gr.jp} Debian Project
Bug#436059: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
Hi,
At Mon, 6 Aug 2007 20:23:14 +0530,
Ritesh Raj Sarraf wrote:
>
> Hi Junichi,
>
> Please ignore my previous mail because that is not containing the correct
> information. It is timing out at the network level only. It was a network
> issue.
>
> Here's the proper log that you were wanting:
> {}forwarded="" {}severity="serious" {}owner="" {}log_modified=1185310445
> {}location="db-h" {}subject="lacks upstream URL in the copyright file"
> {}originator="Josip Rodin <[EMAIL PROTECTED]>" {}pending="done"
> {}tags="" {}fixed_date=[] {}package="xfonts-encodings" {}found_date=[]>
> Retrieving bug reports... Done
> Parsing Found/Fixed information... 0%
A wild guess, how much memory do you have?
It occurred to me that could be memory swap management issue with Hurd.
Looking at memory usage running top might tell you it's using up
memory. From my experience, it may require around 200MB of memory at
this point.
Of course, I could be barking at the wrong tree, but one possibility.
regards,
junichi
--
[EMAIL PROTECTED],netfort.gr.jp} Debian Project
--
To UNSUBSCRIBE, email to [EMAIL PROTECTED]
with a subject of "unsubscribe". Trouble? Contact [EMAIL PROTECTED]
Bug#436059: Fwd: Re: Bug#436059: apt-listbug remains frozen on hurd
Hi Junichi, Please ignore my previous mail because that is not containing the correct information. It is timing out at the network level only. It was a network issue. Here's the proper log that you were wanting: hurd:~# (echo 'VERSION 2'; echo ''; ls -1 /var/cache/apt/archives/*_*.deb | sed 's/^/x x x x /') | /usr/sbin/apt-listbugs apt -d Reading package fields... Done Reading package status... Done Retrieving bug reports... 0%Exception `LoadError' at /usr/lib/ruby/1.8/xml/encoding-ja.rb:12 - no such file to load -- uconv Set XSD::XMLParser::XMLParser as XML processor. Exception `LoadError' at /usr/lib/ruby/1.8/http-access2.rb:31 - no such file to load -- openssl Wire dump: = Request ! CONNECT TO bugs.debian.org:80 ! CONNECTION ESTABLISHED POST /cgi-bin/soap.cgi HTTP/1.1 SOAPAction: "" Content-Type: text/xml; charset=utf-8 User-Agent: SOAP4R/1.5.5 (/114, ruby 1.8.6 (2007-06-07) [i486-gnu]) Date: Mon Aug 06 20:20:49 +0530 2007 Content-Length: 4038 Host: bugs.debian.org http://www.w3.org/2001/XMLSchema"; xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";> http://schemas.xmlsoap.org/soap/encoding/";> http://schemas.xmlsoap.org/soap/encoding/"; xsi:type="n2:Array"> severity critical grave serious package xserver-xorg-video-nsc xserver-xorg-video-newport xserver-xorg-video-apm inetutils-syslogd xserver-xorg xserver-xorg-video-vesa xserver-xorg-video-rendition xkb-data xbitmaps xserver-xorg-video-nv xfonts-encodings xfonts-100dpi type-handling libglu1-mesa xserver-xorg-video-intel xserver-xorg-video-imstt xserver-xorg-video-cyrix xserver-xorg-video-ati xserver-xorg-core xserver-xorg-video-voodoo xserver-xorg-video-siliconmotion xserver-xorg-video-neomagic xserver-xorg-video-i740 libfontenc1 xutils-dev xserver-xorg-video-s3 xserver-xorg-video-trident xserver-xorg-video-tga xserver-xorg-video-savage xserver-xorg-video-s3virge xutils xserver-xorg-video-tdfx xserver-xorg-video-dummy xfonts-base xserver-xorg-video-vga xserver-xorg-video-tseng xserver-xorg-video-i810 xserver-xorg-video-i128 xserver-xorg-input-mouse xserver-xorg-input-all libxfont1 xserver-xorg-video-cirrus xserver-xorg-input-kbd xserver-xorg-video-sis xserver-xorg-video-mga xserver-xorg-video-chips = Response HTTP/1.1 200 OK Date: Mon, 06 Aug 2007 14:50:52 GMT Content-Length: 832 Content-Type: text/xml; charset=utf-8 Server: Apache/2.0.54 (Debian GNU/Linux) mod_python/3.1.3 Python/2.3.5 SOAPServer: SOAP::Lite/Perl/0.60 Via: 1.1 nc-btc01 (NetCache NetApp/6.0.3) http://www.w3.org/2001/XMLSchema-instance"; xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"; xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:xsd="http://www.w3.org/2001/XMLSchema"; SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/";>434290428794434288383465428882422680434286 fetching 434290 428794 434288 383465 428882 422680 434286.. Wire dump: = Request POST /cgi-bin/soap.cgi HTTP/1.1 SOAPAction: "" Content-Type: text/xml; charset=utf-8 User-Agent: SOAP4R/1.5.5 (/114, ruby 1.8.6 (2007-06-07) [i486-gnu]) Date: Mon Aug 06 20:20:50 +0530 2007 Content-Length: 882 Host: bugs.debian.org http://www.w3.org/2001/XMLSchema"; xmlns:env="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";> http://schemas.xmlsoap.org/soap/encoding/";> http://schemas.xmlsoap.org/soap/encoding/"; xsi:type="n2:Array"> 434290 428794 434288 383465 428882 422680 434286 = Response HTTP/1.1 200 OK Date: Mon, 06 Aug 2007 14:50:54 GMT Content-Length: 12336 Content-Type: text/xml; charset=utf-8 Server: Apache/2.0.54 (Debian GNU/Linux) mod_python/3.1.3 Python/2.3.5 SOAPServer: SOAP::Lite/Perl/0.60 Via: 1.1 nc-btc01 (NetCache NetApp/6.0.3) http://xml.apache.org/xml-soap"; xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/"; xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/"; xmlns:xsd="http://www.w3.org/2001/XMLSchema"; SOAP-ENV:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/";>428794Julien Cristau <[EMAIL PROTECTED]>1181811241xorg-server/2:1.3.0.0.dfsg-8xorg-server/2:1.3.0.0.dfsg-8428882sid lenny fixed-upstream<[EMAIL PROTECTED]>428794https://bugs.freedesktop.org/show_bug.cgi?id=11301serious11862570
