Hi fossologists, Well, the foss v1.1 licence analysis job that I wrote about at the start of this thread to analyse the Symbian kernel source, 7497 files, is still running after ~40 days elapsed with the machine up about 2/3 of the total time.
In that period, it has clocked up only 04:06 hrs scheduled time and 03:54 hrs running time. There is no evidence of the scheduler "getting stuck". The job is always clocking up more "items" processed - just extremely slowly - and `top -u fossy` always shows the scheduler, the fo-watchdog and the bsam engine accumulating CPU time - just extremely slowly. At no time is the system overloaded. Both CPUs are normally idling and < 30% of the 3GB memory is in normally in use. The bsam engine gets less than 1/6 of the CPU time that the scheduler itself gets. E.g. right now, top -u fossy shows: Tasks: 197 total, 1 running, 196 sleeping, 0 stopped, 0 zombie Cpu(s): 26.1%us, 6.5%sy, 0.0%ni, 66.7%id, 0.0%wa, 0.5%hi, 0.2%si, 0.0%st Mem: 3092484k total, 1504736k used, 1587748k free, 88888k buffers Swap: 1767108k total, 0k used, 1767108k free, 548452k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 1740 fossy 20 0 74780 68m 1756 S 0 2.3 0:06.40 fossology-sched 1742 fossy 20 0 7660 2812 1672 S 0 0.1 0:00.02 fo_watchdog 2258 fossy 20 0 9280 3256 2124 S 0 0.1 0:00.52 bsam-engine I have experimented with the scheduler.conf file to try to coax the job to consume more cycles (restarting the scheduler each time). I tried changing the default: "%Host localhost 1 1" to "%Host localhost 4 1" and in conjunction with the latter, replicated the line the "agent=licence..." line 4 times, so that 4 bsam engine processes where allowed to run concurrently. I also tried removing the -E parameter from the bsam engine commandline to disable exhaustive matching. I also tried changing the "nice" priority of all the fossy and postgresql processes from 0 to -5. None of these changes made any perceptible difference. I have allowed the job to carry on this long because the "items" counter of the license stage seemed to give me a way of perceiving progress toward the end. I assume that the "items" recorded by the license job-stage are comparisons of license+file pairs. Then since there are 359 licenses in the database and 7497 files in the tarball, there can only be 2,691,423 comparisons. But the items counter surpassed that total sometime over the weekend and now stands at 2,844,760, so I don't know what the items are or how many more there might be. I know that v1.2 offers very much faster licence analysis than 1.1, but the interminable runtime of this job is apparently not governed by the speed of license analysis. It is due to the fact that I cannot get fossology to employ more than a minute fraction of the available processor bandwidth. Is there reason to suppose v1.2 would do better in this respect? Rather than start that experiment myself, I'd be very grateful if someone on the project would download my kernel tarball - and one other that we'd like to benchmark even more - run the license analysis on them on a single 2-4 core Debian or Ubuntu system, and show it can be done in say < 48 hrs per package; then tell me exactly how! Symbian is prepared to invest in heftier kit to run fossology, but only if the software makes decently efficient use of it. Are there any takers for this? The entire ~35M LOC of the Symbian distribution is open source and you are heartily welcome to use as it as experimental fodder for fossology. Cheers, Mike On 17 February 2010 20:53, Mark Donohoe <mark.dono...@hp.com> wrote: > Gobeille, Robert wrote: > >> It sounds like the scheduler is "stuck" did you stop and restart? >> When you restart make sure the -R switch is on (/etc/init.d/fossology). >> What I've seen happen is the scheduler get's confused and doesn't see any >> jobs to run. This is one of a handful of reasons the scheduler is on my >> "hit list" but I won't have time to work on that until 1.2 is done. >> >> Yes the current, in progress, 1.2 is in the svn trunk. But before you do >> this make sure Mark chimes in since he is testing the trunk. He doesn't >> seem to be on IRC yet. >> >> Bob >> >> > > Folks, > > Before I left for skiing, TOT was working. Well except for the extra column > which Bob fixed. I haven't tried to install TOT since I got back and won't > till later this week or early next week. > > -- > Mark Donohoe > OS&T, Cupertino CA. > fossology.org > > -- Mike Kinghan, Test Lead, Symbian +44(0)776 5222 793 Symbian Foundation Limited is a Company Limited by Guarantee and not having a Share Capital registered in England and Wales under its number 6683783 whose registered address is 1 Boundary Row, Southwark, London SE1 8HP.
_______________________________________________ fossology mailing list fossology@fossology.org http://fossology.org/mailman/listinfo/fossology
