Re: Parallelism (was Re: A Few Plans)

2010-06-21 Thread Leo Simons 

On 6/21/10 5:34 AM, Stefan Bodewig wrote:

This would allow for a little more concurrency than we can do on the
Zone or VM... of course we'd have to be able to address all of those
cores.  Wonder whether Python has glue for Grand Central Dispatch?


Most of the weight in gump runs is inside the java processes; the other 
half of the latency is svn checkouts/updates.


For the former, you'd need (a) the JVM to hook up GCD (which is for 
apple to do) and (b) maven to do more stuff in parallel (which is on the 
charts for maven 3 I think).


For the latter, IIRC we have code to run more checkouts in parallel, but 
the code is buggy in gump2; and would mean a load increase on the svn 
server which may not be a good thing?



If you log into one of the machines while Gump is running, the system
feels sluggish and any opration that hits the file system takes ages
which makes me fear we are I/O bound rather than CPU bound - making
those cores do more may not help too much in that case.  I can certainly
be wrong.


You're absolutely right. Builds are almost always I/O bound and you'll 
see a lot of CPU is actually iowait -- so the numbers in top are usually 
misleading and most of CPU busy-ness is due to overhead of waiting for IO.



IIRC Gump's trunk supports parallel SCM checkouts but we've restricted
it to a maximum of one updater because Adam saw problems - it's been a
long time.


Oh, eh, yep, so that's what I remember too :)

In any case to take advantage of multicores it'd be good to re-implement 
parallelism using python's multiprocess module.



Currently we don't support building things in parallel at all.  Starting
several Ant or make builds in parallel would likely do what you expect,
but I don't know how mvn would deal with multiple processes accessing
the same local repository (and writing to it) in parallel.


I don't think there's any special code for it in maven, but nevertheless 
I've never really seen an issue with that. A common hudson deployment is 
to run 4-5 builds in parallel using a 'hudson' user that has one local 
repository.



cheers,


Leo

PS: no I'm not dead just persistently e-mail overloaded :)

-
To unsubscribe, e-mail: general-unsubscr...@gump.apache.org
For additional commands, e-mail: general-h...@gump.apache.org

Re: xml-fop: Can't figure out why it fails

2010-06-21 Thread Jeremias Maerki 
Wonderful. Thanks for your feedback, Stefan!

On 21.06.2010 12:08:07 Stefan Bodewig wrote:
> On 2010-06-21, Jeremias Maerki wrote:
> 
> > It appears as if Gump didn't SVN update the workspace properly and the
> > changes [4] I made to FOP Trunk to adjust for the move are not visible
> > to Gump.
> 
> You are correct, the workspace is locked because of some earlier error
> while we had svn problems
> 
> 
> 
> I'll cleanup thw workspace and grep through the logs to see whether
> there is more to do.  Should work on the next run (after the currently
> building one, that is).
> 
> Thanks
> 
> Stefan
> 



Jeremias Maerki


-
To unsubscribe, e-mail: general-unsubscr...@gump.apache.org
For additional commands, e-mail: general-h...@gump.apache.org

Re: xml-fop: Can't figure out why it fails

2010-06-21 Thread Stefan Bodewig 
On 2010-06-21, Jeremias Maerki wrote:

> It appears as if Gump didn't SVN update the workspace properly and the
> changes [4] I made to FOP Trunk to adjust for the move are not visible
> to Gump.

You are correct, the workspace is locked because of some earlier error
while we had svn problems



I'll cleanup thw workspace and grep through the logs to see whether
there is more to do.  Should work on the next run (after the currently
building one, that is).

Thanks

Stefan

-
To unsubscribe, e-mail: general-unsubscr...@gump.apache.org
For additional commands, e-mail: general-h...@gump.apache.org

Problem running Apache Gump [vmgump-public]

2010-06-21 Thread gump 
There is a problem with run 'vmgump-public' (21062010_02), location : 
http://vmgump.apache.org/gump/public

The log ought be at:
   http://vmgump.apache.org/gump/public/gump_log.txt

The last (up to) 50 lines of the log are :
INFO: Redirecting via client connector to: 
http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.pom
Jun 21, 2010 2:51:50 AM com.noelios.restlet.http.HttpClientCall 
getResponseEntity
INFO: The length of the message body is unknown. The entity must be handled 
carefully and consumed entirely in order to surely release the connection.
Jun 21, 2010 2:51:50 AM com.noelios.restlet.LogFilter afterHandle
INFO: 2010-06-2102:51:50127.0.0.1   -   127.0.0.1   
8192GET 
/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.pom   
-   200 -   -   110 http://localhost:8192   
Apache-Maven/2.2 (Java 1.6.0_06; Linux 2.6.24-28-server) maven-artifact/2.2.1   
-
Jun 21, 2010 2:51:50 AM org.apache.gump.mvnrepoproxy.resources.GumpArtifact log
INFO: proxying .pom sha1 checksum artifact for groupId 
'org.apache.maven.surefire' and artifactId 'surefire-junit4'
Jun 21, 2010 2:51:50 AM org.restlet.Redirector handle
INFO: Redirecting via client connector to: 
http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.pom.sha1
Jun 21, 2010 2:51:50 AM com.noelios.restlet.http.HttpClientCall 
getResponseEntity
INFO: The length of the message body is unknown. The entity must be handled 
carefully and consumed entirely in order to surely release the connection.
Jun 21, 2010 2:51:50 AM com.noelios.restlet.LogFilter afterHandle
INFO: 2010-06-2102:51:50127.0.0.1   -   127.0.0.1   
8192GET 
/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.pom.sha1
  -   200 -   -   294 http://localhost:8192   
Apache-Maven/2.2 (Java 1.6.0_06; Linux 2.6.24-28-server) maven-artifact/2.2.1   
-
Jun 21, 2010 2:51:50 AM org.apache.gump.mvnrepoproxy.resources.GumpArtifact log
INFO: proxying .jar artifact for groupId 'org.apache.maven.surefire' and 
artifactId 'surefire-junit4'
Jun 21, 2010 2:51:50 AM org.restlet.Redirector handle
INFO: Redirecting via client connector to: 
http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.jar
Jun 21, 2010 2:51:50 AM com.noelios.restlet.http.HttpClientCall 
getResponseEntity
INFO: The length of the message body is unknown. The entity must be handled 
carefully and consumed entirely in order to surely release the connection.
Jun 21, 2010 2:51:50 AM com.noelios.restlet.LogFilter afterHandle
INFO: 2010-06-2102:51:50127.0.0.1   -   127.0.0.1   
8192GET 
/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.jar   
-   200 -   -   394 http://localhost:8192   
Apache-Maven/2.2 (Java 1.6.0_06; Linux 2.6.24-28-server) maven-artifact/2.2.1   
-
Jun 21, 2010 2:51:50 AM org.apache.gump.mvnrepoproxy.resources.GumpArtifact log
INFO: proxying .jar sha1 checksum artifact for groupId 
'org.apache.maven.surefire' and artifactId 'surefire-junit4'
Jun 21, 2010 2:51:50 AM org.restlet.Redirector handle
INFO: Redirecting via client connector to: 
http://repo1.maven.org/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.jar.sha1
Jun 21, 2010 2:51:51 AM com.noelios.restlet.http.HttpClientCall 
getResponseEntity
INFO: The length of the message body is unknown. The entity must be handled 
carefully and consumed entirely in order to surely release the connection.
Jun 21, 2010 2:51:51 AM com.noelios.restlet.LogFilter afterHandle
INFO: 2010-06-2102:51:51127.0.0.1   -   127.0.0.1   
8192GET 
/maven2/org/apache/maven/surefire/surefire-junit4/2.5/surefire-junit4-2.5.jar.sha1
  -   200 -   -   95  http://localhost:8192   
Apache-Maven/2.2 (Java 1.6.0_06; Linux 2.6.24-28-server) maven-artifact/2.2.1   
-
Traceback (most recent call last):
  File "bin/integrate.py", line 114, in 
irun()
  File "bin/integrate.py", line 91, in irun
result = getRunner(run).perform()
  File "/srv/gump/public/gump/python/gump/core/runner/runner.py", line 262, in 
perform
return self.performRun()
  File "/srv/gump/public/gump/python/gump/core/runner/demand.py", line 199, in 
performRun
self.performBuild(project)
  File "/srv/gump/public/gump/python/gump/core/runner/demand.py", line 127, in 
performBuild
self.builder.buildProject(project)   
  File "/srv/gump/public/gump/python/gump/core/build/builder.py", line 153, in 
buildProject
self.performPostBuild( project, languageHelper, stats )
  File "/srv/gump/public/gump/python/gump/core/build/builder.py", line 272, in 
performPostBuild

xml-fop: Can't figure out why it fails

2010-06-21 Thread Jeremias Maerki 
Hi there,

I need some help figuring out why xml-fop currently fails. I've moved
some newly added classes from package org.apache.xmlgraphics.java2d to
org.apache.xmlgraphics.java2d.color [1]. Now, xmlgraphics-commons
happily builds and if I look at the generated JAR [2], the classes are
in the right place.

But xml-fop fails [3] since it can't find those classes. It appears as
if Gump didn't SVN update the workspace properly and the changes [4] I
made to FOP Trunk to adjust for the move are not visible to Gump.

[1] http://svn.apache.org/viewvc?rev=954487&view=rev
[2] http://vmgump.apache.org/gump/public-jars/xmlgraphics-commons/jars/
[3] 
http://vmgump.apache.org/gump/public/xml-fop/xml-fop/gump_work/build_xml-fop_xml-fop.html
[4] http://svn.apache.org/viewvc?rev=954512&view=rev

Thanks!
Jeremias Maerki


-
To unsubscribe, e-mail: general-unsubscr...@gump.apache.org
For additional commands, e-mail: general-h...@gump.apache.org