[JIRA] (JENKINS-15120) Minimize round trips for slave class loading

2013-03-21 Thread jgl...@cloudbees.com (JIRA)














































Jesse Glick
 commented on  JENKINS-15120


Minimize round trips for slave class loading















remoting #37248f1 also suggests using


sudo tc qdisc add dev lo root netem delay 100ms


to simulate a laggy network.



























This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira







-- 
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 


[JIRA] (JENKINS-15120) Minimize round trips for slave class loading

2013-03-20 Thread jgl...@cloudbees.com (JIRA)














































Jesse Glick
 commented on  JENKINS-15120


Minimize round trips for slave class loading















https://github.com/jenkinsci/mock-slave-plugin useful for testing impact on performance more controllably than simply connecting to some node in the cloud.



























This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira







-- 
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 


[JIRA] (JENKINS-15120) Minimize round trips for slave class loading

2013-01-04 Thread jgl...@cloudbees.com (JIRA)














































Jesse Glick
 commented on  JENKINS-15120


Minimize round trips for slave class loading















https://github.com/jenkinsci/remoting/pull/10



























This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira






[JIRA] (JENKINS-15120) Minimize round trips for slave class loading

2012-12-10 Thread jgl...@cloudbees.com (JIRA)














































Jesse Glick
 started work on  JENKINS-15120


Minimize round trips for slave class loading
















Change By:


Jesse Glick
(10/Dec/12 11:50 PM)




Status:


Open
In Progress



























This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira






[JIRA] (JENKINS-15120) Minimize round trips for slave class loading

2012-12-07 Thread scm_issue_l...@java.net (JIRA)














































SCM/JIRA link daemon
 commented on  JENKINS-15120


Minimize round trips for slave class loading















Code changed in jenkins
User: Jesse Glick
Path:
 src/main/java/hudson/remoting/Channel.java
http://jenkins-ci.org/commit/remoting/2b1ec8ab152805f01b4063dabc4dcdef64421fed
Log:
  JENKINS-15120 Kohsuke’s explanation of why preloadJar does not really help.































This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira






[JIRA] (JENKINS-15120) Minimize round trips for slave class loading

2012-09-12 Thread jgl...@cloudbees.com (JIRA)














































Jesse Glick
 commented on  JENKINS-15120


Minimize round trips for slave class loading















Can add a dependency from remoting on ASM via http://kohsuke.org/2012/03/03/potd-package-renamed-asm/ if needed for prefetch.



























This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira






[JIRA] (JENKINS-15120) Minimize round trips for slave class loading

2012-09-11 Thread jgl...@cloudbees.com (JIRA)














































Jesse Glick
 created  JENKINS-15120


Minimize round trips for slave class loading















Issue Type:


Improvement



Assignee:


Jesse Glick



Components:


core



Created:


11/Sep/12 5:53 PM



Description:


Currently each attempt to load a class in a remote JVM makes a round-trip request to the master, which over a laggy network can make class loading quite slow, thus adding considerable overhead to the first build on a new slave.

Two possible solutions have been put forward.

Optimistic prefetch

The idea: when sending a class file to be loaded, scan its bytecode for other statically linked classes which have not yet been loaded, and send those along as well. In the common case that a network of classes is loaded around the same time, this would avoid some round trips.

Details

The first part is for one side to keep track of what classes the other side has already loaded (into which class loader), which is basically memorizing the response from IClassLoader.fetch2.

The second part is to add Collection IClassLoader.fetch3 that works like fetch2, except it will also parse the class, figure out some of the referenced classes that are not yet loaded by the other side, then send them along.

Those prefetched class files would need to be remembered by RemoteClassLoader so that when those are actually requested it can load a class in the right classloader without calling back RemoteClassLoader.proxy. (Assuming there are no side effects, it could also eagerly call RemoteClassLoader.loadClassFile on the prefetched classes.)

The remoting layer supports talking to an earlier version of the remoting layer. We do this by a bitmask in Capability, so this needs one more bit defined there. There is no point in tracking the classes the other side has loaded if the other side will never call fetch3.

Bulk transfer

Send entire JAR files at a time, rather than individual classes; can wind up transferring more than is needed, but the reduction in latency is probably worth it. Since arbitrary class loader graphs might be in use, not just a flat classpath, some custom code needs to be run remotely which will implement the class loader delegation model without hitting the network for each class.

Details

An API sketch:


class Channel {
  void setClassLoaderTrafficCop(TrafficCop cop);
}
interface TrafficCop {
  /** do I know/control/own this classloader? */
  boolean controls(ClassLoader cl);
  Set getJarFilesOf(ClassLoader cl);
  RemotePartOfTrafficCop getRemotePart();
}
class JarFile {
  String checksum();
  InputStream data();
}
/** runs in remote agent */
interface RemotePartOfTrafficCop implements Serializable {
  /** given a class/resource and an originating loader, what is the defining loader? */
  RemoteClassLoader trafficControl(RemoteClassLoader origin, String resourceName);
}





Project:


Jenkins



Labels:


remoting
performance
slave




Priority:


Major



Reporter:


Jesse Glick

























This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira