[JIRA] (JENKINS-15120) Minimize round trips for slave class loading
Jesse Glick commented on JENKINS-15120 Minimize round trips for slave class loading remoting #37248f1 also suggests using sudo tc qdisc add dev lo root netem delay 100ms to simulate a laggy network. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] (JENKINS-15120) Minimize round trips for slave class loading
Jesse Glick commented on JENKINS-15120 Minimize round trips for slave class loading https://github.com/jenkinsci/mock-slave-plugin useful for testing impact on performance more controllably than simply connecting to some node in the cloud. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/groups/opt_out.
[JIRA] (JENKINS-15120) Minimize round trips for slave class loading
Jesse Glick commented on JENKINS-15120 Minimize round trips for slave class loading https://github.com/jenkinsci/remoting/pull/10 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-15120) Minimize round trips for slave class loading
Jesse Glick started work on JENKINS-15120 Minimize round trips for slave class loading Change By: Jesse Glick (10/Dec/12 11:50 PM) Status: Open In Progress This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-15120) Minimize round trips for slave class loading
SCM/JIRA link daemon commented on JENKINS-15120 Minimize round trips for slave class loading Code changed in jenkins User: Jesse Glick Path: src/main/java/hudson/remoting/Channel.java http://jenkins-ci.org/commit/remoting/2b1ec8ab152805f01b4063dabc4dcdef64421fed Log: JENKINS-15120 Kohsuke’s explanation of why preloadJar does not really help. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-15120) Minimize round trips for slave class loading
Jesse Glick commented on JENKINS-15120 Minimize round trips for slave class loading Can add a dependency from remoting on ASM via http://kohsuke.org/2012/03/03/potd-package-renamed-asm/ if needed for prefetch. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-15120) Minimize round trips for slave class loading
Jesse Glick
created JENKINS-15120
Minimize round trips for slave class loading
Issue Type:
Improvement
Assignee:
Jesse Glick
Components:
core
Created:
11/Sep/12 5:53 PM
Description:
Currently each attempt to load a class in a remote JVM makes a round-trip request to the master, which over a laggy network can make class loading quite slow, thus adding considerable overhead to the first build on a new slave.
Two possible solutions have been put forward.
Optimistic prefetch
The idea: when sending a class file to be loaded, scan its bytecode for other statically linked classes which have not yet been loaded, and send those along as well. In the common case that a network of classes is loaded around the same time, this would avoid some round trips.
Details
The first part is for one side to keep track of what classes the other side has already loaded (into which class loader), which is basically memorizing the response from IClassLoader.fetch2.
The second part is to add Collection IClassLoader.fetch3 that works like fetch2, except it will also parse the class, figure out some of the referenced classes that are not yet loaded by the other side, then send them along.
Those prefetched class files would need to be remembered by RemoteClassLoader so that when those are actually requested it can load a class in the right classloader without calling back RemoteClassLoader.proxy. (Assuming there are no side effects, it could also eagerly call RemoteClassLoader.loadClassFile on the prefetched classes.)
The remoting layer supports talking to an earlier version of the remoting layer. We do this by a bitmask in Capability, so this needs one more bit defined there. There is no point in tracking the classes the other side has loaded if the other side will never call fetch3.
Bulk transfer
Send entire JAR files at a time, rather than individual classes; can wind up transferring more than is needed, but the reduction in latency is probably worth it. Since arbitrary class loader graphs might be in use, not just a flat classpath, some custom code needs to be run remotely which will implement the class loader delegation model without hitting the network for each class.
Details
An API sketch:
class Channel {
void setClassLoaderTrafficCop(TrafficCop cop);
}
interface TrafficCop {
/** do I know/control/own this classloader? */
boolean controls(ClassLoader cl);
Set getJarFilesOf(ClassLoader cl);
RemotePartOfTrafficCop getRemotePart();
}
class JarFile {
String checksum();
InputStream data();
}
/** runs in remote agent */
interface RemotePartOfTrafficCop implements Serializable {
/** given a class/resource and an originating loader, what is the defining loader? */
RemoteClassLoader trafficControl(RemoteClassLoader origin, String resourceName);
}
Project:
Jenkins
Labels:
remoting
performance
slave
Priority:
Major
Reporter:
Jesse Glick
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
