[JIRA] (JENKINS-3922) Slave is slow copying maven artifacts to master
[ https://issues.jenkins-ci.org/browse/JENKINS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=162143#comment-162143 ] Jimmi Dyson commented on JENKINS-3922: -- The fix is present, but doesn't actually seem to fix the problem. Rather it just improves the speed a bit, but it is still very slow compared to native SSH. I have done some tests this seems to be due to the library that Jenkins uses for SSH - org.jvnet.hudson:trilead-ssh2:build212-hudson-5. Simple tests show really slow SFTPing. I've experimented with [JSch|http://www.jcraft.com/jsch] get comparable speeds against native SFTP. I'm working on porting the SSH slaves plugin to use JSch (which is released under a BSD-style license - any compatibility issues there?). One drawback of the JSch library is the lack of Putty key support. I don't know if this is such a big deal as users can always convert Putty keys to OpenSSH format keys using puttygen? I notice that the org.jvnet.hudson:trilead-ssh2:build212-hudson-5 dependency comes as transitive from jenkins-core. Should the SSH library actually be a part of core dependencies? Slave is slow copying maven artifacts to master --- Key: JENKINS-3922 URL: https://issues.jenkins-ci.org/browse/JENKINS-3922 Project: Jenkins Issue Type: Bug Components: master-slave Affects Versions: current Environment: Platform: All, OS: All Reporter: John McNair Assignee: Kohsuke Kawaguchi Priority: Critical Attachments: pom.xml The artifact transfer is currently a 3-4x penalty for the project that I am working on. I have reproduced the issue with a simple test pom that does nothing but jar hudson.war. I performed this test on a heterogeneous environment. Both master and slave are running Fedora 10, but the master is a faster machine. Still, it highlights the issue. Here are some stats (all stats are after caching dependencies in the local repos): Master build through Hudson: 19s Master build from command line (no Hudson): 9s Slave build through Hudson: 1m46s Slave build from command line (no Hudson): 16s To be fair we should at least add time to do a straight scp of the artifact from slave to master. The two nodes share a 100 Mbit switch: $ scp target/slow-rider-1.0.0-SNAPSHOT.jar master_node: slow-rider-1.0.0NAPSHOT.jar 100% 25MB 12.7MB/s 00:02 Of course this example exaggerates the issue to make it more clear but not by too much. I originally noticed this in a completely separate environment that was all virtual. I reproduced this on two physical machines using a different switch and different ethernet drivers (both virtual and physical). The reproducibility plus the comparison against command line + scp leads me to suspect eager flushing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-3922) Slave is slow copying maven artifacts to master
[ https://issues.jenkins-ci.org/browse/JENKINS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmi Dyson reopened JENKINS-3922: -- The fix slightly speeds it up, but it is still slow enough to extend build times considerably with big artifacts. Experimenting with a different SSH library that shows good initial signs of speeding things up to near-native SSH speed. Slave is slow copying maven artifacts to master --- Key: JENKINS-3922 URL: https://issues.jenkins-ci.org/browse/JENKINS-3922 Project: Jenkins Issue Type: Bug Components: master-slave Affects Versions: current Environment: Platform: All, OS: All Reporter: John McNair Assignee: Kohsuke Kawaguchi Priority: Critical Attachments: pom.xml The artifact transfer is currently a 3-4x penalty for the project that I am working on. I have reproduced the issue with a simple test pom that does nothing but jar hudson.war. I performed this test on a heterogeneous environment. Both master and slave are running Fedora 10, but the master is a faster machine. Still, it highlights the issue. Here are some stats (all stats are after caching dependencies in the local repos): Master build through Hudson: 19s Master build from command line (no Hudson): 9s Slave build through Hudson: 1m46s Slave build from command line (no Hudson): 16s To be fair we should at least add time to do a straight scp of the artifact from slave to master. The two nodes share a 100 Mbit switch: $ scp target/slow-rider-1.0.0-SNAPSHOT.jar master_node: slow-rider-1.0.0NAPSHOT.jar 100% 25MB 12.7MB/s 00:02 Of course this example exaggerates the issue to make it more clear but not by too much. I originally noticed this in a completely separate environment that was all virtual. I reproduced this on two physical machines using a different switch and different ethernet drivers (both virtual and physical). The reproducibility plus the comparison against command line + scp leads me to suspect eager flushing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-3922) Slave is slow copying maven artifacts to master
[ https://issues.jenkins-ci.org/browse/JENKINS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=162156#comment-162156 ] Jimmi Dyson commented on JENKINS-3922: -- I've updated the ssh slaves plugin to use JSch all works fine. But it doesn't solve the SFTP speed issue... that would appear to be because it uses _hudson.util.ssh.SFTPClient_ instead of the one in the ssh slaves plugin. I can't figure out where this class is created though? Am I missing something? I'd like to refactor this as well to use JSch, but the constructor uses _com.trilead.ssh2.Connection_ so need to know where it's created to refactor as well. Any ideas? Slave is slow copying maven artifacts to master --- Key: JENKINS-3922 URL: https://issues.jenkins-ci.org/browse/JENKINS-3922 Project: Jenkins Issue Type: Bug Components: master-slave Affects Versions: current Environment: Platform: All, OS: All Reporter: John McNair Assignee: Kohsuke Kawaguchi Priority: Critical Attachments: pom.xml The artifact transfer is currently a 3-4x penalty for the project that I am working on. I have reproduced the issue with a simple test pom that does nothing but jar hudson.war. I performed this test on a heterogeneous environment. Both master and slave are running Fedora 10, but the master is a faster machine. Still, it highlights the issue. Here are some stats (all stats are after caching dependencies in the local repos): Master build through Hudson: 19s Master build from command line (no Hudson): 9s Slave build through Hudson: 1m46s Slave build from command line (no Hudson): 16s To be fair we should at least add time to do a straight scp of the artifact from slave to master. The two nodes share a 100 Mbit switch: $ scp target/slow-rider-1.0.0-SNAPSHOT.jar master_node: slow-rider-1.0.0NAPSHOT.jar 100% 25MB 12.7MB/s 00:02 Of course this example exaggerates the issue to make it more clear but not by too much. I originally noticed this in a completely separate environment that was all virtual. I reproduced this on two physical machines using a different switch and different ethernet drivers (both virtual and physical). The reproducibility plus the comparison against command line + scp leads me to suspect eager flushing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[JIRA] (JENKINS-3922) Slave is slow copying maven artifacts to master
[ https://issues.jenkins-ci.org/browse/JENKINS-3922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=162156#comment-162156 ] Jimmi Dyson edited comment on JENKINS-3922 at 4/27/12 3:31 PM: --- I've updated the ssh slaves plugin to use JSch all works fine for connection, starting, running builds, disconnecting, etc. But it doesn't solve the SFTP speed issue... I now realise that it doesn't actually use SFTP for archiving artifacts back to the master - that is done through the FilePath abstraction I believe. So the speed issue should exist in all slave types, such as JNLP? So why is this slow? In our environment, doing native SSH transfers takes around 10 seconds for a 100MB transfer. Jenkins archiving a 100MB artifact takes about 50 seconds using an SSH slave. was (Author: jimmidyson): I've updated the ssh slaves plugin to use JSch all works fine. But it doesn't solve the SFTP speed issue... that would appear to be because it uses _hudson.util.ssh.SFTPClient_ instead of the one in the ssh slaves plugin. I can't figure out where this class is created though? Am I missing something? I'd like to refactor this as well to use JSch, but the constructor uses _com.trilead.ssh2.Connection_ so need to know where it's created to refactor as well. Any ideas? Slave is slow copying maven artifacts to master --- Key: JENKINS-3922 URL: https://issues.jenkins-ci.org/browse/JENKINS-3922 Project: Jenkins Issue Type: Bug Components: master-slave Affects Versions: current Environment: Platform: All, OS: All Reporter: John McNair Assignee: Kohsuke Kawaguchi Priority: Critical Attachments: pom.xml The artifact transfer is currently a 3-4x penalty for the project that I am working on. I have reproduced the issue with a simple test pom that does nothing but jar hudson.war. I performed this test on a heterogeneous environment. Both master and slave are running Fedora 10, but the master is a faster machine. Still, it highlights the issue. Here are some stats (all stats are after caching dependencies in the local repos): Master build through Hudson: 19s Master build from command line (no Hudson): 9s Slave build through Hudson: 1m46s Slave build from command line (no Hudson): 16s To be fair we should at least add time to do a straight scp of the artifact from slave to master. The two nodes share a 100 Mbit switch: $ scp target/slow-rider-1.0.0-SNAPSHOT.jar master_node: slow-rider-1.0.0NAPSHOT.jar 100% 25MB 12.7MB/s 00:02 Of course this example exaggerates the issue to make it more clear but not by too much. I originally noticed this in a completely separate environment that was all virtual. I reproduced this on two physical machines using a different switch and different ethernet drivers (both virtual and physical). The reproducibility plus the comparison against command line + scp leads me to suspect eager flushing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.jenkins-ci.org/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
