[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title Joseph Green commented on JENKINS-50429 Re: Shell command are really slower than before Fabrice Pipart - This looks promising! Just for completeness, could you share more information about the fix? Which file are you applying this fix to? Is there a PR you could share (even if its just a Work In Progress or PoC)? What test are you running that took 1m43 and now takes 8s? Could you share the reference code (or link to the comment above if its one of the previous examples?) Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title
Joseph Green commented on JENKINS-50429
Re: Shell command are really slower than before
I've tested combining all the env vars into a single command and that makes no difference. My test command was this:
def label = "slow-${UUID.randomUUID().toString()}"
podTemplate(label: label) {
node(label){
stage('20 calls outside container'){
for (i = 0; i <20; i++) {
sh "date"
}
}
stage('20 calls inside container'){
container("jnlp"){
for (i = 0; i <20; i++) {
sh "date"
}
}
}
}
}
Results were pretty consistently: Outside container: ~12 seconds Inside container: ~1min 10 seconds Individual sh calls for the `date` command were: Outside container: ~400ms Inside container: ~3 seconds Commenting out all the env variable setting only reduced the individual container time to 1s... This seems to tally with the stack overflow post reporting the piped input stream bug. (https://github.com/karianna/jdk8_tl/blob/master/jdk/src/share/classes/java/io/PipedInputStream.java#L274) ?? I tried combining the env variables into a single set before sending them, which reduced the time spent inside the container to 30 seconds. There's a PR for that here... https://github.com/jenkinsci/kubernetes-plugin/pull/393
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title Joseph Green commented on JENKINS-50429 Re: Shell command are really slower than before Yeah I saw those comments. I will test the example PR later and post my findings. I just wanted to make sure we have some concrete record of this having been tested and discussed. It seems that there is an issue in this area (as you found that commenting out this block sped up the pipeline), and this seemed like the natural place to start the investigation. If this proves to make no difference, I will look into other causes. Could the issues discussed in this SO post be a reasonable place too look next? https://stackoverflow.com/questions/28617175/did-i-find-a-bug-in-java-io-pipedinputstream. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title
Joseph Green edited a comment on JENKINS-50429
Re: Shell command are really slower than before
If its writing the env variables thats causing the problem, can we try doing something like this?{code:java}private void setupEnvironmentVariable(EnvVars vars, ExecWatch watch) throws IOException { StringBuilder envVars = new StringBuilder(); for (Map.Entry entry : vars.entrySet()) {//Check that key is bash compliant.if (entry.getKey().matches("[a-zA-Z_][a-zA-Z0-9_]*")) { envVars.append(String.format( "export %s='%s'%s", entry.getKey(), entry.getValue().replace("'", "'\\''"), NEWLINE ));} } watch.getInput().write(envVars.toString());}{code} I've created a PR with an initial attempt at this. We need to address any failing tests as a result of this change. https://github.com/jenkinsci/kubernetes-plugin/pull/390
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title Joseph Green commented on JENKINS-50429 Re: Shell command are really slower than before Additionally, this post (also mentioned earlier in this thread) https://stackoverflow.com/questions/28617175/did-i-find-a-bug-in-java-io-pipedinputstream seems to suggest that `OutputStream.write()` can hang for up to a second and that calling `OutputStream.flush()` can help with this Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title
Joseph Green edited a comment on JENKINS-50429
Re: Shell command are really slower than before
If its writing the env variables thats causing the problem, can we try doing something like this?{code:java}private void setupEnvironmentVariable(EnvVars vars, ExecWatch watch) throws IOException { StringBuilder envVars = new StringBuilder(); for (Map.Entry entry : vars.entrySet()) {//Check that key is bash compliant.if (entry.getKey().matches("[a-zA-Z_][a-zA-Z0-9_]*")) { envVars.append(String.format( "export %s='%s'%s", entry.getKey(), entry.getValue().replace("'", "'\\''"), NEWLINE ));} } watch.getInput().write(envVars.toString()) .getBytes(StandardCharsets.UTF_8) ;}{code}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title
Joseph Green edited a comment on JENKINS-50429
Re: Shell command are really slower than before
If its writing the env variables thats causing the problem, can we try doing something like this?{code:java}private void setupEnvironmentVariable(EnvVars vars, ExecWatch watch) throws IOException { StringBuilder envVars = new StringBuilder(); for (Map.Entry entry : vars.entrySet()) {//Check that key is bash compliant.if (entry.getKey().matches("[a-zA-Z_][a-zA-Z0-9_]*")) { envVars.append(String.format( "export %s='%s'%s", entry.getKey(), entry.getValue().replace("'", "'\\''"), NEWLINE ));} } watch.getInput().write(envVars.toString()).getBytes(StandardCharsets.UTF_8);}{code}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title
Joseph Green commented on JENKINS-50429
Re: Shell command are really slower than before
If its writing the env variables thats causing the problem, can we try doing something like this?
private void setupEnvironmentVariable(EnvVars vars, ExecWatch watch) throws IOException {
StringBuilder envVars = new StringBuilder();
for (Map.Entry entry : vars.entrySet()) {
//Check that key is bash compliant.
if (entry.getKey().matches("[a-zA-Z_][a-zA-Z0-9_]*")) {
envVars.append(String.format(
"export %s='%s'%s",
entry.getKey(),
entry.getValue().replace("'", "'\\''"),
NEWLINE
));
}
}
watch.getInput().write(envVars.toString()).getBytes(StandardCharsets.UTF_8);
}
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-50429) Shell command are really slower than before
Title: Message Title Joseph Green commented on JENKINS-50429 Re: Shell command are really slower than before We are also experiencing this issue, and have spent time recreating the test pipeline to highlight the overhead of commands ran inside containers - the overhead is around 3 - 4 seconds. Our pipeline runs many shell commands and so this time adds up to be quite significant. As others have suggested, the regression seems to be between v1.2.0 and v1.2.1. Pascal Laporte your suggestion seems sensible... do you know of any existing effort towards a solution for the Watchexec PipeInput performance? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
