[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Jim Borden commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Another +1 for firewalls! Our only remedy to poll and the whole arbitrary 1 day poll on a job quite a ridiculous way to set things up at that. It took me days to figure out that there were two separate polls going on and left me wondering why I had to constantly click "scan organization now" to get the results I wanted. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.187771.1516099379000.217.1565397720794%40Atlassian.JIRA.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Kevin Wren commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository +1, I also cant use webhooks due to firewalls (and dont have admin access to my jenkins instance). I'm assuming this isnt being worked on after 7 months of low activity? As anybody found a workaround in in the repo JenkinsFile? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.187771.1516099379000.18155.1563820861461%40Atlassian.JIRA.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Anthony Benedetti commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Another +1 org that can't use webhooks. Any progress here? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-issues/JIRA.187771.1516099379000.11800.1561741200666%40Atlassian.JIRA. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Bryce Schober commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Count me as +1 org with a firewalled Jenkins instance that can't currently accept webhooks. On that point, does the Jenkins community have a guide for best-security-practices forwarding proxy setup for specific webhooks that orgs could follow? Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title James Hardwick commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Yea, we're in the same boat of keeping our Jenkins instance firewall'd off from the outside world, hence the polling. It's not a big deal for us since the Organization being polled has a fairly limited set of repos. Every company I've ever been with has used Jenkins this way, and usually has had very legitimate reasons for doing so. I think it'd be a mistake to have a default worldview of "webhooks everywhere". Just my $.02. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Andy Airey commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Sure Stephen Connolly, it's just that you were listing a limited set of use-cases. Wanted to share mine as well since you were trying to create the bigger picture here (which is appreciated). I agree with all your points, I wish already for more than a year now that I could use webhooks, but I simply cannot because it does not exist or is not worked on. (These are good reasons to change CI system btw ...). I cannot install my own custom plugins on Bitbucket server because I am not in control of that, corporate is and they don't allow us to. They only support the native webhooks. So yeah, you can argue that this is a problem within my corporate atmosphere - but I am fairly certain there are more businesses out there using Jenkins that are in a similar situation (be it with firewalls or what have you). I understand that you cannot support each and every environment but it comes down to "polling is dead" - we all wish it were, but it isn't. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Stephen Connolly commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Jim Klimov so the correct solution to that would be something like the GitHub SNS plugin (if it was updated to replicate events correctly... last time I checked it wasn't ) Andy Airey The point is that those are not issues for the branch-api plugin itself, they are issues with the SCM sources... or the SCMs themselves... or your environment. Polling is exceedingly expensive in terms of overhead to check and also in terms of delay of notification. Especially when you have an entire organization of repositories to scan. For something like GitHub or Bitbucket Cloud, because of API limits, polling may not even be in any way practicable. If you are really stuck, then there are other paths you can take, for example the Apache Software Foundation uses its own custom org folder and event forwarding... https://github.com/stephenc/asf-gitpubsub-jenkins-plugin (also see the video series where I show how this was developed https://www.youtube.com/watch?v=W8BRtJmq_2Y=PLvBBnHmZuNQLXcL6CBZPKbX9OstMNIhTV ... note that the video is real time development of the plugin. The only code I wrote was during the video recordings) Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Jim Klimov commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Then there's also a problem of the corporate networks. Our build farm may poll github, but won't be permitted to get webhooks or any other messages initiated from the internet, directly or through tunnels. Hmm... a case for email-hooks... rss-hooks... out of scope here though. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Andy Airey commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository If it were that simple ... I guess you did not read the linked issue Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Stephen Connolly commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Andy Airey then fix your webhooks Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Andy Airey commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository There's a third case for the need of periodic scanning of underlying multibranch projects; when webhooks don't work (JENKINS-47891). Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Stephen Connolly commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository There are two periodic scans "configured". One is user configurable, namely the one at the org folder level. The other is not currently configurable, namely the one at the branch folder level. Now the way this is designed is on the basis that you will set up webhooks because "polling must die" to quote a blog post from KK ages back. So the periodic scanning is more to cover two cases: 1. To allow controlled pruning of orphaned items. For technical reasons, the code was actually written originally to allow multiple sources in the same org folder. So you could have one source be a github organization and another be a bitbucket team. Then if you moved a repository from the bitbucket team to the github organization the multibranch project would still stay there. This was to mirror the fact that a multibranch project can have multiple sources. However this was too confusing, so we locked the org folders to one source only. In any case, because there originally could be (and for a multibranch project there still can be) multiple things that can claim ownership of a child name, we cannot decide if an event that a name has been deleted by one source will actually result in the deletion of the child. We have to ask all the candidate sources in the priority order: "is this yours?" until we get someone answering "yes"... So the orphaned item strategy only runs with a full scan. This also allows the orphaned item strategy to correctly decide the order in which orphans should be removed. 2. Events may be dropped or lost. We should find those changes eventually. The scan fills that gap. If you have webhooks set up correctly then you only need scanning say once per day or less frequently. All the cron triggers store a random offset as a transient field. This random offset is used to decide when in the `H` portion of a trigger rule to actually pick. The periodic folder trigger converts the times into rules like H * * * * (for hourly) or H H * * * (for daily) so this transient field is important that it remain consistent. What I think is happening is that the org folder scan is resetting the transient field. So if your org folder is scanning every 5 minutes, you will have a 5 in 60 chance that the transient gets reset to a value in the next 5 minutes. Of course we have two H's for the daily scan that the multibranch has. So basically if my theory that the transient is getting reset is correct then there is a 1 in 288 chance that the transient will have a value resulting in a scan during the next 5 minutes... and then the transient gets reset again. Each 5 minute interval gas a 287 in 288 chance of not scanning... so we are naïvely looking at the probability of a multibranch project not scanning being (287/288)^288 = 0.36... it's actually worse, because that is actually the probability of it checking whether a scan is due... and if the transient field for H is being cleared then so too will this one: https://github.com/jenkinsci/cloudbees-folder-plugin/blob/23367538c5c6b405a3ba63d8033ca459aa2f65a0/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L71 and so if we win the 63% lottery then one of those will run through to initialize lastTriggered to a random value https://github.com/jenkinsci/cloudbees-folder-plugin/blob/23367538c5c6b405a3ba63d8033ca459aa2f65a0/src/main/java/com/cloudbees/hudson/plugins/folder/computed/PeriodicFolderTrigger.java#L211-L217 So if my theory is correct then you would expect:
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title
James Hardwick edited a comment on JENKINS-48960
Re: Org folders repository folder never runs scans on repository
If it's any help, I had to setup a fresh Jenkins server for a new project. Again using GitHub Organization plugin to scan a repository for all available branches containing a Jenkinsfile. The repository scan worked the first time as expected, and then immediately started reproducing the original issue as described here. So, pretty easy to setup and reproduce as far as I can tell. I ended up just creating a job that contains [~tario]'s console script but instead as a pipeline script, so when the scan interval occasionally resets for whatever reason, I could just run the job and it's fixed. A la...{code:java}node { for (f in Jenkins.instance.getAllItems(jenkins.branch.MultiBranchProject.class)) {if (f.parent instanceof jenkins.branch.OrganizationFolder) { f.addTrigger(new com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger("5m"));} }}{code}[~stephenconnolly] I'm looking back and trying to understand your original request but not sure it makes sense. Can you break that down with a bit more detail as to just what exactly *should* be happening?
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title
James Hardwick commented on JENKINS-48960
Re: Org folders repository folder never runs scans on repository
If it's any help, I had to setup a fresh Jenkins server for a new project. Again using GitHub Organization plugin to scan a repository for all available branches containing a Jenkinsfile. The repository scan worked the first time as expected, and then immediately started reproducing the original issue as described here. So, pretty easy to setup and reproduce as far as I can tell. I ended up just creating a job that contains Patrick Ruckstuhl's console script but instead as a pipeline script, so when the scan interval occasionally resets for whatever reason, I could just run the job and it's fixed. A la...
node {
for (f in Jenkins.instance.getAllItems(jenkins.branch.MultiBranchProject.class)) {
if (f.parent instanceof jenkins.branch.OrganizationFolder) {
f.addTrigger(new com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger("5m"));
}
}
}
Stephen Connolly I'm looking back and trying to understand your original request but not sure it makes sense. Can you break that down with a bit more detail as to just what exactly should be happening?
Add Comment
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Craig Dunford commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository This is an issue for me. Patrick's workaround works for me, but is fairly cumbersome. What needs to be done to get this looked at and fixed. It is a fairly highly voted issue. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Patrick Artounian edited a comment on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Yeah this is still an issue . , basically a blocker for my team Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Patrick Artounian commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Yeah this is still an issue. Add Comment This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title
Andy Airey edited a comment on JENKINS-48960
Re: Org folders repository folder never runs scans on repository
Tried [~tario] workaround againd again and got it working this time .Was trying [~jimklimov]'s one first but that didn't seem to find any jobs for me.Also, it requires the GroovyPostBuild plugin or it will not be able to use {{manager}}.
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Andy Airey commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Tried Patrick Ruckstuhl workaround againd and got it working. Was trying Jim Klimov's one first but that didn't seem to find any jobs for me. Also, it requires the GroovyPostBuild plugin or it will not be able to use manager. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Jim Klimov commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Hello all, I posted a similar issue some time ago (now linked it as a duplicate of this one and closed); while waiting for a better solution, I can offer this workaround: to add a job to regularly rescan MBPs that are inside folders : https://pastebin.com/xKM7CStM It still is sub-optimal and has its quirks, but at least seems to work. Beware to not set it to too-often, or you could hit Github REST API rate limiting for example. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Tristan Cole commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Andy Airey I am in the process of setting up a Bitbucket Team Project with the Bitbucket Branch Source plugin and I ran into this issue. I tried Patrick Ruckstuhl's fix and it worked for me. Otherwise, you could access the Jenkins server and edit the multibranch job configs manually (read: via script) and reload configuration from disk. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Andy Airey commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Patrick Ruckstuhl That workaround does not work for me, also using Bitbucket Branch Source plugin. Is there any progress on solving the core of the issue? This is kind of blocking. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title
Patrick Ruckstuhl commented on JENKINS-48960
Re: Org folders repository folder never runs scans on repository
Found an easier workaround than to manually edit all the configs. Using the script console, this will change all the triggers to 5 minutes for the folders inside Organizations
for (f in Jenkins.instance.getAllItems(jenkins.branch.MultiBranchProject.class)) {
if (f.parent instanceof jenkins.branch.OrganizationFolder) {
f.addTrigger(new com.cloudbees.hudson.plugins.folder.computed.PeriodicFolderTrigger("5m"));
}
}
Add Comment
This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-48960) Org folders repository folder never runs scans on repository
Title: Message Title Patrick Ruckstuhl commented on JENKINS-48960 Re: Org folders repository folder never runs scans on repository Having the same issue (with bitbucket branch source instead of github). I also tried to set it through one of the jobs with pipelineTriggers([[$class: 'PeriodicFolderTrigger', interval: '5m']]), But didn't work either. Add Comment This message was sent by Atlassian JIRA (v7.3.0#73011-sha1:3c73d0e) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
