Re: Update center generation has been disabled
On 27.05.2014, at 17:25, Slide slide.o@gmail.com wrote: +10 plugins should be self contained as much as possible. JENKINS-17134 mentions a related issue: There are different description sources used for installed and available plugins, leading to confusion. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
Ah, that's why our releases aren't showing up. But what exact part of the update center generation causes load on Jira/Confluence? It looked like the job shouldn't mess a lot with it, just request each plugin's page once. There are lots of com.atlassian.confluence.rpc.RemoteException: You're not allowed to view that page, or it does not exist. messages in the log for the pages that are generally publicly accessible. On Tuesday, May 27, 2014 2:41:42 AM UTC+4, R Tyler Croy wrote: See this ticket for more details https://issues.jenkins-ci.org/browse/INFRA-70 I'm waiting to hear back from KK on how we can fix the update-center generation scripts from scraping/DoSing our JIRA instance before I re-enable the job. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
Why would the update center access JIRA or confluence at all? To display the data on confluence the flow is right in the other direction, confluence connects to the update center to get the info. And JIRA, hmm, I have no idea why JIRA would be used for anything here... Domi On 27.05.2014, at 08:46, Ivan Kalinin pupss...@gmail.com wrote: Ah, that's why our releases aren't showing up. But what exact part of the update center generation causes load on Jira/Confluence? It looked like the job shouldn't mess a lot with it, just request each plugin's page once. There are lots of com.atlassian.confluence.rpc.RemoteException: You're not allowed to view that page, or it does not exist. messages in the log for the pages that are generally publicly accessible. On Tuesday, May 27, 2014 2:41:42 AM UTC+4, R Tyler Croy wrote: See this ticket for more details https://issues.jenkins-ci.org/browse/INFRA-70 I'm waiting to hear back from KK on how we can fix the update-center generation scripts from scraping/DoSing our JIRA instance before I re-enable the job. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
The update center accesses confluence to get the {excerpt} stuff for the
plugin descriptions.
The (as KK likes to call it NIH) CloudBees update center that we use also
scrapes Confluence... but we only scrape a plugin when we have a new
version of that plugin. If scraping fails we fall back to the description
in the pom and don't repeat scraping again until there is a new version.
We also do other crazy things like being able to generate the update
center JSON on the fly so we can offer different content for LTS lines vs
the TIP line from the same URL (which is actually a redirect to the
permanent URL that has very high cache control for the specific JSON
content being served)
Madness I know!
OTOH we have more lines of Jenkins that we have to support and consequently
less load per line whereas Jenkins OSS has the mirrors which favours static
content.
The real issue here is that the Jenkins OSS update center does not use a
database-like[1] back-end to store the data so it has to scrape *every*
plugin *every* time it runs.
-Stephen
[1] does not actually need to be a formal database, it could be a cache
file on disk for example... if the cache is missing then hammer confluence
away!
On 27 May 2014 08:08, domi d...@fortysix.ch wrote:
Why would the update center access JIRA or confluence at all?
To display the data on confluence the flow is right in the other
direction, confluence connects to the update center to get the info. And
JIRA, hmm, I have no idea why JIRA would be used for anything here…
Domi
On 27.05.2014, at 08:46, Ivan Kalinin pupss...@gmail.com wrote:
Ah, that's why our releases aren't showing up.
But what exact part of the update center generation causes load on
Jira/Confluence? It looked like the job shouldn't mess a lot with it, just
request each plugin's page once.
There are lots of com.atlassian.confluence.rpc.RemoteException: You're
not allowed to view that page, or it does not exist. messages in the log
for the pages that are generally publicly accessible.
On Tuesday, May 27, 2014 2:41:42 AM UTC+4, R Tyler Croy wrote:
See this ticket for more details https://issues.jenkins-ci.
org/browse/INFRA-70
I'm waiting to hear back from KK on how we can fix the update-center
generation
scripts from scraping/DoSing our JIRA instance before I re-enable the
job.
- R. Tyler Croy
--
Code: https://github.com/rtyler
Chatter: https://twitter.com/agentdero
% gpg --keyserver keys.gnupg.net --recv-key 3F51E16F
--
--
You received this message because you are subscribed to the Google Groups
Jenkins Developers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
Jenkins Developers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
Jenkins Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
On Tue, May 27, 2014 at 4:13 AM, Stephen Connolly
stephen.alan.conno...@gmail.com wrote:
The update center accesses confluence to get the {excerpt} stuff for the
plugin descriptions.
And, apparently, to find the list of wiki page tags by which to
classify the plugin. This seems dumb. To continue domi’s post, IMHO
everything needed for generation of an UC ought to be in the
*.hpi/META-INF/MANIFEST.MF. For example, regarding the excerpt, you
could simply pull out the Specification-Title (taken from
pom.xml#/project/description), or load /index.jelly which is what is
rendered in Plugin Manager. Why do you need the excerpt from the wiki?
If anything, the wiki macro should display an excerpt taken from the
plugin.
--
You received this message because you are subscribed to the Google Groups
Jenkins Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
Well the classification needs to be editable after the fact of the release.
I am with you on the excerpt though
On 27 May 2014 16:16, Jesse Glick jgl...@cloudbees.com wrote:
On Tue, May 27, 2014 at 4:13 AM, Stephen Connolly
stephen.alan.conno...@gmail.com wrote:
The update center accesses confluence to get the {excerpt} stuff for the
plugin descriptions.
And, apparently, to find the list of wiki page tags by which to
classify the plugin. This seems dumb. To continue domi’s post, IMHO
everything needed for generation of an UC ought to be in the
*.hpi/META-INF/MANIFEST.MF. For example, regarding the excerpt, you
could simply pull out the Specification-Title (taken from
pom.xml#/project/description), or load /index.jelly which is what is
rendered in Plugin Manager. Why do you need the excerpt from the wiki?
If anything, the wiki macro should display an excerpt taken from the
plugin.
--
You received this message because you are subscribed to the Google Groups
Jenkins Developers group.
To unsubscribe from this group and stop receiving emails from it, send an
email to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
Jenkins Developers group.
To unsubscribe from this group and stop receiving emails from it, send an email
to jenkinsci-dev+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
(replies inline) On Tue, 27 May 2014, Stephen Connolly wrote: OTOH we have more lines of Jenkins that we have to support and consequently less load per line whereas Jenkins OSS has the mirrors which favours static content. The real issue here is that the Jenkins OSS update center does not use a database-like[1] back-end to store the data so it has to scrape *every* plugin *every* time it runs. Is there anybody that wants to volunteer to take this work on sooner rather than later? I'd like to turn the UC back on without taking JIRA and Confluence offline which consumes lots of my time (I lost ~2 hours friday figuring this out). If there's not enough information in the repository, I'd be happy to help provide a DB or file dump of the data from confluence that's needed [1] does not actually need to be a formal database, it could be a cache file on disk for example... if the cache is missing then hammer confluence away! stahp. plz dont hammer On 27 May 2014 08:08, domi d...@fortysix.ch wrote: Why would the update center access JIRA or confluence at all? To display the data on confluence the flow is right in the other direction, confluence connects to the update center to get the info. And JIRA, hmm, I have no idea why JIRA would be used for anything here??? Domi On 27.05.2014, at 08:46, Ivan Kalinin pupss...@gmail.com wrote: Ah, that's why our releases aren't showing up. But what exact part of the update center generation causes load on Jira/Confluence? It looked like the job shouldn't mess a lot with it, just request each plugin's page once. There are lots of com.atlassian.confluence.rpc.RemoteException: You're not allowed to view that page, or it does not exist. messages in the log for the pages that are generally publicly accessible. On Tuesday, May 27, 2014 2:41:42 AM UTC+4, R Tyler Croy wrote: See this ticket for more details https://issues.jenkins-ci. org/browse/INFRA-70 I'm waiting to hear back from KK on how we can fix the update-center generation scripts from scraping/DoSing our JIRA instance before I re-enable the job. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- pgpcR7nM9lJ86.pgp Description: PGP signature
Re: Update center generation has been disabled
On Tue, May 27, 2014 at 12:13 PM, Stephen Connolly stephen.alan.conno...@gmail.com wrote: the classification needs to be editable after the fact of the release. Why? If you want to change it, just cut a new release, like you would for the plugin display name or anything else. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
On Tuesday, May 27, 2014 6:50:07 PM UTC+2, Jesse Glick wrote: On Tue, May 27, 2014 at 12:13 PM, Stephen Connolly stephen.al...@gmail.com javascript: wrote: the classification needs to be editable after the fact of the release. Why? If you want to change it, just cut a new release, like you would for the plugin display name or anything else. While I like the general idea of having a highly dynamic plug-in classification system based on tags in the wiki, there is another problem: The plug-in categories that are shown in the update center are hard-coded into the *messages*.properties* files in *core\src\main\resources\hudson\model* (e.g. UpdateCenter.PluginCategory.builder=Build Tools). When someone adds a new category-tag in the wiki, plug-ins still show up under misc in the update center. If the new category-tag gets added to Jenkins core eventually, only the following Jenkins releases show the new category. Everyone else still sees them under misc. So I'd suggest to pull this out of Jenkins core, either into a plug-in or some infra module that generates the update center. Apart from that I'm in favor of self-contained plug-ins. There might be some meta-data (e.g. some kind of plug-in rating) that should be stored independently. At the very least as a fallback (if the plug-in does not provide a description, category, then use the one stored in the update center/wiki), because cutting new releases for 800+ plug-ins just to update the meta-data might be a bit tedious. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
Seems crazy rolling a new release of a plugin just to alter metadata about the plugin. Our update center allows editing the metadata after the release (yes yes I know the UI doesn't let you have access to the metadata but the release notes git log import was more important ;-) ) On Tuesday, 27 May 2014, Jesse Glick jgl...@cloudbees.com wrote: On Tue, May 27, 2014 at 12:13 PM, Stephen Connolly stephen.alan.conno...@gmail.com javascript:; wrote: the classification needs to be editable after the fact of the release. Why? If you want to change it, just cut a new release, like you would for the plugin display name or anything else. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com javascript:;. For more options, visit https://groups.google.com/d/optout. -- Sent from my phone -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
On Tue, May 27, 2014 at 3:35 PM, FredG fred.g...@googlemail.com wrote: When someone adds a new category-tag in the wiki, plug-ins still show up under misc in the update center. According to the code, it should display as “Misc (the-new-category)”. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
Stephen and Jesse's observations are basically correct. We scrape Wiki to fill in a quick description of the plugin, categorizations, and so on. UC generator does not touch JIRA as far as I know. I think there are two issues here, one short-term and the other long-term. The short-term problem is that we need to get the update center generator code running ASAP, without assuming anything more from plugin metadata. That primarily involves separating Wiki crawler and UC generator, so that the former happens less frequently. I'll do that right away. The long-term problem is if such a scraping of Wiki is necessary to begin with, by making plugin metadata more descriptive and self-contained. I think we do need to define manifest entries for excerpt and labels, which isn't even possible today. That said, the motivation for scraping the Wiki was to crowd-source the categorizations of plugins, and I think that is still valid. I see 26 classifications in [1] and we want more categories (see relevant meta packaging discussion in JENKINS-9598), only allowing metadata in the *.jpi files to do this is wrong, IMHO. Plus it's not practical at this late in game when we have 800+ plugins that do not have such metadata to begin with. [1] https://wiki.jenkins-ci.org/display/JENKINS/Plugins On 05/26/2014 03:41 PM, R. Tyler Croy wrote: See this ticket for more details https://issues.jenkins-ci.org/browse/INFRA-70 I'm waiting to hear back from KK on how we can fix the update-center generation scripts from scraping/DoSing our JIRA instance before I re-enable the job. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
Can we use the description tag in pom.xml of the plugin to generate description ? On Monday, May 26, 2014 5:41:42 PM UTC-5, R Tyler Croy wrote: See this ticket for more details https://issues.jenkins-ci.org/browse/INFRA-70 I'm waiting to hear back from KK on how we can fix the update-center generation scripts from scraping/DoSing our JIRA instance before I re-enable the job. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
On Tue, May 27, 2014 at 5:11 PM, Kohsuke Kawaguchi kkawagu...@cloudbees.com wrote: I think we do need to define manifest entries for excerpt and labels, which isn't even possible today. As I have mentioned, the excerpt is generally identical or at least similar to the POM description or /index.jelly, so why not use those instead? (You need an index.jelly anyway.) the motivation for scraping the Wiki was to crowd-source the categorizations of plugins This does not make much sense to me. The plugin author is well placed to decide on a category, just as they would decide on a proper display name and so on. If the “crowd” wants to adjust that, well then we have pull requests. it's not practical at this late in game when we have 800+ plugins that do not have such metadata to begin with. I think it is practical. As a one-time batch task we scrape up existing tags and record those in the UC generation tool as historical fallback values; and edit the POMs of @jenkinsci plugins to define the current labels (just like Nicolas did a batch update of repo definitions a couple years back). Subsequent plugin releases would use the value defined in the POM. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
I've added caching. As a bonus, UC gen now runs in 3 min, whereas it used to take 20 mins. The cache expires after a day. Those two UC gen processes are scheduled to run 6 times a day, so we've gone from 12 full sweep of Wiki per day to 1 full sweep a day. Hopefully this will keep the usage under check. On 05/26/2014 03:41 PM, R. Tyler Croy wrote: See this ticket for more details https://issues.jenkins-ci.org/browse/INFRA-70 I'm waiting to hear back from KK on how we can fix the update-center generation scripts from scraping/DoSing our JIRA instance before I re-enable the job. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- -- Kohsuke Kawaguchi | CloudBees, Inc. | http://cloudbees.com/ Try Jenkins Enterprise, our professional version of Jenkins -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
On Tuesday, May 27, 2014 10:06:25 PM UTC+2, Jesse Glick wrote: On Tue, May 27, 2014 at 3:35 PM, FredG fred...@googlemail.comjavascript: wrote: When someone adds a new category-tag in the wiki, plug-ins still show up under misc in the update center. According to the code, it should display as “Misc (the-new-category)”. You are right, it actually does show up... 25 times Misc (some category) plus a real Miscellaneous category. Sorry, but this is still a crutch! m( -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Update center generation has been disabled
On Tue, May 27, 2014 at 6:56 PM, Kohsuke Kawaguchi kkawagu...@cloudbees.com wrote: the categorization comes after the plugin comes into being. Really? Whenever I have created a plugin I have set a category in the wiki as just one of several routine tasks. I'm all for allowing pluin authors to definitely describe excerpt, labels, and so on. But I also don't really see why we have to bend over backward to avoid hitting Confluence. For me that is a side benefit, but the real motivation is that I expect the official, signed update center JSON to be a simple function of the plugin artifacts that are fed into it from the Maven repository, without mysterious screen scraping of some wiki. If JENKINS-22926 is implemented and some extended columns in Plugin Manager want to show installation counts, ratings, etc. pulled from other databases or sites or astrological almanacks, that is fine. -- You received this message because you are subscribed to the Google Groups Jenkins Developers group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Update center generation has been disabled
See this ticket for more details https://issues.jenkins-ci.org/browse/INFRA-70 I'm waiting to hear back from KK on how we can fix the update-center generation scripts from scraping/DoSing our JIRA instance before I re-enable the job. - R. Tyler Croy -- Code: https://github.com/rtyler Chatter: https://twitter.com/agentdero % gpg --keyserver keys.gnupg.net --recv-key 3F51E16F -- pgplkJ16zaoX2.pgp Description: PGP signature
