[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Title: Message Title dogfood commented on JENKINS-24050 Re: All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Integrated in jenkins_main_trunk #4292 [JENKINS-23471 JENKINS-24050] (Revision 9c82fc42eb08b89047c544aaa586291ad1485472) Result = UNSTABLE ogondza : 9c82fc42eb08b89047c544aaa586291ad1485472 Files : pom.xml changelog.html Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
SCM/JIRA link daemon commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/9c82fc42eb08b89047c544aaa586291ad1485472 Log: JENKINS-23471 JENKINS-24050 Integrated the fix in remoting to Jenkins 1.580. (cherry picked from commit 8fc609fe0952b285d5b26a59fd5ff4c29704d33d) This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
SCM/JIRA link daemon commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Code changed in jenkins User: Kohsuke Kawaguchi Path: changelog.html pom.xml http://jenkins-ci.org/commit/jenkins/8fc609fe0952b285d5b26a59fd5ff4c29704d33d Log: JENKINS-23471 JENKINS-24050 Integrated the fix in remoting to Jenkins 1.580. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
dogfood commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Integrated in jenkins_main_trunk #3685 JENKINS-23471 JENKINS-24050 (Revision 8fc609fe0952b285d5b26a59fd5ff4c29704d33d) Result = SUCCESS kohsuke : 8fc609fe0952b285d5b26a59fd5ff4c29704d33d Files : changelog.html pom.xml This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Patricia Wright commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting After this change, the slaves no longer disconnect.Instead, the underlying issue causes the slaves to just stop doing whatever they were doing. Running jobs on those slaves hang forever and can not be cancelled. The Jenkins server starts to spam this in the logs until the filesystem fills up: Sep 11, 2014 10:38:13 AM org.kohsuke.stapler.export.Property writeValue WARNING: null org.kohsuke.stapler.export.NotExportableException: class hudson.plugins.parameterizedtrigger.CapturedEnvironmentAction doesn't have @ExportedBean so cannot write hudson.model.Actionable.actions at org.kohsuke.stapler.export.Model.init(Model.java:73) at org.kohsuke.stapler.export.ModelBuilder.get(ModelBuilder.java:51) at org.kohsuke.stapler.export.Property.writeValue(Property.java:231) at org.kohsuke.stapler.export.Property.writeValue(Property.java:187) at org.kohsuke.stapler.export.Property.writeValue(Property.java:139) at org.kohsuke.stapler.export.Property.writeTo(Property.java:116) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:190) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:185) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:185) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:185) at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:185) at org.kohsuke.stapler.export.Model.writeTo(Model.java:157) at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:267) at hudson.model.Api.doPython(Api.java:216) at sun.reflect.GeneratedMethodAccessor387.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:622) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:120) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:210) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.MetaClass$12.dispatch(MetaClass.java:390) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:248) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:728) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:858) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:631) at org.kohsuke.stapler.Stapler.service(Stapler.java:225) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:58) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:99) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Daniel Beck commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Patricia Wright: The stack traces and log size are a completely unrelated issue (note that this is about HTTP and the Python API), see JENKINS-24458 and issues linked from there. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
SCM/JIRA link daemon commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Code changed in jenkins User: Kohsuke Kawaguchi Path: src/main/java/org/jenkinsci/remoting/nio/NioChannelHub.java http://jenkins-ci.org/commit/remoting/1083a97145b83f88d9eee0a920a9495e192cd480 Log: FIXED JENKINS-24050 don't let canceled keys kill the selector thread In looking at the proposed PR #24 (https://github.com/jenkinsci/remoting/pull/24), I feel bit uneasy to mask the problem like it does. The code in question is looping through selected keys and processing it one by one. The only code that calls key.cancel() is done from the selector thread that runs this loop. So I don't understand how it is possible that the key picked up from selected key set is already cancelled here. I wonder if something more is going on. Regardless, I agree that this shouldn't kill the selector thread, which breaks all the slaves in one go. This change flags and reports the problem, kill the connection related to that key, then continue to serve other connections. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
SCM/JIRA link daemon resolved JENKINS-24050 as Fixed All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Change By: SCM/JIRA link daemon (05/Sep/14 9:16 PM) Status: Open Resolved Resolution: Fixed This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting To fill this in I hear Koshuke is on break so a remoting release wont happen until he comes back (I don't know when that is). Is there any way to hack in a new remoting jar into our production Jenkins (without building all of Jenkins)? Is this wise (we're not on the latest jenkins so are these things cross compatible, additionally will update continue to work)? Basically we're getting crashes a 1-3 times a day, just wondering if there's anything else anyone would recommend so we could get back to normal faster. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting If it wasn't clear (re-reading my last message I guess it wasn't); yes this represents a different variant to issue initially presented in JENKINS-22932, basically a different exception get's thrown which I think I've fixed in the pull request above, it's probably possible to refactor the fix for both issues in such a way that other exceptions don't kill the main NIO/select loop in the future but this specific issue can be fixed without that (it's easier for you guys to review this anyways). This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder edited a comment on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting @James: So I think the closed channel exception is actually closer to the Jenkins-22932 bug (if so you should repopen, since I had reopened before realizing I had a different root cause I then closed). However one could argue that the "selector" loop should actually catch all NIO errors and try again instead of it's current behavior of killing the loop entirely so it might be the case that the fix ends up being the same. Additionally I've implemented a patch that implements the key.isValid() check above: -https://github.com/kbrowder/remoting/commit/e2fae6f798480cef1753cd3a1fe8d49d7dcd528e- (making a new fork, one moment) I've not had a chance to test this out though (not sure how I would, need to figure out the unittests). Additionally I think changing how the error checking is done might be a safer strategy and it might fix both issues. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder edited a comment on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting @James: So I think the closed channel exception is actually closer to the Jenkins-22932 bug (if so you should repopen, since I had reopened before realizing I had a different root cause I then closed). However one could argue that the "selector" loop should actually catch all NIO errors and try again instead of it's current behavior of killing the loop entirely so it might be the case that the fix ends up being the same. Additionally I've implemented a patch that implements the key.isValid() check above: github.com/kbrowder/remoting/commit/e2fae6f798480cef1753cd3a1fe8d49d7dcd528e (making a new fork, one moment) I've not had a chance to test this out though (not sure how I would, need to figure out the unittests). Additionally I think changing how the error checking is done might be a safer strategy and it might fix both issues. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Jesse Glick commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Assuming the purported fix in JENKINS-22932 did in fact correct at least some variants of the bug, it should be left closed; if this issue represents some other variants, then fine—a follow-up fix can close this one, and it can be backported separately if marked lts-candidate. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
James Noonan commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting We updated to take in fix 22932 today. If the issue reoccurs for us, I'll raise a new defect. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder edited a comment on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting @James: So I think the closed channel exception is actually closer to the Jenkins-22932 bug (if so you should repopen, since I had reopened before realizing I had a different root cause I then closed). However one could argue that the "selector" loop should actually catch all NIO errors and try again instead of it's current behavior of killing the loop entirely so it might be the case that the fix ends up being the same. Additionally I've implemented a patch that implements the key.isValid() check above: https://github.com/kbrowder/remoting/commit/d52cef17a789bac0d1478c561c6696a82eb9ab6a Additionally I've got another change that captures CancelledKeyExceptions: https://github.com/kbrowder/remoting/commit/1dc29075e26c382b593d189a3a04cd1ab859f7c5 Actually I think with some minor modification you could extend this last approach to catch a number of potential pitfalls This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder edited a comment on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting @James: So I think the closed channel exception is actually closer to the Jenkins-22932 bug (if so you should repopen, since I had reopened before realizing I had a different root cause I then closed). However one could argue that the "selector" loop should actually catch all NIO errors and try again instead of it's current behavior of killing the loop entirely so it might be the case that the fix ends up being the same. Additionally I've implemented a patch that implements the key.isValid() check above: https://github.com/kbrowder/remoting/commit/d52cef17a789bac0d1478c561c6696a82eb9ab6a Additionally I've got another change that captures CancelledKeyExceptions: https://github.com/kbrowder/remoting/commit/1dc29075e26c382b593d189a3a04cd1ab859f7c5 And have filed a pull request: https://github.com/jenkinsci/remoting/pull/24; I freely admit having both is a bit paranoid but it seems like it's reasonably OK? Actually I think with some minor modification you could extend this last approach to catch a number of potential pitfalls This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting I have filed a pull request: https://github.com/jenkinsci/remoting/pull/24 Some might consider avoiding the error and catching it a bit paranoid, but I'm not entirely sure about concurrency issues with NIO. Additionally I still don't have a good test for this, I guess you'd need to cancel the key before calling key.isReadable(), but there's probably a very narrow window there. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder updated JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting Change By: Kevin Browder (31/Jul/14 8:29 PM) Summary: Allslavesdisconnectandnonewslavescanconnect dueto CancelledKeyExceptioninorg.jenkinsci.remoting This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [core] (JENKINS-24050) All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting
Kevin Browder commented on JENKINS-24050 All slaves disconnect and no new slaves can connect due to CancelledKeyException in org.jenkinsci.remoting @James: So I think the closed channel exception is actually closer to the Jenkins-22932 bug (if so you should repopen, since I had reopened before realizing I had a different root cause I then closed). However one could argue that the "selector" loop should actually catch all NIO errors and try again instead of it's current behavior of killing the loop entirely so it might be the case that the fix ends up being the same. Additionally I've implemented a patch that implements the key.isValid() check above: https://github.com/kbrowder/remoting/commit/e2fae6f798480cef1753cd3a1fe8d49d7dcd528e I've not had a chance to test this out though (not sure how I would, need to figure out the unittests). Additionally I think changing how the error checking is done might be a safer strategy and it might fix both issues. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
