Re: [ossec-list] Debugging Unprocessed Log Entries
Thanks Dan. Is there a way to get OSSEC to provide more details on the
messages it actually processes? I'd like to gain a better understanding of
this application because it has a lot of seemingly random behaviour.
On Thursday, February 9, 2017 at 9:59:24 PM UTC+2, dan (ddpbsd) wrote:
>
> On Thu, Feb 9, 2017 at 9:48 AM, Quintin Beukes > wrote:
> > Hi group,
> >
> > Server uname: Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11
> 20:56:24
> > UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
> > Agent uname: Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11 20:56:24
> UTC
> > 2017 x86_64 x86_64 x86_64 GNU/Linux
> >
> > I am generating 5 log messages at 2 second intervals to trigger rule
> 1002.
> > 16:40:43 [quintinb@ho-pri-vm-quintindev ~]$ for x in {11..15}; do logger
> > test error$x; date; sleep 2; done
> > Thu Feb 9 16:40:48 SAST 2017
> > Thu Feb 9 16:40:50 SAST 2017
> > Thu Feb 9 16:40:52 SAST 2017
> > Thu Feb 9 16:40:54 SAST 2017
> > Thu Feb 9 16:40:56 SAST 2017
> > A tcpdump on the server indicates all 5 are received:
> > 16:40:49.197974 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length
> 121
> > 16:40:51.200118 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length
> 121
> > 16:40:53.202224 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length
> 121
> > 16:40:55.204480 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length
> 121
> > 16:40:57.206407 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length
> 121
> >
> > Though alerts.log only shows 3 of the 5.
> > ** Alert 1486651295.2432248: mail - syslog,errors,
> > 2017 Feb 09 16:41:35 (quintindev) 10.10.10.100->/var/log/messages
> > Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.'
> > Feb 9 16:40:48 ho-pri-vm-quintindev quintinb: test error11
> >
> > ** Alert 1486651298.2432494: mail - syslog,errors,
> > 2017 Feb 09 16:41:38 (quintindev) 10.10.10.100->/var/log/messages
> > Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.'
> > Feb 9 16:40:52 ho-pri-vm-quintindev quintinb: test error13
> >
> > ** Alert 1486651305.2432740: mail - syslog,errors,
> > 2017 Feb 09 16:41:45 (quintindev) 10.10.10.100->/var/log/messages
> > Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.'
> > Feb 9 16:40:56 ho-pri-vm-quintindev quintinb: test error15
> >
> >
> > Sometimes it alerts on all 5. Though upon inspection it seems OSSEC
> misses
> > 50%+ of my messages, even though I see the packets delivered to the
> server.
> >
> > Is there an explanation for this? Any way I can get more verbose
> logging on
> > this to investigate deeper?
> >
>
> OSSEC does discard some duplicate messages, and I'm not sure if the
> timestamp is taken into account or not off hand.
>
> > Quintin
> >
> > --
> >
> > ---
> > You received this message because you are subscribed to the Google
> Groups
> > "ossec-list" group.
> > To unsubscribe from this group and stop receiving emails from it, send
> an
> > email to ossec-list+...@googlegroups.com .
> > For more options, visit https://groups.google.com/d/optout.
>
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[ossec-list] Debugging Unprocessed Log Entries
Hi group,
Server uname: Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11 20:56:24
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Agent uname: Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11 20:56:24
UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
I am generating 5 log messages at 2 second intervals to trigger rule 1002.
16:40:43 [quintinb@ho-pri-vm-quintindev ~]$ for x in {11..15}; do logger
test error$x; date; sleep 2; done
Thu Feb 9 16:40:48 SAST 2017
Thu Feb 9 16:40:50 SAST 2017
Thu Feb 9 16:40:52 SAST 2017
Thu Feb 9 16:40:54 SAST 2017
Thu Feb 9 16:40:56 SAST 2017
A tcpdump on the server indicates all 5 are received:
16:40:49.197974 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length 121
16:40:51.200118 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length 121
16:40:53.202224 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length 121
16:40:55.204480 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length 121
16:40:57.206407 IP 10.10.10.100.41074 > 10.10.12.171.1514: UDP, length 121
Though alerts.log only shows 3 of the 5.
** Alert 1486651295.2432248: mail - syslog,errors,
2017 Feb 09 16:41:35 (quintindev) 10.10.10.100->/var/log/messages
Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.'
Feb 9 16:40:48 ho-pri-vm-quintindev quintinb: test error11
** Alert 1486651298.2432494: mail - syslog,errors,
2017 Feb 09 16:41:38 (quintindev) 10.10.10.100->/var/log/messages
Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.'
Feb 9 16:40:52 ho-pri-vm-quintindev quintinb: test error13
** Alert 1486651305.2432740: mail - syslog,errors,
2017 Feb 09 16:41:45 (quintindev) 10.10.10.100->/var/log/messages
Rule: 1002 (level 2) -> 'Unknown problem somewhere in the system.'
Feb 9 16:40:56 ho-pri-vm-quintindev quintinb: test error15
Sometimes it alerts on all 5. Though upon inspection it seems OSSEC misses
50%+ of my messages, even though I see the packets delivered to the server.
Is there an explanation for this? Any way I can get more verbose logging
on this to investigate deeper?
Quintin
--
---
You received this message because you are subscribed to the Google Groups
"ossec-list" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to ossec-list+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Re: [ossec-list] Debugging agent connectivity
The ownership and permissions are the same as yours. An unfortunate and rare event just occurred: all the agents are now showing online. This happens occasionally and sticks for a few days. I'll keep monitoring it and when the agents start giving problems again I'll refer back to this message. They're problematic most of the time, so I'm hoping for failures again soon. -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ossec-list] Debugging agent connectivity
Hi group, I'm trying to debug why my agent's are always showing disconnected. They would work for a bit, and then randomly stop working. Some agents will disconnect permanently, some intermittently switch between connected/disconnected. Any advice on how to increase logging verbosity or why my agents are not working properly. I enabled debugging which had no increase in logging verbosity. I did so by editing internal_options.conf and setting on server: remoted.debug=2 run "/var/ossec/bin/ossec-control enable debug" and restart service on agent: agent.debug=2, and restart service This is happening with many agents both outside and inside the OSSEC subnet. I disabled both iptables firewalls for this test. Server IP: 10.10.12.171 Agent IP: 10.10.12.170 Server uname: Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11 20:56:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Agent uname: Linux 2.6.32-642.13.1.el6.x86_64 #1 SMP Wed Jan 11 20:56:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux My agent always shows disconnected: ID: 003, Name: safetynet1, IP: 10.10.12.170, Disconnected The ossec server log doesn't show anything related. The ossec agent log just repeatedly shows: - 2017/02/08 12:20:29 ossec-agentd: INFO: Trying to connect to server ossec.jeoffice, port 1514. 2017/02/08 12:20:29 INFO: Connected to ossec.jeoffice at address 10.10.12.171, port 1514 2017/02/08 12:20:50 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: 'ossec.jeoffice'. - Content of server /etc/ossec-init.conf - DIRECTORY="/var/ossec" VERSION="2.9.0" DATE="Wed Jan 25 09:55:39 EST 2017" TYPE="server" - Content of server /etc/ossec-init.conf - DIRECTORY="/var/ossec" VERSION="2.9.0" DATE="Wed Jan 25 09:55:39 EST 2017" TYPE="agent" - A server tcpdump shows: - 14:14:54.281902 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:14:59.280963 ARP, Request who-has 10.10.12.171 tell 10.10.12.170, length 28 14:14:59.280987 ARP, Reply 10.10.12.171 is-at f2:1e:73:71:3e:c8, length 28 14:15:00.282405 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:04.282833 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:09.283445 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:15.284415 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:32.803559 IP 10.10.12.171.1514 > 10.10.12.170.50637: UDP, length 73 - An agent dump shows: - 14:14:54.280480 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:00.281305 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:04.281914 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:09.282433 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:15.283291 IP 10.10.12.170.50637 > 10.10.12.171.1514: UDP, length 73 14:15:32.803186 IP 10.10.12.171.1514 > 10.10.12.170.50637: UDP, length 73 - Quintin -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ossec-list] Agents going offline intermittnently
I have enabled the debug logging as you described, and additionally set the remoted.debug=2 and logcollector.debug=2 in internal_options.conf (was the latter even necessary)? I'll monitor the agents and report back here. Quintin On Wed, Jul 20, 2016 at 1:55 PM dan (ddp) wrote: > On Tue, Jul 19, 2016 at 10:19 AM, Quintin Beukes > wrote: > > The logs on the agent show this: > > 2016/07/19 16:18:27 ossec-agentd(4101): WARN: Waiting for server reply > (not > > started). Tried: 'ossec.jeoffice/10.10.12.171'. > > 2016/07/19 16:18:29 ossec-agentd: INFO: Trying to connect to server > > (ossec.jeoffice/10.10.12.171:1514). > > 2016/07/19 16:18:29 ossec-agentd: INFO: Using IPv4 for: 10.10.12.171 . > > 2016/07/19 16:18:44 ossec-logcollector: WARN: Process locked. Waiting for > > permission... > > > > Try turning on debug on the manager (`/var/ossec/bin/ossec-control > enable debug && /var/ossec/bin/ossec-control restart`) > > > Quintin > > > > On Tue, Jul 19, 2016 at 4:13 PM Quintin Beukes > > wrote: > >> > >> Hi, > >> > >> A few days ago some of my OSSEC agents started going offline and stop > >> sending alerts, and then a long while after come back online again like > >> nothing's wrong. Restarting the agents don't help fix the offline > status. > >> This affects both agents running through a router/firewall to reach the > >> server, and agents running in the same subnet as the server. > >> > >> I removed all iptables filters and did a tcpdump on both offline and > >> online agents, but couldn't notice anything out of the ordinary. > >> > >> Here are packets from an offline agent showing successful traffic from > >> server to client and vice versa, as well as some curious port > unreachable > >> errors. Even though there is traffic, the agent shows as offline and no > >> alerts are generated for events on this agent. > >> > >> OSSEC Server IP: 10.10.12.171 > >> Agent IP: 10.10.13.8 > >> > >> agent_control -l: > >>ID: 019, Name: devjerm1, IP: 10.10.13.8, Disconnected > >> > >> tcpdump: > >> 15:47:36.515777 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port > >> 58989 unreachable, length 109 > >> 15:47:36.517646 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 > >> 15:47:40.526516 IP 10.10.12.171.1514 > 10.10.13.8.58989: UDP, length 73 > >> 15:47:40.526567 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port > >> 58989 unreachable, length 109 > >> 15:47:41.518182 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 > >> 15:47:47.518732 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 > >> 15:47:59.581518 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > >> 15:48:07.897110 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > >> 15:48:14.725335 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > >> 15:48:19.395627 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > >> 15:48:25.521404 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 > >> 15:48:31.522261 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 > >> 15:48:35.522794 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 > >> > >> Any insights are appreciated. > >> > >> Quintin > >> > >> -- > >> > >> --- > >> You received this message because you are subscribed to the Google > Groups > >> "ossec-list" group. > >> To unsubscribe from this group and stop receiving emails from it, send > an > >> email to ossec-list+unsubscr...@googlegroups.com. > >> For more options, visit https://groups.google.com/d/optout. > > > > -- > > > > --- > > You received this message because you are subscribed to the Google Groups > > "ossec-list" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to ossec-list+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > > --- > You received this message because you are subscribed to the Google Groups > "ossec-list" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ossec-list+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [ossec-list] Agents going offline intermittnently
The logs on the agent show this: 2016/07/19 16:18:27 ossec-agentd(4101): WARN: Waiting for server reply (not started). Tried: 'ossec.jeoffice/10.10.12.171'. 2016/07/19 16:18:29 ossec-agentd: INFO: Trying to connect to server (ossec.jeoffice/10.10.12.171:1514). 2016/07/19 16:18:29 ossec-agentd: INFO: Using IPv4 for: 10.10.12.171 . 2016/07/19 16:18:44 ossec-logcollector: WARN: Process locked. Waiting for permission... Quintin On Tue, Jul 19, 2016 at 4:13 PM Quintin Beukes wrote: > Hi, > > A few days ago some of my OSSEC agents started going offline and stop > sending alerts, and then a long while after come back online again like > nothing's wrong. Restarting the agents don't help fix the offline status. > This affects both agents running through a router/firewall to reach the > server, and agents running in the same subnet as the server. > > I removed all iptables filters and did a tcpdump on both offline and > online agents, but couldn't notice anything out of the ordinary. > > Here are packets from an offline agent showing successful traffic from > server to client and vice versa, as well as some curious port unreachable > errors. Even though there is traffic, the agent shows as offline and no > alerts are generated for events on this agent. > > OSSEC Server IP: 10.10.12.171 > Agent IP: 10.10.13.8 > > agent_control -l: >ID: 019, Name: devjerm1, IP: 10.10.13.8, Disconnected > > tcpdump: > 15:47:36.515777 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port > 58989 unreachable, length 109 > 15:47:36.517646 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 > 15:47:40.526516 IP 10.10.12.171.1514 > 10.10.13.8.58989: UDP, length 73 > 15:47:40.526567 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port > 58989 unreachable, length 109 > 15:47:41.518182 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 > 15:47:47.518732 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 > 15:47:59.581518 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > 15:48:07.897110 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > 15:48:14.725335 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > 15:48:19.395627 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 > 15:48:25.521404 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 > 15:48:31.522261 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 > 15:48:35.522794 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 > > Any insights are appreciated. > > Quintin > > -- > > --- > You received this message because you are subscribed to the Google Groups > "ossec-list" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to ossec-list+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ossec-list] Agents going offline intermittnently
Hi, A few days ago some of my OSSEC agents started going offline and stop sending alerts, and then a long while after come back online again like nothing's wrong. Restarting the agents don't help fix the offline status. This affects both agents running through a router/firewall to reach the server, and agents running in the same subnet as the server. I removed all iptables filters and did a tcpdump on both offline and online agents, but couldn't notice anything out of the ordinary. Here are packets from an offline agent showing successful traffic from server to client and vice versa, as well as some curious port unreachable errors. Even though there is traffic, the agent shows as offline and no alerts are generated for events on this agent. OSSEC Server IP: 10.10.12.171 Agent IP: 10.10.13.8 agent_control -l: ID: 019, Name: devjerm1, IP: 10.10.13.8, Disconnected tcpdump: 15:47:36.515777 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port 58989 unreachable, length 109 15:47:36.517646 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 15:47:40.526516 IP 10.10.12.171.1514 > 10.10.13.8.58989: UDP, length 73 15:47:40.526567 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port 58989 unreachable, length 109 15:47:41.518182 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 15:47:47.518732 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 15:47:59.581518 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:07.897110 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:14.725335 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:19.395627 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:25.521404 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 15:48:31.522261 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 15:48:35.522794 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 Any insights are appreciated. Quintin -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[ossec-list] Random OSSEC Agents Offline
Hi, A few days ago some of my OSSEC agents started going offline and stop sending alerts, and then a long while after come back online again like nothing's wrong. Restarting the agents don't help fix the offline status. This affects both agents running through a router/firewall to reach the server, and agents running in the same subnet as the server. I removed all iptables filters and did a tcpdump on both offline and online agents, but couldn't notice anything out of the ordinary. Here are packets from an offline agent showing successful traffic from server to client and vice versa, as well as some curious port unreachable errors. Even though there is traffic, the agent shows as offline and no alerts are generated for events on this agent. OSSEC Server IP: 10.10.12.171 Agent IP: 10.10.13.8 agent_control -l: ID: 019, Name: devjerm1, IP: 10.10.13.8, Disconnected tcpdump: 15:47:36.515777 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port 58989 unreachable, length 109 15:47:36.517646 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 15:47:40.526516 IP 10.10.12.171.1514 > 10.10.13.8.58989: UDP, length 73 15:47:40.526567 IP 10.10.13.8 > 10.10.12.171: ICMP 10.10.13.8 udp port 58989 unreachable, length 109 15:47:41.518182 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 15:47:47.518732 IP 10.10.13.8.49382 > 10.10.12.171.1514: UDP, length 73 15:47:59.581518 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:07.897110 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:14.725335 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:19.395627 IP 10.10.12.171.1514 > 10.10.13.8.49382: UDP, length 73 15:48:25.521404 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 15:48:31.522261 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 15:48:35.522794 IP 10.10.13.8.59490 > 10.10.12.171.1514: UDP, length 73 Any insights are appreciated. Quintin -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to ossec-list+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
