RE: [qmailtoaster] Distressing strange behavior
Thanks Eric, I'll make that change. -Original Message- From: Eric Broch [mailto:ebr...@whitehorsetc.com] Sent: Monday, August 17, 2020 9:21 AM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior In your .qmail-default file for the domain it's recommended to use 'delete' instead of 'bounce-no-mailbox' On 8/17/2020 8:14 AM, Chas Hockenbarger wrote: > Thanks, Angus. I searched the whole system for a .forward and there aren't > any on the system I can find. > > I'm not seeing anything that is obvious here. I appreciate all the feedback > and help, there were definitely suggestions made I hadn't chased yet. I'm > perplexed to say the least. I deleted all the messages from the bounce queue > and will see if that rectifies the situation or not. I'm watching this > system like a hawk so hopefully if something that is more 'normal' looking is > going on I'll be able to catch it. > > If I find the culprit I'll absolutely update this thread. If anyone has any > other ideas, I'd love to hear them as well. > > -Original Message- > From: Angus McIntyre [mailto:an...@pobox.com] > Sent: Monday, August 17, 2020 5:44 AM > To: qmailtoaster-list@qmailtoaster.com; Chas Hockenbarger > Subject: Re: [qmailtoaster] Distressing strange behavior > > Check for a '.forward' file in '/root'? > > That could account for the status report going somewhere other than where > it's supposed to, but might not explain the other issues you're seeing. > > Angus > > > > Chas Hockenbarger wrote on 8/16/20 6:09 PM: >> I just got another piece of information. I got a failure message a >> few hours ago to the postmaster account for this domain that a message >> from root to root was not delivered to 5 different Gmail accounts. >> The email was the cron.daily status report. There is no way that >> should be going to these Gmail accounts. They are accounts I don’t >> know and root at this server is supposed to go to postmaster. >> >> This just keeps getting weirder. >> >> *From:* Eric Broch [mailto:ebr...@whitehorsetc.com] >> *Sent:* Sunday, August 16, 2020 4:13 PM >> *To:* qmailtoaster-list@qmailtoaster.com >> *Subject:* Re: [qmailtoaster] Distressing strange behavior >> >> Yes forwards can be in a .qmail file or in the vpopmail database. >> >> So, the bounces occurring presently, what's the originating account? >> >> Is there anything in your queue (# qmailctl queue)? >> >> On 8/16/2020 2:46 PM, Charles Hockenbarger wrote: >> >> As I understand the forwards setup in qmailadmin those are in the >> database, right? >> >> The address that was compromised hasn't sent any email since the >> password change. >> >> I hadn't thought about looking at qmail-inject. I'll dig into >> watching that part of the process. >> >> Get TypeApp for Android <http://www.typeapp.com/r?b=15986> >> >> On Aug 16, 2020, at 3:14 PM, Eric Broch > <mailto:ebr...@whitehorsetc.com>> wrote: >> >> How do you have your forwards set up? >> >> Is there any mail in your queue? >> >> If someone hacked an account on your server with forwards to >> gmail accounts they aren't limited to just these forwards, they >> also have the option in the email client to add gmail accounts >> in the "To:" field of the email they're sending, thus bounces >> from gmail accounts that aren't in your forwards file. >> >> Also, qmail-inject puts mail in the queue and you'll see it in >> the send log. >> >> On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: >> >> I'm hoping someone has encountered this weird behavior or >> something like it before and can point me down a path, >> because all my research has turned up nothing so far. >> >> I had an email account recently get breached due to a >> re-used password, and that account was used to send a bunch >> of spam out from a server I help manage. We changed the >> password on the account as soon as we found it happening and >> the outbound flood stopped. >> >> Shortly after that, however, I started seeing a very, very >> strange behavior. Sometimes, and I haven’t yet been able to >&
RE: [qmailtoaster] Distressing strange behavior
Thanks, Angus. I searched the whole system for a .forward and there aren't any on the system I can find. I'm not seeing anything that is obvious here. I appreciate all the feedback and help, there were definitely suggestions made I hadn't chased yet. I'm perplexed to say the least. I deleted all the messages from the bounce queue and will see if that rectifies the situation or not. I'm watching this system like a hawk so hopefully if something that is more 'normal' looking is going on I'll be able to catch it. If I find the culprit I'll absolutely update this thread. If anyone has any other ideas, I'd love to hear them as well. -Original Message- From: Angus McIntyre [mailto:an...@pobox.com] Sent: Monday, August 17, 2020 5:44 AM To: qmailtoaster-list@qmailtoaster.com; Chas Hockenbarger Subject: Re: [qmailtoaster] Distressing strange behavior Check for a '.forward' file in '/root'? That could account for the status report going somewhere other than where it's supposed to, but might not explain the other issues you're seeing. Angus Chas Hockenbarger wrote on 8/16/20 6:09 PM: > I just got another piece of information. I got a failure message a > few hours ago to the postmaster account for this domain that a message > from root to root was not delivered to 5 different Gmail accounts. > The email was the cron.daily status report. There is no way that > should be going to these Gmail accounts. They are accounts I don’t > know and root at this server is supposed to go to postmaster. > > This just keeps getting weirder. > > *From:* Eric Broch [mailto:ebr...@whitehorsetc.com] > *Sent:* Sunday, August 16, 2020 4:13 PM > *To:* qmailtoaster-list@qmailtoaster.com > *Subject:* Re: [qmailtoaster] Distressing strange behavior > > Yes forwards can be in a .qmail file or in the vpopmail database. > > So, the bounces occurring presently, what's the originating account? > > Is there anything in your queue (# qmailctl queue)? > > On 8/16/2020 2:46 PM, Charles Hockenbarger wrote: > > As I understand the forwards setup in qmailadmin those are in the > database, right? > > The address that was compromised hasn't sent any email since the > password change. > > I hadn't thought about looking at qmail-inject. I'll dig into > watching that part of the process. > > Get TypeApp for Android <http://www.typeapp.com/r?b=15986> > > On Aug 16, 2020, at 3:14 PM, Eric Broch <mailto:ebr...@whitehorsetc.com>> wrote: > > How do you have your forwards set up? > > Is there any mail in your queue? > > If someone hacked an account on your server with forwards to > gmail accounts they aren't limited to just these forwards, they > also have the option in the email client to add gmail accounts > in the "To:" field of the email they're sending, thus bounces > from gmail accounts that aren't in your forwards file. > > Also, qmail-inject puts mail in the queue and you'll see it in > the send log. > > On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: > > I'm hoping someone has encountered this weird behavior or > something like it before and can point me down a path, > because all my research has turned up nothing so far. > > I had an email account recently get breached due to a > re-used password, and that account was used to send a bunch > of spam out from a server I help manage. We changed the > password on the account as soon as we found it happening and > the outbound flood stopped. > > Shortly after that, however, I started seeing a very, very > strange behavior. Sometimes, and I haven’t yet been able to > identify the trigger or pattern, when users on this server > send email to a forward that contains around 50 or so email > addresses (they use it like a private distribution list) > they will get anywhere from 1-10 bounces from Gmail. Not > every email sent to the forward has this happen, and not > even every email from a particular user. > > The outbound spamming caused the server’s reputation to go > in the tank with Google, and if it weren’t for that, I > wouldn’t know this was happening, because they get the > bounces from Gmail accounts that absolutely ARE NOT in the > forward or part of the email chain AT ALL. > > I’m kind of fr
RE: [qmailtoaster] Distressing strange behavior
There are only 2 files found. One is in a user’s directory, and the file contains this line: /Maildir The other is in the top of the domain, labeled .qmail-default, which contains | /home/vpopmail/bin/vdelivermail ‘’ bounce-no-mailbox From: Eric Broch [mailto:ebr...@whitehorsetc.com] Sent: Sunday, August 16, 2020 7:40 PM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior Do this: # ls -la /home/vpopmail/domains/'mydomain'/postmaster/ look for a .qmail file. In fact you could do this # find /home/vpopmail/domains/ -name ".qmail*" The .qmail is also a way to forward. On 8/16/2020 4:49 PM, Chas Hockenbarger wrote: So I looked at a few of the files in the bounce folder and every one of them is bounces back from Gmail for either bad addresses or just the reputation bounce. Is there a down side to just blowing those away? From: Remo Mattei [mailto:r...@mattei.org] Sent: Sunday, August 16, 2020 5:43 PM To: qmailtoaster-list@qmailtoaster.com <mailto:qmailtoaster-list@qmailtoaster.com> Subject: Re: [qmailtoaster] Distressing strange behavior BTW, I always use the -L on the qmHandle it should not change much but my 2 cents. Remo On Aug 16, 2020, at 3:32 PM, Chas Hockenbarger mailto:chash...@gmail.com> > wrote: Yes, I did check those, that was my first thought is that the server had been compromised and someone modified those files to do some weird thing. However, .qmail-root has one line &postmaster@ .qmail-postmaster has one line &postmaster@ .qmail-mailer-daemon has one line &postmaster@ I see no other files in that directory. One more piece of info I just discovered. Even though qmHandle –l reports 0 messages in either the remote or local queue, the bounce queue directory has over 2000 messages in it. Could that be a contributing factor here? I don’t see how that would create random emails going to Gmail accounts from (seemingly) random other messages, but is it possible something is borked up in the queue processing there since Gmail is bouncing everything back to me? From: Remo Mattei [mailto:r...@mattei.org] Sent: Sunday, August 16, 2020 5:26 PM To: qmailtoaster-list@qmailtoaster.com <mailto:qmailtoaster-list@qmailtoaster.com> Subject: Re: [qmailtoaster] Distressing strange behavior did you check your qmail aliases? cd /var/qmail/alias/ what do those files say? On Aug 16, 2020, at 3:10 PM, Chas Hockenbarger < <mailto:chash...@gmail.com> chash...@gmail.com> wrote: Thanks, Boheme, and yes that’s a problem, but it’s a symptom of this problem. Emails are going to Gmail accounts when users aren’t sending them. Legit emails to Gmail accounts are definitely getting bounced, too, which I have to deal with later. If I can’t stop this weird spamming to them, I can’t recover the reputation. From: Boheme [ <mailto:boh...@gmail.com> mailto:boh...@gmail.com] Sent: Sunday, August 16, 2020 4:59 PM To: <mailto:qmailtoaster-list@qmailtoaster.com> qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior It doesn’t sound like you are being repeatedly hacked. It sounds like your reputation dropped with google, and certain emails trigger their anti-spam filtering now. Not all of them, just some. I have problems with Google accepting email regularly sometimes, and dropping other emails into people’s spam folders, as a result of too many of my users forwarding email to google and those forwards passing along a lot of spam to their addresses on my server. -Sent from my Pip-Boy 3000 On 17/08/2020, at 8:46 AM, Charles Hockenbarger < <mailto:chash...@gmail.com> chash...@gmail.com> wrote: As I understand the forwards setup in qmailadmin those are in the database, right? The address that was compromised hasn't sent any email since the password change. I hadn't thought about looking at qmail-inject. I'll dig into watching that part of the process. Get <http://www.typeapp.com/r?b=15986> TypeApp for Android On Aug 16, 2020, at 3:14 PM, Eric Broch < <mailto:ebr...@whitehorsetc.com> ebr...@whitehorsetc.com> wrote: How do you have your forwards set up? Is there any mail in your queue? If someone hacked an account on your server with forwards to gmail accounts they aren't limited to just these forwards, they also have the option in the email client to add gmail accounts in the "To:" field of the email they're sending, thus bounces from gmail accounts that aren't in your forwards file. Also, qmail-inject puts mail in the queue and you'll see it in the send log. On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: I'm hoping someone has encountered this weird behavior or something like it before and can point me down a path,
RE: [qmailtoaster] Distressing strange behavior
So I looked at a few of the files in the bounce folder and every one of them is bounces back from Gmail for either bad addresses or just the reputation bounce. Is there a down side to just blowing those away? From: Remo Mattei [mailto:r...@mattei.org] Sent: Sunday, August 16, 2020 5:43 PM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior BTW, I always use the -L on the qmHandle it should not change much but my 2 cents. Remo On Aug 16, 2020, at 3:32 PM, Chas Hockenbarger mailto:chash...@gmail.com> > wrote: Yes, I did check those, that was my first thought is that the server had been compromised and someone modified those files to do some weird thing. However, .qmail-root has one line &postmaster@ .qmail-postmaster has one line &postmaster@ .qmail-mailer-daemon has one line &postmaster@ I see no other files in that directory. One more piece of info I just discovered. Even though qmHandle –l reports 0 messages in either the remote or local queue, the bounce queue directory has over 2000 messages in it. Could that be a contributing factor here? I don’t see how that would create random emails going to Gmail accounts from (seemingly) random other messages, but is it possible something is borked up in the queue processing there since Gmail is bouncing everything back to me? From: Remo Mattei [mailto:r...@mattei.org] Sent: Sunday, August 16, 2020 5:26 PM To: qmailtoaster-list@qmailtoaster.com <mailto:qmailtoaster-list@qmailtoaster.com> Subject: Re: [qmailtoaster] Distressing strange behavior did you check your qmail aliases? cd /var/qmail/alias/ what do those files say? On Aug 16, 2020, at 3:10 PM, Chas Hockenbarger < <mailto:chash...@gmail.com> chash...@gmail.com> wrote: Thanks, Boheme, and yes that’s a problem, but it’s a symptom of this problem. Emails are going to Gmail accounts when users aren’t sending them. Legit emails to Gmail accounts are definitely getting bounced, too, which I have to deal with later. If I can’t stop this weird spamming to them, I can’t recover the reputation. From: Boheme [ <mailto:boh...@gmail.com> mailto:boh...@gmail.com] Sent: Sunday, August 16, 2020 4:59 PM To: <mailto:qmailtoaster-list@qmailtoaster.com> qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior It doesn’t sound like you are being repeatedly hacked. It sounds like your reputation dropped with google, and certain emails trigger their anti-spam filtering now. Not all of them, just some. I have problems with Google accepting email regularly sometimes, and dropping other emails into people’s spam folders, as a result of too many of my users forwarding email to google and those forwards passing along a lot of spam to their addresses on my server. -Sent from my Pip-Boy 3000 On 17/08/2020, at 8:46 AM, Charles Hockenbarger < <mailto:chash...@gmail.com> chash...@gmail.com> wrote: As I understand the forwards setup in qmailadmin those are in the database, right? The address that was compromised hasn't sent any email since the password change. I hadn't thought about looking at qmail-inject. I'll dig into watching that part of the process. Get <http://www.typeapp.com/r?b=15986> TypeApp for Android On Aug 16, 2020, at 3:14 PM, Eric Broch < <mailto:ebr...@whitehorsetc.com> ebr...@whitehorsetc.com> wrote: How do you have your forwards set up? Is there any mail in your queue? If someone hacked an account on your server with forwards to gmail accounts they aren't limited to just these forwards, they also have the option in the email client to add gmail accounts in the "To:" field of the email they're sending, thus bounces from gmail accounts that aren't in your forwards file. Also, qmail-inject puts mail in the queue and you'll see it in the send log. On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: I'm hoping someone has encountered this weird behavior or something like it before and can point me down a path, because all my research has turned up nothing so far. I had an email account recently get breached due to a re-used password, and that account was used to send a bunch of spam out from a server I help manage. We changed the password on the account as soon as we found it happening and the outbound flood stopped. Shortly after that, however, I started seeing a very, very strange behavior. Sometimes, and I haven’t yet been able to identify the trigger or pattern, when users on this server send email to a forward that contains around 50 or so email addresses (they use it like a private distribution list) they will get anywhere from 1-10 bounces from Gmail. Not every email sent to the forward has this happen, and not even every email from a particula
RE: [qmailtoaster] Distressing strange behavior
Yes, I did check those, that was my first thought is that the server had been compromised and someone modified those files to do some weird thing. However, .qmail-root has one line &postmaster@ .qmail-postmaster has one line &postmaster@ .qmail-mailer-daemon has one line &postmaster@ I see no other files in that directory. One more piece of info I just discovered. Even though qmHandle –l reports 0 messages in either the remote or local queue, the bounce queue directory has over 2000 messages in it. Could that be a contributing factor here? I don’t see how that would create random emails going to Gmail accounts from (seemingly) random other messages, but is it possible something is borked up in the queue processing there since Gmail is bouncing everything back to me? From: Remo Mattei [mailto:r...@mattei.org] Sent: Sunday, August 16, 2020 5:26 PM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior did you check your qmail aliases? cd /var/qmail/alias/ what do those files say? On Aug 16, 2020, at 3:10 PM, Chas Hockenbarger mailto:chash...@gmail.com> > wrote: Thanks, Boheme, and yes that’s a problem, but it’s a symptom of this problem. Emails are going to Gmail accounts when users aren’t sending them. Legit emails to Gmail accounts are definitely getting bounced, too, which I have to deal with later. If I can’t stop this weird spamming to them, I can’t recover the reputation. From: Boheme [mailto:boh...@gmail.com] Sent: Sunday, August 16, 2020 4:59 PM To: qmailtoaster-list@qmailtoaster.com <mailto:qmailtoaster-list@qmailtoaster.com> Subject: Re: [qmailtoaster] Distressing strange behavior It doesn’t sound like you are being repeatedly hacked. It sounds like your reputation dropped with google, and certain emails trigger their anti-spam filtering now. Not all of them, just some. I have problems with Google accepting email regularly sometimes, and dropping other emails into people’s spam folders, as a result of too many of my users forwarding email to google and those forwards passing along a lot of spam to their addresses on my server. -Sent from my Pip-Boy 3000 On 17/08/2020, at 8:46 AM, Charles Hockenbarger < <mailto:chash...@gmail.com> chash...@gmail.com> wrote: As I understand the forwards setup in qmailadmin those are in the database, right? The address that was compromised hasn't sent any email since the password change. I hadn't thought about looking at qmail-inject. I'll dig into watching that part of the process. Get <http://www.typeapp.com/r?b=15986> TypeApp for Android On Aug 16, 2020, at 3:14 PM, Eric Broch < <mailto:ebr...@whitehorsetc.com> ebr...@whitehorsetc.com> wrote: How do you have your forwards set up? Is there any mail in your queue? If someone hacked an account on your server with forwards to gmail accounts they aren't limited to just these forwards, they also have the option in the email client to add gmail accounts in the "To:" field of the email they're sending, thus bounces from gmail accounts that aren't in your forwards file. Also, qmail-inject puts mail in the queue and you'll see it in the send log. On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: I'm hoping someone has encountered this weird behavior or something like it before and can point me down a path, because all my research has turned up nothing so far. I had an email account recently get breached due to a re-used password, and that account was used to send a bunch of spam out from a server I help manage. We changed the password on the account as soon as we found it happening and the outbound flood stopped. Shortly after that, however, I started seeing a very, very strange behavior. Sometimes, and I haven’t yet been able to identify the trigger or pattern, when users on this server send email to a forward that contains around 50 or so email addresses (they use it like a private distribution list) they will get anywhere from 1-10 bounces from Gmail. Not every email sent to the forward has this happen, and not even every email from a particular user. The outbound spamming caused the server’s reputation to go in the tank with Google, and if it weren’t for that, I wouldn’t know this was happening, because they get the bounces from Gmail accounts that absolutely ARE NOT in the forward or part of the email chain AT ALL. I’m kind of freaking out here because while I haven’t found a breach of the actual server / OS, this feels like someone has been able to inject something somewhere into my server that I simply can’t find. It is especially troubling because a user who is not on this domain, but is part of the group and therefore uses the forward from time to time, sent something to the forward today and got Gmail bounces.
RE: [qmailtoaster] Distressing strange behavior
Thanks, Boheme, and yes that’s a problem, but it’s a symptom of this problem. Emails are going to Gmail accounts when users aren’t sending them. Legit emails to Gmail accounts are definitely getting bounced, too, which I have to deal with later. If I can’t stop this weird spamming to them, I can’t recover the reputation. From: Boheme [mailto:boh...@gmail.com] Sent: Sunday, August 16, 2020 4:59 PM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior It doesn’t sound like you are being repeatedly hacked. It sounds like your reputation dropped with google, and certain emails trigger their anti-spam filtering now. Not all of them, just some. I have problems with Google accepting email regularly sometimes, and dropping other emails into people’s spam folders, as a result of too many of my users forwarding email to google and those forwards passing along a lot of spam to their addresses on my server. -Sent from my Pip-Boy 3000 On 17/08/2020, at 8:46 AM, Charles Hockenbarger mailto:chash...@gmail.com> > wrote: As I understand the forwards setup in qmailadmin those are in the database, right? The address that was compromised hasn't sent any email since the password change. I hadn't thought about looking at qmail-inject. I'll dig into watching that part of the process. Get TypeApp for Android <http://www.typeapp.com/r?b=15986> On Aug 16, 2020, at 3:14 PM, Eric Broch mailto:ebr...@whitehorsetc.com> > wrote: How do you have your forwards set up? Is there any mail in your queue? If someone hacked an account on your server with forwards to gmail accounts they aren't limited to just these forwards, they also have the option in the email client to add gmail accounts in the "To:" field of the email they're sending, thus bounces from gmail accounts that aren't in your forwards file. Also, qmail-inject puts mail in the queue and you'll see it in the send log. On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: I'm hoping someone has encountered this weird behavior or something like it before and can point me down a path, because all my research has turned up nothing so far. I had an email account recently get breached due to a re-used password, and that account was used to send a bunch of spam out from a server I help manage. We changed the password on the account as soon as we found it happening and the outbound flood stopped. Shortly after that, however, I started seeing a very, very strange behavior. Sometimes, and I haven’t yet been able to identify the trigger or pattern, when users on this server send email to a forward that contains around 50 or so email addresses (they use it like a private distribution list) they will get anywhere from 1-10 bounces from Gmail. Not every email sent to the forward has this happen, and not even every email from a particular user. The outbound spamming caused the server’s reputation to go in the tank with Google, and if it weren’t for that, I wouldn’t know this was happening, because they get the bounces from Gmail accounts that absolutely ARE NOT in the forward or part of the email chain AT ALL. I’m kind of freaking out here because while I haven’t found a breach of the actual server / OS, this feels like someone has been able to inject something somewhere into my server that I simply can’t find. It is especially troubling because a user who is not on this domain, but is part of the group and therefore uses the forward from time to time, sent something to the forward today and got Gmail bounces. I don’t see anything in the send log that shows the server even trying to send to Gmail, which only adds to the ghost story. Any ideas, paths to go down, anything would be greatly appreciated here. I’m about to just rebuild the whole thing from scratch on a new VM, but if I’m overlooking something simple don’t want to put the users through that. Thanks in advance. Chas
RE: [qmailtoaster] Distressing strange behavior
I just got another piece of information. I got a failure message a few hours ago to the postmaster account for this domain that a message from root to root was not delivered to 5 different Gmail accounts. The email was the cron.daily status report. There is no way that should be going to these Gmail accounts. They are accounts I don’t know and root at this server is supposed to go to postmaster. This just keeps getting weirder. From: Eric Broch [mailto:ebr...@whitehorsetc.com] Sent: Sunday, August 16, 2020 4:13 PM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior Yes forwards can be in a .qmail file or in the vpopmail database. So, the bounces occurring presently, what's the originating account? Is there anything in your queue (# qmailctl queue)? On 8/16/2020 2:46 PM, Charles Hockenbarger wrote: As I understand the forwards setup in qmailadmin those are in the database, right? The address that was compromised hasn't sent any email since the password change. I hadn't thought about looking at qmail-inject. I'll dig into watching that part of the process. Get TypeApp for Android <http://www.typeapp.com/r?b=15986> On Aug 16, 2020, at 3:14 PM, Eric Broch mailto:ebr...@whitehorsetc.com> > wrote: How do you have your forwards set up? Is there any mail in your queue? If someone hacked an account on your server with forwards to gmail accounts they aren't limited to just these forwards, they also have the option in the email client to add gmail accounts in the "To:" field of the email they're sending, thus bounces from gmail accounts that aren't in your forwards file. Also, qmail-inject puts mail in the queue and you'll see it in the send log. On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: I'm hoping someone has encountered this weird behavior or something like it before and can point me down a path, because all my research has turned up nothing so far. I had an email account recently get breached due to a re-used password, and that account was used to send a bunch of spam out from a server I help manage. We changed the password on the account as soon as we found it happening and the outbound flood stopped. Shortly after that, however, I started seeing a very, very strange behavior. Sometimes, and I haven’t yet been able to identify the trigger or pattern, when users on this server send email to a forward that contains around 50 or so email addresses (they use it like a private distribution list) they will get anywhere from 1-10 bounces from Gmail. Not every email sent to the forward has this happen, and not even every email from a particular user. The outbound spamming caused the server’s reputation to go in the tank with Google, and if it weren’t for that, I wouldn’t know this was happening, because they get the bounces from Gmail accounts that absolutely ARE NOT in the forward or part of the email chain AT ALL. I’m kind of freaking out here because while I haven’t found a breach of the actual server / OS, this feels like someone has been able to inject something somewhere into my server that I simply can’t find. It is especially troubling because a user who is not on this domain, but is part of the group and therefore uses the forward from time to time, sent something to the forward today and got Gmail bounces. I don’t see anything in the send log that shows the server even trying to send to Gmail, which only adds to the ghost story. Any ideas, paths to go down, anything would be greatly appreciated here. I’m about to just rebuild the whole thing from scratch on a new VM, but if I’m overlooking something simple don’t want to put the users through that. Thanks in advance. Chas
RE: [qmailtoaster] Distressing strange behavior
Thanks Eric and Remo, I appreciate the assistance. I’d forgotten about the simscan setting for the cdb to up the logging, it’s been a LONG time since I’ve had to do that. My queue is empty. Nothing clogged up, it’s not residual stuff; that said, I’m watching it pretty closely right now. No .qmail files. I logged into the db and looked – the forward is all in the database, and I don’t have any .qmail files that I can find outside of the skel folder. My users aren’t ‘real’ users on the system, they’re all virtual users. Part of the problem is that the bouncing from Gmail has happened to different users at different times, and at other times it doesn’t happen to them. It is so very bizarre. Hopefully with an increased logging level I can find enough to trace this down to its actual origins. From: Eric Broch [mailto:ebr...@whitehorsetc.com] Sent: Sunday, August 16, 2020 4:13 PM To: qmailtoaster-list@qmailtoaster.com Subject: Re: [qmailtoaster] Distressing strange behavior Yes forwards can be in a .qmail file or in the vpopmail database. So, the bounces occurring presently, what's the originating account? Is there anything in your queue (# qmailctl queue)? On 8/16/2020 2:46 PM, Charles Hockenbarger wrote: As I understand the forwards setup in qmailadmin those are in the database, right? The address that was compromised hasn't sent any email since the password change. I hadn't thought about looking at qmail-inject. I'll dig into watching that part of the process. Get TypeApp for Android <http://www.typeapp.com/r?b=15986> On Aug 16, 2020, at 3:14 PM, Eric Broch mailto:ebr...@whitehorsetc.com> > wrote: How do you have your forwards set up? Is there any mail in your queue? If someone hacked an account on your server with forwards to gmail accounts they aren't limited to just these forwards, they also have the option in the email client to add gmail accounts in the "To:" field of the email they're sending, thus bounces from gmail accounts that aren't in your forwards file. Also, qmail-inject puts mail in the queue and you'll see it in the send log. On 8/16/2020 10:05 AM, Chas Hockenbarger wrote: I'm hoping someone has encountered this weird behavior or something like it before and can point me down a path, because all my research has turned up nothing so far. I had an email account recently get breached due to a re-used password, and that account was used to send a bunch of spam out from a server I help manage. We changed the password on the account as soon as we found it happening and the outbound flood stopped. Shortly after that, however, I started seeing a very, very strange behavior. Sometimes, and I haven’t yet been able to identify the trigger or pattern, when users on this server send email to a forward that contains around 50 or so email addresses (they use it like a private distribution list) they will get anywhere from 1-10 bounces from Gmail. Not every email sent to the forward has this happen, and not even every email from a particular user. The outbound spamming caused the server’s reputation to go in the tank with Google, and if it weren’t for that, I wouldn’t know this was happening, because they get the bounces from Gmail accounts that absolutely ARE NOT in the forward or part of the email chain AT ALL. I’m kind of freaking out here because while I haven’t found a breach of the actual server / OS, this feels like someone has been able to inject something somewhere into my server that I simply can’t find. It is especially troubling because a user who is not on this domain, but is part of the group and therefore uses the forward from time to time, sent something to the forward today and got Gmail bounces. I don’t see anything in the send log that shows the server even trying to send to Gmail, which only adds to the ghost story. Any ideas, paths to go down, anything would be greatly appreciated here. I’m about to just rebuild the whole thing from scratch on a new VM, but if I’m overlooking something simple don’t want to put the users through that. Thanks in advance. Chas
[qmailtoaster] Distressing strange behavior
I'm hoping someone has encountered this weird behavior or something like it before and can point me down a path, because all my research has turned up nothing so far. I had an email account recently get breached due to a re-used password, and that account was used to send a bunch of spam out from a server I help manage. We changed the password on the account as soon as we found it happening and the outbound flood stopped. Shortly after that, however, I started seeing a very, very strange behavior. Sometimes, and I haven't yet been able to identify the trigger or pattern, when users on this server send email to a forward that contains around 50 or so email addresses (they use it like a private distribution list) they will get anywhere from 1-10 bounces from Gmail. Not every email sent to the forward has this happen, and not even every email from a particular user. The outbound spamming caused the server's reputation to go in the tank with Google, and if it weren't for that, I wouldn't know this was happening, because they get the bounces from Gmail accounts that absolutely ARE NOT in the forward or part of the email chain AT ALL. I'm kind of freaking out here because while I haven't found a breach of the actual server / OS, this feels like someone has been able to inject something somewhere into my server that I simply can't find. It is especially troubling because a user who is not on this domain, but is part of the group and therefore uses the forward from time to time, sent something to the forward today and got Gmail bounces. I don't see anything in the send log that shows the server even trying to send to Gmail, which only adds to the ghost story. Any ideas, paths to go down, anything would be greatly appreciated here. I'm about to just rebuild the whole thing from scratch on a new VM, but if I'm overlooking something simple don't want to put the users through that. Thanks in advance. Chas
[qmailtoaster] CentOS 6 scripts?
I've searched the wiki and gotten zero results for the page Dan references in a couple of posts for his CentOS 6 scripts. I also don't find any script on mirror4. Apologies if I'm just blind or looking in the wrong place. Have these been taken down? I have to move a server to new hardware and standardization requirements force me to go to CentOS 6 so I was hoping to use these rather than having to go build a non-toaster qmail server from scratch (I've gotten spoiled by the toaster!). Thanks Chas
