Re: [outages] Problem with credit card machine processing? "Datawire"

[Frank Bulk via Outages]

Just received this afternoon:

 

==

Support Team,

You have several business customers being affected by an ongoing issue. In 
order to resolve this, First Data is requesting that you clear the cache on all 
DNS servers being used to support them. We propagated a correction over 16 
hours ago and know that Google DNS and others are translating correctly. Would 
you please help us assist your customers?

The correct resolutions are:
vxn.datawire.net   216.220.36.75
vxn1.datawire.net   205.167.140.10
vxn2.datawire.net   64.243.142.36
vxn3.datawire.net   206.112.91.167
vxn4.datawire.net   63.240.199.76

If you are resolving it as anything starting with 45.x.x.x, it is incorrect. 
Please feel free to compare to the Google DNS resolution for confirmation.

Please either reply all or call First Data's Network Operations at 888-377-8726 
Option 3.




First Data, 240 North Roosevelt Av 

Chandler, Arizona 85226



==

 

That kind of confirms that the TTL for the 45.x.x.x record(s) were a bit too 
long – if they had been short, like they are now at 300 seconds, the issue 
would mostly have cleared up.

 

From: Outages  On Behalf Of frnkblk--- via Outages
Sent: Friday, July 13, 2018 9:56 PM
To: 'Luke Guillory' ; jay...@peakinter.net; 
outages@outages.org
Subject: Re: [outages] Problem with credit card machine processing? "Datawire"

 

Yes, we learned of issues late Wednesday morning after receiving reports from 
two and then three business customers.  Indications suggest the issue started 
Tuesday evening.   One local Dairy Queen and another 20 minutes away couldn’t 
accept credit cards on Wednesday.

 

The request to preform DNS flushes of vxn.datawire.net came to us Thursday 
afternoon from two of three customers who (eventually) called their credit card 
partners/processors.  So we flushed our (ISP) caches and then encouraged those 
customers to power cycle their router and then their credit card machines, but 
that wasn’t 100% successful for them, either.  At that point we directed them 
back to their credit card partners/processors. It was interesting to see DNS 
resolution for vxn.datawire.net pointing to a mixture of 216.220.36.75 
(vxn.datawire.net) and 45.227.252.17 (hosting-by.net4web.org).  Maybe it's 
normal that they have multiple, but on Wednesday it was just 216.220.36.75. The 
TTL for 45.227.252.17 was much longer (over 430,000) than 216.220.36.75 (about 
300 seconds) and had a bad SSL certificate for https://vxn.datawire.net.  I 
suspect they moved some operations to another data center, but made a mistake 
with TTL.

 

All told we probably heard from six or seven different businesses.

 

More here:

https://twitter.com/ExecPro/status/1016860164983611392

https://status.cayan.com/issues/5b45477e8dc35afae9000fe6

https://status.cayan.com/issues/5b4546508dc35a5975000fdc

https://status.cayan.com/issues/5b479ad48dc35ad03a0030e7

https://status.cayan.com/issues/5b478b918dc35aff310030c9

https://twitter.com/TriphenTech/status/1016852856408690693

https://twitter.com/C_Forrest/status/1017819893704593410

https://twitter.com/Vicinity_7/status/1017800989347401728

https://twitter.com/pokehbar/status/1017796090052128769

https://twitter.com/glyngh/status/1017790958610493440

https://twitter.com/tallbaby21/status/1017121159526133760

https://twitter.com/devin_ledude/status/1017451556000522241

https://status.cayan.com/issues/5b478ba38dc35a3da80030d9

 

Frank 

 

 

# whob 216.220.36.75

IP: 216.220.36.75

Origin-AS: 12188

Prefix: 216.220.32.0/20

AS-Path: 18106 6939 12188

AS-Org-Name: Q9 Networks Inc.

Org-Name: Q9 Networks Inc.

Net-Name: Q9-NET1

Cache-Date: 1531374425

Latitude: 43.508330

Longitude: -79.88

City: Milton

Region: Ontario

Country: Canada

Country-Code: CA

 

# whob 45.227.252.17

IP: 45.227.252.17

Origin-AS: 58271

Prefix: 45.227.252.0/24

AS-Path: 34224 12389 44125 201765 48882 58271

AS-Org-Name: VSERVER-AS

Org-Name: This network range is not fully allocated to APNIC.

Net-Name: IANA-NETBLOCK-45

Cache-Date: 1531374425

Latitude: 0.00

Longitude: 0.00

City: NULL

Region: NULL

Country: NULL

Country-Code: NULL

 

From: Outages mailto:outages-boun...@outages.org> 
> On Behalf Of Luke Guillory via Outages
Sent: Friday, July 13, 2018 9:18 PM
To: jay...@peakinter.net  ; outages@outages.org 
 
Subject: Re: [outages] Problem with credit card machine processing? "Datawire"

 

We had a customer call saying they needed is to clear dns cache because they 
couldn’t process CCs.

 

One of my guys read about the large outage so when it came in we knew it wasn’t 
anything to do with us. 

Sent from my iPhone


On Jul 13, 2018, at 9:04 PM, Jayson Baker via Outages mailto:outages@outages.org> > wrote:

Our folks have spent the better part of a day 

Re: [outages] Amazon www returns 503 error

[John Knutson via Outages]

almost certainly due to increased traffic for Prime Day
https://twitter.com/CNBCnow/status/1018937466177359872

On Mon, Jul 16, 2018 at 2:24 PM Chaim Rieger via Outages <
outages@outages.org> wrote:

> Eom.
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Amazon www returns 503 error

[Chaim Rieger via Outages]

Eom.
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Issues with Dell.com email server?

[Sean Rose - TeleGo via Outages]

Still experiencing this issue from G Suite.

The response was:

550 Too many invalid recipients


On Mon, Jul 16, 2018 at 12:09 PM Scott Howard via Outages <
outages@outages.org> wrote:

> Best I can make out this was only affecting email from Google Apps/Gmail,
> and it appears to be working now.
>
> I suspect there is a directory harvest setting configured on the IronPorts
> that got tripped and has now timed out so it's working again.
>
> I've raised a ticket with out IT department to get it investigated.
>
>   Scott
>   Dell EMC
>
>
> On Mon, Jul 16, 2018 at 8:15 AM, Kenneth Padgett via Outages <
> outages@outages.org> wrote:
>
>> Anyone else having issues emailing @dell.com addresses? Error coming
>> back is:
>>
>> 550 Too many invalid recipients
>>
>> KP
>>
>> ___
>> Outages mailing list
>> Outages@outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>


-- 

Sean Rose

Senior Technical Engineer

Main: (212) 477-1000 x 2065 

Direct: (929) 477-2065 

Website: www.telego.com 



___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Issues with Dell.com email server?

[Scott Howard via Outages]

Best I can make out this was only affecting email from Google Apps/Gmail,
and it appears to be working now.

I suspect there is a directory harvest setting configured on the IronPorts
that got tripped and has now timed out so it's working again.

I've raised a ticket with out IT department to get it investigated.

  Scott
  Dell EMC


On Mon, Jul 16, 2018 at 8:15 AM, Kenneth Padgett via Outages <
outages@outages.org> wrote:

> Anyone else having issues emailing @dell.com addresses? Error coming back
> is:
>
> 550 Too many invalid recipients
>
> KP
>
> ___
> Outages mailing list
> Outages@outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

Re: [outages] Issues with Dell.com email server?

[Kenneth Padgett via Outages]

Thanks all for the confirmations, seems to be senders using Google's mail
servers that Dell is rejecting.

KP

On Mon, Jul 16, 2018 at 11:15 AM Kenneth Padgett 
wrote:

> Anyone else having issues emailing @dell.com addresses? Error coming back
> is:
>
> 550 Too many invalid recipients
>
> KP
>
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages

[outages] Issues with Dell.com email server?

[Kenneth Padgett via Outages]

Anyone else having issues emailing @dell.com addresses? Error coming back
is:

550 Too many invalid recipients

KP
___
Outages mailing list
Outages@outages.org
https://puck.nether.net/mailman/listinfo/outages