Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Randy Bush]

>>> It surprises that important sites don't do mirroring.
>> depends on what you mean by 'mirroring.' think latency.
> Though a best effort to mirror would be acceptable.  Maybe not up to
> the minute but at least a recovery.

depends on if the writer has to wait for it to hit the spinning oxide
1,000km away.  as i said, depends on what you mean by 'mirroring.'  i
would not recommend raid, drbd, ...  maybe periodic rsync, or an app
level sync designed for latency.

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
signatures are back, thanks to dmarc header butchery

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Chris Cariffe]

On Thu, Mar 11, 2021 at 7:40 PM Randy Bush  wrote:

> It surprises that important sites don't do mirroring.
>
> depends on what you mean by 'mirroring.' think latency.
>
> randy
> --
>
> Though a best effort to mirror would be acceptable.  Maybe not up to the
> minute but at least a recovery.
>

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Randy Bush]

> It surprises that important sites don't do mirroring.

depends on what you mean by 'mirroring.'  think latency.

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
signatures are back thanks to dmarc header mangling

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Rod Beck]

There are plenty of websites still down: https://www.centrepompidou.fr/en/.

It surprises that important sites don't do mirroring.

-R.


From: Johann 
Sent: Friday, March 12, 2021 12:41 AM
To: Rod Beck 
Cc: Randy Bush ; Lukas Tribus ; Eilers, Laura via 
NANOG 
Subject: Re: OVH datacenter SBG2 in Strasbourg on fire 

Several government websites have been impacted by fire. But >90% was 
operationnel during the same day after the incident.
There are DRP (seems to be working) and backups in other datacenters. There 
must remain exceptions or minor things broken.
But I think it's a pretty good result.

Johann

Le ven. 12 mars 2021 à 00:24, Rod Beck 
mailto:rod.b...@unitedcablecompany.com>> a 
écrit :
It is terrible timing with the company planning an IPO and also because the 
French government has heavily backed OVH. I think some government websites went 
down. France has a statist tradition that leads it to get involved in 
industries where has no expertise.




From: NANOG 
mailto:unitedcablecompany@nanog.org>>
 on behalf of Randy Bush mailto:ra...@psg.com>>
Sent: Thursday, March 11, 2021 9:43 PM
To: Lukas Tribus mailto:lu...@ltri.eu>>
Cc: Eilers, Laura via NANOG mailto:nanog@nanog.org>>
Subject: Re: OVH datacenter SBG2 in Strasbourg on fire 

>> Statement (in French) from Octave Klaba, containing some discussion
>> of the development of the fire (starts at ~ 4:30):
>> https://www.ovh.com/fr/images/sbg/index-fr.html
> English:
> https://www.ovh.com/fr/images/sbg/index-en.html

and a few hundred of us hoping we never have to stand in front of that
camera to explain a similar incident

a few things stood out: where backups are located, the ability to build
a lot of new servers and the supporting infra, ...  but in a week or two
i hope he can tell us results of more analysis.

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd 
ra...@psg.com`
signatures are back, thanks to dmarc header butchery

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Johann]

Several government websites have been impacted by fire. But >90% was
operationnel during the same day after the incident.
There are DRP (seems to be working) and backups in other datacenters. There
must remain exceptions or minor things broken.
But I think it's a pretty good result.

Johann

Le ven. 12 mars 2021 à 00:24, Rod Beck  a
écrit :

> It is terrible timing with the company planning an IPO and also because
> the French government has heavily backed OVH. I think some government
> websites went down. France has a statist tradition that leads it to get
> involved in industries where has no expertise.
>
>
>
> --
> *From:* NANOG 
> on behalf of Randy Bush 
> *Sent:* Thursday, March 11, 2021 9:43 PM
> *To:* Lukas Tribus 
> *Cc:* Eilers, Laura via NANOG 
> *Subject:* Re: OVH datacenter SBG2 in Strasbourg on fire 
>
> >> Statement (in French) from Octave Klaba, containing some discussion
> >> of the development of the fire (starts at ~ 4:30):
> >> https://www.ovh.com/fr/images/sbg/index-fr.html
> > English:
> > https://www.ovh.com/fr/images/sbg/index-en.html
>
> and a few hundred of us hoping we never have to stand in front of that
> camera to explain a similar incident
>
> a few things stood out: where backups are located, the ability to build
> a lot of new servers and the supporting infra, ...  but in a week or two
> i hope he can tell us results of more analysis.
>
> randy
>
> ---
> ra...@psg.com
> `gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
> signatures are back, thanks to dmarc header butchery
>

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Rod Beck]

It is terrible timing with the company planning an IPO and also because the 
French government has heavily backed OVH. I think some government websites went 
down. France has a statist tradition that leads it to get involved in 
industries where has no expertise.




From: NANOG  on behalf 
of Randy Bush 
Sent: Thursday, March 11, 2021 9:43 PM
To: Lukas Tribus 
Cc: Eilers, Laura via NANOG 
Subject: Re: OVH datacenter SBG2 in Strasbourg on fire 

>> Statement (in French) from Octave Klaba, containing some discussion
>> of the development of the fire (starts at ~ 4:30):
>> https://www.ovh.com/fr/images/sbg/index-fr.html
> English:
> https://www.ovh.com/fr/images/sbg/index-en.html

and a few hundred of us hoping we never have to stand in front of that
camera to explain a similar incident

a few things stood out: where backups are located, the ability to build
a lot of new servers and the supporting infra, ...  but in a week or two
i hope he can tell us results of more analysis.

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
signatures are back, thanks to dmarc header butchery

Re: DOD prefixes and AS8003 / GRSCORP

[Siyuan Miao]

So this company (Global Resource Systems, LLC) was formed on 2020-10-13 and
ARIN assigned AS8003 to them even earlier than it.

Here's a simple timeline in case anyone want to have a check:

9/8/2020 GLOBAL RESOURCE SYSTEMS, LLC registered in Delaware
9/10/2020 Nameserver of grscorp.com was changed from AfterNIC (a website to
sell premium / expired domains) to UltraDNS
9/11/2020 GLOBAL RESOURCE SYSTEMS, LLC (FL) registered their organization
in ARIN
9/14/2020 GLOBAL RESOURCE SYSTEMS, LLC (FL) got AS8003 from ARIN
9/21/2020 MAINT-GRSL-AS8003 is registered in RADB
10/13/2020 GLOBAL RESOURCE SYSTEMS, LLC registered in Florida

Around 21/01/2021, AS8003 registered numerous route objects in RADB and
started announcing DOD space.
In addition to AS8003, they also added AS95 to their AS-set and registered
some objects under AS95.

Based on RIPEstats, Last seen of AS8003 before 2021 is around 2003.
And there's another GLOBAL RESOURCE SYSTEMS, LLC in FL which has been
inactive since 2013.


On Thu, Mar 11, 2021 at 10:31 PM Alain Hebert  wrote:

> I scratch it out to hiding in plain sight...
>
> -
> Alain Hebertaheb...@pubnix.net
> PubNIX Inc.
> 50 boul. St-Charles
> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
> Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443
>
> On 3/11/21 9:14 AM, Filip Hruska wrote:
>
> Contacted HE NOC earlier regarding these announcements, they are
> "legitimate".
>
> Filip
>
> On 11/03/2021 14:56, Javier Henderson wrote:
>
>
> On Mar 11, 2021, at 8:43 AM, Eric Dugas via NANOG 
>  wrote:
>
> I would be really curious to see the LOA presented to AS6939 to announce
> 54 million IPs out of government IP space and what type of verification was
> done because it doesn't seem legit at all.
>
> Did you try calling the number on the WHOIS for AS8003, or maybe HE’s NOC
> to follow up?
>
> -jav
>
>
>

Re: How to Fix IP GEO for google/youtube tv

[Grant Taylor via NANOG]

On 3/11/21 2:28 PM, Jared Brown wrote:

Out of interest, why does it take multiple weeks to edit a GEO IP entry?


I have no idea /why/.  I only know that it /is/.

I /assume/ -- from a place of ignorance about the innards of the process 
-- that there is validation and manual approval.  I also /assume/ that 
there are enough requests that there can be a backlog, hence why there 
is variance.  There may also be something as silly as the process that 
amalgamates things is run every other week.  I simply do not know /why/ 
I only know that it is.


I wonder why Google even has this problem at all. If you've so much 
as looked at Google maps or used any app that uses location services 
then Google knows with a high certainty where you are.


I think that there is more to it than meats the eye.  Especially when 
you consider how to deal with things like the 2nd hand IP market and 
ISPs from neighboring countries that aren't exactly friendly with each 
other.



This is a problem Google actually *can* automate away.


I assume -- from a place of ignorance -- that you are correct in that 
there is room for process improvement.


I do trust that colleagues are adhering to and executing the process to 
the best of their ability.




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: How to Fix IP GEO for google/youtube tv

[Jared Brown]

Grant Taylor wrote:
> The process takes multiple weeks.

Out of interest, why does it take multiple weeks to edit a GEO IP entry?

I wonder why Google even has this problem at all. If you've so much as looked 
at Google maps or used any app that uses location services then Google knows 
with a high certainty where you are.

This is a problem Google actually *can* automate away.


Jared

Re: Re: OVH datacenter SBG2 in Strasbourg on fire 

[Sean Donelan]

In the 1990s, I spent long time researching and talking to people about 
the history of the old Automated Data Processing room code requirements. 
You can tell by the terminology "Automated Data Processing" the age of the 
original requirements. IBM helped write the original requirements in the 
1950s/1960s when one mainframe computer cost more than a building.


Times have changed.


On Thu, 11 Mar 2021, b...@theworld.com wrote:

I remember many years ago spec'ing a machine room at BU and coming to
loggerheads with the VP of building and grounds.

He (well, their rules) wanted low-temp sprinkler triggers, I wanted
the high-temp ones (I forget but I think 135F vs 175F.)

Me: I'll have over $2M in electrical equipment in that room!

Him: I have $20M in building surrounding your machine room, I win!

Re: How to Fix IP GEO for google/youtube tv

[Grant Taylor via NANOG]

On 3/11/21 1:05 PM, Josh Luthman wrote:

His email: Mar 8, 2021, 10:08 AM (3 days ago)
The current date and time: 3:03 PM Thursday, March 11, 2021 (EST)


The process takes multiple weeks.


You do the math.


My email with recommendations: 10:23 AM Thursday, March 11, 2021 (MST)
The current date and time: 13:47Thursday, March 11, 2021 (MST)

Again, the process takes multiple weeks.

You do the math again.


Well since I don't get any spam in my mailbox, I'd say it is fixed.


Good for you.

I suspect that there are *MANY* who would disagree with you.

Not sure this thread is going anywhere besides downhill so unless 
there's anything productive to contribute on the topic at hand, 
resolving Google's geoip, I'm out.


As previously stated:

 - Contact GGC operations
 - Contact Google NOC

Both teams can escalate the issue to the GeoIP team to fix the problem.

Let me try spelling it out this way.

I have worked on the GGC operations team and we always take any GeoIP 
issues very seriously and we /want/ /to/ /fix/ them.


So, does 1st hand knowledge from someone who has fixed multiple GeoIP 
issues count for anything in regards to your opinion?


I can tell you for a fact that colleagues /do/ /care/ and /do/ /want/ 
/to/ /fix/ /things/.  So, if you have a GGC node, contact the GGC 
operations team with GeoIP issues.  They will do what they can to fix 
the problem.  It does usually take multiple weeks to get something 
fixed.  But I have seen multiple corrections completed through the 
process that starts with the GGC operations team.


I can say, 1st hand, as someone that has processed multiple GeoIP 
corrections, that Google employees /do/ /wanna/ /improve/ GeoIP 
information.  I can also say that the vast majority of Google employees 
do want to improve anything and everything they can.




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Re: OVH datacenter SBG2 in Strasbourg on fire 

[bzs]

From: George Herbert 

...Interesting overview of fire damage.

I remember many years ago spec'ing a machine room at BU and coming to
loggerheads with the VP of building and grounds.

He (well, their rules) wanted low-temp sprinkler triggers, I wanted
the high-temp ones (I forget but I think 135F vs 175F.)

Me: I'll have over $2M in electrical equipment in that room!

Him: I have $20M in building surrounding your machine room, I win!


-- 
-Barry Shein

Software Tool & Die| b...@theworld.com | http://www.TheWorld.com
Purveyors to the Trade | Voice: +1 617-STD-WRLD   | 800-THE-WRLD
The World: Since 1989  | A Public Information Utility | *oo*

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Gavin Henry]

He looks sooo tired.

Re: How to Fix IP GEO for google/youtube tv

[Grant Taylor via NANOG]

On 3/11/21 12:57 PM, Josh Luthman wrote:
The lack of response to the problem from Google's team is the 
knowledge that we're working with.


It may be the knowledge that /you/ are working with.  /I/ am working 
with different knowledge.


As stated in another reply, "I know / have witnessed multiple Google 
employees that have submitted and followed up on multiple GeoIP 
corrections."  So /I'm/ working with /1st/ /hand/ knowledge that Google 
employees /do/ /wanna/ /improve/.



What more do you need to know?


For starters, any dissenting opinions or knowledge would be good to 
know.  Seeing as I have both 1st hand knowledge and dissenting opinions


If it quacks like a duck, walks like a duck, and looks like a duck, 
it's probably a duck.


If it sounds like a herd of stampeding horses, moves like a herd of 
stampeding horses, and looks like a herd of stampeding horses, it's 
probably a herd of stampeding horses /except/ when it's a herd of 
stampeding zebras.



You don't need a DNA test.


Hopefully you don't /need/ a DNA test to identify the difference between 
a horse and a zebra.  But hopefully you realize that your initial 
assumption that it was a heard of stampeding horses was wrong, and 
likely biased towards horses / away from zebras.


You're entitled to your opinion just as the rest of us are entitled to 
ours.  It doesn't make it wrong.


Opinions are only facts (in and of themselves) to the person that holds 
them.  {Your,my} opinion is fact to {you,me} but just {your,my} opinion 
to everyone else.


Also, opinions often to take everything into account.



--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Randy Bush]

>> Statement (in French) from Octave Klaba, containing some discussion
>> of the development of the fire (starts at ~ 4:30):
>> https://www.ovh.com/fr/images/sbg/index-fr.html
> English:
> https://www.ovh.com/fr/images/sbg/index-en.html

and a few hundred of us hoping we never have to stand in front of that
camera to explain a similar incident

a few things stood out: where backups are located, the ability to build
a lot of new servers and the supporting infra, ...  but in a week or two
i hope he can tell us results of more analysis.

randy

---
ra...@psg.com
`gpg --locate-external-keys --auto-key-locate wkd ra...@psg.com`
signatures are back, thanks to dmarc header butchery

Re: How to Fix IP GEO for google/youtube tv

[Josh Luthman]

>How do you know that the OP's problem hasn't been resolved?

I asked him.  He said "no".  Do we have to debate the opinion of fixed or
what no means?

>Presuming the OP did leverage the suggestions that I provided, how can
you say "no resolution" when it's been ~2 hours for a process that I
indicated takes order of weeks?

His email: Mar 8, 2021, 10:08 AM (3 days ago)
The current date and time: 3:03 PM Thursday, March 11, 2021 (EST)

You do the math.

>> once again is a testament to how Google responds to this problem.

>Based on your logic, the entire email industry must actually want spam
because all efforts to the contrary have failed to stop it.

Well since I don't get any spam in my mailbox, I'd say it is fixed.

Not sure this thread is going anywhere besides downhill so unless there's
anything productive to contribute on the topic at hand, resolving Google's
geoip, I'm out.

Josh Luthman
24/7 Help Desk: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Thu, Mar 11, 2021 at 2:58 PM Grant Taylor via NANOG 
wrote:

> On 3/11/21 12:28 PM, Josh Luthman wrote:
> > Based on how difficult it is to correct their data for them at no
> > charge, I'm not sure he's entirely wrong in that statement.
>
> Difficult of doing something is not directly related to people's
> willingness / desire to do it.
>
> I can guarantee you that there are Google employees who do want to have
> GeoIP (and other) information as correct as possible.  Said employees
> are grateful for such corrections.  Said employees try to process such
> corrections as expediently and efficiently as possible.
>
> I can make this guarantee because I know / have witnessed multiple
> Google employees that have submitted and followed up on multiple GeoIP
> corrections.
>
> I'm certain that there are still errors in Google's GeoIP data.  But I'm
> equally certain that multiple Google employees /do/ /wanna/ /improve/
> /it/ (the GeoIP data).
>
> > The fact that this thread exists, the latest of multiple, and has ended
> > in no resolution to the problem
>
> How do you know that the OP's problem hasn't been resolved?
>
> How do you know that the OP hasn't leveraged either of the suggestions
> that I provided?
>
> Presuming the OP did leverage the suggestions that I provided, how can
> you say "no resolution" when it's been ~2 hours for a process that I
> indicated takes order of weeks?
>
> > once again is a testament to how Google responds to this problem.
>
> Based on your logic, the entire email industry must actually want spam
> because all efforts to the contrary have failed to stop it.
>
> No, the email industry does not /want/ spam.
> Yes Google has problems.
> No, Google's problems do not accurately reflect the desires of the
> employees.
>
>
>
> --
> Grant. . . .
> unix || die
>
>

Re: How to Fix IP GEO for google/youtube tv

[Mike Hammett]

I think the distinction needs to be made between some number of {company} 
employees and the effectiveness of the organization. 

One, two, ten, or fifty dedicated employees working on the problem doesn't make 
up for management that doesn't allocate appropriate resources to solve the 
problem. Those resources could be number of bodies, developers to create 
systems to make the process easier, etc. 




- 
Mike Hammett 
Intelligent Computing Solutions 
http://www.ics-il.com 

Midwest-IX 
http://www.midwest-ix.com 

- Original Message -

From: "William Guo"  
To: "Nate Burke"  
Cc: nanog@nanog.org 
Sent: Thursday, March 11, 2021 3:36:43 AM 
Subject: Re: How to Fix IP GEO for google/youtube tv 


Google has its internal GeoIP team. 


But the data quality is not so good, but they don't wanna improve it for some 
reason. 


On Mon, Mar 8, 2021 at 8:00 AM Nate Burke < n...@blastcomm.com > wrote: 


If you find out, let me know the secret sauce. I've had a ticket open 
with YoutubeTV for 7 months and they can't get the GeoIP fixed for one 
of my /21's that I've had for 20 years. Every reply on my ticket from 
Google is that the YoutubeTV service has major GeoIP issues, and they 
don't know how to fix them. It's been escalated to engineering for the 
last 7 months with no ETA on a fix. That is the only service for me 
that is having a location problem. 


On 3/7/2021 7:19 PM, t...@nwohiobb.com wrote: 
> Can anybody tell me how to fix my IP block for google websites and 
> youtube/tv because I am getting customers saying they are out of the 
> country and I tried to apply for the google isp portal and I have not 
> gotten anything back on this. 
> 
> Thanks Tim 

Re: How to Fix IP GEO for google/youtube tv

[Josh Luthman]

The lack of response to the problem from Google's team is the
knowledge that we're working with.  What more do you need to know?  If it
quacks like a duck, walks like a duck, and looks like a duck, it's probably
a duck.  You don't need a DNA test.

You're entitled to your opinion just as the rest of us are entitled to
ours.  It doesn't make it wrong.

Josh Luthman
24/7 Help Desk: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Thu, Mar 11, 2021 at 2:40 PM Tom Beecher  wrote:

> Based on how difficult it is to correct their data for them at no charge,
>> I'm not sure he's entirely wrong in that statement.
>>
>
> No, it's still wrong.
>
> Stating that Google's team 'doesn't want to improve things' , while also
> having zero knowledge of the reasons why the current issues are present, is
> at best uninformed , and at worst just plain ignorant.
>
> On Thu, Mar 11, 2021 at 2:29 PM Josh Luthman 
> wrote:
>
>> Based on how difficult it is to correct their data for them at no charge,
>> I'm not sure he's entirely wrong in that statement.
>>
>> The fact that this thread exists, the latest of multiple, and has ended
>> in no resolution to the problem once again is a testament to how Google
>> responds to this problem.
>>
>> Josh Luthman
>> 24/7 Help Desk: 937-552-2340
>> Direct: 937-552-2343
>> 1100 Wayne St
>> Suite 1337
>> Troy, OH 45373
>>
>>
>> On Thu, Mar 11, 2021 at 12:23 PM Grant Taylor via NANOG 
>> wrote:
>>
>>> On 3/11/21 2:36 AM, William Guo wrote:
>>> > but they don't wanna improve it for some reason.
>>>
>>> That is both unfair and wrong.
>>>
>>> I know for a fact that they do want their internal GeoIP to be as
>>> accurate as possible and that they do want to improve the inaccuracies
>>> if and when possible.
>>>
>>> I have personally submitted a handful of GeoIP changes through channels.
>>>   It does take a wile O(weeks) to complete.  But it /does/ complete.
>>>
>>> If you have a GGC node, you can reach out to the GGC operations team for
>>> help resolving GeoIP issues.  Ostensibly the GGC operations team focuses
>>> on GGC related issues.  But they should be able to document the issue
>>> and send it to proper internal teams anyway.
>>>
>>> I would assume that Google's NOC can also re-route issues as necessary.
>>>
>>> I believe that saying "they don't wanna improve" as a fact, vs an
>>> opinion, is disingenuous and somewhat rude.
>>>
>>>
>>>
>>> --
>>> Grant. . . .
>>> unix || die
>>>
>>>

Re: How to Fix IP GEO for google/youtube tv

[Grant Taylor via NANOG]

On 3/11/21 12:28 PM, Josh Luthman wrote:
Based on how difficult it is to correct their data for them at no 
charge, I'm not sure he's entirely wrong in that statement.


Difficult of doing something is not directly related to people's 
willingness / desire to do it.


I can guarantee you that there are Google employees who do want to have 
GeoIP (and other) information as correct as possible.  Said employees 
are grateful for such corrections.  Said employees try to process such 
corrections as expediently and efficiently as possible.


I can make this guarantee because I know / have witnessed multiple 
Google employees that have submitted and followed up on multiple GeoIP 
corrections.


I'm certain that there are still errors in Google's GeoIP data.  But I'm 
equally certain that multiple Google employees /do/ /wanna/ /improve/ 
/it/ (the GeoIP data).


The fact that this thread exists, the latest of multiple, and has ended 
in no resolution to the problem


How do you know that the OP's problem hasn't been resolved?

How do you know that the OP hasn't leveraged either of the suggestions 
that I provided?


Presuming the OP did leverage the suggestions that I provided, how can 
you say "no resolution" when it's been ~2 hours for a process that I 
indicated takes order of weeks?



once again is a testament to how Google responds to this problem.


Based on your logic, the entire email industry must actually want spam 
because all efforts to the contrary have failed to stop it.


No, the email industry does not /want/ spam.
Yes Google has problems.
No, Google's problems do not accurately reflect the desires of the 
employees.




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: How to Fix IP GEO for google/youtube tv

[Tom Beecher]

>
> Based on how difficult it is to correct their data for them at no charge,
> I'm not sure he's entirely wrong in that statement.
>

No, it's still wrong.

Stating that Google's team 'doesn't want to improve things' , while also
having zero knowledge of the reasons why the current issues are present, is
at best uninformed , and at worst just plain ignorant.

On Thu, Mar 11, 2021 at 2:29 PM Josh Luthman 
wrote:

> Based on how difficult it is to correct their data for them at no charge,
> I'm not sure he's entirely wrong in that statement.
>
> The fact that this thread exists, the latest of multiple, and has ended in
> no resolution to the problem once again is a testament to how Google
> responds to this problem.
>
> Josh Luthman
> 24/7 Help Desk: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
>
> On Thu, Mar 11, 2021 at 12:23 PM Grant Taylor via NANOG 
> wrote:
>
>> On 3/11/21 2:36 AM, William Guo wrote:
>> > but they don't wanna improve it for some reason.
>>
>> That is both unfair and wrong.
>>
>> I know for a fact that they do want their internal GeoIP to be as
>> accurate as possible and that they do want to improve the inaccuracies
>> if and when possible.
>>
>> I have personally submitted a handful of GeoIP changes through channels.
>>   It does take a wile O(weeks) to complete.  But it /does/ complete.
>>
>> If you have a GGC node, you can reach out to the GGC operations team for
>> help resolving GeoIP issues.  Ostensibly the GGC operations team focuses
>> on GGC related issues.  But they should be able to document the issue
>> and send it to proper internal teams anyway.
>>
>> I would assume that Google's NOC can also re-route issues as necessary.
>>
>> I believe that saying "they don't wanna improve" as a fact, vs an
>> opinion, is disingenuous and somewhat rude.
>>
>>
>>
>> --
>> Grant. . . .
>> unix || die
>>
>>

Re: How to Fix IP GEO for google/youtube tv

[Josh Luthman]

Based on how difficult it is to correct their data for them at no charge,
I'm not sure he's entirely wrong in that statement.

The fact that this thread exists, the latest of multiple, and has ended in
no resolution to the problem once again is a testament to how Google
responds to this problem.

Josh Luthman
24/7 Help Desk: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373


On Thu, Mar 11, 2021 at 12:23 PM Grant Taylor via NANOG 
wrote:

> On 3/11/21 2:36 AM, William Guo wrote:
> > but they don't wanna improve it for some reason.
>
> That is both unfair and wrong.
>
> I know for a fact that they do want their internal GeoIP to be as
> accurate as possible and that they do want to improve the inaccuracies
> if and when possible.
>
> I have personally submitted a handful of GeoIP changes through channels.
>   It does take a wile O(weeks) to complete.  But it /does/ complete.
>
> If you have a GGC node, you can reach out to the GGC operations team for
> help resolving GeoIP issues.  Ostensibly the GGC operations team focuses
> on GGC related issues.  But they should be able to document the issue
> and send it to proper internal teams anyway.
>
> I would assume that Google's NOC can also re-route issues as necessary.
>
> I believe that saying "they don't wanna improve" as a fact, vs an
> opinion, is disingenuous and somewhat rude.
>
>
>
> --
> Grant. . . .
> unix || die
>
>

Re: How to Fix IP GEO for google/youtube tv

[Grant Taylor via NANOG]

On 3/11/21 2:36 AM, William Guo wrote:

but they don't wanna improve it for some reason.


That is both unfair and wrong.

I know for a fact that they do want their internal GeoIP to be as 
accurate as possible and that they do want to improve the inaccuracies 
if and when possible.


I have personally submitted a handful of GeoIP changes through channels. 
 It does take a wile O(weeks) to complete.  But it /does/ complete.


If you have a GGC node, you can reach out to the GGC operations team for 
help resolving GeoIP issues.  Ostensibly the GGC operations team focuses 
on GGC related issues.  But they should be able to document the issue 
and send it to proper internal teams anyway.


I would assume that Google's NOC can also re-route issues as necessary.

I believe that saying "they don't wanna improve" as a fact, vs an 
opinion, is disingenuous and somewhat rude.




--
Grant. . . .
unix || die



smime.p7s
Description: S/MIME Cryptographic Signature

Re: OVH datacenter SBG2 in Strasbourg on fire ????

[Lukas Tribus]

On Thu, 11 Mar 2021 at 18:10, Enno Rey  wrote:
>
> Statement (in French) from Octave Klaba, containing some discussion of the 
> development of the fire (starts at ~ 4:30):
>
> https://www.ovh.com/fr/images/sbg/index-fr.html

English:
https://www.ovh.com/fr/images/sbg/index-en.html

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Andy Ringsmuth]

> On Mar 11, 2021, at 10:41 AM, Sean Donelan  wrote:
> 
> If you pump 100s of megawatts of energy into a confined space (i.e. a large 
> data center), and there is an uncontrolled release of that energy (i.e. a 
> fire), that's a lot of energy which is going to go somewhere.
> Stored energy, batteries, backup fuel for generators, server plastic, etc.
> There is a lot of stuff that can burn in a data center, especially if you 
> start over-stuffing it.
> 
> China has the top 7 data centers over 100 megawatts.
> 
> http://worldstopdatacenters.com/power/
> 
> Ok, an ISP may not have a mega datacenter.  Even "small" data centers have a 
> large fuel load. They are major industrial operations.  Like other major 
> industrial fires, often the firefighter response is to rescue any trapped 
> people and then contain the fire until it burns itself out.
> 
> The "cloud" is just someone else's physical data center.

Obligatory:

https://xkcd.com/908/




Andy Ringsmuth
5609 Harding Drive
Lincoln, NE 68521-5831
(402) 304-0083
a...@andyring.com

“Better even die free, than to live slaves.” - Frederick Douglas, 1863

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Chris Boyd]

> On Mar 11, 2021, at 5:06 AM, Matt Harris  wrote:
> 
> There are plenty of effective options besides environmentally-destructive 
> Halon, dangerous-to-equipment water sprinkler, or dangerous-to-personnel CO2 
> for fire suppression these days. Some of the most common today are foam 
> systems like FM-200 or 3m's Novec. 

Novec and Solvay’s Galden are not really that much better than Halon. I guess 
it come down to which halogen do you want to release? Chlorine or Ffuorine?

https://www.engineeredfluids.com/post/are-pfas-the-next-pcbs

—Chris

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Sean Donelan]

If you pump 100s of megawatts of energy into a confined space (i.e. a 
large data center), and there is an uncontrolled release of that energy 
(i.e. a fire), that's a lot of energy which is going to go somewhere.

Stored energy, batteries, backup fuel for generators, server plastic, etc.
There is a lot of stuff that can burn in a data center, especially if you 
start over-stuffing it.


China has the top 7 data centers over 100 megawatts.

http://worldstopdatacenters.com/power/

Ok, an ISP may not have a mega datacenter.  Even "small" data centers have 
a large fuel load. They are major industrial operations.  Like other major 
industrial fires, often the firefighter response is to rescue any trapped 
people and then contain the fire until it burns itself out.


The "cloud" is just someone else's physical data center.


1975 AT New York City fire, documentary how it recovered:
https://www.youtube.com/watch?v=f_AWAmGi-g8

Re: DoD IP Space

[Christopher Morrow]

On Thu, Mar 11, 2021 at 10:54 AM j k  wrote:
>
> Two questions...
>
> 1. How many on this list already have dual-stack or IPv6 only in operation?

we're coming up on the 10yr anniversary of 'world ipv6 day'.. so I
would HOPE 'lots' :)
probably that's not entirely a good 'hope' :(

> 2. If you are running IPv4 only, and a major service was to switch to IPv6 
> only,..
>  a. How fast would you move to a dual-stack of IPv6 only?
>  b. What would it impact your customers?
>  c. How would it impact your business?
>

This is REALY now a days: "people will learn when they get bit"
much like 'gosh, password is not a great password, who knew?'
 or: "well, who needs windows updates anyway?"

evangelizing ipv6 is... not worth the effort :( because if you didn't
get them memo over the last 10yrs
you are verizon and you are not changing stance until something
significant enough bites you.
(yearly email about verizon residential service and lack of ipv6 support.. )

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Andy Ringsmuth]

> 
> https://us.ovhcloud.com/about/company/green-tech
> 
> Further I conjecture that a halon system would not be feasible in such a 
> structure.
> 
> Daniel
> 
> There are plenty of effective options besides environmentally-destructive 
> Halon, dangerous-to-equipment water sprinkler, or dangerous-to-personnel CO2 
> for fire suppression these days. Some of the most common today are foam 
> systems like FM-200 or 3m's Novec. 
> 
> - Matt
> 

I suppose I could be wrong, but I would imagine any of the various fire 
prevention/mitigation systems are less environmentally damaging than an entire 
building burning to the ground.


-Andy

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Christopher Morrow]

On Thu, Mar 11, 2021 at 10:45 AM Dale W. Carder  wrote:
> Sadly, the foams are not without environmental impact either with
> respect to PFAS.

what's the trade off on environmental impact for:
  "foam fire retardant" (or halon or ...)

vs:
  "Burning plastics and other carcinogen like materials"
  (or just blowing co2 into the space)

yes, halon isn't used and was 'not great' for hoomuns.
yes, co2 envelopment is unlikely to go well for hoomuns either.
yes, of course greenhouse gas release isn't great for gaia...

I'm wondering if the tradeoff is accounted for though :)

Re: DoD IP Space

[j k]

Two questions...

1. How many on this list already have dual-stack or IPv6 only in operation?

2. If you are running IPv4 only, and a major service was to switch to IPv6
only,..
 a. How fast would you move to a dual-stack of IPv6 only?
 b. What would it impact your customers?
 c. How would it impact your business?

Joe Klein

"inveniet viam, aut faciet" --- Seneca's Hercules Furens (Act II, Scene 1)
"*I skate to where the puck is going to be, not to where it has been."
-- *Wayne
Gretzky
"I never lose. I either win or learn" - Nelson Mandela


On Thu, Feb 11, 2021 at 12:56 PM William Herrin  wrote:

> On Thu, Feb 11, 2021 at 6:13 AM Izaac  wrote:
> > On Wed, Feb 10, 2021 at 10:38:00AM -0800, William Herrin wrote:
> > > None whatsoever. You just have to be really big.
> >
> > Hi Beel,
>
> That was unnecessary. Sorry I used an S instead of a Z.
>
> > Thanks for backing me up with an example of an organization with
> > competent network engineering.  Their ability to almost infinitely
> > leverage the existing rfc1918 address space to serve an appreciable
> > fraction of all Internet attached hosts is a real demonstration of the
> > possible.
>
> Except they don't. One of the reasons you can't put vms in multiple
> regions into the same VPC is they don't have enough IP addresses to
> uniquely address the backend hosts in every region. They end up with a
> squirrelly VPC peering thing they relies on multiple gateway hosts to
> overcome the address partitioning from overlapping RFC1918.
>
> In other words, it proves the exact opposite of your assertion.
>
> Regards,
> Bill Herrin
>
>
>
> --
> William Herrin
> b...@herrin.us
> https://bill.herrin.us/
>

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Dale W. Carder]

Thus spake Matt Harris (m...@netfire.net) on Thu, Mar 11, 2021 at 05:06:46AM 
-0600:
> 
> There are plenty of effective options besides environmentally-destructive
> Halon, dangerous-to-equipment water sprinkler, or dangerous-to-personnel
> CO2 for fire suppression these days. Some of the most common today are foam
> systems like FM-200 or 3m's Novec.

Sadly, the foams are not without environmental impact either with
respect to PFAS.

Dale

Re: How to Fix IP GEO for google/youtube tv

[George Michaelson]

Google honour https://tools.ietf.org/html/rfc8805 Which they also authored.

A bunch of people are proposing a geofeed: RPSL marker to catalog how
to find the feed.

-G

On Thu, Mar 11, 2021 at 7:38 PM William Guo  wrote:
>
> Google has its internal GeoIP team.
>
> But the data quality is not so good, but they don't wanna improve it for some 
> reason.
>
> On Mon, Mar 8, 2021 at 8:00 AM Nate Burke  wrote:
>>
>> If you find out, let me know the secret sauce.  I've had a ticket open
>> with YoutubeTV for 7 months and they can't get the GeoIP fixed for one
>> of my /21's that I've had for 20 years.  Every reply on my ticket from
>> Google is that the YoutubeTV service has major GeoIP issues, and they
>> don't know how to fix them.  It's been escalated to engineering for the
>> last 7 months with no ETA on a fix.  That is the only service for me
>> that is having a location problem.
>>
>>
>> On 3/7/2021 7:19 PM, t...@nwohiobb.com wrote:
>> > Can anybody tell me how to fix my IP block for google websites and
>> > youtube/tv because I am getting customers saying they are out of the
>> > country and I tried to apply for the google isp portal and I have not
>> > gotten anything back on this.
>> >
>> > Thanks Tim
>>

Re: DOD prefixes and AS8003 / GRSCORP

[Alain Hebert]

    I scratch it out to hiding in plain sight...

-
Alain Hebertaheb...@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911  http://www.pubnix.netFax: 514-990-9443

On 3/11/21 9:14 AM, Filip Hruska wrote:
Contacted HE NOC earlier regarding these announcements, they are 
"legitimate".


Filip

On 11/03/2021 14:56, Javier Henderson wrote:


On Mar 11, 2021, at 8:43 AM, Eric Dugas via NANOG  
wrote:


I would be really curious to see the LOA presented to AS6939 to 
announce 54 million IPs out of government IP space and what type of 
verification was done because it doesn't seem legit at all.
Did you try calling the number on the WHOIS for AS8003, or maybe HE’s 
NOC to follow up?


-jav

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Sander Steffann]

> Again: all conjecture, which seems to be tolerated here. ;-)

It's all good food for thoughts! It's important to learn from these
things, because I (and I expect many others) assumed that fire
suppression systems would prevent something like this from happening.
It is good to think and talk about the limits of such systems.

Cheers!
Sander



signature.asc
Description: This is a digitally signed message part

Re: DOD prefixes and AS8003 / GRSCORP

[Filip Hruska]

Contacted HE NOC earlier regarding these announcements, they are 
"legitimate".


Filip

On 11/03/2021 14:56, Javier Henderson wrote:



On Mar 11, 2021, at 8:43 AM, Eric Dugas via NANOG  wrote:

I would be really curious to see the LOA presented to AS6939 to announce 54 
million IPs out of government IP space and what type of verification was done 
because it doesn't seem legit at all.

Did you try calling the number on the WHOIS for AS8003, or maybe HE’s NOC to 
follow up?

-jav

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Daniel Karrenberg]

On 11 Mar 2021, at 11:46, Daniel Karrenberg wrote:

… Further I conjecture that a halon system would not be feasible in 
such a structure. …


Several of you have pointed out to me at varying level of politeness 
that ‘halon’ is no longer used. Sometimes I cannot hep but reveal my 
age … ;-)


I meant to say ‘Further I conjecture that the building could not have 
had a fire suppression system that requires reasonably airtight 
spaces.’. The between-the-lines there being that it may have been the 
very design of the building that lead to the rapid spreading of the 
fire.


Again: all conjecture, which seems to be tolerated here. ;-)

Daniel

Re: DOD prefixes and AS8003 / GRSCORP

[Javier Henderson]

> On Mar 11, 2021, at 8:43 AM, Eric Dugas via NANOG  wrote:
> 
> I would be really curious to see the LOA presented to AS6939 to announce 54 
> million IPs out of government IP space and what type of verification was done 
> because it doesn't seem legit at all.

Did you try calling the number on the WHOIS for AS8003, or maybe HE’s NOC to 
follow up?

-jav

Re: DOD prefixes and AS8003 / GRSCORP

[Eric Dugas via NANOG]

Single-homed on AS6939, no website setup on gsrcorp.com. (gsrcorp.com)

The address listed is in Plantation, PL and shows is a typical commercial 
office building. You can even get virtual office address here: 
https://www.davincivirtual.com/loc/us/florida/plantation-virtual-offices/facility-2492
https://bgp.he.net/net/11.0.0.0/8#_dns shows a lot of .cn domains pointing to 
these IPs
https://bgp.he.net/net/11.0.0.0/8#_irr shows route-object created for AS95 
(real DoD) and AS8003 by the same maintainer, probably to make it seem more 
legit.
I would be really curious to see the LOA presented to AS6939 to announce 54 
million IPs out of government IP space and what type of verification was done 
because it doesn't seem legit at all.
Eric
On Mar 11 2021, at 7:56 am, Siyuan Miao  wrote:
> Hi Folks,
>
> Just noticed that almost all DOD prefixes (7.0.0.0/8,11.0.0.0/8,22.0.0.0/8 
> (http://7.0.0.0/8,11.0.0.0/8,22.0.0.0/8) and bunch of /22s) are now announced 
> under AS8003 (GRSCORP) which was just formed a few months ago.
>
> It looks so suspicious. Does anyone know if it's authorized?
>
> Regards,
> Siyuan
>

DOD prefixes and AS8003 / GRSCORP

[Siyuan Miao]

Hi Folks,

Just noticed that almost all DOD prefixes (7.0.0.0/8,11.0.0.0/8,22.0.0.0/8
and bunch of /22s)  are now announced under AS8003 (GRSCORP) which was just
formed a few months ago.

It looks so suspicious. Does anyone know if it's authorized?

Regards,
Siyuan

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Matt Harris]

Matt Harris|Infrastructure Lead
816-256-5446|Direct
Looking for something?
Helpdesk Portal|Email Support|Billing Portal
We build and deliver end-to-end IT solutions.
On Thu, Mar 11, 2021 at 4:46 AM Daniel Karrenberg  wrote:

> Maybe the innovative ‘green’ design had something to do with the
> rapid spread of the fire and the way fire suppression was engineered:
>
> https://us.ovhcloud.com/about/company/green-tech
>
> Further I conjecture that a halon system would not be feasible in such a
> structure.
>
> Daniel
>

There are plenty of effective options besides environmentally-destructive
Halon, dangerous-to-equipment water sprinkler, or dangerous-to-personnel
CO2 for fire suppression these days. Some of the most common today are foam
systems like FM-200 or 3m's Novec.

- Matt

RE: an IP hijacking attempt

[Brian Turnbow via NANOG]

Hi Daniel,


> 
> Tracing it back to the originator of the route is of course a good first step.

Yes, we have done that and the results were not good.
The company that created the LOA is registered in the Seychelles and they have 
IPs that were/are being revoked by Afrinic
remarks:* * * * * * * * * * * * * * * * * * * * * * * * *
remarks:*   *
remarks:*  This IP prefix will be reclaimed and *
remarks:*  returned to the free pool by AFRINIC *
remarks:* on the 5th March 2021.*
remarks:*   *
remarks:* For more information, please contact  *
remarks:*   AFRINIC at hostmas...@afrinic.net   *
remarks:*   *
remarks:* * * * * * * * * * * * * * * * * * * * * * * * *

> 
> I would send an FYI to the RIR that allocated the prefix; preferably after the
> initial investigation established that it was not a genuine mistake. In that
> message I would make very clear if any action is requested from the RIR or
> not. If it is just an FYI the RIR will take note of it, watch for trends and 
> take it
> into account before doing anything with the registration.
> 
> Just what I would do.

Thanks for the Advice, I will do so



Brian 

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Daniel Karrenberg]

On 10 Mar 2021, at 16:42, Andy Ringsmuth wrote:

Sad to see of course, but also a little surprising that fire 
suppression systems didn’t, well, suppress the fire.


Maybe the innovative ‘green’ design had something to do with the 
rapid spread of the fire and the way fire suppression was engineered:


https://us.ovhcloud.com/about/company/green-tech

Further I conjecture that a halon system would not be feasible in such a 
structure.


Daniel

Re: OVH datacenter SBG2 in Strasbourg on fire 

[Rod Beck]

It is amazing that they can burn so well. Would never have imagined it.

Regards,

Roderick.


From: NANOG  on behalf 
of George Herbert 
Sent: Thursday, March 11, 2021 11:37 AM
To: Andy Ringsmuth 
Cc: nanog@nanog.org list 
Subject: Re: OVH datacenter SBG2 in Strasbourg on fire 

Sent from my iPhone

On Mar 10, 2021, at 7:45 AM, Andy Ringsmuth 
mailto:a...@andyring.com>> wrote:

Sad to see of course, but also a little surprising that fire suppression 
systems didn’t, well, suppress the fire.

Unless they didn’t exist?


I am assuming you haven’t had a real datacenter fire before.

I’ve had one fire, seen another, and had an accidental system firing of the gas 
system.

In the actual fire, caused by cooling system partial failure, there was no gas 
and it turned out the system to disable power on sprinkler discharge failed, 
then the power circuit breakers stayed live despite significant electrical 
short circuiting in the room as sprinklers fired, and then after 5 minutes the 
fire department arrived and sprayed water in for over 10 minutes without 
reducing arcing and fire before building power was successfully disabled and 
they could put the rest of the fire out.  This one could easily have destroyed 
its building, also the building Palo Alto Frys was in.

Fun fact: a rack of burning servers displaces sprinkler water around the rack, 
if it has a top on it, and even if not a vertical stack of burning servers 
pushes water down the front and back and slightly to the sides of burning 
servers, not through the systems themselves.

Fun fact: motherboards burn, as does chip encapsulation epoxy and all the 
wiring, the fan frames, board standoffs, essentially all the RAM and PCI board 
slots, some capacitors and surface mount devices... hard drives melt and the 
aluminum casings burn, SSD plastic casings generally burn.  GBICs and other 
laser diodes smell awful after they catch fire, though it can be hard to tell 
with everything else that smells.

Fun fact: It does not necessarily take many burning servers to put the room 
integrity at risk, even with sprinklers going and the fire department spraying 
water in.

Fun fact: All the servers that get wet but don’t burn will rust.  And 
everything in the room near the sprinklers that are going off will get wet.  
This is very sad as you watch many millions of dollars of brand new 42U racks 
prestuffed with HP enterprise servers oxidizing away while you wait for the 
garbage truck.

Fun fact: Combustion soot is conductive and even things that didn’t get wet 
probably are dead.

Fun fact: late era Sun Microsystems server boxes were very nice waxed 
cardboard, very well made, apparently fire resistant more than anyone would 
have thought but that were also water resistant enough that you’d have a new in 
box server submerged in a box still full to the brim with water days later.

Fun engineering advice: The window for critical data recovery from hard drives 
that are visibly corroding from water damage short of immersion is probably 
48-72 hours, but run them outside any casing on a fire resistant table and have 
all of CO2 and dry chemical fire extinguishers ready... and a fire hose, and 
handy building fire alarm.  Corroding water damaged SSDs are lower power draw 
and somewhat less likely to start another fire, but take the same precautions.


The fire I saw but that wasn’t mine burned a building with halon and pre action 
sprinklers more or less to the steel columns and roof beams, despite the fire 
department arriving in 5 minutes and trying to actively suppress.  Not enough 
openings for them to safely get to the fire before it was out of control, and 
not enough water flow available in all the sprinklers once it took off.

That one had evidently burned aluminum rack posts by the time it was over, not 
just melted them...


Systems and datacenters aren’t built to eliminate fire risk.  They just aren’t. 
 You can contain or control most office fires with sprinklers, and certainly 
evacuate.  Datacenters with emergency power batteries in the envelope often 
have enough stored energy to set the room on fire despite sprinklers.  If AC 
cutoffs fail and circuit breakers don’t trip mains power will as well.  Too 
many systems and storage and networking hardware components can burn.


-george

Re: OVH datacenter SBG2 in Strasbourg on fire 

[George Herbert]

Sent from my iPhone

> On Mar 10, 2021, at 7:45 AM, Andy Ringsmuth  wrote:
> 
> Sad to see of course, but also a little surprising that fire suppression 
> systems didn’t, well, suppress the fire.
> 
> Unless they didn’t exist?


I am assuming you haven’t had a real datacenter fire before.

I’ve had one fire, seen another, and had an accidental system firing of the gas 
system.  

In the actual fire, caused by cooling system partial failure, there was no gas 
and it turned out the system to disable power on sprinkler discharge failed, 
then the power circuit breakers stayed live despite significant electrical 
short circuiting in the room as sprinklers fired, and then after 5 minutes the 
fire department arrived and sprayed water in for over 10 minutes without 
reducing arcing and fire before building power was successfully disabled and 
they could put the rest of the fire out.  This one could easily have destroyed 
its building, also the building Palo Alto Frys was in.

Fun fact: a rack of burning servers displaces sprinkler water around the rack, 
if it has a top on it, and even if not a vertical stack of burning servers 
pushes water down the front and back and slightly to the sides of burning 
servers, not through the systems themselves.

Fun fact: motherboards burn, as does chip encapsulation epoxy and all the 
wiring, the fan frames, board standoffs, essentially all the RAM and PCI board 
slots, some capacitors and surface mount devices... hard drives melt and the 
aluminum casings burn, SSD plastic casings generally burn.  GBICs and other 
laser diodes smell awful after they catch fire, though it can be hard to tell 
with everything else that smells.

Fun fact: It does not necessarily take many burning servers to put the room 
integrity at risk, even with sprinklers going and the fire department spraying 
water in. 

Fun fact: All the servers that get wet but don’t burn will rust.  And 
everything in the room near the sprinklers that are going off will get wet.  
This is very sad as you watch many millions of dollars of brand new 42U racks 
prestuffed with HP enterprise servers oxidizing away while you wait for the 
garbage truck.

Fun fact: Combustion soot is conductive and even things that didn’t get wet 
probably are dead.

Fun fact: late era Sun Microsystems server boxes were very nice waxed 
cardboard, very well made, apparently fire resistant more than anyone would 
have thought but that were also water resistant enough that you’d have a new in 
box server submerged in a box still full to the brim with water days later.

Fun engineering advice: The window for critical data recovery from hard drives 
that are visibly corroding from water damage short of immersion is probably 
48-72 hours, but run them outside any casing on a fire resistant table and have 
all of CO2 and dry chemical fire extinguishers ready... and a fire hose, and 
handy building fire alarm.  Corroding water damaged SSDs are lower power draw 
and somewhat less likely to start another fire, but take the same precautions.


The fire I saw but that wasn’t mine burned a building with halon and pre action 
sprinklers more or less to the steel columns and roof beams, despite the fire 
department arriving in 5 minutes and trying to actively suppress.  Not enough 
openings for them to safely get to the fire before it was out of control, and 
not enough water flow available in all the sprinklers once it took off.

That one had evidently burned aluminum rack posts by the time it was over, not 
just melted them...


Systems and datacenters aren’t built to eliminate fire risk.  They just aren’t. 
 You can contain or control most office fires with sprinklers, and certainly 
evacuate.  Datacenters with emergency power batteries in the envelope often 
have enough stored energy to set the room on fire despite sprinklers.  If AC 
cutoffs fail and circuit breakers don’t trip mains power will as well.  Too 
many systems and storage and networking hardware components can burn.  


-george

Re: an IP hijacking attempt

[Daniel Karrenberg]

Tracing it back to the originator of the route is of course a good first 
step.


I would send an FYI to the RIR that allocated the prefix; preferably 
after the initial investigation established that it was not a genuine 
mistake. In that message I would make very clear if any action is 
requested from the RIR or not. If it is just an FYI the RIR will take 
note of it, watch for trends and take it into account before doing 
anything with the registration.


Just what I would do.

Daniel
(Full disclosure: I work for the RIPE NCC)

On 9 Mar 2021, at 18:58, Brian Turnbow via NANOG wrote:


Hello everyone,

We received a strange request that I wanted to share.
An email was sent to us asking to confirm a LOA from a diligent ISP.
The Loa was a request to open bgp for an AS , that is not ours, to 
announce a /23 prefix that is ours.
So basically this entity sent to their upstream a request to announce 
a prefix from one our allocated ranges.
We have the allocation correctly registered and ROAs in place , but it 
is worrisome that someone would attempt this.
Obviously we have informed the ISP that the LOA is not valid and are 
trying to contact the originating party.
Aside from RIRs for the offending AS and our IPs,  Is there anywhere 
to report this type of activity?
We have dealt with hijacking technically speaking in the past but this 
is the first time, to my knowledge, of someone forging a LOA with our 
IPs.


Thanks in advance for any advice

Brian

P.S. a big thanks to Chris for checking the boxes before activating 
the filter if you are on the list!

Re: How to Fix IP GEO for google/youtube tv

[William Guo]

Google has its internal GeoIP team.

But the data quality is not so good, but they don't wanna improve it for
some reason.

On Mon, Mar 8, 2021 at 8:00 AM Nate Burke  wrote:

> If you find out, let me know the secret sauce.  I've had a ticket open
> with YoutubeTV for 7 months and they can't get the GeoIP fixed for one
> of my /21's that I've had for 20 years.  Every reply on my ticket from
> Google is that the YoutubeTV service has major GeoIP issues, and they
> don't know how to fix them.  It's been escalated to engineering for the
> last 7 months with no ETA on a fix.  That is the only service for me
> that is having a location problem.
>
>
> On 3/7/2021 7:19 PM, t...@nwohiobb.com wrote:
> > Can anybody tell me how to fix my IP block for google websites and
> > youtube/tv because I am getting customers saying they are out of the
> > country and I tried to apply for the google isp portal and I have not
> > gotten anything back on this.
> >
> > Thanks Tim
>
>