Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Nico CARTRON]

On 15 Dec 2015, at 18:25, Brett Carr  wrote:
> 
> Thanks for the information Romeo I wonder if perhaps you would consider doing 
> a presentation at the next WG meeting on the issues you encountered and 
> mitigation techniques you used.
> 
> Thanks
> 
> Brett
> 

+1

Cheers,

-- 
Nico

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Brett Carr]

Thanks for the information Romeo I wonder if perhaps you would consider doing a 
presentation at the next WG meeting on the issues you encountered and 
mitigation techniques you used.

Thanks

Brett

--
Brett Carr
Senior DNS Engineer
Nominet UK

> On 15 Dec 2015, at 12:35, Romeo Zwart  wrote:
> 
> Dear colleagues,
> 
> Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services
> were functioning in a severely degraded state during parts of the day.
> 
> This was due to an attack on one of the ccTLDs for which the NCC hosts a
> secondary DNS service. The attack traffic started around 08:00 UTC. RIPE
> NCC staff applied various countermeasures during the day. These
> mitigations were effective for some time. However, after implementing
> each of these mitigations, the traffic patterns were modified to evade
> them. Towards the end of the day, the volume of the attack traffic
> targeted at our servers had increased to such a level that it was
> overloading our incoming links and our mitigation measures were no
> longer sufficiently effective.
> 
> At that time we were forced to contact our upstream peers to assist us
> with mitigation measures. Apart from the ccTLD service for the attacked
> domain, normal services were restored at around 18:30 UTC.
> 
> The attack is ongoing, and we continue with mitigation measures in order
> to provide the best service possible under the circumstances.
> 
> We note that attacks like this rely on spoofing source addresses in the
> attack packets. Therefore, Source Address Validation and BCP-38 should
> be used wherever possible to reduce the ability to abuse networks to
> transmit spoofed source packets.
> 
> Kind regards,
> Romeo Zwart
> 

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Romeo Zwart]

Hi Brett,

On 15/12/15 18:25 , Brett Carr wrote:
> Thanks for the information Romeo I wonder if perhaps you would consider doing 
> a presentation at the next WG meeting on the issues you encountered and 
> mitigation techniques you used.

We will consider it. As you will understand, and will have noticed in
our communication about this, we are trying to balance providing
operationally relevant information about the event with a desire to not
aid in designing any future events. So the information we give will
likely be unsatisfactory for many people in the technical audience we
have here.

However, we might be able to present more information in a somewhat
generalised way that is still useful to the community. As said, we will
consider it.

Regards,
Romeo



> Thanks
> 
> Brett
> 
> --
> Brett Carr
> Senior DNS Engineer
> Nominet UK
> 
>> On 15 Dec 2015, at 12:35, Romeo Zwart  wrote:
>>
>> Dear colleagues,
>>
>> Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services
>> were functioning in a severely degraded state during parts of the day.
>>
>> This was due to an attack on one of the ccTLDs for which the NCC hosts a
>> secondary DNS service. The attack traffic started around 08:00 UTC. RIPE
>> NCC staff applied various countermeasures during the day. These
>> mitigations were effective for some time. However, after implementing
>> each of these mitigations, the traffic patterns were modified to evade
>> them. Towards the end of the day, the volume of the attack traffic
>> targeted at our servers had increased to such a level that it was
>> overloading our incoming links and our mitigation measures were no
>> longer sufficiently effective.
>>
>> At that time we were forced to contact our upstream peers to assist us
>> with mitigation measures. Apart from the ccTLD service for the attacked
>> domain, normal services were restored at around 18:30 UTC.
>>
>> The attack is ongoing, and we continue with mitigation measures in order
>> to provide the best service possible under the circumstances.
>>
>> We note that attacks like this rely on spoofing source addresses in the
>> attack packets. Therefore, Source Address Validation and BCP-38 should
>> be used wherever possible to reduce the ability to abuse networks to
>> transmit spoofed source packets.
>>
>> Kind regards,
>> Romeo Zwart
>>
> 
> 

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Marek Vavruša]

Same question as for the root incident - would you be willing to share
more information OTR with software implementors (such as well, me)?
Pinky swear that I'm not the perpetrator.

Best,
Marek

On 15 December 2015 at 18:48, Romeo Zwart  wrote:
> Hi Brett,
>
> On 15/12/15 18:25 , Brett Carr wrote:
>> Thanks for the information Romeo I wonder if perhaps you would consider 
>> doing a presentation at the next WG meeting on the issues you encountered 
>> and mitigation techniques you used.
>
> We will consider it. As you will understand, and will have noticed in
> our communication about this, we are trying to balance providing
> operationally relevant information about the event with a desire to not
> aid in designing any future events. So the information we give will
> likely be unsatisfactory for many people in the technical audience we
> have here.
>
> However, we might be able to present more information in a somewhat
> generalised way that is still useful to the community. As said, we will
> consider it.
>
> Regards,
> Romeo
>
>
>
>> Thanks
>>
>> Brett
>>
>> --
>> Brett Carr
>> Senior DNS Engineer
>> Nominet UK
>>
>>> On 15 Dec 2015, at 12:35, Romeo Zwart  wrote:
>>>
>>> Dear colleagues,
>>>
>>> Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services
>>> were functioning in a severely degraded state during parts of the day.
>>>
>>> This was due to an attack on one of the ccTLDs for which the NCC hosts a
>>> secondary DNS service. The attack traffic started around 08:00 UTC. RIPE
>>> NCC staff applied various countermeasures during the day. These
>>> mitigations were effective for some time. However, after implementing
>>> each of these mitigations, the traffic patterns were modified to evade
>>> them. Towards the end of the day, the volume of the attack traffic
>>> targeted at our servers had increased to such a level that it was
>>> overloading our incoming links and our mitigation measures were no
>>> longer sufficiently effective.
>>>
>>> At that time we were forced to contact our upstream peers to assist us
>>> with mitigation measures. Apart from the ccTLD service for the attacked
>>> domain, normal services were restored at around 18:30 UTC.
>>>
>>> The attack is ongoing, and we continue with mitigation measures in order
>>> to provide the best service possible under the circumstances.
>>>
>>> We note that attacks like this rely on spoofing source addresses in the
>>> attack packets. Therefore, Source Address Validation and BCP-38 should
>>> be used wherever possible to reduce the ability to abuse networks to
>>> transmit spoofed source packets.
>>>
>>> Kind regards,
>>> Romeo Zwart
>>>
>>
>>
>
>

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Jacques Latour]

Hi Romeo,
Perhaps you can share more details in the member only part of the next DNS-OARC 
session?
Jacques


> -Original Message-
> From: dns-wg [mailto:dns-wg-boun...@ripe.net] On Behalf Of Romeo Zwart
> Sent: December-15-15 12:48 PM
> To: Brett Carr
> Cc: RIPE DNS Working Group
> Subject: Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on
> Monday 14 December
> 
> Hi Brett,
> 
> On 15/12/15 18:25 , Brett Carr wrote:
> > Thanks for the information Romeo I wonder if perhaps you would consider
> doing a presentation at the next WG meeting on the issues you encountered
> and mitigation techniques you used.
> 
> We will consider it. As you will understand, and will have noticed in our
> communication about this, we are trying to balance providing operationally
> relevant information about the event with a desire to not aid in designing any
> future events. So the information we give will likely be unsatisfactory for
> many people in the technical audience we have here.
> 
> However, we might be able to present more information in a somewhat
> generalised way that is still useful to the community. As said, we will 
> consider
> it.
> 
> Regards,
> Romeo
> 
> 
> 
> > Thanks
> >
> > Brett
> >
> > --
> > Brett Carr
> > Senior DNS Engineer
> > Nominet UK
> >
> >> On 15 Dec 2015, at 12:35, Romeo Zwart  wrote:
> >>
> >> Dear colleagues,
> >>
> >> Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS
> >> services were functioning in a severely degraded state during parts of the
> day.
> >>
> >> This was due to an attack on one of the ccTLDs for which the NCC
> >> hosts a secondary DNS service. The attack traffic started around
> >> 08:00 UTC. RIPE NCC staff applied various countermeasures during the
> >> day. These mitigations were effective for some time. However, after
> >> implementing each of these mitigations, the traffic patterns were
> >> modified to evade them. Towards the end of the day, the volume of the
> >> attack traffic targeted at our servers had increased to such a level
> >> that it was overloading our incoming links and our mitigation
> >> measures were no longer sufficiently effective.
> >>
> >> At that time we were forced to contact our upstream peers to assist
> >> us with mitigation measures. Apart from the ccTLD service for the
> >> attacked domain, normal services were restored at around 18:30 UTC.
> >>
> >> The attack is ongoing, and we continue with mitigation measures in
> >> order to provide the best service possible under the circumstances.
> >>
> >> We note that attacks like this rely on spoofing source addresses in
> >> the attack packets. Therefore, Source Address Validation and BCP-38
> >> should be used wherever possible to reduce the ability to abuse
> >> networks to transmit spoofed source packets.
> >>
> >> Kind regards,
> >> Romeo Zwart
> >>
> >
> >
> 

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Jaap Akkerhuis]

 Romeo Zwart writes:

 > Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services
 > were functioning in a severely degraded state during parts of the day.
 > 
 > etc.

According a message from Stephane Bortzmeyer

   "The RIPE name server was retired on 16 december, for unknown
reasons (as far as I know, the RIPE-NCC did not communicate on
that)."

Can you comment on that? 

Thanks,

jaap

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Romeo Zwart]

Hi Jaap,

On 15/12/29 13:08 , Jaap Akkerhuis wrote:
>  Romeo Zwart writes:
> 
>  > Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services
>  > were functioning in a severely degraded state during parts of the day.
>  > 
>  > etc.
> 
> According a message from Stephane Bortzmeyer
> 
>"The RIPE name server was retired on 16 december, for unknown
> reasons (as far as I know, the RIPE-NCC did not communicate on
> that)."
> 
> Can you comment on that? 

With this limited amount of information, that would be hard. Which zones
are we talking about and what does 'retired' mean in this context?

I haven't seen Stephane's message (yet). Was that a private message or
sent to a mailing list? Can you forward the whole message or have
Stephane provide more detail about his observations directly to me?

Thanks,
Romeo

> 
> Thanks,
> 
>   jaap
> 

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Jaap Akkerhuis]

 Romeo Zwart writes:

 > Hi Jaap,
 > 
 > On 15/12/29 13:08 , Jaap Akkerhuis wrote:
 > >  Romeo Zwart writes:
 > > 
 > >  > Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services
 > >  > were functioning in a severely degraded state during parts of the day.
 > >  > 
 > >  > etc.
 > > 
 > > According a message from Stephane Bortzmeyer
 > > 
 > >"The RIPE name server was retired on 16 december, for unknown
 > > reasons (as far as I know, the RIPE-NCC did not communicate on
 > > that)."
 > > 
 > > Can you comment on that? 
 > 
 > With this limited amount of information, that would be hard. Which zones
 > are we talking about and what does 'retired' mean in this context?
 > 
 > I haven't seen Stephane's message (yet). Was that a private message or
 > sent to a mailing list? Can you forward the whole message or have
 > Stephane provide more detail about his observations directly to me?
 > 

It seems that I have indeed removed to much of the context.

Stephane's message was on the centr security list which archives
seem to be sealed (contrary to what I thought). It was refering to
the attack on the .tr name servers about which you reported in

that it had impacted RIPE's DNS service. Apparently Stephan wanted
to know why RIPE NCC dropped serving the .tr zone. (My guess, since
de RIPE NCC dropped out of the root zone as well, it was done in
coordination with the tr people).

So I was just curious wat happened on RIPE's end.

jaap

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Romeo Zwart]

Hi Jaap,

> On 29 dec. 2015, at 19:58, Jaap Akkerhuis  wrote:
> 
> Romeo Zwart writes:
> 
>> Hi Jaap,
>> 
>>> On 15/12/29 13:08 , Jaap Akkerhuis wrote:
>>> Romeo Zwart writes:
>>> 
 Yesterday, Monday 14 December 2015, RIPE NCC Authoritative DNS services
 were functioning in a severely degraded state during parts of the day.
 
 etc.
>>> 
>>> According a message from Stephane Bortzmeyer
>>> 
>>>   "The RIPE name server was retired on 16 december, for unknown
>>>reasons (as far as I know, the RIPE-NCC did not communicate on
>>>that)."
>>> 
>>> Can you comment on that?
>> 
>> With this limited amount of information, that would be hard. Which zones
>> are we talking about and what does 'retired' mean in this context?
>> 
>> I haven't seen Stephane's message (yet). Was that a private message or
>> sent to a mailing list? Can you forward the whole message or have
>> Stephane provide more detail about his observations directly to me?
> 
> It seems that I have indeed removed to much of the context.
> 
> Stephane's message was on the centr security list which archives
> seem to be sealed (contrary to what I thought). It was refering to
> the attack on the .tr name servers about which you reported in
> 
> that it had impacted RIPE's DNS service.

Ah ok, some context helps. :) 

> Apparently Stephan wanted
> to know why RIPE NCC dropped serving the .tr zone. (My guess, since
> de RIPE NCC dropped out of the root zone as well, it was done in
> coordination with the tr people).

Indeed it was. 

> So I was just curious wat happened on RIPE's end.

We can share some more detail next week. 

Kind regards,
Romeo

> 
>jaap
> 

Re: [dns-wg] RIPE NCC Authoritative and Secondary DNS services on Monday 14 December

[Romeo Zwart]

Dear Jaap and colleagues,

On 29 December you wrote to the list:

> Stephane's message was on the centr security list which archives
> seem to be sealed (contrary to what I thought). It was refering to
> the attack on the .tr name servers about which you reported in
> 
> that it had impacted RIPE's DNS service. Apparently Stephan wanted
> to know why RIPE NCC dropped serving the .tr zone. (My guess, since
> de RIPE NCC dropped out of the root zone as well, it was done in
> coordination with the tr people).
>
> So I was just curious wat happened on RIPE's end.

In the incident report you reference above, I did not mention the .TR
zone explicitly, which apparently led to unnecessary confusion and an
undesired atmosphere of secrecy around the incident.

I did mention in the same message that, after applying various
mitigation measures during the day, we turned to our upstreams to assist
us with mitigation in the late afternoon of Monday 14th. In practice
this meant we asked for upstream blackholing of the attack traffic,
which effectively meant we were no longer serving the .TR zone.

While the event was ongoing, we were of course communicating with the
.TR staff frequently. On Tuesday morning, 15 December, the .TR staff
informed us that they removed the RIPE NCC secondary server from the .TR
zone altogether.

I hope this clarifies matters sufficiently. If you have more questions
please feel free to ask. I should add, however, that we do not intend to
share more details about the attack itself, or the mitigation applied,
on this list.

An observation that we have made during the past months is that the
impact of attacks upon our DNS infrastructure is increasing. This seems
to be a more general trend that readers on this list are likely to be
aware of, but this may not be the case for the community at large. For
the RIPE NCC this means that we are investigating the options to
increase the capacity and robustness of our DNS services further.

Kind regards,
Romeo Zwart