Re: [atlas] [NTP measurements] Minimum offset if negative

[Philip Homburg]

In your letter dated Fri, 28 Jun 2024 13:24:21 +0200 you wrote:
>On Fri, Jun 28, 2024 at 12:14:45PM +0200,
> Philip Homburg  wrote 
> a message of 11 lines which said:
>
>> In math. min(-2, -1) = -2
>> 
>> Why would that be different for offset?
>
>Because this is not Math 101 but NTP. You typically don't care about
>the sign of the offset, only about its absolute value.

So you want the smallest and largest absolute offset. You already get
the minimum and maximum offset. So the maximum absolute offset should be
easy to spot.

Showing only the maximum absolute offset is not great. It is useful to know
if on average probes are behind or ahead of the time source. Only showing
absolute offsets doesn't show that.

Why would you want to know the smallest absolute offset?  That's just
measuring noise.

-- 
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] [NTP measurements] Minimum offset if negative

[Philip Homburg]

> Measurement #74488034
> says on : "Min Offset:
> -1173,799". Since offsets can be positive or negative, I suggest
> it is not a good idea to take the minimum without first removing
> the sign. Here, the real minimum offset was -0.000345 (the closest
> from the server) and the maximum -1.173799 (the farthest from the
> server).

In math. min(-2, -1) = -2

Why would that be different for offset?


-- 
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] Probe DHCPv6 support

[Philip Homburg]

In your letter dated Tue, 9 May 2023 19:17:41 +0200 you wrote:
>That you do not see more of it is mostly Android's fault - because
>enterprises get told "it will not work with half the mobile devices",
>so they go and postpone IPv6 deployment instead (in places where the
>control bit is strictly required)...

Android devices show up at relatively untrusted wifi networks. I can see
the problem there. Or are you saying the these days enterprises are
100% cloud based, and the only networks they have left are wifi?

In many cases untrusted devices only communicate with the internal network over 
a VPN. In that case, the local access protocol doesn't matter much. The VPN
can hand out IPv6 address even if the host is IPv4-only.

In any case, Atlas probes run a stripped down version of OpenWRT. It may help
if somebody can figure out which OpenWRT packages need to be added to 
enable DHCPv6.

-- 
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] Probe DHCPv6 support

[Philip Homburg]

>Dear RIPE Atlas folks, why doesn't Atlas probes support DHCPv6 in 2023?
>
>All the stubborn reasons given why Android can't support DHCPv6 do not
>apply, and for the environment my v4 probe is running in, DHCPv6 is what
>we use - so it's DHCPv6 or no v6...

A random question about the state of the art in this area. Does OpenWRT
out of the box support DHVPv6? Did you ever try that? If it does, it may
help the Atlas team to just add the relevant OpenWRT packages.

It seems that networks that support only DHCPv6 IN_NA for IPv6 are quite rare. 

-- 
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] Proposal: Remove support for non-public measurements [ONLY-PUBLIC]

[Philip Homburg]

>I might be troubleshooting something in our network where I have no
>interest in making the results public.  So I value the option to have
>non-public measurements.
>
>There's no "right to see all measurements" here - if someone wants to
>see something, they are free to run their own measurements with their
>own credits.  What I do with my credits (which do not come for free)
>and who can see the results should be my decision.

The problem with non-public measurements for the RIPE NCC (at least, when I
was still working there, but I doubt the situation has changed), is
that keeping track of whether a measurement result is public or not causes 
significant code complexity.

So the question is really whether the relatively small amount of use that is
made of non-public measurements is worth the cost to RIPE NCC members 
for the ongoing maintainance of this feature.

Without non-public measurements, measurement results could be stored anywhere,
meta-data of measurements could be stored aynwhere.

With non-public measurrements, some measurement results have to stored such
that they are non generally accessible. Of course API endpoints do need
access to that data. At some point there is just too much code that needs
to know about non-public measurements.

-- 
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

[atlas] Moving on

[Philip Homburg]

Hi Atlas Users,

After ten years at the RIPE NCC it is time for me to move on. Normally,
that is not something that would be announced on a public mailing list.
However, I get quite a bit of email that is directly sent to me.

This is to let you know that I will not be able to read future email 
sent to this address. I have two new colleagues who will work on probe
firmware development. They are Guy Meyer  and Michel 
Stam . I'm sure they will be happy to help.


As for me, I will be  from now on.

Philip Homburg

--
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] Problem with DoT measurements?

[Philip Homburg]

On 2022/03/24 13:33 , Stephane Bortzmeyer wrote:

DoT measurements seem partially broken, only 10 % of the probes do
report. (See for instance #39675721.)


Unfortunately, a small configuration error on the Atlas backend throws 
out the results. I hope we can fix this soon.


--
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] monthly information about probe availability

[Philip Homburg]

On 2022/02/14 16:47 , MAYER Hans wrote:

many thanks for your answer.
Do you know is this a single server ? What if the server is down ? Maybe it’s 
anycast.


There is a collection of about 30 servers. When a server is down for 
longer time (a number hours) probes are automatically distributed over 
other servers.


Philip

--
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] How to run the RIPE Atlas software probe on FreeBSD

[Philip Homburg]

On 2022/02/07 15:08 , Viktor Naumov wrote:
I'm running it in centos in bhyve using cbsd. 
https://cbsd.io/cbsd/freebsd-bhyve/


Not sure if anyone succeeded compiling native binaries.


I just tried to build the binaries on FreeBSD. It will be quite a bit of 
work (looking at the errors from the compiler). One known issue is that 
it will be hard to get TCP traceroute to work on FreeBSD.


Philip

--
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] My probe's not doin' IPv6

[Philip Homburg]

On 2022/01/22 0:12 , Edward Lewis wrote:

General question…

My probe was doing IPv6 until 15 July 2021 but not it is not.  Is there 
any way to discover why it is no longer IPv6’ing?


Assuming this is about probe 22382, then probe doesn't seem to receive 
any (useful) IPv6 router advertisement. I have no clue why.


To find out, you could connect a switch between the probe and the router 
and use another computer to take a look at the router advertisements.


If your IPv6 prefix is stable enough, you could manually assign an IPv6 
address to the probe and see what happens.


Philip

--
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] probe V2 status lights

[Philip Homburg]

On 2022/01/18 10:35 , Jaap Akkerhuis wrote:

There used to be information telleing what the lights mean. It seems
to be gone. After a lot of searching around one finds some information
about blinking orage lights but all leds in version 2 are green
colored so I guess this cvers V3 (or V1?) probes.


The LEDs on V1 and V2 probes are regular ethernet activity LEDs. I think 
one is link status and the other activity, but I'm not completely sure.


If they are not blinking then there is something wrong with the probe's 
ethernet connection. It could also be a lack of power. V2 probes are 
getting more hardware failures, but not at a rapid rate.


Philip

--
ripe-atlas mailing list
ripe-atlas@ripe.net
https://lists.ripe.net/mailman/listinfo/ripe-atlas

Re: [atlas] swapping a v1 for a v3

[Philip Homburg]

On 2021/12/10 15:46 , Sander Steffann wrote:

Data is firmly associated with a probe ID. And hardware probes all have their 
own unique IDs.


So… How do we get someone to do "UPDATE data SET probe_id=25898 WHERE 
probe_id=2285 ;)


Simple, download the data dumps, import them into Google Bigtable, run 
the update and convince the RIPE NCC to drop the hadoop cluster. :-)

Re: [atlas] swapping a v1 for a v3

[Philip Homburg]

On 2021/12/10 1:37 , Randy Bush wrote:

a sweet old v1 probe, 2285, in a hard to access (due to covid) place
died 19 months ago.  i want to swap in a v3, 25898, this weekend.  i
know i could create a new entry, but how do i delete the old; or better
yet, move the data from old to new?  stop laughing.  :)


For measurements, you can add the new probe and remove the old one. The 
API is called 'participation-requests'.


Data is firmly associated with a probe ID. And hardware probes all have 
their own unique IDs.


Philip

Re: [atlas] ripe atlas Probe 32256 is disconnected firm ware upgrade com326.193.612

[Philip Homburg]

On 2021/10/12 11:54 , kiran.gun...@colruytgroup.com wrote:
Noticed our probe's firmware got upgraded and from there on our 
measurements stopped working.


2021-10-10 03:31:00 UTC Probe firmware upgraded Your probe #32256 upgraded its 
firmware from version 5020 to version 5030

Any clue on how we can fix this, current measurements does not work and  
tried creating new measurements which also does not work.


  * Old Measurement : Ping measurement to www.hln.be: ID23382497
  o Last successfull measurment was at 2021-10-10 02:46
  * New Measurement : Ping measurement to www.f5.com from probe 32256 :
ID32499756

*Recevied following email from ripe:*

/Your probe 32256 has been disconnected from the RIPE Atlas 
infrastructure since 2021-10-10 02:39:46 UTC. You may be able to learn 
more about the possible cause from the "Status" tab on your probe's 
information page: https://atlas.ripe.net/probes/32256/#!tab-status/


Current status says connected and no measurement works.


Hi,

In this case it seems that the USB stick might be corrupted. Please try 
to the trick with turning the probe on without USB stick.


Philip

Re: [atlas] All-Probe Traceroute + detect RFC1918 addresses

[Philip Homburg]

On 2021/09/15 21:59 , Bjørn Mork wrote:

I believe it's better to ignore the formalities here and forward those
packets.  It's certainly harmless.  At least as harmless as forwarding
any other ICMP error messages.


In my opinion this is the wrong approach. If there is a good reason to 
violate the standard, then the thing to do is to bring up in the IETF.


Philip

Re: [atlas] All-Probe Traceroute + detect RFC1918 addresses

[Philip Homburg]

On 2021/09/15 19:06 , Gert Doering wrote:

On Wed, Sep 15, 2021 at 06:58:50PM +0200, Philip Homburg wrote:

My router at home gets a lot of error icmps with link local source, from
quite a few hops away...


*wake up*  IPv4 "link local" (169.254.*) or IPv6 LLA (fe80::)?


IPv6 link local


The latter would make me very curious what sort of intermediate devices
are forwarding these (which is strictly forbidden).


As far as I know, this a well known Juniper bug where their routers 
forward errors ICMPs without checking whether the source address is link 
local. I don't know what happens to other packets with link local source.


Philip

Re: [atlas] All-Probe Traceroute + detect RFC1918 addresses

[Philip Homburg]

On 2021/09/15 17:32 , Jeroen Massar via ripe-atlas wrote:

Has anybody ever run a all-probe traceroute and then to detect any RFC1918 
addresses in there? (though many probes will have locally some RFC1918)


Atlas performs quite a few traceroutes on all probes. For example, 
https://atlas.ripe.net/measurements/5017/ which is a unicast destination.


So anything close to the probes should show up.


Of course, one should also do that for IPv6; though I expect outside the stray ULA 
address (thank you apple; though they are fixing that ULA issue with homepods apparently) 
very little of it, though "meten is weten" (measuring is knowing).


My router at home gets a lot of error icmps with link local source, from 
quite a few hops away...

Re: [atlas] DNS resolution

[Philip Homburg]

On 2021/08/09 13:47 , Thomas Schäfer wrote:
I have a simple question. After crating a measurement how often the name 
of the target is resolved?


E.g. I want to ping a target with a stable dyndns name, but with 
changing ip addresses.


It seems to me that most probes don't refresh the ip address once the 
measurement has started, even it is just a daily ping.


There is an option in the measurement specification called 'Resolve on 
Probe'. Without this option, the name is resolved when the measurement 
specification is submitted and probe's target a fixed IP address.


With 'Resolve on Probe', the probe will try to resolve the name each 
time it performs the measurement.


Philip

Re: [atlas] Getting RRSIG records when do bit is set

[Philip Homburg]

On 2021/07/06 17:21 , pravicha wrote:

This is another measurement, which has the do bit set, 31426253.


This one does indeed have the DO flag set.

Where it goes wrong is that Atlas DNS measurements default to a UDP 
buffer size of 512 octets. The result that comes back is truncated.


In measurement 31428793, I set the buffer to 2048.

Philip

Re: [atlas] Getting RRSIG records when do bit is set

[Philip Homburg]

On 2021/07/06 16:18 , pravicha wrote:

The measurement ID is 31426231. I query for DNSKEY records, have the do bit set 
in this and yet don’t get back the RRSIGs.


Hi,

DNSSEC OK flag is false in this measurement. See 
https://atlas.ripe.net/measurements/31426231/#general and then 'DNS 
Specific Settings'. Maybe something went wrong in the creation of the 
measurement. What tool did you use?


Philip

Re: [atlas] Getting RRSIG records when do bit is set

[Philip Homburg]

On 2021/07/06 3:41 , pravicha wrote:

When I set the do bit in dig, I get back RRSIG records associated with the 
Query Type, but I don’t get the RRSIGs when I create a measurement with do bit 
set in Ripe Atlas. Can I replicate the same behavior in RIPE atlas?


Hi,

Can you give a measurement ID that didn't work as expected?

Philip

Re: [atlas] Feature request for Validated Timestamps

[Philip Homburg]

On 2021/04/14 6:00 , Will wrote:
To Philip's point, the basic protocol I am hoping to enable looks as 
follows:

(I'll describe the roughtime variant)
* A machine wants to attest that it is in (e.g. Seattle). It picks a set 
of machines that are believed to be in Seattle (perhaps atlas machines, 
or more generally machines that the measurement system has agreed on 
locations of in advance.)

* It generates a statement saying "i am doing a measurement", and signs it.
* It requests the time with it's own signature as a nonce and receives a 
(timestamp, uncertainty, signature) from the server, per 
https://blog.cloudflare.com/roughtime/ 

* It then immediately req-requests the time, using the signature it 
received as the nonce this time, and receives a second timestamp, 
uncertainty, signature.


It seems to me that this can be simplified a bit, the first step can be 
just a nonce. Basically:

- The client generates a random nonce
- The client uses to nonce to obtain the time from a server using the 
nonce and the roughtime protocol

- The client generates a new nonce from the reply by hashing the reply
- The client uses the second nonce to once again obtain the time.
- The client packs the two nonces and the two replies and reports it as 
measurement result.


Later, a verifier can check the signatures in the two replies to verify 
that responses came from the right server. Then a check is made that the 
second nonce is derived from the first reply. Finally a check is made 
that the replies correspond to the nonces. Note that the first nonce 
serves no value, so could be omitted from the measurement result and 
verification.


Open issue: key management. How to store all public keys of all servers 
for ever.


This seems fine, however...

There are some complexities - e.g could the machine attesting it's 
location delegate the request to a different machine closer to the 
anchor? Depending on the situation there are mitigations for this, like 
asking that some piece of data the machine that's attesting it's 
location be hashed into the nonce, in a way that's difficult for the 
attesting machine to predict ahead of time (so that it would need to 
move all of its data to the delegate machine at which point it's already 
in a sense also in that secondary location at that time.) But the 
ability to locate client software deployment via latency with some 
guarantee that someone running it isn't spoofing their location is useful.


If we assume that the client is malicous, what prevents the client from 
using a collaborating machine to execute the protocol, proving the 
location of the collaborating machine. The client just submits the 
results as its own?


Philip

Re: [atlas] Feature request for Validated Timestamps

[Philip Homburg]

On 2021/04/06 17:04 , Will wrote:
I'm not sure if there's a process for this sort of feature request 
beyond this mailing list. Would it help if I propose a more concrete PR 
against https://github.com/RIPE-NCC/ripe-atlas-software-probe 



To the extend that you would like secure geolocation in presence of a 
malicious probe, it would make sense to me to start with documenting the 
protocol you would like to use.


The current way of geolocating probes works by having to probe report 
round trip times to a number locations. Obviously, a malicious probe 
could report any rtt value.


I'm sure we can come up with a protocol if we have a sufficient number 
of trusted servers. However, such a protocol would need to be documented 
and deployed on those servers.


Philip

Re: [atlas] What to do with broken v1 probe?

[Philip Homburg]

On 2021/03/21 8:18 , Lorenzo Colitti via ripe-atlas wrote:
After more than 10 years (!) of 99% uptime (see below), my probe v1 
appears to have given up the ghost. When connected to power, the lights 
still come on, but after a few seconds link bounces, and it never sends 
any packets.


Yes, it is a pity that they stop working.

Any idea what to do? Should I just apply for a new probe? Are new probes 
being given out these days?


It will take a while before we have new hardware probes. Probably at the 
end of this year. At the moment the best option is a software probe.


Philip

Re: [atlas] [mat-wg] RIPE Atlas testing of reserved IPv4 addresses

[Philip Homburg]

On 2021/02/18 14:13 , Chriztoffer Hansen wrote:

On Thu, 18 Feb 2021 at 12:06, Philip Homburg wrote:

For a binary install of the CentOS RPM it should be automatic. For
debian based, it is a matter of compiling a new .deb and upgrading
(manually).


An idea for a system-tag in the RIPE Atlas system?

I.e. based on the (client) atlas probe software revision. Restrict
which probes/anchers would be permitted/allowed to run tests for IP's
in e.g. 240/4 space...


That is part of the API backend. When probes are selected for a 
measurement, only those probes are selected that have a high enough 
firmware version. Though I have to admit, that will make this feature 
slightly more complex than I thought.

Re: [atlas] [mat-wg] RIPE Atlas testing of reserved IPv4 addresses

[Philip Homburg]

On 2021/02/17 17:56 , Randy Bush wrote:

I just did a small experiment with the probe firmware. Allowing
240.0.0.0/4 and 0.0.0.0/8 is a two line change. I tested this on V3
and V4 probes and on CentOS 7. In all 3 cases, the linux kernel does
support 240.0.0.0/4 and does not support 0.0.0.0/8.


so, what experiment(s) do you think would be useful once some capable
probes deploy?


Traceroute seems to be the only measurement that makes sense. I noticed 
that my first hop router sends an error ICMP, but I don't get anything 
beyond that.



We are not going to releasing new firmware for V1 and V2 (except to
address security issues).


i can not get to my v1 probes during covid lockdown even if you shipped
replacements to me :(  < snif >


Yes, it is amazing that corona doesn't seem to have much in the number 
of probes that are connected.



will you upgrade soft probes?  or send instructions?


For a binary install of the CentOS RPM it should be automatic. For 
debian based, it is a matter of compiling a new .deb and upgrading 
(manually).


Philip

Re: [atlas] [mat-wg] RIPE Atlas testing of reserved IPv4 addresses

[Philip Homburg]

On 2021/02/16 19:39 , Randy Bush wrote:

I think the answer very much depends on who you ask.


with my researcher hat on, i am curious yellow.  with a minor concern
for how much of the lab's resources it would consume.


I just did a small experiment with the probe firmware. Allowing 
240.0.0.0/4 and 0.0.0.0/8 is a two line change. I tested this on V3 and 
V4 probes and on CentOS 7. In all 3 cases, the linux kernel does support 
240.0.0.0/4 and does not support 0.0.0.0/8.


We are not going to releasing new firmware for V1 and V2 (except to 
address security issues).


So this a relatively small change that can go out with a future firmware 
release. Currently the Atlas API also blocks these netblocks. I assume, 
but did not verify, that this will also be a small change.


In short, supporting this request is going to take only a tiny bit of 
time to support.


Philip

Re: [atlas] atlas probe down : ID 35603

[Philip Homburg]

On 2021/02/08 11:26 , MAYER Hans wrote:

Now I tried a third USB memory stick. It has 8GB and it seems working.
When I check https://atlas.ripe.net/probes/35603/#tab-network it says connected 
for more then one hour.
So it should be OK for the future ? What do you think ?


... Unfortunately we are out of hardware probes due to Corona


Just in case it would die completely, do you think you will get some in the 
future or will it go into soft probe only ?


Hi Hans,

The probe looks fine now.

We are working on new hardware probes, but there is no clear date yet 
when we will have them. For the coming years we want to have hardware 
probes next to software probes (and anchors).


Philip

Re: [atlas] atlas probe down : ID 35603

[Philip Homburg]

On 2021/02/03 11:53 , MAYER Hans wrote:

many thanks for your response.
This I didn’t see, even if it is listed several times.
So will I try a third USB stick. Is there a recommended size ? Is 64 GB too big 
?
And what if this will not help too ?
Due to covid-19 I am working from home most of the time. So it will take some 
days until I can do the change.


Hi,

I never tried a bigger USB stick. But I can't think of any reason why it 
wouldn't work. If the probe gives 'NO-USB' even with a known good USB 
stick then it best to assume that the probe is broken. Unfortuantely we 
are out of hardware probes due to Corona. So in that case, it is best to 
replace the probe with a software probe.


Philip

Re: [atlas] atlas probe down : ID 35603

[Philip Homburg]

On 2021/01/29 14:34 , MAYER Hans wrote:
But when I go to https://atlas.ripe.net/probes/35603/ 
  I see it is offline. The web 
page says „no flash drive“ and „trying to connect“.


On the page https://atlas.ripe.net/probes/35603/#tab-network at the 
bottom it says NO-USB.


If you get that with a known good USB stick then there is a chance that 
the USB interface of the probe is broken.


Philip

Re: [atlas] Automating software probe creation?

[Philip Homburg]

On 2021/01/02 8:25 , Steve Gibbard wrote:

As far as I can tell from the documentation, it looks like the probes are 
identified by the encryption keys in /var/atlas-probe/etc/probe-key* .  If 
creating new probes from an image, is it sufficient to make each probe generate 
its own encryption keys, or are there other files that need to change?


If you build the debian package, then you can install that as many times 
as you like.


Probes are indeed identified by their public keys.


For registering the probes, I see a web form that requires being logged into a 
RIPE account, and don’t see any equivalent in the API documentation.  Is there 
a way to register new software probes via the API?


We are working on an API. However, the only thing the API will do is to 
copy the public key to the atlas backend systems. You still have go to 
the web size, log in, etc. We want to associate a user with each probe.


Philip

Re: [atlas] DoH / DoT test in Atlas Probes

[Philip Homburg]

On 2020/10/29 15:28 , Imre Tuskes wrote:
> Hello,
> Is there any plan to measure DNS over HTTPs (or TLS).
> We are interested in the performance of this kind of service.
> Regards, Imre  

DNS over TLS is available today.

DNS over HTTPS require new code, in particular because HTTPS should be
HTTPS/2.

I'm not sure how representative Atlas measurements would be when it
comes to performance. Atlas sets up a new connection for each DNS
request so you cannot measure whether the service can reorder requests, etc.

Re: [atlas] probe offline/lock up

[Philip Homburg]

On 2020/07/05 19:55 , Bryan Fields wrote:
> Is this normal or is there a problem with my probe?  I've never really taken
> this close of a look before, I'm unsure if it has an issue or if this is 
> normal.

Do you see any ARP request coming from the probe? It seems that the
probe cannot reach it's default router on IPv4.

Philip

Re: [atlas] dead v1 probe

[Philip Homburg]

On 2020/06/03 21:08 , Wilfried Wöber wrote:
> As I do not have a static machine there, just a laptop which goes with me
> onto the road, my Q is: what is the cheapest / recommended / smallest piece
> of hardware which can support a software probe? Any pointers for more info?

The debian version of the software probe runs on a raspberry pi with
raspbian. Any other small device with debian is probably fine as well.



signature.asc
Description: OpenPGP digital signature

Re: [atlas] RIPE Atlas Software Probes and Ubuntu 20.04 / glibc 2.31

[Philip Homburg]

On 2020/05/24 13:04 , Bernd Strehhuber wrote:
> with newer Linux Dists / glibc Versions (tested on Ubuntu 20.04 LTS)
> stime() has been deprecated and replaced with clock_settime().
> Busybox got the following commit back in 11/2019:
> 
> https://git.busybox.net/busybox/commit/?id=d3539be8f27b8cbfdfee460fe08299158f08bcd9
> 
> Compiling on the latest Ubuntu version I applied the attached diff
> to get things working again.

I put a version in the devel branch of the software probe that refers to
a newer version of busybox code, which no longer uses stime. Could you
please try that version and see if it works on Ubuntu?

Philip

Re: [atlas] dead v1 probe

[Philip Homburg]

On 2020/06/02 20:28 , Gert Doering wrote:
> have there been software updates for v1 probes recently?  I've heard
> about a few v1 probes that have died "just last week" - mine among
> them (#461), fell off the net with "Firewall Problems Suspected",
> but it's no longer even ARPing properly (= either it totally lost
> its config and wants DHCP now, which there isn't in this network,
> or it just died)...

No there haven't been any software updates for a very long time. For
this reason they should be considered obsolete. However, we currently
still support them, and we will try to release new firmware if there is
a security issue.

It seems that v1 and v2 probes are now starting to die faster. I have no
 idea what part is wearing out.

Unfortuantely, due to corona, we are out of v4 probes. So the best thing
to do at the moment is to replace dead v1/v2 probes with software probes.

Philip



signature.asc
Description: OpenPGP digital signature

Re: [atlas] RIPE Atlas Software Probes and Ubuntu 20.04 / glibc 2.31

[Philip Homburg]

Hi Bernd,

On 2020/05/24 13:04 , Bernd Strehhuber wrote:
> with newer Linux Dists / glibc Versions (tested on Ubuntu 20.04 LTS)
> stime() has been deprecated and replaced with clock_settime().
> Busybox got the following commit back in 11/2019:
> 
> https://git.busybox.net/busybox/commit/?id=d3539be8f27b8cbfdfee460fe08299158f08bcd9
> 
> Compiling on the latest Ubuntu version I applied the attached diff
> to get things working again.

I was working on removing busybox code that is not used by the Atlas
measurement code. That makes it unnecessary to patch the busybox code.

Replacing an stime call directly with clock_settime is not going to work
(that code never runs on software probes, so I understand why it works
for you). I'll work on a better patch. In any case, thanks.

Philip

Re: [atlas] Feature request DNS DoH measurement

[Philip Homburg]

On 2020/05/23 8:35 , Dave . wrote:
> Would it be possible for your servers to first verify whether a DOH
> address is really a DNS before running actual atlas tests? If you can do
> it from an IP address that also hosts a web page that explains the
> purpose of the test, anyone investigating traffic coming to them is
> easily informed.

Some people want to use DoH from within a browser. If that gets popular,
it could be that many webservers would also have DoH endpoints.

In any case, for now that might be a sensible solution. Some time ago it
was proposed that the MAT working group would handle policy proposals
for Atlas. So, whoever wants to make the effort to push the policy
proposal through, please contact the chairs of the MAT wg on how they
would like to handle this.

Philip

Re: [atlas] Feature request DNS DoH measurement

[Philip Homburg]

On 2020/05/20 22:00 , Yang Yu wrote:
> As DoH is getting more adoption, it would be interesting to have DoH
> query support on Atlas. With support added as an additional protocol
> for DNS measurement (currently TCP/UDP), most measurement
> creation/result parsing settings can be reused.

>From a technical point of view it is not that simple. RFC 8484
recommends at least HTTP/2. Currently there is no support for HTTP/2 in
the Atlas measurement code.

The bigger problem however is that there is a policy for RIPE Atlas to
not allow http requests to arbitrary destinations. The reasoning is that
connecting to certain webservers from certain countries could bring
trouble to the probe hosts.

Of course policies are not set in stone. However, nobody has come up
with a better policy proposal.

Note that Atlas does support DNS over TLS.

Re: [atlas] status-check redirects to itself?

[Philip Homburg]

On 2020/04/29 19:58 , Stephane Bortzmeyer wrote:
> Since today, status-check seems to redirect to itself?
> 
> % curl -v 
> https://atlas.ripe.net/api/v2/measurements/2060427/status-check\?permitted_total_alerts=3

> < Location: 
> /api/v2/measurements/2060427/status-check/?permitted_total_alerts=3

Note the backward slash and the forward slash.

The redirect is to a slightly different URL.

Re: [atlas] Setting the DNS AD bit

[Philip Homburg]

On 2020/04/15 8:58 , Stephane Bortzmeyer wrote:
> It seems there is currently no way to set the AD bit in DNS queries?
> (Through the API, we can only control RD, CD and DO bits.)

Hi Stephane,

That sounds like a useful addition. I'll put it on the list of things to
implement.

Philip

Re: [atlas] Probe 2177 disconnected since Tuesday

[Philip Homburg]

On 2020/03/20 11:38 , Werner Wiethege wrote:
> I didn't see any ARP requests. So I order a new probe?

Yes, that's best.

Philip

Re: [atlas] Probe 2177 disconnected since Tuesday

[Philip Homburg]

On 2020/03/20 11:14 , Werner Wiethege wrote:
> No change. The orange light flickers when the switch handles
> broadcasts.

The orange light is just ethernet traffic.

If the probe never transmits anything then the probe is probably broken.

Because the probe has static address configuration, it will typically
send an ARP for the configured router address.

Philip

Re: [atlas] Software probe using port 8080

[Philip Homburg]

On 2020/03/13 13:48 , Paul Eagles wrote:
> Is there any way to change the ports that the software probes use?  Port
> 8080 is used by the probe which conflicts with another piece of software
> I wanted to use.

Unfortunately, that is not possible at the moment. I'll look into making
this more flexible.

Philip

Re: [atlas] Software probes offline since updating to 5010-3

[Philip Homburg]

On 2020/03/11 13:54 , Paul Eagles wrote:
> About an hour ago I updated my 4 software probes (1000136, 1000137,
> 1000149 & 1000177) to 5010-3 from 5010-1 and they’re all still showing
> as offline.  For each probe I went through and updated the keys within a
> minute or so of the installation finishing so I don’t think that’s the
> problem.
> 
> Am I missing something or just being impatient?

Hi Paul,

After some investigation we found that the new keys do not fully
propagate until just after midnight Amsterdam time.

We found what is causing this and we will deploy a fix soon.

Sorry for the inconvenience,

Philip

[atlas] New on RIPE Labs: What’s the Deal with IPv6 Link-Local Addresses?

[Philip Homburg]

Dear colleagues,

Stephen Strowes, Emile Aben and I published a new article on RIPE Labs
on how link-local addresses in IPv6, and specifically the '%eth0'-part
of link-local addresses, can have an impact on RIPE Atlas measurements.

https://labs.ripe.net/Members/philip_homburg/whats-the-deal-with-ipv6-link-local-addresses

We think this will be an interesting read for network operators and
others who are using RIPE Atlas measurement data or are interested in IPv6.

Kind regards,

Philip Homburg
RIPE NCC

[atlas] Broken traffic graphs in latest firmware (5010)

[Philip Homburg]

Hi,

A change in the RIPE Atlas probe code to make reporting traffic
statistics optional in software probes (and disabled by default) has the
unfortunate side-effect that reporting of traffic statistics is now
disabled on Version 3 and Version 4 hardware probes, as well as all
anchors. This change was rolled out in firmware version 5010.

Unfortunately, this effect was not spotted during testing. We will roll
out a new firmware release that corrects this issue in coming weeks.

Philip

Re: [atlas] RIPE Atlas Software Probes

[Philip Homburg]

On 2020/03/04 11:14 , Enrico Ardizzoni wrote:
> This is the "prerm" script ( a little bit overkill :) )

I pushed v5010.3 which fixes the prerm script. Unfortuantely, during an
upgrade it runs the old script, so upgrading to 5010-3 will still remove
the keys.

Philip

Re: [atlas] RIPE Atlas Software Probes

[Philip Homburg]

On 2020/03/04 11:14 , Enrico Ardizzoni wrote:
> This is the "prerm" script ( a little bit overkill :) )

I'm not an expert in debian packaging. Does the prerm get executed
during an upgrade?

What I remember is that upgrading the debian package works, but maybe I
remember wrong.

Philip

Re: [atlas] check dns64/nat64 behavior ?

[Philip Homburg]

On 2020/02/21 14:50 , Thomas Schäfer wrote:
> Am 21.02.20 um 14:38 schrieb Thomas Schäfer:
>> Hi,
>>
>> is there a simple way to check the reachability of IPv6 targets by
>> probes behind NAT64?
> 
> IPv4 targets of course.

On NAT64, an IPv4 address is accessed over IPv6. You do need to disable
the check if the domainname exists.

Re: [atlas] RIPE Atlas Software Probes

[Philip Homburg]

On 2020/02/16 16:23 , Jacques Lavignotte wrote:
> *Connection & Traffic* is firmly showing *No data available for Probe #
> 1000165 for this time period* since start.
> 
> Do I have something to do more ?

Hi,

There are two issues:

The first is that by default, the software probe does not report traffic
statistics. This is done to protect people who install a software probe
on a router. The readme in the software probe repo describes how to
enable reporting of traffic statistics
(https://github.com/RIPE-NCC/ripe-atlas-software-probe/blob/master/README.rst)

The second reason is that our processing of traffic statistics needs to
be improved, to avoid reporting statistics on down-stream interfaces of
a router. We made some changes to make this happen but not all code is
there yet.

Philip

Re: [atlas] RIPE Atlas Software Probes

[Philip Homburg]

On 2020/02/14 10:50 , Jaap Akkerhuis wrote:
> When the software probe projects started I offered to make a FreeBSD
> port version. The developers said that that was a good idea and
> they would come back to me.

A FreeBSD port would be nice, but may be a significant amount of work.

Philip

Re: [atlas] RIPE Atlas Software Probes

[Philip Homburg]

On 2020/02/12 11:11 , Borja Marcos wrote:
> I am wondering, is the RIPE Atlas probe software fully Linux centric or can 
> it run
> on other modern *IX derviatives?

The sort answer is yes, it is fully Linux centric.

The longer answer is that the code consists of two parts. One part is a
collection of shell scripts that manages the probe and the second is C
code that performs the actual measurements.

The shell scripts are probably easy to move to a different platform as
long that platform has a bourne shell.

The tricky part is porting the measurement code. For example, on Linux
is relatively straightforward to implement a tcptraceroute. On BSD
systems that requires a lot more effort.

Note that the measurement code was added to busybox (for historical
reasons). Busybox is no longer needed, so that code needs to be removed
at some point.

Philip

Re: [atlas] RIPE Atlas Software Probes

[Philip Homburg]

On 2020/02/12 12:42 , Philip Homburg wrote:
> On 2020/02/12 12:03 , Paul Eagles wrote:
> 
>> fatal: reference is not a tree: 0615d3803b7b68cf85f959957b433f1c9fe2f3e6
>> Unable to checkout '0615d3803b7b68cf85f959957b433f1c9fe2f3e6' in submodule 
>> path 'probe-busybox'
>> [pauleagles@dns2 ~]$
> 
> Weird. It seems that the two repos are out of sync. I'll take a look.

Thanks for reporting the problem. The software probe repo is updated and
I set the v5010 tag to the new commit.

Philip

Re: [atlas] RIPE Atlas Software Probes

[Philip Homburg]

On 2020/02/12 12:03 , Paul Eagles wrote:

> fatal: reference is not a tree: 0615d3803b7b68cf85f959957b433f1c9fe2f3e6
> Unable to checkout '0615d3803b7b68cf85f959957b433f1c9fe2f3e6' in submodule 
> path 'probe-busybox'
> [pauleagles@dns2 ~]$

Weird. It seems that the two repos are out of sync. I'll take a look.

Philip

[atlas] RIPE Atlas Software Probes

[Philip Homburg]

Dear colleagues,

We are glad to announce that, after several months of development and
testing, we are now accepting applications for RIPE Atlas software
probes. With several different installation options to choose from,
software probes provide future hosts a whole new way to get involved
with RIPE Atlas.

For more details, see our new article on RIPE Labs:

https://labs.ripe.net/Members/alun_davies/ripe-atlas-software-probes

Kind regards,
Philip

Re: [atlas] DNS issue with Atlas or with the ISP?

[Philip Homburg]

On 2019/09/09 9:39 , Stephane Bortzmeyer wrote:
> So, people have tested, and confirm that 1) this ISP advertises
> link-local address via RA 2) it works with all DNS clients, but the
> RIPE Atlas probes. Any idea?

I just tried a bare link local address with dig on FreeBSD and it
failed. For some reason my attempts on a few linux VMs failed for
completely different reasons.

The problem is that for a link local address you need a scope. The Atlas
measurement code doesn't know how to add one and the addresses in
/etc/resolv.conf don't have one.

I sort of knew that this was going to be a problem, but link local for
the resolvers seems to be quite rare.

I'll make a note to see if I can make it work somehow.

Philip

Re: [atlas] DNS RTT over TCP: twice as long than UDP?

[Philip Homburg]

On 2019/07/05 14:33 , Giovane Moura wrote:
> https://atlas.ripe.net/results/maps/root-server-performance/
> 
> By definition,  however, RTT is "the length of time it takes for a
> signal to be sent plus the length of time it takes for an
> acknowledgement of that signal to be received."[2]. So if DNS TCP
> measurements starts measuring from the SYN packet, that', in fact, be
> two RTTs. Is this the case with Atlas?

The use of RTT on that page is wrong. The DNS measurement reports
response time.

Philip

Re: [atlas] Probe off-line for 6 days due to bad firmware signature

[Philip Homburg]

On 2019/06/24 17:54 , Novak Jirka wrote:
> My probe is connected to my network infrastracture again, but not
> connected to RIPE ATLAS infrastracture
> 
> Could you help me now?

Your probe is now fully connected.

Philip

Re: [atlas] Probe off-line for 6 days due to bad firmware signature

[Philip Homburg]

On 2019/06/19 18:38 , Novak Jirka wrote:
> my probe 22424 is off-line due to bad firmware signature error.
> I try to boot it without USB stick many times, then i change the USB
> stick (replace the old one with new one), but it doesnt solve my problem.
> Could you help me?

Hi,

A bad firmware signature error can happen if the USB stick is broken.
But if it still happens after the USB stick has been replaced then it is
usually a network problem.

The probe seems disconnected that the moment. If you connect it again, I
can see if I can find out what is going on.

Philip

Re: [atlas] Feature Request to consider: DNS response IP header

[Philip Homburg]

On 2019/05/30 15:22 , Ponikierski, Grzegorz wrote:
> Can be useful to estimate proximity between probe and DNS and to detect
> nasty middle boxes. To limit overhead on Atlas side, IP header can be
> provided as raw data to decode on end-user side.

As far as I know there is no option provided by the Linux kernel to
receive the original IP header with recvmsg.

Fields that are returned by recvmsg, such as TTL, can be added without
to much trouble. I'll make a note to add that that at some point.

Philip

Re: [atlas] All our atlas probes have reverted to DHCP

[Philip Homburg]

On 2019/04/17 8:52 , Sebastian Wiesinger wrote:
> all of our four atlas probes have reverted from their fixed network
> configuration to DHCP which is a shame because there is no DHCP at
> their present locations. Also this not the first time it happend. Can
> someone explain why this happens and how to prevent this? This wastes
> a lot of time and effort every time it happens.

Hi Sebastian,

I have to admit I have no idea what could be going on with your probes.
With static IPv4 configuration, there is a bit of complexity in how the
probes switches back up DHCP if the static config doesn't work. But your
probes have IPv6 as well.

For v3 probes, there is always the possibility that the filesystem on
the USB stick become corrupt, and then the static config gets lost as
well. But 3 of your probes are V1/V2.

Maybe you can let me know when you plug them in a network that does have
DHCP? There might be something in the probes' logs that gets reported
when they connect.

Philip

[atlas] Measurement result issues for some RIPE Atlas anchors

[Philip Homburg]

In April, we started investigating a strange traceroute result reported
by one of the RIPE Atlas anchors. The conclusion of our investigation is
that the probe upgrade process for the v3 anchor firmware has been
leaving old measurement daemons running as well as starting new ones.

This means that a gradually increasing number of measurements from the
affected anchors have been performed multiple times more or less
simultaneously. Because those processes were writing to the same file,
this resulted in a lot of garbled results. Purely by chance, the weird
result that triggered the investigation had the first part of one result
and the last part of another result in a way that generated
syntactically correct JSON.

The net result is that for the last few months, measurements done by
anchors may have resulted in multiple results at roughly the same time
or no results because the results were rejected as invalid. For those
who wish to check whether results for specific measurements were
affected, the following links will direct you to lists of probe IDs for
the affected anchors:

https://atlas.ripe.net/api/v2/probes/?is_anchor=true=system-virtual=id
(VM anchors)

https://atlas.ripe.net/api/v2/probes/?is_anchor=true=system-v3-pc-engines=id
(v3 hardware anchors)

We are working to resolve the issue and will keep you updated as and
when we have more to report. We apologise for any inconvenience this
might cause.

Philip

Re: [atlas] DNS-over-TLS and DNS-over-HTTPS measurement

[Philip Homburg]

On 2019/04/08 17:04 , Petr Špaček wrote:
> Anyway, are there plans for supporting DNS-over-HTTPS?

Hi Petr,

A couple of years ago we created a policy regarding HTTP measurements on
RIPE Atlas. The concern was that probe hosts located in certain
countries could get into trouble should their probes try to reach
certain HTTP targets. So it was decided at the time that since these
measurements do not add much to the goal of RIPE Atlas, which is to
measure the Internet as a network and not the higher level protocols
that run on top of it, we restricted HTTP measurements such that they
are only able to target RIPE Atlas anchors.

Obviously there are benefits in measuring DNS-over-HTTPS. However, the
risk for probe hosts in certain countries remains the same. For this
reason, although we would be open to the creation of a new policy should
there be sufficient interest from the community, there are no plans to
support DNS-over-HTTPS until such a policy is in place.

Philip

Re: [atlas] Probe not coming online anymore after firmware upgrade

[Philip Homburg]

On 2019/03/21 16:41 , Annika Wickert wrote:
> I bought a new USB Stick for this probe as the old one was tagged as
> readonly etc..
> https://atlas.ripe.net/probes/23913/
>
> But for some reason it is not coming online at all .
>
> Any idea what could be the issue? I also see no network traffic at all
> originating from the probe.

Hi Annika,

I looked at the settings for your probe and noticed that two DNS
resolvers were configured with IPv6 addresses.

The probe was able to connect to our reg. servers and download firmware.
So I suspected that forcing the probe to use those DNS resolvers could
be the cause of the problem.

Shortly after disabling that configuration the probe managed to connect.
Unfortunately, for reasons unknown to me, it disconnected 12 minutes later.

Philip

[atlas] Issue with RIPE Atlas HTTP measurements

[Philip Homburg]

We recently discovered an issue with RIPE Atlas HTTP measurements. In
trying to fix an issue that caused the HTTP body size to be reported
incorrectly, we inadvertently introduced a bug that prevents the
measurement from working under certain conditions. This bug affects
firmware release 4940 and the upcoming releases 4950 and 4960.

The combination that fails is a recurring measurement after the first
run with an HTTP server that returns the body as chunks. This means that
one-off measurements are fine. Unfortunately, HTTP measurements that are
part of anchoring measurements are affected.

We will work to fix this issue in the first firmware release after 4960,
which can be expected in the next couple of months. We apologise for the
ongoing issue and will keep you informed as we work to resolve it.

Philip

Re: [atlas] New on RIPE Labs: RIPE Atlas Probes: Delays in Distribution

[Philip Homburg]

On 2019/02/08 13:45 , Micha Bailey wrote:
> Interesting. This seems like a much higher power device. What are the
> actual power requirements? So far my probe has been powered from the USB
> port on the router, and I’m concerned that this may make the
> installation of future probes more difficult.

The power requirements are quite reasonable when used as an Atlas probe.
There should be no problem powering a v4 probe from a USB port.

Philip

[atlas] Bug in DNS client subnet measurement

[Philip Homburg]

Hi,

We got a bug report that using the DNS client subnet option in a
measurement results in a form error. It turns out that probe firmware
version 4940 will send a malformed DNS query. Code cleanup related to
other changes fixes this issue in the upcoming firmware. We expect to
release the new firmware in about one month from now.

Philip

Re: [atlas] Probe v1 disconnecting every 2 hours

[Philip Homburg]

On 2019/01/30 19:49 , Cornelius Keck wrote:
> I'm wondering, in case the probe continues to have issues, is there a
> way to get a replacement?

Yes, you can always ask for a replacement. We are still quite a bit
behind in shipping probes because the transition from v3 to v4 probes
was quite a bit of work.

> Or, come to think of it, replace the probe
> with some code on smallish hardware, like a Raspberry Pi?

We are looking into this. There is no timeline, or even a commitment to
provide this option. But we are actively preparing for a pilot.

Philip

Re: [atlas] EDNS Client Subnet

[Philip Homburg]

On 2019/01/28 14:33 , Rami Al-Dalky wrote:
> When I tried to create a DNS measurement, I found that the only way to
> send DNS query with option is to set default_client_subnet to True.
> However, by setting this option, a DNS query will be sent with 0.0.0.0/0
>  as client subnet. 
> 
> Is there a reason why ECS is implemented that way? If it for privacy
> issue, the RFC recommends to sent the client IP with /24 prefix for IPv4
> and /56 for IPv6 to preserve the privacy.

Let me point out that we chose 0.0.0.0/0 to avoid all privacy issues.
The recommendation just reduces privacy issues.

At the same time, it was not clear to us what additional benefit it
would bring to RIPE Atlas measurements to include longer prefixes. In
particular, we assumed that the main purpose of this option would be to
measure interference by firewalls or other middle boxes.

Philip

Re: [atlas] Probe firmware source code

[Philip Homburg]

On 2019/01/13 16:41 , Evgeniy S. wrote:
> thank you for helping I overlooked that information on website and
> Github repository. 
> 
> Btw, I see some discrepancy there (which can be improved):
> 1) commit to Gihub repository - last was Jun 14, 2018;
> 2) listed firmware (last 4790 from Jun 27 2017)
> on https://atlas.ripe.net/resources/source-code/ page;
> 3) actual firmware version we have installed on probes 4940 (I found no
> way to fund when it was released and when probe was updated);

Thank you for your interest in the measurement source code. Based on
community feedback we switched from releasing tar-balls to publishing
our git repository on github.

However, this transition requires a bit more work:
1) The structure of the git repository needs to be improved. At the
moment there is one branch per release. That should be switched to a
release branch with tags.
2) It is not clear what should be done with the old tar files.

Note that currently we publish only the measurement source code. We are
working toward releasing the remaining code (a collection of shell
scripts) as well to allow people to create software probes. The complete
project for making software probes possible is more involved, it
requires backend and procedural changes as well.

Philip

Re: [atlas] Long response times using one-off measurements with old probes

[Philip Homburg]

On 2018/12/14 15:03 , Moritz Muller wrote:
> Some small additional delays have been documented before on the RIPE Atlas 
> website and in research papers [1, 2], but the big delay with one off 
> measurements was new to me.
> 
> Also to others? Is our assumption correct that the scheduler is the culprit?

So far my investigation shows that the measurement actually takes that
long.

Why is not clear at the moment, but it doesn't seem to be anything
external to the probe. It also doesn't seem to be interference from
other measurements.

Philip



signature.asc
Description: OpenPGP digital signature

Re: [atlas] shared fate for my two probes

[Philip Homburg]

On 2018/11/19 17:53 , Jay Borkenhagen wrote:
> Philip Homburg writes:
>  > On 2018/11/16 22:16 , Jay Borkenhagen wrote:
>  > 
>  > > probe 11171  2018-11-16 14:42:28 UTC.
>  > > probe 11203  2018-11-16 14:42:33 UTC.
>  > 
>  > Both probes lost all internet connectivity for a couple of hours. I have
>  > no idea what happened but Atlas traceroutes stop at hop 1 (and don't
>  > even reach the local router).
>  > 
> 
> Yes, but the problem was not with the networking infrastructure: 
For me the most obvious approach is to put the probes behind a switch
that supports port mirroring and look at the actual traffic during this
period.

The fact that it happens to both probes at the same time makes it very
unlikely that it is a probe specific hardware problem.

I'm not aware of any bugs in the Linux kernel that would affect IPv4 and
IPv6 at the same time.

So the obvious next step is to check what actually goes over the wire.

Re: [atlas] shared fate for my two probes

[Philip Homburg]

On 2018/11/16 22:16 , Jay Borkenhagen wrote:

> probe 11171  2018-11-16 14:42:28 UTC.
> probe 11203  2018-11-16 14:42:33 UTC.

Both probes lost all internet connectivity for a couple of hours. I have
no idea what happened but Atlas traceroutes stop at hop 1 (and don't
even reach the local router).

Philip

Re: [atlas] DNS measurement using the Probe's resolvers

[Philip Homburg]

On 2018/07/06 22:30 , Rami Al-Dalky wrote:
> I have a question about using the probe's list of resolvers when
> creating a DNS measurement. It is unclear to me whether the probe would
> send the query to all resolvers at the same time or not! Also, does the
> probe send the query to all resolvers in its list or to a subset of
> those resolvers? Is DNS eyeball implemented in the probes?

Hi,

The Atlas DNS measurement code sends one request at a time, in the order
in which the DNS resolvers are listed in /etc/resolv.conf on the probe.

In addition to measuring DNS, probe can also resolve a DNS name for the
target of a measurement. In that case, the stub resolver that is built
into libevent is used. That stub resolver does some clever and funky
stuff. I don't know the details of how DNS queries are scheduled.

Philip

Re: [atlas] Running a probe behind a NAT66

[Philip Homburg]

On 2018/06/11 20:42 , Roman Mamedov wrote:
> I fully expect that you didn't account for such a setup in the controller
> infrastructure or possibly various "local IP is valid" checks and whatnot, but
> why this used to work fine before? Has there been any change on your side in
> April that would break this kind of setup?
> 

I started a TCP traceroute on your probe toward it's controller, but it
consistently fails after hop 2. I have no idea if that is related to
your NAT box.

https://atlas.ripe.net/measurements/14364300/

In general it is worth noting that Atlas has support for IPv4 NAT, such
as keeping track of the public address of a probe. This support in not
implemented for IPv6. So if you put a probe behind some kind of IPv6
NAT, the results may be confusing to other Atlas users.

Philip

Re: [atlas] TLS error when the certificate is expired?

[Philip Homburg]

On 2018/04/15 15:33 , Stephane Bortzmeyer wrote:
> I'm reasonably certain that it has been possible to use 'sslcert'
> measurements even when the certificate is expired.
> 
> Today, I try to use 'sslcert' with trigger-happy.eu and it fails:
> 
> "alert": {
>   "description": 40,
>   "level": 2
> },
> 
> And no certificate in the JSON output (this is measurement #12166428)
> 
> 40 is the very general "handshake failure" of
> TLS. 
> 
> 
> Was there a change in Atlas recently? 

Hi Stephane,

What is typically the case is that the server side has all kinds of
restrictions on the ciphers that it is willing to accept.

I recently made some changes to support a server where the sslcert
measurement would also fail. With these changes 'trigger-happy.eu' is
happy as well.

At moment I have no clear idea when the next firmware will be released.
It may take a few months.

Philip

Re: [atlas] False positives on SSL check?

[Philip Homburg]

Hi Ruben,

On 2018/01/31 13:46 , Ruben Herold wrote:
> I tried to monitor on of our services running on https://login.afterbuy.de. 
> I configured an SSL check like (Meassure ID #11090260). All probes response 
> with:
> 
>  Error: timeout reading hello
> 
> But the service is online and reachable. SSLabs SSL check confirmed this.
> So can someone explain what the error message will tell me?

The SSL check is written from scratch and does not contain all of the
extensions and features you find in common SSL libraries. We do update
the code as we find issues.

In this case, login.afterbuy.de just terminates the connection without
sending any kind of error message.

After looking at the difference between what the atlas measurement code
sends and what wget and curl do, my guess is that it is the lack of the
'signature_algorithms' extension that causes the server to drop the
connection.

I will investigate if that is indeed the case. If so, I'll add the
necessary code. Though it may take a couple of months before that will
be available on the probes.

Philip

Re: [atlas] Probe disconnects every 2 hours

[Philip Homburg]

On 2017/12/20 20:41 , Daniel AJ Sokolov wrote:
> It is 1118.

Your probe seems to reboot every two hours. V1 probes suffer from memory
fragmentation, so when the ssh connection breaks, they often reboot.
However I cannot find anything related to that in the logs.

A reboot can be caused by a power failure. Do you have anything that
might be causing that?

Alternatively, there may be a software bug that triggers the reboot.

Other V1 probes that connect to the same controller seem fine. So either
you probe gets a specific measurement that triggers a bug, or there is a
power issue.

How is the probe powered?

Philip

Re: [atlas] Probe disconnects every 2 hours

[Philip Homburg]

Hi,

Maybe you can mention your probe ID. That makes it easier to figure out
what is going on.

Philip

Re: [atlas] Effect of "Resolve on Probe"-option

[Philip Homburg]

On 2017/12/03 17:36 , Stephane Bortzmeyer wrote:
> Atlas rejects 'use_probe_resolver': False if you did not specify a
> target:
> 
> Status 400, reason 
> "{"error":{"status":400,"errors":[{"source":{"pointer":"/definitions/0"},"detail":"`target`
>  cannot be null if `use_probe_resolver` is not 
> specified"}],"code":102,"detail":"There was a problem with your 
> request","title":"Bad Request"}}"
> 
> 'resolve_on_probe' seems ignored.
> 

> 2) Use an external server (here, Quad9, option -e):
> 
> % atlas-resolve -r 5 -v -e 9.9.9.9 thepiratebay.org 
> {'definitions': [{'protocol': 'UDP', 'description': 'DNS resolution of 
> thepiratebay.org via nameserver 9.9.9.9', 'af': 4, 'query_argument': 
> 'thepiratebay.org', 'query_type': 'A', 'query_class': 'IN', 'target': 
> '9.9.9.9', 'set_rd_bit': True, 'type': 'dns', 'use_probe_resolver': False}], 
> 'is_oneoff': True, 'probes': [{'requested': 5, 'type': 'area', 'value': 'WW', 
> 'tags': {'include': ['system-ipv4-works']}}]}
> Measurement #10397471 for thepiratebay.org/A uses 5 probes
> Nameserver 9.9.9.9
> [104.27.216.28 104.27.217.28] : 5 occurrences 
> Test #10397471 done at 2017-12-03T16:19:41Z

It may be worth pointing out that resolve_on_probe has effect if two
conditions are met:
1) use_probe_resolver is false
2) the measurement target is a dns name (as opposed to an IP address
literal)

In your quad-9 example, the target is a literal, so resolve_on_probe has
no effect.

The following DNS measurement uses resolve_on_probe:
https://atlas.ripe.net/measurements/10404214/

Here is an example measurement result:
{"af":4,"dst_addr":"193.0.9.7","from":"73.34.225.118","fw":4780,"group_id":10404214,"lts":38,"msm_id":10404214,"msm_name":"Tdig","name":"manus.authdns.ripe.net","prb_id":21675,"proto":"UDP","result":{"ANCOUNT":1,"ARCOUNT":0,"ID":55944,"NSCOUNT":0,"QDCOUNT":1,"abuf":"2oiEAAABAAEAA3d3dwRyaXBlA25ldQABwAwAAQABAABUYAAEwQAGiw==","rt":131.954,"size":46},"src_addr":"192.168.29.154","stored_timestamp":1512385062,"timestamp":1512385044,"type":"dns"}

The probe reports that it resolved 'name' manus.authdns.ripe.net to
'dst_addr' 193.0.9.7

Philip

Re: [atlas] Recent TLS supported by probes?

[Philip Homburg]

Hi Stephane,

On 2017/11/13 10:10 , Stephane Bortzmeyer wrote:
> I cannot get the certificate for 
> (measurement #10174881): "alert" is "{u'description': 40, u'level':
> 2}".
> 
> It works with other, more "mainstream", HTTPS sites (see #10174883). I
> suspect this is because the probes don't handle the recent TLS
> stuff. Can anyone confirm or infirm?

The only issue I'm aware of is SNI. We added that in 4780.

SNI or no SNI doesn't seem to make a difference to dns-resolver.yeti.eu.org

Hmm, looking at the list of ciphers that the measurement code sends, I
can see how it can fail if somebody applies a rather script security policy.

I'll create a ticket to add improved ciphers.

Philip

Re: [atlas] IPv4 leading zeroes and weird interface behaviour

[Philip Homburg]

On 2017/10/25 23:16 , Peter J. Holzer wrote:
> Possibly gethostbyname in the libc, but I haven't tested that. At least
> interpreting numbers with leading zeroes as octal is traditional on
> unix-like OSs (I probably first noticed that on Ultrix around 1990). I
> don't know why anyone thought this was a good idea. I wouldn't expect
> anyone to rely on it, but apparently nobody dares to get rid of that
> "feature".

Technically it is C feature. If you call, for example, strtol with the
third argument (base) equal to 0, then it will parse numbers just like a
C compiler.

In many contexts is is convenient if you can just type a hex number by
prefixing the number with 0x.

Unfortunately, this use of strtol is so common, that it gets used even
in a context where it doesn't make sense, such as parsing an IPv4 address.

However, the way IPv4 addresses are parsed is completely weird. This is
from the inet_aton manual page:

 Values specified using the `.' notation take one of the following
forms:

   a.b.c.d
   a.b.c
   a.b
   a

 When four parts are specified, each is interpreted as a byte of
data and
 assigned, from left to right, to the four bytes of an Internet address.
 Note that when an Internet address is viewed as a 32-bit integer
quantity
 on the VAX the bytes referred to above appear as ``d.c.b.a''.  That is,
 VAX bytes are ordered from right to left.

 When a three part address is specified, the last part is
interpreted as a
 16-bit quantity and placed in the right-most two bytes of the network
 address.  This makes the three part address format convenient for
speci-
 fying Class B network addresses as ``128.net.host''.

 When a two part address is supplied, the last part is interpreted as a
 24-bit quantity and placed in the right most three bytes of the network
 address.  This makes the two part address format convenient for
specify-
 ing Class A network addresses as ``net.host''.

 When only one part is given, the value is stored directly in the
network
 address without any byte rearrangement.

 All numbers supplied as ``parts'' in a `.' notation may be decimal,
 octal, or hexadecimal, as specified in the C language (i.e., a
leading 0x
 or 0X implies hexadecimal; otherwise, a leading 0 implies octal; other-
 wise, the number is interpreted as decimal).



signature.asc
Description: OpenPGP digital signature

Re: [atlas] Can Atlas probe handle Truncated response ?

[Philip Homburg]

On 2017/09/12 17:19 , Stephane Bortzmeyer wrote:
> I don't think tou can ask for several tests. (Source:
> )

There is a retry option hidden somewhere in the docs.

"[dns] retry (integer): Number of times to retry,"

Philip

Re: [atlas] Link aggregation for anchors?

[Philip Homburg]

On 2017/07/20 7:44 , Tore Anderson wrote:
> I'd rather you spent that time implementing LLDP support, come to think
> of it. (That would be useful on the non-Anchor probes as well.)

I have a ticket open for LLDP on regular probes. Though no time frame is
assigned when it should be done. I assume the same code would work for
anchors, but I haven't looked into that.

Re: [atlas] Probe does not request IPv4 address via DHCP

[Philip Homburg]

On 2017/06/23 16:34 , Stanish Stanishev wrote:
> Thank you for the details.
> Do I need to assist you somehow for finding the reason, why the automatic way 
> for providing the probe with literal did not happen ?

Thanks for the offer. We already found the reason and this situation
should now be covered.

Philip

Re: [atlas] Probe does not request IPv4 address via DHCP

[Philip Homburg]

On 2017/06/13 14:07 , Stanish Stanishev wrote:
> Ah, Thank you so much for the support.
> May I ask you to share what did you do in order to make the probe connect 
> again to the Atlas ?

Normally the probe receives the name of a controller to connect to. If
the probe doesn't have access to DNS resolvers then it cannot connect.

We have a trick to provide the probe with either an IPv4 or an IPv6
literal. In theory the system will detect that the probe has troubles
resolving DNS and will switch to that automatically.

We have to figure out why that didn't happen for your probe. But it can
also be activated manually.

Philip

Re: [atlas] TLS tests with login.live.com

[Philip Homburg]

On 2017/05/29 14:02 , Stephane Bortzmeyer wrote:
>> My guess is that the measurement code sends something the other side
>> doesn't like and then the server responds with something the Atlas
>> code doesn't understand.
> 
> Any way to debug it in more detail?

I guess the first step would be look at the interaction in wireshark to
see what the server is sending back.

Note that at the moment, the probes do not include any SNI information.
But a quick test suggests that that is not the cause of the problem.

Philip

Re: [atlas] TLS tests with login.live.com

[Philip Homburg]

On 2017/05/29 13:53 , Stephane Bortzmeyer wrote:
> On Mon, May 29, 2017 at 01:41:45PM +0200,
>  Stephane Bortzmeyer  wrote 
>  a message of 5 lines which said:
> 
>> TLS tests from the Atlas probes work for me, for every server... *but*
>> login.live.com, for which I always get timeouts (it works with other
>> TLS clients).
> 
> Could it be related to this error
>  which is
> reported by many people on the socnets today?
> 
> The Atlas probe checks the OCSP answer?

Hi Stephane,

No, the Atlas code is nowhere near advanced enough to do that.

My guess is that the measurement code sends something the other side
doesn't like and then the server responds with something the Atlas code
doesn't understand.

Philip

Re: [atlas] Many timeout-failures for DNS over TCP

[Philip Homburg]

On 2017/05/09 10:03 , Davey Song(宋林健) wrote:
> I setup measurements for DNS queries over both TCP and UDP. I found the
> timeout and failures of TCP is far more than UDP.  

> DNS over TCP: https://atlas.ripe.net/measurements/8552847/#!probes

Hi,

Unfortunately, due to a bug, DNS over TCP measurements do not work in
firmware 4760. I went over the results of this measurement. And the in
most cases the probe was indeed still running 4760. Over time, probes
upgrade 4770 and this issue will be fixed.

There are a few probes that have a timeout but are running 4770. I
created a TCP traceroute measurement using the same probes of
measurement 8552847. https://atlas.ripe.net/measurements/8552847/#!probes

In those remaining cases, the traceroute fails to complete. So for those
probes there a network related issue.

Philip

Re: [atlas] DNS over TCP problems?

[Philip Homburg]

On 2017/03/21 19:10 , Stephane Bortzmeyer wrote:
> On Tue, Mar 21, 2017 at 05:57:15PM +0100,
>  Philip Homburg <philip.homb...@ripe.net> wrote 
>  a message of 16 lines which said:
> 
>> The patch is simply to delete the offending line and, if all goes well,
>> will be rolled out on the anchors in the next few days.
> 
> What about the other probes?

The other probes as soon as possible. But releasing probe firmware is a
long process.

So both are in progress at the moment. But it is easier to rush a
release on anchors than on probes.

Re: [atlas] DNS over TCP problems?

[Philip Homburg]

Hi Stephane,

On 2017/03/21 17:25 , Stephane Bortzmeyer wrote:
> According to DNSmon, there is indeed a problem since 14 March:
> 
> https://atlas.ripe.net/dnsmon/group/fr.?dnsmon.session.color_range_pls=0-10-10-50-100=true=zone-servers=fr.=undefined=1488974400=1490112000=false=both=true

Unfortunately, a debug statement was left behind in the DNS over TCP
code path. This was not noticed until that code was deployed on the
anchors and started affecting DNSMON.

The patch is simply to delete the offending line and, if all goes well,
will be rolled out on the anchors in the next few days.

Philip

Re: [atlas] Failing probe, all lights blink

[Philip Homburg]

Hi Phillip,

On 2017/03/07 0:41 , Phillip Remaker wrote:
> A V3 probe, when powered on, lights the power light, and then blinks all
> other lights for about a second every few seconds. Forever.

If it does this without the USB stick then it is very likely that the
probe is bricked. That can happen if the probe loses power when it is
trying upgrade it's built-in firmware.

Though we haven't release a new firmware for the built-in flash for
quite a while now.

If this is about probe 16110 then the logs are consistent with the probe
trying to upgrade it's built-in flash.

Philip

Re: [atlas] traceroute measurement with abnormally small RTT

[Philip Homburg]

Hi,

On 2017/01/18 18:01 , Wenqin SHAO wrote:
> It’s a IPv4 traceroute measurement toward DNS b root, 192.228.79.201.
> What you’ll see is:
> starting from hop 14, the 3 measurements for the same hop come from
> different IP address:
> 
> hop 14: 184.105.80.202, 72.52.92.122
> hop 15: 72.52.92.122, 216.218.223.28
> hop 16: 216.218.223.28, 130.152.181.189
> hop 17: 130.152.181.189, 206.117.5.22
> hop 18: 206.117.5.22, 192.228.79.201
> hop 19: 192.228.79.201
> 
> 
> My first doubt is with Paris traceroute is it still possible that
> response IP can change due to LB mechanism on the returning path?

Obviously, middle boxes can do anything including changing IP addresses
in packets. However, normally you see one the addresses of the router
that generated the error ICMP there and the actual return path only
affects rtt and ttl.

> At hop 18, the traceroute actually already reached the destination, but
> it continued to perform the hop 19, why?

Traceroute continues if a hop generates a TTL exceeded. For a UDP
traceroute the expected value is a UDP port unreachable.

Sometimes a middle box in front of the destination sends a TTL exceed
when you expect the target to handle the packet.

> At hop 19, there is one measurement from 192.228.79.201 reporting
> abnormally small RTT of 0.449ms. How come? Can that happen to other
> measurements as well, under what circumstances?
> 
> The above listed neighbouring hops always share one IP address, which
> makes it seem like a data formatting issue. Is there a known bug?

There is something really weird going on here. I have no idea what is
causing this.

Philip

Re: [atlas] tp-link probev3 tl-mr3020

[Philip Homburg]

On 2016/11/25 13:27 , folkert wrote:
> I have one of those tp-link probes, a v3 it looks like.
> Now when I got it, I was warned to put it on a 1A powersource. Now that
> is a bit of a problem so I started to measure its power usage to verify
> if I could maybe run it on a smaller power supply.
> What I measure is 0.14A upto 0.23A - that should easily be powered by a
> 0.5A power supply I think.
> As I'm not an electronics expert I'm now writing this message for
> confirmation: am I right about the power usage? Or ar there situations
> where it would be using a lot more power?

The actual TP-Link is not very sensitive to voltage changes. You
can run it close to 3 volts and it will still work.

On the other hand, the first USB sticks that we used (from SanDisk) are
very sensitive to low voltage, and will often not show up if there is
too much voltage drop somewhere (either in the power supply or in the
cable).

However, the main problem at the moment is filesystem corruption and USB
sticks dying. There is no hard evidence as to what causes this. So to be
on the safe side, it is best to use a good power supply.

Typically, you can't tell from the outside whether something is well
designed or not. And most people use unlabeled cheap versions, both for
power supplies and cables. So it seems best to err on the side of
caution and recommend slightly higher specs.

In the end, if it works, fine. If you find USB sticks dying, then it
worth looking if there is anything that can be changed.

Philip

Re: [atlas] meaning of error message in Ping measurements

[Philip Homburg]

Hi,

> Could someone please help me figure out what these error messages in result 
> field of ping measurement mean?
> - error: sendto failed: Invalid argument (related to name resolve?)
> - error: sendto failed: Bad file descriptor

These are errors returned by the sendto system call on the probe.

The sendto(2) manual page on Linux gives a description of when sendto
returns those errors.

By and large I would say that these errors are not supposed to happen.
So this is probably a bug in the probe code.

Philip

Re: [atlas] Reconnecting probe 3508 - troubleshooting

[Philip Homburg]

On 2016/08/22 22:14 , Michael-John Turner wrote:
> The reason why the probe was offline originally was that it stopped
> accepting DHCPOFFERs for some reason, with the result that it would
> continually send DHCPDISCOVERs every few seconds but never ACK an OFFER and
> never connect properly to the network. That seemed to be fixed when I
> powered it on again yesterday but now I see the problem is occurring again
> (it started again as soon as the initial lease it received yesterday
> expired):
> 
> ...
> Aug 22 21:10:18 dreamland dhcpd: DHCPOFFER on 192.168.1.247 to 
> 00:80:a3:91:38:ac via br0
> Aug 22 21:10:21 dreamland dhcpd: DHCPDISCOVER from 00:80:a3:91:38:ac via br0
> Aug 22 21:10:21 dreamland dhcpd: DHCPOFFER on 192.168.1.247 to 
> 00:80:a3:91:38:ac via br0
> ...
> 
> This poor probe seems cursed :( Is there any way to reset it (it's a v1
> probe, according to the Atlas website on firmware 4480)?

One possible explanation is that the probe doesn't actually receive any
packets (i.e. the receive circuitry at the ethernet level is broken).
The only way to know for sure is to open up the probe and attach a
serial console. But I'm not aware of any software failure mode that
leads to this behavior.

Philip

Re: [atlas] Incrementing LTS For probe 6106

[Philip Homburg]

On 2016/08/09 17:21 , Clinton Work wrote:
> Probe 6106 has an LTS value of at least 187450 and it continues to grow.

The code that manages the lts value does a consistency check with the
time on the controller. The problem with the anchor is that the
connection to the controller (in Amsterdam) is slow enough that the
consistency check fails.

I don't know why. The latency to Amsterdam doesn't seem exceptionally
high. See

https://stat.ripe.net/m/widget/atlas-ping-measurements#w.mode=condensed_id=2030_id=6106

Philip

Re: [atlas] Verbatim stick falled in read only

[Philip Homburg]

Hi Francesco,

On 2016/08/04 14:23 , Francesco Molitierno wrote:
> The verbatim stick of my probe seems falled in a read-only state, there
> is something that i can try to fix?

Unfortunately, when the USB stick is read-only it should be considered
broken an has to be replaced. Any USB stick of 4 GB or bigger will do.
Note that all existing data on the stick will be lost.

Philip

Re: [atlas] Probe reported as disconnected but is actually online

[Philip Homburg]

Hi,

On 2016/08/04 14:18 , Colin Johnston wrote:
> surely a fsck via the mini operating system, of the affected flash drive file 
> system should able to be attempted remotely before reinstall of new image ?

The problem is that the mini operating system on the built-in flash
doesn't notice that the filesystem on the USB stick is corrupt and then
tries to boot from it.

It is not clear what the most fool-proof way would be to detect
corruption. We are working on changing the filesystem from ext2 to ext4.
Hopefully that helps. If it doesn't, then we have to look into detecting
corrupt filesystems.

Philip