RIPE 91
Main room
Plenary session
11 a.m.
BABAK FARROKHI: Hello, welcome to the second part of today's session. Please take your seats. We are about to start. For the first presentation for this segment, for those of you you were not following IETF development recently, we have an interesting talk. Suresh, Erik and Mirja are going to tell us what's new at the IETF.
SURESH KRISHNAN: Hi, welcome everybody. So, I am a member of the IAB, I work for Cisco but I'm not here for Cisco, I am talking as IAB. That's Erik on my left here, he is an Internet area director and he also works for Cisco but.
ERIC VYNCKE: Actually I am paid by Cisco but I work for the IETF, that's different.
SURESH KRISHNAN: On my right is Mirja, she is a web of the IAB and she works for Ericsson.
So, just to talk a little bit about the ideas of IETF. I think it's pretty well known to a lot of you but in case you are new, we do want people to come in too, so the ethos of the IETF that is everybody it participate, all or working product and the final results are all freely available and free to participate in, and we judge everything by technical merits, and we determine access by deployment. This is not like RIPE database, it's a community led organise and we see our standards getting deployed.
This is like a slide we like to show. There is a bunch of regular stats stuff. We have a whole bunch of areas in there. And this is not supposed to be read through but read it in your free time. But we have a bunch of technologies that show up. If you look at some technology and it looks interesting, you can go back into what area it is and look at other things in that same area. For your curiosity.
So this is like a set of topics we thought could be interesting to an audience here who is operationally very well versed, and we just start putting it up. At the end I do have a question to see if these are the kind of things you want to see further or not, but I'll leave it till then. So, the IETF have recently started putting in AI, just like everybody else, including my friend Erik here who has got some A2B built pictures in the slide. Those are the ones with the typos, if you look at the bad typos or stuff like that, it's from the gemini AI.
STENOGRAPHER: Thank God, it won't be from me.
ERIC VNYNKE: You really like to make fun of me.
SURESH KRISHNAN: We started workhing on AI preferences. Text is something that colours, whether to access the site and can colours for search. That's how it started off. We are extending this to see if people can scrape content for AI training, or like generate AI or inference and stuff like that. This is something that got recently kicked off, this there is Jan and we are making progress and we have a whole bunch of of people from content providers, AI platforms and so on coming and talking here. We'll be continuing to have this discussion. There is also this thing call web bot auth. So with others colours, it's become very difficult to check whether somebody is who they say they are. So somebody is could say I am a Google bot but nobody checks. Will then you could get permissions to do things that you may not want to. There is work to kind of start doing tomorrow cryptographic checking for this to check they are who they are.
Of course, like any hot topic, theres a whole bunch of side meetings happening in the IETF. I don't know if there is a similar concept in RIPE, we have a bunch of side meeting rooms where anybody can book with two weeks advance and get a community to go and talk. This is something like akin to a BoF but without the drinking. There is a whole bunch of stuff that's happening in the AI area. Will again not to be read through here, but it's something you can click through. There is a bunch of links following this presentation to go look at it.
Also, Mirja chairs like map RG and there was a session at the last IETF which talks about how colour traffic is impacted by all these
ERIC VYNCKE: What is map RG?
MIRJA KUHLEWIND: Map RG is part of the research task force, we are looking at papers from the research communities that have interesting results from the IETF communities. Usually we look at protocol deployments but we did a special session on crawling traffic last IETF. You can watch a recordings, I think it's really interesting, we had people from CloudFlare and also from our own tool team but then also Wikipedia also had a presentation about problems because they see that nowadays like 30% of all of the traffic comes from bots and that is actually loading the system unnecessarily.
SURESH KRISHNAN: Also if you look at the results that came out, like the TLD version of it is there is a lot of blocking happening, so a lot of the stuff that's happening for preserving Internet history, a lot of the educational research stuff is getting blocked. It's collateral damage for a the lot of AI stuff that's happening. You can go look at the whole presentation a bit further.
Now, passing on to Erik for the DNS stuff.
ERIC VYNCKE: As we all know and I am pretty sure that the other people in the room understand that DNS is the foundation of the Internet right. So we see what is capable of this.
What's putting there that would have a deep impact on you? What is the newly form for one year now delegation Working Group. It's about oh you can delegate to a zone. Right now you use an S, DS and then you need to look for A or AAAA later. We want to simplify this, and basically have a single record looking like service dB or services dB where you get one shot all the information you want. And yet it's quite new. That's a very narrow scope Working Group. So,ing the Charter is only about this requirement and solution. So two documents only. Very dense Working Group, heavily working, with a lot of people coming from the RIPE community working there. For instance, I don't know, Florence is somewhere in the room, and Jim is here as well. You wrote the requirement.
We changed a lot, not so much, right, so the same DNS traffic, simply the zone changing. Other stuff, deacon, domain connect, it's coming in from an operator, so it's coming from people like you, GoDaddy in this case, this is oh, if you are running a hosting service and you have a web service there for customers of yours but you need to then update the A and the AAAA or whatever, but you don't run the DNS for your customer, so you need, you A the hosts provider tell your DNS server, hey please update this record. This is about domain connect. Just newly form working with the matter already.
DNS op, DNS operation, this has become a huge huge bot. I think there are 150 people participating in meetings, they are typically meeting twice during the IETF week with another loaded agenda and a big queue of documents. It's actually a mixed between operation and the main DNS op, as well as maintenance and extensions like DNSSEC. So two kinds of people, right. So one with the people in the black are Board and one of the operators. It's nice to have both of them in the same room of course, so they don't advance something which is not deployable, and vice versa, it happens sometimes.
So, there is a thought to split DNS up in two, one doing still the operation and one doing the protocol extension and maintenance.
At the bottom, it looks like people questioning this. There is a new top reverse zone in the IPv6, the tree or something like that at the bottom line. It will appear, it's already there, right, the that he mandatory the route servers and they know to address this. There is nothing below as far as I know. This will be for registering all the drugs. Of course the one that obey to the low, not the one that's used nearby, alas. You will know who is operating this, which is the mission in CERN. So you will see this appearing in your DNS cache sometimes. Most of you will not be running it, the zone, because it will be managed by international civil aviation organisation.
OK. Operation:
SURESH KRISHNAN: On the operation side the ideal is is work on energy efficiency, mainly metrics for that. There is the green Working Group, we very good at cute acronyms, that's actually a working acronym for this. There is a bunch of he metrics that have been been defined for turning on and off the switch and so on. This is all in scope for this RPKI with. They had a bunch of meetings to start working on a modelling things and there is going to be a read out in Montreal. Please feel free to participate in the meeting, or look at the recording. You can certainly sign up for a remote registration and if you cannot afford it, you can certainly get a waiver for that too. So please try to join in because this is very important like with all the energy that's getting consumed by data centres more than networks right now but it's something that everything is covered in here.
Again NMRG, they do a lot of cutting edge research in network management. So this is something that either is something like issues that are unidentified and are being researched or something that's in the leading edge and it's going to lead into engineering soon.
There was an Working Group, some of my colleagues came and talked here about this workshop about what do we do like what happens in the last 20 years in network management, how did the YANG NETCONF solutions work out for people. We did find out some very interesting findings from it. People still use SNTP a lot but that's a shocking thing. In retrospect not so shocking. But there is a report for this thing that it's out, look at it but it's also following work that's going to happen. There is thing called Onions, they plan to meet in Montreal and online.
ERIC VYNCKE: So just one draft of the testify. It's a draft, right. It's about the prefix lengths of the end site. You, as a provider, you get a /29 for your v6. And you give out /48 to your end customer, so you will be able to present into the Internet the data that they, I have my /29 but what I'm giving to my customer is a /48, so if you want to be reputation database for instance, one customer equals a /48. So you know exactly the size of your end zone. It could be very helpful for management, at least we hope. Comments are welcome. Other stuff: On the Sunday morning, you cannot attend for free, right, IETF ‑‑ you can have a free waiver, you can attend for free if you want. IEPG, that's where many providers and operators, including enterprise, for instance, like Google or others, are coming and presenting experience. It is most probably the most RIPE like or NANOG like part of the IETF. It's on your Sunday morning. If you are in Europe it's Sunday afternoon, it will be six hours, you are welcome too. Any comment on my gemini sketch.
SURESH KRISHNAN: No, I am good.
ERIC VYNCKE: What are we doing there? So you know there are two main groups, v6 operations and of 6man and we have the chair here, about the maintenance. One is about if you want to do NAT64, the NAT ‑‑ you need to understand which is the prefix used by NAT64. There are two ways of doing it. One using DNS in a specific domain and one using areas, what advertisement. This draft, or this RFC says hey let's use route advertisement, it's much more convenient and much more reliable.
There is also a way where the other one, this RFC 981800 is about if I have a CPE, you receive a dedicated prefix from the provider, but the CPE itself should be ready to also send prefix delegation to those three routers. Allowing all network like mine where I would love this to get delegated prefix in making a cascade.
Another one, you know about the 2001 BD8, about the documentation prefix, it's only a /22. If you want to make a big big document about I don't know, VPNs and so on as a service provider, it's not enough, right, so you need to get another one, a shorter one, which is the one that has been allocated there. IP IPv6‑mostly, right that's what you are running and I was talking last time on this. 6man. There is one RFC about how do you do it this interface on a defier with the link look out. Typically to a person but if you use a person in a URI, it doesn't go, right. The person in the UI URL are used basically to escape characters.
There is one draft SKA, if I want to do it DHCP in my network, but it's nice to talk to the host here, you can do PD there, so you can request a delegated prefix. So there is a flag there.
And source address selection. The nightmare, right, it always takes me one hour or two to understand how it works after one year, they want to update it and basically prioritise ULA compared to IPv4 in some case.
SURESH KRISHNAN: There is like on the routing stuff, there is a base to BGP 4, that would be interesting for to you follow and contribute to. So BGP is getting updated, so there is a draft for 6271 base. And grow really works on BGP scaling and if you are interested in BGP monitoring, the work happens there.
CIDR is really about RPKI, and like how the certificates are there and how we do the original validation and stuff and so on. It happens in CIDR ops.
MIRJA KUHLEWIND: Transport. That's me. Oh it's not, sorry ‑‑
SURESH KRISHNAN: Just to also point out it's like a whole bunch of post quantum work happening. So there is a lot of work that has been kicked off regarding post quantum crypto because it's something that's happening and multi.comments are pushing for it and so on. There is HPKI, that's something that a group has never done, like publish in a couple of months. That is something that's ongoing and they are on track for doing that. The plants is something that's a new work, it's a BoF that's interesting. It's because like post quantum signatures are really, really big, they are thousands of bytes long and the certificates are going to have issues when you do certificate transfer. So they are trying to do something interest with merkle trees, so you should probably follow that too.
I wanted to point out one thing in the slide that's super interesting. This is this thing called post quantum cryptographic for engineers, this is like an excellent document, whether or not you are interested in IETF, anything like this. This is like a really, really nice document for to you look at, to understand the engineers' perspective of post quantum. So it's a proofed for publication. I have a link to it here, so please go take a look.
MIRJA KUHLEWIND: So, in the last year, there are two new Working Groups in the are transport, it's not an area anymore, it's the transport working field, which might be of interest to you. One is the happy Working Group. The happy Working Group is working on a new version 3 of the Happy Eyeballs proposal. The Happy Eyeballs was developed to raise IPv4 and IPv6 with the a time when IPv6 was still buggy and problems. The version 3 we're working on now is not only doing this racing but takes more protocols into account. So for example, it's also racing QUIC and TCP. I think the other thing that is new here is we also know now that IPv6 actually works better so the algorithms are slightly changing in order to favour IPv6 a little bit more and to make sure there is kind of report out on logging, because one the problems with Happy Eyeballs is that you can see problems with IPv6 and this he never get fixed. That's kind of the scope here.
The constitutional Working Group is working in mobile phone network you see that traffic gets trotted. They are developing a small protocol to provide guidance to end points so a media server can swaddle itself rather than the network doing it. That gives better user experience because you don't have delays and drops and whatever. The thing I am mentioning here, might be interested for people here, the way it works is it's focused over QUIC and it uses a new QUIC version number. It's not QUIC anymore. It uses a number and does something completely different, it provides a guidance. It's not a transport protocol anymore. You meet see this on your network at some point.
Then another interesting part is about tunnelling is there was a BoF at the last IETF meeting in the summer about protocols about transposed transactions over TCP also called PTT H, or potato. And this is kind of a set of collections where somehow proxying is used in a different way, either reverse proxy or peer‑to‑peer protocols and this effort is still ongoing to create a working group here that extends different protocols, HTTP but also discovery of proxies and so on to support these use cases. That's coming up.
Quick updates on existing Working Groups. The QUIC working group just rechartered in order to work on QUIC over TCP, or QUIC over streams. TCP is one solution. Why are we doing that. That enables application to always use TCP 3, but still use it if UDP is blocked. It's a deployment situation and we always like to stack protocol on each other. That's coming.
Then the MASQUE Working Group. This is kind of tunnelling over QUIC Working Group, so modelen VPNs plus plus that use QUIC instead of IP secretary, they completed the base work about the base protocols, they are working on extensions right now. Optimisations for QUIC over QUIC basically. The one thing I'm mentioning this is because the last bullet point here, the Working Group is about to recharter to work on ament do which is called the MASQUE proxy. So that's more like explaining the concept behind that. So that might be an interrogatory read for if you you never heard about MASQUE before.
Just to say more about QUIC. There is a bunch of whatever over QUIC going on in the IETF, QUIC is everywhere, one the biggest efforts is actually media over QUIC. This is a new media protocol that uses Mick and the interesting part is having a relay where you can subscribe and distribute your media over is part of the architecture there. So somewhere in the newcomer you have a relay and then they split out your video traffic from there.
ERIC VYNCKE: So, we are all engineers, right, so we read Star Wars, Star Trek and the like. All of us knows better than the rocket scientist, we know about space and rockets. Can you imagine the IETF creating a working group called tiptop, taking IP to other planets. It is basically all we can use existing protocols mainly DNS and QUIC, and not TCP, right. To go to the moon, which is three seconds, or to mars, which is, whether mars is nearer the earth four minutes or the other side of the sun, 20 minutes and of course if it's behind the sun you don't get communication obviously. So not only you get huge delay, but you get also intermittent communication. That's where TVR, and DTN can help there. There is also a space research group, look about the acronym right. It's space but it means something can be expanded.
So, feel free to join again. It's on Thursday the full day in two weeks.
DM. Again I was talking about drones and drones, nobody should not target ambulance or hospital. Of course we know, it's not true. But if you target an hospital with a red cross or whatever on the top of it, it's a war crime. That does not protect the hospital per se but it's a war crime. So what about now the server, a DNS server, a web server giving red crescent data. If you target such a server by your cyber attack, then really it's a war crime as well. So that's how you can signal this.
MIRJA KUHLEWIND: We recently, the IAB does workshops for interesting topics. We had a workshop on age restrictions. I won't say much but this is a hot topic in the policy space so we are trying to connect policy and technology here and we have an upcoming workshop in December on IP geolocation, that might be interesting for you. A lot of servers use IP address for geolocation and it doesn't work well. So we are looking at the problems figuring out what we can do better.
SURESH KRISHNAN: And so, just to kind of close on this. So we put together this topic because we thought this would be interesting for the audience here. Will of course like we kind of speed ran through the whole thing, and we would like feedback from you on what you found interesting and what you didn't find interesting, so we can go into a little bit more detail in a future meeting, and we do want to have this continuing engagement with the RIPE community and all the operator communities. We also did this at NANOG, we want to keep the protocol development people in sync with the operative people so we get the feedback much faster. We are seen when we don't do that things don't go well. We want to keep this as an ongoing engagement between people.
As I said like we are, we can talk will a lot of things for a lot of time. We would really like your feedback on what you think is going to be interesting for you.
ERIC VYNCKE: On this one quickly, the Cooperation Working Group tomorrow I will be running a panel with four other people that are RIPE community members also working for IETF and to get their feedback. See you there.
MIRJA KUHLEWIND: We will be around, at least until Thursday, if you have any questions that you can't ask right now, please approach us and talk to us.
BRIAN NISBET: Okay. Perfect timing. Okay, I didn't see which one of the two you got there first.
RUDIGER VOLK: I wanted to add for a gap that I did see in your presentation for network management operations which is very much of concern to this audience. I missed the mentioning of the Nmop Working Group, which actually in its initial discussions kind of triggered the name ops workshop which of course was important and is important, but kind of the NMop Working Group is doing really relevant and active work at the moment actually with a major European participation that's really practicable stuff and actually the NMRG is doing interesting stuff, but that's really for the far out future.
AUDIENCE SPEAKER: Wolfgang Tremmel. I wonder why you didn't mention that UDP got its first major update after about 40 years.
MIRJA KUHLEWIND: The extensions you mean? Okay, I wouldn't consider this like really news because we're working on this for ten years or something but you are right, the RFC got published, so yes.
AUDIENCE SPEAKER: Hi. Andrew Capling, IETF, enthusiast. Impressive by the way to run through all that stuff in such a short time, given all the scope of things. I thought the recent age sort of restrictions workshop was very interesting, I look forward to the output from that.
But what I wanted to say was more directed at the group here. It would be fantastic to have even more people with operational experience from the community present in the IETF, because the more actual hands‑on sort of practitioners that are present we get much better standards, so it would be great to have more people here. As you said, I think remote access, they are freely available, fee waivers, so, attend at no cost using Meetecho, so the remote experience is very simple for anyone were RIPE.
MIRJA KUHLEWIND: Just contribute to the mailing list. You don't have to show up in person at a meeting too.
SURESH KRISHNAN: That's a really good point. We want everybody to come to the IETF too. We are also coming here to collect the feedbacks, because time zones may not work. Sometimes it's better to meet somebody on their home turf because it's easier for them to be more open about the issues and so on. This is something we realised during the of workshop, is that we got much Bergin put from the forums than people coming to the IETF. We would love both. That's what we're trying to do to get the inputs here as well.
AUDIENCE SPEAKER:
JIM REID: Thanks very much, thanks to you for coming along here and explaining more about what the IETF does. I think there is a couple of fundamental things that maybe could have been brought out earlier on is that participation in the IETF has a very, very low barrier for entry. All you really need to do is having a working e‑mail address and some command of the English language. That's all you need to do to take part. Join the mailing list because every Working Group has got a mailing list. All the key decisions that are taken are taken on the mailing list. They are not taken at the meetings so you don't have to physically attend the meetings. And you can also participate remotely, and again the fees for participating remotely are very low. I think it's the order of a couple of hundred bucks, and in many cases you can give fee waivers for that so for the students and others in a case of hardship, there are waivers there, so those charges don't even have to apply. So anybody who wants to take part in the IETF can. There is no real restrictions or limitations on that. It might feel a bit intimidating because a lot of grey beards and old guys like myself hang around the corridors, but everyone is welcome and you can take part and everyone has something to say. Don't feel intimidated because of the fact that you are not a relevant contributor to the IETF. Come along, take part, follow the failing lists and make contributions when the occasion arises.
ERIC VYNCKE: For the fee waiver, just to be clear, it's for remote and there is no question asked. You don't need to be a student or a retired yet, anybody, right.
MIRJA KUHLEWIND: We didn't put any participation information on the slides but if you have any further questions, also come to talk to us, we are really happy to talk to you.
BRIAN NISBET: One question from myself. Because obviously some of the things you are talking about encryption and all of these pieces. Is there any law enforcement engagement in the IETF?
SURESH KRISHNAN: No so much but there is policy maker enforcement, so the ISOC in collaboration with the IETF, they get a bunch of policy makers on the specific region. The last time we had a lot of European policy makers. It trickles down that way that the policy makers later get into the programme where they are kind of walk through a programme and look at the relevant Working Groups and get the feedback and take it back to the comments to push back. That's how it works. Andrew mentioned this, the workshop, it gets input from the regulators and policy makers directly from law enforcement.
BRIAN NISBET: Cool. Thank you very much.
(Applause)
.
So, time is really important, as the Programme Committee are very clearly aware. So, we now have Shreyas talking about NTP clients and are they always right?
SHREYAS KONJERLA: Hi everyone. Good morning and welcome to this PC talk about NTP. My name is Shreyas and I am going to present my master thesis. Also this is my first RIPE meeting.
So today we'll talk a bit about NTP and we'll analyse the behaviour of NTP clients and normal operations optimal operations. Then we'll look at the two types of attacks I did on the NTP clients.
I do want to mention that this project was done in collaboration with SIDN Labs and tweed.
So, we all know time sync is important for many essential applications. RPKI, TLS, DNS, all of these protocols depend on the right time. And if they are, if they have bad time, they can have ‑‑ they can be significantly affected.
NTP has become the default of time synchronisation protocol on the Internet, and it was designed to mitigate the effects of changing networks changing latency, and it can still provide exceptional accuracy in terms of milliseconds over the Internet. NTP takes accurate time from satellites and distributes them across billions of clients very efficiently.
So the latest version of NTP is the version 4, which was standardised in 2011 and it's good for clock selection, offset calculation. They also added a kiss of death mechanism which we'll talk about a bit later. In 2020 a new extension was added to increase then authenticity and integrity of the packets.
Talking about the background, the research, which has been done in this field. A lot of research has been done on the protocol itself and the servers. But there is very little work has been done on the client side and this client is actually the main thing. It's going to decide whether or not to trust the servers, how much to change the clock back, and if there is any issue, any vulnerability at this point, it can something the best of infrastructures out there.
In total we looked at eight different clients. After eight there are three which are the system defaults for Apple, Windows and Ubuntu.
.
So a quick summary of what we will discuss. We measured the behaviour and analysed the behaviour of NTP clients under completely normal conditions and see how they performed. And then we exposed them to two types of NTP attacks. Time shift and the kiss of death.
Before we get to any attacks, we need to understand how NTP baseline performance in optimal conditions. That means how often do they query the NTP servers, how many servers do they use at any given point of time and how does it compare to everyone else?
.
So, we created a virtual environment using virtual machines and in that virtual environment we had three systems configured at NTP servers and these servers were synchronised with external time sources from Apple and Ubuntu. And all ‑‑ three three servers were behind a specific domain address, and all the clients, client virtual machines were configured to use the specific domain. This is the what was presented at the role world set‑ups where usually the clients are just configured to use a specific domain address and the local DNS server directs them to the nearest NTP servers.
So, when we run all of these experiments for ‑‑ when we let the clients run for about 60 hours we can get to analysing the data. Here, the first three which you see are the SNTP clients. Also, they are the default clients. You can see that they use ‑‑ they send significantly less queries than everyone else. They generally have a fixed polling rate. And ‑‑ which is usually a standard queries every 15 minutes‑ish. If you look at the other normal NTP clients, you also see a point where they are something their behaviour. For example the NTP client, pretty much the reference client, it's sent queries, 12 queries every hour. By all the other NTP DRS sends up to 250 queries. This is significantly higher than the rest. But when we talked to the developers, they mentioned that the client dynamically is trying to send a network condition the latency the jitter and if it sends ‑‑ if it finds NTP server which is close by, it generally sends out more queries to improve accuracy.
So this data, the previous data, this slide was aggregated on three servers we had. I mentioned we had three NTP servers in our virtual environment and these three were aggregated on the three servers. But if you split that data, we can make another oh and that observation is at the last row. Time sync D only used one server. You might that this is because it's a simple NTP client and the RFC does mention that these clients should only use one source. But if at all this one source is malfunctioning or even malicious at that point, it becomes a single point of failure. MacOS and Windows both are also SNTP clients but both of them utilise all the servers. So, they are less dependant on the single point of failure.
.
All of them utilise all the servers pretty much equal. And they also use all of them statement.
Now here in this graphic you can see the difference between a normal NTP client and a simple NTP client. The top one you can see it starts off with a low polling rate, which is the time interval between consecutive queries. You can see it starts off low and then it goes up and down and up and down, and it various, depending on the network conditions. But on the other hand, simple NTPD, it just starts over the specific polling rate and it stays over there forever. We can also see that the NTP clients are using all three servers at the same time. On the other hand, SNTP is just using one server. Again we mentioned this to the developers and they mentioned the limitations of SNTP are widely known and if the users want higher accuracy and better protection, higher integrity, they should switch to alternative, more robust alternatives.
So now, I mentioned TimesynD is the default NTP client for Ubuntu. But in a few weeks, Ubuntu will be rolling out an update where they'll switch to NTS by default. It operates its own private pool of NTP servers and they control all the servers in that pool.
So, NTP packets are generally 48, but NTS packets on the other hand are over 250 bytes which is five times increase over the normal packets. Plus we saw NTS time sync sent significantly less packets than normal NTP. How does it reflect on the increase in the surge in traffic to these NTP servers.
It does it quite a lot. Switching to NTS, there will be a 25 times increase in that amount of newcomer traffic, and a 11 times increase in the number of queries sent to these servers. We reported this to the developers, and it turned out they had not thought of this, and fortunately along with this update they are also, they will also be rolling out additional servers to handle the surge.
Now we get to the attacks. So, the first one is the time shifting attacks where the attacker tries to manipulate the client into changing its time. A similar situation happened a few years ago in 2012 where one of the NTP servers at the US naval observatory had an unexpected incident and it reported the time significantly wrong by about twelve years and all the servers and all the services depended on this specific server. All of them had outages and disruptions. This was not an attack as far as we know but it highlighted what will happen in the case of a successful attack. And this is one of the most feared situations for the network operators.
So,s to combat this and to resist this massive jumps, NTP has specified a panic threshold which defines a maximum allowed offset during operation, and it recommends the panic threshold to be around 1000 seconds, which is about 15 minutes. In our experiments we tried to force the client to jump beyond this.
.
Talking about the threat model. So, in this environment, in this experiment, we ‑‑ the attacker controls NTP servers and we had, we had configured clients to use attacker servers. This might seem impossible or a difficult position to obtain, but this is very easily possible by just adding malicious servers in the NTP pool which is volunteered programme or by DNS hijacking or IP spoofing or even like BGP hijacking.
So, how did the NTP clients perform? It turns out not that good. At least the SNTP clients were available to do time shifting to a certain level. Keep in mind, the SNTP clients mentioned over here are the system defaults. So billions of devices are affected.
.
MacOS was, I was able to change the time for macOS anywhere from 15 minutes all the way up to two years, for fun we also tried an offset of 30 years, and in the same, in a predictable amount of time it changed to 1993, very easily.
Fortunately Windows has at least some protection where it does not allow time jumps over two weeks. So that's why you can see any time jump over two weeks would was not successful.
The time needed for a successful time shift that was also different. For macOS needed two hours, like the same amount of time for all the different off sets. TimesynD needed just 13 minutes, which is nothing. Windows, on the other hand, needed a significantly more time, which is about 36 hours. This probably means it has some sanity check or it takes longer for it to trust the servers.
Now, we move onto the next type of attacks which is the kiss of death attacks. A quick background about the kiss of death. It was introduced in version 4 to protect the servers from aggressive clients. By aggressive, I mean clients which sent out way too many queries in a short amount of time. It was used by servers to send control messages, for example to tell clients who send less queries or just stop using it completely. But it has ‑‑ it uses same packet structure as normal NTP packets and it was again able to see limitations of NTP. It be be easily spoofed and it can be easily inspected because it's unencrypted. An attacker can use spoofed packets, it can manipulate the client into stopping synchronisation completely.
The goal of our experiments was to see if we can do that. There was an RFC which was published a few years ago, which detailed the best practices for NTP. To help mitigate some of the drawbacks of NTP. This RFC does mention, it recommends honouring, the client should honour the kiss of death packets but instead of completely stopping synchronisation, it should only slow down up to a certain point. So we tried to manipulate the.clients into pushing it beyond this limit and seeing if they actually, if the developers have actually implemented this safety mechanisms.
.
So for the experiment we set up we had a similar setup from the previous excerpts where the attack, where the clients were configured to the attackers servers and tried out two different types of kiss of death packets. The first one is the rate which means the client should reduce the polling rate and the second is denial where the compliant should just stop using it. Again this scenario where the client is automatically directed to the attacker servers can be achieved by DNS spoofing having malicious servers in the NTP pool etc.
.
So, after running the experiments for a bit, we can see most of the clients had not vulnerable and are unaffected by these kiss of death packets.
This does imply the developers are deviating from the established standards, but this actually providing better security for the users. We had some ‑‑ we observed some bugs, some issues with the results also, for example one of the clients, NTP DRS sent out 5 hundred thousand queries in just five minutes, which is about 615 million queries in an hour. Fortunately it was a bug which we did report and which was subsequently fixed and NTPD, which is the reference implementation, had a weird behaviour. It stopped querying when it was told to slow down. And it sent too many queries when it was told to stop completely.
So, closing off:
.
We tested three different ‑‑ we see the three major OSs have vulnerable SNTP compliance. Fortunately one of them will be going out of the list, which is Ubuntu, just leave to NTP with Croney soon. Time shifting issue we saw with Apple, I did report this to them, but after six to seven months of back and forth, they were not convinced this is a problem and concluded that this is not a bug or any problem anyway. We do recommend like both macOS and Windows deploy stronger NTP clients or at least at some, some alternative, some sanity checks to help secure and help secure the use, because it is the system default, so billions of users are dependant on them. Most of them are not like us who understand NTP or understand the protocols.
Lastly, every system D based Linux distrust issues should use time sync D, for their NTP client should strongly consider moving away from TimesyncD and switching to an alternative like Croney or NTP DRS or many other options.
So in total this project found about ten issues/bugs, most of them were reported to the respective developers.
Finally, a quick takeaways from this project. NTP lurks in the ground of all modern computer systems. It plays a key role in its well functioning. The default clients defaults for major OSes are vulnerable. And it's creating unnecessary risk for billions of users.
So, if you want to learn ‑‑ if you want more details on my research, want to see more detail, more fancy graphs, you can follow this QR code to a tech report.
So thank you very much for having me here.
(Applause)
BABAK FARROKHI: Thank you, interesting presentation, and I see there are some questions in the room.
AUDIENCE SPEAKER: Tom Hill, BT. I really enjoyed this presentation, thank you very much for doing the work. It's actually very fascinating to see such differences in clients.
The graph that you had comparing TimesyncD to Croney and the Ubuntu releases, there was obviously quite a big difference in query time, but was any of that attributable to NTS rather than simply the differences between TimesyncD?
SHREVAS KONJERLA: Yeah.
AUDIENCE SPEAKER: So what proportion of that was Daemon versus protocol?
SHREVAS KONJERLA: So, the graph on the left the increase in network traffic. So that includes the bigger packet sizes and the more frequency of packets. And the right is just looking at the queries. So, if you just look at queries there is an eleven time increase but if you look at the network traffic in total, there is a 25 times increase. Sorry, before it was not 11%, like 11 times increase, sorry.
AUDIENCE SPEAKER: Fascinating, thank you very much.
AUDIENCE SPEAKER: So, I really would like to see follow‑up of this NTP if you lot of things was stuff we looked at when we did NTP. That's 8915. One thing that I'd really be curious B we defined a new kiss of death code with NTS, which is the NTS snack code. I was wondering if there was going to be any attacks possible on that to see. This is super good, but like I would love to see some follow‑up with NTS when the Chrony stuff goes on.
SHREVAS KONJERLA: We did look at that also. So NTS knack is basically a server who is ‑‑ who can not decrypt your authentication packet and it sends out a normal NTP packet with a knack. We tried, we thought of that where an attacker can spoof that packet, making it look like a normal NTP with a knack. But we did like, I don't know if was a limitation in my step but we could not get any good results from it.
AUDIENCE SPEAKER: And the traffic thing is interesting but the goal was to fit it into a lady because that's the v6 MTU. That's the goal we're look working towards but it's good to see the numbers are fairly high but we need to know how that scales. Thank you.
BRIAN NISBET: Just from online, we have David: "Have you tried jumping Windows time over several steps rather than one big jump to try and circumvent the sanity check?"
SHREVAS KONJERLA: Yes, as I mentioned it has a two‑week threshold, but if you go step by step, like one week at a time, you can make it go beyond a month.
AUDIENCE SPEAKER: Peter Hessler. We're the upstreams of open NTP and one the tricks that we do in open NTPD to prevent the time skew attack is we have a feature called constraints and what this does is reach out to an HTTP server and gets the time from the HTTP headers, and we validate the certificate and all that. While this is only one second resolution, it's very good to see if the NTP server we're talking to is giving us bogus or realistic time. I wanted to mention that for the audience.
SHREVAS KONJERLA: We also found that recently, like the constraint feature. But one small limitation, it's not on by default. You need to manually configure to a website so...
PETER HESSLER: In open BSD datastream configured by the default, I do not know what status is of the portable distribution on the other operating system, it's entirely possible that Ubuntu or any third party packager has turned that off by default. So from our perspective we have it on by default but we should look at the details to see if everyone can ‑‑
SHREVAS KONJERLA: For the portable it's not on by default. So we need to manually configure it and direct to a specific HTTP server.
AUDIENCE SPEAKER: Raymond Jetton. I don't know if you have time for it in your presentation, but I am missing the GPS jamming, which is a very current thing in Europe. That GPS jamming can cause the clock to think they are somewhere else. Have you had time to look at it?
SHREVAS KONJERLA: Unfortunately that. That GPS jamming part, it fell a bit outside of my project scope, so we did not look at that.
BABAK FARROKHI: We are one person in the queue. Or two.
AUDIENCE SPEAKER: This is Karen Donoghue, I am the Chair of the NTP Working Group in the IETF. There is a couple of things that I wanted to mention. First of all, I would invite you to bring this presentation to the NTP Working Group, that would be very helpful.
The second thing I wanted to mention is I don't think the concept of a reference implementation is vowed at this point. I realise that that's the way it's documented in the RFC, but there is no single reference implementation for NTP going forward.
I mean, the third thing, I just need to understand this, so I might be half asleep. But the distinction between NTP clients and SNTP clients, I think SNTP was never intended to be as robust as the NTP clients. I think you need better clarification between those two. And also, just as a side note, not to you but to the person who just asked, there is a number of efforts going on in the GPS jamming, and it has been a topic in a number of other venues, I can get you some links to that if you'd like. Anyway I do thank you for the presentation, I do ask you to come to the NTP Working Group. Contact me if you are interested in more details.
.
(APPLAUSE)
BABAK FARROKHI: Next up we have another talk on NTP.
CALIN OLARU: All right. So, welcome everyone. Thank you for joining us here for our first RIPE presentation. We have come ‑‑ we are here today to show case or new application NTP info which is a tool that allows users to measure the accuracy of NTP servers.
So, this tool was built by us. Only four of us are here because our colleague George is doing a matter in Korea, that's a bit far away so you are stuck with only us.
So just to give an outline of what I'm going to talk about. We are going to introduce you a bit into the problem and understand the problem at hand to better understand why our application is important, and give some context into how it came to be. Then dive a bit deeper in the research that we have done to ensure the best results, the most accurate results, and to get all the features right.
We are then going to have a life demonstration of the application and the finally some community feedback and thoughts.
So, first of all, as an introduction. Why such a tool is needed, why is it so important? Well time synchronisation is essential in most modern systems especially in the online medium. It allows trains to run, it ensures security in financial transactions, and online communication. On the Internet clocks are synchronised by the network protocol servers, also known as NTP servers or time servers. Despite the importance of NTP, there are currently very few available tools, publicly available tools that allow the users to actually get accurate measurements for in servers and even those offer very little relevant data.
A failing or inaccurate NTP server is a huge security risk and as an example as I said in the previous presentation, in 2012 the US naval observatory had a problem where all the clocks were set back to the year 2000, and anything that was related to logging failed across multiple systems in the USA affecting millions of people.
So, to understand the problem and why NTP is so important. This is the solution. You will see a preview of what you will see in the live demonstration. So our tool offers the following key features.
Users can perform a live NTP measurement and get the information about the accuracy of the server such as offset, RTT and stratum from the time server in the Netherlands, second set of measurements is performed by RIPE Atlas from the probes very close to the location of the client. So it's more for the user. After that the user also has its of visualising the historical data of the server management in the past via a few nice graphs. Also have the option of downloading said data. Some newer features are analysing whether the server supports NTS and what NTP versions it works on
.
My colleague will give you a bit of context about thousand this application came to be.
MIHAI‑VALENTIN NICOLAE: Thank you. Now, you can see we did this project as part of our software project course in the end of our year in university. We were coordinated by a teaching assistant, where we held weekly meetings with him, we had a coach and we also had a client for whom we actually made the project. We are students, we worked over 32 hours per week with the goal of producing a working project at the end of project.
So, now some context about our client. His name is Dr. Giovane Moura. He is a data scientist at SIDN Labs and assistant professor at TU Delft. He has worked in the field of NTP for years and proposed the probably to the university. He has collaborated closely with us, holding weekly meetings and providing us useful feedback. And helped us to get the project deployed via SIDN labs. Now we're going to dive into the research that we have done and I am going to let my colleague talk to you about that.
HORIA‑ANDREI BOTEZATU: Thank you for the introduction. Now in the research and the challenges that we had during this project.
So, one the first challenges was getting the right IP to measure. Let's say that a user inputs a domain name and the scanner uses multiple IPs. We decided that we wanted to take the APs that are most probably used by the user. So how did we do this? Our solution was using can EDNS. EDNS has a client sub‑net option that let's us look like we did the request from the part of the user, and for doing this, we also masked the IP of the user because of privacy user, we did a 40 P MASQUE for IPv4 and 60 bit for IPv6 because this gave us enough granularity to really get accurate, more accurate location for the user.
Now, the second challenge is related to the external things that influence the NTP measurements. We have a lot of them, like geographic distance, latency, and this too comes in really handy because they affect the RTT time. And the next one is packet loss. We used UDP which we know that is not necessarily ‑‑ it doesn't have retransmission, so if we perform a measurement, it could be lost or we could have incomplete data. And the last one it's IP type. We'll touch upon that later in the presentation.
Now, my colleague will talk to you more about a really interesting scenario.
SERBAN ORZA: Thank you. So, now imagine this scenario. You are somewhere in Japan and you have some NTP servers near you but our server is in Amsterdam. If we measure or server in Amsterdam you get results like this one, which is pretty inaccurate for your case because the distance will interfere too much. So there would be confusing and misleading. This is why we use RIPE Atlas because you all know it is a global distributed measuring platform that has almost 11,000 probes all over the world and they can perform measurements, and for example, in this image we measured the NTP server from London and we used the 100 probes, each probe behaved differently. Not every measurement result is relevant for our client, because sometimes the distance in the network delay interferes too much and destroys the measurement. So now, therefore we need to get to select the best probes because we clearly cannot use all of them. We need to get the best probes to get the best accuracy and results for our client. How do we do this? You may say take the closest one to the client. But close in terms of what? For this, the information that we had to use the client IP. But from the client IP we can get a lot of useful information. Such as the local, which will help us finding geographically close probes near the client. We can also get the ASN which is the network of the client. And also the prefix which helps us routing. With all of this information only, we need to find the best probes. It's important to say that we currently use only three probes per measurement, but it's configurable. If you want you can use ten or more depending on your needs.
So, now after analysing the different options to combine those tables and discussing with Dr. Giovane, we decided that this is the best way we should choose the probes. Will as you can see in the left side, the having probes that have the same ASN and prefix as the client is the best thing because the network latency is very small and the distance is usually just a few hops to the client. Then if you don't find enough probes in this category, we move often to the second one and the third one and so on until the last one where we are guaranteed to have some probes. Also I think it's important to note that in each category, we saw the probes again by the distance to the client. And we use maximum databases for this.
I let my colleague tell you about the problem that we came up this approach.
HORIA‑ANDREI BOTEZATU: Thank you. Now, let's talk about IPv4 and IPv6. Let's say we have the following scenario: Somebody with only an IPv4 address wants to measure IPv6, to do IPv6 measurements in our application, and we need some IPv6 details obviously for this. So, what can we harvest from the IPv4? Well, we can harvest the location. The prefix, it's related just to IPv4 so we can't use that. And maybe the ASN, but there is some special parts about that. Next thing, we found that a solution for this problem is trying to use a reverse DNS queries to get the IPv6 address for a user. And how it works. Our server ‑‑ asked for a PTR record for the client IP to the DNS server. The DNS server then sends back the PTR record and our server then sends a request for an IPv6 query for the PTR, and the DNS server can then send back either the IPv6 address or nothing, but this depends a lot on how the provider configured the whole network that you are on.
And now I'll talk about the control flow application, like this is a more high level overview about how we perform the measurements. So, the client inputs let's say a domain name to our server, the question goes to our server, and then we do what I also told you earlier in the presentation, the configured DNS server gives us the NTP server's IP that were resolved through the EDNS and then our server measures the NTP servers, along with that we also send these requests to RIPE Atlas who delegates everything to the probes that also perform the measurements. The probes are specially chosen to be really close to the client. So they provide more, really accurate data, and the more relevant data, and then at the end everything is aggregated and presented back to the client, to our really nice front‑end.
Now I'll let my colleague talk to you about the latest feature that we added, which is also measurements for NTS. Back to you.
SERBAN ORZA: Since we finished the course we continued working on this project and we managed to measure time. Let me remind you what NTS is. NTS is ‑‑ NTP wasn't designed with security in mind so this is why NTS exists. It basically ads cryptographic to NTP packets. It has key exchange and NTP protocol. In the first one, you can exchange some keys and cookies over a TLS connection. And also the AEAD algorithm to use. The second part you just encrypt, so you'd authenticate the packet and send it. And it can measure both key exchange part and... can analyse the output, can also validate the cookies, see if the packet is corrupted or modified, also it can detect which server is NTS and an interesting feature is that I don't know, I think you know but sometimes the key exchange provides alternative IP port on which to continue the measurement. Our tool can detect this and tells you hey this NTP server wants to do the measurement on that IP not on this one.
So basically, we think that adding NTS to our OpenSource tool is very useful, because there aren't many things that, many public things on NTS and NTS is trying turning out to be something very useful. Like we said in the previous presentation Ubuntu will update in a few weeks. We believe it's a good option, it's tried and tested.
Now it's time for a live demo, which I hope will appear on the screen.
If you want, we hosted an AWS instance, which you can access on this link, and look at the measurements. But please keep your phone on the landscape mode because it's a desktop application, so on the phone it may look strange.
CALIN OLARU: Also if you want to try the project in the state that it was at the end of our software project, that's hosted by SIDN Labs, but this is the newest version that we have created in the last weeks on AWS. We are going to let you guys try it and we are going to try and figure out if we can actually show it to you guys on our screen as well.
SERBAN ORZA: Until they configure the file, do you want to measure a specific NTP server?
CALIN OLARU: We're just fixing something technical issues with the live demo, it should be here shortly.
HORIA‑ANDREI BOTEZATU: I still argue it might be something HDMI ports.
BRIAN NISBET: If it makes you feel any better, this is a rite of passage in RIPE presentations and demos, so welcome to the club.
STENOGRAPHER: Anyone in the room have any technical knowledge?
.
AUDIENCE SPEAKER: One question I had is like do you have any IPv6 support on this?
SERBAN ORZA: Yeah, we support IPv6 measurements, both NTS and NTP.
AUDIENCE SPEAKER: I would like to see a measurement for the NTS D server at NIS. If you can.
STENOGRAPHER: We are sucking diesel!! Oh, no we're not.
MIHAI‑VALENTIN NICOLAE: It's not our fault.
STENOGRAPHER: Nobody move!!!
BRIAN NISBET: I don't think this is looking good. I think we may have to ‑‑ try one more time. But...
SERBAN ORZA: Until they finish this I want to say that in future we also want to add the ability to send the measurement links, for example, to send links to the measurements on our website, to be able to I don't know, share with others. I discovered something here again, look this is a problem and so we can share it. Maybe another interesting stuff is that I don't know if you know but in April, we found a bug in how the ...was correlating the offset for the probes, it was an inverted sign. So I think this is pretty awesome.
MIHAI‑VALENTIN NICOLAE: We have it now. You guys can see the screen now. So this is the version that we have worked on for ten weeks doing the software project. (Live demo).
So for starters I am going to just type like an NTP server domain name, I am going to measure using IPv4, and ‑‑ now that's going to load. The first results that you see here are from our time sync server that's situated in the Netherlands, and now it's also going to load results from RIPE Atlas probes that are chosen close to this location. So, as you can see here, on the map, the probes are chosen here in Bucharest, close to our location, using RIPE Atlas. And the vantage point is the server in the Netherlands. Will you can also see the NTP servers that we measured. For more information you can also see the measurement on the RIPE Atlas website. And right here, you can also see like a historical graph with offset the round trip time for recent measurements. And if you want to see more historical data, you can see them here and you can choose whatever time interval you want to see trends over time. And if you want, you can actually compare servers. For example, I am going to compare time.google.com with time.Apple.com, and you can see that it displays it very nicely. But what's more interesting I think is the new feature that we implemented here. So this is the AWS version that we have shown you the link for, I am going to run, for example, time dot.apple.com again, but this time using IPv6, for example.
You can see now we have NTS measurement as well, it's going to load quickly, but time.apple.com doesn't support NTS. You can also download the results in JSON and CSV format. We can also see NTP versions for different NTP versions. Last but not least, let me just try to run this server. I want to show you something interesting. I want to show you an actually successful NTS measurement on a server that supports it. Will so, for this server, it does support the NTS measurement, and you can see more details about our measurement as well right here.
I think that's everything that we could do in this time for the live demo. Thank you for sticking around and we are going to touch upon some community feedback back then on the stage, Calin is going to talk about that.
CALIN OLARU: Yes, thank you, so just to move quickly through the community feedback and thoughts. So we received a large amount of feedback from numerous members of the NTP community, most of them very useful for researchers and for other people who are interested in the Internet measurements but do not have time to make some of their own. We also received some very interesting suggestion that is led to the addition of features such as analysis. Over all we'd like to thank the NTP community for their continued supported and allow us to improve or application. We don't have any more time but sorry we'd like to extend a special thanks to SIDN lab, especially our client, for the continued support of hosting the app and for providing the services, RIPE Atlas services for using the app and for giving us the opportunity to be here and present to you.
.
I'll leave you at that, but if you have an idea for a project you can scan the QR code. You can test the tool and the students get working experience in the industry. So, we thank you. I'll leave it at that and I'll now open the Q&A session if there is any more time. Thank you.
(Applause)
BRIAN NISBET: So first off, I think we have Karen online.
AUDIENCE SPEAKER: Hi. I'd like to thank you again for this presentation as well, I'd also invite you to the NTP Working Group.
The question I had was with respect to your measuring individual servers ‑‑ you are measuring against individual servers and I was curious if you had any aggregation type of data. I am always curious about what percentages of the Internet are supporting various versions or various large scale data sets, I was curious if you had any look at any.
SERBAN ORZA: If I want to input into the input field, for the NTP versions analysis, we performed an NTP measurement for participation but it's important to note that NTP v1 is no longer in use and NTP 5 is still developing and we used the draft 5. Currently our server is configurable to be able to use other graphs, but... yeah, basically we used the applications for NTP 5, this is I didn't some servers don't accept it, some servers accept it but the ones that accepted it they usually just pretend they accept NTP 5 and in reality it's just in 4 but with some modifications.
BRIAN NISBET: Okay.
AUDIENCE SPEAKER: Thank you for that. That's really cool. I was wondering if you got some, a bunch of free credits from the Atlas community? If you need more, we have got a bunch I guess.
SERBAN ORZA: Thank you, we got a lot of credits, we were almost billionaires in credits.
HORIA‑ANDREI BOTEZATU: Thank you very much for the openness and making this project possibly. We really would like to thank you the whole RIPE community that helped us get a long and also for giving us the opportunity to be here.
MIHAI‑VALENTIN NICOLAE: And we also thank RIPE Atlas for temporarily increasing our limit on our account. So you guys can test it without worrying that we might break something.
BRIAN NISBET: I am sure we can find a way to break something. Okay. Thank you very much.
(Applause)
.
Okay, so that was the third presentation. Thanks again to all of our speakers. Just a very quick reminder, if you would like more awesome academic content, we have for the first time in a long time posters, a poster session during the coffee breaks Tuesday, Wednesday and Thursday, by the RACI, some of the RACI team, the RIPE Academic Cooperation Initiative team, so they are out in the foyer.
A quick reminder to rate the talks and there is still time to volunteer for the PC elections that stops at 3:30 this afternoon local time. And there are buses at the social this evening which are all in your packs, there is information you can read yourself on the Internet.
Enjoy your lunch. Thank you all very much.
(Applause)
.
(Lunch break)