RIPE 91
Bucharest
Main hall
22 October 2025
9 a.m.
Connect Working Group
WILL VAN GULIK: Let's give it one or two minutes for everybody to get into the room: I think we will try to start now, because we have a packed agenda.
So so welcome to the Connect Working Group. If you are looking for OpenSource, that's not this room. Thank you very much all for being around. I am here with amazing co‑chairs, Paul and staf Ross, thank you. So, I would like also to thank the stenographer, and our scribe, and with that, you have all read the minutes of the previous meeting.
STAVROS KONSTANTARAS: All of us, and we had so many comments, we don't know how to address.
WILL VAN GULIK: Perfect, we are all on same line. We actually had a slide with the agenda. Oh I forgot that. It's even readable, you know. I'm making some progress now.
So, that's how our agenda and we will start right now with the first presentation.
LEO VEGODA: Hello everyone, I have an update for you on PeeringDB. I start with the people news. So, this is the Board of Directors and officers, Shauna is the secretary and Treasurer, the person who deals with sponsorship, if you want to become a sponsor, she is the person who will be helping you. Job, in the bottom right, is also Chair of the volunteers operations committee. He is standing down. We're looking to recruit more people to the operations committee, that's the group of people that makes PeeringDB run as an operation. So if you or Ä.O1 of your colleagues are interested, please let the stewards know, you can send me an e‑mail. The e‑mail address is on the front slide. This is the admin committee. They are the people who answer your support queries and help you when the website doesn't do what you want it to do.
This is the outreach committee, will be joined Yolanda who is moving from the product committee. And this is the product committee, and Yolande's picture should be on the other slide, she is moving as of this week.
I think operations news is probably the most interesting, we have an apology to make, we're very sorry about the recent performance of the site. We have been the subject of am lots of AI scrapers and they scrape faster than we would like. We do have rate limiting, but the rate limiting was put in place for the API and not for the web pages because we didn't anticipate that the AI scrapers would ignore the robot text and just scrape, scrape, scrape. The good news is that we are completing this month a move from VMs which didn't scale well to containers which can scale much better. So the immediate resolution for the operations issues is that we will be able to just scale horizontally and add more capacity. We're also this month introducing metrics for service level indicators, things like how quickly does it take for you to load a web page, and we will then be putting some work in over the new year so that we go and make improvements based on the the measurements that we discover as a results. We do have ideas for improving operations. Things like distributing the service geographically, segregating the service for logged in users versus anonymous users, that sort of thing. Adding PeeringDB which is our local mirror cache that you can run on your laptop or in a server in your own organisation. At the moment it's a local version of the API. We're thinking maybe because 4 and 5 of the people who respond to our sure is survey say they use the website versus the API. Maybe we need to add a web interface to DB‑PY so that you can go and do web queries that run on your own laptop or a server in your own organisation, but the immediate thing that we're switching to containers, we're going to be adding more capacity, and then we're going to be working on further improvements.
So, outreach news. I am developing a little micro learning about peering DB‑PY explaining why you should be using it and how simple it is to install. We will hopefully have that ready early in the new year, and then the ‑‑ it will be on the RIPE NCC Academy. The RIPE NCC and other organisations will be able to refer to that in any training they do when you get an AS number or something like that so that anyone who needs access to PeeringDB can go and run all their queries locally and of course that means that they are going to get the very best performance.
Ben, as Chair of the outreach committee, has a big plan for the next year, and how he is going to manage community engagement. I don't want to steal his thunder so I'm not going to say too much about it, but, you know, we have a lot planned for next year.
Product news:
We now have the network comparison feature. It's been released, it lets you compare networks by facility. We will be expanding this so that you can compare networks based on IX, including country data, make the columns sortable, and improve the colour scheme with a key and all that good stuff.
There is more product stuff that we could talk about, but time is short. So, I will just push the survey. Even if you just answer one or two questions, leave a comment, please tell us what is important to you so that we can focus future product development based on what it is that is valuable, because obviously we like it when you send us e‑mails and you engage in GitHub or whatever it is to tell us what you want, but this is a good way of telling us in a very cheap way to you, in that it can take just a couple of minutes, but because there are hopefully going to be many of you who will look at that QR code and go and complete the survey, that will mean that we'll have statistically meaningful input from our users.
And I'd like to remind you that PeeringDB is entirely supported by sponsors. You can become a sponsor if PeeringDB is important to you. And these are the sponsors who are supporting the work that we do, and we'd like to thank them very much.
Thank you.
(Applause).
STAVROS KONSTANTARAS: Thank you very much. Anyone has any questions for Leo? I don't see anything. Thank you Leo.
So, I would like to introduce the next speaker, Stefano from from Italy. It's a first RIPE meeting for him, so please be kind to him.
STEFANO SERVILLO: Hello, today I will explore the blind spot of Internet Exchange point route servers in combination a NaMeX and AMS‑IX.
In the BGP, there are two types of peering. Private peering among two autonomous system and public peering, multiple autonomous system and IXP will facilitate this public peering that are route servers that will announce BGP announcements upon IX members in this way annist can make only a one BGP session with the route server instead of N minus one with old holders. In in way they can interconnect each other to exchange traffic. Internet Exchange point became really important critically in the ecosystem. As you can see from the traffic that they handle, most of them now handle more than 1 terabytes of traffic. And to avoid large traffic disruption, IXP apply BGP filtering policies at the route server. These policy can be divided in two types. We have the basic filters that are based on chat like reserved prefixes, prefix and AS path length. They are mainly inserting the inbound session. Then we have the advanced filters that are based on RPKI, IRR and also ASPA. For this part there are no strict rules. Operators can apply filters based on best practices, their needs and their clients and for that reason there are different approaches. Someone has put all inbound, some all in outbound or split between in and out.
So, let's see how they work.
Well when an announcement is received by the route server, there is, for example, the validation. And the invalid is filtered. The other two goes to the IRR validation and that is more strict and have only two outcomes, valued and invalid. Only the valid is accepted. When the validation is unknown it means the PI had no information for that prefix so all the ways of the validation is based on the IRR. In the IXP, this validation is made with the IS set. They are a election of AS numbers and AS‑SETs that are used for route filtering. This is to insert all the clients. Here is an example that is the AS that gives connectivity to Italian university and research, and here there are all the members, all the clients. And then GAR is using this AS‑SET in the session, they are saying that through that session, they can propagate the announcement of all these ASes.
I set up some problems, for example, the application, in a non‑authoritative IRR, this makes it difficult for the IXP to choose the right AS‑SET and also the biggest are problem that is not maintaining them. In fact the size can grow inactive ASes. Here are the top AS‑SETs ranking by number of ASes and all of them are more than 100,000 of ASes. With the first one which in 102 thousand of ASes. This number can look normal, but instead they are problematic because for example, they are located at 130,000 while the active ASes are only 77 thousand of ASes. Here there are also the IPv4 and IPv6 prefixes that are related to those ASes inside the route object. Also we have more than 4 million and more for IPv4 and more than 800,000 for IPv6. The first question is how IXP use this AS‑SET for IRR validation?
.
Well most of the IXP uses BGP Q4, which is a tool that retrieves the information from the IRR specifically AS‑SET and route object and creates the filtering policy for each member. What it does is for each member, it will retrieve the AS‑SET and expand this AS in a list of autonomous system numbers and for each ASN retrieve the prefixes inside the route object, and then with that prefix, it compiles the prefix list. And these prefix list contains only the prefixes that are inside the route object, and is linked to the peer session inside the route server configuration. Here there is an example. AS1 is connected to the route server, it retrieves the information and creates the filtering policies, that it goes from AS1 if the prefix it inside this set, accept. Otherwise drop.
So, there is one question is are route servers safe from prefix hijacks and misconfiguration? Well, not completely. Why? Because as you probably see, there are only information related to the prefixes and not to the real owner. So in this scenario, if AS3 is the real owner of the prefix and AS4 is announcing the same prefix, if the announcement is reaching the route server, it will be accepted. Why? Because the prefix is inside the list and is received through AS1. So, AS4 did the prefix hijacking or a miss configuration.
So, all of this was just found. We checked the configuration, we found this problem, this blank spot. What we did was to analyse the data from NaMeX and AMS‑IX and we did two types of analysis.
The first one is that feasibility analysis when we want to see what can really happen? And the control plane analysis in which we show that it's happening.
Let's start with the first one.
We defined some requirements that have to be fulfilled by the announcement to be considered valid by the route server and be accepted. And for each requirement, we check inside the RIB if there are specifically prefixes that are fulfilled only that requirement. So the first one is prefix must not be covered by a ROA. We found that 32% of announcements received by a route server are in the unknown category. Then obviously the update must not be filtered by intermediate ASes, otherwise you will never reach the route server. And we found that 9 percent of announcements are invalid for the route server. So these ASs are not filtering invalid announcements. Then there are some requirements based on the AS‑SET. The update must reach the route server through the authorised member. We found that 52% are via the authorised member. With 28 percent via two or more authorised members. So this gives the possibility let the announcement reach the route server to multiple session. And the malicious AS is not required to be inside the AS‑SET. Because the route servers is doing the filtering at the prefix level, not AS level. In fact, 5.4% of announcement considered valid by route servers, the origin AS is not inside the AS‑SET. The last two requirements are here only to let the prefix be inside the route server configuration. So, the prefix should be inside the route object with an AS‑SET. And that AS should be inside the AS‑SET of a member.
Here, there are AS‑SETs specifically for the peer of the IXP, and there are some near number to the holders with the first one reaching 99,000 of ASes. So the value that AS is inside 1 the IXP member AS‑SET is really high.
Then we show, so the IPv4 and IPv6 prefixes that are related to this AS inside the route object, but without a ROA. So, this is the upper band of number of prefixes that are affected by this blind spot.
So, after this analysis, it can really happen. So, let's see what is happening. What we did was to take data from the IXP and use the requirements, try to identify the prefixes that are related to the blind spot. And we found that among 1,000 of prefixes are related to this blind spot. It means that they are valid by ‑‑ considered invalid by the route server but the origin AS has no route object. Obviously not all these cases are malicious, there are some misconfigurations, some business agreements. So what we did was a little bit of high level analysis where we checked if the origin AS has some relationship with the owner or with the prefix. And we found that some, most of the cases are related to the business agreement. So, customer provided relationship or a sibling relationship.
At the end we found that among 200 of prefixes for us are considered suspicious because we haven't found any information that related to the origin AS to the real owner. So these cases should be subject of further analysis.
So, it's happening. But we found other interesting things that this type of filtering affecting also the correct announcements. For us a correct announcement is the case in which the origin AS as a valid route object for the prefix. And we found that for 65.5 percent of these announcements are filtered due to outdated AS‑SET. So, there are received by a non‑authorised IXP member. And this is the problem for the operators because they are announcement be filtered, and our worst scenario is that for the majority of these prefixes, all the announcements are filtered. So the route server has no route for that prefix and the operators is losing the connectivity within the IXP. The others are considered valid and based on the BGP selection based on that are propagated or not.
We found some cases of ASes MOAS. That's where it's announced by more that you know two or more ASes. Among these cases which are 153 prefixes, we found tomorrow specific case in which the same prefix is announced by both invalid and valid autonomous system. And they are both considered corrects valid by the route server. And so, as I said before, I accept that doesn't mean propagation. So we checked the propagation, and we found that for all of them, the route server is always choosing the invalid AS, so it is propagating a route via an illegitimate AS.
Then obviously another question is it an isolated case or not? What we did was to search for some of the IXP inside the IX community, and what we did was to check inside their website if they are using the AS‑SET, and then we looked the LookingGlass we validate or prefix inside the LookingGlass to see if they are being to be accepted or not. We found that most of these IXPs are affected, they are not protected by this blind spot, with only four that are really protected. In this case, there are some IXP that have the green symbol, because we found some information. So, only the analysis on the RIB can give us an outcome. So, if there are some IXPs that want to contribute to my research, we can look into Europe and to validate also our results.
The last thing obviously, there should be a solution for this blind spot. The solution is try to do the same thing that you are doing with FBI but based on the route object, so keep that information about the real owner or the prefix. So, you should validate all the announcements against the route object. But obviously this could be problematic and inefficient. So we found some of the solution still using the AS‑SET, because a lot of information relating to the prefix and to the ASN, and so we can combine this information in order to create our filtering strategy. But the route object validation should always be at the end.
What we did was a numerical analysis. Here we are comparing our strategy in order to see how many announcements should be checked, validated against the route object validation. And we found that obviously the one containing only the route object validation is the worst, while the best solution is combining the filtering on the prefix level, ASN level and then route object validation. They did use the number of announcements that are checked by the route object validation.
Obviously, we have some recommendations both for network operators and IXP operators. For network operators, obviously they have to create ROAs for each prefix because this blind spot is happening only if the AP validation is unknown and they have to keep IRR AS‑SET objects up to date and try to use authoritative IRR for the AS‑SET. For the IXP operators, the combination is just review your filtering policies. Check your prefixes inside your RIB to see if you are affected by this blind spot. And if you are affected, try to apply one of the proposed solutions.
As a conclusion, we exposed a blind spot in the IXP route server filtering and we show how invalid announcements can still be accepted and propagated by the route server.
We show that this is not an isolated case but a widespread challenge and more important, I show as case studies some IXP, but this is not happening only in the IXP. All ASes that are performing such filtering, this type of filtering based on AS‑SET can be affected by the blind spot, so even network operators can learn by this analysis.
What we want to extend our analysis to IPv6 to compare with IPv4. Then we want to try our solution on the route server in order to measure the performance of the route server, both in terms of time to take to update the configuration, or time to possess a single announcement and then obviously some analysis on what the consequences of this filtering, how much traffic IXP are losing due to this filtering.
And that's it. Thanks for your attention.
(Applause)
STAVROS KONSTANTARAS: Thank you. Amazing presentation. Please keep in mind when it's about to test the solution and to measure the RS implementation because I have something. I know there are people that are waiting to ask questions.
AUDIENCE SPEAKER: Radu. So I do not fully agree that the best way to validate is using IRR data. So, IRR data is important, it has to be for certain parts which are not implemented in another way, but one big problem that we IXs have not only with IRR data is the authenticate. There is still a lot of providers, especially in the US, but not only that are still using RADB which is not an authoritative source. There are still some of them but which use some other IRR, non‑authoritative IRR sources, but let's say that those ones are small enough so that we say don't use, okay.
For me, the most important is RPKI. So, for the origin, if RPKI says it's okay it's okay, I am not checking the data for the prefix, for the mapping prefix origin. What we still have to use IRR data for and let's hope ASPA will advance to a point where things will probably enhance, is AS path, so yes, AS path as far as I understood there is not many IXPs that do this check. We do it, some others do it but not all of them do it, and this is something that ‑‑ so basically what you are saying in your presentation is do AS path filtering, use AS‑SETs, but even that one is something which you may or may not rely on. I know for example, you know, right now there is all that stuff with listing IP addresses. There is some people that take transit from cogent, the lease an IP block from cogent and they want to announce that lease from cogent to an IX, depending on which block could only gives to the customer it may be a legacy block or it may be a block with a full feature depending I suppose on how do they get to the support, they may not get IRR information, and RPKI information on that block, so, what some of them do, they say, but wait I can just include the Cogent test and in my AS‑SET or even worse, I can include cogent AS‑SET. So... this gives what you told us in the beginning, you have AS‑SETs with tonnes and tonnes of AS‑SETs inside which don't have to be there, there is no authoritative way of validating AS‑SET content.
STEFANO SERVILLO: Yeah. This problem is only because FBD I harks not reached 100 percent. When it reaches 100% this solution is not needed. But in the meantime there is still this blank spot can be solved by doing this type of validation.
AUDIENCE SPEAKER: It goes pretty far but not to the end.
STAVROS KONSTANTARAS: I would like to go to the next speakers.
AUDIENCE SPEAKER: Will, My question was like, it would be great actually to do some small test for the smaller IXPs because obviously a bunch of people there are not in the list of your IXPs, so I'm representing those now. Saying we would like to test that, if we can do it it's going to be fine and great and I'd be happy to help and to point to you small IXPs for that. So please come to me. If you have got the tool reachable and usable, we could also share that, that would be really nice. Thank you.
AUDIENCE SPEAKER: More just to come on to what Radu was saying earlier, it might be worth for the others to know that cogent has now signed an ARIN agreement which means that space is now RPKI capable. So the last reason for people to import legacy IRRs, unauthenticated IRRs has kind of gone away. You can push them harder because they can actually do it.
STAVROS KONSTANTARAS: That's a little bit correct because it's for cogent. There is also another space from ARIN legacy that's being used from other operators as well.
AUDIENCE SPEAKER: Mick O'Donovan. This is probably a $64,000 question, but I guess like the biggest problem with AS‑SET is trying to maintain it as an ISP or as a network operator. Ours is gracefully succinct and it's okay, my previous employer not so much, that gets very unyieldey and hard to manage. Have you got any commentary about potentially how to, I suppose, brow beat some of the operators into keeping their AS‑SETs properly maintained?
STEFANO SERVILLO: For that reason I put in a slide for the recommendation that obviously network maintainers should maintain this AS object because as I said before there is also the problem of the duplication. If they are creating AS‑SET in non‑IRR, the IXP has no problems to get that AS set and probably the announcement can be filtered by the IXP due to this duplication.
AUDIENCE SPEAKER: I suppose in the past we have received kind of prods from RIPE NCC even to say oh by the way, that member ‑‑ but it's generally when an AS disappears from use. So, I'm not sure if it's something that could potentially be looked at but I think we as a community need to be better at managing AS‑SETs.
AUDIENCE SPEAKER: Dave from APNIC. One thing you are missing is most IXs at least implement prefix count limits, so in the event that somebody inadvertently has an AS‑SET that is overly optimistic,ing at least if those announcements do turn up and the prefix limits are set as per PeeringDB, then everything should be fine. Number two, I don't often defend RADB, but at least since the IRD update if there is a conflicting ROA object that is valid, they will not accept any third party entries. So, it sort of keeps it a bit cleaner from that side.
STAVROS KONSTANTARAS: That's still not enough.
STEFANO SERVILLO: It's limit the number of prefixes but they are still passing.
STAVROS KONSTANTARAS: As far as I know the amount of RPKI invalids globally is only 1,000. If case you have a huge AS‑SET that goes to 100 K thousand prefixes you are still opening a huge hole to the Internet
RUDIGER VOLK: I just walked in during the Q and A, and ‑‑ well, okay, hearing the discussion about AS and maintenance reminded me that five and a half years ago, two weeks before Covid measures coincided with my retirement, I gave a presentation at NANOG about the tools that I had finished at that time for maintaining stuff or helping people to analyse what garbage they are accepting from their customers, and I think the ‑‑ well, I figured out that actually reviving that project might make sense. We have the tools and the ideas of how to use it effectively, and kind of support by any of interested IXPs for doing that work or for hosting a service would be welcome. I think I'll do it anyway, but...
STAVROS KONSTANTARAS: Rudiger, can I request from you to send an e‑mail to the mailing list of connect with the URLs of the tools if they are public
RUDIGER VOLK: They are not yet. I will have to do a little bit of publicity and I'm sure that my former employer will allow me to make it publicly available. I'm not completely sure of the source or container set for delivering service.
STAVROS KONSTANTARAS: Thank you.
(Applause)
.
Now for our next talk.
MIKE BLANCHE: Good morning all. I am Mike and I am here to talk about interconnection regulation. If you were at the European peering forum about a month ago you might have seen a similar presentation before, I apologise, as the saying goes repetition doesn't spoil the prayer. So hopefully it's still interesting.
I am Mike Blanche. I have worked in Spain infrastructure for about 30 years, I haven't been to a RIPE meeting for a long time. It's great to be back. It's great to see a few old faces but there is many new faces as well which is fantastic as well.
I was the first member of the Google peering team in Europe about 15 years ago. I was on the Board of Linx for about nine years and I am now an independent consultant working on infrastructure, Internet infrastructure telecoms policy and so on. This is all my personal views, and nothing to do with any of the clients that I work with. Although if you'd like to work with me, that would be great. Come talk to me afterwards.
First of all, Europe is great. Of tele‑geographies, digital gravity Index of the most connected cities in the world, Europe has four cities in the top ten, which is fantastic. Good work team, you are doing a great job. Keep it up. I think this is fantastic, especially considering like if you think back 120 years ago, the centre of gravity of the Internet was very much in the US, and the places, there are still three US cities on there but they are somewhat further down the list. What I think is also interesting is that some of the countries that certain large telecom operators, which we'll get on to, are pushing as great examples that should be followed, such as China and South Korea, don't appear on this list at all.
So, Europe is doing great on interconnection. We want that to continue as a European success story.
When it comes to disagreements, we never disagree, we never have a dispute, we all love each other very much. I would argue IXP can interconnection it the world's most efficient market that we can have 99.99 whatever percent of these arrangements without any contracts, without any paperwork, traffic flows across the Internet, just on a handshake, or not even on a handshake, just an e‑mail or increasingly automated is fantastic and it's a great example of how we can reduce friction, we can help make the Internet for resilient and reliable.
BEREC were here at the last meeting and talked about their IP interconnection study they did at the end of last year and they consider the market to be well functioning. You can go back and watch the video on that from RIPE 90. Where there is a disagreement, people generally just kind of agree to disagree. The Internet allows traffic to flow other ways, you can use alternative routes, you can use transit, there is alternative options for traffic to get from A to B. You don't have to have a direct interconnection.
But where there is an actual dispute, like a dispute is some sort of appeal to a higher power, how are these resolved currently?
.
So there is basically three different mechanisms, which is the top three sections in this table inside the EU. The first one is consumer law and the open Internet regulation. And there's been a couple of examples of this where like a consumers association has gone to a regulator to complain about poor performance which has been based on interconnection issues. Will there was one about 12 years ago between are Free and Google. There is one ongoing between a group of consumers association and civil society groups going to the German regulator about Deutsche Telekom.
The second group is contract law, and the dispute between Deutsche Telekom and Meta is an example of a contract law dispute. They had a contract. It didn't work out. It went to court. That was how it went that way. The third one is competition law. Theres a few examples here both reactive, so, ex post as regulators call it, so for example Orange and Cogent, Cogent complained about Orange's behaviour, that they wouldn't connect, there is also proactive ones, where if, the two liberty global examples here with liberty global was acquiring customers from another operator and as part of the merger conditions, the European Commission put in conditions about their interconnection to make sure that they couldn't, cheats the phrase, they couldn't leverage their position.
So, telecoms regulators have generally not been involved in this. The two examples at the top where, the one that was actually done, between free and Google, the French regulator went to great pains to actually say free and Google must interconnect, they must do this. You can read the decision online, it's in available. And they say well it's free's job to provide a connection for the rest of the Internet to their customers and that's up to free. And the one case where telecoms regulators have actually made a decision, which is this last one at the bottom, and I see Freddie sat in the front row over here, it's been outside the EU and it's the longstanding challenges that INIT 7 had with Swisscom that took more than a decade to resolve through the courts an through the regulator and the competition commission. And the decision that the regulator made there was that interconnection should be free. Which we'll come back to later. That's the only case I am a aware of in the greater Europe where a regulator has actually made a decision about the terms of an interconnection.
Some stakeholders are not very happy with this. In particular, the European Commission, GSMA, the association of mobile operators, connect Europe which is the association of large incumbent telecoms operators. They are not very happy with BEREC and BEREC's position everything is working fine.
This goes back to the European electronic communications code. It's the current regulatory regime for telecoms, it was completed in 2018, but then completed converting into the law of each European country. Will and that took four or five years. And it's due for its first review in 2025.
The European electronic communications code does no cover IP interconnection. It's vague but it's never been used for t the main features of the EECC are all about universal service, driving 5G and what they call very high capacity networks, so gigabit networking and so on. There is lots in there about consumer protection and contractual rights. But these stakeholders who are not very happy have been lobbying the European Commission and as a result there is a proposed for something called the Digital Networks Act which is coming at the end of this year, the announcement and the draft is coming, this won't need transposition into each individual country's law so we'll go a bandwidth faster but it's still going to be a couple of years before it's implemented. The key features we think are coming are simplification, making a telecoms single market. Addressing what's claimed to be a lack of innovation and investment in Telecom's networks. Spectrum policy reform. Something called level playing field, but also IP interconnect dispute resolution. If you want to learn more about Digital Networks Act as a whole I'm doing a talk on the whole thing in the other room in the next session in the Cooperation Working Group, so come along to that.
IP interconnection cooperation in particular, what the European Commission is proposing is potentially one of these four options which they circulated to stakeholders in June as part of a questionnaire, what do you think ‑‑ how should cooperation be enforced in IP interconnection? There is four approaches they talked about. One the a vague cooperation mechanism, including on IP interconnection. My issue with this is there are many different sorts of relationships between ISPs and CAPSA content and application providers. So people are providing content and services on the Internet. There is many sorts of relationships. It's not just about network interconnection. So the first one is perhaps far too large in scope.
The second proposal is to change the definition of interconnection in the EECC to explicitly include IP interconnection. So previously, the EECC has only looked at a voice interconnection. There is 30 years of history of disputes in voice interconnection because voice works very differently to IP so it's needed regulators to get involved in many ways for many years.
But by changing the definition, you then potentially bring this 30 years of case law for voice interconnection into scope for IP interconnection.
The third option is about empowering NRAs, that's national regulatory authorities, to impose IP interconnection obligations under certain conditions. So a regulator can come and go you must peer with these people so that seems like outsourcing your network management which seems problematic to me anyway. The last one is including IP connection under some sort of a compute mechanism. Some sort of arbitration or mediation where a regulator again gets to decide how and where you interconnect.
There is some challenges and problems with this. The biggest one is that as a peering coordinator, you are no longer looking at the technical considerations and perhaps the commercial considerations of connecting to another party in different places and the trade‑offs between that. You now have to optimise for the regulatory environment which could be different in different European countries or different within the EU versus outside the EU. And there is already some confusion. The Italian regulator, AGCOM, has jumped the gun a bit and decided that content delivery networks are now regulated, should now be regulated like telecoms operators and therefore would be, are in the scope of this sort of cooperation or dispute resolution mechanism, and it's entirely unclear what they consider a CDN. The diagrams in their proposal are quite vague. And so we already have an issue where Italy is different to the rest of Europe.
There is the potential for this to be dispute creation rather than dispute resolution. If you, as a network provider, think you can get a better deal by going to a regulator, appealing to a regulator rather than trying to revolve your differences then you are going to do that. That means that this is the potential here to create disputes rather than resolve them.
And we explored some of the challenges with this in a report that I was a co‑author of. You can find it online. About what might happen if this was brought in.
So, there is kind of three scenarios here. At the top there is flow charts on the right and the kind of network diagrams on the left here. So a large ISP that say is unhappy with their interconnection relationship with a content provider or CDN, they seek dispute resolution on terms of the interconnection relationship and an arbitrator or regulator imposes possibly payment on the content provider or CDN, because that's what large telecoms operators want from the rest of the Internet. And there is three different ways this might play out.
The content provider or the CDN might stay peered with the ISP. They will pay the ISP so the ISP gets network fees, that's that purple box. The content provider, their costs go up as they have to pay money to other ISPs. But there is all sorts of indirect impacts that affect the broader ecosystem as well. It's not just between these two parties. Who what I might describe two relatively large companies large telecoms operator, a large content provider.
Transit providers might have some challenges now. Because it could be that an ISP doesn't like the relationship they have with a transit provider and similarly goes to the regulator and goes well I think the transit provider should pay me, I shouldn't have to pay them. Other ISPs, smaller ISPs are at a competitive disadvantage because they don't have the leverage that the big ISPs have to get paid.
Other content providers might have to pay as well. Other CDNs may have to pay as well or may exit the market.
Internet exchanges are likely to see less peering in the market if there's been decisions where payment is kind of forced from one party to the other. And ultimately the effect on users is negative because the costs of the content that the content provider has are passed onto the users. For example, if NetFlix is having to pay a Kelly comes operator to deliver NetFlix traffic, NetFlix is going to have to pass that cost on to users.
The second approach could be that the cap or the content provider or the CDN depeers directly in the market and uses a third party. This is probably not great for everyone, but it's kind of what happens today, but without the payment.
More costs for the ISP because the traffic is coming through transit. More costs for the content provider. Again you have this situation where there is this precedent of being asked to pay, which can have negative impacts on other players in the ecosystem in that country which is going to say maybe I won't build my network into that country if I am being forced to pay to connect and peer.
The third approach could be we'll just leave Europe, leave this country entirely and connect somewhere else. Connect maybe outside the EU. And this has greater cost, worse performance, the cost of the content provider might go down because they don't have to carry the traffic as close to the user as before, and it's a negative impact for most of the other players in the ecosystem as well, there is going to be less peering in the market for Internet exchanges. There is going to be a less rich interconnection environment and less resilience in the Internet.
Overall, there is probably many losers in this and only one winner in one situation.
.
So, regulatory intervention in peering and interconnection is a risk, and we ‑‑ it could lead to poor interconnection within the EU. Maybe strengthened interconnection hubs outside the EU if the EU takes this rule as a whole, people might do more peering in London or Zurich where it's outside the EU. Worse performance for consumers, higher costs for networks and users, and ultimately weakening that success story we saw at the beginning of Europe as a leading global interconnection hub.
This isn't just theory, this is South Korea, which you may have heard about at previous meetings, has IP interconnection regulation which forces payment from the rest of the Internet to large ISPs in South Korea, and a recent paper which I was not involved in, but it's a good paper, talks about the negative impacts on South Korea's broader digital ecosystem as a result of this. They see less peering in IXPs, less traffic over IXPs, they even see less investment in submarine cables and data centres in South Korea than would be expected otherwise given the rules that are in place.
It's a worry.
What can you do? It I know there is probably a variety of views in this room and different people have different ‑‑ their companies have different views on this but I would strongly encourage that you engage with your organisation's public policy and regulatory teams, as this progresses. Please explain IP interconnection to them that it's not the same as voice interconnection and that it doesn't need the same regulatory approach as 30 years of voice interconnection.
Please stay up to date on developments in Brussels as this Digital Networks Act emerges towards the end of the year. If you join the connect and cooperation mailing lists you will see what the latest is there and you can follow me on LinkedIn because I post on this regularly, and please contribute to the conversation, and we need the voice of the technical community in this, not just people in suits like me. So, thank you.
(Applause)
STAVROS KONSTANTARAS: Thank you, Mike, very much. I saw some disputes on your slides which I didn't know that existed. Very interesting. We have some questions.
AUDIENCE SPEAKER: Alex, a lawyer for AMS‑IX. I am going to read you something, it's from the recent US EU Tariffs and Trade Agreement.
Digital trade. "The parties committed to address unjustified digital trade barriers with the EU confirming that it will not adopt or maintain network users fees."
So I think the problem is solved.
MIKE BLANCHE: You are an optimist and I think there is many ways that we could ‑‑ of those other options in terms of dispute resolution, that could not be construed as network fees in the strict sense, but could still cause problems for people trying to do IP interconnection.
TOM STRICKX: Thank you for doing this, I think we need to start doing the same thing about the DNA as we have been doing with RPKI. We need at least one talk at RIPE about the DNA and network users fees. Thank you.
I think a really important case to make with the Korean example is that technically the Korean regulation is only about ISPs intersection with interconnections ISPs at that point have abused the regulation and have said well effective to pay to interconnect with the other ISPs you are also going to pay as a CAP. They are just technically no regulatory environment that insist that is needs to happen. It's that everyone has decided that we are going to abuse the legislation that exists so I think that's helpful to kind of add as a context to the European side of things as well. Even if the regulation is toned down significantly, which is unlikely to happen, you are still creating an environment that is incredibly toxic and can cause the level of shenanigans that we are seeing in Korea.
AUDIENCE SPEAKER: Freddie, from INIT 7. I am the culprit. Actually, I want to make a small addition. The slide you showed with our case it's not quite correct. It's still ongoing, we are on the last court now in the Federal Administration Court, so, Swisscom took it further despite that they already lost once there. TTL so, I am confident we are going to bring this to an end maybe this year, maybe next year. And the good effect of our case that the Swiss regulator is that we have, we had another network which our current interconnection was doing packet loss because it was full, so, we talked to them and said we need to upgrade and then they said oh we need to discuss this, etc., etc., I said well if you don't make the upgrades we can discuss this in front of the regulator so they same along and said here are the 200 gig ports you need. Well it took a while but they are finally installed.
If you want to add me here something. I mean, as you all know, there is another player involved in this case because of this court thing, and once that case is over, we will look into the other operator.
STAVROS KONSTANTARAS: We have one last question.
AUDIENCE SPEAKER: Thank you for the fantastic talk, in my experience with some of the meetings with the European Commission, I can say that their understanding of the Internet is different than mine, that you know ours. That's nicely put. But can you speculate a bit how they have the idea that this will actually benefit innovation? That's what's really bothering me.
MIKE BLANCHE: I think this is largely there was a lot of discussion between large telecoms operators and Breton, who was the previous Commissioner, and it's kind of percolated into the thinking of parts of the European Commission, not all of it, parts of it, that this is, it has "benefits" for Europe.
AUDIENCE SPEAKER: Thanks. So if you want to learn more about the DNA as a whole, come to my talk at the cooperation group next please.
(Applause)
STAVROS KONSTANTARAS: Before we go to Ben I would like to remind everyone to rate the talks, right, so there is this button "rate the talk" in the RIPE website, and now we can welcome Ben from BGP tools.
BEN CARTWRIGHT‑COX: Hi, I'll get started straight away because I think most of you know who I am.
So, Internet exchanges provide services, there is quite a lot of them provide services on the LAN. If you are lucky some of them will do things like AS112, NTP time, some of the other IXPs have these mandatory route collectors that you have to peer with or sometimes don't have to. But most of the time you have to. Critically the most important service that Internet Exchange offers route servers.
So route servers attempt to solve ‑‑ I suspect you all know what this is. Attempt to solve a simple problem in that networks typically are lazy and don't want to respond to e‑mails and so when you join the exchange, especially if it's big, you endpoint want to send 3 hundred e‑mails to people who will not reply. Instead you peer with these route servers which distribute these routes, thanks to IXPs for better or worse being giant Layer2 networks. The traffic doesn't actually go throughout the route server, it gets, it's rerouted. The route server can control terabits of traffic with 100 megabits of port. And the system kind of works.
Your average ‑‑ the other advantage I think a very understated of a route server is that the average peer quality of config that you are peering with is not he very good, despite what people say at conferences and despite what people write down on their peering policies most networks are not filtering their sessions almost at all. That means not even doing basic is this thing a /24 or is it a /25, they'll still accept the 25.
So, peering with route servers if you are less clueful or your platform limits your ability to write good peering policy because the thing may explode before it ever gets to convert a full table. You are in much safer scenario if you say I'm not interested in doing bilateral peerings, I just am interested in peering with the route servers.
So, you may have a magic config that you are very confident in but one it's probably not going to stay up to date because business needs probably trump your ability to make your config extra nice. But also, it's just going to rot over time because write a lot of things in routers do.
The other problem is that it's very hard to functionally filter route servers. This is AS DECIX, this is the official AS‑SET for Frankfurt. This is an old screenshot. The number has got bigger, but there is only about 80 thousand ASNs in the Internet now, or visible and DE‑CIX has somehow managed to do 101,000 if you totally go through the full recursion. Now AS‑SETs interest also a large number of problems attached to them, some of which was discussed before. It's not ‑‑ this number can vary depending on your implementation. BGP tool has a number of axes to grind about AS‑SETs. What a fascinating is make in the industry.
So, also of course the v4 prefixes we have got about one million today for better or worse, rest in piece your sub‑720s, because it just crossed 1 million. We got 2 million v4 prefixes. Ridiculous, if you try to import this thing in your router, I'd be amazed if it accepts.
There are some disadvantages though. Some networks have different policies on their route servers. I haven't seen an IXP that enforces, if you are going to peer with a route server, I have at the seen an IXP that enforces a policy for route servers, which is annoying because it would be nice if they did. Some networks import ‑‑ some networks do sort of what I would consider the right thing, which is everything that they are going to send to a standard peer, they send to the route servers. And everything they get from the route servers, they import.
Some networks just import from the route servers. They don't send anything. This is potentially the most valid exception to this configuration because there are genuine traffic reasons to not send routes to a route server especially if you are say a CDN or a VoIP where you are really interested in controlling the latency because if you join networks like Linx which have, for better or worse, peers in like Indonesia, what you don't want to happen is traffic harpooning from Indonesia back to London, because that may not be the point of what you are trying to do with peering, that may be the point of Indonesian network is trying to do but incentives don't always align.
But there are weird combinations of this setup.
If you don't already know what this is, it's my general routing information site. It runs its own route collector and it tries to help answer the question of was that a small outage or has CloudFlare already started writing the blog post about the outage. It's generally used a a quick will you please tool to figure out basically what the rest of the Internet sees your network as. As a side effect of that it has a large Internet Exchange presence.
I was wondering ‑‑ to demonstrate the point, the Internet Exchange presence is shown up on the website. If you look up an Internet Exchange point you see these TIX here. The regular green tick means that the website knows that that peer router is online. If it says RS, it means that they are on the route servers and they are advertising prefixes. So, if you click that, it tells you which routes they are sending to the route servers. However, not everyone does that. But, they could still be importing from the route servers, and I would have no idea that they are doing so because BGP has no feedback mechanism.
Incidentally of course the other way are around could be happening. These people or some of these people could also just not be importing from the route servers, they could be sending their routes and ignoring everything that comes back at them. Very annoying. So this is the question I want to ask is: How good is the reach for people who are an outbound heavy network, say delivering traffic, say your CDNs of the world, how good are they, how good are route servers just to reach those users? At the time of writing, which is clearly not what I just said because we are at one million routes now, about 980,000 routes, if you just combined all of the route servers that BGP tools is on together, you get about 56 percent of the table. Which is a relatively impressive amount of routes. On v6 you get 60%. So that's pretty good actually.
A lot of this is combined ‑‑ I want to show you what the thing is in a more readable way. I made a graph. If you combine all of the Internet exchanges sort them by their largest route server table size and then go one by one and remove ‑‑ so ‑‑ this is a complicated graph. I just wanted to show you because it took a lot of time. So, the blue bar is the total size of the route server, not necessarily very interesting because this basically controls whether Hurricane Electric is on your exchange or not. But the orange bar is much more interesting in that it is the cumulative gain that you get if you were running in this order and adding these Internet exchanges, how many new routes, how many new unique routes are added to your network in peering on the route servers do you get.
As you can see there is a huge drop‑off, massive diminishing returns to the point that by the time you are at NetNod you are basically only adding like 1,000 routes but sounds reasonable. The whole of Sweden is probably ‑‑ or about 1,000 routes. If you are a Swedish ISP they are 1,000 routes that really matter, but, you know, a lot of the time a whole country's eyeball networks can compress down to 100 entries. The idea that you genuinely need to import a full table in a lot of people's networks is kind of silly. But I don't know. The technology isn't there yet or something.
There are some caveats obviously to that graph because there is always caveats to graphs. That's a survey of exchanges that I am on. Also you are almost certainly not going to build a real network in the way that I have built the route collector. Just because it's on some very weird exchanges and some very weird locations.
But, critically, also, just because it's 60 percent of your prefixes does no the mean that it's 60% of your traffic. There are a lot of networks out there, especially eyeballs ones, where almost all of your serious traffic is concentrated into what I have acronymed to call the MANGA networks, which is Meta, Google, NetFlix and Amazon. The bulk of your traffic is from these guys. Obviously this doesn't account for weird shenanigans that people do in places like Brazil where you announce for specifics to the route servers where you try to lock in things like IX BR into your network. Most people don't do it, apart from reasons where traffic has been traditionally been really expensive that you can't afford for things to not go over peering.
We have covered the other direction. So we have covered the content network site where you are sending stuff to peers. The question is how do we measure the other direction. Because we can't use control plane stuff for that because BGP as I said famously has no feedback mechanism for saying I accepted something. So, we need other way to see if they have accepted a prefix. One way of doing this potentially on the control plane using BGP data is to just go and see if we send stuff, if I send something to the route servers, do I see my sessions get it back? Unfortunately BGP tools does not peer with the entire world. You could help change this. But I think that's probably a little difficult.
So, instead we need a way to do that on the data plane, which is fine. Using ping or packets or what not. So, strategy was developed. We announced a prefix to all the route servers. With any luck they accept them. Not all of them. Somewhere else you basically send or spoof pings to the entire Internet or a very tactical selection of the Internet, and if they are accepting the route server route, then the reply is surely going to come back through to the route server. Otherwise it has nowhere to go.
So, what do you get out of this? Sorry, some brief slides about how the BGP tools satellite stuff works but it's in the hugely interesting. Also I don't have IPs, because IP address is very hard to APNIC very thankfully lent me some IP space, which I am thankful for, so thanks to Geoff Huston and also APNIC for letting me borrow some of their IP space. I said a tactical amount of IP space. BGP tools does regular scans of the Internet, and can figure out per /24 the most likely IP address that will respond to ping. This speeds it up massively. Otherwise you are pinging three million things and this you are only pinging about 900,000 things which you can complete in about an hour.
.
Obviously this doesn't help in all cases because CGNAT boxes. A lot of CGNAT boxes will not respond to pings. When you hit mobile phone networks or you hit sufficiently new networks in terms of IP space, you can't really do this experiment, which sucks, but I have no good way of working around that yet.
If you all all route servers, then about 15% can reach you if you announce ‑‑ 15% of the Internet can reach you if you announce something to a route server, which to me is lower than expected. I have no answer for that. This massively trumped my expectations. Most packets go to these exchanges so the power of these route servers are sufficiently powerful. Some of this is explainable by policy of certain networks, but I think this mostly attracts with what I would expect. Obviously the long tail here is very long. It's 47 percent. It turns out a lot of regional exchanges do their job very well.
I wrote this talk mostly for European peering forum. I am reusing the slides. I am focusing on the EPF people who happen to be most of the biggest changes in Europe. If you just use these, it's still is pretty dominant on DECIX, which I think is pretty interesting.
The difference between, however the EPF hosted, like the big euro exchanges and basically all 118 exchanges, only 27 thousand routes, which as I said before in Sweden you can probably fit the entire Sweden into 1,000 real prefixes so the difference between those 27,000 prefixes to you may actually matter a lot. It's probably dominated by a small amount of networks that accept ‑‑ with a large amount of downstream people who accept prefixes off route servers. But, I don't know. This is actually a very strange result to me. I am pretty sure the numbers work. But, some of this is almost certainly limited, or will get more limited over time because of the slow rollout of CGNAT everywhere. I don't think I have any other ‑‑ yes, because I used the same stuff that I used to name and shame people who publish, or have bad router configuration and do silly things, I actually know the MAC addresses of everybody who replied to me. I was surprised to learn that China Telecom publicly peer on the route servers in Linx, which is odd but also explained some of the DDoS traffic. So, that's what I found at least. You may find some of these things interesting.
I think that's all. If you have any questions, I would happily take them. If not, then thank you very much.
(Applause)
WILL VAN GUILK: Thank you Ben. We have one question.
AUDIENCE SPEAKER: Tomorrow, CloudFlare. Thank you. That's a massive difference between the import and the export side of things. Will do you have any explanation as to how, why, or what the hell is going on? Because it's a bit erred pooh.
BEN CARTWRIGHT‑COX: Some of this I suspect is probably policy of people genuinely exporting routes and for some reason or another not importing routes, say configuration is make or whatever. I think there is a lot of just bad configuration sitting out there. I think also some of the exchanges, I never managed to get to all of them simply because once again BGP tools has no feedback mechanism. If you push something to a route server there is something to say to the route server itself to say I accepted. You have to go through hundreds of pages to find it was accepted. I think some of the exchanges route servers filter policy is very annoying so there is a really annoying IXP Manager default, because I have APNIC space, even though I am a RIPE ASN, a lot of IXPs for some reason using IXP Manager will only build my AS‑set ‑‑ sorry, filter using RIPE's IRR database which is a very optimistic view of the world is my best politest way of saying that, that is kind of bonkers. That's probably made a massive difference.
WILL VAN GUILK: I have an answer for that, because it was more like a is make and I noticed that I didn't read out the prefix from Quad9 and I was like what the hell did I do wrong? And when I saw that the small box I touched RIPE, ARIN and so on and so on and then suddenly Cloud 9 was announced.
BEN CARTWRIGHT‑COX: The ISP is in the room. Check your route servers over config, because maybe the difference between you and DE‑CIX could be the fact that your policy generation sucks. You may have more traffic from to offer free route servers, whether it matters or not, I don't know.
AUDIENCE SPEAKER: Coming from a big network, it doesn't. But that's a separate conversation. Thank you.
WILL VAN GULIK: Do we have ‑‑ I didn't think we have anything online. No. Okay. Thank you, Ben.
And we have the final talk.
JELENA COSIC: First of all, thank you for this last minute addition. Very much appreciated. And for giving me this chance to share that the report on the IXP landscape in the southeast Europe, that my colleagues and I have been working on for the past few months, is finally out, it has been published yesterday. And you can download the full report by scanning the QR code.
So very briefly, in the report we introduced four bench marks on what makes an IXP successful, and those will be: Is it keeping the traffic local? Is it fostering local interconnection to support digital economy of its country?
Is it able to attract global hyper‑scalers, content providers and all other various other players?
And the last one would be: Is it becoming the hub for the regional traffic exchange? And I just wanted to say if you are interested in discussing, talking to us, discussing our findings of the report, you can find me during the meeting, you can also drop me an e‑mail, but at the same time I wanted to announce that you can also join us at the Open House online session that we scheduled for November 10, and again you can register for free by scanning the QR codes on the slides.
So, very briefly that's it from me. Thank you so much.
(Applause)
WILL VAN GULIK: Thank you so much. I am actually amazed that we made it. That's crazy. Wow. Thank you to all the speakers because that was like a nice, an interesting exercise that we made here.
So with that, I would like to remind you all to please rate the talks. So, as already said previously, you should remember to go and follow things on the mailing list because it seems that there will be stuff happening in the next few months hopefully with some stuff from Rudiger and some other people, Michael. So, please do that.
What else? So I was handed that small piece of paper that says remember to vote for the Programme Committee election by Thursday at 5 p.m. The other one is like if you registered to vote for the NRO NC election, remember to vote by Friday at 5 p.m.
With this I think we can close our session for ‑‑ AOB? Anyone has anything else to ask? Is it with this, we will see you all in Edinburgh, and have a good coffee. Thank you very much.
(Applause)
.
Coffee break.