After Nearly a Decade, Bolivia’s IXP Gets a Boost

After several years of stagnation, Bolivia’s Internet Exchange Point (IXP), PIT Bolivia, now doubles its exchanged traffic every six months.

Unlike the path taken to create many IXPs, Bolivia’s was imposed rather than built by the community. It shows that it takes a human element, trust, to help IXPs soar.

In 2011, Law 164 forced the country’s Internet service providers (ISPs) to interconnect through an IXP, with regulations stipulated by Supreme Decree 1391 a year later.

Given its imposition, many operators were slow to trust or believe in the IXP’s benefits, according to local experts. They couldn’t even agree on a location. After more than a year of negotiations, it was finally housed at state-owned Entel, the country’s main Internet service provider, because it had the necessary physical space.

Bolivia’s IXP was born on 13 November 2013. Its members initially only exchanged 90 Megabits per second (Mbps) in total.

“Because it wasn’t created organically, the operators didn’t take ownership,” says Roberto Zambrana, former president of the Internet Society’s Bolivia Chapter. “They are there because the law requires it, but they don’t release all their traffic.”

Lia Solis, current president of the Bolivia Chapter and networks manager at Entel, has a similar take. “The operators didn’t see it as something that would benefit them. They met there but the interconnection rules weren’t clear, so they did it with restrictive traffic.”

A Change of Venue

In 2016, the new executive director of Bolivia’s Telecommunications and Transportation Regulation and Supervision Authority (ATT), decided to move the IXP from Entel to its own offices, seeing the regulator as a more neutral location.

Carlos Sanabria, head of the ATT’s ICT Unit at the time, says the change was only possible thanks to the Internet Society’s support. “The IXP’s first switch was donated by Internet Society. And for its change of venue to the ATT in 2016, Internet Society donated four switches and supported us to install an UPS [uninterruptible power supply].”

He adds, “The Internet Society is like our angel. They see to it that things work.”

Still, as trust among the IXP members remained low, the volume of traffic they exchanged remained low, too. They also made little effort to attract new members.

“The Internet Society began working on the Bolivian IXP in 2012 with the goal of creating trust among operators to make it profitable for all,” says Christian O’Flaherty, the Internet Society’s regional vice-president for Latin America and the Caribbean. “A law can’t force the collaboration needed to make an IXP successful. That requires trust between Internet service-providers.”

Expanding Its Membership

Today, most of PIT Bolivia’s new members have joined on their own accord and through their own investments in an international connection.

In 2017, Digital TV Cable Edmund, a medium-sized operator with some 35,000 residential clients in Santa Cruz, invested in laying fiber optic cable to obtain an international Internet connection in La Paz, some 855km away. It meant having to connect to the IXP.

“We have noticed an increase in traffic. When we joined, we had 100 Mbps and now our traffic to the IXP has grown to 1 Gbps,” says Daniel Guzmán, Digital TV Cable Edmund’s technical manager.

Installation of the UPS.

He says IXP content has grown alongside its membership, and the more it continues to grow, the more traffic and benefits there will be for everyone.

In 2019, PIT Bolivia obtained legal status as a nonprofit entity—an important step that created an institution independent of the government, allowing it to start making its own plans and projects. Still, the IXP wasn’t looking to grow its membership.

Guzmán believes the IXP needs to integrate more small operators to increase traffic and provide greater benefits to all users. He says that IXP membership shouldn’t be limited to operators, but extended to more government institutions, banks, universities, telemedicine services, and other entities.

With Vuela’s integration, the IXP has 10 operator members, including the three mobile companies with the market’s highest concentration, and one government agency, the Agency for Electronic Government and Information and Communication Technologies (AGETIC).

Carlos Sanabria, General Manager of PIT Bolivia, has extensive experience in both the public and private sector.

A Pandemic and a New Director

In February 2020, only 1.4 Gigabits per second (Gbps) of traffic were being exchanged through the PIT—an increase from its early levels, but still below the country’s potential.

But the increase in Internet consumption during the pandemic forced a change, says Solis. “The operators started changing their filtering policies and releasing more traffic. But the size of the interfaces was small, which motivated an upgrade to 10Gb ports to create more capacity to handle the greater volume of traffic.”

New regulations were also created. According to Solis, the regulatory group started creating policies with greater vision, they instituted membership fees, and an IXP general manager job announcement was circulated, which led to the recruitment of Carlos Sanabria.

Sanabria assumed the role in April 2020, in the midst of the pandemic, giving the IXP visibility and leadership at a crucial moment—just as traffic was hitting peaks of 25-75% more than usual.

Having worked for 10 years with the state regulator, Sanabria understands the government and the operator point-of-view, which has been an advantage for bringing them closer together.

“This is something that gets constructed,” he says. “Entel has 5.1 million connections, compared to a few small operators that have 1,000 connections. But you have to sit them all down at the table and ensure that none of them feel bigger than the other.”

In this process, Veliz believes that Sanabria “is a skilled diplomat and his approach is perceived to be the right one, the one that the country and all interested parties need today.”

Two new small operators have already joined under Sanabria’s leadership—Megalink in 2020 and Sirio Telecomunicaciones in 2021. But Sanabria isn’t satisfied with having 13 members. He aspires to integrate each of the 51 autonomous systems (networks with their own Internet presence) that exist in the country.

He also recognizes the need for the largest operators to be there.

He says the IXP will become even more relevant as it grows and as members themselves keep investing.

“We are already more advanced and more mature,” says Solis. “With the help of the Internet Society, other doors—on educational content creation and reporting systems—also started opening.”

Under its new leader and members, the IXP’s traffic started doubling, reaching peaks of almost 3 Gbps in July 2020.

Routing Improvements

On 1 October 2020, changes were implemented that made the root server, a critical Internet service responsible for the functioning of the Domain Name System (DNS), available to everyone. The IXP’s board of directors authorized the publication of all the prefixes of Bolivian autonomous systems directly connected to the Internet. (Once autonomous system numbers become public, it facilitates direct peering with IXPs and help operators optimize network management and have better control of their traffic.) This helped their clients or ISPs remotely connect. Likewise, the root server installed at the COMTECO Cooperative became accessible. With this, the DNS became available to all ISPs, including those that weren’t members of the IXP.

With the opening of these prefixes, the local traffic exchanged increased, as can be seen in this graph. (Photo courtesy of PIT Bolivia)

With these changes, within a month the amount of local traffic exchanged doubled again, reaching peaks of almost 6 Gbps.

PIT also made crucial security enhancements. In early 2021, the Internet Society provided support for the IXP members to adhere to its Mutually Agreed Standards for Routing Security (MANRS) global initiative. This initiative provides crucial fixes to reduce the most common routing threats. According to Sanabria, 80% of his members have already generated and signed Route Origin Authorizations (ROA), enabling them to join the MANRS network operator program.

This has also allowed Bolivia’s IXP to join LAC-IX, the organization that brings together IXPs from across Latin America and the Caribbean. With the support of Internet Society, LAC-IX, and the Latin America and Caribbean Network Information Centre (LACNIC), PIT Bolivia also installed a route monitoring system.

LAC-IX is the Association of Internet Exchange Points Operators in Latin America and Caribbean. It promotes the exchange of Internet traffic within the region, contributing to the development of IXPs through the implementation of best practices, new tools, and services to improve the critical infrastructure of the Internet.

Donating New Switches

In January 2021, the Internet Society donated two switches. These increased the IXP’s automation and doubled its capacity, expanding it from 500 Gb before the pandemic to a national capacity of close to one terabyte of traffic.

“The Internet Society has supported PIT Bolivia from the beginning,” says Solis. “It is so important to have Internet Society’s global vision[…] When we lack equipment, they send us equipment. They have given us route servers, and supported the IXP’s ROA certification. This support has helped give the IXP a face, give it an image and [helped] interconnect it with other IXPs in the region.”

Access to Educational Content and Improvements in Measurement and Quality

On 31 December 2020, the government of Bolivia introduced regulation to ensure free access to the Ministry of Education through mobile devices. This included its platforms offering classes and educational content for public schools.

“Prior to the implementation of this… we held meetings with the ATT, the Ministry of Education, and Internet providers […] our big concern was that traffic would overflow, with 3,000 simultaneous classrooms, because we had to ensure access to content for two million public school students,” says Sanabria.

As a result, the IXP made another series of improvements in April 2021, increasing its capacity, improving its automation, and allowing its traffic to be better visualized.

“We saw that the previous infrastructure was not going to work for us, so we agreed that the Internet Society would give us a switch and a server. The new equipment began operating on 7 April 2021. With the previous infrastructure, we also couldn’t fully measure the traffic because there were ports that didn’t chart correctly. When we made the change, our traffic measurement also improved.”

Currently almost all local traffic is passing through the IXP, which Sanabria says wasn’t the case before. This has decreased latency—what the user perceives as the delay in downloading content. Before, a download took 600–700 milliseconds on average. It’s now been reduced to 11 milliseconds.

According to Solis, customer service has improved, as has the resilience of the Internet in Bolivia, by eliminating dependence on a single provider.

Another important milestone for the IXP is on the horizon, as it expects to incorporate its first content delivery network (CDN) in the coming months. According to calculations, this CDN would help increase the IXP’s traffic once again, from 13 to 23 Gbps.

Much of Bolivia’s total Internet traffic, currently around 400 Gb, is generated by these large content providers, so Sanabria is focusing on attracting more CDNs in order to keep more traffic local.

“When we bring in more CDNs, traffic will get cheaper,” says Sanabria. He says some smaller providers are currently buying Internet for US$50/Mb, but the IXP will charge only US$4/Mb.

Veliz already sees an improvement with the cloud and cache that were integrated. “We’ve advanced. I see a difference. It is helpful. But it is still insufficient because Bolivia has a terrible depletion problem. We need to measure the quality of Internet service for users at a neutral point and for large CDNs to have a reference point to route their traffic to Bolivia in a more optimal way, for example by peering with Bolivian ASNs through the IXP.”