Aurich Lawson
App developer Panic Inc. knew it had a network problem when customers began complaining about trouble downloading and updating Panic apps.
“Geez, your downloads are really slow!” was the common complaint that started coming in a few months ago, Panic co-founder Cabel Sasser explained in a blog post titled, “The Mystery of the Slow Downloads.”
But once the mystery cleared up, it all made sense. Panic and its users were the innocent victims of a longstanding network interconnection battle between cable ISP Comcast and Cogent, which operates a global network that carries traffic across the Internet.
Not ancient history
Comcast/Cogent battles caused repeated problems for customers back in 2013 and 2014, as we documented in several articles at the time. For a refresher, Cogent carries Internet traffic on behalf of many businesses that need to reach the home Internet customers of residential ISPs like Comcast. Cogent exchanges traffic directly with Comcast at various data centers across the US.
The companies have long been exchanging traffic without Comcast paying Cogent or Cogent paying Comcast, to the mutual benefit of their customers. This is called “settlement-free interconnection.”
Comcast wanted Cogent to start paying for this interconnection (also known as peering), and Cogent refused. Comcast responded by delaying upgrades to the ports that allow traffic to flow swiftly between the companies, and customers suffered with poor Netflix quality and other Internet problems.
Netflix had been a Cogent customer, but it built its own network in order to cut out the middleman. Netflix also refused to pay ISPs at first, but the company ended up paying Comcast and others for interconnection starting in 2014.
Cogent continued to refuse to pay and seemed to win its battle in February 2015 when the Federal Communications Commission passed net neutrality rules and used its Title II authority to reclassify ISPs like Comcast as common carriers. The Title II rules didn’t ban interconnection payments, but the FCC set up a complaint process that would let companies like Cogent complain about unreasonable or unjust payment demands.
The companies apparently settled their differences without Cogent having to file a complaint. After the net neutrality rules took effect, Comcast and Cogent continued to exchange traffic without any payments, and each company has continued to upgrade their ports to let the traffic flow quickly.
The latest flare-up
But this week, Cogent founder and CEO Dave Schaeffer told Ars that Comcast still delays capacity upgrades, though not to the same extent as a few years ago. Comcast generally takes 90 days to add ports after Cogent alerts Comcast that ports are becoming congested, he said.
“They meet the letter of the agreement but they drag their feet,” Schaeffer said. “I think it’s fair to say they don’t like the deal. They would like to get paid.” While Comcast isn’t breaking any contract terms, Schaeffer argues that the company is not “honoring the spirit” of the settlement-free interconnection agreement.
Cogent upgrades ports within an average of eight calendar days, and it guarantees upgrades within 17 days, Schaeffer said.
Comcast could make the upgrades for Cogent “almost instantaneously if they wanted because the capacity already exists, cross-connects [between Comcast equipment and Cogent equipment] exist,” Schaeffer said. “All they literally need to do is take a port that already exists in their router and allow us to connect to it.”
But Comcast has a “vested interest in its Internet product performing sub-optimally,” because it wants customers to buy Comcast TV services that compete against online streaming, Schaeffer said.
A Comcast spokesperson didn’t dispute Schaeffer’s statement that Comcast takes 90 days to upgrade ports. Instead, Comcast told Ars that it has no contractual obligation to add new capacity for Cogent. Comcast says it adds capacity anyway in order to prevent problems for customers.
The net neutrality rules will be coming off the books as soon as the FCC can finalize the repeal it voted on in December. After that, Cogent won’t have the FCC complaint process to fall back on, and Comcast could increase the pressure on Cogent to pay for interconnection.
Panic problem
The net neutrality rules didn’t stop Panic and its customers from being victimized by the Comcast/Cogent battle—though perhaps that’s because the FCC’s current Republican leadership doesn’t want those rules on the books at all.
Panic initially didn’t know why its customers were having trouble downloading or updating the apps the company makes for iPhones, iPads, and Macs. The company also didn’t know why its own employees were struggling to access Panic systems from their homes in Portland, Oregon.
But Panic knew that its network was simple enough that the problem could be isolated:
The Panic web servers have a single connection to the Internet via Cogent. We co-locate our own servers, rather than using AWS or any other PaaS, and we also don’t currently use a CDN or any other cloud distribution platform. So, if something is making our downloads slow, it ought to be pretty easy to do some analysis and figure out why, or at least where.
Panic set up a speed test page to find out how fast customers could download from Panic’s website and how fast people could download from a “control” website hosted by Linode rather than on the Panic network. The test would also show what home Internet provider people were using; Panic tweeted out a link and started collecting results.
At 356kBps, the connection to Panic via Cogent was far slower for Comcast customers than it was for anyone else. Cox, Charter, Verizon, and other companies were providing access to Panic via Cogent at more than 5MBps:
Sasser wrote:
Well, well, well. It doesn’t take statistical genius to see one glaring outlier—and that was Comcast, with download speeds often being as low as 300 kilobytes/second. And you’ll never guess what provider is used by virtually every Panic employee when they work from home? Yeah, Comcast. There is, in fact, no other cable ISP available to Portland residents.
Panic downloads were slow, “but seemingly only to Comcast users, and only during peak Internet usage times,” he wrote.
Panic contacted Cogent, but the company said it couldn’t do anything, according to Sasser. Panic did some research and found Ars articles detailing the Comcast/Cogent battles.
“We felt certain history was repeating itself: the peering connection between Comcast and Cogent was once again saturated,” Sasser wrote. “Cogent said their hands were tied. What now?”
Panic contacted Comcast, and Comcast fixed the problem—after a while.
“[P]retty soon a call came back [from Comcast] with a definitive-sounding statement: ‘Give us one to two weeks, and if you re-run your test I think you’ll be happy with the results,'” Sasser wrote.
Two weeks later, Panic asked its users to re-run the speed tests and the results were markedly different:
There was about three weeks between Panic’s first email to Comcast and the fix, Sasser told Ars.
Comcast told Panic that it “added more capacity for Cogent traffic,” and that “Cogent made some unspecified changes to their traffic engineering,” according to Sasser’s blog.
“Here’s where I have to give Comcast credit where credit is due: they really did care about this problem, and they really did work quickly to make it go away,” Sasser wrote.
But Sasser thinks Comcast may simply be on its best behavior for now while the net neutrality debate is still raging. The net neutrality repeal is subject to pending lawsuits, Congressional legislation, and state government efforts to require net neutrality.
“If I had to guess, I’d say it’s simple: in the middle of a serious ongoing debate over net neutrality, the last thing Comcast wanted to look like was a network-throttling bad guy in this blog post,” Sasser wrote.
Be the first to comment