Verizon Business Transit Congestion

Hi -

I'm a Comcast Business customer in Seattle, WA.  Over the past couple of weeks I've noticed that my connectivity with addresses that take Comcast' transit through Verizon Business get latent and lossy in the evening hours between 16:00 and 20:00 PT.  This only affects a subset of addresses, but one happens to be a server of mine hosted by Internap in NYC that connect to very frequently.

Here's a link to a SmokePing graph showing the latency & loss over two evenings:

http://www.prolixium.com/share/smokeping-comcast-internap.png

Here's a traceroute from Comcast -> Internap:

(destiny:19:18)% mtr --report --report-wide --report-cycles=32 69.9.189.182
Start: Wed May 14 19:19:04 2014
HOST: destiny Loss% Snt Last Avg Best Wrst StDev
1.|-- et3.starfire.prolixium.net 0.0% 32 0.3 0.3 0.2 0.4 0.0
2.|-- et0-0.e.prolixium.net 0.0% 32 0.8 0.7 0.6 0.8 0.0
3.|-- 10.1.10.1 0.0% 32 0.9 1.0 0.9 1.6 0.0
4.|-- 73.98.180.1 0.0% 32 8.5 9.6 8.5 11.2 0.5
5.|-- te-0-2-0-5-ur08.burien.wa.seattle.comcast.net 0.0% 32 9.4 10.2 8.7 12.3 0.8
6.|-- ae-21-0-ar03.seattle.wa.seattle.comcast.net 0.0% 32 9.9 30.5 9.4 64.0 18.4
7.|-- he-1-7-0-0-11-cr01.seattle.wa.ibone.comcast.net 0.0% 32 13.1 13.5 9.5 27.7 3.8
8.|-- be-10-pe03.seattle.wa.ibone.comcast.net 0.0% 32 11.4 11.3 9.9 15.7 0.9
9.|-- 173.167.57.250 9.4% 32 60.3 62.1 60.3 65.8 1.4
10.|-- 0.so-6-0-1.XL3.IAD8.ALTER.NET 18.8% 32 136.7 142.2 136.5 181.6 11.1
11.|-- TenGigE0-6-2-0.GW1.IAD8.ALTER.NET 9.4% 32 138.3 140.9 137.8 153.7 2.9
12.|-- internap-gw.customer.alter.net 9.4% 32 140.1 139.3 137.0 142.6 1.4
13.|-- border10.pc2-bbnet2.wdc002.pnap.net 9.4% 32 140.2 140.4 137.1 151.0 2.9
14.|-- inapvoxcust-1.border10.wdc002.pnap.net 6.2% 32 140.3 143.3 138.5 152.0 3.8
15.|-- 0.te1-3.tsr1.ewr4.us.voxel.net 12.5% 32 149.0 145.7 135.8 156.8 4.7
16.|-- 0.te1-1.tsr1.lga5.us.voxel.net 9.4% 32 148.7 146.6 135.1 157.4 5.8
17.|-- 0.ae57.csr2.lga6.us.voxel.net 6.2% 32 152.7 152.0 145.2 166.7 4.7
18.|-- dax.prolixium.com 6.2% 32 147.0 147.2 137.7 156.5 4.9

 Above, you can clearly see the latency & loss starting with Comcast's peering with Verizon Business (shows up as ALTER.NET in traceroutes).

Here's a traceroute in the other direction to one of my static addresses, from Internap -> Comcast:

(dax:22:19)% mtr --report --report-wide --report-cycles=32 50.248.192.201
Start: Wed May 14 22:19:13 2014
HOST: dax.prolixium.com Loss% Snt Last Avg Best Wrst StDev
1.|-- voxel.prolixium.net 0.0% 32 0.5 1.5 0.4 12.2 2.5
2.|-- 0.ae2.tsr1.lga5.us.voxel.net 0.0% 32 0.6 2.4 0.3 10.8 3.2
3.|-- border3.te8-2.inapvoxcust-8.nym008.pnap.net 0.0% 32 3.9 19.9 0.4 216.2 59.2
4.|-- core1.te6-1-bbnet1.nym007.pnap.net 0.0% 32 0.8 1.0 0.8 2.7 0.3
5.|-- ae-26.r05.nycmny01.us.bb.gin.ntt.net 0.0% 32 6.6 6.9 6.5 8.5 0.3
6.|-- 50-242-151-69-static.hfc.comcastbusiness.net 0.0% 32 29.3 25.4 13.4 55.6 7.5
7.|-- he-1-6-0-0-cr01.newyork.ny.ibone.comcast.net 0.0% 32 29.6 26.4 18.5 35.5 5.3
8.|-- he-0-11-0-0-cr01.350ecermak.il.ibone.comcast.net 0.0% 32 51.1 46.1 32.2 57.4 4.9
9.|-- he-1-13-0-0-cr01.denver.co.ibone.comcast.net 0.0% 32 71.5 67.9 59.1 76.7 4.2
10.|-- he-0-10-0-0-cr01.seattle.wa.ibone.comcast.net 3.1% 32 146.8 141.4 128.1 148.1 4.7
11.|-- 68.86.94.70 9.4% 32 141.9 145.0 132.8 225.2 19.5
12.|-- be-21-ur08.burien.wa.seattle.comcast.net 6.2% 32 143.1 140.7 127.5 148.9 4.6
13.|-- te-0-1-1-ten06.burien.wa.seattle.comcast.net 3.1% 32 139.4 139.2 131.8 149.3 4.0
14.|-- c-67-160-53-176.hsd1.wa.comcast.net 9.4% 32 147.4 148.5 135.7 159.1 5.2
15.|-- starfire.prolixium.com 3.1% 32 149.5 147.1 136.4 154.4 3.7

I've opened up a ticket with Internap and as expected they say the problem is on the Comcast side.

All of my signal levels are fine on my router and only addresses that transit Verizon Business are affected.

Anyone know if Comcast will be either 1) upgrading their transit links with Verizon Business or 2) peering with Internap?

Thanks!

- Mark