VoIP/RTP Traffic Shaping or Congestion Management?

Hello,

This is a recent problem (last week or two), but I am seeing it with multiple Comcast circuits at this time.   I tried to open a trouble ticket, but the technician was Tier I, tested ping, and said everything was fine.  I think I'll need a higher Tier tech to work on this issue.  

I have a few VoIP Phones connected to a Comcast Circuit in our offices in Utah.  (Same thing on another Comcast circuit too).

The phones register with a server on a Level 3 Internet connection in Tennessee.

SIP Traffic (not RTP) seems to be flowing just fine in both directions.   (SIP on Ports 5060 to 5070)

RTP Traffic is flowing fine from Comcast circuit to Level 3.  No packet loss.

RTP Traffic coming from the VoIP Server to offices in Utah (Comcast Download) - We are getting around 2.5% packet loss.

    Note:  UDP Source Port in TN was around 13000    UDP Destination Port (Comcast in UT) around 57000

The service seems to work fine in the morning, and then degrades as the day moves on.

Now the Kicker...   If I VPN connect from the Comcast site to another networkso the SIP and RTP traffic is encapsulated in the VPN tunnel and effectively Comcast doens't know the traffic inside is RTP, then I get 0% packet loss in both directions and the quality is great.

I have verified we don't have traffic bursting on either end which would cause packet loss.  Since I am seeing this on 2 separate Comcast circuits in different parts of Salt Lake City, it isn't the "local loop" portion of the Comcast network.

Since we don't have this problem with sites (phone sites) that use ISPs other than Comcast which connect to the Level 3 circuit in Tennessee and these servers, we can verify it isn't the Level 3 circuit or the servers.

Based on my troubleshooting:   I believe the issue is either in the Level 3 to Comcast connection, or along the Comcast network between Tennessee and Salt Lake City.   It really feels like some form of traffic shaping or congestion management is happening on the circuit targeting this RTP traffic.

Who can I work with on this issue?

I will appreciate any assistance or ideas.

-gnieboer