Problems after DPC3939B firmware upgrade

Hello lindend and welcome,

Please see my questions/comments below.  Thanks.

1. IPv6 DHCP was auto-enabled (was disabled previously) and this broken most of our clients.  Fortunately, this was easily fixable.

Comcast Business class modems are now being setup with a Dual stack IPV6 and IPV4 functional inplementation in an effort for customers to now begin to get their networks prepared for the near future IPV6 migration. The Dual stack supports gradual migration of of customers end points, networks and applications. Could you please share wityh us what you mean by "  Fortunately, this was easily fixable. " ? 

2. The modem periodically times out reaching external destinations such as google and had to be rebooted/power cycled.

I recommend that you contact Comcast 800391300 and use the technical options, provide the technical agent your account number, the ask the technical agent to confirm the follow:
a.) make absolutely sure that your static IP WAN interface has its DNS 75.75.75.75 and 75.75.76.76 address programmed.
b.) make abolutely sure that your LAN DHCP server Lease Time = forever, this insures that your dynamic IP addresses will not break their internet until the device is shutdown, etc.
c.) Also make sure that the LAN DCHP Server has its DNS 75.75.75.75 and 75.75.76.76 address programmed. This will enable any connecting wired or wireless devices performance to be maximized across the board.  

#2 happened twice while we were streaming our services (two consecutive weeks) so its a really big problem that we've got to get a solution for.

Could you provide us a little more detail regarding what means by which you are streaming your services and how your parisioners are able to connect to the services? For example, are you using a video camera connected to your server and then having your parisioners connect to your server ?  Is your server running with a static IP?  Do your parisioners connect both wired and wireless?  

I did notice that running a test from speedtest.net seemed to bring the modem to life twice while in this state (once was temporary and the modem had to be power cycled).

Could you connect a laptop or computer directly to one of the DPC LanPorts 1-4 and run the speedtest.comcast.net and let us know what your 3 time average is for download and upload Mbps speed?

Look forward to hearing from you and hope this helps you out. 

One other thing.  The timeouts aren't due to DNS.

I have an infinite ping going (ping -t) to multiple sites

google.com

10.1.10.1 

both had resolved to physical addresses, were pinging fine and both were timing out.  PCs were ok because they could ping all devices on the network and when the modem rebooted the timeouts went away.

---

 

Comcast Business class modems are now being setup with a Dual stack IPV6 and IPV4 functional inplementation in an effort for customers to now begin to get their networks prepared for the near future IPV6 migration. The Dual stack supports gradual migration of of customers end points, networks and applications. Could you please share wityh us what you mean by "  Fortunately, this was easily fixable. " ? 

 

The immediate symptoms were:

 

1. Win7 client (connected directly to the modem port #3) could only ping ipv6 sites and couldn't reach any ipv4 sites

2. Devices hooked up to a router (connected directly to the modem port #2) were no longer reachable externally from Teamviewer and could not ping any ipv4 sites.

3. Devices hooked up to wifi (we have our own separate Wifi router) were unable to ping any ipv4 sites.

 

#1 was solved by allocating it a static IPv4 address (this was a temporary setup for a special event we were streaming for a week).

 

#2 was solved by disabling the router getting a IPv6 DHCP address and using the previously allocated IPv4 address that was allocated to it.

 

 

2. The modem periodically times out reaching external destinations such as google and had to be rebooted/power cycled.

 

I recommend that you contact Comcast 800391300 and use the technical options, provide the technical agent your account number, the ask the technical agent to confirm the follow:
a.) make absolutely sure that your static IP WAN interface has its DNS 75.75.75.75 and 75.75.76.76 address programmed.

 

It does.  However, we use OpenDNS on our clients (and OpenDNS wasn't down).

b.) make abolutely sure that your LAN DHCP server Lease Time = forever, this insures that your dynamic IP addresses will not break their internet until the device is shutdown, etc.

 

It was set for a week (but I think this is the default and we didn't change it).  I don't see this being the cause because the router has a reserved DHCP address allocated to it.

 

c.) Also make sure that the LAN DCHP Server has its DNS 75.75.75.75 and 75.75.76.76 address programmed. This will enable any connecting wired or wireless devices performance to be maximized across the board.  

 

It does (this is unchanged).

 

#2 happened twice while we were streaming our services (two consecutive weeks) so its a really big problem that we've got to get a solution for.

 

Could you provide us a little more detail regarding what means by which you are streaming your services and how your parisioners are able to connect to the services?

 

We have two streaming services (simulcast for backup purposes):

 

1. 3angelslive.com

2. Youtube (this we started after the issues last week).

 

They watch by visiting one of the sites above.  We use wirecast to capture video (over HDMI) from an ATEM TV Studio and upload via a CAT 6 wired connection.  The 3angelslive.com stream is SD (around 500kbps) and the youtube stream is HD (~2500kbps) well within the 20mbps we're allocated (and confirmed that speedtest typically shows 20-22mbps upload capacity for our connection.

 

 

For example, are you using a video camera connected to your server and then having your parisioners connect to your server ?  Is your server running with a static IP?  Do your parisioners connect both wired and wireless?  

 

That's really up the the viewers, but they are off site.  If I had to guess, we're probably 60/40 fixed/wireless viewers

 

 

Could you connect a laptop or computer directly to one of the DPC LanPorts 1-4 and run the speedtest.comcast.net and let us know what your 3 time average is for download and upload Mbps speed?

 

115mbps down/22-23 up which is essentially what I always see when I run the test.

 

Right now, we've disabled the wifi on the modem in an effort lock down the traffic on the modem so we know exactly what is going on, but the xfinity wifi hotspot is active and I can't disable it (I sent Comcast_John a PM and hopefully he can address on Monday).

 

I definitely see there are xfinity clients on there because there MAC addresses from WiFi devices I don't recognize.  No idea what they're doing or how much they're transmitting.  Its on a different subnet so its tough to snoop w/wireshark.

 

 

Look forward to hearing from you and hope this helps you out. 

 

Thanks for the quick response.

 

 

---

We've narrowed the problem down a bit.

Simulataneously with the firmware upgrade, our intranet server went down.  As a stopgap, we enabled DHCP on the modem (previously it was disabled).  After three weeks of this issue, we noticed that the problems seem to occur right before the sermon when the most people entered the building and when people were leaving (right around 60 DHCP devices were on the network).

This lead to the hypothesis that the modem was struggling with the DHCP leases.  Today, we completely disabled our guest wifi and streamed in SD and HD without incident.

Still a bit more work to do to confirm/deny this hypothesis, but are there any known issues either with a large number of DHCP leases or a burst of DHCP activity?

Interestingly enough, ping times vary (when going through a router to the modem) between 1-11ms but average 3-7ms.  When doing a speed test though, pings are around 9ms so I didn't think much of it.  But now that you raise the issue, I did think it odd that pings to the router were always < 1ms and the modem was so high.

I've not run a test with a direct connection to the modem but we can do so and get some results.  Thanks for the pointer.

also high utilization might also bring your connection to its knees, especialy upload. 

I'm pretty sure we had plenty of upload capacity during the problematic periodic, but I wasn't able to do an exhaustive test.

I'll follow up via email.

BTW, what type of hardware problem would cause this issue?

Following my companies recent problems very simalar to yours I would plug directly into the modem with DHCP enabled. once a address is allocated, ping your default gateway (most likely 10.1.10.1) if your ping responce is normaly above 3ms you may have a hardware problem. the respoce time should actualy be 0-1ms. disregard your first ping or 2 as sometimes there is a lag in finding the route and ping may show high. do a ping 10.1.10.1 -t to continuously ping the gateway. you shoukld get a stable ping of around 1ms

high pings seems to go hand in hand with another problem that can't be tested without specific hardware. The modems with high ping do fine with low throughput. however when packet count increases they start dropping packets and pings increase dramaticly. 

you said that the symptopm happened with high DHCP utilazation. that should have no impact on performance. but packet count would. also high utilization might also bring your connection to its knees, especialy upload. 

please contact me directly at my user name at gmail.com and we can discuse a couple quick tests and maybe some solutions to your issus.

In my opinion the additional CPU demands of the IPv6 stack is overrunning the CPU on that modem.

These modems are built with the cheapest and slowest CPUs the makers can find.  Cisco in particular is an expert at doing this.  Almost certainly the original design of the DPC3939B was vetted with IPv4 only.  Comcast came along later and told Cisco they would stop purchasing the devices unless Cisco provided dual-stack for them.  So Cisco quickly slapped together IPv6 code for the device and gave it to Comcast - who is now being screwed over by this because the additional CPU load of the IPv6 stack is causing the CPU of the device to go to 100% when the additional load of even a reasonable amount of use (as your describing) is put on the device.

If it was me running Comcast I'd stop buying devices from Cisco and I'd tell Cisco when you redesign your device to work PROPERLY on dual stack mode and come out with a NEW hardware rev, then we will look at again - and then I'd put an exception in my IPv6 policy that would disable availability of IPv6 for the DPC3939b and if a customer with one wanted IPv6 I'd replace the modem with an SMC or other modem and tell the customer to go buy their own wifi access points.

Unfortunately this technically logical solution wouldn't be popular with Comcast's customers, their modem vendors, or their accountants - so you can bet it won't be taken.