Cisco Modem Losing Upstream Lock

[JimD]

We upgraded our service back in July and that upgrade required our modem to be replaced. Now, I've been battling this for 2 months now....

Every day the modem loses it's upstream bond and eventually restarts on it's own to start the ranging process again. This is not acceptable, especially when we are trying to process a credit card transaction and the customer must wait up to 10 minutes while the modem reconnects.

The service techs have replaced the cable coming to our building as well as the modem itself, but it still continues.

If there is some way this could be escalated to be traced at the head end or farther down the line, it would be appreciated.

Log of outages:

Monday August 3rd - 1:20pm
Tuesday August 4th - 1:30pm
Thursday August 6th - 1:51pm
Thursday August 6th - 6:30pm Reboot for unknown reason
Monday August 10th - 12:33pm
Tuesday August 11th - 6:39pm
Thusday August 13th - 8:17pm
Friday August 14th - 2:18pm

============================
Saturday August 15th - Service tech
Checked cable by modem
Installed filter on TV cable side
Said to disconnect TV cable if still happening
Said now on device watch
============================

Monday August 17 - 3:14pm
Tuesday August 18 - 5:11pm
Wednesday August 19 - 1:01pm (removed TV cable)
Monday August 24 - 1:07pm
Tuesday August 25 - 1:11pm (removed splitter - cable now direct to outside)
Wednesday August 26 - 1:30pm
Wednesday August 26 – 2:51pm
Thursday August 27 - 1:43pm (called - recording said work completed recently - restarted modem[unplugged/plugged back in] as instructed)
Friday August 28 - 1:50pm (called - agent rebooted remotely - told there are a lot of outages in area)
Saturday August 29 - 2:19pm (called - 25 to 30 minute wait - hung up)
Sunday August 30 - 10:45am
Monday August 31 - 4:55pm (called - setup service call)
Tuesday September 1 - 5:00pm

============================
Wednesday September 2nd - Service tech
Checked cable from outside and by modem
Said replaced ground block outside
Replaced modem
============================

Thursday September 3rd - 3:55pm to 3:56pm
Friday September 4th - 3:42pm
Monday September 7th - 7:06pm
Tuesday September 8th - 1:07pm
Wednesday September 9th - 1:11pm
Thursday September 10th - 1:16pm

============================
Thursday September 10th - Called and escalated

Friday September 11th - Service techs (2)
Gave screenshots of problem to tech
Replaced cable from pole
============================

Saturday September 12th - 4:30pm (upstream lock lost)
Monday September 14th - 4:35pm (upstream lock lost)

[Comcast_Gina]

I'm really glad you let us know about this, we'd be glad to work on figuring this out for you. I know how vital credit transactions are right now. Please click on my handle (Comcast_[Gina]) and send a private message with your name, the business name, the complete service address (including city, state, ZIP, suite number, etc), and the phone or account number, and any other pertinent details you haven't already included above, so we can help. 

[JimD]

Requested info sent.

[Comcast_Joseph]

I appreciate your detailed reply, Jim. I apologize for the connection lose but this is always a great place to talk and get support as we are a dedicated team that is here to help resolve any concern. I promise I will take the right actions to help. I can imagine how inconvenient this is for your business and all users that are trying to get online. My diagnostic check does not show any timeouts, but I can get an appointment scheduled so we can refer work to maintenance if needed. I was wonderland if you have any devices losing connection that connect directly to the modem.

[JimD]

All equipment is hard wired and I don't lose connection to the modem. Normally when it happens LAN remains connected, WAN is what dies.

The only time I lose my LAN connection is if the modem does a complete reboot. But that's expected.

Which you can add to the list of when it's gone down:

Tuesday September 15th - 1:15pm (modem reboot on it's own)

Times I use in my posts are Eastern time.

[Comcast_Gina]

You're awesome, Jim; thank you for so much information! I'll have to ask some questions to make sure that we can schedule a tech. 

• s anyone in this home/business under quarantine for COVID-19?

  YesNo
• Is anyone experiencing flu-like symptoms?

  YesNo
• Has anyone in this home/business been exposed to COVID-19 over the past 14 days?

  YesNo
• Is there any reason you won’t be able to minimize staffing in the area and ensure our technician will have enough space to install services and be able to maintain a 10 foot social distancing guidance?

  YesNo
• Does your business currently support any healthcare activity regarding COVID-19? ( i.e. – doctor’s office, nursing homes, emergency rooms, hospitals, testing sites)

  YesNo

Please let us know! And I really appreciate all the data you've provided so far, it's really helpful to us! 

[JimD]

No to all the questions.

[JimD]

The same thing just happened at 1:32pm eastern..

[Comcast_Robert]

Hi there! I wanted to follow up with to see if you still needed assistance? 

[JimD]

There are several people that have spoken to me in private messages. So, probably not.

But I will continue to post the outages as they happen until they stop happening.

It happened again today at 1:37pm Eastern:

[JimD]

But I will say that this is getting extremely frustrating. This has been going on for 2 months now with no resolution.

[JimD]

Informational for those working on problem: The problem continues...

Monday Sept. 21st 1:41pm Eastern

[Comcast_Gabe]

Hi, there! Thank you for your patience and for sending over those details. I have reviewed the area and I am still seeing that we have a team working to resolve a technical issue affecting your area. I will continue to monitor this issue for another 24 hours and follow up with you to provide an update and to ensure we get this fixed. In the meantime, please feel free to reach out for any additional questions or concerns.

[JimD]

Informational for those working on problem: The problem continues...

Tuesday Sept. 22nd 1:45pm Eastern

[Comcast_Gabe]

Thank you for providing these details and for your patience. In reviewing the account, I am seeing that we still have our awesome field team working diligently to resolve the technical issue affecting your location. I’ll continue to monitor this issue on my end and reach back out to you within 24 hours to provide an update and to ensure a solution. Please feel free to reach out for any additional questions or concerns in the meantime. We are here to help 24/7.

[JimD]

Informational for those working on problem: The problem continues...

Wednesday September 23rd - 1:49pm

[JimD]

Informational for those working on problem: The problem continues...

Thursday September 24th - 9:00am

This time I didn't catch it in time to see if uplink lock was lost. But the modem did go through a US/DS lock sequence (US/DS light blinking).

[SprinklrBiz1]

Happy Friday! Thank you for your continued patience while we work to get this resolved. I just wanted to follow up as promised our amazing local team advised this they have made updates and now showing single levels all now in spec and speeds are good. Can you confirm if you are seeing any improvement on your end? 

[JimD]

Sorry, no I can not.

It happened again today:

Friday September 25th - 3:09pm

[SprinklrBiz1]

Hey there, thank you so much for the updated information and we have reported it back to our dedicated team. I will continue to monitor all communication with them, and reach out as soon as we have an update. I appreciate your continued patience with us!

[JimD]

Informational for those working on problem: The problem continues...

Saturday September 26th - 3:12pm

I don't know if this helps or not, but shortly after the modem obtains a lock again this is logged in the event logs:

[Dhcpc][21584]: erouter0 got new IP <IP REMOVED FOR PRIVACY>

[JimD]

Informational for those working on problem: The problem continues...

Monday September 28th - 3:18pm

Obviously, the modem goes into a ranging mode:

This really needs to be fixed. When this happens all access to the internet, both inbound and outbound is interrupted.

[JimD]

Informational for those working on problem: The problem continues...

Tuesday September 29th - 3:21pm

[JimD]

Informational for those working on problem: The problem continues...

Wednesday September 30th - 8:06am Eastern

The modem reboot on it's own. Therefore I don't have any screenshots because LAN was lost during the reboot.

[JimD]

Informational for those working on problem: The problem continues...

The problem continues....

Wednesday September 30th 10:48pm Lost internet

I can only assume the upstream lock was lost because I was asleep at the time it happened. I have one of my servers pinging Google once every minute and logging failures. This was the log from last night (times are Eastern)...

Wednesday 30th of September 2020 10:48:06 PM - FAILED TO CONNECT
Wednesday 30th of September 2020 10:49:07 PM - FAILED TO CONNECT
Wednesday 30th of September 2020 10:50:07 PM - FAILED TO CONNECT
Wednesday 30th of September 2020 10:51:06 PM - FAILED TO CONNECT
Wednesday 30th of September 2020 10:52:07 PM - FAILED TO CONNECT
Wednesday 30th of September 2020 10:53:06 PM - FAILED TO CONNECT
Wednesday 30th of September 2020 10:54:07 PM - FAILED TO CONNECT
Wednesday 30th of September 2020 10:55:06 PM - FAILED TO CONNECT

[JimD]

If the past pattern holds true, the next outage will be between 10:48pm and 11:00pm Eastern tonight.

[Comcast_Gina]

Good afternoon, thanks so much for sending these details over! We had a tech out to monitor the area and connection yesterday. During that time, they were unable to locate any issues as everything was passing. New equipment at the tap was installed in order to ensure it is good to go there. Could you tell me how everything is running as of this morning?

[JimD]

Informational for those working on problem: The problem continues...

Thursday October 1st - 4:52pm

[JimD]

I'm pretty sure everyone can agree the problem is most likely NOT in my installation.

So that leaves the cable farther down stream from me and the head end. I can only suggest that the cable be inspected between the head end and my location. And maybe even the head end itself.

I can't say for sure that other people aren't experiencing the same problem and not realizing it or even seeing it and accepting the problem. I see it because I have servers running and I am constantly monitoring critical live stream data as part of my work. And when the internet goes down, that stream stops.

Then there is also the problem of our store inventory/sales system going down for 10 minutes each day. That may not seem like a lot, but if a customer is attempting to pay for a purchase, they have to wait until the system comes back online. And that is not always an option for our customers.

[JimD]

Informational for those working on problem: The problem continues...

Friday October 2nd - 4:56pm Eastern

[JimD]

Informational for those working on problem: The problem continues...

The problem continues....

Saturday October 3rd - 5:00pm

I can only assume the upstream lock was lost because I was not in the office when the outage happened. I have one of my servers pinging Google once every minute and logging failures. This was the log from yesterday (times are Eastern)...

Saturday 3rd of October 2020 05:00:06 PM - FAILED TO CONNECT
Saturday 3rd of October 2020 05:01:07 PM - FAILED TO CONNECT
Saturday 3rd of October 2020 05:02:06 PM - FAILED TO CONNECT
Saturday 3rd of October 2020 05:03:07 PM - FAILED TO CONNECT
Saturday 3rd of October 2020 05:04:07 PM - FAILED TO CONNECT
Saturday 3rd of October 2020 05:05:07 PM - FAILED TO CONNECT
Saturday 3rd of October 2020 05:06:07 PM - FAILED TO CONNECT
Saturday 3rd of October 2020 05:16:06 PM - FAILED TO CONNECT

[JimD]

I don't know if any changes took place between Saturday Oct. 3rd and Sunday Oct. 4th, but no outage was experienced on Sunday Oct. 4th.

I hope that continues.

[JimD]

Informational for those working on problem: Unfortunately, the problem continues...

Monday October 5th - 5:04pm

[JimD]

This is an update to the problems I was experiencing.

Last week a tech came out and after going through what I was seeing and my pointing out what was happening in the modem's control panel he was as surprised and stumped as I was about what was happening. He ended up spending about 2 hours here and making a lot of phone calls to people in various other departments. The conclusion was that the Comcast WAN IP the modem was receiving might be being used somewhere else, thus booting the modem offline and causing it to re-range/reconnect. The only way to test that theory was to replace the modem again with an older Netgear modem. Which he did. And he said if this works, then that definitely points to a problem with the Cisco modem(s) that were here.

And now after a week, I have not experienced any problems with my connection. So, I can only assume there is a definite problem with the Cisco modems that were installed here. I can't say if it's what we discussed (WAN IP duplication), or some other problem. I just know that my connection has been good for a week now with the Netgear modem.

This matter can be considered closed.