TCP 3 way handshakes taking forever intermittently on 50 x 10 SMC static IP service

Hi Forums-

We are in the Devens, MA area on a recently new plant expansion with 50 down x 10 up business service w/ 8 address static IP on an SMC modem firmware 1.4.0.49.7-CCR hardware version 1.01.

We regularly can get the advertised speeds for long lived TCP connections where we download up to 50 megabits -- speedtests with speakeasy or speedtest.net are usually no problems shown. We can also easily push our full 10 Mbps upstream.

Our issue is with TCP 3-way handshakes just taking several seconds to complete and a major burst of traffic happening when it finally does complete making short bursts of web browsing unusable at intermittent times. I looked in wireshark and saw SYN packets heading to a web server where it gets deep into the exponential backoff -- about 5 SYN packets go out in a pattern of initial SYN packet, 1 second wait, retrans a SYN packet, 3 seconds, send a SYN packet, 5 seconds etc. until 20 seconds total goes by I finally see a SYN w/ ACK set and ACK completing the connection and a bunch of data and acks goes flying by like it should have 20 seconds earlier!!!

I've been using iperf on linux on a couple of servers I have access to in Cambridge and Chicago in both TCP and UDP mode and haven't been able to replicate the issue easily -- a lot of the time data just goes through w/ no TCP startup latency and no evidence of packet loss (all done on TCP and UDP port 5001).

I wiresharked a 30 megabyte https transfer from the Cambridge connection to see the speed ramp to full 23 meg upstream capability there to my connection here in Devens and only saw a couple of fast retransmits with a large TCP window (250 Kbyte window or so) allowing the speedy transfer. If the SYN packet losses we see on port 80 connections was due to general packet loss, I would suspect I would see poor long lived transfers too -- our people using the Comcast connection here have no issues once past the TCP SYN backoff problem!

I know it's not my firewalls because we have a backup 3 x T1 connection from Verizon (4.5 Mbps symmetrical) and we have no problems like this on that setup -- just we can saturate the bandwidth on those T1's a lot easier than the 50 Meg Comcast connection (which is why we signed up for Comcast business for our primary connection)

Further evidence it's not our firewalls is we can plug a computer into a 2nd LAN port on the SMC modem and use another one of our 5 usable static IPs and still have the "initial connection" problem -- a web browser just says "connecting to xyz.com..." for many many seconds, refreshing sometimes is super fast... it is very hit or miss.

I think there is some kind of connection based traffic management on our business connection that is just failing to do things right! I'm trying to spend more time w/ UDP on iperf w/ the servers I have access to to try and verify it's just TCP problem versus UDP.... or maybe even only TCP port 80 related issues!

I've also logged into the SMC modem to see 4 x Downstreams w/ Snr of 37 dB on each channel, 11.5 dbm Receive power on each channel and about 30 dBmv transmit power on 2 x Upstreams (I understand this needs to get up into the 40's to be considered too high)

The modem has had 4 days of uptime, so I know it's not rebooting.

I don't know if it's re-syncing -- I doubt it.

We have 70 people working in the building here and we regularly use internet based services like salesforce and people are extremely frustrated by these connection "lag" issues -- Are there other people on the same plant as us with similar problems? Should I call support and try to explain all of this on the phone -- I don't want to be on the phone with someone for hours on end and would prefer to do it via forum, e-mail, or online chat.

Thank you.

Responses

Accepted Solution

malk315

Visitor

•

5 Messages

12 years ago

I think we nailed this on our own!

After examining our SMC buisness gateway (setup for static IP) we noticed this option:

"Disable Gateway Smart Packet Detection"

The above option was NOT checked, so we googled for "SMC Disable Gateway Smart Packet Detection"

The first result was this post:

http://ckdake.com/content/2008/disable-gateway-smart-packet-detection.html

The above describes exactly the problem we've been experiencing -- right down to the author of the above article complaining about what seems to be TCP SYN packets getting dropped so sessions don't setup. After turning off "Disable Gateway Smart Packet Detection" on our modem, it has so far made a HUGE difference. Hopefully this will continue to be the resolution to our problem.

We've still got a tech coming here tomorrow to look at signal levels -- it would be best for them to be optimal, but right now I'm quite convinced this option in the modem tries to do some kind of packet accounting to help w/ DOS attacks or something, but whatever it does it makes our connection turn to crap. We have our own firewall that does fine.

Hope this helps anyone else who might be having this problem! I think we're going to be a lot happier.

0

malk315

Visitor

•

5 Messages

12 years ago

I wanted to make one more point to prove it's not our equipment:

I've got a Soekris box running a linux based firewall going right into the cable modem. I've got simple shell access to it.

To see the SYN packets going unanswered 5 in a row over a 20 second period, I used tcpdump right on the eth0 interface which goes right into the cable modem. We see this issue with several sites -- not just the one example. I then use any browser from any box behind the firewall and collect the evidence w/ tcpdump and then look at it in wireshark later or even live in the tcpdump output.

We'll have a period of horrid web surfing to various sites (google is a good one since it's such a simple front page -- it will take several seconds to show the page due to deep TCP initial connection backoff) for about 15 to 30 seconds, and then it will clear up and pages load more or less instantly for a while... then it goes lousey again and it does that all day all the time.

If I switch people over to our 2nd connection on 3 x T1's I have no problems -- sites are fast all the time.

I can even switch my entire office over to the 3 x T1's and it's lightning fast ... the reason we can't use the T1's all the time is for the times when a few people start transferring a lot of data and we easily saturate the limited bandwidth of the T1's. I could easily shape long lived connections on the T1's to keep bursty traffic doing well but we decided to get the Comcast connection instead and use the bigger bandwidth. So far our experience has been terrible...

Comcast_John -- can you please respond and let me know if at least the stats for the modem that you can see at the headend has anything alarming going on?

I did read that for a Docsis 3 modem (which we have) the valid reliable range for downstream receive power is -5 to +5 as opposed to older Docsis 2 and Docsis 1.1 setups where -15 to +15 was the range. Our modem sees receive power on the 4 downstream channel's it's locked on to at +11 Db which is more than double +5. Is this a problem?

When the installer was here, they changed the 2 port tap at our demarc to a 4 port tap to deliberately lose some signal since the installer had noted the receive power was really hot coming in from the new section of plant that went in enabling us to get service to begin with.

0

CC_John

Retired Employee

•

1.9K Messages

12 years ago

Hi Malk315. We checked the line and the levels are still out of spec at at the SMC gateway. We can schedule a service call for you today in the afternoon timeframe.

0

malk315

Visitor

•

5 Messages

12 years ago

Ok -- Thanks. They have opened Ticket # is CR312322352 and will be here tomorrow morning 9:00 - 11:00 to check it out.

It would be nice if getting the signal levels into spec makes this issue go away... it's been a pretty poor experience for our approximately 70 folks in this building... usable, but not good unfortunately.

0

malk315

Visitor

•

5 Messages

12 years ago

Got a window from 9:00-11:00 this morning from a late afternoon call yesterday and tech was here at 10:00 a.m.

He checked signals at the tap and a plant maintenance guy will be out at the amp (or node if it's fiber to coax on the pole) where to adjust the power levels.

Regardless of the power levels, Turning off smart gateway has made a world of difference as reported by our 70 or so employees using the service. A lot of happier people. Tech said the level adjustment should get done today the guy doing it was in by 10:30 EST and he said it shouldn't affect us -- even if it requires a modem reboot that's fine. Here's a screenshot of the current levels:

I'm going to mark disabling smart packet detection on the modem as the solution.

Will check to see how the signal balancing goes towards end of day.

I think it would be worth while for the smart gateway thing to be shared with eveyone possible in support at Comcast as it seems this has been a common issue since 2008 according to the web link in my previous post. Anyone with an SMC gateway should watch out for this!

I'm pretty impressed with the support response from finally calling yesterday at 3:00 p.m. EST -- we missed one of 5 slots for 3:00-5:00 p.m. truck roll and got someone here by 10:00 the next morning isn't too bad -- considering the windows were 4+ hours in the past.

Hopefully service will be good for a while. If it proves reliable we will look into migrating our phone system over and keep just a single T1 for emergency backup.

0