We are an internet based company. Our connectivity defines our success and or failure. At this time we are loosing customers again. We have had Comcast refund 6+ months of service charges this year alone. We thought you had everything fixed, and now we seen to be back to square one. We have constant lag spikes which last 10 to 60 seconds where there is NO traffic being routed through your modems. We call these outages, and we know you like to argue that they are not outages. We do not care what you want to call them, they are killing our business which is effecting our ability to take care of our employees and their families. We keep seeing that "Comcast Cares", but letters to those Supervisors and Network Engineers we have been told to work with are NEVER answered or addressed ever. And this way of dealing with your customers has gone on the same way for years. The only time we get any attention is when we start talking with our Lawyers, which we are doing so again today.
Take a look at just how wonderful Comcast Bulsines Class is... it stinks!
Solved! Go to Solution.
The link for yesterday was wrong...
This is the correct link for 10-23-12
The last four days of monitoring: http://184.108.40.206/r3/cricket/grapher?type=gif&target=%2Fping-pktloss%2F220.127.116.11&dslist=ping-...
We are again losing our larger customers due to the poor internet quality. For the last two months we have repeatedly emailed both Eric Behr and David Andersen of Comcast for their assistance in this matter. Due to our past issues, our company has been told that these two Comcast employees are our direct point of contact to resolve any future issues. Not one single email has been responded to, even the multiple calls and emails over a 2 hour outage in September. Since this time our internet has gone from 100% operational to 81%. We have had constant lag spikes where ZERO packets flow through both of my modems. These complete lag spikes will last anywhere from a few seconds to a few minutes. On average we are having between 1 to 3 lag spikes per minute after 3:30 pm until almost 9:AM daily.
Here is a printout of the quality connection we are having in the good part of the day. http://vindicator.us/images/ComcastBusiness.10-25-2012-TWG.jpg
Wow... 2 direct calls from Comcast Supervisors tonight. They even see the problem and it is in their head end... All it took was me demanding they roll a truck and I let them know I will have them start rollnig a truck every 4 hours until it is fixed.
TheWebGamer. While the outage issue was cleared last evening we are still monitoring the gateway for resets and errors. If you will send me any of the terminating IP's that you are seeing failures to we can test thru the network path to locate the source of the issue.
18.104.22.168 is directly on your SMC-D3 modem. I test only to your modem, and never to any IP on my dise of the modem.
You can watch it live as you wish. This is to the 22.214.171.124 on your modem.
Our internet is down the toilet yet again tonight. Worse then the normal poor internet we have had for the last two months.
We are unable to reach anyone in Tier 2 Corporate Escalation tonight. None of my other primary contacts are answering email or their phones. I am back to demanding that Comcast toll a truck which I know is totally useless, but it does hit them in the pocketbook. All I can do is have them roll a truck 2 to 4 times a day to get their attention. If I do not go out of my way to totally hammer them constantly, they will ignore you and hope you go away.
Sorry forgot to post the new numbers. They said they will have a truck here in 4 hours. We will see and I will post the results later tonight or tomorrow morning. 11:30 PM now, so I am looking for a 3:30Am service call.
BTW: Just for kicks.. here is a monotor on another Comcast modem in the same building.
Only slightly off topic... Why are you routing IP traffic From Sacramento, to Fresneo CA, to another router in Fresno then back to Sacramento before sending the packets down to the LA Ibone? You just doubled the latency to LA... Why?
Comcast Escalation called this morning and asked how the service call went last night. I was sorry to tell them they not shown up again, and they never called. They assured me someone will be here in 45 minutes and someone did show up. Too bad this is the time of the day when my internet is at its best (as you can see from the charts).
Last night I ran a ping test to Comcast.com to keep packets within their network. My settings were to send 32767 packets which lasted around 10 hours for the test. Packet size was 256, timeout was 500ms. 32767 packets sent, 30478 packets received at 6% lost. Round trip times min 91ms, max 125ms average was 94ms. There were no packets received between 125ms and 500ms. They just vanished.
New Ticket CR302176697
The tech pulled in a new F2 underground to the Office and put both our modems on a single dedicated cable that goes directly to the pedistal.
Several techs have told me today what they have been telling me for the last 18 months. The problem is in between the head end and my office. There are no problems at the head end, and there are none at all at my office.
We have a single direct cable from the pedistal to the modems. The issue is still out there, in the neighborhood. bad cable, faulty amp, box gone bad... Something that is feeding massive line noise into the network causing a constant 6% packet loss. A 24 hour test from my office to comcast.com (To keep it all in the Comcast Network) has shown a constant 6% average packet loss. The line noise is so bad that Comcast equipment is timing out and restarting to sync up again.
This is to inform you that every single ticket that "WE" (Comcast Tier-2 and our company) have opened for the Web Gamer has been closed and marked RESOLVED. There are no current pending tickets noting that we have any issue or problems with our network. I am calling in and placing yet another report for a new ticket right now.
There you go… check them for youself…
On the phone with Tech Suport right now. Asking for a Cr# and an ER# so I know that Comcast knows this is still an active issues and not resolved.
The tech said specifically that she could see “Signal Power issues in my up stream RX Power that was not Green, but were Yellow and Red” She said with this she could send out a truck, but she was told that it was unlikely anyone could show up before 10AM to 12:Noon tomorrow. By then the nightly line noise will be gone and they shall close the ticket and mark it resolved. Now if I keep this up every single night for the next year, I wonder if anyone at Comcast will notice a trend?
New active Tickets: CR302214916 ER012481826
Checking my Modem....
I normally see upstream power at 46 or 47 when everything is running good.
Lets look at the downstream that was not addressed by the over the phone tech support.
Looking at the downstream....
Should the downstream power be 3 point something and not -0 point something?
New active Tickets from last night: CR302214916 ER012481826
Comcast was again unsuccessful in their ability to roll a truck during the worst of our packet loss last night. Nor were they able to locate a telephone to call us as we were told they would last night. This morning, 13 hours after an escalated ticket was made, a line tech did show up and ask me what the problem was.
Please note, that at this time of the day, this is our BEST internet connection in a 24 hour period always...
So with a line tech sitting in the chair next to me, I ran the following tests and with the results posted below.
When we started running the tests the only lag we had in the last 3 hours was an hour ago...
Here are the modem power levels...
Jitter and Packet loss test
Looking at the local 4% local packet loss from the head end to my modem.
I was up again for the second night in a row till 4AM waiting for a truck or a phone call which again never happened.
Thank you again Comcast!
Called into Business Class Tech Support tonight at 11:30 PM PST. All of the above CR and ER numbers were marked closed and resolved. Wish someone would have called and told me that so I could laugh in their face.
The last 3 hours my average packet loss was 3.19% Highest was 18.81 and it is currently 2.86%
The new CR number for tomorrow is CR302272701. Tech #31333 offered me an ER number which I declined. The last two days in a row I had a nice escalated ER number and that failed to even get anything done, so why bother.
I could post images of my modem signals etc, but they were as bad as last night... so same old news. I did save them and document them in our files though.
Morning John! What is the word from the inside about our issues?
As for me, I am looking forward to another call from Britney and or Sanji from Comcast Corporate Escalations tomorrow to go over the past 3 days with them. I have kept Britney in the loop with every ticket and the responce, or lack there of.
I will post more status reports tomorrow. Right now it is 00:30 AM Pacific, My internet dropped 20-30 ms of latency instantly and I have a nice solid green line with zero packet loss. Just like someone just flips a switch and all the line noise goes away... every single night, at the same time. You can set your watch on it.
William, We are still monitoring form the gateway upstream to the node. We are testing to the node as well as running a smokeping test from DSL reports. The results are posted below.
GATEWAY TO NODE results
packet size 1024
--- 126.96.36.199 ping statistics ---
200 packets transmitted, 200 packets received, 0% packet loss
round-trip min/avg/max = 0/8/60 ms
--- 188.8.131.52 ping statistics ---
packet size 1400
500 packets transmitted, 500 packets received, 0% packet loss
round-trip min/avg/max = 0/8/80 ms
I also checked smokeping result on DSLReports
As you are aware all results are dependent on traffic volume at the time of test.
Comcast is continuing to investigate the issue of Latency from your gateway to the node.
For the past few days we have had an average of 4 to 14 people using our hosting. Normally we see over a thousand people within the 30+ virtual servers on 7 physical servers. So almost all of our customers have left. I have been told that there is nothing really more they can do to correct whatever is going on. I have totaly lost heart and faith.
Here is out traffic on 10-29 last night at prime time... No one was really even on. 4 People playing Minecraft can not cause a 50 MB down and 10MB up connection lag out.
Ok Comcast John... This one is for you. Thank you for giving me the very very close to me IP address you ran those very very short tests from during at 10:07 AM my time at which time we hav ethe very best of what Comcast can offer. You really need to read and look at the graphs to know we FAIL at night. Waving that crap test under my nose like I have no idea what I am talking about insults me and makes you look foolish.
But what is more to the point is that you failed to set the correct date after you photo shopped the immage I suspect... You posted on the 29 an image for October 31st, which was several days in the future...
Let me show you a real test using the same IP address information you so kindly pointed out. Time of al lthe tests was 10:54 PM Pacific Time. Date and time on DSLReports Smoke ping is Eastern Time and shows 00:45:49 10-31-2012
Lets start with my traffic that you are all trying to say is the problem... 4 whole users... I have had 3000 users this summer with no lag.
Lets look at the ping route that you used to test my modem... Looks like you are pinging me from Roseville Road at Elkhorn Blvd in Sacramento CA where the Head End is located?
Lets now look what DSLReports is saying from the smoke ping...
Now at the same time lets send 1000 256k packets with a 500ms timeout and see that tonight I only have 2% packet loss between here and the head end. AGAIN!!! I am complaining about my traffic lag AT NIGHT!!! AT NIGHT!!! AT NIGHT!!!
Pissing away your tech and our time in the morning out here after the problem goes away for 15 months does not solve this problem... So your worthless test at 10:AM means nothing at all. You just wasted your time just like all the other caring Comcast employees.
So there you go... We are having a good night here at only 2% packet loss... Next time to a real test that counts when it counts. And as for my bandwidth, you are monitoring it and know dam well I am not using very much at all.
Time for my morning update. Wow, was I in a mood last night or what. I did not take the news very well that there is nothing Comcast can do for me and the fact that you are trying to blame everything on my traffic...
Ok Lets start by looking at the DSL Reports for the last 3 hours and the last 30 hours on 184.108.40.206
Now lets look at a smoke ping from CA1 to the Head End at 220.127.116.11. Please note that the date/time stamps for both pings are within the same day, hour and minute of each other.
Now lets take a look at how many of my customers I had online during this test. Normally this time of day I would see 3 or 400 customers... Humm.. looks like only 18 at the moment. One of them is in the process of canceling their server and using FTP to download all their files in order to move to a new server hosting company. So you might see some higher level upload traffic due to the one slow FTP. If you have ever tried to use FTP to transfer thousands of little tiny files you will understand just how slow that process is.
Now for your favorite IP to test my server from. Here is a 12 hour ping test from 18.104.22.168 to the Comcast Head End at 22.214.171.124. Which ran from mindnight 00:00 AM Pacific time to 12:00 PM Pacific time. This covers the normal 4 of 5 hours a day when we have our best internet as well. You will note that we still had a 2% packet loss average over the last 12 hours in just the 3 hops it takes to get from my modem to the head end.
It is my hope that you will find this test usefull. I will start another 12 hour ping here shortly to cover the worst part of the day/evening/night when Comcast Internet is at it's worst between my modem and the Head End. Thank you again for an IP address we can use in common to match tests against each other.
WebGamer. Thank you and good catch on the stats post. The CA1 image was posted as an embedded live link which will allow for data updates by clicking the link, which also updates the time stamp. Ongoing we will post screen shots only.
Thanks again for pointing this out.
Face palm.... Ok John... I feel like an ass for that then. I am still pulling 2% packet loss from my office to the head end. So far this is going on around the clock. Mt tests have been running at 12 hours at a shot.
Pinging 126.96.36.199 from 188.8.131.52
Last night was 32767 packets sent at 256k wiht a 500ms time out going 3 hops to the head end. We received 31965 packets back for a 2% loss. Ping times ranged from 7ms to 199ms with an average of 10ms.
Thanks again for all your time and attention to this topic. Local operations and the Tier 2 support group have worked together to determine a solution for latency issues during high traffic times. We show that you purchased Business Class Internet Deluxe with 50Mbps x 10 Mbps. However, you consistently demand more than 50x10 from the circuit, and the consistent demands above the service levels are causing degradation of service. If you consistently require more than 50x10, we recommend some of our higher bandwidth services which may be more appropriate to meet your demands. From what we can tell, some of our enterprise services may be more appropriate. If you want, we can connect you with an enterprise service rep to discuss what may be best to meet your bandwidth usage needs.
Thanks... but as you can see from my connection, almost no one is connected and it is still failing. We are talking to Comcast Fiber sales and to other providers.
Oh by the way.... I am multi-line load balanced with 2 x 50/10 modems. Both modem are failing at the same time.
Please supply the records proving that we are pulling more than the 50/10 MB/s please.
Have not heard back from anyone about our problems, but the connectivity issues seem to be getting better. I loaded up my second modem to see what effect it would have on our main Business Class, and I could not even tell. So we think someone is working on this area, just not telling us of any progress.
Anyone know why the head end went down yesterday for a while?
Also at this time we are working with local Comcast sales towards the deployment of fiber to this location or to a Co-Locaton supprted by Comcast. Also Comcast took us out for Sushi at Mizu then packed us off to some fantastic seats for the Kings game against the Pistons last night. My business partner is a huge Kings fan and was both very impressed and very excited. Yes, I was very happy to go too. Thank you Comcast!
Here is the direct effect the outage at the head end had upon our servers. We did loose 2 more customers which have also been documented and clearly stated by our customer that connectivity was the issue again.
Looking at both images, you can clearly see that this issue had nothing to do with any connectivity or bandwidth issues between out company and our node at the Comcast head end...
Here is the LIVE link to the Smoke ping to your head end at 184.108.40.206
Here again is the live link to our modem at the Offices of The Web Gamer
As I said before, we are looking rather darn good at 2:07 PM my time today. I nice stable looking green line.
For anyone reading this. The "Official" word from Comcast is that this is the best they can deliver, and there is nothing they can do to correct any issues we have with our connectivity. So if you have Comcast Business Class Cable, expect moments of up to 98% packet loss, extended lag spikes etc. with the full knowledge that this is the best you can expect. If you need to host some servers, take them to a Data Center in your local area and look into "Blended Internet".
Check thnis out.... I have not seen a connection like this in years! This is amazing!!! This is a graph of the last 9 days. What in the heck did they do to fix it!?!?!?! I am floored! I am Stunned and I am still connected to see it...
I would really like to know what they did to fix our problem. No one has called me in just over a week, but I can see they worked on it and fixed it.
Could we get a response about how this issue was fixed? I had an unexplained 8 hour outage and morning outages over several days which prompted me to start inspecting my uptimes and packet loss as well. See https://secure.dslreports.com/r3/smokeping.cgi?target=network.1bc86f75dc1ddfd6b583ac439d748c53.CA1 and https://secure.dslreports.com/r3/smokeping.cgi?target=network.1bc86f75dc1ddfd6b583ac439d748c53.KS. Ignore the blip around 2PM today as that was the Comcast tech wasting time disconnecting the line just to say everything was "working great". On Sunday you'll notice a near-constant 5%-10% packet loss.
Could I also have the cusadmin password reset on the SMC gateway so that I can view the signal information?