Only getting <100mb throughput on gigabit network via Netgear CG3000DCR gateway

Hello aburt and welcome,

First, your 50/10 Megabits/second tier Internet service is always measured directly from a wired device connected to any NetGear 3000 (NG3K) Lanport(s) 1-4 connecting to a browser running http://speedtest.comcast.net.  

Second, as far as your " I notice that on my gigabit internal network, traffic sent to the hosts on the static IPs only runs at <100mb speeds. That is, if I "scp  1-Gb-File  foo.com:/path..." from an internal box on, say, 10.1.10.99 to a box on the static IP (75.xx.xx.xx - as the hostname "foo.com" resolves to via public DNS), I get a speed of 2MByte/sec. (Yes, about 20Mbit/sec.)  It also saturates the connection such that ssh session typing is painfully slow.  ", networking traffic between the NG3K DHCP to any StaticIP device 2MBytes/second or 16 Megabits per second is actually quite good. However, as I am sure your know, this actual bandwidth is a direct function of both your device(s) OS(s) networking interface and transport, not related at all to the Comcast 50/10 Internet service bandwidth. 

Thirdly, as far as your "  However, if I assign foo.com a virtual, internal 10.1.10.88 address, the same ssh zips along at 80MB/s. (11 seconds for 900Mbyte file, and no typing lag via ssh.) So it's not the network or the hosts. It's the netgear. (Which in theory is a gigabit device.)  ",  I believe this in fact proves my previous premise that this is a direct function your device(s) OS(s) and networking interface and transport. It would be interesting for you to share if logging into  NG3K.LAN.PORTS to see what your device(s) auto-negotiation speed it set to? As you know, if both devices are not able to interconnect @ 1Gbits/second speed, then this means that more than likely 100 Mbits/second is being used between both devices. So, this would easily explain your 80Mbits/second max you have attained.

Lastly, as fas a you "

Now, I can sort of work around this by hardwiring -some- boxes with their own DNS that says "foo.com" is 10.1.10.88 (e.g. the hosts file on a win7 box), but I can't easily jerryrig all the boxes (e.g. an ipad...).

I'd assume the netgear gateway would be smart enough to know, hey, 75.xx is really on the internal net, and talk gigabit speeds to it.", I totally agree if you are only using any wireless N which sort of maxes out at 150Mbits/second. If you are going from a Gbit Win7 Box into a "can be wired" ipad, then we both know that the ipad network interface and transport will be the determining speed factor in this interconnect.

Inconclusion, sounds like you are doing some really fun stuff with your NG3K network implementation. Great stuff ! 

Hi, Rich, thanks for the reply.

I'm not sure I was clear, though. I would expect internal traffic to the 75.xxx static IPs to remain internal (and thus at full gigabit speeds). That is, comcast's netgear box -should- say, ok, this internal traffic from a 10.xxx host is going to 75.xxx, which is mac address yy:yy:... and send it directly out the appropriate switch port (all at gigabit speeds). What seems to be happening is that comcast's netgear box is instead saying, hmmm, 75.xxx, that's the static IP, I don't see the mac address for that (when it should), so I'll that packet out -this- way -- which I'm guessing goes through some part of the WAN port, which, I'm guessing, is only a 100Mb/s port (to save costs, for example -- since the WAN port isn't likely to need gigabit speeds itself).

I did check the netgear's "switch status" page, and it says it's doing 1000 full duplex.

The actual wiring is this: There is exactly -one- cable going into the netgear box, which comes from a gigabit switch I have. The win7 boxes and linux servers are wired directly into that switch. All switch lights indicate all connections are 1000.

It's as if the netgear doesn't store the mac address for the 75.xxx address in the same table as it does the 10.xxx, so it sends 75.xxx traffic half-way via the WAN port (when it doesn't need to and really shouldn't)......

At least that's my guess... ???

(As for the ipads, those should run faster than they do to internal hosts, but I haven't wasted my time testing them to 10.xxx addresses. The wired win7 boxes are proof enough that something's amiss.)

And I didn't think this was anything complex... This is just one set of 10.xxx DHCP + some static IPs, all on the same comcast box. You should have seen the setup I used to have! 🙂

Just to be clear, it sounds like you are asking asking a layer-2 switch to route layer 3 traffic? i.e. assuming because the MAC addresses are in the same table, the networks(10.x.x.x and 75.x.x.x) should be able to talk?  It doesn't work like that, if that's what you are expecting to have happen, unless you have a layer 3 switch(which is able to route traffic between networks because it's aware of layer-3, which is what I think you're trying to make happen)
.

Say you have PC A in 10.x and PC B in 75.x

PC looks at the IP address you want it to connect to and says "Is it local to me?".  10.x.x.x and 75.x.x.x are not local to each other. So PC A says, ok I need to send traffic to the *router* because that's non local traffic. The router looks at it and puts it back on the local switch because it knows both 10.x and 75.x are local to it.

So you have a few different thigns you can do.  Either move machines so they are in the same subnets, and there for local, or put route entries in.  A route table entry would essentially be you putting an override in place that says "this host/network is local to me, don't send it's traffic to the router".

Hi, Jason, routing isn't the problem, and yes, that is indeed what I expect: 10.xxx sends a packet to 75.xxx (which is local) by way of the 10.1.10.1 gateway -- which is the comcast netgear box. The netgear box does -route- the traffic correctly. The problem is that despite it being a gigabit speed device, when it routes the 10. -> 75. internal traffic, the speed slows waaaaaaaaaaay down.

The packets (should) never leave the building here. It's all gigabit wiring between the boxes. (Proof being that when the packets go 10. -> 10. between the same two hosts, everything flies at gigabit speeds.) Comcast's netgear box is -acting- like it's sending should-be-internal-only packets outside and back in.

So, my problem, and my initial question, remains: Why is comcast's netgear box slowing down my internal gigabit throughput? It shouldn't be. Thus: Is there a fix for this?

Maybe I'm the only one who's experience a slowdown when traffic goes between the 10. DHCP and 75. static IPs on the netgear box? (Or the only one noticing it?)

Anyone else seeing this?

Thx.

I think routing is exactly your problem, an actual traceroute would help figure that out though. You need to think of the logical level of connectivity, not the physical one. You said 75 is on the internal network. It's only on the physical internal net, it's not logically on the same physical net, if you haven't done something to make it that way, which is why they are going out the WAN interface. Physical hardware plugged into the same switch does NOT make them on the same internal network.  If you plug those 2 devices directly into each other with a crossover cable, I bet they wouldn't be able to talk. You also said 75 is public facing. By definition that means it is not part of your internal network logically, if it doesn't have another IP in the local subnet that it responds to.

Why do you think your packets shouldn't leave your local network? you yourself said one of the addresses is a public facing one. You mentioned a trace route...are the packets showing a direct hop, or is the gateway in between them? If the gateway is showing up, then they are going out your public interface and then coming right back in.

10.x to 10.x is gigabit speed because your devices consider each other to be on the same subnet. They don't need to be routed. you said if you add a 10.x IP to the machine in 75 it fixes the problem, which is further support that in fact your packets are routing out and then back in.  Adding a 10.x gives a local IP to respond on thus no use of the gateway.

75.x and 10.x are not local to each other as far as your machines are concerned, especially since you said the one is resolving as a public address, your packets are doing exactly what you think they shouldn't, they are going out the public interface and right back in, so they are limited by its speed. It does this because your machines think that they are NOT local to each other.

You can test this easily. Add entries to the route tables on each machine and see if it goes back to gigabit speed.

So on the 10.a.b.c machine:

route add 75.x.y.z mask 255.255.255.255 10.a.b.c

and on the 75.x.y.z machine:

route add 10.a.b.c mask 255.255.255 75.x.y.z

Those will force your machines to treat packets to each other as though they were local, and just drop them on the wire directly to each other, instead of trying to send through the gateway(which you don't want)

Hope that helps!

P.S. Post a tracert, even with obscured IPs it'd help!

Maybe the attach image will help, this is what it sounds to me like you are describing, physically vs logically, which is where my line of thinking is coming from.

Hello JasonColeman and aburt,

Jason: Outstanding networking layer 2 to layer 3 information in your posts. However, you are absolutely correct in that with the NG3K DHCP being at 10.1.10.1 subnet mask 255.255.255.0 and any single (or 5 or 13 block) Static IP routable address being at 75.xxx.yyy.zzz subnet mask being at 255.255.255.252( or 248 or 240) does in fact normally drive this interconnect transport speed.

There are many other Comcast staticIP transport speed inpacting parametric requirements, for instance, authenticating staticIP validation, etc. So your point of simply requiring devices to be on the network subnet clearly is one of the main issues that aburt's internetworking connect is being impacted by. Lastly, your route add test should be able to shed some light onto this, as well.

Nice job by JasonColeman and aburt. Looking forward to hear the results of aburt's testing.   

Hi, Jason, re "your packets are doing exactly what you think they shouldn't, they are going out the public interface and right back in, so they are limited by its speed" -- I agree that's what seems to be happening.

I don't think it -should- happen is my point 🙂 since comcast's netgear box should be smart enough to realize that both the 10.xx and 75.xx networks are local to it. The netgear box -is- the designated router for the 75.xx static IPs (it has to properly route Internet -> 75.xx traffic to the local hosts, and it does); and of course the netgear box is also the router that handles the 10.xx local traffic, via the 10.1.10.1 gateway address. So what should happen is...

1) 10.xx host sends packet to 75.xx, via 10.1.10.1 gateway, arrving at 10.1.10.1 through its local gigabit switch port

2) 10.1.10.1 gateway (netgear comcast box) says "How do I get to 75.xx? Well, gosh, that's ME! So I send that out the local gigabit switch port."

Instead, at step #2, what seems to be happening is the netgear box says, "How do I get to 75.xx? Let me count the ways... slowly... hmm, how about if I send it by turtle!" 🙂

What I suspect is really happening in step #2 is that the netgear is saying this: "Okay, a packet arrived on local interface abc0, local gigabit switch port. How do I get it to 75.xx? Well, that's one of them thar static IPs, so I should send that onto local interface xyz1, which is the one that incoming packets from the Internet cloud arrive through, and which, heh heh, is really only a sort of hokey 100Mb/s or slower interface."

I don't think the netgear is actually sending the packet all the way out the coax cable line to the next comcast router and back. Just sending it internally onto the same internal interface that the coax packets use.

x

Traceroute, by the way, shows all packets taking exactly one hop, directly from source to destination:

Tracing route to reanimus.com [75.148.123.25]
over a maximum of 30 hops:

  1     1 ms    <1 ms    <1 ms  75.148.123.25

Trace complete.

--vs--

Tracing route to 10.1.10.66 over a maximum of 30 hops

  1     2 ms    <1 ms    <1 ms  10.1.10.66

Trace complete.

Note that 10.1.10.1 never itself responds. It's as if the netgear -switch- software handles it all at layer 2 (as it should), -but- it sends the 10 -> 75 traffic through a different internal path than 10 -> 10 packets. It's like it sends the 10->75 packets half way through the conversion to coax stack (still in layer 2). So it's like it goes Gigabit i/f->Coax i/f outbound->Coax i/f inbound->Gigabit i/f for 10->75, but 10->10 goes Gigabit i/f -> Gigabit i/f.  (And, the Coax i/f seems to get really swamped with the true gigabit-speed traffic.)

Your idea of adding static routes with 255.255.255.255 netmasks is worth trying as a test, i.e., trying to prevent the traffic from going to the 10.1.10.1 part of the netgear box. (I.e., arriving at the netgear box with the mac address of the 75.xx box already in the ethernet packet, rather than the 10.1.10.1's mac address.) I'll try it when I have a chance to play (and not worry about messing up live traffic on the servers). 🙂

The problem is that even it worked, while it would verify the problem, it wouldn't really help with the solution: I have many internal boxes where I can't add static routes. (I.e., software you can't muck with the routing - an ipad would be one example.)

So ideally the solution would be for comcast's netgear box to handle local 10. -> 75. traffic properly. 🙂

So after chewing on this a bit, I think it's entirely possible what you are running into is that the router is set to only handle x Mb of traffic/sec corresponding to whatever your plan is.  I don't know how Comcast throttles performance, but it could do it by limiting the packet routing rate, and that could explain the limit you are hitting.  I think you mentioned it's only moving data between the 10.x and 75.x hosts at your plan rate, and that it saturates your traffic at that point? In reality, you may be correct, you may not be using any traffic on the Internet end, but the router probably doesn't care if it's routing between the WAN interface and the local switch, it probably just routes a certain amount of traffic per second and stops there. I think this neatly explains everything you are seeing. So even though it's physically local, because it's being routed, you're hitting that limit on the router side is my guess.

Now, if the router was a bit smarter and it distinguished between routing back to things on its local switch versus the WAN interface, then you'd probably be correct, you'd probably see closer to wire speeds without having to add route table entries between hosts. I think adding route table entries at the hosts will somewhat confirm if router throttling is causing this.

I don't get why the gateway isn't showing up in the trace route, I thought it would, but maybe routing between subnets that are local to each other doesn't show the router in a trace route, I'm not sure on that one.

-Jason

By the way, the simple solution, which I think you implicitly proved would work earlier, rather then have to add lots of route table entries, would seem to be to just add another IP address for the 75.x host to use, only in the 10.x range.  Is there a specific reason you are avoiding doing that?

Hi, Jason, yes, I think your explanation could well be it, that the netgear is the locus of the bandwidth limiting for our plan, and it simply isn't smart enough to realize that not every packing on the 75.xx static IPs are actually arriving from outside. (Although, it should be limiting inbound packets to my plan's -download- limit, which is 50Mb/s. My plan upload bandwidth is 10Mb/s, which is what it seems to be, roughly, limiting it to... Well, twice that. Hmm.)

As for a solution, on -certain- boxes, yes, I can add static routes, change IP#s to 10.xxx, etc. Unfortunately, I can't do that on -all- the boxes. Many have proprietary software (e.g. ios7) where there's simply no way to mess with routes, do virtual interfaces for multiple IPs, or suchlike. The only thing those inflexible boxes can do is a straight DNS lookup -- they will simply do a DNS look up of, say, reanimus.com, and will be told that's 75.148.123.25, and their throughput will be throttled even though they're internal.

Well, bottom line, it appears to be a firmware bug...

Any way to submit a bug report to netgear?

I'm not suggesting you add routes to all the hosts.  The route thing was just to see if it is a route/switch problem.

What I was asking after that was the 75.x host...it's presumably not an IOS device, it sounds like a server of some sort.  Why are you not just assigning it IPs in both subnets? That would address all the issues you are having.

Ah I see.

Can you just put up your own local DNS so that internally those servers resolve to internal addresses?

Hi, Jason, thanks, yeah, that's a possiblity. The ideal solution would be for netgear to fix their firmware, though.

I'll leave this open with that as the pending question: Anyone got contacts at Comcast who can tell Netgear that this needs to be fixed in the firmware?

Thanks.