/59 prefix delegation block changed...

@Comcast_Robert just sent a message to "Comcast Business". Didn't see Comcast_Robert or ComcastRob as an option in the direct message dropdown.

On Monday someone from Comcast contacted me and I re-explained the problem to them.

They were able to confirm the last time a firmware update was pushed was the same time all of this started.  He kept insisting changing firmware would not help and that I should statically configure IP addresses and not pull prefix via DHCP PD.  Every time I call there are different weird responses that make no sense.

Finally relented and said he would roll it back so renumbered everything again and this worked for about a day and a half before all outgoing traffic from the delegated prefix was again dropped.  Can't get into our modem so I can't say if the firmware was actually changed, rolled back and then later a broken version pushed or it was actually rolled back and the same issue persists regardless.  Has been over a week with no IPv6 and no progress.

I really think that the "easy" long-term solution is for the modem to advertise your /56 (assuming you have static) upstream & letting us do static routes downstream.    I put easy in double quotes as I don't know if comcast getting the modem vendors to update firmware is actually easy or not -- thinking more from a routing perspective.

I got a call back from "Tier 2". She understood the problem and escalated it to "Engineering" to have the firmware rolled back for me. Lets see if that helps. In the meantime: If you get the wrong prefix, you can "rotate" them until the right one comes up by deleting the DHCP UUID. I am using this script on opnsense:

(this is a super ugly way of doing it, but well... fits the problem)

#!/usr/local/bin/bash
prefix=$(cat /tmp/em0_pdinfo)
if [ "$prefix" == "<INSERT YOUR DESIRED PREFIX HERE>::/59" ]; then
  echo "all good"
  exit
fi
cat /var/db/dhcp6c_duid | hexdump
rm /var/db/dhcp6c_duid
if [ ! -f /tmp/em0_pdinfo ]; then
  echo "no prefix file yet"
  exit
fi

pid=$(cat /var/run/dhcp6c.pid)
kill -9 $pid
sleep 5
/usr/local/sbin/dhcp6c -c /var/etc/dhcp6c.conf -p /var/run/dhcp6c.pid
cat /tmp/em0_pdinfo

Last night around 1am, my modem rebooted. I hoped this was the "fix", but appears things are worse now. the modem no longer responds at all to the DHCPv6 requests from my router/fw, and I am only able to use the /64 assigned by the modem directly. No other IPv6 addresses within my /56 are usable at all.

I sent my information and the ticket number to Comcast_Business yesterday.

I've been thinking of how to help troubleshoot/resolve this without breaking my servers in the short term.  I've started to use my HE.NET tunnel & I think I may just number my servers with HE.NET addressing & turn DHCPD back on to "test" with on my lan (which is in a different subnet/vlan).  I'll do routing internally between the he.net /48 and the comcast /56 and that way if the DHCPv6-pd stuff breaks on the comast side, my servers aren't broken to the internet.    My lan mostly uses slaac/dhcp & not statics so the clients can "fix themselves" after a short while.

Seems I'm not the only one seeing this from a recent firmware update.

How much of the the IPv6 prefix delegation is handled by the modem as opposed to upstream? At least some of it is if a firmware update can cause us so much havoc. It would be nice to if the modem would let us manually/statically assign prefixes to DUID's like it can for MAC addresses and IPv4 addresses. I wish I had thought to try a solution like flybynight's before re-IP'ing everything (I'm on pfsense).

Does anyone know if other cable modems would allow us such functionality? Netgear has several, but the documentation shares very very little in the way of configuration.

Is there a way to statically assign prefixes on our routers with our current Comcast modems? Perhaps give your router's WAN and LAN devices static address/prefix assignments and have your WAN interface advertise as a route for any of your statically assigned LAN prefixes?

I've managed to get my network stable, at least for the time being, so I can't try this for myself until possibly this weekend to avoid further blocking myself from remote work and 4 kids from attending remote school.

IPv6 broke on its own this morning... But I was able to get static prefix assignment working with PfSense so OpnSense should work too. I'm sure at least a few other routers will work as well.

Assign your WAN interface a static IPv6 address in your public IPv6 range. Add a static IPv6 gateway for your WAN interface (Your modem's v6 IP, you can use the global IP, but I'm using the link-local address). Sadly, obtaining a router automatically doesn't work in PfSense with a static v6 WAN address.

Assign static IP's to your LAN interfaces that fall under your /56 choosing your desired /64's for said LAN interfaces.

Enable route advertising for your WAN interface for your /56.

It works!

Some caveats:

1.  I've had this running for at most 20 minutes. No idea how reliable this will be.
2. For my prefixes, I'm using the same ones that were delegated to me via the modem's DHCPv6 server since I don't want to re-IP. Maybe I'm in a situation where this works because these delegations haven't expired on the modem side?
3. Traffic to my LAN prefixes would not make it through until I enabled router advertising on the router's WAN interface. While this is completely expected behavior... This suggests that #2 may not apply and things are probably working.

I'll report within the next day or few. The previous firmware version for me had IPv6 issues as well, just not as bad. Even if the firmware is fixed completely at some point, this is exactly how I'll want to keep things if this works out. Probably many others will feel the same way.

@flybynight : This exact scenario (exact same symptoms) is what preceeded my call to comcast 2 weeks ago & them determining the modem as "bad" and replacing it. 

yup. they will replace it on Saturday. Let's see how that goes. They figured I got an older model anyway.

Fun fact I learned the hard way. Apparently when not using prefix delegation, static mappings defined in DHCPv6 must include the prefix. The impression I have is that this is the general behavior and not just limited to PfSense (or even just isc-dhpd).

With prefix delegation from the modem, this IP will work, with DHCPv6 filling in the prefix for you:

1234::5

However, if you're not using prefix delegation from upstream (such as your Comcast modem), then you need to enter the whole address:

3300::2323::4545::1234::5.

Otherwise your static mappings will not work.

Otherwise, so far, things have been the most reliable they've been since the update. But it's in the mornings since the update so far that IPv6 tends to have completely fallen over and stopped working so even a preliminary verdict is still out.

The modem swap did help. I now have the same type of IPv6 connectivity as I had before the firmware upgrade. I still need to do a DHCP request, but at least it is working now. (I used to have a "Cisco" modem I think (vertical, 4 ethernet ports). They swapped it for a "Technicolor" modem (horizontal, 8 ethernet ports). I still need the script I posted above to force the modem to give me the right prefix.

@flybynight : I have a vertical with 4 ethernet ports with a label technicolor on it...Comcast Business said they may want to replace my modem again.  What should I be asking for?  Any suggestions?

My modem identifies itself as CGA4131COM . (the "COM" part means it is the Comcast version of this particular product). The modem also has 8 voice ports (I didn't notice them at first. I am not using Comcast voice service).