Issue affecting distro1.dc2

-

Issue has now been resolved. We have seen two issues tonight which we have identified and fixed. First one being a bad breakout cable causing errors on the port and packet loss for affected networks.

Second one is packet loss to and from the router, which we have identified as the control-plane policies kicking in and rate limiting ping.

After adjusting the policies we are now seeing no errors on the ports and no rate-limiting by the CoPP.

We will still proceed with planned maintenance and use the time to make sure the router is still functioning as expected tomorrow night. We will also replace a router in DC1 during this time.

Traffic is flowing fine through the router and we are not seeing any performance issues. We are however seeing packet loss to and from the router.

Performance should now be back to normal. We suspect a bad 40G->10G breakout cable to be the source of the issue. We are still investigating the cause and it's too early to conclude just yet. We will update this issue with any updates.

Router is back online.

The router is up but not switching packets properly. We are working on it.

The router is currently being reloaded.

Issue seems to have root in an error on one of the uplink ports to the router. We are dispatching at tech to the DC to take a look at it. Occasional dropouts while we investigate is to be expected.
Affects subnets:
185.181.63.0/24
185.181.61.0/24
185.243.217.0/24

We're currently aware of the issue and are investigating the cause. We will provide further updates as we have them.