Affected services:

  • Network DC2

Issue affecting distro1.dc2

Opened on Friday 17th July 2020, last updated

Resolved

-

Posted by Andreas Haakonsen

Resolved

Issue has now been resolved. We have seen two issues tonight which we have identified and fixed. First one being a bad breakout cable causing errors on the port and packet loss for affected networks.

Second one is packet loss to and from the router, which we have identified as the control-plane policies kicking in and rate limiting ping.

After adjusting the policies we are now seeing no errors on the ports and no rate-limiting by the CoPP.

We will still proceed with planned maintenance and use the time to make sure the router is still functioning as expected tomorrow night. We will also replace a router in DC1 during this time.

Posted by Andreas Haakonsen

Monitoring

Traffic is flowing fine through the router and we are not seeing any performance issues. We are however seeing packet loss to and from the router.

Posted by Andreas Haakonsen

Monitoring

Performance should now be back to normal. We suspect a bad 40G->10G breakout cable to be the source of the issue. We are still investigating the cause and it's too early to conclude just yet. We will update this issue with any updates.

Posted by Andreas Haakonsen

Monitoring

Router is back online.

Posted by Andreas Haakonsen

Identified

The router is up but not switching packets properly. We are working on it.

Posted by Andreas Haakonsen

Identified

The router is currently being reloaded.

Posted by Andreas Haakonsen

Investigating

Issue seems to have root in an error on one of the uplink ports to the router. We are dispatching at tech to the DC to take a look at it. Occasional dropouts while we investigate is to be expected.
Affects subnets:
185.181.63.0/24
185.181.61.0/24
185.243.217.0/24

Posted by Andreas Haakonsen

Investigating

We're currently aware of the issue and are investigating the cause. We will provide further updates as we have them.

Posted by Andreas Haakonsen