Network Outage - 25 February 2013

Network-wide DNS outage:

At around 10am on Monday morning (25 February 2013) our primary DNS controller at our Cape Town data center went offline.

Our engineers attempted to restore the DNS controller however this was not possible and a new replacement DNS controller had to be provisioned. Admins worked throughout the night to configure and set this up.

The new controller came online on Tuesday morning and started propagating DNS. As the DNS started propagating all services came online.  A fail-over secondary DNS controller is situated in the IS data center in Bryanston, Johannesburg.  Sadly IS started reporting connectivity issues in their DC with intermittent connectivity issues adding to the problem.

At this time DNS have stabilized, our networking team is working with both Cape Town and Johannesburg DC's and looking into building an even more redundant fail-over system.

Telephone system issues:

During the DNS outage we had an influx of calls as one can expect.  We make use of a well known VOIP provider in South Africa for truncating calls to our help-desk. As a result of the large number of incoming calls our provider informed us that we are overloading their trunk and started rate limiting the number of calls being patched through.  We are still experiencing a large number of incoming calls at this time and we are dealing with these calls as swiftly as possible.

We urge clients who are unable to get through to please open a support ticket with us (support@registerdomain.co.za), we will attend to each case in a fast efficient manner.

We do apologise for the inconvenience.

Register Domain SA
Http://www.registerdomain.co.za