30 May 2023

NZ websites down - Security update causes widespread internet outages

2:04 pm on 30 May 2023

By Chris Keall of NZ Herald

Male hands on a laptop

Photo: 123RF

An internet glitch rendered banking apps and a number of .co.nz websites inaccessible for some users on Tuesday morning.

It appears to be related to an attempt by InternetNZ - the non-profit that administers local web domains - to roll out a better system for protecting users from fake versions of websites.

"Our apologies, we're aware that certain Internet Service providers are encountering issues this morning. This means some of our customers will have issues accessing FastNet Classic and ASB mobile," ASB posted this morning on its Facebook page.

And after Sheri Ngaha complained on Kiwibank's Facebook page, "Why can't we get into the app or ring this morning. This is so annoying, I'm needing to transfer money but can't," the bank replied: "We're currently experiencing an issue for some customers when trying to access our app or internet banking. Our teams are looking into this at the moment and we hope to have this resolved soon."

On Twitter, Hamish Mack posted: "NZ sites RNZ, New World shopping online and Kiwibank sites are not working? What the heck??"

And Rebecca McMillan said the outage did not seem to have affected govt.nz, but all New Zealand apps and websites she used were down.

"Can't even listen to @radionz because the mobile app is down. Yikes. Time to get a transistor radio."

A service bulletin from InternetNZ late yesterday noted technical problems that hit .ac.nz (education) addresses yesterday then spread to other local domains from 10:45pm Monday.

InternetNZ on Tuesday said all types of local internet addresses were affected. An update at 9.21am said, "The issue will resolve over time."

Responding to a New Zealand Herald query on Twitter, cloud computing engineer Simon Lyall said, "InternetNZ was changing the key they use to sign .nz and made a mistake. So DNS [domain name server] queries are getting a certificate error."

In other words, it seems a change designed to boost security went haywire, rendering some sites inaccessible. It seems the change was related to a measure to prevent 'DNS spoofing' - or maliciously redirecting a user to a fake version of a website.

InternetNZ acknowledged the issue when approached by the Herald. More information was pending.

Mack said at 10:30am his internet connection was "all good now".

Stuart Laing posted earlier this morning: "Anybody having problems accessing .co.nz sites.nz sites seem to be ok," but told the Herald just after 9am that his connections were now "sorted".

One customer of One NZ (formerly Vodafone NZ) said he had issues accessing multiple .co.nz internet addresses from around 11:30pm.

A spokesman for One NZ told the Herald: "We had an issue that occurred in the wee small hours where some customers couldn't access .nz domains when using a fixed connection, but this has been resolved."

Major internet service providers and banks have been approached for comment. 2degrees and One referred the Herald to InternetNZ.

What happened?

So what was the change that InternetNZ was trying to implement?

Technology writer Juha Saarinen explained that: "The original domain name system (DNS) - that translates between links like http://nzherald.co.nz and Internet Protocol (IP) address like 104.18.2.137 assigned to network hosts - had no security features.

"This led to major security problems such as DNS 'cache poisoning' which meant malicious people were able to redirect users to bogus websites."

The changes introduced overnight were designed to make it easier to authenticate if a website was the real deal. They involved Domain Name System Security Extensions (DNSSEC).

Some people posted that clearing their web browser's cache resolved the issue, but Lyall cautioned that might not work if there are problems further up the food chain. It was better to wait until your internet service provider has followed InternetNZ's instructions to flush its own cache (temporary storage of websites and related data, designed to speed up loading).

This story was originally published on New Zealand Herald.