The man, in his 50s, cut the fence ties at the Hamilton facility, leaving the site for about half an hour.
People traveling to New Zealand must isolate for 14 days and are placed in designated hotel quarantines.
According to the government press release, the man arrived from Sydney on July 1 and was isolated at the Distinction Hotel when it exploded.
He was reportedly negative for COVID-19 on his third day of quarantine and was subjected to another test on Friday when he was scheduled to appear in court.
The incident comes just days after another newcomer was caught walking out of an isolation center in Auckland, only to find a local supermarket.
In a statement, Air Commodore Darryn Webb, chief of managed isolation and quarantine, said the actions were “unacceptable” and added that clear instructions had been given to those arriving in the country.
He added: “Isolation is an essential part of our defense against COVID-19, and it is up to each person entering this country to play their role and obey the law. ”
Since mid-March, almost 28,000 people have been forced into solitary confinement in New Zealand.
This comes as the nation has eradicated community transmission of the virus, now focusing on imported cases.