There were wild cheers when the statue of Edward Colston was ripped from its pedestal and later sprayed with paint and thrown into the harbor.
“Vandalism and disorder are completely unacceptable,” said Priti Patel – who added that it would divert attention from the protesters.
Avon and Somerset police said they were investigating after a “small group” had committed criminal damage.
Colston made his fortune through the slave trade in the 17th century and helped build schools, churches and houses for the poor in Bristol.
A petition to remove the statue – which lasted more than 120 years – received 11,000 signatures.
The Bristol protests united around 10,000 people and there were no arrests, police said.
The protest was one of many protests held in the UK this weekend, sparked by the murder of George Floyd in America almost two weeks ago.
The vast majority have been peaceful, with thousands of people supporting the Black Lives Matter movement.
There were a few skirmishes in central London that evening as a few hundred protesters attempted to enter King Charles Street, near the Foreign Office and the Treasury.
The police fired their batons to prevent the crowd from passing through the stone arches, and the situation subsided after about 20 minutes.
Earlier on Sunday, a large crowd descended on the United States Embassy in London, with masses of people meandering along the road waving cardboard signs and chanting slogans such as “no justice, no peace “,” George Floyd “and” bring out Boris “.
Masks, gloves and hand gel were distributed free of charge.
Other protests took place in Edinburgh’s Holyrood Park and Manchester’s St. Peter’s Square.
Fourteen police were injured on Saturday when violence broke out in part of central London.
Whitehall police were bombarded with bottles and officers on horseback were forced to charge the crowd.
One of the injured officers was taken to hospital after riding a traffic light on his horse. Twenty-nine people were arrested.
The Interior Minister said the violence – a few yards from the Centotaph and Downing Street – was “shameful.”
She called the murder of Mr. Floyd “dismaying” and echoed concerns that the protesters could help spread the word. coronavirus.
While some who turned out to be wearing masks, social distance seemed to have been overlooked by many.
Health Secretary Matt Hancock told Sky’s Sophy Ridge on Sunday “there is no doubt that there is a risk” that the number of coronavirus cases will increase following the protests.
“I very much support the argument made by those who protest for more equality and against discrimination, but the virus itself does not discriminate,” said Hancock.
“Gathering in large groups is temporarily against the rules precisely because it increases the risk of spreading this virus. “
Protests over Mr. Floyd’s death have continued worldwide since his death in Minneapolis on May 25.
The 46-year-old man was killed when a police officer handcuffed him and knelt on his neck for almost nine minutes after he repeatedly said that he could not breathe.
Hundreds of thousands marched through cities in the United States on Saturday, with events taking place in an extremely peaceful manner – including in Washington DC where tear gas and rubber bullets were fired earlier this week .