The Derbyshire Dales district council had yielded to mounting pressure after nearly 35,000 people signed a petition calling for the removal of the controversial listed structure.
But local people reportedly shot their heads to protect them from the council, claiming to have permission.
Hours earlier, authorities announced that the 200-year-old sign was being removed.
The council’s decision was hailed by anti-racism activists, but others have called it “cultural vandalism”.
The move came after protesters shot down a 17th-century statue of the slave trader Edward Colston in Bristol on Sunday.
A few hours after the revelation, the residents gathered under the sign of the city center before hanging a banner around the head of the sculpture on which was written “Save me”.
As darkness fell, two men climbed a ladder and removed their head from its place on the wooden gallows.
A witness said, “They were clapping and clapping, then one person yelled” Black Lives Matter. ” It was quite tense.
“The road was blocked and there were people arguing. It really got out of hand. ”
The first petition described the cartoon as “disgusting” before a counter petition was launched and signed by 3,000 people calling for the historic sign to remain in place.
On Monday evening, activist Mark Redfern, who launched the petition to save the brand, went on social media to publish: “The head is now safe on the ground and in secure storage.
“The people of Ashbourne will give the head a Lick of Black painting and fully restore it when we get the chance.
“Permission has been granted to withdraw what has taken some time. The people of Ashbourne did this only to save their heads from vandalism. The head will be reinstalled at a later date. ”
Earlier today, Derbyshire Dales District Council confirmed that the two-sided head would be removed due to a public safety issue.
They said it was “an issue that requires urgent discussion and consultation”, but declined to confirm whether the decision was permanent.
For many years, the sign, straddling the road, has been the subject of heated debate, given its controversial depiction of a black man’s head with two faces – one smiling, the other sad.
Following the Black Lives Matter protests, a petition was launched on Friday calling for the removal of the historic figure who has stood above the city’s main street since the 1820s.
Although it was 4,000 miles from Derbyshire in Minneapolis where George Floyd died when he was arrested by police, the iconic sculpture has drawn the community into a dispute over racism.
In less than 72 hours, nearly 29,000 people supported the petition on Change.org, demanding the removal of the private structure and classified Grade II *, citing its “repugnant racist images”.
But within hours, a counter-petition to save the sign, which supporters say is of historic importance to Ashbourne, was launched and signed by 3,000 people.
Rebecca Jordan, a local woman, originally requested that the sign be removed in response to the murder of Mr. Floyd.
She wrote: “After the tragic and illegal murder of George Floyd and countless other blacks in police violence, and the ensuing protests around the world, we in Ashbourne can do our part to combat this type of racism. vile starting at home on St John’s Street.
“This is only the first step to repairing racism rooted in our own country, but how can we support racial justice while passing through this monstrosity every day without doing anything about it.
“Ashbourne does not need to be stuck in the dark ages, this singular act is the least we can do to change racist attitudes. ”
Meanwhile, Redfern said: “The black head should be kept because of the history of the city and should only be seen as a reminder of the times. ”
On Monday, however, the council decided to act quickly to broadcast the increasingly controversial line.
A spokesperson said, “We are removing the head from the panel with immediate effect.
“We agree that the sign itself is not only a public safety concern at this time, but that it is an issue that requires urgent discussion and consultation.
“The sign was offered to the section council several years ago and is currently protected by a list of grade II structures.
“Legally, only Heritage England or the Secretary of State can withdraw this list, which means that we must take into account the views of our own advisers and the local population before advancing any representation. It will happen soon. ”
The Green Man pub dates back to 1750 and was a busy hostel for travelers, but by the 1820s business was declining.
The owner then bought the Blackmoor’s Head Inn next door and combined the names.
Officially, the Royal Green Man and the Blackamoor’s Head Family and Commercial Hotel have been recognized as the longest pub name in the country by the Guinness Book of Records.
In recent years, the pub has gone through a rough patch, but the Green Man reopened in 2018 as a restaurant after a major renovation. The other half of the “two” inns is now a café.
The story of the origin of the Black’s Head is unclear, some believe he is related to Othello de Shakespeare, others believe he was a Moroccan coffee merchant who visited the city. Other stories claim that he was Turkish.
Another story is that of a young servant who accompanied Sir Walter Raleigh on visits to the city, where it is said that the explorer owned land.
Some believe the sign has obvious racist overtones and say it should be deleted, while many see no problem with the piece of local history.
Councilor Barry Lewis, head of Derbyshire County Council, said the figure was “clearly culturally insensitive and racist”, but added: “Cultural heritage is there to challenge us sometimes, to hurt us comfortable.
“It is an artifact of an era and of attitudes to which we never want to return, but that does not mean that we have to demolish our heads. To do this, in my opinion, it is simply cultural vandalism. ”
Colin Wright, owner of The Green Man pub, said the two petitions “underscore polarizing views on this characteristic of the city.”
Historical England stated that the inn sign “gallows” of painted iron and wood in the mid-18th century was first listed in 1951.