The fighters, from the Ansar Dine group, which controls much of northern Mali, attacked the mausoleum of Sidi Mahmoud, one of 16 shrines in the city.
Last week, the UN cultural organization Unesco put Timbuktu on its list of endangered world heritage sites, fearing damage following the coup which toppled the Malian government in March.
Islamists regard shrines as idolatrous.
However, some Muslims, especially Sufis, regard them as an accepted part of Muslim worship.
Witnesses said Islamist fighters began attacking the shrines on Saturday morning, using shovels and pickaxes.
“This is tragic news for us all,” Unesco chairperson Alissandra Cummins said in a statement to the AFP news agency. “I appeal to all those engaged in the conflict in Timbuktu to exercise their responsibility.”
Ansar Dine spokesman Sanda Ould Boumama told the AFP news agency the shrines would be destroyed, “all of them, without exception”.
He went on: “God is unique. All of this is haram (forbidden in Islam). We are all Muslims. Unesco is what?”
“They have already completely destroyed the mausoleum of Sidi Mahmoud (Ben Amar) and two others,” Malian journalist Yeya Tandina told the Reuters news agency.
In addition to the shrines, Timbuktu is home to some 700,000 ancient manuscripts held in about 60 private libraries.
On Thursday, Unesco said the city’s capture by armed Islamists could endanger its “outstanding architectural wonders”.
“It looks as if it is a direct reaction to the Unesco decision,” Timbuktu official Sandy Haidara told Reuters.
Timbuktu, an ancient trading city on the edge of the Sahara Desert, is known for its distinctive architectural structures, constructed mainly from mud and wood.
The shrine of Sidi Mahmoud was also attacked in April and set on fire by armed men from Ansar Dine.
Ansar Dine, which has ties to al-Qaeda, seized northern Mali in March, in tandem with ethnic Tuareg rebels.
However, the alliance between the two groups has frayed and the Islamists now say they control the territory after driving out their former allies.
Islamist forces recently seized the town of Gao from the Tuaregs. The 17th Century tomb of Emperor Mohammed Askia, which is in Gao, has also been placed on the Unesco danger list.