He said there has been "strong pressure" from crowds of about 100 people who noisily descended upon the pagoda for three days until Friday, demanding that devotees of the French-based Thich Nhat Hanh leave.
"It's because of that that I had to sign this morning... a written promise of their departure, at the latest December 31."
Nhat Hanh is a Zen monk and was a confidant of slain US civil rights leader Martin Luther King.
Almost 200 of his followers have been staying at Phuoc Hue since September when they fled their monastery at Bat Nha, also in Lam Dong province.
Followers said at the time they left Bat Nha after threats from people armed with hammers and batons.
"For the authorities, we are illegal," Trung Hai, a monk representing the group, said last month in Geneva.
One nun, who declined to be identified, said police were among the crowd, whose members told her they had been paid to intimidate the Buddhists.
"We don't have a place to go. We don't know what to do," the nun said.
A local government spokesman could not be reached for comment but a police chief in nearby Bao Loc town said: "I don't know anything about the situation in this pagoda."
In October a foreign ministry spokeswoman described the matter as an internal Buddhist dispute and denied that hundreds of people had been forced from Bat Nha.
A crowd first arrived at Phuoc Hue on Wednesday during a fact-finding visit by the European Union, witnesses said.
The delegation's visit followed a European Parliament resolution late last month which condemned the violent expulsion of the "peaceful Buddhist community". The resolution called on Vietnam to comply with its international obligations to allow the free practice of religion.
All religious activity remains under state control in Vietnam but the government says it always respects the freedom of belief and religion.
