The Linh Son Temple has not been used as a place of worship for four years.
Subsequent planning applications for it to be used for worship again were refused and now a for sale sign has gone up outside.
Upper Norwood councillor Pat Ryan said: "The folks there are very well-behaved.
"But it is not a suitable place for a place of worship and we have tried to find them alternative accommodation in the past.
"At least two times a year there would be large numbers attending the site.
"I would get numerous complaints about parking on side roads and on the pavement."
Four years ago Croydon Council received complaints that over 500 people had attended the building for worship, even though at the time planning permission for this use had expired.
Video evidence was taken and the Master of the Linh Son Buddhist Association was convicted of failing to comply with an enforcement notice.
Planning applications to use part of the ground floor as a place of worship were then turned down in July 2006 and April 2007.
Cllr Ryan made it clear there was no discrimination over it being a place of worship for Buddhists.
He explained: "It would have been the same if it was Church of England or Roman Catholic.
"I wish them well wherever they go. I hope whoever buys the property will use it for more suitable purposes."
Penge councillor John Getgood claims the temple's owners have their eyes on moving to an old sports ground in Beckenham.
He said: "I have been told by people at the Beulah Hill temple that they want to buy premises in Kings Hall Road.
"But before they can complete on that they have to sell the temple in Beulah Hill."