Will they ruin what we have and how do I ignore them ?

I fear people at work will damage what I have with my best friend. Please dont judge but we have kissed a few times. We.got close as his girlfriend of 5 years treats him badly. For now he says we should cool it and stay good friends . Im fearful because work colleagues are already gossip mongering and im frightened will damage what I have with him