"It (the decision) is off the back of just looking at that channel (X) and the vitriol.
"It was ceasing to be a rational conversation. It was ceasing to be a positive place to be."
X, previously known as Twitter, was bought by South African billionaire Elon Musk for $44bn in 2022.
Brasher, who became race director in 2012, has been involved with the London Marathon since it was set up by his father Chris and John Disley in 1981.
"One of the aims of my father and John was to show that on occasion, the family of humankind could be joyous together and celebrate together," said Brasher.
"That's what the London Marathon is about. It is a force for good.
"We didn't feel that channel shared those values, and therefore we have come off."
This year's event is aiming to break the world record for the most finishers in a marathon.
With more than 56,000 people expected to run, the race could surpass the 55,646 finishers at last year's New York Marathon.