You could try a fungicide. I don’t think it’d kill the mushrooms, but it’d kill the fungus that started them.