So, the Silence turn out to have been around making mankind do things for thousands of years, and they have a perfect defence mechanism in that they mess with your head and make you forget every encounter you had with them. We have the odd scene where they kill indiscriminately, or leave people unable to think properly, and there's generally an impression that they're sinister and parasitic. We don't find out why they've been guiding mankind all this time, or why they want moon landing related technology.
But nevertheless, when the Doctor leaves mankind a subliminal message to destroy the Silence, effectively to commit genocide (an act condemned e.g. at the end of season 4 regarding the Daleks), is that OK? Isn't it just possible that the Silence are responsible for good things as well as bad (e.g. medical and technological advances)? Or that not all of them are responsible for the bad things?
And then there's a lingering question of what mankind would have been without them, if they've been around for so long. Would man have developed in a completely different way? But while that's interesting, I think it's less important than the ethical question, especially in a show that privileges ethics over other considerations a lot of the time.
Disclaimer: For 'ethics', feel free to substitute 'morality' or other similar words. Not that they don't mean different things, but their use is so confused in everyday parlance (as opposed to, say, philosophical theory) that it always seems weird to me to talk about them as if they are absolute concepts.