Just noticed yesterday evening that there is a large wasp's nest in the eves of my house. I counted about a dozen wasps coming and going in the space of a minute (but as they all look the same, I might have double counted

).
My DIY instincts tell me to get a long pole (I've got one about twelve feet long) and knock it down, wait for them to disperse and then destroy the nest or will I end up in serious trouble with a swarm of angry insects after me?
Another query is that the nest is built on top of one of the vents in the soffit. Is there a danger that they might also have gotten through this vent into the attic as well?