It is not a cicada killer which are solitary wasps and do not construct huge paper nests. Most cicada killers dig a hole in the ground, place a paralyzed cicada in it and lay and egg on the cicada. Without a picture of the insect its hard to say what it is...but the fact that you say they are non aggressive indicates a european hornet. They are very large have a kind of orange cast to them and gather in large numbers. Though somewhat non-aggressive, they will sting when provoked. There nests are as you describe as well.