Mine have a dog door as well, into a secure dog run. Normally it has a clip on the gate, but it's gone missing, however there is a padlock I'm going to put on tomorrow instead anyway. So someone wanting to get at the dogs would have to cut the fence or break the lock, or come into the house.
Kilt started using the dog door right away, and hasn't had an issue with it at all. She likes to be outside.
But, they don't have free access to the whole house AND the dog door to the outside. Because they'd track mud, snow, rain, etc. all over the house and anything not nailed down would be dragged out the dog door and destroyed. In the house they're fine, but if they have the chance, apparently in their minds (well ok it's one dog but still...) it's perfectly 'legal' to carry anything outside and shred it. She can be in the house all day and not touch a thing, then grab it and scoot outside and destroy it. So, they are in the pen and can go in and out, or usually one at a time come hang in the house, do some training or whatever.
The only concern with a dog having free access to the outside is safety, both from animals and people (so no way for dogs etc. to get to him, or people opening a gate), as well as him not being a bother to anyone (you don't want him outside at 2 am barking at the moon and waking up people all the time).