It's pretty normal for puppies to put things they shouldn't in their mouths, much like toddlers. You should work on teaching your pup a solid "Leave it" cue with really great treats. Here is a video from Kikopup on how to train a "leave it." You should also work on the "trade game" so if the puppy does get ahold of trash he'll willingly give it up for a tasty treat.
Yes, most puppies do eventually grow out of that phase where they have to mouth everything and see if its edible.