Dog hates to be outside alone

Pete our rescue terrier mix has made HUGE strides in being alone. He used to scream and drool and claw at his crate continuously whenever we were gone but I am happy to report that he is now spending the night in his crate downstairs and handled being left alone in the day for 2 hours with only 10 mins of barking. YAY!!

However, there is one last issue with separation.... he HATES being left alone outside. He has even started refusing to go outside at all unless I am out there (not anyone else). As soon as he realizes he is alone he runs frantically from door to door barking and whining.

Although I want him to enjoy our yard I don't plan to leave him out there unattended for any extended period. I just would like to be able to put him out to pee and have him go and come back (5 mins or so). During the warm weather I can stay out with him but I do NOT want to have to stand outside in the Canadian winter in my pajamas at 6 am. Thoughts?