Neighborly Dog Etiquette Question

I'm having an issue with my neighbor's dog that's making me feel really bad. My dog Cthullu (one yr old male fixed Great Pyr) LOVES other dogs. My neighbors have two dogs, Poochy, and old female fixed toy terrier mix of some kind, and Rudy, a mid-age (I want to say 5-7years ish?) male fixed pit-bull. Now, before I start saying things about Rudy, I just want to state that my neighbors are amazing people, good dog owners, and they do everything they can to make sure both of their dogs are happy and healthy. Poochy is a sweetie, so there's no problem there. But Rudy has issues. I could give you the long story, but I don't feel like typing that much, so I just say he wasn't able to be properly socialized when he was younger, so now he's aggressive to people and animals that he doesn't know. Now for the issue that I'm having. I feel like I keep chasing Rudy and his family back inside. His family doesn't leave him outside unattended. And they spend a lot of time outside with him. But every time Cthullu and I go to play in the back yard, Rudy goes crazy. Thankfully he's on a chain as well as being in a fenced in yard, and like I said, always supervised, so there haven't been any altercations. So usually what happens is Cthullu and I will go outside, Cthullu will start running around the backyard, eventually sniffing the fence against Rudy's yard. Rudy will see him and start going nuts, barking, growling, etc. Cthullu will decide he wants to play with Rudy, and start barking back, and running up and down their fence line. Rudy's family will grab his leash, apologize to me, and take him inside. And I'm left feeling guilty because my dog made their dog go inside, essentially. I feel like that bad guy here. I want both of our dogs to be able to play in our backyards. Right now, I've been trying to take Cthullu out only when I know Rudy is inside. But sometimes I think he's inside, and he's not, and sometimes, they've been outside for a while, and I really want to go play with my dog. I'm not sure what the proper etiquette with this kind of situation is. When I try to talk to my neighbor's about it, they just tell me not to worry about it, that they can bring Rudy inside whenever, etc etc. But I still feel horrid about it. What should I do?

Side, note: I'm pretty sure putting up a privacy fence would help, but I just can't afford it. Those things are bloody expensive...