Is there really evidence of RAH as a "wife-swapper"? I don't remember reading details of that. Hints, maybe. Or I'm dense.

The most explicit discussion I've seen is the letters by Grace Sang reprinted in the appendix of vol 2 of Patterson's biography. It gives the strong impression that Robert enjoyed an open marriage and that Leslyn wasn't completely happy with the situation, but figured fair is fair, and when she tried to do likewise, Robert interfered with her liaisons. They also seem to indicate that Ginny was throwing herself at Robert while he and Leslyn were still married, and that this was a contribution to their divorce.