You didn't have to wear a Speedo, J.D. Some loose fitting pants and no one would ever know. 

From the Daily Beast: 

Salinger Was Born With a Single Testicle

At one point, Salinger called himself as a “condition, not a man.” Based on their research, Salerno and Shields are convinced that the author was referring, at least in part, to the fact that he had been born with only one testicle.

After Pearl Harbor, Salinger tried to enlist in the Army, but he was, as he put it in a letter to his literary mentor, “classified I-B with all the other cripples and faggets [sic].” According to one of his fellow soldiers, however—Salinger later volunteered for a counterintelligence position and went on to serve in Europe—the author once told his hero Ernest Hemingway “that he didn’t think the army would take him… [because] he had only one testicle.” Salerno and Shields write that they initially dismissed the assertion, but “two women independently confirmed that Salinger had this physical deformity, about which, one of them said, he was ‘incredibly embarrassed and frustrated … It was a big deal to him.’”