Background: This study aimed to examine the relationship between depression, self-efficacy, sexual function, and quality of life in middle-aged Korean men, and to identify which of these factors affect their quality of life. Methods: Participants were 216 middle-aged Korean men, aged 41–64 years. Data were collected from October 23, 2020, to November 15, 2020, from three small and medium-sized enterprises, four government offices, university alumni associations, and meetings. Study variables were depression, self-efficacy, sexual function, and quality of life of middle-aged men. Results: There was a negative correlation between quality of life and depression (r = –0.51, p 0.001), while self-efficacy (r = 0.52, p 0.001) and sexual function (r = 0.35, p 0.001) showed a positive correlation. Depression had a significant negative correlation with self-efficacy (r = –0.31, p 0.001) and sexual function (r = –0.30, p 0.001). Self-efficacy was positively correlated with sexual function (r = 0.27, p 0.001). Model 1 included general characteristics that affected the quality of life, two of which were found to have significant effect: monthly household income and subjective health status. The explanatory power was 21.6% (F = 10.86, p 0.001). Model 2 added depression, self-efficacy, and sexual function, and showed that subjective health status, depression, and self-efficacy had a significant effect on quality of life. The explanatory power increased to 43.6% (F = 19.49, p 0.001). Conclusions: Depression, self-efficacy, and sexual function affect the quality of life of middle-aged Korean men. Therefore, these variables should be considered when establishing an intervention strategy to improve the quality of life for this group.