Background: To compare the rate of malignancy in adnexal masses with benign appearance on ultrasound and low Risk of Malignancy Index (RMI) based on their rate of growth. Methods: All patients in our obstetrics and gynecology practice undergoing surgery between 2015 and 2020 for adnexal masses with only benign appearance on ultrasound were analyzed. Ultrasound findings of the adnexa up to 3 years prior to surgery, cancer antigen 125 (CA-125) levels and RMI were evaluated. Results: Patients ranged in age from 22 to 84. All adnexal masses appeared benign on ultrasound using International Ovarian Tumor Analysis (IOTA) Simple Rules. CA-125 levels ranged from 4 to 127 U/mL. The RMI ranged from 7 to 127 (200 is indicative of malignancy). Ultrasound findings up to three years prior to surgery were available for 43 patients. In 39 patients, the adnexal mass grew slowly, and the histopathology was benign. In 4 patients, the adnexal mass grew rapidly (increased in size by more than 50% or first appeared in 6–12 months prior to surgery), and all were found to have borderline or early stage ovarian/tubal carcinomas. The rate of malignancy was 9.3%. Conclusions: Despite benign appearance on ultrasound and low RMI, adnexal masses with a rapid rate of growth were associated with a risk of malignancy of 9.3%.