Previous studies have confirmed the sex difference of gray matter asymmetry in typically developing controls and the abnormal gray matter asymmetry in autism spectrum disorders. However, whether and how sex differences of gray matter asymmetry exist in autism spectrum disorders remains studied. This paper analyzes the above issues and explores correlations between gray matter asymmetry and autistic symptoms. Data from 72 children (36 males and 36 females) with autism spectrum disorders and 72 typically developing-controls (36 males and 36 females) between 8 and 14 years were included and obtained from the autism brain imaging data exchange repository (autism brain imaging data exchange I and autism brain imaging data exchange II). The voxel-based morphometry approach was used to assess gray matter asymmetry in T1-weighted brain data, and gray matter asymmetry was quantified as asymmetry index. A 2 \times 2 analysis of covariance was used to identify asymmetry index differences among the four groups. Pearson correlation analysis was performed for asymmetry index values extracted from the clusters with significant differences between the four groups and autistic symptoms (social impairments, communication difficulties, and restricted, repetitive behaviors) measured by the revised autism diagnostic interview scale. Results showed that specific brain regions showed significant main effects for diagnosis in which autism spectrum disorders patients had more leftward asymmetry than typically developing-controls for the parahippocampal gyrus and the postcentral gyrus; specific brain regions showed significant main effects for sex in which females showed more rightward asymmetry for the middle temporal gyrus, inferior frontal gyrus, angular gyrus, and postcentral gyrus and minor rightward asymmetry for the superior frontal gyrus than males; significant diagnosis \times sex interaction effects were identified in the angular gyrus and middle occipital gyrus. Pearson correlation analysis showed that males with autism spectrum disorders had a positive association between the asymmetry index value in the middle occipital gyrus and more significant verbal impairment measured by the revised autism diagnostic interview (r = 0.387, p = 0.026). The asymmetry index value in the parahippocampal gyrus was positively associated with more severe social impairment in females with autism spectrum disorders (r = 0.422, p = 0.020). We identified that the sex difference of gray matter asymmetry in children with autism spectrum disorders is qualitative rather than quantitative, which is relatively novel. Our findings provide the theoretical basis for conducting separate studies and using sex-specific diagnostic methods and treatments for males and females children with autism spectrum disorders.