White matter asymmetries of the human brain have been well documented using diffusion tensor imaging. The purpose of this study was to investigate white matter asymmetry across the whole brain in cerebral small vessel disease patients and evaluate the relation between the factors which often represent disease existence and white matter asymmetry. A total of 105 nondemented elderly subjects with cerebral small vessel disease aged between 60 and 85 years were included in this study. All subjects underwent T1 MPRAGE, fluid-attenuated inversion recovery, and diffusion tensor imaging scanning. With tract-based spatial statistics in diffusion tensor imaging, this study examined the white matter asymmetries and the correlations between white matter asymmetries and four distinct factors such as deep white matter hyperintensities score, periventricular hyperintensities score, cerebral microbleed number and lacune number. The study suggested the asymmetric microstructural change in small vessel disease patients involving the right side being more injured than the left. The four factors jointly affect right brain anisotropy decrease in the middle cerebellar peduncle, the cerebral peduncle, the pontine crossing tract, the corticospinal tract, the medial lemniscus, the posterior limb of internal capsule, and the frontal pattern of white matter. Results of the study demonstrated the lost right white matter may be the main origin of dysfunction in small vessel disease patients. This asymmetry should help with the evaluation of prognostic indicators of disease progression in lesion-based neuropathology.