Accelerated maturation of brain parenchyma close to term-equivalent age leads to rapid changes in diffusion-weighted imaging (DWI) and diffusion tensor imaging (DTI) metrics of neonatal brains, which can complicate the evaluation and interpretation of these scans. In this study, we characterized the topography of age-related evolution of diffusion metrics in neonatal brains. We included 565 neonates who had MRI between 0 and 3 months of age, with no structural or signal abnormality—including 162 who had DTI scans. We analyzed the age-related changes of apparent diffusion coefficient (ADC) values throughout brain and DTI metrics (fractional anisotropy [FA] and mean diffusivity [MD]) along white matter (WM) tracts. Rate of change in ADC, FA, and MD values across 5 mm cubic voxels was calculated. There was significant reduction of ADC and MD values and increase of FA with increasing gestational age (GA) throughout neonates’ brain, with the highest temporal rates in subcortical WM, corticospinal tract, cerebellar WM, and vermis. GA at birth had significant effect on ADC values in convexity cortex and corpus callosum as well as FA/MD values in corpus callosum, after correcting for GA at scan. We developed online interactive atlases depicting age-specific normative values of ADC (ages 34–46 weeks), and FA/MD (35–41 weeks). Our results show a rapid decrease in diffusivity metrics of cerebral/cerebellar WM and vermis in the first few weeks of neonatal age, likely attributable to myelination. In addition, prematurity and low GA at birth may result in lasting delay in corpus callosum myelination and cerebral cortex cellularity.