The past decades witnessed a surge of interest in neuroimaging study of normal and abnormal early brain development. Structural and functional studies of normal early brain development revealed massive structural maturation as well as sequential, coordinated, and hierarchical emergence of functional networks during the infancy period, providing a great foundation for the investigation of abnormal early brain development mechanisms. Indeed, studies of altered brain development associated with either genetic or environmental risks emerged and thrived. In this paper, we will review selected studies of genetic and environmental risks that have been relatively more extensively investigated-familial risks, candidate risk genes, and genome-wide association studies (GWAS) on the genetic side; maternal mood disorders and prenatal drug exposures on the environmental side. Emerging studies on environment-gene interactions will also be reviewed. Our goal was not to perform an exhaustive review of all studies in the field but to leverage some representative ones to summarize the current state, point out potential limitations, and elicit discussions on important future directions.