A system for automatic detection and characterization of cracks in road flexible pavement surfaces is proposed. It does not require manually labeled samples to minimize human subjectivity. Crack detection is based on unsupervised training of a learning from samples paradigm using a subset of the image database. The system classifies image blocks as containing crack pixels or not. Crack type characterization is accomplished by constructing another classification system to label cracks according to types defined in the Portuguese Distress Catalog. A novel methodology for assigning crack severity levels is introduced. The system’s experimental results are based on images captured during a visual road pavement surface survey over Indian roads. The results show promise, and a quantitative evaluation methodology is introduced, including a comparison with human experts’ manual labeling results. To evaluate the performance of our method, we use various metrics such as precision, recall, and F1-score. We compare our results with manual inspection and other traditional methods to assess the effectiveness of our deep learning-based approach.