Widespread interest in the first successful Out of Africa dispersal of modern humans ∼60-80 thousand years ago via a southern migration route has overshadowed the study of later periods of South Arabian prehistory. In this work, we show that the post-Last Glacial Maximum period of the past 20,000 years, during which climatic conditions were becoming more hospitable, has been a significant time in the formation of the extant genetic composition and population structure of this region. This conclusion is supported by the internal diversification displayed in the highly resolved phylogenetic tree of 89 whole mitochondrial genomes (71 being newly presented here) for haplogroup R0a-the most frequent and widespread haplogroup in Arabia. Additionally, two geographically specific clades (R0a1a1a and R0a2f1) have been identified in non-Arabic speaking peoples such as the Soqotri and Mahri living in the southern part of the Arabian Peninsula where a past refugium was identified by independent archaeological studies. Estimates of time to the most recent common ancestor of these lineages match the earliest archaeological evidence for seafaring activity in the peninsula in the sixth millennium BC.