Angelina
Specific to ancestry, we call the phenomenon this. https://en.m.wikipedia.org/wiki/Pedigree_collapse
Regardless of interbreeding between anatomically modern humans and the Neanderthals and Denisovans, the 2% @curious mentions is specific to populations we see outside of Africa, because those two lineages (and their hybrids) had already left the continent long before our last universal common ancestor (LUCA.)
especially of times when there weren't so many people yet
Genetic studies show that humans most recent bottleneck was between 50,000 and 100,000 years ago, when the population may have been reduced to between 3,000 and 10,000 breeding individuals. One hypothesis revolves around climate impacts from the eruption of the Toba supervolcano in Sumatra 73,000 years ago causing drastic cooling to the Earth's surface for a 1,000 year period.
Whatever the cause, that bottleneck gives humans, even with the previous interbreeding opportunities we had with other human lineages, a lower overall genetic diversity than most other mammalian species.