Sometimes the invading alleles can become fixed (or very prevalent) in the population. This is what is presumed to have happened with Neanderthal or Denisovan alleles in modern humans. Rare matings between modern human ancestors and Neanderthal, for example, led to some gene flow between the populations [How many species of humans were there?].
The spread of invasive mitochondria is an extreme example of the fixation of invasive alleles. The example of polar bear evolution [Speciation in Bears] shows us that mitochondrial DNA can enter a population through a rare mating with an individual from another "species" and then become fixed in the new population over many thousand years.
The question is how does this fixation of "foreign" mitochondria occur? Is it an accident due to random genetic drift or do the foreign mitochondria confer a selective advantage over the original mitochondria? It seems unlikely that the mitochondria in one population (e.g. brown bears) are better than those in a foreign population (e.g. polar bears) that has adapted to a different environment.
John Hawks seems to want to have his cake and eat it too [Polar bear mtDNA replacement] ...
In any single population, the behavior of mtDNA is rarely outside the very wide range of dynamics that happen by genetic drift alone, but that's more a sign of the extremely wide range of possibilities that drift allows. (This is why it took so long to demonstrate a problem with mtDNA in phylogenetic reconstruction). Now we know of many instances like polar bears, where the mtDNA genealogy has a different topology than that typical of nuclear genes. Moreover, we know that across many populations of different species, mtDNA is systematically less variable than the expected ratio from the nuclear genome. So it seems that once it enters a population, mitochondrial DNA sometimes spreads more rapidly and broadly than the typical gene. This dynamic sometimes may reflect extreme population histories, such as population bottlenecks and large-scale migrations. But in many cases it probably reflects selection on the mitochondrial genome.This is one of those issues where a little bit of evidence would be extremely helpful. Is it true that the fixation of foreign mitochondrial DNA is more common than, say, the fixation of Y chromosomes or the fixation of X-linked genes? What about other genes? Does maternal inheritance of mitochondria influence fixation?
Jerry Coyne suggests that there's something special about mitochondria (and chloroplasts) [A new study of polar bears ...].
The problem is that, for reasons we don’t fully understand, mtDNA also moves between species during hybridization much more readily than does nDNA, and that can screw up species relationships. This is true for both animals and plants, and not just for mtDNA either: in plants, DNA from another organelle, the chloroplasts (site of photosynthesis, this DNA is called “cpDNA”) also moves between species more readily than nDNA.Is it really true that mtDNA is more commonly fixed than nuclear alleles? Is it still true when you take into account maternal inheritance? I hope someone assigns this problem to a graduate student and I hope the graduate student takes care to rule out drift and accident.
We see this situation over and over again in biology. We don’t really know why mtDNA (and cpDNA) leak so readily between species, but we do know that this leakage makes it dicey to use only organelle’s DNA to make species trees. But the reason for this leakage compared to nDNA (so common to be almost a “rule of biology”) would make a useful paper topic for some enterprising graduate student.
Image Credit: Moran, L.A., Horton, H.R., Scrimgeour, K.G., and Perry, M.D. (2012) Principles of Biochemistry 5th ed., © Pearson Education Inc. page 419 [Pearson: Principles of Biochemistry 5/E]