The evolution of gene families has been studied for over 50 years. We now recognize three different modes of evolution. The two simplest modes are shown below.
Imagine a gene duplication event occurring in a common ancestor at the left-hand side of these trees. Two genes, A and B, are now present in the genome of every species that descends from this common ancestor. In divergent evolution, each of the genes evolves independently after the duplication. Thus, we have two separate phylogenies: one for the A genes and one for the B genes. The two phylogenies will be identical. This is the most common mode of evolution for gene families, especially if the A gene and the B gene are separated in the genome (i.e., on different chromosomes). The classic example is evolution of the α- and β-globin genes in vertebrates.
In concerted evolution, the trees looks like the one on the right. If we assume that the gene duplication event occurred in the last common ancestor of fish, chickens, mice, and humans, then the pattern we see looks very strange. Instead of showing two independent phylogenies, the A and B genes in each species are much more closely related to each other than to family members in any other species. The prototypical example is the evolution of ribosomal RNA genes in all species.
This form of evolution is termed concerted evolution because the pair of genes (A and B) evolved in a concerted manner. They talk to each other. When a mutation occurs in one gene it is transferred to the other so that both genes change in the same direction. The only differences between family members within a species are those that have only become fixed in the very recent past.
The most important mechanism of concerted evolution is gene conversion. This is a form of recombination where the sequence of one gene "converts" the other. It explains how the two genes can communicate. Gene conversion can take place between any two homologous genes in the genome but it is much more common between two homologous genes that are adjacent to each other, especially if they are transcribed in the same direction as a result of a tandem duplication. Gene conversion has been well studied. It is known to produce the results shown in the figure.
There's another way to explain the result in the right-hand tree without invoking concerted evolution. It's possible that our initial assumption is wrong. Perhaps the common ancestor had only a single gene and gene duplications occurred independently within each lineage. This would give a result similar to the tree shown. While this is possible, it seems very unlikely that for a single pair of genes the same duplication would occur in every lineage. With larger genes families such multiple duplication events are more probable.
The third mode of gene family evolution is a combination of the patterns seen in divergent evolution and in concerted evolution. If duplications of family members occur frequently then this gives rise to the birth of new genes. Newborn genes will closely resemble one another, as in concerted evolution. The number of family members does not keep expanding because some of the genes become inactivated—they become pseudogenes and they die. The resulting pattern of evolution will look like a mixture of divergent and concerted evolution. The mode is called "birth-and-death."
The figure below is from a review by Nei and Rooney (2005). Masatoshi Nei is one of the discoverers of birth-and-death evolution (Nei and Hughes, 1992).
Note that in birth-and-death evolution some genes survive in a lineage and some genes are lost. The birth and death of genes can be random or it can be under selection. The point is that not all members of the gene family in the ancestor will show up in all species descending from the common ancestor, and that sometimes several members of the gene family will be much more closely related than you would expect from divergent evolution.
Niimura and Nei (2006) studied the evolution of olfactory receptor genes. In order to study the evolution of gene families you have to be sure you have included every copy of the gene in your study. If you're going to test birth-and-death hypotheses, you also have to include all the pseudognes.
Niimura and Nei (2006) were able to do this for the mouse and human olfactory receptor genes because the complete genomes have been published. By examining the sequences of all genes and pseudogenes, they were able to determine that the most recent common ancestor (MCRA) of mice and humans had 754 functional genes (see figure below). Of these ancestral genes, 691 are still functional in the mouse genome but only 326 remain functional in the human genome.
This study can now be extended because there are complete genome sequences of chickens, frogs, and fish. In addition, there is enough sequence information from lampreys to estimate the number of olfactory receptor genes in that species.
The result is shown above in (b). The ancestor of jawed and jawless fish had two olfactory receptor genes: one type 1 gene and one type 2 gene. Each of these genes gave rise to subfamilies in fish so that the MCRA between fish and tetrapods had six type 1 genes and three type 2 genes. Various members of these subfamilies expanded or contracted in number in the lineages leading to modern fish, amphibians, birds, and mammals. This is birth-and-death evolution.
The patterns produced by birth-and-death evolution look much more like random fluctuations than something produced by sustained directional selection. Niimura and Nei (2006) caution against adaptationist explanations of the differing numbers of genes in various species of mammals. For example, it is widely assumed that the reason mice have more functional olfactory receptor genes than humans is because mice have a better sense of smell. But Niimura and Nei (2006) point out that dogs are supposed to have an excellent sense of smell even though they have fewer OR genes than rodents. The number of genes could be due to chance and not selection. (Or, most likely, a combination of accident and selection.)
Nei, M. and Hughes, A.L. (1992) Balanced polymorphism and evolution by the birth-and-death process in the MHC loci. In 11th Histocompatibility Workshop and Conference, ed. K. Tsuji, M. Aizawa, and T. Sasazuki, pp. 27-28. Oxford, UK: Oxford Univ. Press
Nei, M. and Rooney, A. P. (2005) Concerted and Birth-and-Death Evolution in Multigene Families. Ann. Rev. Genet. 39: 121-152.
Niimura, Y. and Nei, M. (2006) Evolutionary dynamics of olfactory and other chemosensory receptor genes in vertebrates. J. Hum. Genet. 51: 505-517.