It seemed unlikely that this disparity could have evolved in just a few million years so many scientists have been searching for fossil antecedents in the early Cambrian and Ediacaran (635-541 Ma). Many trace fossils have been found in the past few decades, indicating that the fossil animals of the Cambrian were preceded by small wormlike creatures.
The other approach has been sequence analysis. One can construct molecular phylogenies by comparing the sequences of genes in modern extant organisms. This approach has been highly successful over the past fifty years so that we now know a great deal about the relationship of the various animal phyla. The correspondence between the old morphological taxonomy and molecular evolution is the most powerful evidence we have that evolution explains the history of life [see Twin Nested Hierarchies].
The problem with sequence comparisons has always been getting accurate dates using the molecular clock. It is hard to get an accurate date when dealing with events that occurred 500 million years ago because there aren't very many calibration points. An accurate calibration point is a known time when two lineages diverge.
If there really was a rapid divergence in the Cambrian then one would expect the molecular tree to show this. But it never has. The molecular phylogeny shows that chordates diverged from invertebrates at least one hundred million years before their fossils appear in the Cambrian. Similarly other phyla and classes of animals have their origin long before the Cambrian, according to the molecular clock.
A recent paper in Science extends this comparison by calculating more a more accurate molecular phylogeny using seven housekeeping genes from 118 different species (Erwin et al. 2011). The result is shown in Figure 1 of the paper: "The origin and diversification of animals as inferred from the geologic and genetic fossil records." (Click on the figure to embiggen.)
This is a complicated figure. The fist part shows the rise of new phyla (blue) and new classes (yellow) over time. The scale of taxa numbers is on the left. The rapid rise in the number of phyla and classes, as measured by the fossil record, is evident at the beginning of the Cambrian era (green bars on the y-axis).
The molecular tree is shown as an overlay with the various species on the right-hand x-axis. As you can see, the branch points (nodes) for the various classes and phyla pre-date the Cambrian by millions of years. There is no obvious correlation between the fossil record and the phylogeny.1
The authors refer to this difference as the "Cambrian Conundrum." The problem has been apparent for about 25 years and it seems no closer to being solved than it was in the 1980s. The only difference is that now developmental biology has show us how easy it is to evolve very different organisms with only small changes in regulating gene expression. That part isn't a problem. Whether these developmental changes were triggered by environmental changes, such as the oxygenation of the oceans or the rise of predators, remains an open question.
Here's the abstract of the paper.
Diverse bilaterian clades emerged apparently within a few million years during the early Cambrian, and various environmental, developmental, and ecological causes have been proposed to explain this abrupt appearance. A compilation of the patterns of fossil and molecular diversification, comparative developmental data, and information on ecological feeding strategies indicate that the major animal clades diverged many tens of millions of years before their first appearance in the fossil record, demonstrating a macroevolutionary lag between the establishment of their developmental toolkits during the Cryogenian [(850 to 635 million years ago (Ma)], and the later ecological success of metazoans during the Ediacaran (635 to 541 Ma) and Cambrian (541 to 488 Ma) periods. We argue that this diversification involved new forms of developmental regulation, as well as innovations in networks of ecological interaction within the context of permissive environmental circumstances.
1. Note that this does not conflict with the twin nested hierarchies. It's a different problem—one that deals with dates and not relationships.
Erwin, D.H., Laflamme, M., Tweedt, S.M., Sperling, E.A., Pisani, D., and Peterson, K.J. (2011) The Cambrian conundrum: early divergence and later ecological success in the early history of animals. Science 334:1091-1097. [PubMed] [doi: 10.1126/science.1206375]