The first thing you have to know is that polar bears and brown bears are not distinct species as defined by the biological species concept. This definition of "species" requires that there be no interbreeding with other populations, otherwise it's not a true species. (See Jerry Coyne's analysis of the problem with respect to humans.)
Polar bears and brown bears can mate to form hybrids, although this doesn't happen very often in the wild because the two "species" are geographically separated. The interesting question is when did these two populations diverge.
The result is shown here on the left in a figure taken from the recent paper in Science (Hailer et al, 2012). The Lindvist et al. paper showed conclusively that polar bears are relatively young as a "species" (about 150,000 years) and that they clearly arose from within the brown bear clade.
This result was somewhat surprising but not revolutionary. However, there were some experts on speciation who, questioned the conclusion; notably, Jerry Coyne [Do “polar bears” exist?]. They cautioned that using mitochondrial DNA could be misleading.
The latest study in Science shows that their concerns were justified. Hailer et al. (2012) lookd at 14 nuclear genes and constructed the tree shown shown below on the right.
What does this mean? It probably means that about 150,000 years ago there was a brief fling between a male polar bear and a female brown bear. The female offspring, carrying brown bear mitochondria, mated with male polar bears and all of their progeny contained brown bear mitochondria with mixtures of brown and polar bear alleles in their nuclei. Over time, the brown bear mitochondria became fixed by random genetic drift in the polar bear population.
This explains why the two trees differ and why you have to be cautious about using mitochondrial DNA in constructing phylogenies.
The story doesn't end there.
Just this week, a new paper appeared in PNAS confirming this result (Miller et al., 2012). Many of the authors are the same ones on the Lindqvist et al. (2012) paper. They are refuting the conclusions of their earlier paper by now showing that the sequence of nuclear genes indicate an earlier divergence of brown bears and polar bears.
I put off blogging about the Science paper because there were more important things to do and because it would have been difficult to explain the problem and the explanation. Fortunately, Jerry Coyne has now taken up the task of explaining the result at: A new study of polar bears underlines the dangers of reconstructing evolution using mitochondrial DNA. Please read his explanation of gene flow in bears based on his extensive knowledge of speciation.
The only issue I have with Coyne's explanation is that I don't think he gives enough emphasis to the idea that fixation of brown bear mitochondria in polar bears could be a genetic accident.
UPDATE: Ed Yong of Not Exactly Rocket Science already wrote about this three months ago [Polar bear origins revised – they’re older and more distinct than we thought]. John Hawks emphasizes Jerry Coyne's warning about using mitochondrial DNA and discuss the implications for hominid evolution [Polar bear mtDNA replacement].
Hailer F, Kutschera VE, Hallström BM, Klassert D, Fain SR, Leonard JA, Arnason U, Janke A. (2012) Nuclear genomic sequences reveal that polar bears are an old and distinct bear lineage. Science 336:344-347. [Abstract] [DOI: 10.1126/science.1216424]
Lindqvist, C., Schuster, S.C., Sun, Y., Talbot, S.L., Qi, J., Ratan, A., Tomsho, L.P., Kasson, L., Zeyl, E., Aars, J., Miller, W., Ingólfsson, O., Bachmann, L., and Wiig, O. (2010) Complete mitochondrial genome of a Pleistocene jawbone unveils the origin of polar bear. Proc. Natl. Acad. Sci. (USA) 107:5053-5057. [doi:10.1073/pnas.0914266107]
Millera, W., Schuster, S.C., Welch, A.J., Ratan, A., Bedoya-Reina, O.C., et al, (2012) Polar and brown bear genomes reveal ancient admixture and demographic footprints of past climate change. Proc. Natl. Acad. Sci. (USA) Published online before print July 23, 2012. [doi: 10.1073/pnas.1210506109]