One of the most stupendous discoveries in astronomy came with the recent application of supernovae -- the explosions of massive stars -- to study the shape and fate of the universe. The shocking results: The universe seems to be pervaded with a previously unknown, and still unexplained, "dark energy" that works against gravity and is apparently accelerating the universe to a bleak fate of eternal expansion.
The roots of this development go back to the 1960s when cosmologists sought two numbers: the Hubble constant (related to the age of the universe) and the "deceleration parameter" (related to the total density of matter in the universe). Astronomers assumed that the universe's expansion was slowing down because all of the mass in the universe would be pulling on all the other mass with enormous gravity. The total mass would then indicate whether the universe would expand forever or eventually re-collapse.
In the 1960s, supernova researchers Stirling Colgate and Robert Wagoner recognized a possible way to measure the deceleration parameter: study the light from distant supernovae. When the light from a supernova is spread out into its component wavelengths (called a spectrum), scientists can look at the shift in wavelength (the redshift) of certain features in the spectrum to see how fast that part of the universe is expanding. Coupled with a measurement of the distance to the supernovae, they could determine whether the supernova light showed that the universe expanded faster in the past. If so, then the expansion must now be slowing down, perhaps enough to some day stop and begin contracting.
To test this theory, astronomers needed automated search programs to find a lot of faint, distant supernovae. Supernovae explode at random, so thousands of galaxies have to be searched to find a single one. It would take decades for the necessary technology to catch up with the dream.
It began to bear fruit in the mid-1990s. A team of scientists led first by Rich Muller and later by Saul Perlmutter began to search for distant supernovae on a shoestring budget, using new cameras mounted on modest-sized telescopes. They could survey enough galaxies in one shot that they were nearly guaranteed to find supernovae. They began to discover them by the dozens. Another team led by Brian Schmidt decided they could more carefully obtain the data needed to determine the deceleration parameter. The race was on.
The shock came when both teams, virtually independently and simultaneously, found that their data suggested that the distant supernovae were a little dimmer than expected. This implied that they were farther away than expected, and hence that the universe must have expanded faster than expected. The universe must have accelerated, not decelerated! This is akin to throwing a ball toward the ceiling, and having it proceed ever faster upward rather than falling back to the floor.
The accelerating, anti-gravitating factor has been labeled "dark energy." It is, somehow, an energy associated with empty space, with the vacuum itself. No fundamental theory of physics yet explains how this energy arises.