Genuine Scientific Mystery - The Pioneer Anomaly and Dark Matter / Energy
I've occasionally heard scientists criticized for not appreciating mystery. "Scientists want to take apart something that seems unusual or counterintuitive. They want to explain and measure and figure it out. They don't appreciate mystery in the world. They can't accept things they can't explain or don't know."
Nothing could be further from the truth. There is nothing more enchanting and exciting for scientists than a good mystery. But scientists love real genuine mysteries so much that they're very careful about making sure that something is an actual mystery before they get really excited, because mistakes are a lot more common than genuine mysteries.
The Pioneer Anomaly:
The Pioneer Anomaly is an excellent example. In 1980, John Anderson of JPL noticed that Pioneer 10 was 3000 miles away from where it ought to be, and exiting the solar system slightly slower than we thought it should. At the time, nobody assumed that this was anything special. There were a lot of ways this could be explained with what we already know: fuel leaks, pressure from sunlight, thermal pressure, electrostatic forces from a charged spacecraft, drag from collisions with dust, gravity from the kuiper belt or dark matter, relativity and frame dragging, math errors, observational errors. Then a closer look at Pioneer 11 started showing the same anomaly on the other end of the solar system. These and other observations ruled out some of the more obvious explanations. In early March of this year, the same anomaly was discovered in probes not on the edge of the solar system but nearby as they're doing gravity assists. Of six satellites studied, four of them exhibited the anomaly. Recently, Anderson's team have discovered a formula that seems to address the discrepancies post hoc, but nobody knows what the extra variables are or why they're doing what they're doing. What's interesting about the formula is that the anomaly isn't uniform; it varies depending on the angle that the spacecraft's incoming and outgoing trajectories make with respect to Earth's equator. (more) Now that they think they know what to look for, they're resurrecting the complete data from Pioneer's 30+ year mission to make sure it's consistent or see when the inconsistency might have started appearing.
Scientists can't explain the pioneer anomaly - they're genuinely stumped. The Pioneer Anomaly is a genuine mystery, and scientists are absolutely nuts with excitement. Our understanding of a widely accepted fact of the universe - gravity - might be wrong insufficiently precise. I get goosebumps just thinking about it.
Dark Matter and Energy:
Dark Matter is another serious mystery. Back in 1933 (before World War Two) Fritz Zwicky used galaxies' rotatation to estimate their mass and compared this figure to the amount of stuff we could actually observe inside them. From his observations it looked like the mass of the galaxies he was observing was 400 times bigger than the galaxy actually appeared to be. Galaxies were rotating too fast for their apparent masses to explain.
Nobody paid this much thought, because no single observation proves that something is true, and no single anomalous data point proves that something is false. But in the 1970s Vera Rubin discovered that the velocity of rotation was more constant than it ought to be. Instead of interior stars moving faster than outer stars like a cosmic game of crack the whip, the movement of stars is more uniform than we'd expect. Galaxies don't actually have the big central masses of matter that it looks like they have - their density is closer to being uniform. Lots of conventional explanations were explored, starting with "Vera Rubin is probably wrong about what she thinks she saw", but the observations held up. Skeptics saw the same thing. Something that we couldn't see really was there. Other explanations were graduallly ruled out. Eventually the explanation that requires the least adjustment to the physical laws of the universe was that there is a lot more gravity than visible matter generates. Something else is doing it.
In 1998, Lawrence Berkeley Lab noticed some evidence indicating that the universe wasn't just expanding, its expansion was accelerating. The universe isn't just getting bigger, it's getting bigger faster. It's not just the big bang; something is continuing to push it.
It's not just little traces of dark matter or energy that we're talking about, either. Observations from the WMAP satellite indicate that 73% of everything in the universe is dark energy. Of the 27% that is matter, 4% is matter that we can interact with. Scientists have no idea why we can't see dark matter, or what it actually is. They're genuinely stumped. Dark matter is a genuine mystery, and scientists are absolutely nuts with excitement over it. We can't interact with 96% of the universe. If that doesn't give you goosebumps I don't know what will.
There are major differences between genuine scientific mysteries and pseudoscientific or supernatural mysteries. For one, scientists spend decades ruling out mundane explanations. They start by assuming that what they're seeing can be explained by something simple, natural, and obvious. No one jumps to crazy conclusions until the simple stuff is ruled out multiple times. Extraordinary claims require extraordinary evidence, and it's taken a lot of extraordinary evidence to reach the point where new physics, new matter, or new fundamental forces of the universe are the simplest and most reasonable explanations.
Another difference is that we actually know that the thing in question is a mystery. A distinct hole in our knowlege has resisted being filled by conventional explanations for decades or longer. It's not just a few cases where we don't have enough information - we have gathered sufficient information to verify that something mysterious is definitely going on. To quote Rumsfeld our investigation has uncovered "known unknowns", not "unknown unknowns". Like the Flying Spaghetti Monster we still don't know what exactly it is, but unlike the Flying Spaghetti Monster we've got some very strong evidence that these anomalies - whatever they are - actually exist.
Genuine science can be tested or verified with empirical evidence. Even if we don't know what it is, knowing what it isn't suggests way to prove that it isn't. Using better methodologies to observe the unknown phenomenon closer and look more carefully produces clearer and more obvious results. The thing we're examining doesn't get smaller the more we look at it. When we find more evidence it means that we're onto something. *Not* finding evidence or finding evidence that decreases as your methodology gets better means that those explanations are probably not true. Finding unambiguous evidence in one direction and a lack of evidence in other directions is a pretty good sign that we're onto something.
Science is validated by prospective studies, not just retrospective explanations. When you think you know something you can using your knowlege to make testable predictions. Even if you don't know what you're looking at, knowing what it isn't suggests way to prove that it isn't. Finding answers where you think they ought to be increases your confidence. Not finding answers where you think they ought to be casts doubt on your theory. That's the problem with using creationism to explain the Pioneer Anomaly - it doesn't make any predictions. Before Pioneer nobody was saying "if God created the universe we ought to see a small deceleration in spacecraft over long distances". After Pioneer I can't find any way to verify this explanation. "If God's act of creation is causing this deceleration we ought to also see this influence - and only this influence - appearing in specific other settings."
Genuine scientific mysteries are verified by skeptics, who are welcomed and included as a part of the scientific process. Some of those skeptics are the discoverers themselves, who are genuinely interested in knowing if they're not dealing with something legitimate. Twenty years ago there were a lot of skeptics who assumed with good reason that the anomaly must be a mistake. Decades of challenges from skeptics were welcomed before the phenomenon was verified because everyone wanted to be right. When science is done right, nobody wants to be wrong.
Simple mistakes and mistaken assumptions are a lot more common than genuine mysteries. Scientists love a genuine mystery, but it takes good evidence and a bit of patience to convince them that's what they're looking at. That careful effort is what pays off, and is what makes legitimate scientific mysteries so fascinating.
I am a somewhat well-informed layman, not a professional physicist or astronomer, and I have no doubt that I have made mistakes here. Corrections are welcome, as always.
Nothing could be further from the truth. There is nothing more enchanting and exciting for scientists than a good mystery. But scientists love real genuine mysteries so much that they're very careful about making sure that something is an actual mystery before they get really excited, because mistakes are a lot more common than genuine mysteries.
The Pioneer Anomaly:
The Pioneer Anomaly is an excellent example. In 1980, John Anderson of JPL noticed that Pioneer 10 was 3000 miles away from where it ought to be, and exiting the solar system slightly slower than we thought it should. At the time, nobody assumed that this was anything special. There were a lot of ways this could be explained with what we already know: fuel leaks, pressure from sunlight, thermal pressure, electrostatic forces from a charged spacecraft, drag from collisions with dust, gravity from the kuiper belt or dark matter, relativity and frame dragging, math errors, observational errors. Then a closer look at Pioneer 11 started showing the same anomaly on the other end of the solar system. These and other observations ruled out some of the more obvious explanations. In early March of this year, the same anomaly was discovered in probes not on the edge of the solar system but nearby as they're doing gravity assists. Of six satellites studied, four of them exhibited the anomaly. Recently, Anderson's team have discovered a formula that seems to address the discrepancies post hoc, but nobody knows what the extra variables are or why they're doing what they're doing. What's interesting about the formula is that the anomaly isn't uniform; it varies depending on the angle that the spacecraft's incoming and outgoing trajectories make with respect to Earth's equator. (more) Now that they think they know what to look for, they're resurrecting the complete data from Pioneer's 30+ year mission to make sure it's consistent or see when the inconsistency might have started appearing.
Scientists can't explain the pioneer anomaly - they're genuinely stumped. The Pioneer Anomaly is a genuine mystery, and scientists are absolutely nuts with excitement. Our understanding of a widely accepted fact of the universe - gravity - might be wrong insufficiently precise. I get goosebumps just thinking about it.
Dark Matter and Energy:
Dark Matter is another serious mystery. Back in 1933 (before World War Two) Fritz Zwicky used galaxies' rotatation to estimate their mass and compared this figure to the amount of stuff we could actually observe inside them. From his observations it looked like the mass of the galaxies he was observing was 400 times bigger than the galaxy actually appeared to be. Galaxies were rotating too fast for their apparent masses to explain.
Nobody paid this much thought, because no single observation proves that something is true, and no single anomalous data point proves that something is false. But in the 1970s Vera Rubin discovered that the velocity of rotation was more constant than it ought to be. Instead of interior stars moving faster than outer stars like a cosmic game of crack the whip, the movement of stars is more uniform than we'd expect. Galaxies don't actually have the big central masses of matter that it looks like they have - their density is closer to being uniform. Lots of conventional explanations were explored, starting with "Vera Rubin is probably wrong about what she thinks she saw", but the observations held up. Skeptics saw the same thing. Something that we couldn't see really was there. Other explanations were graduallly ruled out. Eventually the explanation that requires the least adjustment to the physical laws of the universe was that there is a lot more gravity than visible matter generates. Something else is doing it.
In 1998, Lawrence Berkeley Lab noticed some evidence indicating that the universe wasn't just expanding, its expansion was accelerating. The universe isn't just getting bigger, it's getting bigger faster. It's not just the big bang; something is continuing to push it.
It's not just little traces of dark matter or energy that we're talking about, either. Observations from the WMAP satellite indicate that 73% of everything in the universe is dark energy. Of the 27% that is matter, 4% is matter that we can interact with. Scientists have no idea why we can't see dark matter, or what it actually is. They're genuinely stumped. Dark matter is a genuine mystery, and scientists are absolutely nuts with excitement over it. We can't interact with 96% of the universe. If that doesn't give you goosebumps I don't know what will.
There are major differences between genuine scientific mysteries and pseudoscientific or supernatural mysteries. For one, scientists spend decades ruling out mundane explanations. They start by assuming that what they're seeing can be explained by something simple, natural, and obvious. No one jumps to crazy conclusions until the simple stuff is ruled out multiple times. Extraordinary claims require extraordinary evidence, and it's taken a lot of extraordinary evidence to reach the point where new physics, new matter, or new fundamental forces of the universe are the simplest and most reasonable explanations.
Another difference is that we actually know that the thing in question is a mystery. A distinct hole in our knowlege has resisted being filled by conventional explanations for decades or longer. It's not just a few cases where we don't have enough information - we have gathered sufficient information to verify that something mysterious is definitely going on. To quote Rumsfeld our investigation has uncovered "known unknowns", not "unknown unknowns". Like the Flying Spaghetti Monster we still don't know what exactly it is, but unlike the Flying Spaghetti Monster we've got some very strong evidence that these anomalies - whatever they are - actually exist.
Genuine science can be tested or verified with empirical evidence. Even if we don't know what it is, knowing what it isn't suggests way to prove that it isn't. Using better methodologies to observe the unknown phenomenon closer and look more carefully produces clearer and more obvious results. The thing we're examining doesn't get smaller the more we look at it. When we find more evidence it means that we're onto something. *Not* finding evidence or finding evidence that decreases as your methodology gets better means that those explanations are probably not true. Finding unambiguous evidence in one direction and a lack of evidence in other directions is a pretty good sign that we're onto something.
Science is validated by prospective studies, not just retrospective explanations. When you think you know something you can using your knowlege to make testable predictions. Even if you don't know what you're looking at, knowing what it isn't suggests way to prove that it isn't. Finding answers where you think they ought to be increases your confidence. Not finding answers where you think they ought to be casts doubt on your theory. That's the problem with using creationism to explain the Pioneer Anomaly - it doesn't make any predictions. Before Pioneer nobody was saying "if God created the universe we ought to see a small deceleration in spacecraft over long distances". After Pioneer I can't find any way to verify this explanation. "If God's act of creation is causing this deceleration we ought to also see this influence - and only this influence - appearing in specific other settings."
Genuine scientific mysteries are verified by skeptics, who are welcomed and included as a part of the scientific process. Some of those skeptics are the discoverers themselves, who are genuinely interested in knowing if they're not dealing with something legitimate. Twenty years ago there were a lot of skeptics who assumed with good reason that the anomaly must be a mistake. Decades of challenges from skeptics were welcomed before the phenomenon was verified because everyone wanted to be right. When science is done right, nobody wants to be wrong.
Simple mistakes and mistaken assumptions are a lot more common than genuine mysteries. Scientists love a genuine mystery, but it takes good evidence and a bit of patience to convince them that's what they're looking at. That careful effort is what pays off, and is what makes legitimate scientific mysteries so fascinating.
I am a somewhat well-informed layman, not a professional physicist or astronomer, and I have no doubt that I have made mistakes here. Corrections are welcome, as always.