(no subject)
Does astrology work? We tested the ability of 152 astrologers to see if they could demonstrate genuine astrological skill.
Here is how the study was designed and what we found (including a result that really surprised me).
Back in January, we ran a study trying to predict 37 facts about people's lives using their astrological sun signs (whether they are Pisces, Aries, etc.) While personality tests were able to predict these facts decently well, sun signs couldn't predict even a single one of them...
Some astrologers criticized us for this, saying that sun signs/zodiac signs are just tabloid astrology - real astrologers use a person's entire astrological chart. And they're right! Taking into account this criticism, we got the help of 6 astrologers to design a new study.
Here's how the study worked to test astrologers:
• in each round, each astrologer gets LOTS of information about a real person (answers to 43 questions) along with 5 full astrological charts
• they then predict which is the person's real natal chart (the other 4 are decoys)
Why this study design?
One of the most fundamental claims of astrology is that a person's natal chart contains information about that person's life and character. If true, astrologers should be able to correctly choose a person's chart at a rate well above random guessing.
Each astrologer tries to match people to their correct chart 12 times. If they're guessing completely at random, then they'll get about 20% of questions right, or about 2.4 questions right (on average) out of 12.
Neat aspects of this study design are that
(1) if astrology doesn't work, it's impossible for astrologers to do better than random guessing at this task, while
(2) for the study to come out in support of astrology, astrologers only need to do slightly better than random guessing
But this is only a fair test if astrologers believe they can do this task - so we limit our analyses only to participants with prior astrological experience who predicted they would do better than random guessing at the task. Our results are based on 152 such astrologers. These astrologers were quite confident in their ability to match people to charts. Those with the least experience believed (after they had completed participation) that they'd gotten 5 out of 12 right, and those with the most experience thought they'd gotten 10 out of 12 right.
So, how did astrologers do overall? If they'd gotten even 23% of questions right (slightly above the 20% of random guessing), the study would have come out in favor of astrology. But astrologers as a group performed indistinguishable from random guessing, getting <21% right. We can compare how frequently astrologers got different numbers of questions correct to how often we'd expect them to get different numbers correct if they were all guessing totally at random with no skill. The two distributions match very closely.
But perhaps the less experienced astrologers were just dragging down the performance of the group? We looked at how performance varied based on astrological experience. More experienced astrologers did not do better than less experienced ones despite being far more confident.
Even if most astrologers have no skill, there's another way astrology could prove itself. If even 1 of the 152 astrologers performed exceptionally well, that could provide meaningful evidence for astrology. We offered a $1000 prize for anyone getting at least 11 out of 12. Unfortunately, despite more than half of the astrologers believing that they had gotten 6 or more questions right (after completing the task), in actual fact, not a single astrologer got more than 5 right.
Okay, so despite them believing they could do this task, astrologers seemed to have no ability to match people to their astrological charts. But, even if they aren't getting the answers right, do they at least agree with each other on what the right answers are? Much to my surprise, astrologers had very low agreement with each other on the chart for each person. If astrologers picked charts at random, they would agree with each other 20% of the time. In our study, even the most experienced astrologers only agreed 28% of the time.
In conclusion, despite believing they could do it, the 152 astrologers seemed to lack any ability to match people to their astrological charts.
You can learn a lot more about the study (including its limitations and how we sought to address them) here.
If you believe you have astrological skill, you can try the same questions that we used in the study (and find out the right answers at the end) in order to test yourself.
- I know that I’m talking to the wrong population segment here, but the methodology is inherently flawed. It’s sort of like putting 5 books vs one author, and having people guess which book was written by the person described. It’s technically possible but very difficult.
- Two thoughts on that:
(1) for the study to come out in favor of astrology, they would only have had to do a little better than random guessing. For instance, eliminating one or two options per round would have been plenty.
(2) study participants believed they could do the task - even after finishing they largely predicted that they had done much better than chance.
***
- Interesting. But, why did you put time and effort into proving that astrology is nonsense? I would imagine most sensible people know that, and those that believe in it aren’t going to change their minds.
- (1) Astrology is extremely popular. For instance, more than 20% of Americans at least somewhat believe in it. Most people don't have strong beliefs, though.
(2) Every day there are people hearing about astrology for the first time. It's good to have a recent study available for them to find.
(3) Guessing you don't hang out with a lot of gen Zs. Astrology is becoming far more mainstream and is taken very seriously by a large share of young people.
(4) Perhaps you are stereotyping out the skeptics who lean towards science but are also fascinated by some synchronicities with the stars and all else in the sky.
(5) That's how science works. You don't just assume that because something is "obvious" it must be correct. You test the idea to see if it supported by evidence.
(6) There will always lie tidbits of innovation and discovery hidden among even the most egregiously misinformed theories.
The dismissal of an idea because it seems *obviously* wrong or the fixation on it because it is *obviously* right stifles innovation and prevents discovery because we cannot always rely on our ability to fully comprehend the bigger picture even if it were to be presented to us; we must always dig deeper.
(7) Well it also disproves a "weak" case for astrology to which more people can be receptive: hour/date of birth being predictive of something, even very faintly.
(8) Faith in the context of religion makes plenty of sense. Astrology is different, it’s doesn’t provide existential comfort or a moral foundation. IMO it is net harmful - people make decisions of the highest consequence based on these things, such as mating preference.
***
I've always had a *loose* belief in astrology and find this study super interesting. Definitely has me rethinking.
For over a decade I've been one of those people who is like "he's such a Virgo" or "I really get along with Gemini women" or "I hate Aquariuses" or "we get along because his moon is in Pisces," etc etc. While these aren't firmly held beliefs, a lot of people like me look at trends in the people we know throughout our lives and align them with astrology. While it's not something we're *sure* about by any means, we don't totally disbelieve it either. A LOT of my millennial friends are like this. And yeah this study does have me questioning that. So, props to the folks who conducted it.
Here is how the study was designed and what we found (including a result that really surprised me).
Back in January, we ran a study trying to predict 37 facts about people's lives using their astrological sun signs (whether they are Pisces, Aries, etc.) While personality tests were able to predict these facts decently well, sun signs couldn't predict even a single one of them...
Some astrologers criticized us for this, saying that sun signs/zodiac signs are just tabloid astrology - real astrologers use a person's entire astrological chart. And they're right! Taking into account this criticism, we got the help of 6 astrologers to design a new study.
Here's how the study worked to test astrologers:
• in each round, each astrologer gets LOTS of information about a real person (answers to 43 questions) along with 5 full astrological charts
• they then predict which is the person's real natal chart (the other 4 are decoys)
Why this study design?
One of the most fundamental claims of astrology is that a person's natal chart contains information about that person's life and character. If true, astrologers should be able to correctly choose a person's chart at a rate well above random guessing.
Each astrologer tries to match people to their correct chart 12 times. If they're guessing completely at random, then they'll get about 20% of questions right, or about 2.4 questions right (on average) out of 12.
Neat aspects of this study design are that
(1) if astrology doesn't work, it's impossible for astrologers to do better than random guessing at this task, while
(2) for the study to come out in support of astrology, astrologers only need to do slightly better than random guessing
But this is only a fair test if astrologers believe they can do this task - so we limit our analyses only to participants with prior astrological experience who predicted they would do better than random guessing at the task. Our results are based on 152 such astrologers. These astrologers were quite confident in their ability to match people to charts. Those with the least experience believed (after they had completed participation) that they'd gotten 5 out of 12 right, and those with the most experience thought they'd gotten 10 out of 12 right.
So, how did astrologers do overall? If they'd gotten even 23% of questions right (slightly above the 20% of random guessing), the study would have come out in favor of astrology. But astrologers as a group performed indistinguishable from random guessing, getting <21% right. We can compare how frequently astrologers got different numbers of questions correct to how often we'd expect them to get different numbers correct if they were all guessing totally at random with no skill. The two distributions match very closely.
But perhaps the less experienced astrologers were just dragging down the performance of the group? We looked at how performance varied based on astrological experience. More experienced astrologers did not do better than less experienced ones despite being far more confident.
Even if most astrologers have no skill, there's another way astrology could prove itself. If even 1 of the 152 astrologers performed exceptionally well, that could provide meaningful evidence for astrology. We offered a $1000 prize for anyone getting at least 11 out of 12. Unfortunately, despite more than half of the astrologers believing that they had gotten 6 or more questions right (after completing the task), in actual fact, not a single astrologer got more than 5 right.
Okay, so despite them believing they could do this task, astrologers seemed to have no ability to match people to their astrological charts. But, even if they aren't getting the answers right, do they at least agree with each other on what the right answers are? Much to my surprise, astrologers had very low agreement with each other on the chart for each person. If astrologers picked charts at random, they would agree with each other 20% of the time. In our study, even the most experienced astrologers only agreed 28% of the time.
In conclusion, despite believing they could do it, the 152 astrologers seemed to lack any ability to match people to their astrological charts.
You can learn a lot more about the study (including its limitations and how we sought to address them) here.
If you believe you have astrological skill, you can try the same questions that we used in the study (and find out the right answers at the end) in order to test yourself.
- I know that I’m talking to the wrong population segment here, but the methodology is inherently flawed. It’s sort of like putting 5 books vs one author, and having people guess which book was written by the person described. It’s technically possible but very difficult.
- Two thoughts on that:
(1) for the study to come out in favor of astrology, they would only have had to do a little better than random guessing. For instance, eliminating one or two options per round would have been plenty.
(2) study participants believed they could do the task - even after finishing they largely predicted that they had done much better than chance.
***
- Interesting. But, why did you put time and effort into proving that astrology is nonsense? I would imagine most sensible people know that, and those that believe in it aren’t going to change their minds.
- (1) Astrology is extremely popular. For instance, more than 20% of Americans at least somewhat believe in it. Most people don't have strong beliefs, though.
(2) Every day there are people hearing about astrology for the first time. It's good to have a recent study available for them to find.
(3) Guessing you don't hang out with a lot of gen Zs. Astrology is becoming far more mainstream and is taken very seriously by a large share of young people.
(4) Perhaps you are stereotyping out the skeptics who lean towards science but are also fascinated by some synchronicities with the stars and all else in the sky.
(5) That's how science works. You don't just assume that because something is "obvious" it must be correct. You test the idea to see if it supported by evidence.
(6) There will always lie tidbits of innovation and discovery hidden among even the most egregiously misinformed theories.
The dismissal of an idea because it seems *obviously* wrong or the fixation on it because it is *obviously* right stifles innovation and prevents discovery because we cannot always rely on our ability to fully comprehend the bigger picture even if it were to be presented to us; we must always dig deeper.
(7) Well it also disproves a "weak" case for astrology to which more people can be receptive: hour/date of birth being predictive of something, even very faintly.
(8) Faith in the context of religion makes plenty of sense. Astrology is different, it’s doesn’t provide existential comfort or a moral foundation. IMO it is net harmful - people make decisions of the highest consequence based on these things, such as mating preference.
***
I've always had a *loose* belief in astrology and find this study super interesting. Definitely has me rethinking.
For over a decade I've been one of those people who is like "he's such a Virgo" or "I really get along with Gemini women" or "I hate Aquariuses" or "we get along because his moon is in Pisces," etc etc. While these aren't firmly held beliefs, a lot of people like me look at trends in the people we know throughout our lives and align them with astrology. While it's not something we're *sure* about by any means, we don't totally disbelieve it either. A LOT of my millennial friends are like this. And yeah this study does have me questioning that. So, props to the folks who conducted it.