Do my homework
So a colleague in math ed has a hypothesis and does two experiments ("fall" and "spring") which each produce evidence in its favor.
But the sample sizes are pretty small and the instruments are pretty crude, so neither experiment makes it across whatever the current arbitrary bar for "statistical significance" is. Another colleague in ed ed wants to increase the sample size by just combining the data sets, but this would be bullshit because the fall and spring experiments are in many ways apples and oranges.
This raises the question: Is there a theory of composite experiments? That is, can we view the two experiments together as two independent observations, and claim that (proctonumerology: ) observing something with 10% likelihood both times is a 1% event, so both experiments showing significance at the 90% level is actually significant at the 99% level?
Or, if independence is too strong a claim, is there some weaker version of this?
But the sample sizes are pretty small and the instruments are pretty crude, so neither experiment makes it across whatever the current arbitrary bar for "statistical significance" is. Another colleague in ed ed wants to increase the sample size by just combining the data sets, but this would be bullshit because the fall and spring experiments are in many ways apples and oranges.
This raises the question: Is there a theory of composite experiments? That is, can we view the two experiments together as two independent observations, and claim that (proctonumerology: ) observing something with 10% likelihood both times is a 1% event, so both experiments showing significance at the 90% level is actually significant at the 99% level?
Or, if independence is too strong a claim, is there some weaker version of this?