(no subject)
Kind of busy day today. I had a meeting in the morning with Todd Jusko to work on our letter to Neurotoxicology. It seems to be coming along pretty well.
After that I got an email with an interesting problem from Sarah. Basic problem: we have a bunchof samples that have been tested using one assay. We want to compare the sensitivity of this assay with a different assay. Now this is a matched analysis, so infromation on the difference in sensitivities will come from the discordant results. Since we know all these people are positive and that even the new assay should get a positive result most of the time, we will maximize the number of discordant results by testing lots of old negatives and fewer old positives. BUT this means we now can't use a straightforward test. Instead we have to reweight by the inverse probability of choosing a sample. Think of it this way: out of 100 samples and 90 were positive and 10 were negative. We're going to use the new assay on 20 samples: the 10 negatives plus a random 10 positives. We find that 9 of the 10 negatives show a positive on the new assay and 2 of the 10 original positives come up negative. That means the new test is a lot better, right? We gain 7 more right answers and lose only 2. Except that we expect another 16 new negatives among the other 80 unsampled original positives. So we're actually losing 18 to gain those 7. Whoops.
The neat thing here - and I suppose the above was pretty neat when I first learned it but whatever - is that when we do this reweighting we multiplyy the variance of the estimate of the number that are old + and new - by that same factor. So it turns out that preferentially sampling those negaitves isn't really giving us any more power than a simple random sample would. In fact, I'm pretty sure it's exactly the same.
Yet another failed attempt at a free lunch.
And now an evening of Busby Berkeley-watching, Vernor Vinge-meeting and Space Virgin-ConWorksing.
After that I got an email with an interesting problem from Sarah. Basic problem: we have a bunchof samples that have been tested using one assay. We want to compare the sensitivity of this assay with a different assay. Now this is a matched analysis, so infromation on the difference in sensitivities will come from the discordant results. Since we know all these people are positive and that even the new assay should get a positive result most of the time, we will maximize the number of discordant results by testing lots of old negatives and fewer old positives. BUT this means we now can't use a straightforward test. Instead we have to reweight by the inverse probability of choosing a sample. Think of it this way: out of 100 samples and 90 were positive and 10 were negative. We're going to use the new assay on 20 samples: the 10 negatives plus a random 10 positives. We find that 9 of the 10 negatives show a positive on the new assay and 2 of the 10 original positives come up negative. That means the new test is a lot better, right? We gain 7 more right answers and lose only 2. Except that we expect another 16 new negatives among the other 80 unsampled original positives. So we're actually losing 18 to gain those 7. Whoops.
The neat thing here - and I suppose the above was pretty neat when I first learned it but whatever - is that when we do this reweighting we multiplyy the variance of the estimate of the number that are old + and new - by that same factor. So it turns out that preferentially sampling those negaitves isn't really giving us any more power than a simple random sample would. In fact, I'm pretty sure it's exactly the same.
Yet another failed attempt at a free lunch.
And now an evening of Busby Berkeley-watching, Vernor Vinge-meeting and Space Virgin-ConWorksing.