Elementary chaos theory
Back in the dark days when I was a mathematician, I worked about as far away from statistics as possible, but since topology doesn't really apply to Scrabble(R) Brand Crossword Game, I've found myself doing some good, old fashioned calculatin' recently. Anyway, since I started using Quackle's Championship Player reports to analyze practice games, I've been wondering just how reliable the results are. So, I took the first game I posted back in May and asked for another report.
I looked at the top five choices for each play in the first report and compared the winning percentages to those the second report came up with, a total of 102 candidate plays once you factor in cases when the report only looked at a few plays, like when a bingo is the only reasonable option. The median difference between reports was 1.0%, which is pretty close, but that's not all of the story. In 14% of the cases, a play that made the top five in the first report didn't even make the list of options in the second report, and some plays that did varied by as much as 6.4%. Results tended to become more chaotic as the game advanced, which I suppose is unsurprising. In the preendgame, when the report starts labeling plays that aren't atop the win percentage list as best, the reports never chose the same play.
Obviously, this is not nearly enough data to impress statisticians or prove anything, but enough to convince me not to just blindly believe the duck's verdict. We might all be better off not obsessing so much over fractions of a percent, but instead using the reports to see scoring opportunities we didn't consider or words we'd have played if we knew them or as a starting point for questioning why a paradoxical-looking play comes out significantly ahead of one we still believe in.
I looked at the top five choices for each play in the first report and compared the winning percentages to those the second report came up with, a total of 102 candidate plays once you factor in cases when the report only looked at a few plays, like when a bingo is the only reasonable option. The median difference between reports was 1.0%, which is pretty close, but that's not all of the story. In 14% of the cases, a play that made the top five in the first report didn't even make the list of options in the second report, and some plays that did varied by as much as 6.4%. Results tended to become more chaotic as the game advanced, which I suppose is unsurprising. In the preendgame, when the report starts labeling plays that aren't atop the win percentage list as best, the reports never chose the same play.
Obviously, this is not nearly enough data to impress statisticians or prove anything, but enough to convince me not to just blindly believe the duck's verdict. We might all be better off not obsessing so much over fractions of a percent, but instead using the reports to see scoring opportunities we didn't consider or words we'd have played if we knew them or as a starting point for questioning why a paradoxical-looking play comes out significantly ahead of one we still believe in.