instrumentalism vs realism, part 4: KvN
As I mentioned in part 3, I had never heard of the Koopman-von Neumann formulation of classical mechanics until reading Tom Banks' 2011 post about probability in quantum mechanics on cosmic variance.
But finding out about it makes so many things about quantum clear to me that were murky in the past. The main thing that's now crystal clear is this: quantum mechanics is a generalization of statistical mechanics. They aren't really two different theories, rather quantum mechanics is statistical mechanics... it's just that a central assumption of statistical mechanics had to be dropped in light of the evidence.
I had made it most of the way to understanding this when I wrote my series on Wandering Sets in 2013. In some ways, I think it's probably the best thing I've ever written on this blog, even though I think it ended up being too long, meandering, and esoteric for my friends to follow all the way through. I want to write a popular physics book at some point where I explain these ideas more clearly, with pictures and more analogies and examples. What I've learned via KvN solidifies my hunch that QM and SM are really the same theory.
I think one of the first things any student is struck with when they take their first course on quantum mechanics is how different the math is from classical mechanics or stat mech. In classical mechanics, you have lots of differential equations that come from a single important master entity called a Lagrangian, and if you want you can write this in an alternate way as something similar called a Hamiltonian. But all of the variables in the theory just stand for regular real numbers (like 2, pi, 53.8, etc.) that describe the world. In quantum mechanics, you start from the assumption that there is a complex Hilbert space of operators. And you can write down a Hamiltonian, which you're told is an analog of the Hamiltonian used in classical mechanics. The Hamiltonian seemed like a weird way of writing the Lagrangian in classical mechanics, but in quantum mechanics it takes on a more important role. But the "variables" used in the quantum Hamiltonian are not ordinary real numbers, they're operators. These operators correspond to observables (things you can observe about the world), but instead of being a single number they are more like a technique used for making measurements and getting a set of possible results out with associated probabilities. And instead of these operators acting on states in a more familiar space (like the ordinary 3-dimensional space we live in, or the phase space used in statistical mechanics), they act on states in a complex Hilbert space. Complex numbers like 5+i play an important role in this space, and yet as a student there's really no way of understanding why or what the purpose is. You're just asked to accept that if you start with these assumptions, somehow they end up predicting the results of experiments correctly where the corresponding classical predictions fail.
There were many reasons why I ended up leaning towards many worlds rather than other interpretations. I've always preferred representational realism to instrumentalism, so that was one reason. Another was locality (reading David Deutsch's 1999 paper on how quantum mechanics is entirely local as long as you assume that wave functions never collapse was the most influential piece of evidence that convinced me.) But there was a third reason.
The third reason was that whenever I had asked myself "what's the essential difference between classical mechanics and quantum mechanics?" it came down to the idea that instead of regular numbers representing a single outcome, you have operators which represent a set of possible outcomes. In other words, instead of reality being single threaded (one possibility happens at a time), it's multi-threaded. Things operate in parallel instead of in series. This especially resonated with my computing background, and my hope that one day quantum computers would be developed. I knew that it was a little more complicated than just "replace single-threaded process with multi-threaded process", but I thought it was the biggest difference between how the two theories work and what they say.
Learning about the KvN formalism hasn't completely destroyed my preference for Many Worlds, but it has obliterated my view that this is the most important difference between the theories. I now understand that this is just not true.
While I was writing my wandering set series in 2013, I discovered the phase space formalism of quantum mechanics (and discussed it a bit in that series, I believe). This was very interesting to me, and I wondered why it wasn't taught more. It demonstrates that you can write quantum mechanics in a different way, using a phase space like you use in statistical mechanics, instead of using the usual Hilbert space used in quantum mechanics. That was surprising and shocking to me. It hinted that maybe the two theories are more similar than I'd realized. But even more surprising and shocking was my discovery this year of KvN, which shows that you can write statistical mechanics... ordinary classical statistical mechanics... in an alternate formalism using a Hilbert space! What this means is that I was just totally wrong about the number/operator distinction between quantum and classical. This is not a difference in the theories, this is just a difference in how they are written down. Why was I mistaken about this for so long? Because the standard procedure for taking any classical theory and making it a quantum theory is called "canonical quantization", and the procedure says that you just take whatever variables you had in the classical theory and "promote" them to operators. It's true that this is how you can convert one theory to the other, but it's extremely misleading because it obscures the fact that what you're doing is not making it quantum but just rewriting the math in a different way. What makes it quantum is solely the set of commutation relations used!
to be continued in part 5...
But finding out about it makes so many things about quantum clear to me that were murky in the past. The main thing that's now crystal clear is this: quantum mechanics is a generalization of statistical mechanics. They aren't really two different theories, rather quantum mechanics is statistical mechanics... it's just that a central assumption of statistical mechanics had to be dropped in light of the evidence.
I had made it most of the way to understanding this when I wrote my series on Wandering Sets in 2013. In some ways, I think it's probably the best thing I've ever written on this blog, even though I think it ended up being too long, meandering, and esoteric for my friends to follow all the way through. I want to write a popular physics book at some point where I explain these ideas more clearly, with pictures and more analogies and examples. What I've learned via KvN solidifies my hunch that QM and SM are really the same theory.
I think one of the first things any student is struck with when they take their first course on quantum mechanics is how different the math is from classical mechanics or stat mech. In classical mechanics, you have lots of differential equations that come from a single important master entity called a Lagrangian, and if you want you can write this in an alternate way as something similar called a Hamiltonian. But all of the variables in the theory just stand for regular real numbers (like 2, pi, 53.8, etc.) that describe the world. In quantum mechanics, you start from the assumption that there is a complex Hilbert space of operators. And you can write down a Hamiltonian, which you're told is an analog of the Hamiltonian used in classical mechanics. The Hamiltonian seemed like a weird way of writing the Lagrangian in classical mechanics, but in quantum mechanics it takes on a more important role. But the "variables" used in the quantum Hamiltonian are not ordinary real numbers, they're operators. These operators correspond to observables (things you can observe about the world), but instead of being a single number they are more like a technique used for making measurements and getting a set of possible results out with associated probabilities. And instead of these operators acting on states in a more familiar space (like the ordinary 3-dimensional space we live in, or the phase space used in statistical mechanics), they act on states in a complex Hilbert space. Complex numbers like 5+i play an important role in this space, and yet as a student there's really no way of understanding why or what the purpose is. You're just asked to accept that if you start with these assumptions, somehow they end up predicting the results of experiments correctly where the corresponding classical predictions fail.
There were many reasons why I ended up leaning towards many worlds rather than other interpretations. I've always preferred representational realism to instrumentalism, so that was one reason. Another was locality (reading David Deutsch's 1999 paper on how quantum mechanics is entirely local as long as you assume that wave functions never collapse was the most influential piece of evidence that convinced me.) But there was a third reason.
The third reason was that whenever I had asked myself "what's the essential difference between classical mechanics and quantum mechanics?" it came down to the idea that instead of regular numbers representing a single outcome, you have operators which represent a set of possible outcomes. In other words, instead of reality being single threaded (one possibility happens at a time), it's multi-threaded. Things operate in parallel instead of in series. This especially resonated with my computing background, and my hope that one day quantum computers would be developed. I knew that it was a little more complicated than just "replace single-threaded process with multi-threaded process", but I thought it was the biggest difference between how the two theories work and what they say.
Learning about the KvN formalism hasn't completely destroyed my preference for Many Worlds, but it has obliterated my view that this is the most important difference between the theories. I now understand that this is just not true.
While I was writing my wandering set series in 2013, I discovered the phase space formalism of quantum mechanics (and discussed it a bit in that series, I believe). This was very interesting to me, and I wondered why it wasn't taught more. It demonstrates that you can write quantum mechanics in a different way, using a phase space like you use in statistical mechanics, instead of using the usual Hilbert space used in quantum mechanics. That was surprising and shocking to me. It hinted that maybe the two theories are more similar than I'd realized. But even more surprising and shocking was my discovery this year of KvN, which shows that you can write statistical mechanics... ordinary classical statistical mechanics... in an alternate formalism using a Hilbert space! What this means is that I was just totally wrong about the number/operator distinction between quantum and classical. This is not a difference in the theories, this is just a difference in how they are written down. Why was I mistaken about this for so long? Because the standard procedure for taking any classical theory and making it a quantum theory is called "canonical quantization", and the procedure says that you just take whatever variables you had in the classical theory and "promote" them to operators. It's true that this is how you can convert one theory to the other, but it's extremely misleading because it obscures the fact that what you're doing is not making it quantum but just rewriting the math in a different way. What makes it quantum is solely the set of commutation relations used!
to be continued in part 5...