Optimization & Learning
I've been looking into optimization algorithms recently. That is, given f(x), find x to minimize f(x) where x may be multidimensional (or even infinite-dimensional). It turns out that surprisingly many useful problems can be cast as an optimization problem. For example, solving the equation Ax=b (where A is a symmetric matrix) can be thought of as looking for x that minimizes the “residual”, r = Ax−b.
As another example, consider designing a ramp for a marble. Suppose the marble starts at (0, 1) and you want it to roll to (1, 0) as quickly as possible subject to gravity. You want to optimize the shape of the track (x) with respect to rolling time (f).
An interesting thing about these and many other useful problems (as I understand it) is that these functions are convex. In a loose sense sense, convex functions are those for which moving “down hill” always gets you closer to the minimum. Since there's always a clear way to improve a given solution, it should be no surprise that there are many iterative solvers for convex optimization problems (conjugate gradient, for example). (As it turns out, simply wandering down hill isn't necessarily the best way to go.)
Some problems are only locally convex. That is, there may be many local minimums. Given a starting point, an optimization algorithm will be able to work its way towards the local minimum, like rainwater finding its way to a pond or a lake rather than to the ocean.
Learning
With the above in mind, I overheard someone talking about how he recently improved his golf swing. Like many, he had some notion that he needed to twist his hips and hands in just the right ways. He had become frustrated with all of those details and his hit hadn't improved. Recently, though, he had been shown that the important part was where the club's head was and where it was moving upon impact - that the rest is just to get the club head there.
In terms of optimization, the objective function is the distance you send the ball (with some accounting for accuracy); the function being optimized is the path his body and club take through the swing. It strikes me that this isn't a convex optimization problem - that there are many local “best swings” even if there's only one (global) perfect swing. His previous advice, involving hips and wrists and things, started him out in a particular part of “swing space” from which no amount of independent practice would get him to converge on the global optimum.
The advice he got was simply to pay attention to how the club was hitting the ball, along with some simple instructions about where his hands should go. This moved him to a different part of swing space from which he could improve without supervision by practicing on his own. That is, a teacher can see when we are stuck in a local optimum and suggest a better place from which to start looking.
In other words, if a horse is thirsty, you needn't lead it to water, you just need to lead it to a valley with a pond in it; it's all down hill from there.
I find all this a particularly satisfying explanation of why practice works but why teachers are tremendously helpful, even when their advise is simple.
As another example, consider designing a ramp for a marble. Suppose the marble starts at (0, 1) and you want it to roll to (1, 0) as quickly as possible subject to gravity. You want to optimize the shape of the track (x) with respect to rolling time (f).
An interesting thing about these and many other useful problems (as I understand it) is that these functions are convex. In a loose sense sense, convex functions are those for which moving “down hill” always gets you closer to the minimum. Since there's always a clear way to improve a given solution, it should be no surprise that there are many iterative solvers for convex optimization problems (conjugate gradient, for example). (As it turns out, simply wandering down hill isn't necessarily the best way to go.)
Some problems are only locally convex. That is, there may be many local minimums. Given a starting point, an optimization algorithm will be able to work its way towards the local minimum, like rainwater finding its way to a pond or a lake rather than to the ocean.
Learning
With the above in mind, I overheard someone talking about how he recently improved his golf swing. Like many, he had some notion that he needed to twist his hips and hands in just the right ways. He had become frustrated with all of those details and his hit hadn't improved. Recently, though, he had been shown that the important part was where the club's head was and where it was moving upon impact - that the rest is just to get the club head there.
In terms of optimization, the objective function is the distance you send the ball (with some accounting for accuracy); the function being optimized is the path his body and club take through the swing. It strikes me that this isn't a convex optimization problem - that there are many local “best swings” even if there's only one (global) perfect swing. His previous advice, involving hips and wrists and things, started him out in a particular part of “swing space” from which no amount of independent practice would get him to converge on the global optimum.
The advice he got was simply to pay attention to how the club was hitting the ball, along with some simple instructions about where his hands should go. This moved him to a different part of swing space from which he could improve without supervision by practicing on his own. That is, a teacher can see when we are stuck in a local optimum and suggest a better place from which to start looking.
In other words, if a horse is thirsty, you needn't lead it to water, you just need to lead it to a valley with a pond in it; it's all down hill from there.
I find all this a particularly satisfying explanation of why practice works but why teachers are tremendously helpful, even when their advise is simple.