Sysyphus, misunderstood

[misha_b]

It is often believed that Sysyphus was a king who was forced to forever roll a stone up the mountain slope as a punishment by Zeus for cheating death.

The truth, as is often the case with the Greek myths, is different. In fact, Sysyphus was not a king at all. He was an investor and operator of an early manual gravity energy storage device.

Comments

[anhinga_anhinga]

!

[misha_b]

I've read a comment somewhere recently that it is stupid to heat you house with electricity without mining some bitcoin first :)

[anhinga_anhinga]

I have a friend who used to heat his house by running his bitcoin mining servers in the early years (but was not wise enough to keep those bitcoin).

Offtopic: I presume you've seen the new "Grokking" paper by a small OpenAI team at "The Role of Mathematical Reasoning in General Artificial Intelligence" ICLR workshop :-)

[misha_b]

I took a quick look at the paper. It seems like a nice demonstration of something we already knew. Am I missing something?

[anhinga_anhinga]

I noticed 3 things:

1) The effect in the left image of Figure 1 is quite striking. Figure 4 is also quite remarkable.

2) It might be the case that for precisely defined synthetic tasks the effects tend to be "more pronounced" (and tend to lead to the ability to solve the task exactly). It's premature to make this kind of general pronouncement, especially about the ability to solve the task exactly, but this paper seems to push us to at least consider this kind of conjecture.

3) If 2 is actually true, then one notices that these are the conditions for a typical program synthesis problem (a modestly-sized problem precisely defined by few constraints (expected test results)). So it might be that modestly-sized models (like a small transformer used in this paper) will actually be able to solve these tasks, because the data size is really small, so the desirable "superoverfitting area" is not too far...

[misha_b]

It took me some time to take a more careful look :)

1) ->
Agreed, both are pretty striking. Although still in line with what we have seen all along.

2) ->
It is an interesting conjecture but I would like to see more evidence.

3) I suspect that it is exactly the case. For small problems the highly over-parameterized solution is still within reach. For large problems of this type it may be impossible to get to it at all.