Dataflow matrix machines as generalized recurrent neural networks

[anhinga_anhinga]

A year ago I posted about dataflow programming and linear models of computation:

http://anhinga-anhinga.livejournal.com/82757.html

It turns out that those dataflow matrix machines are a fairly powerful generalization of recurrent neural networks.

Comments

[datjko]

Misha, thank you, looks very interesting but I'm rather a newbie here. Could you answer a first bunch of my silly questions here or give me a link to some introduction? I started learning ml more or less seriously only recently and some looking into the papers you gave makes it clear for me: it's yet another interesting story I started to listen from the middle of the book

[anhinga_anhinga]

> Do you have a simple hint of what is that matrix transformation which would apply the neuron functions to appropriate data and produce a new computational matrix. Or, may be an example?Currently, we use a very simple form of Self. It has two arguments, and on the up movement it simply

[anhinga_anhinga]

4 in an interesting question

[datjko]

Misha, thank you a lot for so detailed answers! I'll keep making small steps and asking new questions.

[anhinga_drafts]

Privet!

This is one of the most interesting talks I've ever heard in my life:

http://anhinga-drafts.livejournal.com/29954.html

[datjko]

Misha, they say that "There's no sparse variable in TensorFlow

[anhinga_anhinga]

I'll also try to think about this...

[anhinga_anhinga]

(more on 1. Of course, if one has unary neurons with "log" and "exp" activation functions, then multiplication is doable at a relatively moderate cost. Multiplication only becomes difficult, if one has the usual (rather impoverished) set of activation functions.

But even with "log" and "exp", there is some difficulty, because the most important case of multiplication is multiplication by zero. So, a bit more thought would be needed on handling this near zero. And, of course, sign reversal would be a separate unary operation. Still the idea of presenting multiplication in this fashion is neat, might be useful for some things, so let it be recorded here.)