Self-referential neural nets in 2018
Two series of experiments with self-referential neural nets with vector flows ("dataflow matrix machines") were done by us in 2018.
The ability of a neural net to modify itself on the fly was used to edit it interactively while it is running ("livecoding"). This also opens the way to have populations of neural nets editing each other.
Emerging "sleep-wake" behavior and other emerging bistability patterns were observed in randomly initialized neural nets (May 2019 update: a couple of video recordings of those behaviors are posted: https://youtu.be/_mZVVU8x3bs and https://youtu.be/CKVwsQEMNjY ). There is no theoretical understanding of this emerging dynamics yet.
These experiments are described in Section 1 of https://github.com/jsa-aerial/DMM/tree/master/technical-report-2018 (DMM technical report 11-2018, "Dataflow matrix machines: recent experiments and notes for next steps").
Also https://arxiv.org/abs/1610.00831 got an extra page (page 7) with two new appendices, and the main paper, https://arxiv.org/abs/1712.07447, has a new more compact abstract:
1) Dataflow matrix machines (DMMs) generalize neural nets by replacing streams of numbers with linear streams (streams supporting linear combinations), allowing arbitrary input and output arities for activation functions, countable-sized networks with finite dynamically changeable active part capable of unbounded growth, and a very expressive self-referential mechanism.
2) DMMs are suitable for general-purpose programming, while retaining the key property of recurrent neural networks: programs are expressed via matrices of real numbers, and continuous changes to those matrices produce arbitrarily small variations in the associated programs.
3) Spaces of V-values (vector-like elements based on nested maps) are particularly useful, enabling DMMs with variadic activation functions and conveniently representing conventional data structures.
We also have a better slide deck: https://github.com/jsa-aerial/DMM/blob/master/doc/DMM-IBM-Talk-Oct2018.pdf
Crosspost: https://anhinga-anhinga.dreamwidth.org/83296.html
Other blogs by this author:
https://dmm.dreamwidth.org/ (partial mirror: https://anhinga-travel.livejournal.com/ )
https://anhinga-drafts.livejournal.com/ (mirror: https://anhinga-drafts.dreamwidth.org/ )
The ability of a neural net to modify itself on the fly was used to edit it interactively while it is running ("livecoding"). This also opens the way to have populations of neural nets editing each other.
Emerging "sleep-wake" behavior and other emerging bistability patterns were observed in randomly initialized neural nets (May 2019 update: a couple of video recordings of those behaviors are posted: https://youtu.be/_mZVVU8x3bs and https://youtu.be/CKVwsQEMNjY ). There is no theoretical understanding of this emerging dynamics yet.
These experiments are described in Section 1 of https://github.com/jsa-aerial/DMM/tree/master/technical-report-2018 (DMM technical report 11-2018, "Dataflow matrix machines: recent experiments and notes for next steps").
Also https://arxiv.org/abs/1610.00831 got an extra page (page 7) with two new appendices, and the main paper, https://arxiv.org/abs/1712.07447, has a new more compact abstract:
1) Dataflow matrix machines (DMMs) generalize neural nets by replacing streams of numbers with linear streams (streams supporting linear combinations), allowing arbitrary input and output arities for activation functions, countable-sized networks with finite dynamically changeable active part capable of unbounded growth, and a very expressive self-referential mechanism.
2) DMMs are suitable for general-purpose programming, while retaining the key property of recurrent neural networks: programs are expressed via matrices of real numbers, and continuous changes to those matrices produce arbitrarily small variations in the associated programs.
3) Spaces of V-values (vector-like elements based on nested maps) are particularly useful, enabling DMMs with variadic activation functions and conveniently representing conventional data structures.
We also have a better slide deck: https://github.com/jsa-aerial/DMM/blob/master/doc/DMM-IBM-Talk-Oct2018.pdf
Crosspost: https://anhinga-anhinga.dreamwidth.org/83296.html
Other blogs by this author:
https://dmm.dreamwidth.org/ (partial mirror: https://anhinga-travel.livejournal.com/ )
https://anhinga-drafts.livejournal.com/ (mirror: https://anhinga-drafts.dreamwidth.org/ )