Croft Press
/ David Wallace Croft
/ Research
Neural Networks
David Wallace Croft
2005-04-15
Frustrated Synapse Learning
In the Spring of 1993, I invented a neural network learning
algorithm which I called Frustrated Synapse learning. My novel idea was
to combine Hebbian learning with an additional rule that would make a
synaptic weight more negative if it were triggered during the
hyperpolarization of the target neuron. Through a number of computer
simulation experiments that Summer, I observed that the ability of the
algorithm to overcome the stability-plasticity dilemma and to stabilize
fully recurrent networks.
In the Fall of 1993, I entered Caltech as a graduate student and threw
myself into my simulations and course work in an effort to determine
whether this was the learning rule used by real-life biological
neurons. During that time, I was able to discuss the learning rule
with many of my professors and classmates and increase the sophistication
of my models, as demonstrated in some of my notes, student papers,
and presentation slides from that period as listed below.
When I left graduate school to enter industry in 1995, I was even more
convinced that this learning algorithm was biologically plausible.
Upon returning to the field in 2003, I was pleasantly surprised to learn
that this learning rule was demonstrated to exist in biological
neurons through experiments performed by neuroscientists in 1997.
What I have previously labeled Frustrated Synapse, Phase Covariance,
or Hebbian Phase learning is now known as antisymmetric Hebbian or
spike timing dependent synaptic plasticity (STDP).
Synchronicity and Periodicity Research
Mel, B.W., Niebur, E., & Croft, D.W.,
"How neurons may respond to temporal structure in their inputs."
Proceedings of CNS*96, Computational Neuroscience Meeting, Boston, MA, 1996.
http://quake.usc.edu/abstracts/SNP.html
http://www.bbb.caltech.edu/cns96/
Mel, B.W., Niebur, E., & Croft, D.W.,
"When neurons crave regularity and shun cooperativity in their synaptic input stream".
In Proc. of the 3rd Joint Symposium on Neural Computation, Caltech and UCSD, 1996.
http://lnc.usc.edu/publications.html
Bartlett W. Mel, David Croft, and Ernst Niebur,
"Why Neurons Make Bad Coincidence Detectors But Good Periodicity Detectors".
Abstract submitted to 1995 Neurosciences Meeting.
http://quake.usc.edu/abstracts/Temporal.html
Other Neural Network Papers
Neural Projects
-
2005 -
Newt Cyborg: neuro-micro-transponder interface software
1993 - 1997
Theoretical analysis of biologically realistic neuron model with
dendritic tree using the simulation tool Neuron. B. Mel, E. Niebur,
and D. Croft, "Why Neurons Make Bad Coincidence Detectors but Good
Periodicity Detectors", presented at the 1995 Neurosciences Meeting.
1993 - 1997
Research of the neural network Phase Correlation learning algorithm,
novel activation and learning rules for spiking neurons.
1996
Implementation of neural network and genetic algorithm
simulations in the Java programming language.
http://www.croftsoft.com/portfolio/
1995 Jun - 1996 Jul
Systems Engineer
Tanner Research Inc., Pasadena, CA
http://www.tanner.com/
-
Design and implementation of parameterizable VLSI layout
language software code in C for the automated generation of
digital neural network and subthreshold analog VLSI
neuromorphic circuits as part of the Neural Network
Silicon Compiler research contract.
Demonstrated at the
1996 NSF Workshop on Neuromorphic Engineering.
-
Design and fabrication of scalable, programmable, stochastic pulse
CMOS VLSI Digital Neural Network Architecture (DNNA) circuitry.
-
Laboratory testing of analog and digital CMOS VLSI chips
for speech processing and neural network applications.
-
Documentation of reusable VLSI circuit layout language code
components and cell libraries in HTML.
-
Wrote the "Fuzzy Logic Silicon Compiler" government
research proposal, identifying low-power analog circuits
to be used for Fuzzy Logic processing.
-
Experience with the full suite of EDA tools for VLSI
design including schematic editors, layout editing,
and simulators in the process of carrying circuit
designs from concepts to the test bench.
1995
With lab partners, injected mRNA for neural channels into frog
oocytes and later observed neural spiking when current was injected.
1994 - 1995
Design, fabrication, and testing of a novel analog VLSI
depolarizing-hyperpolarizing neuron with an analog synapse
adapted using an integrated learning algorithm with floating
gate tunneling and injection. Presented in a talk at the
kickoff for the NSF Center for Neuromorphic Systems Engineering
at the California Institute of Technology.
1994
Developed simulation software in MatLab for the implementation of a
spatiotemporal filter for accurate velocity estimation over a
range of spatial frequencies using passive and integrate-and-fire
neuron models.
1990
Design, implementation, and demonstration of the neural network
ART-1 learning algorithm for pattern recognition. System included
photodiode input, digital to analog conversion, RS-232 serial I/O
circuitry, serial I/O software, software implementation of the
ART-1 learning algorithm, and graphical output.
Demos
-
Newt Cyborg: neuro-micro-transponder interface software
-
Backprop_XOR:
a neural network simulation of backpropagation error learning
-
Dice:
click the mouse and watch a simple neural net learn whether to
flee or fight
-
Insight:
goblins hunt kobolds in the dark using a neural network
Links
Books
-
Software Agents and Soft Computing:
Towards Enhancing Machine Intelligence
1997; Hyacinth S. Nwana and Nader Azermi (Eds.)
-
Intelligent Java Applications
for the Internet and Intranets
1997; Mark Watson
-
Advances in Knowledge Discovery and Data Mining
1996; Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy (Eds.)
-
Data Mining with Neural Networks: Solving Business Problems
-- from Application Development to Decision Support
1996; Joseph P. Bigus
- An Introduction to Natural Computation
1996 June (draft); Dana H. Ballard
- C++ Neural Networks & Fuzzy Logic, 2nd Edition
1995; Valluru Rao and Hayagriva Rao
- Neural Networks for Pattern Recognition
1995; Christopher M. Bishop
- Neural Networks: A Comprehensive Foundation
1994; Simon Haykin
- Analog VLSI: Signal and Information Processing
1994; Mohammed Ismail and Terri Fiez
- Biophysics of Computing:
Information Processing in Single Neurons
1994 September 08 (draft); Christof Koch
- Advanced Methods in Neural Computing
1993; Philip D. Wasserman
- The Book of Genesis: Exploring Realistic Neural Models
with the GEneral NEural SImulation System
1993; James M. Bower and David Beeman
- From Neuron to Brain, 3rd Edition
1992; Nicholls, Martin, and Wallace
- Frontiers in Cognitive Neuroscience
1992; Stephen M. Kosslyn and Richard A. Andersen (Eds.)
- Principles of Neural Science, 3rd Edition
1991; Kandel, Schwartz, and Jessell
- Introduction to the Theory of Neural Computation
1991; Hertz, Krough, and Palmer
- Visual Perception: The Neurophysiological Foundations
1990; Lothar Spillmann and John S. Werner (Eds.)
- The Synaptic Organization of the Brain, 3rd Edition
1990; Gordon M. Shepherd (Ed.)
- Mind and Cognition: A Reader
1990: William G. Lycan (Ed.)
- The Representational Theory of Mind: An Introduction
1990: Kim Sterelny
- Methods in Neuronal Modeling:
From Synapses to Networks
1989; Christof Koch and Idan Segev (Eds.)
- Analog VLSI and Neural Systems
1989; Carver Mead
- Neural Computing: Theory and Practice
1989; Philip D. Wasserman
- Eye, Brain, and Vision
1988; David H. Hubel
- Robot Vision
1986; Berthold Klaus Paul Horn
- An Introduction to the Mathematics of Neurons
1986; F. C. Hoppensteadt
- The Human Brain Coloring Book
1985; Diamond, Scheibel, and Elson
|