Croft Press / David Wallace Croft / Research

Neural Networks

David Wallace Croft

2005-04-15


Frustrated Synapse Learning

In the Spring of 1993, I invented a neural network learning algorithm which I called Frustrated Synapse learning. My novel idea was to combine Hebbian learning with an additional rule that would make a synaptic weight more negative if it were triggered during the hyperpolarization of the target neuron. Through a number of computer simulation experiments that Summer, I observed that the ability of the algorithm to overcome the stability-plasticity dilemma and to stabilize fully recurrent networks.

In the Fall of 1993, I entered Caltech as a graduate student and threw myself into my simulations and course work in an effort to determine whether this was the learning rule used by real-life biological neurons. During that time, I was able to discuss the learning rule with many of my professors and classmates and increase the sophistication of my models, as demonstrated in some of my notes, student papers, and presentation slides from that period as listed below. When I left graduate school to enter industry in 1995, I was even more convinced that this learning algorithm was biologically plausible.

Upon returning to the field in 2003, I was pleasantly surprised to learn that this learning rule was demonstrated to exist in biological neurons through experiments performed by neuroscientists in 1997. What I have previously labeled Frustrated Synapse, Phase Covariance, or Hebbian Phase learning is now known as antisymmetric Hebbian or spike timing dependent synaptic plasticity (STDP).

Synchronicity and Periodicity Research

Other Neural Network Papers

Neural Projects

  • 2005 -
    Newt Cyborg: neuro-micro-transponder interface software

  • 1993 - 1997
    Theoretical analysis of biologically realistic neuron model with dendritic tree using the simulation tool Neuron. B. Mel, E. Niebur, and D. Croft, "Why Neurons Make Bad Coincidence Detectors but Good Periodicity Detectors", presented at the 1995 Neurosciences Meeting.

  • 1993 - 1997
    Research of the neural network Phase Correlation learning algorithm, novel activation and learning rules for spiking neurons.

  • 1996
    Implementation of neural network and genetic algorithm simulations in the Java programming language.
    http://www.croftsoft.com/portfolio/

  • 1995 Jun - 1996 Jul
    Systems Engineer
    Tanner Research Inc., Pasadena, CA
    http://www.tanner.com/

    • Design and implementation of parameterizable VLSI layout language software code in C for the automated generation of digital neural network and subthreshold analog VLSI neuromorphic circuits as part of the Neural Network Silicon Compiler research contract. Demonstrated at the 1996 NSF Workshop on Neuromorphic Engineering.
    • Design and fabrication of scalable, programmable, stochastic pulse CMOS VLSI Digital Neural Network Architecture (DNNA) circuitry.
    • Laboratory testing of analog and digital CMOS VLSI chips for speech processing and neural network applications.
    • Documentation of reusable VLSI circuit layout language code components and cell libraries in HTML.
    • Wrote the "Fuzzy Logic Silicon Compiler" government research proposal, identifying low-power analog circuits to be used for Fuzzy Logic processing.
    • Experience with the full suite of EDA tools for VLSI design including schematic editors, layout editing, and simulators in the process of carrying circuit designs from concepts to the test bench.

  • 1995
    With lab partners, injected mRNA for neural channels into frog oocytes and later observed neural spiking when current was injected.

  • 1994 - 1995
    Design, fabrication, and testing of a novel analog VLSI depolarizing-hyperpolarizing neuron with an analog synapse adapted using an integrated learning algorithm with floating gate tunneling and injection. Presented in a talk at the kickoff for the NSF Center for Neuromorphic Systems Engineering at the California Institute of Technology.

  • 1994
    Developed simulation software in MatLab for the implementation of a spatiotemporal filter for accurate velocity estimation over a range of spatial frequencies using passive and integrate-and-fire neuron models.

  • 1990
    Design, implementation, and demonstration of the neural network ART-1 learning algorithm for pattern recognition. System included photodiode input, digital to analog conversion, RS-232 serial I/O circuitry, serial I/O software, software implementation of the ART-1 learning algorithm, and graphical output.

Demos

  • Newt Cyborg: neuro-micro-transponder interface software
  • Backprop_XOR: a neural network simulation of backpropagation error learning
  • Dice: click the mouse and watch a simple neural net learn whether to flee or fight
  • Insight: goblins hunt kobolds in the dark using a neural network

Links

Books

  • Software Agents and Soft Computing: Towards Enhancing Machine Intelligence
    1997; Hyacinth S. Nwana and Nader Azermi (Eds.)
  • Intelligent Java Applications for the Internet and Intranets
    1997; Mark Watson
  • Advances in Knowledge Discovery and Data Mining
    1996; Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy (Eds.)
  • Data Mining with Neural Networks: Solving Business Problems -- from Application Development to Decision Support
    1996; Joseph P. Bigus
  • An Introduction to Natural Computation
    1996 June (draft); Dana H. Ballard
  • C++ Neural Networks & Fuzzy Logic, 2nd Edition
    1995; Valluru Rao and Hayagriva Rao
  • Neural Networks for Pattern Recognition
    1995; Christopher M. Bishop
  • Neural Networks: A Comprehensive Foundation
    1994; Simon Haykin
  • Analog VLSI: Signal and Information Processing
    1994; Mohammed Ismail and Terri Fiez
  • Biophysics of Computing: Information Processing in Single Neurons
    1994 September 08 (draft); Christof Koch
  • Advanced Methods in Neural Computing
    1993; Philip D. Wasserman
  • The Book of Genesis: Exploring Realistic Neural Models with the GEneral NEural SImulation System
    1993; James M. Bower and David Beeman
  • From Neuron to Brain, 3rd Edition
    1992; Nicholls, Martin, and Wallace
  • Frontiers in Cognitive Neuroscience
    1992; Stephen M. Kosslyn and Richard A. Andersen (Eds.)
  • Principles of Neural Science, 3rd Edition
    1991; Kandel, Schwartz, and Jessell
  • Introduction to the Theory of Neural Computation
    1991; Hertz, Krough, and Palmer
  • Visual Perception: The Neurophysiological Foundations
    1990; Lothar Spillmann and John S. Werner (Eds.)
  • The Synaptic Organization of the Brain, 3rd Edition
    1990; Gordon M. Shepherd (Ed.)
  • Mind and Cognition: A Reader
    1990: William G. Lycan (Ed.)
  • The Representational Theory of Mind: An Introduction
    1990: Kim Sterelny
  • Methods in Neuronal Modeling: From Synapses to Networks
    1989; Christof Koch and Idan Segev (Eds.)
  • Analog VLSI and Neural Systems
    1989; Carver Mead
  • Neural Computing: Theory and Practice
    1989; Philip D. Wasserman
  • Eye, Brain, and Vision
    1988; David H. Hubel
  • Robot Vision
    1986; Berthold Klaus Paul Horn
  • An Introduction to the Mathematics of Neurons
    1986; F. C. Hoppensteadt
  • The Human Brain Coloring Book
    1985; Diamond, Scheibel, and Elson
 
 
Google
Croft Press Web