Perceptrons vs. neurons

[quintopia]

I've been thinking today about the comparative computational power of perceptrons and neurons.

Background: A perceptron is a function from Rⁿ to {0,1} that first linearly projects an input vector x to 1 dimension by taking its dot product witha fixed weight vector, then returns 1 iff the result is greater than some fixed threshold t. In other words, it answers the question "Does the input vector lie in some fixed halfspace?"

A neuron, on the other, takes as input a bunch of neurotransmitters binding to its dendritic membrane. These neurotransmitters trigger the opening of ion channels, whereupon the influx or efflux or ions can either increase (depolarize) or decrease (hyperpolarize) the potential difference across the cell membrane. (The resting potential of a neuron is ~-70mV.) When a neuron becomes sufficiently depolarized (>~55mV) at the top of the axon (these local changes in the potential are called "graded potentials"), an action potential is triggered down the axon, which is essentially a cascade of ion channel triggerings which spike the potential difference up to ~40mV before it drops again. Importantly, every such spike looks about the same. When the action potential reaches the axon terminal, it opens more ion channels, and the resulting ion influx triggers the release of neurotransmitters into the synaptic cleft.

The parallels are obvious: The input vector corresponds to each of the input neurons of a neuron, with the value of the input being proportional to the amount and type of neurontransmitters in the synaptic cleft. The weight vector corresponds to number and type of receptors on the post-synaptic membrane, which can translate the presence of a neurotransmitter into either an excitatory (depolarizing) or inhibitory (hyperpolarizing) response in the post-synaptic neuron. The threshold parameter is obvious: the threshold voltage at which an action potential is triggered. Unlike with the perceptron, the threshold here never changes, but those familiar with homogeneous coordinate systems can see that this is no necessarily a limitation on a neuron's computational power. Finally, since every action potential is essentially identical ("all-or-none"), when one reaches the axon terminal, it triggers a fixed-size influx of Ca²⁺ ions, resulting in an essentially fixed release of neurotransmitters. In other words, a binary output.

But this isn't the whole story. After an action potential fires, eventually the voltage-gated sodium channels repoen, and some essentially fixed amount is subtracted from the graded potential that triggered it to fire. However, if the original graded potential that caused the spike were sufficiently high to begin with, or has been continually increased while the action potential was travelling, the residual potential might still be above the threshold potential when the neuron "resets". In this case, it will immediately refire. This does happen: some neurons in the brain continually fire, 10-100 times per second. And every time an action potential reaches the axon terminal, neurotransmitters are re-released again and again, and the continual presence of neurotransmitters in the synaptic cleft holds open the ion channels in the post-synaptic membrane, continuing to excite or inhibit the post-synaptic neuron.

In other words, the neuron, over time, is delivering a non-binary output. Past signals accumulate with the current one in such a way that a neuron that cannot, in one firing excite another neuron to fire, can, by repeatedly firing, eventually trigger a spike in the post-synaptic neuron. Of course, past excitatory signals have less weight, since the neuron is constantly running its ion pumps to try to regain its resting potential, but enough ions fast enough can overwhelm their ability to polarize the cell. As far as inhibitory signals go, they can only hyperpolarize the cell so far before electrostatic pressure (as well as resulting osmotic pressure) are overwhelmingly powerful.

There is a maximum amount of neurotransmitters that can be delivered to a synapse as well, since the nature of the action potential limits its firing rate, and limitations of the axon terminal determine how many neurotransmitters can get into the synaptic cleft.

Anyway, the point is, it's more accurate to say that the output of a neuron could be any real number in a specific range. As a result, I came up with the following modification to the perceptron to reflect this difference:

Let f(r)=a*(1-q^(int(r/t)+1))/(1-q) for some fixed parameters a, q, and t. Then, the function computed by our neu-perceptron is f(w.x). Note that t here is like the threshold value we had earlier. int(r/t) is the like the number of spikes a single graded potential could trigger, and the geometric sum surrounding this indicates the degradation of past signal contributions over time (at a constant rate).

Questions:
-Are there other things a neuron can do computationally that I'm missing with this revised model?
-Do we lose any computational power when converting a neuron that takes a vector of values from an interval of the reals and returns a real value in that interval to a perceptron that takes binary input and returns binary output?
-In particular, is a network of binary perceptrons computationally equivalent to a network of biological neurons? If there is a fixed limit on the number of neurons/perceptrons? If they are only feed-forward?
-If no, is the new, modified, perceptron?
-Can some computations using a binary perceptron be simplified or compacted using the new perceptron?

[Yakk]

Glancing at the preceptron, it looks like a network of same can generate a dense (wrt Lⁿ, not L^inf) subset of the characteristic functions on Rⁿ? (ie, I'm thinking that the only real limit a network of preceptrons has is efficiency and the inability to output anything besides 0-1 (or {0,1}^n))

So you'd want to make sure that your neo-preceptron can manage at least that much.

Anyway, the point is, it's more accurate to say that the output of a neuron could be any real number in a specific range.

That number being (roughly) the local firing frequency of the neuron?

-Are there other things a neuron can do computationally that I'm missing with this revised model?

IIRC, even primitive neurons are capable of approximating arbitrary functions. It isn't what it is computationally capable of, but other properties that matter (the Turing Tar Pit is in full force -- computational equivalence is boring and not that useful).

I'd be more interested in efficiency and fragility, which is a far from boring problem, and hence ridiculously hard.

[quintopia]

No, I don't think the Turing tarpit is in full force at the level of the individual neuron. This is why I find it interesting to ask about the computational power of a single neuron.

And when I ask about the computational power of nets of these things, I'm implicitly asking about efficiency: Can one compute some functions asymptotically faster than the other? Can we compute more functions with exactly n neurons than with exactly n perceptrons? Etc.

[serpent]

In terms of computing, although a perceptron with a real number output would be nice - and do-able, it would require more processor time to work out something like your function that it would to add a few more normal neurons. On your questions:

-Are there other things a neuron can do computationally that I'm missing with this revised model?
From what I can tell, not really. Using real-number outputs for the neurons should make them as computationally 'powerful' as those in the brain, excluding the variety of learning processes in the brain, and possible variations of the standard neuron in the brain (I'm not sure, but don't different areas of the brain use slightly different neurons?)

-Do we lose any computational power when converting a neuron that takes a vector of values from an interval of the reals and returns a real value in that interval to a perceptron that takes binary input and returns binary output?
This depends on what you want to talk about in terms of computational power. Each neuron should have more computational power using real values, but in terms of running a neural network on a computer, using a relatively complex function to determine the output of a neuron rather than a simply check of the input value against a threshold, the CPU requirements for simulating each neuron will be considerably higher when using real number neurons rather than simple threshold, binary-response neurons. In this way, one could simulate more binary neurons than 'real' neurons with the same amount of processor time. I think the real question here is: "Does a neural network of binary response neurons have more computational power than a network of 'real' number response neurons that requires similar processing power to simulate?"
i.e. Will more binary neurons be better than fewer 'real' neurons?

-In particular, is a network of binary perceptrons computationally equivalent to a network of biological neurons? If there is a fixed limit on the number of neurons/perceptrons? If they are only feed-forward?
A feed-forward binary neural network cannot hope to match a biological neural network like the human brain. There would be many feedback loops in the brain, all of which are of extreme importance to its functioning and indeed our own though. A computer-based neural network needs to incorporate these feedback loops in order to match a brain. I would say that the 'computational' power of a network with feedback is greater than a simple feed-forward one because information from the feedback loop can be processed again and again in a sense.

-If no, is the new, modified, perceptron?
The new, modified perceptron, although individually more powerful than a binary, threshold based perceptron would still not match a biological network if the network is only feed-forward.

-Can some computations using a binary perceptron be simplified or compacted using the new perceptron?
Less 'real' perceptrons would be able to perform the same task as more binary perceptrons because they would have greater computational power in that they can respond with a greater range of values to their inputs, however the computational processes required to simulate these 'new' perceptrons would be a lot more complicated than the simple threshold based binary perceptron.


A few last notes:

Ultimately, you're talking about computer based neural networks. These networks will be 'simulated' using a computer. Because of this, unless there is redundancy in the code, in order to simulate a neural network with more computational power, more CPU time will be required because ultimately all of the computations are CPU-based. This means that if the code for both your 'new' perceptrons and 'old' binary perceptrons is optimised as much as possible, the increased computational power of your new perceptrons will be offset by the increased CPU time required to 'simulate' them.

Also, you seem to think that the binary perceptron is the only one that has been conceived so far. There have been many that have been proposed - many similar to yours. There are some that mimic biological neurons closely and some that don't. However, the binary perceptron would be favoured in computing because it can be easily, quickly 'simulated' on a BINARY computer, whereas other, more complicated ones can't. The increased CPU time needed to simulate more complex perceptrons generally offsets their increased individual computational power.

Note: I've probably incorrectly refered to perceptrons as neurons a few times, but you should be able to work out whether I'm talking about computer based perceptrons or biological neurons.

[quintopia]

1) I was asking for a comparison between a feed-forward network of biological neurons and a feed-forward network of binary/real perceptrons, not a comparison of a feed-forward network to a recurrent network. The answer to the latter question is obvious.

2) I am unconcerned with the simulation of these things on a CPU, but merely the computational power of these things as pure mathematical models.

[Jplus]

I have not taken the effort to read those long posts above, so forgive me if I repeat something that has already been said.

Before you can start comparing, you must state very clearly what kind of artificial neural nodes is assumed. What is the input function, what is the activation function, and do you allow bias nodes or not?
I think that in general it will be more fair to compare a single biological neuron with a single artificial node, especially if you use a sigmoid activation function and bias nodes. Both output a real value that is bound to a maximum and a minimum, and both only fire if the amount of activation is higher than a certain treshold (which can vary from node to node). Also, the weights of the connections in the artificial nodes correspond to the differences that can be caused by variation in neurotransmitters and synapse location in biological neurons.

So I think the biological neuron and the (theoretically analysed) artificial node have exactly the same computational capabilities. The biological equivalent of a perceptron would be an array of sensor cells innervating a neuron (containing sensors or not is a very relevant difference for the computational capabilities of an entity).