Since the 1940s, simple artificial neurons have been developed as the building blocks of artificial intelligence (AI) technologies, including brain-inspired AI algorithms and neuromorphic devices [25, 26]. The artificial neurons are simple mathematical models conceived as a model of biological neurons; in general, they are designed to perform only basic arithmetical and Boolean logic operations [27, 28, 29]. Traditionally, only the basic dynamics and coding properties of biological neurons have been considered in developing simple neuron models. Specifically, details of individual synaptic currents and distinct dynamics of different types of spines are often disregarded because a single excitatory postsynaptic potential is typically much smaller in amplitude than the threshold for an action potential. The underlying notion of the simple neuron models is that a neuron can fire only when a sufficiently large number of excitatory synapses are activated simultaneously to drive its voltage over the threshold.

In 1943 Warren McCulloch and Walter Pitts developed the first mathematical neuron model [8], which takes multiple binary inputs and produces a single binary output. The neuron is characterized by the parameter $\theta$ denoting the minimum number of excitatory synapses that can generate an action potential: if the number of synapses is greater than or equal to $\theta$, the neuron is active (labeled as “1”); otherwise, it is inactive (“0”). This simple mathematical treatment also allows the Boolean logic operations, in which the binary values 1 and 0 correspond to “true” and “false”. They remarked that the combination of such simple neuron models is capable of universal logic computations; this seminal work laid the foundations of developing brain-inspired digital electronic circuits for AI systems. Analyzing the McCulloch-Pitts (MP) neuron, one can establish basic Boolean logic operations (i.e., AND and OR operations) at the single neuron level (Fig. 1). With the threshold parameter $\theta$ set at a high value (e.g., equal to the total number of inputs), the neuron is active only if all the synaptic inputs are active, leading to the logical AND operation (Fig. 1A). Alternatively, with the threshold set at a low value, only a small portion of active synaptic inputs is enough to fire; this corresponds to the logical OR operation (Fig. 1B).

Similarly, the dynamics of linear-threshold (LT) [30] and firing-rate (FR) [31] models can be interpreted as AND/OR Boolean operations, with the given threshold $\theta$ (Fig. 1). The LT model neuron employs continuously graded input values to describe different contributions of synaptic inputs to the neuronal activation. Each synaptic input is assigned a weight (according to the relative contribution); the weighted sum of all the inputs is compared with $\theta$to decide whether or not to activate the neuron. In the FR model, not only the input but also the output is treated as a continuously graded quantity. While MP and LT neuron models describe the integration of synaptic inputs using the Heaviside step function defined by H(u) = 1 for u $\ge$0 and H(u) = 0, the FR model is based on a differential equation for the firing rate with a continuous-time domain.

The integrate-and-fire (IF) model is the most widely used simple spiking neuron model in artificial neural network algorithms [25, 32, 33, 34, 35]. It is a single-compartment model describing the dynamics of membrane potential, with the morphologies of dendrite branches and axons not explicitly included.

The one variable IF models describe the relationship between the time-dependent voltage V(t) and current I(t):

(1)$${\tau }_{\mathrm{m}}\frac{\mathrm{d}V}{\mathrm{d}t}=\frac{R_{\mathrm{m}}}{A}I+f(V),$$

where $\tau$${}_m$ and A denote the membrane time constant and the effective surface area, respectively, and f(V) describes the leak and spike-generating currents as a function of V. If the voltage V(t) exceeds Ṽ, which stands for the cutoff voltage V${}_c$ or threshold voltage V${}_T$ (depending on whether or not the spike-generating part of f(V) exists), the voltage V(t${}_{\mathrm{+}}$) at time t${}_{\mathrm{+}}$ right after spiking becomes equal to the resetting voltage V${}_r$. After the membrane potential crosses Ṽ, it is reset to V${}_r$ and is inactivated for a brief time corresponding to the absolute refractory period t${}_{r\mathit{}e\mathit{}f}$ of the neuron.

Two-variable IF models include the additional time-dependent adaptive variable $u$:

(2)${\tau }_{\mathrm{m}}{\displaystyle \frac{\mathrm{d}V}{\mathrm{d}t}}={\displaystyle \frac{R_{\mathrm{m}}}{A}}I+f(V)-u$ ${\tau }_u{\displaystyle \frac{\mathrm{d}u}{\mathrm{d}t}}=a\left(V-V_{\mathrm{L}}\right)-u$

with constant a controlling the adaptation to voltage and V${}_L$ denoting the leak reversal potential. If the voltage V(t) exceeds Ṽ, the voltage V(t${}_{\mathrm{+}}$) right after spiking reduces to V${}_r$, similarly to the case of Eqn. 1; in addition, the adaptive variable u(t${}_{\mathrm{+}}$) increases by the amount b controlling the magnitude of the spike event, namely, u(t${}_{\mathrm{+}}$) is set equal to u(t${}_{\mathrm{-}}$)+b with t${}_{\mathrm{-}}$ being the time just before spiking.

Table 1 lists the five models, i.e., LIF (leaky integrate-and-fire); QIF (quadratic integrate-and-fire) with and without an adaptive variable; EIF (exponential integrate-and-fire) with and without an adaptive variable.

The information transfer capabilities are assessed in the information-theoretic framework originally suggested by Denève and colleagues [37, 38, 39, 40, 41]. Fig. 2 illustrates the framework used in the present study. In brief, the binary hidden state triggers a presynaptic neuron. Then the presynaptic neuron fires a spike train via a Poisson process with the firing rate q${}_{o\mathit{}n}$ or q${}_{o\mathit{}f\mathit{}f}$, depending on the hidden state. Synaptic input current I is generated by convolving the spike train with the double exponential kernel: $k\left(t\right)=exp\mathrm{}(-t/{\tau }_1)-exp\mathrm{}(-t/{\tau }_0)$ with ${\tau }_0$ = 0.2 ms and ${\tau }_1$ = 2 ms, followed by multiplying by the synaptic weight, which is modified to control the average input current $\overline{I}$. In general, mutual information measures the overlapping information of two random variables. The mutual information $H(X;I_{0\to t})$ between the hidden state $X\equiv \{x\}$ and the history of postsynaptic spike trains $I_{0\to t}$ is defined as

(3)$H(X;I_{0\to t})={\displaystyle \sum _{x,I_{0\to t}}}p(x;I_{0\to t})\log {\displaystyle \frac{p(x;I_{0\to t})}{p(x)p(I_0\to t)}}$ $=S(X)-S\left(X\mid I_{0\to t}\right),$

where $p(x;I_{0\to t})$ is the joint probability, and $S(X)$ and $S\left(X\mid I_{0\to t}\right)$ denote the information entropy of X and the conditional entropy of $X$ given $I_{0\to t}$, respectively. They are given by

(4)$S(X)\equiv -{\displaystyle \sum _x}p(x)\log p(x)$ $=-⟨x⟩\log ⟨x⟩-(1-⟨x⟩)\log (1-⟨x⟩)$

(5)$S\left(X\mid I_{0\to t}\right)=$ $-⟨x\log p(x=1\mid I_{0\to t})⟩-⟨(1-x)\log (1-p(x=1\mid I_{0\to t}))⟩,$

where the average, defined to be taken with respect to the probability measure p(x), may be estimated as the time average.

The conditional probability $p(x=1|I_{0\to t})$ is equivalent with posterior log-likelihood of the hidden state, $L(t)={\log }_2\frac{p(x=1\mid I_{0\to t})}{p(x=0\mid I_{0\to t})}$, where $p(x=1,t\mid I_{0\to t})$ is the conditional probability of on-state $(x=1)$ at time t, given the history $I_{0\to t}\equiv (I_0,I_1,\mathrm{\dots },I_t)$ of the input current from time 0 to t. The log-odds ratio can be estimated via the following differential equation:

(6)${\displaystyle \frac{dL}{\mathrm{d}t}}=r_{\mathrm{on}}\left(1+e^{-L}\right)-r_{\mathrm{off}}\left(1+e^L\right)+w\delta \left(I_t-1\right)-\theta ,$

where $w\equiv \log \left(q_{\mathrm{on}}/q_{\mathrm{off}}\right)$ and $\theta \equiv q_{\mathrm{on}}-q_{\text{off}}$ with the mean postsynaptic firing rates q${}_{o\mathit{}n}$ and q${}_{o\mathit{}f\mathit{}f}$ for x = 1 and 0, respectively. The Dirac delta function $\delta$ produces a discontinuous jump when the postsynaptic neuron fires.

The information-theoretic framework is used for comparing the neural dynamics and coding properties of the five IF neuron models (Table 1). The train of hidden state X is presented to each of the neuron models to induce the presynaptic input current I and the resulting postsynaptic spike train $I_{0\to t}$. The time evolution of the hidden state and the postsynaptic spike is used to calculate the mutual information $H(X;I_{0\to t})$ as a measure of information transfer by a neuron model.

Fig. 3 displays the current-rate (I-f) curves (the left column) and the time evolutions of the hidden state x and output spike trains (the right column). The I-f curve, which expresses the relationship between the applied current to a neuron and the firing rate (i.e., the frequency of output spikes), is used as the basic measure for characterizing neural dynamics. The firing rate f of the LIF model is the highest, followed by QIF and EIF. The models possessing adaptive variables (QIF* and EIF*) exhibit reduced firing rates compared with the corresponding models without adaptive variables. The right column displays the time evolutions of the hidden states (the first row) and resulting output spikes upon $\overline{I}$ = 50 pA (the second row) and 100 pA (the third row). In all models, the timing of spikes is generally well-matched with the hidden state “1”; however, the spike events do not reflect the fast transitions of hidden states between “0” and “1”.

The dynamics and information processing of IF models are mapped to Boolean operations in Fig. 4. At a given threshold for firing rate f or that for mutual information H, if f or H is greater than or equal to the threshold, then the neuron is active (“1” or “true”); if it is under the threshold, the neuron is considered as inactive (“0” or “false”). Both OR and AND operations occur as a function of the fold change of the difference between the threshold voltage V${}_T$ and the resetting voltage V${}_r$ with respect to the default value V${}_d$/V${}_d$${}^{\mathrm{(}\mathrm{0}\mathrm{)}}$, where V${}_d$ = V${}_T$ – V${}_r$ and V${}_d$${}^{\mathrm{(}\mathrm{0}\mathrm{)}}$ = V${}_T$${}^{\mathrm{(}\mathrm{0}\mathrm{)}}$ – V${}_r$${}^{\mathrm{(}\mathrm{0}\mathrm{)}}$ (with superscript “${}^{(0)}$” denoting the default parameters). V${}_d$ may indicate the voltage required for generating subsequent action potentials: a smaller value of V${}_d$corresponds to the increased membrane excitability (i.e., greater tendency to fire). Several biological contexts, giving rise to the decrease of V${}_d$, include (1) depolarization of the resting membrane potential, (2) reduction in GABAergic inhibition, (3) increased neuronal responsiveness to subthreshold input, and (4) increased conductance that dictates the rate of action potential firing [42]. The OR operations (red shaded regions) arise when both weak (e.g., $\overline{I}$ = 50 pA) and strong ($\overline{I}$ = 100 pA) inputs activate the neuron; on the other hand, AND operations (blue shaded regions) occur only when strong presynaptic input (e.g., $\overline{I}$ = 100 pA) can activate the neuron. These Boolean operations correspond to the schematic illustrations in Fig. 1B: the input current $\overline{I}$ = 50 pA may denote the active input ‘1’, and $\overline{I}$ = 100 pA thus corresponds to two active inputs. Depending on the thresholds for f and H, the regions can be mapped to OR or AND operations.

Although these IF models are simplified versions of the biophysically realistic multi-compartment neuron models (which are explored in the following section), they appear to characterize the neural logic operations successfully. In particular, the neural dynamics and coding properties of exponential integrate-and-fire models (EIF and EIF*) are overall similar to the biophysical models, compared with other IF models [29]. The neural dynamics of IF models vary, depending on the mathematical form of the leak and spike-generating currents. In brief, the leak current term is necessary for responding to the changes of the hidden states in a timely manner; the IF models without the term usually fire in response to inactive hidden states (‘0’) because depolarization of the membrane potential during previous active hidden states (‘1’) is maintained during inactive states. The spike-generating currents [Eqn. 1 and Table 1] determine the speed of spiking: the IF models except EIF and EIF* exhibit much slower spike generation, compared with the biophysical model [29].

This section describes the characterization of information processing of the biophysically realistic multi-compartment neuron models, which describe how action potentials are initiated and propagated, based on Hodgkin-Huxley (HH) type conductance models for ion channels [43, 44]. Containing the axon and dendrites explicitly, the model has highly realistic structures via three-dimensional morphological reconstruction of biological neurons [45].

Two representative neuron models for neural computation are compared: the pyramidal neuron in CA1 in the hippocampal circuit (ModelDB accession 7907) [46] and PF in the cerebellum (ModelDB accession 7907) [46]. In the CA1 pyramidal neuron model, all dendrites are divided into compartments with a maximum length of 7 mm. Spines are incorporated where appropriate by scaling membrane capacitance and conductance [47, 48]. Two Hodgkin–Huxley-type conductances (g${}_{Na}$ and g${}_K$) are inserted into the soma and dendrites at uniform densities. The model is tuned by attaching a synthetic axon. The uniform passive parameters of the model are given by R${}_i$ = 150 $\mathrm{\Omega }$$\cdot$cm, C${}_m$ = 1 mF/cm${}^2$, and R${}_m$ = 12 k$\mathrm{\Omega }$$\cdot$cm${}^2$. The standard values for g${}_{N\mathit{}a}$ and g${}_K$ are 35 and 30 pS/mm${}^2$, respectively. For the Purkinje cell, we use the morphology of a 21-day-old Wistar rat PF [49]. The model consists of an axon, a soma, smooth dendrites, and spiny dendrites. The model has passive parameters as follows: R${}_m$ = 12 k$\mathrm{\Omega }$$\cdot$cm${}^2$, R${}_i$ = 150 $\mathrm{\Omega }$$\cdot$cm, and C${}_m$ = 1 $\mu$F/cm${}^2$. To compensate for the absence of spines in the reconstructed morphology, we scale the conductance of passive current and C${}_m$ by a factor of 5.34 in the spiny dendrite and 1.2 in the smooth dendrite. Two Hodgkin–Huxley-type conductances (g${}_{Na}$ and g${}_K$) are inserted into the soma and dendrites at uniform densities. The model is tuned by attaching a synthetic axon. The standard values for g${}_{N\mathit{}a}$ and g${}_K$ are 35 and 30 pS/mm${}^2$, respectively.

The information-theoretic framework is similar to that used in Section 3 for IF models, except that the stimulus sites are carefully chosen based on biological knowledge and two inputs are provided simultaneously, with their competitive and cooperative effects assessed [29, 50, 51]. Fig. 5 displays the schematic of the framework. Each of the two hidden states X${}_{\mathrm{1}}$ and X${}_{\mathrm{2}}$ triggers a set of presynaptic neurons connected to the postsynaptic neuron via synapses. The synapses from each set (corresponding to X${}_{\mathrm{1}}$ or X${}_{\mathrm{2}}$) are colored blue or red, respectively. The coherence between X${}_{\mathrm{1}}$ and X${}_{\mathrm{2}}$ is measured by parameter $\alpha$ in the range [0, 1]: $\alpha$ vanishes for the two states behaving independently while it is equal to unity for the two fully synchronized. The mutual information $H(X_i;I_{0\to t})$ between each hidden state $X_i$ and the output spike train is measured as in Eqn. 3 (with $X_i$ replacing $X$).

In the rest of this section, we explore the information processing of homosynaptic plasticity (Section 4.1) and analyze Boolean logic operations triggered by one hidden state. Then we examine heterosynaptic transmission in the two hidden state scheme, which allows us to assess the synaptic cooperation and competition (Section 4.2).

Homosynaptic transmission refers to the specific modification of a synapse by the activity of the corresponding presynaptic and postsynaptic neurons. The most widely used realization of this concept, first proposed by Hebb in 1949 [52], is the spike-timing-dependent plasticity (STDP) rule [53], which is often adopted in spike-based artificial neural networks. This rule states that a presynaptic stimulus immediately followed by a postsynaptic spike results in potentiation of the synapse while the opposite results in depression. Another well-known synaptic plasticity rule is the Bienenstock, Cooper, and Munro (BCM) rule [54, 55], according to which modification of the synapse depends on the instantaneous postsynaptic firing rate. In the original formulation of the rule, depression occurs when the postsynaptic firing rate is below a threshold while potentiation occurs when the rate is above the threshold. In particular, the threshold separating depression and potentiation is itself a slow variable that changes as a function of the postsynaptic activity.

Here, we implement a learning rule in which the STDP rule is combined with the BCM rule [50, 56, 57, 58]. According to the STDP rule, weight changes occur for each pair of presynaptic and postsynaptic spikes separated by time interval $\mathrm{\Delta }$t in the following way:

(7)$\mathrm{\Delta }{w}_{\mathrm{p}}(\mathrm{\Delta }t)=A_{\mathrm{p}}(t)\exp \left(-\mathrm{\Delta }t/{\tau }_{\mathrm{p}}\right)\text{for}\mathrm{\Delta }t>0$

where the subscripts p and d label potentiation and depression, respectively, and $\mathrm{\Delta }$t$\equiv$t${}_{p\mathit{}o\mathit{}s\mathit{}t}$ – t${}_{p\mathit{}r\mathit{}e}$ is the time difference between the presynaptic and postsynaptic spikes. The sign of $\mathrm{\Delta }$t determines whether the presynaptic spike precedes the postsynaptic spike (positive) or follows (negative). The BCM rule is implemented by allowing the amplitudes A${}_p$ and A${}_d$ to vary with the postsynaptic firing rate $⟨c⟩$ according to

(8)$A_{\mathrm{p}}(t)={⟨c⟩}^{-1}{A}_{\mathrm{p}}(0)$
$A_{\mathrm{d}}(t)=⟨c⟩A_{\mathrm{d}}(0),$

where $⟨c⟩$ is a weighted sum of postsynaptic spikes, given by

(9)$⟨c⟩={\displaystyle \frac{{\alpha }_c}{{\tau }_c}}{\displaystyle {\int }_{-\mathrm{\infty }}^t}𝑑t^{\prime }c\left(t^{\prime }\right)\exp \left(-{\displaystyle \frac{t-t^{\prime }}{{\tau }_c}}\right).$

The BCM rule has a balancing effect that allows robust synaptic learning [55, 56]. The parameters for the learning rule are as follows: $\tau$${}_p$ = 20 ms; $\tau$${}_d$ = 70 ms; A${}_p$(0) = 0.006 $\mu$S; A${}_d$(0) = 0.002 $\mu$S; $\tau$${}_c$ = 1500 ms;$\alpha$${}_c$ = 62.5.

Boolean logic mappings of the dynamics and information processing of the biophysical neurons are illustrated in Fig. 6. Note that the Boolean operations depend on the location of the synaptic input. For the CA1 pyramidal neuron, synapses are placed in the distal section or proximal section of the apical dendrite (a or b in Fig. 6A) in one setting or the other. These locations are selected based on neuroanatomical knowledge: The apical dendrite of the CA1 pyramidal has a long, extended structure to receive inputs from distinctly organized regions. Its distal region directly receives input via the perforant pathway from the entorhinal cortex while the proximal region indirectly receives via the granule cell and CA3 pyramidal neuron [59] (Fig. 5).

Presynaptic neurons fire at an average rate of 100 Hz, provided that the hidden state is on. When the stimulus is given to section a, nine synaptic inputs are required to cross the firing threshold of 10 Hz and the mutual information threshold of 0.1. In the case of the stimulus given to section b, the required number of synapses for the same threshold is just four. This manifests that the Boolean logic operations occurring with synaptic transmission depend on the location on the dendrite: in the distal region, the operation is closer to AND, with many concurrent inputs required to exceed the threshold, while in the proximal region, it is closer to OR, with only a few required.

The Purkinje cell has a highly branching, flattened structure built to receive inputs from up to 200,000 parallel fibers that pass orthogonally through the dendritic arbor [60]. In addition, each Purkinje cell receives input from one climbing fiber, which enwraps the dendrite and forms a vast number of synapses [61]. Unlike pyramidal neurons, there is no need for a vertical extension. Fig. 6B shows the logic operations performed by the Purkinje cell. Two inputs are given to two sections c and d along a main dendritic branch. The presynaptic neurons are assigned to a firing rate of 200 Hz when the hidden state is on. For the relatively distal section c, the firing rate threshold is crossed at eight synapses and the mutual information threshold at ten; for the relatively proximal section d the thresholds are exceeded at six and seven. Namely, there is a less drastic difference in the two sections compared with the CA1 case.

The CA1 pyramidal neuron and the Purkinje cell form well-established functional neural circuits that integrate multiple inputs from distinct sources. In particular, they can perform complex computation via homo and heterosynaptic mechanisms [50]. Heterosynaptic transmission refers to the modification of synaptic strength by unspecific presynaptic stimuli. The activity of a presynaptic neuron alters the strength of a synapse of the postsynaptic neuron not directly connected to the presynaptic neuron in action [62]. Unlike the homosynaptic case, there is no widely accepted computational model for heterosynaptic transmission.

The CA1 pyramidal neuron receives inputs mainly from two sources, one directly from the entorhinal cortex and one from the CA3 region [63]. The processing of these inputs is a critical step in the role of the hippocampus in memory, which is postulated to play a comparator role [64]. Synaptic plasticity in the CA1 neuron is well studied and exhibits rich heterosynaptic plasticity mechanisms [65, 66]. The Purkinje cell provides the sole output from the cerebellar cortex. It receives inputs from parallel fibers, axons of granule cells, and just one climbing fiber originating from the inferior olive, which nevertheless comprises about 1500 synapses. The Purkinje cell is believed to play a key role in motor learning, yet the synaptic mechanism remains elusive [67]. It was suggested in the early theories of learning in the cerebellum that heterosynaptic interactions between the parallel and climbing fibers play a key role [68, 69]. These neuronal systems have in common that heterosynaptic interactions between multiple inputs, which can be cooperative or competitive, play a crucial role in their computations [70, 71]. To understand the processing of these multiple inputs, we study them in our information-theoretic framework (Fig. 5).

Fig. 7 displays the Boolean logical operations of the neuron models at given two multimodal inputs. For the CA1 neuron, X${}_{\mathrm{1}}$ stimulates twelve synapses in section a and X${}_{\mathrm{2}}$ stimulates eight synapses in b (see Fig. 6A); for the PF, X${}_{\mathrm{1}}$ stimulates five synapses in c and X${}_{\mathrm{2}}$ stimulates five synapses in d (see Fig. 6B). Varying the firing rates of the two sets of presynaptic neurons, q${}^{\mathrm{1}}$${}_{o\mathit{}n}$ and q${}^{\mathrm{2}}$${}_{o\mathit{}n}$, we compute the mutual information $H(X_1;I_{0\to t})$ and $H(X_2;I_{0\to t})$ and show the resulting maps in the first and second columns, respectively. The third column presents the total information transmitted, $H(X_1;I_{0\to t})$ + $H(X_2;I_{0\to t})$, with the threshold set to 80% of the maximum. For both CA1 and PF, when the coherence is low ($\alpha$ = 0.1; first row), synaptic competition occurs. In case that just one input is on, the total mutual information exceeds the threshold and the output becomes unity; in the case of both on, the output is zero. It is remarkable that the resulting Boolean operation is the exclusive OR (XOR) operation. On the other hand, when the coherence is high ($\alpha$ = 0.9; first row), the two inputs exhibit cooperation, which results in OR and AND-like operations.