TimeAveragedDense

Bases: tf.keras.layers.Dense

This layer simulates continuous-temporal dynamics with time-averaged input/output.

Note

This layer is for single input, i.e., signal comes from one precedent layer. For multiple inputs equivalent, see MultiInputTimeAveraging.

Parameters:

Name	Type	Description	Default
tau	float	Time-averaging parameter, from 0 to 1.	required
average_at	str	Select where to average, 'before_activation' or 'after_activation'.	required
kwargs	dict	Any argument in keras.layers.Dense.	{}

Time-averaged input

See Plaut, McClelland, Seidenberg, and Patterson (1996) equation (15).

Defines as:

$$a_t = act(s_t)$$

$$s_t = \tau \cdot (x_t w + b) + (1-\tau) \cdot s_{t-1}$$

Warning

"$\cdot$" is element-wise multiplication.

• $a_t$: activation at time $t$.
• $act$: activation function (provided by this layer if activation is used).
• $s_t$: state at time $t$.
• $\tau$: time constant, smaller means slower temporal dynamics.
• $x_t$: input at time $t$.
• $w$: weight matrix (provided by this layer).
• $b$: bias vector, provided by this layer if use_bias is True (Default).
• $s_{t-1}$: state at time $t-1$, stored in self.states.

Example

layer = TimeAveragedDense(
    tau=0.1, average_at="before_activation", units=10, activation="sigmoid"
    )

Time-averaged output

Defines as:

$$a_t = \tau \cdot act(x_t w + b) + (1-\tau) \cdot a_{t-1}$$

Warning

"$\cdot$" is element-wise multiplication.

• $a_t$: activation at time $t$.
• $\tau$: time constant, smaller means slower temporal dynamics.
• $act$: activation function (provided by this layer if activation is used).
• $x_t$: input at time $t$.
• $w$: weight matrix (provided by this layer).
• $b$: bias vector, provided by this layer if use_bias is True (Default).
• $a_{t-1}$: activation at time $t-1$, stored in self.states.

Example

layer = TimeAveragedDense(
    tau=0.1, average_at="after_activation", units=10, activation="sigmoid"
    )

call(inputs)

Forward pass.

Note

This function reused keras.layers.Dense.call() without activation. Time-averaging and activation are manually implemented here.

Parameters:

Name	Type	Description	Default
inputs	tf.Tensor	Input tensor $x_t$.	required

Returns:

Name	Type	Description
outputs	tf.Tensor	Output tensor $a_t$.

reset_states()

Resetting self.states to None.

Note

This is typically called when RNN unrolling is done, so that the next batch of data can start with a clean state.