MultiInputTimeAveraging

Bases: tf.keras.layers.Layer

This layer simulates continuous-temporal dynamics with time-averaged input/output for multiple inputs.

Note

This layer is for multiple inputs, and assuming they had ALREADY been multiplied by weights. i.e., This layer has no weights matrix, but a bias vector if use_bias is True. For single input equivalent, see TimeAveragedDense.

Parameters:

Name	Type	Description	Default
tau	float	Time-averaging parameter, from 0 to 1.	required
average_at	str	Select where to average, 'before_activation' or 'after_activation'.	required
activation	str	Activation function to use.	None
use_bias	bool	Whether the layer uses a bias vector.	True
bias_initializer	optional	Initializer for the bias vector.	'zeros'
bias_regularizer	optional	Regularizer function applied to the bias vector.	None
bias_constraint	optional	Constraint function applied to the bias vector.	None

Time-averaged input

See Plaut, McClelland, Seidenberg, and Patterson (1996) equation (15).

Defines as:

$$a_t = act(s_t)$$

$$s_t = \tau \cdot (\sum_i^n x_{i,t} + b) + (1-\tau) \cdot s_{t-1}$$

Warning

"$\cdot$" is element-wise multiplication.

• $a_t$: activation at time $t$.
• $act$: activation function (provided by this layer if activation is used).
• $s_t$: state at time $t$.
• $\tau$: time constant, smaller means slower temporal dynamics.
• $x_{i,t}$: input at time $t$ coming from input $i$.
• $n$: number of input layers.
• $b$: bias vector, provided by this layer if use_bias is True (Default).
• $s_{t-1}$: state at time $t-1$, stored in self.states.

Example

layer = MultiInputTimeAveraging(
    tau=0.1, average_at="before_activation", activation="sigmoid"
    )

Time-averaged output

Defines as:

$$a_t = \tau \cdot act(\sum_i^n x_{i,t} + b) + (1-\tau) \cdot a_{t-1}$$

Warning

"$\cdot$" is element-wise multiplication.

• $a_t$: activation at time $t$
• $\tau$: time constant, smaller means slower temporal dynamics
• $act$: activation function (provided by this layer if activation is used)
• $x_{i,t}$: input at time $t$ coming from input $i$
• $b$: bias vector, provided by this layer if use_bias is True (Default)
• $a_{t-1}$: activation at time $t-1$, stored in self.states

Example

layer = MultiInputTimeAveraging(
    tau=0.1, average_at="after_activation", activation="sigmoid"
    )

build(input_shape)

Build the underlying layers/weights.

call(inputs)

Forward pass.

Parameters:

Name	Type	Description	Default
inputs	List[tf.Tensor]	List of input tensors $x_{i,t}$.	required

Returns:

Name	Type	Description
outputs	tf.Tensor	Output tensor $a_t$.

reset_states()

Resetting self.states to None.

Note

This is typically called when RNN unrolling is done, so that the next batch of data can start with a clean state.