PMSP

Bases: tf.keras.Model

PMSP sim 3 model. A recurrent neural network with time-averaging input that contains 3 inter-connected layers: hidden, phonology, and cleanup.

See Plaut, McClelland, Seidenberg and Patterson (1996), simulation 3.

This model provides extra functionality for "brain damage" experiments, including:

• shrink_layer: Reduce the number of units in a layer.
• zero_out: Write zero values to a portion of units in a weight matrix and make those units not trainable.
• cut_connections: Remove the specified connections.
• add_noise: Add Gaussian noise to a layer.
• apply_l2: Apply L2 regularization to all trainable weights and biases.

Parameters:

Name	Type	Description	Default
tau	float	Time-averaging parameter, from 0 to 1.	required
h_units	int	Number of units in the hidden layer.	required
p_units	int	Number of units in the phonological layer.	required
c_units	int	Number of units in the cleanup layer.	required
h_noise	float	Gaussian noise parameter (in stddev) for hidden layer.	0.0
p_noise	float	Gaussian noise parameter (in stddev) for phonological layer.	0.0
c_noise	float	Gaussian noise parameter (in stddev) for cleanup layer.	0.0
connections	List[str]	List of connections to use, each connection consists of two letters (from, to). Default is ["oh", "ph", "hp", "pp", "cp", "pc"].	None
zero_out_rates	Dict[str, float]	Dictionary of zero-out rates for each connection. Default is {c: 0.0 for c in self.connections}. See PMSP for more details.	None
l2	float	L2 regularization parameter, apply to all trainable weights and biases.	0.0

Example

import tensorflow as tf
from connectionist.data import ToyOP
from connectionist.models import PMSP

data = ToyOP()
model = PMSP(tau=0.2, h_units=10, p_units=9, c_units=5)
model.compile(
    optimizer=tf.keras.optimizers.Adam(),
    loss=tf.keras.losses.BinaryCrossentropy(),
)
model.fit(data.x_train, data.y_train, epochs=3, batch_size=20)

weights_abbreviations: Dict[str, str] property

Weight abbreviation to internal name mapping.

Note

Technically the full internal name will change when creating more than one instance of the same model. As a workaround, we use the last section of the internal name here as a matching string.

add_noise(layer, stddev)

Add noise to the target layer.

The noise is active in both training and inference.

Parameters:

Name	Type	Description	Default
layer	str	the target layer, choose from ['hidden', 'phonology', 'cleanup']	required
stddev	float	the standard deviation of the noise	required

Returns:

Type	Description
tf.keras.Model	A new model with the same architecture, but with new noise.

Example

from connectionist.models import PMSP

model = PMSP(tau=0.2, h_units=10, p_units=9, c_units=5)
model.build(input_shape=[1, 30, 10])
new_model = model.add_noise('hidden', stddev=0.1)

apply_l2(l2)

Add L2 regularization to all the weights and biases in the model.

Parameters:

Name	Type	Description	Default
l2	float	the L2 regularization rate.	required

Returns:

Type	Description
tf.keras.Model	A new model with the same architecture, but with L2 regularization applied to all the weights and biases.

Example

from connectionist.models import PMSP

model = PMSP(tau=0.2, h_units=10, p_units=9, c_units=5)
model.build(input_shape=[1, 30, 10])
new_model = model.apply_l2(l2=0.1)

call(inputs, training=False, return_internals=False)

Forward pass, identical to PMSPLayer.

Parameters:

Name	Type	Description	Default
inputs	tf.Tensor	Input tensor.	required
training	bool	Whether the model is in training mode.	False
return_internals	bool	Whether to return intermediate inputs to each connection.	False

Returns:

Name	Type	Description
outputs	Dict[str, tf.Tensor]	a dictionary that stores all outputs.

cut_connections(connections)

Cut connections between two layers.

Parameters:

Name	Type	Description	Default
connections	List[str]	the connections to be cut, the connections must be found in the original model, i.e., in model.connections.	required

Returns:

Type	Description
tf.keras.Model	A new model with the same architecture, but with new connections.

Example

from connectionist.models import PMSP

model = PMSP(tau=0.2, h_units=10, p_units=9, c_units=5)
model.build(input_shape=[1, 30, 10])
new_model = model.cut_connections(['pp', 'pc'])

shrink_layer(layer, rate)

Shrink the number of units in a layer, and all its dependent connections by random sampling.

Parameters:

Name	Type	Description	Default
layer	str	the target layer, choose from ['hidden', 'phonology', 'cleanup']	required
rate	float	the shrink rate	required

Returns:

Type	Description
tf.keras.Model	A new model with the same architecture, but with new weights shapes that match with the shrank layer.

Example

from connectionist.models import PMSP

model = PMSP(tau=0.2, h_units=10, p_units=9, c_units=5)
model.build(input_shape=[1, 30, 10])
new_model = model.shrink_layer('hidden', rate=0.5)

to_units(layer)

Get the number of units in a target layer.

Parameters:

Name	Type	Description	Default
layer	str	the target layer, choose from ['hidden', 'phonology', 'cleanup']	required

Returns:

Name	Type	Description
int	int	the number of units in the target layer

zero_out(rates)

Zero out weights of the target connections.

Parameters:

Name	Type	Description	Default
rates	Dict[str, float]	the zero out rates for each connection. e.g., {'hc': 0.5, 'ph': 0.4}. Higher zero out rates means more weights will be zeroed out.	required

Returns:

Type	Description
tf.keras.Model	A new model with the same architecture, but with new weights.

Example

from connectionist.models import PMSP

model = PMSP(tau=0.2, h_units=10, p_units=9, c_units=5)
model.build(input_shape=[1, 30, 10])
new_model = model.zero_out(rates={'hp': 0.5, 'pc': 0.4})