Machine Learning Granularity by Splitting Neurons

This is just a simple Sequential model with one hidden Dense layer and 1,600 parameters. Now let’s compile and fit the model.

batch_size = 128
# epochs won't matter, we'll be using the EarlyStopping callback
epochs = 1000
model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1, callbacks=keras.callbacks.EarlyStopping(monitor="val_accuracy", min_delta=.01))

You may note the EarlyStopping callback in our fit method. This will ensure the model stops once the validation accuracy is changing less than 1% on two consecutive epochs. Now we want to get the weights and biases from the model. We can do that via Keras’ built in get_weights method.

# this gets a list of the weights (2 arrays) and biases (2 arrays)
weights=model.get_weights()

Each layer contains two arrays, one for weights and one for biases. The weights get multiplied via matrix multiplication. The biases then get added to the resulting vector. Our model contains a Dense layer and an Output layer. So the structure of the weights and biases is this:

for i in weights:
print(i.shape)------------------
(784, 2)  <-----Layer1 Dense Weights
(2,)      <-----Layer1 Biases
(2, 10)   <-----Output Layer Weights
(10,)     <-----Output Layer Biases

We do NOT want to split the output as we still should have 10 categories. We will need to do the following:

1. Duplicate all elements in the Layer1 Dense Weights, Layer1 Biases, and Output Layer Weights.
2. Divide the elements of the Output Layer Weights by 2.
3. The new neuron pairs in Layer1 will be identical. And so the loss function may end up adjusting them both the exact same. So we will want to nudge their biases a little in opposite directions so each identical neuron pair has a dominant neuron and a passive one that can continue to diversify on further training.

c=0
# loop through list of numpy arrays
for i in weights:
if c < 3:
# repeat elements on layer 1 weights and biases
if c < 2:
layerpre = np.repeat(i, 2, axis=-1)
#nudge biases
if c==1:
layerpre[::2] += 0.001
layerpre[1::2] += -0.001
weights[c] = layerpre
# repeat elements on output layer weights and divide by two
if c==2:
layerpre = np.repeat(i, 2, axis=0)
layerpre = layerpre/2
weights[c] = layerpre
#Note that we do not change output layer biases
else:
break
c+=1

This gives us a new list of arrays:

for i in weights:
print(i.shape)------------------
(784, 4)  <-----Layer2 Dense Weights
(4,)      <-----Layer2 Biases
(4, 10)   <-----Output Layer Weights
(10,)     <-----Output Layer Biases

And we can see that the above arrays are the shapes we want. Let’s now create a second model architecture with two times the neurons.

# create second model architecture, only difference is twice the neurons in the first layer
model2 = keras.Sequential(
[
keras.Input(shape=input_shape),
layers.Flatten(),
layers.Dense(num_neurons*2, activation="relu"),
layers.Dense(num_classes, activation="softmax"),
]
)
model2.summary()
model2.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
# Set the weights before training
model2.set_weights(weights)

Use the set_weights method to pass all our processed weights and biases to the new model. Let’s get the summary:

model2.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1,
callbacks=keras.callbacks.EarlyStopping(monitor="val_accuracy", min_delta=.005))