What is a fully connected layer?

A fully connected layer, also known as a dense layer, is a type of neural network layer where every neuron is connected to every neuron in the preceding layer. This means each input feature from the previous layer is multiplied by a weight, summed with a bias term, and passed through an activation function to produce an output. These layers are commonly used in the final stages of neural networks for tasks like classification or regression, where combining all learned features into a unified prediction is necessary. For example, in image classification, a fully connected layer might take flattened pixel values or features extracted by convolutional layers and map them to class probabilities.

The structure of a fully connected layer is defined by its weights matrix and biases vector. If a layer has (n) inputs and (m) outputs, the weights matrix will have dimensions (n \times m), and the biases vector will have (m) elements. Each output neuron computes a weighted sum of all inputs, adds its bias, and applies an activation function like ReLU or sigmoid. This design allows the layer to model complex relationships between features but also introduces a large number of parameters. For instance, a fully connected layer with 1,000 inputs and 500 outputs requires (1,000 \times 500 = 500,000) weights and 500 biases, which can make training computationally intensive.

In practice, fully connected layers are often paired with other layer types like convolutional or recurrent layers. For example, in a convolutional neural network (CNN) for image recognition, convolutional layers detect edges and textures, while fully connected layers at the end interpret these features to predict the image class. However, their dense connectivity can lead to overfitting, especially with limited data. Techniques like dropout (randomly deactivating neurons during training) or L2 regularization (penalizing large weights) are commonly used to mitigate this. Despite their computational cost, fully connected layers remain a staple in neural networks for their flexibility in combining global information.