Large Language models (LLMs) have witnessed impressive progress and these large models can do a variety of tasks, from generating human-like text to answering questions. However, understanding how these models work still remains challenging, especially due a phenomenon called superposition where features are mixed into one neuron, making it very difficult to extract human understandable representation from the original model structure. This is where methods like sparse Autoencoder appear to disentangle the features for interpretability.\u00a0<\/p>\n
In this blog post, we will use the Sparse Autoencoder to find some feature circuits on a particular interesting case of subject-verb agreement ,and understand how the model components contribute to the task.<\/p>\n
Key concepts\u00a0<\/h2>\nFeature circuits\u00a0<\/h3>\n
In the context of neural networks, feature circuits<\/strong> are how networks learn to combine input features to form complex patterns at higher levels. We use the metaphor of \u201ccircuits\u201d to describe how features are processed along layers in a neural network because such processes remind us of circuits in electronics processing and combining signals.<\/p>\n These feature circuits form gradually through the connections between neurons and layers, where each neuron or layer is responsible for transforming input features, and their interactions lead to useful feature combinations that play together to make the final predictions.<\/p>\n Here is one example of feature circuits: in lots of vision neural networks, we can find \u201ca circuit as a family of units detecting curves in different angular orientations. Curve detectors are primarily implemented from earlier, less sophisticated curve detectors and line detectors. These curve detectors are used in the next layer to create 3D geometry and complex shape detectors\u201d [1].\u00a0<\/p>\n In the coming chapter, we will work on one feature circuit in LLMs for a subject-verb agreement task.\u00a0<\/p>\n In the context of Machine Learning<\/a>, we have sometimes observed superposition, referring to the phenomenon that one neuron in a model represents multiple overlapping features rather than a single, distinct one. For example, InceptionV1 contains one neuron that responds to cat faces, fronts of cars, and cat legs.\u00a0<\/p>\n This is where the Sparse Autoencoder<\/a> (SAE) comes in.<\/p>\n The SAE helps us disentangle the network\u2019s activations into a set of sparse features. These sparse features are normally human understandable,m allowing us to get a better understanding of the model. By applying an SAE to the hidden layers activations of an LLM mode, we can isolate the features that contribute to the model\u2019s output.\u00a0<\/p>\n You can find the details of how the SAE works in my former blog post<\/a>.\u00a0<\/p>\n Subject-verb agreement is a fundamental grammar rule in English. The subject and the verb in a sentence must be consistent in numbers, aka singular or plural. For example:<\/p>\n Understanding this rule simple for humans is important for tasks like text generation, translation, and question answering. But how do we know if an LLM has actually learned this rule?\u00a0<\/p>\n We will now explore in this chapter how the LLM forms a feature circuit for such a task.\u00a0<\/p>\n Let\u2019s now build the process of creating the feature circuit. We would do it in 4 steps:<\/p>\n We start by building a toy language model which might have no sense at all with the following code.\u00a0This is a network with two simple layers.\u00a0<\/p>\n For the subject-verb agreement, the model is supposed to:\u00a0<\/p>\n It is unclear what happens inside the hidden layer. So we introduce the following sparse AutoEncoder:\u00a0<\/p>\n We train the original model Now we visualise the feature circuit. As introduced before, a feature circuit is a unit of neurons for processing specific features. In our model, the feature consists:\u00a0<\/p>\n You can see in the plot that we visualize the feature circuit as a graph:\u00a0<\/p>\n For a real case, we run the similar code on GPT2-small. We show the graph of a feature circuit representing the decision to choose the singular verb. <\/p>\n Feature circuits help us to understand how different parts in a complex LLM lead to a final output. We show the possibility to use an SAE to form a feature circuit for a subject-verb agreement task.\u00a0<\/p>\n However, we have to admit this method still needs some human-level intervention in the sense that we don\u2019t always know if a circuit can really form without a proper design.<\/p>\n [1] Zoom In: An Introduction to Circuits<\/a><\/p>\n The post Formulation of Feature Circuits with Sparse Autoencoders in LLM<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\nSuperposition and Sparse AutoEncoder\u00a0<\/h3>\n
Case study: Subject-Verb Agreement<\/h2>\n
Subject-Verb Agreement\u00a0<\/h3>\n
\n
Building the Feature Circuit<\/h3>\n
\n
\n
\n
\n
Toy Model\u00a0<\/h3>\n
\n
# ====== Define Base Model (Simulating Subject-Verb Agreement) ======\nclass SubjectVerbAgreementNN(nn.Module):\n def __init__(self):\n super().__init__()\n self.hidden = nn.Linear(2, 4) # 2 input \u2192 4 hidden activations\n self.output = nn.Linear(4, 2) # 4 hidden \u2192 2 output (runs\/run)\n self.relu = nn.ReLU()\n\n\n def forward(self, x):\n x = self.relu(self.hidden(x)) # Compute hidden activations\n return self.output(x) # Predict verb<\/code><\/pre>\n# ====== Define Sparse Autoencoder (SAE) ======\nclass c(nn.Module):\n def __init__(self, input_dim, hidden_dim):\n super().__init__()\n self.encoder = nn.Linear(input_dim, hidden_dim) # Decompress to sparse features\n self.decoder = nn.Linear(hidden_dim, input_dim) # Reconstruct\n self.relu = nn.ReLU()\n\n\n def forward(self, x):\n encoded = self.relu(self.encoder(x)) # Sparse activations\n decoded = self.decoder(encoded) # Reconstruct original activations\n return encoded, decoded<\/code><\/pre>\nSubjectVerbAgreementNN<\/code> and the SubjectVerbAgreementNN<\/code> with sentences designed to represent different singular and plural forms of verbs, such as \u201cThe cat runs\u201d, \u201cthe babies run\u201d. However, just like before, for the toy model, they may not have actual meanings.\u00a0<\/p>\n\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-19-at-12.53.18%25E2%2580%25AFPM.png?resize=894%2C549&ssl=1\")
\n
GPT2-Small\u00a0<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/towardsdatascience.com\/wp-content\/uploads\/2025\/02\/Screenshot-2025-02-19-at-12.54.48%25E2%2580%25AFPM-1024x121.png?resize=1024%2C121&ssl=1\")
Conclusion\u00a0<\/h2>\n
Reference\u00a0<\/h2>\n