In this post, I will explain Graphsage<\/a> and how it solves common problems of GCNs and GATs. We will train GraphSAGE and use it for graph predictions to compare performance with GCNs and GATs.<\/p>\n New to GNNs? You can start with\u00a0post 1 about GCNs<\/a>\u00a0(also containing the initial setup for running the code samples), and post 2 about GATs<\/a>.\u00a0<\/p>\n<\/blockquote>\n I shortly touched upon it in the introduction, but let\u2019s dive a bit deeper. What are the problems with the previous GNN models?<\/p>\n GCNs and GATs struggle with generalizing to unseen graphs. The graph structure needs to be the same as the training data. This is known as\u00a0transductive learning<\/em>, where the model trains and makes predictions on the same fixed graph. It is actually overfitting to specific graph topologies. In reality, graphs will change: Nodes and edges can be added or removed, and this happens often in real world scenarios. We want our GNNs to be capable of learning patterns that generalize to unseen nodes, or to entirely new graphs (this is called\u00a0inductive<\/em>\u00a0learning<\/em>).<\/p>\n Training GCNs and GATs on large-scale graphs is computationally expensive. GCNs require repeated neighbor aggregation, which grows exponentially with graph size, while GATs involve (multihead) attention mechanisms that scale poorly with increasing nodes. Let\u2019s take a look at GraphSAGE to fix these issues.<\/p>\n GraphSAGE<\/a>\u00a0makes training much faster and scalable. It does this by sampling only a subset of neighbors<\/em>. For super large graphs it\u2019s computationally impossible to process all neighbors of a node (except if you have limitless time, which we all don\u2019t\u2026), like with traditional GCNs. Another important step of GraphSAGE is\u00a0combining the features of the sampled neighbors with an aggregation function<\/em>.\u00a0 With tabular data, sampling is easy. It\u2019s something you do in every common machine learning project when creating train, test, and validation sets. With graphs, you cannot select random nodes. This can result in disconnected graphs, nodes without neighbors, etcetera:<\/p>\n What you\u00a0can<\/em>\u00a0do with graphs, is selecting a random fixed-size subset of neighbors. For example in a social network, you can sample 3 friends for each user (instead of all friends):<\/p>\n After the neighbor selection from the previous part, GraphSAGE combines their features into one single representation. There are multiple ways to do this (multiple\u00a0aggregation functions<\/em>). The most common types and the ones explained in the paper are\u00a0mean aggregation<\/em>,\u00a0LSTM<\/em>, and\u00a0pooling<\/em>.\u00a0<\/p>\n With mean aggregation, the average is computed over all sampled neighbors\u2019 features (very simple and often effective). In a formula:<\/p>\n LSTM aggregation uses an\u00a0LSTM<\/a>\u00a0(type of neural network) to process neighbor features sequentially. It can capture more complex relationships, and is more powerful than mean aggregation.\u00a0<\/p>\n The third type, pool aggregation, applies a non-linear function to extract key features (think about\u00a0max-pooling<\/a>\u00a0in a neural network, where you also take the maximum value of some values).<\/p>\n After sampling and aggregation, the node\u00a0combines its previous features with the aggregated neighbor features<\/em>. Nodes will learn from their neighbors but also keep their own identity, just like we saw before with GCNs and GATs. Information can flow across the graph effectively.\u00a0<\/p>\n This is the formula for this step:<\/p>\n The aggregation of step 2 is done over all neighbors, and then the feature representation of the node is concatenated. This vector is multiplied by the weight matrix, and passed through non-linearity (for example ReLU). As a final step, normalization can be applied.<\/p>\n The first three steps can be repeated multiple times, when this happens, information can flow from distant neighbors. In the image below you see a node with three neighbors selected in the first layer (direct neighbors), and two neighbors selected in the second layer (neighbors of neighbors).\u00a0<\/p>\n To summarize, the key strengths of GraphSAGE are its scalability (sampling makes it efficient for massive graphs); flexibility, you can use it for Inductive learning<\/a> (works well when used for predicting on unseen nodes and graphs); aggregation helps with generalization because it smooths out noisy features; and the multi-layers allow the model to learn from far-away nodes. <\/p>\n Cool! And the best thing, GraphSAGE is implemented in\u00a0PyG<\/a>, so we can use it easily in PyTorch.<\/p>\n In the previous posts, we implemented an MLP, GCN, and GAT on the Cora<\/a>\u00a0dataset (CC BY-SA). To refresh your mind a bit, Cora is a dataset with scientific publications where you have to predict the subject of each paper, with seven classes in total. This dataset is relatively small, so it might be not the best set for testing GraphSAGE. We will do this anyway, just to be able to compare. Let\u2019s see how well GraphSAGE performs.<\/p>\n Interesting parts of the code I like to highlight related to GraphSAGE:<\/p>\n This is the complete script, we train 100 epochs and repeat the experiment 10 times to calculate average accuracy and standard deviation for each model:<\/p>\n And here are the results:<\/p>\n Impressive improvement! Even on this small dataset, GraphSAGE outperforms GAT and GCN easily! I repeated this test for CiteSeer and PubMed datasets, and always GraphSAGE came out best.\u00a0<\/p>\n What I like to note here is that GCN is still very useful, it\u2019s one of the most effective baselines (if the graph structure allows it). Also, I didn\u2019t do much hyperparameter tuning, but just went with some standard values (like 8 heads for the GAT multi-head attention). In larger, more complex and noisier graphs, the advantages of GraphSAGE become more clear than in this example. We didn\u2019t do any performance testing, because for these small graphs GraphSAGE isn\u2019t faster than GCN.<\/p>\n GraphSAGE brings us very nice improvements and benefits compared to GATs and GCNs. Inductive learning is possible, GraphSAGE can handle changing graph structures quite well. And we didn\u2019t test it in this post, but neighbor sampling makes it possible to create feature representations for larger graphs with good performance.\u00a0<\/p>\n Optimizing Connections: Mathematical Optimization within Graphs<\/strong><\/a><\/p>\n Graph Neural Networks Part 1. Graph Convolutional Networks Explained<\/strong><\/a><\/p>\n Graph Neural Networks Part 2. Graph Attention Networks vs. GCNs<\/strong><\/a><\/p>\n<\/blockquote>\n The post Graph Neural Networks Part 3: How GraphSAGE Handles Changing Graph Structure<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n\n
\nTwo Key Problems with GCNs and\u00a0GATs<\/h2>\n
Problem 1. They don\u2019t generalize<\/h3>\n
Problem 2. They have scalability issues<\/h3>\n
In big production recommendation systems that have large graphs with millions of users and products, GCNs and GATs are impractical and slow.<\/p>\nGraphSAGE (SAmple and aggreGatE)<\/h2>\n
We will walk through all the steps of GraphSAGE below.<\/p>\n1. Sampling Neighbors<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/disconnected.drawio-3.png?ssl=1\")
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/neighborselection.drawio-5.png?ssl=1\")
2. Aggregate Information<\/h3>\n
![](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/1%2AmvSOnRmQxgY4X-KokooSMw.png?ssl=1\")
<\/p>\n3. Update Node Representation<\/h3>\n
![](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/1600\/1%2A8QkSpHp70K4bq1e4EsG1Qg.png?ssl=1\")
<\/p>\n4. Repeat for Multiple\u00a0Layers<\/h3>\n
![\"\"](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/03\/graphsage.drawio-4.png?ssl=1\")
Predicting with GraphSAGE<\/h2>\n
\n
NeighborLoader<\/code>\u00a0that performs selecting the neighbors for each layer:<\/li>\n<\/ul>\nfrom torch_geometric.loader import NeighborLoader\n\n# 10 neighbors sampled in the first layer, 10 in the second layer\nnum_neighbors = [10, 10]\n\n# sample data from the train set\ntrain_loader = NeighborLoader(\n data,\n num_neighbors=num_neighbors,\n batch_size=batch_size,\n input_nodes=data.train_mask,\n)<\/code><\/pre>\n\n
SAGEConv<\/code>\u00a0layer. The default is\u00a0mean<\/code>, you can change this to\u00a0max<\/code>\u00a0or\u00a0lstm<\/code>:<\/li>\n<\/ul>\nfrom torch_geometric.nn import SAGEConv\n\nSAGEConv(in_c, out_c, aggr='mean')<\/code><\/pre>\n\n
model_type<\/code>\u00a0(GCN, GAT, or SAGE). This makes it easy to compare or make small changes.<\/li>\n<\/ul>\nimport torch\nimport torch.nn.functional as F\nfrom torch_geometric.nn import SAGEConv, GCNConv, GATConv\nfrom torch_geometric.datasets import Planetoid\nfrom torch_geometric.loader import NeighborLoader\n\n# dataset_name can be 'Cora', 'CiteSeer', 'PubMed'\ndataset_name = 'Cora'\nhidden_dim = 64\nnum_layers = 2\nnum_neighbors = [10, 10]\nbatch_size = 128\nnum_epochs = 100\nmodel_types = ['GCN', 'GAT', 'SAGE']\n\ndataset = Planetoid(root='data', name=dataset_name)\ndata = dataset[0]\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\ndata = data.to(device)\n\nclass GNN(torch.nn.Module):\n def __init__(self, in_channels, hidden_channels, out_channels, num_layers, model_type='SAGE', gat_heads=8):\n super().__init__()\n self.convs = torch.nn.ModuleList()\n self.model_type = model_type\n self.gat_heads = gat_heads\n\n def get_conv(in_c, out_c, is_final=False):\n if model_type == 'GCN':\n return GCNConv(in_c, out_c)\n elif model_type == 'GAT':\n heads = 1 if is_final else gat_heads\n concat = False if is_final else True\n return GATConv(in_c, out_c, heads=heads, concat=concat)\n else:\n return SAGEConv(in_c, out_c, aggr='mean')\n\n if model_type == 'GAT':\n self.convs.append(get_conv(in_channels, hidden_channels))\n in_dim = hidden_channels * gat_heads\n for _ in range(num_layers - 2):\n self.convs.append(get_conv(in_dim, hidden_channels))\n in_dim = hidden_channels * gat_heads\n self.convs.append(get_conv(in_dim, out_channels, is_final=True))\n else:\n self.convs.append(get_conv(in_channels, hidden_channels))\n for _ in range(num_layers - 2):\n self.convs.append(get_conv(hidden_channels, hidden_channels))\n self.convs.append(get_conv(hidden_channels, out_channels))\n\n def forward(self, x, edge_index):\n for conv in self.convs[:-1]:\n x = F.relu(conv(x, edge_index))\n x = self.convs[-1](x, edge_index)\n return x\n\n@torch.no_grad()\ndef test(model):\n model.eval()\n out = model(data.x, data.edge_index)\n pred = out.argmax(dim=1)\n accs = []\n for mask in [data.train_mask, data.val_mask, data.test_mask]:\n accs.append(int((pred[mask] == data.y[mask]).sum()) \/ int(mask.sum()))\n return accs\n\nresults = {}\n\nfor model_type in model_types:\n print(f'Training {model_type}')\n results[model_type] = []\n\n for i in range(10):\n model = GNN(dataset.num_features, hidden_dim, dataset.num_classes, num_layers, model_type, gat_heads=8).to(device)\n optimizer = torch.optim.Adam(model.parameters(), lr=0.01, weight_decay=5e-4)\n\n if model_type == 'SAGE':\n train_loader = NeighborLoader(\n data,\n num_neighbors=num_neighbors,\n batch_size=batch_size,\n input_nodes=data.train_mask,\n )\n\n def train():\n model.train()\n total_loss = 0\n for batch in train_loader:\n batch = batch.to(device)\n optimizer.zero_grad()\n out = model(batch.x, batch.edge_index)\n loss = F.cross_entropy(out, batch.y[:out.size(0)])\n loss.backward()\n optimizer.step()\n total_loss += loss.item()\n return total_loss \/ len(train_loader)\n\n else:\n def train():\n model.train()\n optimizer.zero_grad()\n out = model(data.x, data.edge_index)\n loss = F.cross_entropy(out[data.train_mask], data.y[data.train_mask])\n loss.backward()\n optimizer.step()\n return loss.item()\n\n best_val_acc = 0\n best_test_acc = 0\n for epoch in range(1, num_epochs + 1):\n loss = train()\n train_acc, val_acc, test_acc = test(model)\n if val_acc > best_val_acc:\n best_val_acc = val_acc\n best_test_acc = test_acc\n if epoch % 10 == 0:\n print(f'Epoch {epoch:02d} | Loss: {loss:.4f} | Train: {train_acc:.4f} | Val: {val_acc:.4f} | Test: {test_acc:.4f}')\n\n results[model_type].append([best_val_acc, best_test_acc])\n\nfor model_name, model_results in results.items():\n model_results = torch.tensor(model_results)\n print(f'{model_name} Val Accuracy: {model_results[:, 0].mean():.3f} \u00b1 {model_results[:, 0].std():.3f}')\n print(f'{model_name} Test Accuracy: {model_results[:, 1].mean():.3f} \u00b1 {model_results[:, 1].std():.3f}')\n<\/code><\/pre>\nGCN Val Accuracy: 0.791 \u00b1 0.007\nGCN Test Accuracy: 0.806 \u00b1 0.006\nGAT Val Accuracy: 0.790 \u00b1 0.007\nGAT Test Accuracy: 0.800 \u00b1 0.004\nSAGE Val Accuracy: 0.899 \u00b1 0.005\nSAGE Test Accuracy: 0.907 \u00b1 0.004<\/code><\/pre>\n
\nConclusion<\/h2>\n
Related<\/h3>\n
\n