The inspiration for this project came from a desire to make better tooling for grant writing, namely to suggest research topics, funding bodies, research institutions, and researchers. I have made\u00a0Innovate UK<\/a>\u00a0grant applications in the past, so I have had an interest in the government funding landscape for some time.<\/p>\n Concretely, a lot of the recent political discourse focuses on government spending, namely Elon Musk\u2019s\u00a0Department of Government Efficiency (DOGE<\/a>) in the United States and similar sentiments echoed here in the UK, as Kier Starmer looks to integrate\u00a0AI into government<\/a>.<\/p>\n Perhaps the release of this project is quite timely. Albeit not the original intention, I hope as a secondary outcome of this article is that it inspires more exploration into open source datasets for public spending.<\/p>\n I have used Networkx<\/a> & PyVis to visualise the graph of UKRI API data. Then, I detail a LlamaIndex graph RAG implementation. For completeness, I have also included my initial LangChain-based solution. The web framework is Streamlit, the demo is hosted on Streamlit community cloud.<\/p>\n This article contains the following sections.<\/p>\n UK Research and Innovation is a non-departmental public body sponsored by the Department for Science, Innovation and Technology (DSIT) that allocates funding for research and development. Generally, funding is awarded to research institutions and businesses.<\/p>\n \u201cWe invest \u00a38 billion of taxpayers\u2019 money each year into research and innovation and the people who make it happen. We work across a huge range of fields \u2014 from biodiversity conservation to quantum computing, and from space telescopes to innovative health care. We give everyone the opportunity to contribute and to benefit, bringing together people and organisations nationally and globally to create, develop and deploy new ideas and technologies.\u201d \u2014\u00a0UKRI Website<\/a><\/p>\n<\/blockquote>\n A graph is a convenient data structure showing the relationships between different entities (nodes) and their relationships to each other (edges). In some instances, we also associate those relationships with a numerical value.<\/p>\n \u201cIn computer science, a graph is an abstract data type that is meant to implement the undirected graph and directed graph concepts from the field of graph theory within mathematics. <\/p>\n A graph data structure consists of a finite (and possibly mutable) set of vertices (also called nodes or points), together with a set of unordered pairs of these vertices for an undirected graph or a set of ordered pairs for a directed graph. These pairs are known as edges (also called links or lines), and for a directed graph are also known as edges but also sometimes arrows or arcs.\u201d \u2014\u00a0Wikipedia<\/a><\/p>\n<\/blockquote>\n NetworkX is a useful library in this project to construct and store our graph. Specifically, a digraph though the library supports many graph variants such as multigraphs, the library also supports graph-related utility functions.<\/p>\n \u201cNetworkX is a Python package for the creation, manipulation, and study of the structure, dynamics, and functions of complex networks.\u201d \u2014\u00a0NetworkX Website<\/a><\/p>\n<\/blockquote>\n We use the PyVis Python package to create dynamic network views for our graph, screenshots of these can be found throughout the article.<\/p>\n \u201cThe pyvis library is meant for quick generation of visual network graphs with minimal python code. It is designed as a wrapper around the popular Javascript visJS library\u201d \u2014\u00a0PyVis Docs<\/a><\/p>\n<\/blockquote>\n LlamaIndex is a popular library for LLM applications, including support for agentic workflows, we use it to perform the graph RAG component of this project.<\/p>\n \u201cLlamaIndex (GPT Index) is a data framework for your LLM application. Building with LlamaIndex typically involves working with LlamaIndex core and a chosen set of integrations (or plugins).\u201d \u2014\u00a0LlamaIndex Github<\/a><\/p>\n<\/blockquote>\n Retrieval-augmented generation, or RAG as it is commonly known, is an AI framework for which additional context from an external knowledge base is used to ground LLM answers. Graph RAG, by extension, pertains to the use of a Graph to provide this additional context.<\/p>\n \u201cGraphRAG is a powerful retrieval mechanism that improves GenAI applications by taking advantage of the rich context in graph data structures\u2026 Basic RAG systems rely solely on semantic search in vector databases to retrieve and rank sets of isolated text fragments. While this approach can surface some relevant information, it fails to capture the context connecting these pieces. For this reason, basic RAG systems are ill-equipped to answer complex, multi-hop questions. This is where GraphRAG comes in. It uses knowledge graphs to represent and connect information to capture not only more data points but also their relationships. Thus, graph-based retrievers can provide more accurate and relevant results by uncovering hidden connections that aren\u2019t often obvious but are crucial for correlating information.\u201d \u2014\u00a0Neo4j Website<\/a><\/p>\n<\/blockquote>\n Streamlit is a lightweight Python web framework we will use to create the web application for this project.<\/p>\n \u201cStreamlit is an open-source Python framework for data scientists and AI\/ML engineers to deliver dynamic data apps with only a few lines of code. Build and deploy powerful data apps in minutes.\u201d \u2014\u00a0Streamlit website<\/a><\/p>\n<\/blockquote>\n The UKRI API is a service that facilitates access to the public UKRI grant funding dataset, authentication is not required and the docs can be found\u00a0here<\/a>. I use only two endpoints for our application, they are the\u00a0Search projects endpoint<\/a>\u00a0and the\u00a0Projects endpoint<\/a>. This allows a user to search for projects based on a keyword search and retrieve all project-specific information.<\/p>\n A search term, page size and page number are provided as query string parameters. The query string parameters;<\/p>\n Ensure that the results are returned by funded value descending.<\/p>\n I have also included the code I used for asynchronous pagination.<\/p>\n The following function is used to get project-specific data using the unique UKRI project reference. The project reference is derived from the aforementioned project search results.<\/p>\n Similarly, we parse out the relevant data for the construction of the graph and remove superfluous information.<\/p>\n There are different types of graphs, and I elected for a directed graph where the direction of the edges are important. More formally;<\/p>\n \u201cA DiGraph stores nodes and edges with optional data, or attributes. DiGraphs hold directed edges. Self loops are allowed but multiple (parallel) edges are not.\u201d \u2014\u00a0NetworkX Website<\/a><\/p>\n<\/blockquote>\n To construct the NetworkX graph, we must add nodes and edges \u2014 including the sequential updating of node attributes.<\/p>\n The standard attributes, compatible with PyVis graph rendering for nodes are as follows;<\/p>\n We also use the custom attribute \u201cfunding\u201d, which we will use to sum all of the funding for research and funding organizations. This will be normalized to set the node size according to the percentage of total funding for a particular group.<\/p>\n For our graph, we have nodes from four groups. They are classified as: HTML links can be used in the node title to allow the user to easily click through to a URL. I have included a helper function to do this below. There are project, people, and research organisation-specific links that, if redirected to provide additional information to the user.<\/p>\n The code to construct the NetworkX graph can be seen below. The DiGraph class has methods to check if a graph already has a node and similarly for edges. There are also methods for adding nodes and edges. As we iterate through projects, we want to sum the total funding amount for the funding organization and lead research institution. There are methods to both get an attribute from a node in the graph and set an attribute on a node. Depending on the source and destination node, we also apply different titles and labels to reflect that specific predicate. These can be seen in the code below.<\/p>\n Once the graph has been constructed and as previously described, I wanted to normalize the node sizes depending on the percentage of the total amount of funding for particular groups. I also append the total funding, both as a summation and as a percentage to the node label so it can be more easily viewed by a user.<\/p>\n The scale factor is just a multiple applied for aesthetic reasons, such that the node sizes appear relative to the other node groups present.<\/p>\n I allow the user to filter nodes via the UI to create a subgraph. The form to do this in Streamlit is below. I also find the neighbors of neighbors for the filtered nodes. I had some issues with Pylint raising unnecessary comprehension errors from the generator, which I have disabled\u200a\u2014\u200amore on Pylint later in the article. A smaller graph will take less time to render and will ensure that irrelevant context will be excluded.<\/p>\n NetworkX makes it easy to create a subgraph from a list of nodes with the subgraph_view<\/a> function, which takes a callable as a parameter. The callable takes a graph node as a parameter and if the boolean True value is returned, the node would be included in the subgraph.<\/p>\n To produce the visualizations I have presented earlier in the article, we must first convert the NetworkX graph to a PyVis network and then render the HTML file within the Streamlit UI.<\/p>\n If you are unfamiliar with Streamlit<\/a>, you can see one of my other articles that explore the topic here<\/a>.<\/p>\n Converting a NetworkX graph to PyVis format is relatively trivial and can be achieved with the code below. The Network class is the main class for visualization functionality, first we instantiate the class and in this example, the graph is directed. The To render the Graph to the UI, we first create a unique user ID as we use the PyVis save_graph<\/a> method to save the HTML file for the graph on the server. The uuid<\/a> ensures a unique file name, which is then read into the streamlit UI and after the file is deleted.<\/p>\n Through graph retrieval-augmented generation, we can query our graph data directly, an example can be seen in the prior screenshot. Extracted entities from the user query are looked up in the graph to give specific context to the AI to ground its response, as this information would likely not have been in the training corpus, and hence any answer given would have had an increased likelihood of being a hallucination.<\/p>\n We create a chat engine to pass a user\u2019s previous query history into the model. Usually, the Open AI API key is read as an environment variable within LlamaIndex\u200a\u2014\u200ahowever, since this is user-submitted for our application and we don\u2019t want to save users\u2019 Open AI credentials, we need to pass credentials to the LLM and embedding model classes as keyword arguments.<\/p>\n We then create an empty LlamaIndex Knowledge Graph Index and populate the knowledge graph by inserting triple<\/a>s. The triples come from traversing the edges of our NetworkX graph and calling the Since the graph is directed, we can interchange the subjects and objects so that the graph is traversable in either direction. The chat engine uses the tree_summarize option for the response builder.<\/p>\n \u201cTree summarize response\u00a0builder. This response builder recursively merges text chunks and summarizes them in a bottom-up fashion (i.e. building a tree from leaves to\u00a0root). More concretely, at each recursively step: 1. we repack the text chunks so that each chunk fills the context window of the LLM 2. if there is only one chunk, we give the final response 3. otherwise, we summarize each chunk and recursively summarize the summaries.\u201d\u2014 LlamaIndex Website<\/a><\/p>\n<\/blockquote>\n Calling the chat method with the user\u2019s query and constructing the chat history from the Streamlit state object is included here.<\/p>\n Similarly, I initially explored a LangChain<\/a> implementation, though during some experimentation, I decided to continue wth the LlamaIndex-based approach previously demonstrated. For reference, I have included this below if it is useful to you.<\/p>\n In the interest of brevity, the explanation is omitted, though it should be self-explanatory for the reader.<\/p>\n Since I have left some comments in the code to disable the linter in the examples above (examples are referenced from the GitHub repo), I thought I\u2019d cover the topic of linting briefly.<\/p>\n For those unfamiliar, linting helps to check your code for potential bugs and stylistic issues. Linters automatically enforce coding standards.<\/p>\n To get started, install Pylint<\/a> by running the command.<\/p>\n Secondly, we need to create a\u00a0.pylintrc file at the root of the project (we can also set default global and user-specific settings depending on where we create the\u00a0.pylintrc file). To do this, you will need to run.<\/p>\n We can configure this file to fit our preferences by updating the default values within the\u00a0.pylintrc file.<\/p>\n To run the linter manually, you can use.<\/p>\n When the Docker<\/a> image is built, it will automatically run Pylint and raise an error should it detect an issue with the code. This can be seen in the Dockerfile.<\/p>\n A popular formatter that you might also find useful is\u00a0Black\u00a0<\/a>\u2014<\/p>\n \u201cBlack<\/em>\u00a0is a\u00a0PEP 8<\/a>\u00a0compliant opinionated formatter.\u00a0Black<\/em>\u00a0reformats entire files in place.\u201d<\/p>\n<\/blockquote>\n Running Black will automatically resolve some of the issues that would be raised by the linter. <\/p>\n With\u00a0Streamlit Community Cloud<\/a>, anyone can host their application for free. If you have an application you\u2019d like to deploy, you can follow this\u00a0tutorial<\/a>.<\/p>\n To see the hosted demo, please click the link below.<\/p>\n https:\/\/governmentfundinggraph.streamlit.app<\/a><\/p>\n Thanks for reading my article \u2014 as promised, you can find all the code in the GitHub repo\u00a0here<\/a>.<\/p>\n Any and all feedback is valuable to me as it provides direction for my future projects. If you found this article useful, please let me know.<\/p>\n You can also find me over on\u00a0LinkedIn<\/a>\u00a0if you have specific questions.<\/p>\n Interested in open-source AI grant writing projects? Sign up for our mailing list\u00a0here<\/a>.<\/p>\n *All images, unless otherwise noted, are by the author.<\/p>\n The post Government Funding Graph RAG<\/a> appeared first on Towards Data Science<\/a>.<\/p>\n<\/div>\n![\"Government](\"https:\/\/i0.wp.com\/contributor.insightmediagroup.io\/wp-content\/uploads\/2025\/04\/graph-1.gif?ssl=1\")
\n\n
\n1. Definitions<\/h2>\n
What is UKRI?<\/h3>\n
\n
What is a Graph?<\/h3>\n
\n
![\"Government](\"https:\/\/i0.wp.com\/miro.medium.com\/v2\/resize%3Afit%3A557\/1%2AwPrq07Fs_aqg-iOgVyh6rw.png?ssl=1\")
What is\u00a0NetworkX?<\/h3>\n
\n
What is PyVis?<\/h3>\n
\n
What is LlamaIndex?<\/h3>\n
\n
What is Graph RAG?<\/h3>\n
![\"Graph](\"https:\/\/i0.wp.com\/miro.medium.com\/v2\/resize%3Afit%3A630\/1%2Aw7xjKLxEAf51wrph4ht2jg.png?ssl=1\")
\n
What is Streamlit?<\/h3>\n
\n
2. UKRI API<\/h2>\n
selectedSortableField=pro.am&selectedSortOrder=DESC<\/code><\/em><\/p>\nimport math\nimport requests\nimport concurrent.futures\nimport os \nfrom itertools import chain\nimport urllib.parse\nimport logging\n\ndef search_ukri_projects(args):\n \"\"\"\n Search UKRI projects based on a search term page size and page number.\n More details can be found here: https:\/\/gtr.ukri.org\/resources\/api.html\n \"\"\"\n search_term, page_size, page_number = args\n try:\n encoded_search_term = urllib.parse.quote(search_term)\n if (\n (\n response := requests.get(\n f\"https:\/\/gtr.ukri.org\/api\/search\/project?term={encoded_search_term}&page={page_number}&fetchSize={page_size}&selectedSortableField=pro.am&selectedSortOrder=DESC&selectedFacets=&fields=project.abs\",\n timeout=10,\n )\n )\n and (response.status_code == 200)\n and (\n items := response.json()\n .get(\"facetedSearchResultBean\", {})\n .get(\"results\")\n )\n ):\n return items\n except Exception as error:\n logging.exception(\"ERROR search_ukri_projects: %s\", error)\n return []\n\ndef search_ukri_paginate(search_term, number_of_results, page_size=100):\n \"\"\"\n Asynchronous pagination requests for project lookup.\n \"\"\"\n args = [\n (search_term, page_size, page_number + 1)\n for page_number in range(int(math.ceil(number_of_results \/ page_size)))\n ]\n with concurrent.futures.ThreadPoolExecutor(os.cpu_count()) as executor:\n future = executor.map(search_ukri_projects, args)\n results = [result for result in future if result]\n return list(chain.from_iterable(results))[:number_of_results]<\/code><\/pre>\nimport requests \nimport logging\n\ndef get_ukri_project_data(project_grant_reference):\n \"\"\"\n Search UKRI project data based on grant reference.\n \"\"\"\n try:\n if (\n (\n response := requests.get(\n f\"https:\/\/gtr.ukri.org\/api\/projects?ref={project_grant_reference}\",\n timeout=10,\n )\n )\n and (response.status_code == 200)\n and (items := response.json().get(\"projectOverview\", {}))\n ):\n return items\n except Exception as error:\n logging.exception(\"ERROR get_ukri_project_data: %s\", error)<\/code><\/pre>\ndef parse_data(projects):\n \"\"\"\n Parse project data into a usable format and validate.\n \"\"\"\n data = []\n for project in projects:\n project_composition = project.get(\"projectComposition\", {})\n project_data = project_composition.get(\"project\", {})\n fund = project_data.get(\"fund\", {})\n funder = fund.get(\"funder\")\n value_pounds = fund.get(\"valuePounds\")\n lead_research_organisation = project_composition.get(\"leadResearchOrganisation\")\n person_roles = project_composition.get(\"personRoles\")\n if all(\n [\n project_composition,\n project_data,\n fund,\n funder,\n value_pounds,\n lead_research_organisation,\n ]\n ):\n record = {}\n record[\"funder_name\"] = funder.get(\"name\")\n record[\"funder_link\"] = funder.get(\"resourceUrl\")\n record[\"project_title\"] = project_data.get(\"title\")\n record[\"project_grant_reference\"] = project_data.get(\"grantReference\")\n record[\"value\"] = value_pounds\n record[\"lead_research_organisation\"] = lead_research_organisation.get(\n \"name\", \"\"\n )\n record[\"lead_research_organisation_link\"] = lead_research_organisation.get(\n \"resourceUrl\", \"\"\n )\n record[\"people\"] = person_roles\n record[\"project_url\"] = project_data.get(\"resourceUrl\")\n data.append(record)\n return data<\/code><\/pre>\n3. Construct NetworkX Graph<\/h2>\n
\n
\n
funder_name<\/code>, lead_research_organisation<\/code>, project_title<\/code> and person_name<\/code>.<\/p>\n![\"Government](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/720\/1%2Ae5y6LrCEWLnG8iSuZ5P3Ow.png?ssl=1\")
import networkx as nx\n\ndef get_link_html(link, text):\n \"\"\"\n Helper function to construct a HTML link.\n \"\"\"\n return f\"\"\"<a href=\"{link}\" target=\"_blank\">{text}<\/a>\"\"\"\n\ndef set_networkx_attribute(graph, node_label, attribute_name, value):\n \"\"\"\n Helper to set attribute for networkx graph.\n \"\"\"\n attrs = {node_label: {attribute_name: value}}\n nx.set_node_attributes(graph, attrs)\n\ndef append_networkx_value(graph, node_label, attribute_name, value):\n \"\"\"\n Helper to append value to current node attribute scalar value.\n \"\"\"\n current_map = nx.get_node_attributes(graph, attribute_name, default=0)\n current_value = current_map[node_label]\n current_value = current_value + value\n set_networkx_attribute(graph, node_label, attribute_name, current_value)\n\ndef create_networkx(data):\n \"\"\"\n Create networkx graph from UKRI data.\n \"\"\"\n graph = nx.DiGraph()\n for row in data:\n if (\n (funder_name := row.get(\"funder_name\"))\n and (project_title := row.get(\"project_title\"))\n and (lead_research_organisation := row.get(\"lead_research_organisation\"))\n ):\n\n project_data_lookup = row.get(\"project_data_lookup\", {})\n\n if not graph.has_node(funder_name):\n graph.add_node(\n funder_name, title=funder_name, group=\"funder_name\", size=100\n )\n if not graph.has_node(project_title):\n link_html = get_link_html(\n row.get(\"project_url\", \"\").replace(\"api\/\", \"\"), project_title\n )\n graph.add_node(\n project_title,\n title=link_html,\n group=\"project_title\",\n project_data_lookup=project_data_lookup,\n size=25,\n )\n if not graph.has_edge(funder_name, project_title):\n graph.add_edge(\n funder_name,\n project_title,\n value=row.get(\"value\"),\n title=f\"{'\u00a3{:,.2f}'.format(row.get('value'))}\",\n label=f\"{'\u00a3{:,.2f}'.format(row.get('value'))}\",\n )\n\n if not graph.has_node(lead_research_organisation):\n link_html = get_link_html(\n row.get(\"lead_research_organisation_link\").replace(\"api\/\", \"\"),\n lead_research_organisation,\n )\n graph.add_node(\n lead_research_organisation,\n title=link_html,\n group=\"lead_research_organisation\",\n size=50,\n )\n if not graph.has_edge(lead_research_organisation, project_title):\n graph.add_edge(\n lead_research_organisation, project_title, title=\"RELATES TO\"\n )\n\n append_networkx_value(graph, funder_name, \"funding\", row.get(\"value\", 0))\n append_networkx_value(graph, project_title, \"funding\", row.get(\"value\", 0))\n append_networkx_value(\n graph, lead_research_organisation, \"funding\", row.get(\"value\", 0)\n )\n\n person_roles = row.get(\n \"people\", []\n ) \n\n for person in person_roles:\n if (\n (person_name := person.get(\"fullName\"))\n and (person_link := person.get(\"resourceUrl\"))\n and (project_title := row.get(\"project_title\"))\n and (roles := person.get(\"roles\"))\n ):\n if not graph.has_node(person_name):\n link_html = get_link_html(\n person_link.replace(\"api\/\", \"\"), person_name\n )\n graph.add_node(\n person_name, title=link_html, group=\"person_name\", size=10\n )\n for role in roles:\n if (not graph.has_edge(person_name, project_title)) or (\n not graph[person_name][project_title][\"title\"]\n == role.get(\"name\")\n ):\n graph.add_edge(\n person_name,\n project_title,\n title=role.get(\"name\"),\n label=role.get(\"name\"),\n )\n return graph<\/code><\/pre>\nimport networkx as nx\nimport math \nimport utils.config as config # pylint: disable=consider-using-from-import, import-error\n\ndef set_networkx_attribute(graph, node_label, attribute_name, value):\n \"\"\"\n Helper to set attribute for networkx graph.\n \"\"\"\n attrs = {node_label: {attribute_name: value}}\n nx.set_node_attributes(graph, attrs)\n\ndef calculate_total_funding_from_group(graph, group):\n \"\"\"\n Helper to calculate total funding for a group.\n \"\"\"\n return sum(\n [\n data.get(\"funding\")\n for node_label, data in graph.nodes(data=True)\n if data.get(\"funding\") and data.get(\"group\") == group\n ]\n )\n\ndef set_weighted_size_helper(graph, node_label, totals, data):\n \"\"\"\n Create normalized weights based on percentage funding amount.\n \"\"\"\n if (\n (group := data.get(\"group\"))\n and (total_funding := totals.get(group))\n and (funding := data.get(\"funding\"))\n ):\n div = funding \/ total_funding\n funding_percentage = math.ceil(((100.0 * div)))\n set_networkx_attribute(graph, node_label, \"size\", funding_percentage)\n\ndef annotate_value_on_graph(graph):\n \"\"\"\n Calculate normalized graph sizes and append to title.\n \"\"\"\n totals = {}\n for group in [\"lead_research_organisation\", \"funder_name\"]:\n totals[group] = calculate_total_funding_from_group(graph, group)\n\n for node_label, data in graph.nodes(data=True):\n if (\n (funding := data.get(\"funding\"))\n and (group := data.get(\"group\"))\n and (title := data.get(\"title\"))\n ):\n new_title = f\"{title} | {'\u00a3 {:,.0f}'.format(funding)}\"\n if total_funding := totals.get(group):\n div = funding \/ total_funding\n funding_percentage = math.ceil(((100.0 * div)))\n set_networkx_attribute(\n graph,\n node_label,\n \"size\",\n config.NODE_SIZE_SCALE_FACTOR * funding_percentage,\n )\n new_title += f\" | {' {:,.0f}'.format(funding_percentage)} %\"\n\n set_networkx_attribute(graph, node_label, \"title\", new_title)<\/code><\/pre>\n4. Filter a NetworkX\u00a0Graph<\/h2>\n
![\"Government](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/720\/1%2ArQEB2duPJiODJ6oqZu9KNw.png?ssl=1\")
import networkx as nx\nimport streamlit as st\n\ndef find_neighbor_nodes_helper(node_list, graph):\n \"\"\"\n Find unique node neighbors and flatten.\n \"\"\"\n successors_generator_array = [\n # pylint: disable=unnecessary-comprehension\n [item for item in graph.successors(node)]\n for node in node_list\n ]\n predecessors_generator_array = [\n # pylint: disable=unnecessary-comprehension\n [item for item in graph.predecessors(node)]\n for node in node_list\n ]\n neighbors = successors_generator_array + predecessors_generator_array\n flat = sum(neighbors, [])\n return list(set(flat))\n\ndef render_filter_form(annotated_node_data, graph):\n \"\"\"\n Render form to allow the user to define search nodes.\n \"\"\"\n st.session_state[\"filter\"] = st.radio(\n \"Filter\", [\"No filter\", \"Filter results\"], index=0, horizontal=True\n )\n if (filter_determinant := st.session_state.get(\"filter\")) and (\n filter_determinant == \"Filter results\"\n ):\n st.session_state[\"node_group\"] = st.selectbox(\n \"Entity type\", list(annotated_node_data.keys())\n )\n if node_group := st.session_state.get(\"node_group\"):\n ordered_lookup = dict(\n sorted(\n annotated_node_data[node_group].items(),\n key=lambda item: item[1].get(\"neighbor_len\"),\n reverse=True,\n )\n )\n st.session_state[\"search_nodes_label\"] = st.multiselect(\n \"Filter projects\", list(ordered_lookup.keys())\n )\n if search_nodes_label := st.session_state.get(\"search_nodes_label\"):\n filter_nodes = [\n ordered_lookup[label].get(\"label\") for label in search_nodes_label\n ]\n search_nodes_neighbors = find_neighbor_nodes_helper(filter_nodes, graph)\n search_nodes = find_neighbor_nodes_helper(search_nodes_neighbors, graph)\n st.session_state[\"search_nodes\"] = list(\n set(search_nodes + filter_nodes + search_nodes_neighbors)\n )<\/code><\/pre>\nimport networkx as nx\nimport streamlit as st\n\ndef filter_node(node):\n \"\"\"\n Check to see if the filter term is in the nodes selected.\n \"\"\"\n if (\n (filter_term := st.session_state.get(\"filter\"))\n and (filter_term == \"Filter results\")\n and (search_nodes := st.session_state.get(\"search_nodes\"))\n ):\n if node not in search_nodes:\n return False\n return True\n\ngraph = nx.subgraph_view(graph, filter_node=filter_node)<\/code><\/pre>\n5. Graph Visualisation Using\u00a0PyVis<\/h2>\n
barnes_hut<\/code> method is then called, which is a gravity model. The from_nx<\/code> method takes an existing NetworkX graph as an argument and translates it to PyVis, which is called in place.<\/p>\nfrom pyvis.network import Network\n\ndef convert_graph(graph):\n \"\"\"\n Convert networkx to pyvis graph.\n \"\"\"\n net = Network(\n height=\"700px\",\n width=\"100%\",\n bgcolor=\"#222222\",\n font_color=\"white\",\n directed=True,\n )\n net.barnes_hut()\n net.from_nx(graph)\n return net<\/code><\/pre>\nimport uuid\nimport contextlib\nimport os\nimport streamlit as st\n\ndef render_graphs(net):\n \"\"\"\n Helper to render graph visualization from pyvis graph.\n \"\"\"\n uuid4 = uuid.uuid4()\n file_name = f\".\/output\/{uuid4}.html\"\n with contextlib.suppress(FileNotFoundError):\n os.remove(file_name)\n net.save_graph(file_name)\n with open(file_name, \"r\", encoding=\"utf-8\") as html_file:\n source_code = html_file.read()\n st.components.v1.html(source_code, height=650, width=650)\n os.remove(file_name)<\/code><\/pre>\n6. Graph RAG Using LlamaIndex<\/h2>\n
![\"Government](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/720\/1%2ACAGDreLCUXY7j6VyXnBs0w.png?ssl=1\")
upsert_triplet_and_node<\/code> method, which will create the triple and node if they don\u2019t already exist.<\/p>\n\n
from llama_index.core import KnowledgeGraphIndex\nfrom llama_index.core.schema import TextNode\nfrom llama_index.embeddings.openai import OpenAIEmbedding\nfrom llama_index.llms.openai import OpenAI\nfrom llama_index.core.llms import ChatMessage, MessageRole\nimport streamlit as st\nimport utils.ui_utils as ui_utils # pylint: disable=consider-using-from-import, import-error\n\ndef init_llama_index_graph(graph_nx, open_ai_api_key):\n \"\"\"\n Construct a knowledge graph using llama index.\n \"\"\"\n llm = OpenAI(model=\"gpt-3.5-turbo\", api_key=open_ai_api_key)\n embed_model = OpenAIEmbedding(api_key=open_ai_api_key)\n\n graph = KnowledgeGraphIndex(\n [], llm=llm, embed_model=embed_model, api_key=open_ai_api_key\n )\n\n for subject_entity, object_entity in graph_nx.edges():\n predicate = graph_nx[subject_entity][object_entity].get(\"label\", \"relates to\")\n graph.upsert_triplet_and_node(\n (subject_entity, predicate, object_entity), TextNode(text=subject_entity)\n )\n graph.upsert_triplet_and_node(\n (object_entity, predicate, subject_entity), TextNode(text=subject_entity)\n )\n\n chat_engine = graph.as_chat_engine(\n include_text=True,\n response_mode=\"tree_summarize\",\n embedding_mode=\"hybrid\",\n similarity_top_k=5,\n verbose=True,\n llm=llm,\n )\n\n return chat_engine\n\ndef add_result_to_state(question, response):\n \"\"\"\n Add model output to state.\n \"\"\"\n if response:\n graph_answers = st.session_state.get(\"graph_answers\") or []\n graph_answers.append((question, response))\n st.session_state[\"graph_answers\"] = graph_answers\n else:\n st.error(\"Query failed, please try again later.\", icon=\"![\"\u26a0\"](\"https:\/\/i0.wp.com\/s.w.org\/images\/core\/emoji\/15.0.3\/72x72\/26a0.png?ssl=1\")
\")\n\ndef query_llama_index_graph(query_engine, question):\n \"\"\"\n Query llama index knowledge graph using graph RAG.\n \"\"\"\n graph_answers = st.session_state.get(\"graph_answers\", [])\n chat_history = []\n for query, answer in graph_answers:\n chat_history.append(ChatMessage(role=MessageRole.USER, content=query))\n chat_history.append(\n ChatMessage(role=MessageRole.ASSISTANT, content=answer)\n )\n\n if response := query_engine.chat(question, chat_history):\n add_result_to_state(question, response.response)<\/code><\/pre>\nfrom langchain_community.chains.graph_qa.base import GraphQAChain\nfrom langchain_community.graphs import NetworkxEntityGraph\nfrom langchain_community.graphs.networkx_graph import KnowledgeTriple\nfrom langchain_openai import ChatOpenAI\nimport streamlit as st\n\ndef add_result_to_state(question, response):\n \"\"\"\n Add model output to state.\n \"\"\"\n if response:\n graph_answers = st.session_state.get(\"graph_answers\") or []\n graph_answers.append((question, response))\n st.session_state[\"graph_answers\"] = graph_answers\n else:\n st.error(\"Query failed, please try again later.\", icon=\"\")\n\ndef construct_graph_langchain(graph_nx, open_ai_api_key, question):\n \"\"\"\n Construct a knowledge graph in Langchain and preform graph RAG.\n \"\"\"\n graph = NetworkxEntityGraph()\n for node in graph_nx:\n graph.add_node(node)\n\n for subject_entity, object_entity in graph_nx.edges():\n predicate = graph_nx[subject_entity][object_entity].get(\"label\", \"relates to\")\n graph.add_triple(KnowledgeTriple(subject_entity, predicate, object_entity))\n\n llm = ChatOpenAI(\n api_key=open_ai_api_key, model=\"gpt-4\", temperature=0, max_retries=2\n )\n\n chain = GraphQAChain.from_llm(llm=llm, graph=graph, verbose=True)\n\n if response := chain.invoke({\"query\": question}):\n answer = response.get(\"result\")\n add_result_to_state(question, answer)<\/code><\/pre>\n7. Linting With\u00a0Pylint<\/h2>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/720\/1%2ALNF849PxzGqpSt10hXQ3ag.png?ssl=1\")
pip install pylint<\/code><\/pre>\npylint --generate-rcfile > .pylintrc<\/code><\/pre>\npylint .\/main.py && pylint .\/**\/*.py<\/code><\/pre>\nFROM python:3.10.16 AS base \n\nWORKDIR \/app\n\nCOPY requirements.txt .\n\nRUN pip install --upgrade pip\n\nRUN pip install -r requirements.txt \n\nCOPY . .\n\nRUN mkdir -p \/app\/output\n\nRUN pylint .\/main.py && pylint .\/**\/*.py\n\nRUN python -m unittest -v tests.test_ukri_utils.Testing\n\nCMD [\"streamlit\", \"run\", \".\/main.py\"]<\/code><\/pre>\n\n
8. Streamlit Community Cloud Demo App<\/h2>\n
\nReferences<\/h2>\n
\n