Learn how to create an agent that understands your home\u2019s context, learns your preferences, and interacts with you and your home to accomplish activities you find valuable.<\/h4>\n![\"\"](\"https:\/\/cdn-images-1.medium.com\/max\/1024\/0*FmWlPYUgsHV6Wo2G\")
Photo by Igor Omilaev<\/a> on\u00a0Unsplash<\/a><\/figcaption><\/figure>\nIntroduction<\/h3>\n
This article describes the architecture and design of a Home Assistant<\/a> (HA) integration called home-generative-agent. This project uses LangChain<\/a> and LangGraph<\/a> to create a generative AI agent<\/a> that interacts with and automates tasks within a HA smart home environment. The agent understands your home\u2019s context, learns your preferences, and interacts with you and your home to accomplish activities you find valuable. Key features include creating automations, analyzing images, and managing home states using various LLMs (Large Language Models). The architecture involves both cloud-based and edge-based models for optimal performance and cost-effectiveness. Installation instructions, configuration details, and information on the project\u2019s architecture and the different models used are included and can be found on the home-generative-agent<\/a> GitHub. The project is open-source and welcomes contributions.<\/p>\nThese are some of the features currently supported:<\/p>\n
\n- Create complex Home Assistant automations.<\/li>\n
- Image scene analysis and understanding.<\/li>\n
- Home state analysis of entities, devices, and\u00a0areas.<\/li>\n
- Full agent control of allowed entities in the\u00a0home.<\/li>\n
- Short- and long-term memory using semantic\u00a0search.<\/li>\n
- Automatic summarization of home state to manage LLM context\u00a0length.<\/li>\n<\/ul>\n
This is my personal project and an example of what I call learning-directed hacking<\/em><\/a>. The project is not affiliated with my work at Amazon nor am I associated with the organizations responsible for Home Assistant or LangChain\/LangGraph in any\u00a0way.<\/p>\nImportant Considerations<\/h3>\n
Creating an agent to monitor and control your home can lead to unexpected actions and potentially put your home and yourself at risk due to LLM hallucinations and privacy concerns, especially when exposing home states and user information to cloud-based LLMs. I have made reasonable architectural and design choices to mitigate these risks, but they cannot be completely eliminated.<\/p>\n
One key early decision was to rely on a hybrid cloud-edge approach. This enables the use of the most sophisticated reasoning and planning models available, which should help reduce hallucinations. Simpler, more task-focused edge models are employed to further minimize LLM\u00a0errors.<\/p>\n
Another critical decision was to leverage LangChain\u2019s capabilities, which allow sensitive information to be hidden from LLM tools and provided only at runtime. For instance, tool logic may require using the ID of the user who made a request. However, such values should generally not be controlled by the LLM. Allowing the LLM to manipulate the user ID could pose security and privacy risks. To mitigate this, I utilized the InjectedToolArg<\/em><\/a> annotation.<\/p>\nAdditionally, using large cloud-based LLMs incurs significant cloud costs, and the edge hardware required to run LLM edge models can be expensive. The combined operational and installation costs are likely prohibitive for the average user at this time. An industry-wide effort to \u201cmake LLMs as cheap as CNNs\u201d is needed to bring home agents to the mass\u00a0market.<\/p>\n
It is important to be aware of these risks and understand that, despite these mitigations, we are still in the early stages of this project and home agents in general. Significant work remains to make these agents truly useful and trustworthy assistants.<\/em><\/strong><\/p>\nArchitecture and\u00a0Design<\/h3>\n
Below is a high-level view of the home-generative-agent architecture.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/940\/1%2AXjehp4jA_Ptbu3_U391Adg.png?ssl=1\")
Diagram by Lindo St.\u00a0Angel<\/figcaption><\/figure>\nThe general integration architecture follows the best practices as described in Home Assistant Core<\/a> and is compliant with Home Assistant Community Store<\/a> (HACS) publishing requirements.<\/p>\nThe agent is built using LangGraph and uses the HA conversation<\/em> component to interact with the user. The agent uses the Home Assistant LLM API to fetch the state of the home and understand the HA native tools it has at its disposal. I implemented all other tools available to the agent using LangChain. The agent employs several LLMs, a large and very accurate primary model for high-level reasoning, smaller specialized helper models for camera image analysis, primary model context summarization, and embedding generation for long-term semantic search. The primary model is cloud-based, and the helper models are edge-based and run under the Ollama<\/a> framework on a computer located in the\u00a0home.<\/p>\nThe models currently being used are summarized below.<\/p>\n
\n- \nGPT-4o<\/a> | OpenAI Cloud | High-level reasoning and\u00a0planning<\/li>\n
- \nllama-3.2-vision-11b<\/a> | Ollama Edge | Image scene\u00a0analysis<\/li>\n
- \nllama-3.2-vision-11b<\/a> | Ollama Edge | Primary model context summarization<\/li>\n
- \nmxbai-embed-large<\/a> | Ollama Edge | Embedding generation for sematic\u00a0search<\/li>\n<\/ul>\n
LangGraph-based Agent<\/h4>\n
LangGraph powers the conversation agent, enabling you to create stateful, multi-actor applications utilizing LLMs as quickly as possible. It extends LangChain\u2019s capabilities, introducing the ability to create and manage cyclical graphs essential for developing complex agent runtimes. A graph models the agent workflow, as seen in the image\u00a0below.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/356\/1%2AUJDKU7QaYwzMwXOTlVVIew.png?ssl=1\")
Diagram by Lindo St.\u00a0Angel<\/figcaption><\/figure>\nThe agent workflow has five nodes, each Python module modifying the agent\u2019s state, a shared data structure. The edges between the nodes represent the allowed transitions between them, with solid lines unconditional and dashed lines conditional. Nodes do the work, and edges tell what to do\u00a0next.<\/p>\n
The __start__<\/em> and __end__<\/em> nodes inform the graph where to start and stop. The agent<\/em> node runs the primary LLM, and if it decides to use a tool, the action<\/em> node runs the tool and then returns control to the agent<\/em>. The summarize_and_trim<\/em> node processes the LLM\u2019s context to manage growth while maintaining accuracy if agent<\/em> has no tool to call and the number of messages meets the below-mentioned conditions.<\/p>\nLLM Context Management<\/h4>\n
You need to carefully manage the context length of LLMs to balance cost, accuracy, and latency and avoid triggering rate limits such as OpenAI\u2019s Tokens per Minute restriction. The system controls the context length of the primary model in two ways: it trims the messages in the context if they exceed a max parameter, and the context is summarized once the number of messages exceeds another parameter. These parameters are configurable in const.py<\/em>; their description is\u00a0below.<\/p>\n\n- \nCONTEXT_MAX_MESSAGES<\/em> | Messages to keep in context before deletion | Default =\u00a0100<\/li>\n
- \nCONTEXT_SUMMARIZE_THRESHOLD<\/em> | Messages in context before summary generation | Default =\u00a020<\/li>\n<\/ul>\n
The summarize_and_trim<\/em> node in the graph will trim the messages only after content summarization. You can see the Python code associated with this node in the snippet\u00a0below.<\/p>\nasync def _summarize_and_trim(
state: State, config: RunnableConfig, *, store: BaseStore
) -> dict[str, list[AnyMessage]]:
\"\"\"Coroutine to summarize and trim message history.\"\"\"
summary = state.get(\"summary\", \"\")
if summary:
summary_message = SUMMARY_PROMPT_TEMPLATE.format(summary=summary)
else:
summary_message = SUMMARY_INITIAL_PROMPT
messages = (
[SystemMessage(content=SUMMARY_SYSTEM_PROMPT)] +
state[\"messages\"] +
[HumanMessage(content=summary_message)]
)
model = config[\"configurable\"][\"vlm_model\"]
options = config[\"configurable\"][\"options\"]
model_with_config = model.with_config(
config={
\"model\": options.get(
CONF_VLM,
RECOMMENDED_VLM,
),
\"temperature\": options.get(
CONF_SUMMARIZATION_MODEL_TEMPERATURE,
RECOMMENDED_SUMMARIZATION_MODEL_TEMPERATURE,
),
\"top_p\": options.get(
CONF_SUMMARIZATION_MODEL_TOP_P,
RECOMMENDED_SUMMARIZATION_MODEL_TOP_P,
),
\"num_predict\": VLM_NUM_PREDICT,
}
)
LOGGER.debug(\"Summary messages: %s\", messages)
response = await model_with_config.ainvoke(messages)
# Trim message history to manage context window length.
trimmed_messages = trim_messages(
messages=state[\"messages\"],
token_counter=len,
max_tokens=CONTEXT_MAX_MESSAGES,
strategy=\"last\",
start_on=\"human\",
include_system=True,
)
messages_to_remove = [m for m in state[\"messages\"] if m not in trimmed_messages]
LOGGER.debug(\"Messages to remove: %s\", messages_to_remove)
remove_messages = [RemoveMessage(id=m.id) for m in messages_to_remove]
return {\"summary\": response.content, \"messages\": remove_messages}<\/pre>\nLatency<\/h4>\n
The latency between user requests or the agent taking timely action on the user\u2019s behalf is critical for you to consider in the design. I used several techniques to reduce latency, including using specialized, smaller helper LLMs running on the edge and facilitating primary model prompt caching by structuring the prompts to put static content, such as instructions and examples, upfront and variable content, such as user-specific information at the end. These techniques also reduce primary model usage costs considerably.<\/p>\n
You can see the typical latency performance below.<\/p>\n
\n- HA intents (e.g., turn on a light) | < 1\u00a0second<\/li>\n
- Analyze camera image (initial request) | < 3\u00a0seconds<\/li>\n
- Add automation | < 1\u00a0second<\/li>\n
- Memory operations | < 1\u00a0second<\/li>\n<\/ul>\n
Tools<\/h4>\n
The agent can use HA tools as specified in the LLM API<\/a> and other tools built in the LangChain framework as defined in tools.py<\/em>. Additionally, you can extend the LLM API with tools of your own as well. The code gives the primary LLM the list of tools it can call, along with instructions on using them in its system message and in the docstring of the tool\u2019s Python function definition. You can see an example of docstring instructions in the code snippet below for the get_and_analyze_camera_image<\/em> tool.<\/p>\n@tool(parse_docstring=False)
async def get_and_analyze_camera_image( # noqa: D417
camera_name: str,
detection_keywords: list[str] | None = None,
*,
# Hide these arguments from the model.
config: Annotated[RunnableConfig, InjectedToolArg()],
) -> str:
\"\"\"
Get a camera image and perform scene analysis on it.
Args:
camera_name: Name of the camera for scene analysis.
detection_keywords: Specific objects to look for in image, if any.
For example, If user says \"check the front porch camera for
boxes and dogs\", detection_keywords would be [\"boxes\", \"dogs\"].
\"\"\"
hass = config[\"configurable\"][\"hass\"]
vlm_model = config[\"configurable\"][\"vlm_model\"]
options = config[\"configurable\"][\"options\"]
image = await _get_camera_image(hass, camera_name)
return await _analyze_image(vlm_model, options, image, detection_keywords)<\/pre>\nIf the agent decides to use a tool, the LangGraph node action<\/em> is entered, and the node\u2019s code runs the tool. The node uses a simple error recovery mechanism that will ask the agent to try calling the tool again with corrected parameters in the event of making a mistake. The code snippet below shows the Python code associated with the action<\/em>\u00a0node.<\/p>\nasync def _call_tools(
state: State, config: RunnableConfig, *, store: BaseStore
) -> dict[str, list[ToolMessage]]:
\"\"\"Coroutine to call Home Assistant or langchain LLM tools.\"\"\"
# Tool calls will be the last message in state.
tool_calls = state[\"messages\"][-1].tool_calls
langchain_tools = config[\"configurable\"][\"langchain_tools\"]
ha_llm_api = config[\"configurable\"][\"ha_llm_api\"]
tool_responses: list[ToolMessage] = []
for tool_call in tool_calls:
tool_name = tool_call[\"name\"]
tool_args = tool_call[\"args\"]
LOGGER.debug(
\"Tool call: %s(%s)\", tool_name, tool_args
)
def _handle_tool_error(err:str, name:str, tid:str) -> ToolMessage:
return ToolMessage(
content=TOOL_CALL_ERROR_TEMPLATE.format(error=err),
name=name,
tool_call_id=tid,
status=\"error\",
)
# A langchain tool was called.
if tool_name in langchain_tools:
lc_tool = langchain_tools[tool_name.lower()]
# Provide hidden args to tool at runtime.
tool_call_copy = copy.deepcopy(tool_call)
tool_call_copy[\"args\"].update(
{
\"store\": store,
\"config\": config,
}
)
try:
tool_response = await lc_tool.ainvoke(tool_call_copy)
except (HomeAssistantError, ValidationError) as e:
tool_response = _handle_tool_error(repr(e), tool_name, tool_call[\"id\"])
# A Home Assistant tool was called.
else:
tool_input = llm.ToolInput(
tool_name=tool_name,
tool_args=tool_args,
)
try:
response = await ha_llm_api.async_call_tool(tool_input)
tool_response = ToolMessage(
content=json.dumps(response),
tool_call_id=tool_call[\"id\"],
name=tool_name,
)
except (HomeAssistantError, vol.Invalid) as e:
tool_response = _handle_tool_error(repr(e), tool_name, tool_call[\"id\"])
LOGGER.debug(\"Tool response: %s\", tool_response)
tool_responses.append(tool_response)
return {\"messages\": tool_responses}<\/pre>\nThe LLM API instructs the agent always to call tools using HA built-in intents<\/a> when controlling Home Assistant and to use the intents `HassTurnOn` to lock and `HassTurnOff` to unlock a lock. An intent describes a user\u2019s intention generated by user\u00a0actions.<\/p>\nYou can see the list of LangChain tools that the agent can use\u00a0below.<\/p>\n
\n- \nget_and_analyze_camera_image<\/em> | run scene analysis on the image from a\u00a0camera<\/li>\n
- \nupsert_memory<\/em> | add or update a\u00a0memory<\/li>\n
- \nadd_automation<\/em> | create and register a HA automation<\/li>\n
- \nget_entity_history<\/em> | query HA database for entity\u00a0history<\/li>\n<\/ul>\n
Hardware<\/h4>\n
I built the HA installation on a Raspberry Pi 5 with SSD storage, Zigbee, and LAN connectivity. I deployed the edge models under Ollama on an Ubuntu-based server with an AMD 64-bit 3.4 GHz CPU, Nvidia 3090 GPU, and 64 GB system RAM. The server is on the same LAN as the Raspberry Pi.<\/p>\n
Results<\/h3>\n
I\u2019ve been using this project at home for a few weeks and have found it useful but frustrating in a few areas that I will be working on to address. Below is a list of pros and cons of my experience with the\u00a0agent.<\/p>\n
Pros<\/h4>\n\n- The camera image scene analysis is very useful and flexible since you can query for almost anything and not have to worry having the right classifier as you would for a traditional ML approach.<\/li>\n
- Automations are very easy to setup and can be quite complex. Its mind blowing how good the primary LLM is at generating HA-compliant YAML.<\/li>\n
- Latency in most cases is quite acceptable.<\/li>\n
- Its very easy to add additional LLM tools and graph states with LangChain and LangGraph.<\/li>\n<\/ul>\n
Cons<\/h4>\n\n- The camera image analysis seems less accurate than traditional ML approaches. For example, detecting packages that are partially obscured is very difficult for the model to\u00a0handle.<\/li>\n
- The primary model clould costs are high. Running a single package detector once every 30 mins costs about $2.50 per\u00a0day.<\/li>\n
- Using structured model outputs for the helper LLMs, which would make downstream LLM processing easier, considerably reduces accuracy.<\/li>\n
- The agent needs to be more proactive. Adding a planning step to the agent graph will hopefully address\u00a0this.<\/li>\n<\/ul>\n
Example Use\u00a0Cases<\/h3>\n
Here are a few examples of what you can do with the home-generative-agent (HGA) integration as illustrated by screenshots of the Assist dialog taken by me during interactions with my HA installation.<\/p>\n
\n- Create an Automation.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/400\/1%2AqP8fNmqKJq4wABMCCAyOhg.png?ssl=1\")
Image by Lindo St.\u00a0Angel<\/figcaption><\/figure>\n\n- Create an automation that runs periodically.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/400\/1%2A7mxA0LVZJvRhlmsz3HvQBQ.png?ssl=1\")
Image by Lindo St.\u00a0Angel<\/figcaption><\/figure>\nThe snippet below shows that the agent is fluent in YAML based on what it generated and registered as an HA automation.<\/p>\n
alias: Check Litter Box Waste Drawer
triggers:
- minutes: \/30
trigger: time_pattern
conditions:
- condition: numeric_state
entity_id: sensor.litter_robot_4_waste_drawer
above: 90
actions:
- data:
message: The Litter Box waste drawer is more than 90% full!
action: notify.notify<\/pre>\n
\n- Check a\u00a0camera.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/400\/1%2AZxu-tsT9QkIwy6zGzF7_AA.png?ssl=1\")
Image by Lindo St.\u00a0Angel<\/figcaption><\/figure>\n\n- Check multiple cameras (video by the\u00a0author).<\/li>\n<\/ul>\n
https:\/\/github.com\/user-attachments\/assets\/230baae5-8702-4375-a3f0-ffa981ee66a3<\/a><\/p>\n\n- Summarize the home state (video by the\u00a0author).<\/li>\n<\/ul>\n
https:\/\/github.com\/user-attachments\/assets\/96f834a8-58cc-4bd9-a899-4604c1103a98<\/a><\/p>\n\n- Long-term memory with semantic\u00a0search.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/402\/1%2AAtCYRf5MUp_9_oEU5kYy0Q.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/400\/1%2A88vfzObxYGatSDxfvAW4Zw.png?ssl=1\")
<\/figure>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/403\/1%2Ahn3Yk6jkxq9IPr3DoYLrEw.png?ssl=1\")
Images by Lindo St.\u00a0Angel<\/figcaption><\/figure>\nYou can see that the agent correctly generates the automation below.<\/p>\n
alias: Prepare Home for Arrival
description: Turn on front porch light and unlock garage door lock at 7:30 PM
mode: single
triggers:
- at: \"19:30:00\"
trigger: time
actions:
- target:
entity_id: light.front_porch_light
action: light.turn_on
data: {}
- target:
entity_id: lock.garage_door_lock
action: lock.unlock
data: {}<\/pre>\n
\n- Check a camera for packages.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/402\/1%2Arg3NMW1PTrRGRRf1lvcmqA.png?ssl=1\")
Image by Lindo St.\u00a0Angel<\/figcaption><\/figure>\nBelow is the camera image the agent analyzed, you can see that two packages are\u00a0visible.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/486\/1%2Ab6Hk2T0TiM32qpWcYHM-NQ.png?ssl=1\")
Photo by Lindo St.\u00a0Angel<\/figcaption><\/figure>\n\n- Proactive notification of package delivery.<\/li>\n<\/ul>\n
![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/403\/1%2Albq7HT0FTHjF_8J3IrilVA.png?ssl=1\")
Image by Lindo St.\u00a0Angel<\/figcaption><\/figure>\nBelow is an example notification from this automation if any boxes or packages are\u00a0visible.<\/p>\n![\"\"](\"https:\/\/i0.wp.com\/cdn-images-1.medium.com\/max\/457\/1%2AiHF_bKOlkuBACO_ohuLpFg.png?ssl=1\")
Image by Lindo St.\u00a0Angel<\/figcaption><\/figure>\nConclusions<\/h3>\n
The Home Generative Agent offers an intriguing way to make your Home Assistant setup more user-friendly and intuitive. By enabling natural language interactions and simplifying automation, it provides a practical and useful tool for everyday smart home\u00a0use.<\/p>\n
Using a home generative agent carries security, privacy and cost risks that need further mitigation and innovation before it can be truly useful for the mass\u00a0market.<\/p>\n
Whether you\u2019re new to Home Assistant or a seasoned user, this integration is a great way to enhance your system\u2019s capabilities and get familiar with using generative AI and agents at home. If you\u2019re interested in exploring its potential, visit the Home Generative Agent GitHub page<\/a> and get started\u00a0today.<\/p>\nAppendix<\/h3>\nInstallation<\/h4>\n
1. Using the tool of choice, open your HA configuration\u2019s directory (folder) (where you find configuration.yaml<\/em>).<\/p>\n2. If you do not have a `custom_components` directory (folder), you must create\u00a0it.<\/p>\n
3. In the custom_components<\/em> directory (folder), create a new folder called home_generative_agent<\/em>.<\/p>\n4. Download _all_<\/em> the files from the custom_components\/home_generative_agent\/<\/em> directory (folder) in this repository.<\/p>\n4. Place the files you downloaded in the new directory (folder) you\u00a0created.<\/p>\n
6. Restart Home Assistant<\/p>\n
7. In the HA UI, go to \u201cConfiguration\u201d -> \u201cIntegrations\u201d click \u201c+,\u201d and search for \u201cHome Generative Agent\u201d<\/p>\n
8. Install all the Blueprints in the blueprints<\/em> directory (folder).<\/p>\nConfiguration<\/h4>\n
Configuration is done in the HA UI and via the parameters in const.py<\/em>.<\/p>\n![\"\"](\"https:\/\/medium.com\/_\/stat?event=post.clientViewed&referrerSource=full_rss&postId=f318732d9d84\")
<\/p>\n
\nLangChain Meets Home Assistant: Unlock the Power of Generative AI in Your Smart Home<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n
Introduction<\/h3>\n
This article describes the architecture and design of a Home Assistant<\/a> (HA) integration called home-generative-agent. This project uses LangChain<\/a> and LangGraph<\/a> to create a generative AI agent<\/a> that interacts with and automates tasks within a HA smart home environment. The agent understands your home\u2019s context, learns your preferences, and interacts with you and your home to accomplish activities you find valuable. Key features include creating automations, analyzing images, and managing home states using various LLMs (Large Language Models). The architecture involves both cloud-based and edge-based models for optimal performance and cost-effectiveness. Installation instructions, configuration details, and information on the project\u2019s architecture and the different models used are included and can be found on the home-generative-agent<\/a> GitHub. The project is open-source and welcomes contributions.<\/p>\n These are some of the features currently supported:<\/p>\n This is my personal project and an example of what I call learning-directed hacking<\/em><\/a>. The project is not affiliated with my work at Amazon nor am I associated with the organizations responsible for Home Assistant or LangChain\/LangGraph in any\u00a0way.<\/p>\n Creating an agent to monitor and control your home can lead to unexpected actions and potentially put your home and yourself at risk due to LLM hallucinations and privacy concerns, especially when exposing home states and user information to cloud-based LLMs. I have made reasonable architectural and design choices to mitigate these risks, but they cannot be completely eliminated.<\/p>\n One key early decision was to rely on a hybrid cloud-edge approach. This enables the use of the most sophisticated reasoning and planning models available, which should help reduce hallucinations. Simpler, more task-focused edge models are employed to further minimize LLM\u00a0errors.<\/p>\n Another critical decision was to leverage LangChain\u2019s capabilities, which allow sensitive information to be hidden from LLM tools and provided only at runtime. For instance, tool logic may require using the ID of the user who made a request. However, such values should generally not be controlled by the LLM. Allowing the LLM to manipulate the user ID could pose security and privacy risks. To mitigate this, I utilized the InjectedToolArg<\/em><\/a> annotation.<\/p>\n Additionally, using large cloud-based LLMs incurs significant cloud costs, and the edge hardware required to run LLM edge models can be expensive. The combined operational and installation costs are likely prohibitive for the average user at this time. An industry-wide effort to \u201cmake LLMs as cheap as CNNs\u201d is needed to bring home agents to the mass\u00a0market.<\/p>\n It is important to be aware of these risks and understand that, despite these mitigations, we are still in the early stages of this project and home agents in general. Significant work remains to make these agents truly useful and trustworthy assistants.<\/em><\/strong><\/p>\n Below is a high-level view of the home-generative-agent architecture.<\/p>\n The general integration architecture follows the best practices as described in Home Assistant Core<\/a> and is compliant with Home Assistant Community Store<\/a> (HACS) publishing requirements.<\/p>\n The agent is built using LangGraph and uses the HA conversation<\/em> component to interact with the user. The agent uses the Home Assistant LLM API to fetch the state of the home and understand the HA native tools it has at its disposal. I implemented all other tools available to the agent using LangChain. The agent employs several LLMs, a large and very accurate primary model for high-level reasoning, smaller specialized helper models for camera image analysis, primary model context summarization, and embedding generation for long-term semantic search. The primary model is cloud-based, and the helper models are edge-based and run under the Ollama<\/a> framework on a computer located in the\u00a0home.<\/p>\n The models currently being used are summarized below.<\/p>\n LangGraph powers the conversation agent, enabling you to create stateful, multi-actor applications utilizing LLMs as quickly as possible. It extends LangChain\u2019s capabilities, introducing the ability to create and manage cyclical graphs essential for developing complex agent runtimes. A graph models the agent workflow, as seen in the image\u00a0below.<\/p>\n The agent workflow has five nodes, each Python module modifying the agent\u2019s state, a shared data structure. The edges between the nodes represent the allowed transitions between them, with solid lines unconditional and dashed lines conditional. Nodes do the work, and edges tell what to do\u00a0next.<\/p>\n The __start__<\/em> and __end__<\/em> nodes inform the graph where to start and stop. The agent<\/em> node runs the primary LLM, and if it decides to use a tool, the action<\/em> node runs the tool and then returns control to the agent<\/em>. The summarize_and_trim<\/em> node processes the LLM\u2019s context to manage growth while maintaining accuracy if agent<\/em> has no tool to call and the number of messages meets the below-mentioned conditions.<\/p>\n You need to carefully manage the context length of LLMs to balance cost, accuracy, and latency and avoid triggering rate limits such as OpenAI\u2019s Tokens per Minute restriction. The system controls the context length of the primary model in two ways: it trims the messages in the context if they exceed a max parameter, and the context is summarized once the number of messages exceeds another parameter. These parameters are configurable in const.py<\/em>; their description is\u00a0below.<\/p>\n The summarize_and_trim<\/em> node in the graph will trim the messages only after content summarization. You can see the Python code associated with this node in the snippet\u00a0below.<\/p>\n The latency between user requests or the agent taking timely action on the user\u2019s behalf is critical for you to consider in the design. I used several techniques to reduce latency, including using specialized, smaller helper LLMs running on the edge and facilitating primary model prompt caching by structuring the prompts to put static content, such as instructions and examples, upfront and variable content, such as user-specific information at the end. These techniques also reduce primary model usage costs considerably.<\/p>\n You can see the typical latency performance below.<\/p>\n The agent can use HA tools as specified in the LLM API<\/a> and other tools built in the LangChain framework as defined in tools.py<\/em>. Additionally, you can extend the LLM API with tools of your own as well. The code gives the primary LLM the list of tools it can call, along with instructions on using them in its system message and in the docstring of the tool\u2019s Python function definition. You can see an example of docstring instructions in the code snippet below for the get_and_analyze_camera_image<\/em> tool.<\/p>\n If the agent decides to use a tool, the LangGraph node action<\/em> is entered, and the node\u2019s code runs the tool. The node uses a simple error recovery mechanism that will ask the agent to try calling the tool again with corrected parameters in the event of making a mistake. The code snippet below shows the Python code associated with the action<\/em>\u00a0node.<\/p>\n The LLM API instructs the agent always to call tools using HA built-in intents<\/a> when controlling Home Assistant and to use the intents `HassTurnOn` to lock and `HassTurnOff` to unlock a lock. An intent describes a user\u2019s intention generated by user\u00a0actions.<\/p>\n You can see the list of LangChain tools that the agent can use\u00a0below.<\/p>\n I built the HA installation on a Raspberry Pi 5 with SSD storage, Zigbee, and LAN connectivity. I deployed the edge models under Ollama on an Ubuntu-based server with an AMD 64-bit 3.4 GHz CPU, Nvidia 3090 GPU, and 64 GB system RAM. The server is on the same LAN as the Raspberry Pi.<\/p>\n I\u2019ve been using this project at home for a few weeks and have found it useful but frustrating in a few areas that I will be working on to address. Below is a list of pros and cons of my experience with the\u00a0agent.<\/p>\n Here are a few examples of what you can do with the home-generative-agent (HGA) integration as illustrated by screenshots of the Assist dialog taken by me during interactions with my HA installation.<\/p>\n The snippet below shows that the agent is fluent in YAML based on what it generated and registered as an HA automation.<\/p>\n https:\/\/github.com\/user-attachments\/assets\/230baae5-8702-4375-a3f0-ffa981ee66a3<\/a><\/p>\n https:\/\/github.com\/user-attachments\/assets\/96f834a8-58cc-4bd9-a899-4604c1103a98<\/a><\/p>\n You can see that the agent correctly generates the automation below.<\/p>\n Below is the camera image the agent analyzed, you can see that two packages are\u00a0visible.<\/p>\n Below is an example notification from this automation if any boxes or packages are\u00a0visible.<\/p>\n The Home Generative Agent offers an intriguing way to make your Home Assistant setup more user-friendly and intuitive. By enabling natural language interactions and simplifying automation, it provides a practical and useful tool for everyday smart home\u00a0use.<\/p>\n Using a home generative agent carries security, privacy and cost risks that need further mitigation and innovation before it can be truly useful for the mass\u00a0market.<\/p>\n Whether you\u2019re new to Home Assistant or a seasoned user, this integration is a great way to enhance your system\u2019s capabilities and get familiar with using generative AI and agents at home. If you\u2019re interested in exploring its potential, visit the Home Generative Agent GitHub page<\/a> and get started\u00a0today.<\/p>\n 1. Using the tool of choice, open your HA configuration\u2019s directory (folder) (where you find configuration.yaml<\/em>).<\/p>\n 2. If you do not have a `custom_components` directory (folder), you must create\u00a0it.<\/p>\n 3. In the custom_components<\/em> directory (folder), create a new folder called home_generative_agent<\/em>.<\/p>\n 4. Download _all_<\/em> the files from the custom_components\/home_generative_agent\/<\/em> directory (folder) in this repository.<\/p>\n 4. Place the files you downloaded in the new directory (folder) you\u00a0created.<\/p>\n 6. Restart Home Assistant<\/p>\n 7. In the HA UI, go to \u201cConfiguration\u201d -> \u201cIntegrations\u201d click \u201c+,\u201d and search for \u201cHome Generative Agent\u201d<\/p>\n 8. Install all the Blueprints in the blueprints<\/em> directory (folder).<\/p>\n Configuration is done in the HA UI and via the parameters in const.py<\/em>.<\/p>\n LangChain Meets Home Assistant: Unlock the Power of Generative AI in Your Smart Home<\/a> was originally published in Towards Data Science<\/a> on Medium, where people are continuing the conversation by highlighting and responding to this story.<\/p>\n<\/div>\n\n
Important Considerations<\/h3>\n
Architecture and\u00a0Design<\/h3>\n
\n
LangGraph-based Agent<\/h4>\n
LLM Context Management<\/h4>\n
\n
async def _summarize_and_trim(
state: State, config: RunnableConfig, *, store: BaseStore
) -> dict[str, list[AnyMessage]]:
\"\"\"Coroutine to summarize and trim message history.\"\"\"
summary = state.get(\"summary\", \"\")
if summary:
summary_message = SUMMARY_PROMPT_TEMPLATE.format(summary=summary)
else:
summary_message = SUMMARY_INITIAL_PROMPT
messages = (
[SystemMessage(content=SUMMARY_SYSTEM_PROMPT)] +
state[\"messages\"] +
[HumanMessage(content=summary_message)]
)
model = config[\"configurable\"][\"vlm_model\"]
options = config[\"configurable\"][\"options\"]
model_with_config = model.with_config(
config={
\"model\": options.get(
CONF_VLM,
RECOMMENDED_VLM,
),
\"temperature\": options.get(
CONF_SUMMARIZATION_MODEL_TEMPERATURE,
RECOMMENDED_SUMMARIZATION_MODEL_TEMPERATURE,
),
\"top_p\": options.get(
CONF_SUMMARIZATION_MODEL_TOP_P,
RECOMMENDED_SUMMARIZATION_MODEL_TOP_P,
),
\"num_predict\": VLM_NUM_PREDICT,
}
)
LOGGER.debug(\"Summary messages: %s\", messages)
response = await model_with_config.ainvoke(messages)
# Trim message history to manage context window length.
trimmed_messages = trim_messages(
messages=state[\"messages\"],
token_counter=len,
max_tokens=CONTEXT_MAX_MESSAGES,
strategy=\"last\",
start_on=\"human\",
include_system=True,
)
messages_to_remove = [m for m in state[\"messages\"] if m not in trimmed_messages]
LOGGER.debug(\"Messages to remove: %s\", messages_to_remove)
remove_messages = [RemoveMessage(id=m.id) for m in messages_to_remove]
return {\"summary\": response.content, \"messages\": remove_messages}<\/pre>\nLatency<\/h4>\n
\n
Tools<\/h4>\n
@tool(parse_docstring=False)
async def get_and_analyze_camera_image( # noqa: D417
camera_name: str,
detection_keywords: list[str] | None = None,
*,
# Hide these arguments from the model.
config: Annotated[RunnableConfig, InjectedToolArg()],
) -> str:
\"\"\"
Get a camera image and perform scene analysis on it.
Args:
camera_name: Name of the camera for scene analysis.
detection_keywords: Specific objects to look for in image, if any.
For example, If user says \"check the front porch camera for
boxes and dogs\", detection_keywords would be [\"boxes\", \"dogs\"].
\"\"\"
hass = config[\"configurable\"][\"hass\"]
vlm_model = config[\"configurable\"][\"vlm_model\"]
options = config[\"configurable\"][\"options\"]
image = await _get_camera_image(hass, camera_name)
return await _analyze_image(vlm_model, options, image, detection_keywords)<\/pre>\nasync def _call_tools(
state: State, config: RunnableConfig, *, store: BaseStore
) -> dict[str, list[ToolMessage]]:
\"\"\"Coroutine to call Home Assistant or langchain LLM tools.\"\"\"
# Tool calls will be the last message in state.
tool_calls = state[\"messages\"][-1].tool_calls
langchain_tools = config[\"configurable\"][\"langchain_tools\"]
ha_llm_api = config[\"configurable\"][\"ha_llm_api\"]
tool_responses: list[ToolMessage] = []
for tool_call in tool_calls:
tool_name = tool_call[\"name\"]
tool_args = tool_call[\"args\"]
LOGGER.debug(
\"Tool call: %s(%s)\", tool_name, tool_args
)
def _handle_tool_error(err:str, name:str, tid:str) -> ToolMessage:
return ToolMessage(
content=TOOL_CALL_ERROR_TEMPLATE.format(error=err),
name=name,
tool_call_id=tid,
status=\"error\",
)
# A langchain tool was called.
if tool_name in langchain_tools:
lc_tool = langchain_tools[tool_name.lower()]
# Provide hidden args to tool at runtime.
tool_call_copy = copy.deepcopy(tool_call)
tool_call_copy[\"args\"].update(
{
\"store\": store,
\"config\": config,
}
)
try:
tool_response = await lc_tool.ainvoke(tool_call_copy)
except (HomeAssistantError, ValidationError) as e:
tool_response = _handle_tool_error(repr(e), tool_name, tool_call[\"id\"])
# A Home Assistant tool was called.
else:
tool_input = llm.ToolInput(
tool_name=tool_name,
tool_args=tool_args,
)
try:
response = await ha_llm_api.async_call_tool(tool_input)
tool_response = ToolMessage(
content=json.dumps(response),
tool_call_id=tool_call[\"id\"],
name=tool_name,
)
except (HomeAssistantError, vol.Invalid) as e:
tool_response = _handle_tool_error(repr(e), tool_name, tool_call[\"id\"])
LOGGER.debug(\"Tool response: %s\", tool_response)
tool_responses.append(tool_response)
return {\"messages\": tool_responses}<\/pre>\n\n
Hardware<\/h4>\n
Results<\/h3>\n
Pros<\/h4>\n
\n
Cons<\/h4>\n
\n
Example Use\u00a0Cases<\/h3>\n
\n
\n
alias: Check Litter Box Waste Drawer
triggers:
- minutes: \/30
trigger: time_pattern
conditions:
- condition: numeric_state
entity_id: sensor.litter_robot_4_waste_drawer
above: 90
actions:
- data:
message: The Litter Box waste drawer is more than 90% full!
action: notify.notify<\/pre>\n\n
\n
\n
\n
<\/figure>\n<\/figure>\nalias: Prepare Home for Arrival
description: Turn on front porch light and unlock garage door lock at 7:30 PM
mode: single
triggers:
- at: \"19:30:00\"
trigger: time
actions:
- target:
entity_id: light.front_porch_light
action: light.turn_on
data: {}
- target:
entity_id: lock.garage_door_lock
action: lock.unlock
data: {}<\/pre>\n\n
\n
Conclusions<\/h3>\n
Appendix<\/h3>\n
Installation<\/h4>\n
Configuration<\/h4>\n
<\/p>\n
\n