The Apology

I’ve been thinking about how to write this for a solid half a year. My technical writing skills are lackluster, so a lot of it will read like drivel, and I’m afraid you’re going to have to bear with me on this. I apologize in advance.

Here’s basically what I want:

tldr #1: Different LLM chains that get data from sensors every x minutes/hours/days to perform immediate/pattern inspection, and then proactively make changes in house settings + reminders/todos etc. They do this by arguing among themselves, coming to an agreement(or have a fundamental disagreement), which then gets relayed to a more powerful House entity which makes the decisions.

tldr #2: An “Always listening” house entity, that processes data on a low(relatively) powered system, and when user pauses, it decides if whatever was said was directed at it or if an ‘intervention is needed’ eg two people talking wondered something aloud. If so, send snippet to larger LLM, speak out teh intervention/answer.

tldr #2 is something I will save for later.

The Longer Writeup

Still with me? Great. I have no special skills. I am a software engineer working as an infrastructure engineer for last year or so. I dabble in homelabs and self-hosting at home. This entire idea was inspired by Iain M. Banks’ Culture series, which I heavily recommend. The entire idea grew from “Wow a suit can have rudimentary systems and opinions and then when it needs more data processing prowess, it talks to the drone/big ship” in Player of Games(which I recommend you start with, rather than Consider Phlebas, thh first book in the series, which you should read LAST).

Anyhow, back to reality - the idea is this: When the idea of smart homes was touted about a decade ago, they were simple if-else machines, just expanded throughout the house: If motion is detected, turn on bathroom lights; if temperature reaches 30C, turn on the AC; when phone connects to car and it’s a workday, open maps and set directions to work, etc etc etc.

None of this takes into account the actual context of you, your day, your todos for the day, your house, or what’s happening inside your house. There’s a distinct lack of pattern recognition and true environmental awareness. Is your kitchen warm because of an anomaly or is it because during the summer the sun shines through that gap in the backdoor blinds? does your cat move to the living room 10 minutes before your alarm every day?

Why is there not a system that is actually ‘intelligent’(Yes, I know people are going to yell at me about this, more on this later)? Not Google’s overt promises of “It learns your habits!” (which has never really held true, by the way, as a user of Android and early adopter of many, many Google products - I miss Google Now), but truly learns - stores sensor data across your house across different time scales, stores discoveries, experiments, tidbits it observes from its many sensors spread around your house?

That’s what my end goal is. This is what I’ve been trying to build for the last month(well, more like trying to organize my thoughts, order the sensors I might need, and so on and so forth). Think of it as having multiple observers, each watch your and your home’s patterns at different intervals.

What you may (or rather, I may, since I doubt anyone else might want to until I figure this out) need:

• A local semi-decent GPU or an API subscription (warning: not private, and costs may vary)
• All of the sensors below (or pick and choose)
• Semi-rudimentary programming skills
• Semi-rudimentary soldering/electrical skills
• A 3D printer+CAD skills are nice to have, otherwise you can just buy project boxes and hole them out.

The Devices

The sensors I have received so far measure:

BME280: Temp, Humidity, Pressure (Lower class - higher range of error: )

BME680: Temp, Humidity, Pressure, eVOC

SHT41 : High Precision Temp, Humidity

Pre-made Zigbee Temp/Humidity sensors(Redundancy+Commercial backup spread around the house separate from the ‘sensor array box’)

MHZ19B : CO2 levels

TSL2591 : Light intensity, Infrared, Full spectrum, Lux (high dynamic range)

BH1750 : Light intensity/illumination

TCS34725 : Color recognition (RGB + Clear)

LTR390 : UV light detection

MPU6500 : Acceleration, Gyroscope, Vibration (6-axis motion)

VL53L0X : Time of Flight (precise distance measurement)

AM312 IR sensors : Motion detection/presence

RCWL-0516 : Microwave radar motion detection

GPS Neo 6M : Location/positioning

PMS7003/PMSD7 : Laser particle sensor (PM2.5, dust)

Analog mics (INMP551) : Sound detection

MAX9814 : Microphone with automatic gain control

PAM8302A : Audio amplification

DF9-40 : Bed Pressure Resistor

Most of these will be in every room, doorways where it makes sense (TOF for example) etc. There will be more in the future and I will add that list to an addendum to this list.

The Plan

The basic conceptual idea is this:

There are at least 3 LLM chains:

1. a five minute chain that studies immediate anomalies, mishaps and so forth, and it also has the data for last hour.
2. an hourly chain with average sensor data for each hour with max/mins, that also has the data for the last 24h and a summary for last 7 days.
3. A daily chain that has average sensor data for the duration of 24h, with min/max, any anomalous events and such, and has recent data of 7 days/30 days.

Effectively, Every 5 minutes, the system looks at immediate sensor data: “Hey, something interesting is happening right now with the temperature and humidity or x or y sensor.”

Every hour, it steps back for a broader view: “The pattern of CO2 levels this afternoon is different from usual.”

Daily, it considers lifestyle patterns: “On workdays, the study gets too warm around 2PM when the sun hits directly.”

Weekly and monthly, it spots longer trends: “You know, the air quality in the kitchen has been improving since you started using the hood fan more regularly.” (I know it’s a lame example but hey)

Then there are two other chains:

1. this chain would gather insights, tidbits etc that are generated by “lifestyle patterns or broader views,” or a summary of those. This isn’t actionable for the most part.
2. this chain would offer ’experiments’ (and keep track of running experiments) based on what the first three chains. x levels rise at y time when z person is present, how about we move the fan from p position to q for 3 days and measure x?

Then there are ‘controls" - the bread and butter of this entire rant; The “intelligent” part; The “Proactive” part of the house. After thorough testing, I aim to either have it automagically control the non-critical parts of the house(with critical systems having limits placed on them OUTSIDE the control of LLMs (eg, Ecobee Thermostat will have max/min temperature ranges set so it won’t magically set the Cool mode to 65)), or give user a prompt before doing so (which I dislike, since Proactive, but it’s a choice).

The “Entities”

There are two ways to go on about this:

1. One way is that you have one central entity, that performs all the tasks. Let’s call it a House Entity. That entity does everything. Problem with this is that there’s no consensus. You don’t know if that entity is wrong, or right. I am cautious about this approach. It CAN work, given proper safeguards, but there is wisdom in the alternative approach.
2. Another way is that each of the ‘classes’ of your sensors are their own entities. Security System would have a name and a separate class, so would sensors that have to do with health, or HVAC/blinds/fan systems. When a serious anomaly is detected and/or there is a need for a proactive change or user needs alerting, they present their own facts, disagree/agree with each other based on their sensor data + sensor data presented by their respective entities in a conversation format, and the ‘house’ entity which has been neutral so far, makes a recommendation and decides, overriding any function call requests and so forth. You can specify your own same LLMs (but different thread/recording chains) for each of the entities, or separate LLMs (weaker for each entity, and a more powerful House LLM).

What I have so far

I have multiple rudimentary programs written in python running in a VM taking dummy data that I generated with help of an LLM in a timeseries database, and running it through chains.

The technical aspects I’ll cover in another “rant,” maybe part 2/3 of this series, but the basic gist is effectively this:

The specialzied ’entities’ (as I mentioned above, effectively a specialized cluster of sensors) have access to function calling, a program or two monitor which functions are being called, and who is calling them. The ‘functions’ defined so far are generic, and not tied to anything specific, but they can be tied to HomeAssistant running currently in my house, and I plan to do so soon enough.

The clincher is that the entities never call a function. They request the function call, alongside their explanation, observation, and sensor data they deem to share. Then other entities form a “conversation chain” for that specific event. Based on the conversation agreement/disagreement, other entities may request other function calls instead. If we have six entity groups in total, the conversation goes around in a loop until either the entities come to an agreement, and a function call is requested to the house entity, or the house entity is asked to intervene, after evaluating all the data, it may approve a specific function requested by a specific entity. This isn’t perfect, and I am still tinkering with it, but you get the gist.

A template sorta kinda looks like this:

{
  "trigger_event": {
    "type": "temperature_anomaly",

    "timestamp": "2024-01-20T14:30:00",

    "data": {
      "temperature": 28,

      "rate_of_change": "+2.3C/30min",

      "location": "office"
    }
  },

  "conversation_sequence": [
    {
      "entity": "serious_tactical_disadvantage",

      "trigger_type": "primary",

      "data_used": ["temperature_sensors", "thermal_trends"],

      "response": "Your office is a furnace. Fix it.",

      "function_requests": [
        {
          "type": "system_action",

          "function": "hvac.fan",

          "params": {
            "state": "on",

            "speed": "high",

            "location": "office"
          },

          "priority": "high"
        }
      ]
    },

    {
      "entity": "anticipating_several_problems",

      "trigger_type": "secondary",

      "data_used": [
        "equipment_specs",

        "productivity_metrics",

        "environmental_history"
      ],

      "response": "Oh no, this could affect productivity, health, equipment lifespan, air quality...",

      "function_requests": null
    },

    {
      "entity": "suspicious_of_small_coincidences",

      "trigger_type": "secondary",

      "data_used": ["voc_sensors", "appliance_status", "temperature_sensors"],

      "response": "Temperature 28°C and the VOC sensor just detected ethanol at the exact same moment your dishwasher finished",

      "function_requests": null
    },

    {
      "entity": "documented_historical_precedent",

      "trigger_type": "secondary",

      "data_used": ["historical_events", "maintenance_logs"],

      "response": "Reminds me of October 14th, when we had that similar spike right before you reset your HVAC filter...",

      "function_requests": [
        {
          "type": "user_recommendation",

          "action": "check_hvac_filter",

          "priority": "medium",

          "reason": "historical_pattern_match"
        }
      ]
    },

    {
      "entity": "pragmatic_observer",

      "trigger_type": "secondary",

      "data_used": ["hvac_status", "temperature_sensors"],

      "response": "It's 28°C. The AC is off. That's literally all that's happening.",

      "function_requests": null
    },

    {
      "entity": "serious_tactical_disadvantage",

      "trigger_type": "secondary",

      "data_used": ["hvac_status", "previous_responses"],

      "response": "I'm turning on the fan. You're all exhausting.",

      "function_requests": [
        {
          "type": "system_action",

          "function": "hvac.fan",

          "params": {
            "state": "on",

            "speed": "high",

            "location": "office"
          },

          "priority": "critical",

          "override_conflicts": true
        }
      ]
    }
  ],

  "executed_functions": [
    {
      "timestamp": "2024-01-20T14:30:05",

      "function": "hvac.fan",

      "params": {
        "state": "on",

        "speed": "high",

        "location": "office"
      },

      "requested_by": "serious_tactical_disadvantage",

      "status": "success"
    }
  ]
}

Again, inspired by Culture. You don’t have to have all the naming/gibberish in there. Conversational “Reasoning” might be nice, alongside the data provided by each entity, eg "data_used": ["temperature_sensors", "thermal_trends", "maintennace logs"] etc. For the dummy data I’ve used so far, every entity has a specific personality and behaves in a specific way. An entity whose primary function is wellbeing/comfort of the occupants may try to object to changes made by other entities which may impede on the wellbeing/comfort of the occupants (like the HVAC system being requested to be at 80F because of a ‘community event’ by your utility company).