So, you bought a Google Nest thermostat - only to find that setting up it's API access is just too far out for you? This article comes to the rescue.
Learn how to get access to your Google Nest thermostat on any Internet connected Linux box.
Comments on hardware and wiring are specific for the European version of the Nest thermostat.
Google's Nest (3rd Generation) is an Internet connected self learning thermostat. It works with the Google Hub, the Google Home app, or the special Nest app. The Nest learns how to cool or heat your home in the most energy efficient way. Physically it looks a lot like a 1970's Honeywell thermostat on steroids. It's a big round knob that sits on your wall which you turn to set the desired room temperature. But that is where the similarity with the mechanical thermostat ends.
The Nest has a bright and very readable display. It uses its built-in motion sensors to turn the display off if you are at sufficient distance, and on again if you approach the Nest. The Nest uses these sensors too to check if anybody is at home. Also, it can check for the absence or presence of mobile phones. If no activity is seen in your house and all configured phones are absent, the Nest switches the heater to ECO mode to save energy.
The Nest can learn your heating patterns and create schedules from that. So, by simply using the thermostat for a week, the Nest will "know" when to turn the heater on. Assuming your weeks are fairly similar, of course. You can also set your schedule manually. The Nest is capable of learning how long it will take to heat your home to the desired temperature, figuring in the weather conditions and the specifics of your house or flat. Nice. And they are starting to fall in price too, I had mine for roughly 200 credits.
Setting up the Nest physically can be a bit daunting. The Nest can't just be hooked up to your system, it uses a special connection box (the "heatlink"). It can be a bit risky to install this yourself, as getting the wiring wrong may damage your thermostat, heatlink or even your heater. I took the risk and hooked up the Nest to my old ATAG A series heater (built in 2009) myself. Initially I simply used the on/off option, later I found out that the old Atag actually supports OpenTherm, which allows a finer grained set of controls to be used by the Nest. For example to dynamically set the temperature of the water that flows through your radiators. As the Nest also can use OpenTherm, I rewired it all and it seems to work fine.
The nest can work with a HVAC system and so has options to control fans and cooling systems. I haven't tested that, as I don't have such a system.
There is a specific setting in the Nest's menus that confused me. It is the "supply" setting. You can find it by opening up the gear wheel menu on the Nest, then select (Equipment) → (Continue) → (Pro Setup) → (Continue) → (select your hardware setup).
In the screen that opens next you will see four fields. The first field allows you to set the protocol the Nest should use to communicate with the heater. It can either be "on/off" or "OpenTherm¨. Next, there is a field that can be set to indicate the energy source for your heater (gas, electricity). The third field indicates how heat is dispersed in your rooms (e.g. radiators). All quite self-explaining. But the fourth field had me scratching my head. It is labeled "supply" and you can set a temperature there. But what does it do?
Well, it sets the desired flow temperature. That is: the temperature that your central heating water should have as it flows through your radiators (or floor or what have you). Why did I find it puzzling: because when the Nest is configured to work with "on/off", the Nest has no means to set the flow temperature, so this setting is superfluous. But if the Nest has been configured to work with OpenTherm - it is also superfluous, as the Nest should dynamically change the flow temperature itself, and not adhere to some fixed value. What then, pray tell, does this setting do?
I asked around and finally got an answer: the "supply" setting in the Pro-configuration menu is only used if you have set the first field to "OpenTherm". It serves to set the upper limit used by the Nest to set the CH flow temperature, In other words: say I set this to 70 degrees (C), it may well be that the Nest orders the heater to use a flow temperature of say 50 degrees, or 60, or 59.2.. whatever, but NEVER higher than 70 degrees. Note that setting this value too low in an older house may result in slow heating of your house and higher bills.
The Nest is an Internet connected thermostat. It uses weather data from other sources to check the outside temperature and weather for example, which it uses to determine when to start heating and for how long. You can connect your Google Nest to your Google account. I did, and my Google Hub that uses the same account immediately was able to tell me the room temperature. Also, I hooked up the Nest App on my smart phone to that account, and now I can control my Nest from the comfort of my own phone.
But I would like to do a bit more with my Nest. For example, I would like to register the behaviour of my Nest for a longer period of time. The Nest app has a builtin feature to do just that. It allows you to see when the Nest switched the heater on and for how long. It will also tell you about circumstances that reduced your energy usage. For example when nobody was at home, or the outside temperature was higher of lower than normal. But the catch here is that it has a very limited memory: just a week. If you want to store more than one week of data, you need to write your own software. Google provides API's to access the Nest. But to prevent anybody being able to heat the bejeezes out of you at will, they require you to authenticate properly. And though they provide instructions, it took me a while to get it to work. So, I will share what I did here, hoping you may find this useful.
Once you have access to the API's from your local or cloud based server, you can integrate the Nest with other stuff, for example you might create a project that uses the Google calendar to check if a special heating program should be started (think: holidays, absence due to travel etc.). I will do that later on, for now, let's just get things up and running. I will use a Linux box, the jq utility (available on most distro's) and create a POSS (Plain Old Shell Script).
This script is NOT fit for production use, but merely serves to demonstrate how the API's can be used and which data you need to get it up and running. A serious flaw of this script is that it stores the secrets inside the script. Anybody with access to the code can read them. Also it is not sufficiently robust, e.g. error handling is not good enough. Caveat!
#!/bin/bash
#
# (C) 2023 Heinrich W. Klöpping
# Licensed under GPL-3.0, see: https://opensource.org/licenses/GPL-3.0
#
# This script will use the Google SDM API's to extract data from your
# Google Nest thermostat.
#
# *** WARNING ***
#
# This is a demonstration script, not fit for production.
# Storing secrets inside source code is bad practice.
# Use at your own risk.
#
#
############################### configuration ##############################
#
# See https://developers.google.com/nest/device-access/ on how to obtain
# a ProjectId, ClientID/ClientSecret and a Refreshtoken. Replace the
# variables below with your data. Once you have a refreshtoken you can
# use that - TTBOMK - for an undetermined time to obtain access tokens.
# However, protect this data "with your life" as it provides access to your
# home heating system and may provide valuable data on your absences to
# burglars.
#
ProjectId=' fill in your own data here '
ClientID=' fill in your own data here '
ClientSecret=' fill in your own data here '
RefreshToken=' fill in your own data here '
# Credentials will be stored in a local credential file. It can be anywhere
# you want to. Note that these credentials have a limited lifetime
# (typically 1 hour) but should be protected well nevertheless. Check file
# permissions and pick a secure location to store these credentials.
#
CredFile=~/.credentials
#
# This variable extracts a number of seconds from the value specified by
# Google, to compensate for any slow responses which might lead to
# invalid results. Default of 60 is just fine.
#
MinValidity=60
########################## configuration ends here #########################
# Graceful exit on failure
#
fail() {
echo "Failed: $*"
exit 1
}
# Get new access token from refresh token and store it in the credentials file
#
GetNewBearer() {
[ -z "$1" ] && fail missing filename
CredJson=$(curl -s -d "" -L -X POST 'https://www.googleapis.com/oauth2/v4/token?client_id='$ClientID'&client_secret='$ClientSecret'&refresh_token='$RefreshToken'&grant_type=refresh_token')
[ -z "$CredJson" ] && fail can not get bearer token
echo "$CredJson" |jq -r '. | "Bearer=\(.access_token) Expiry=\(.expires_in+'"$(date +%s)"')"' >$1
}
CheckCredFileFormat() {
[ -z "$1" ] && fail missing filename
T=$(sed 's/Bearer=[^ ]\+\ Expiry=[0-9]\+$//g' $1)
[ ! -z "$T" ] && fail "invalid credentials file format"
}
TokenWorker() {
# if current token is unavailable or has expired, get a new one
#
[ ! -f $CredFile ] && GetNewBearer $CredFile
# we either have an old one or we have just gotten a new one. Check if it is in proper format:
#
CheckCredFileFormat $CredFile
# If we got here, we can safely dot in the file to obtain the variables
#
. $CredFile
Validity=$[ $Expiry - $(date +%s) ]
}
GetWorkableToken() {
TokenWorker
[ $Validity -lt $MinValidity ] && rm $CredFile && TokenWorker
}
GetWorkableToken
#echo "Bearer: $Bearer"
#echo "Expires: $Expiry"
echo "Token still valid for $Validity(-${MinValidity}) seconds "
JsonResults=$(curl -s -X GET 'https://smartdevicemanagement.googleapis.com/v1/enterprises/'$ProjectId'/devices' -H 'Content-Type: application/json' -H "Authorization: Bearer $Bearer")
Humidity=$(echo "${JsonResults}" | jq -r '.devices[].traits["sdm.devices.traits.Humidity"].ambientHumidityPercent')
Scale=$(echo "${JsonResults}" | jq -r '.devices[].traits["sdm.devices.traits.Settings"].temperatureScale')
TargetTemp=$(echo "${JsonResults}" | jq -r '.devices[].traits["sdm.devices.traits.ThermostatTemperatureSetpoint"].heatCelsius')
CurrentTemp=$(echo "${JsonResults}" | jq -r '.devices[].traits["sdm.devices.traits.Temperature"].ambientTemperatureCelsius')
CurrentMode=$(echo "${JsonResults}" | jq -r '.devices[].traits["sdm.devices.traits.ThermostatMode"].mode')
DisplayName=$(echo "${JsonResults}" | jq -r '.devices[].parentRelations[].displayName')
echo "$DisplayName humidity now $Humidity percent"
echo "Desired minimum temperature $TargetTemp degrees $Scale"
echo "Current room temperature $CurrentTemp degrees $Scale"
echo "Mode: $CurrentMode"
echo "Have a nice day"
# echo $JsonResults
The script needs access to TWO API's:
The first call is easy to understand: your Nest's data is available to Google because you gave permission to Google to store and use that data. And so, the data is available (for authenticated users) over Google's API's. Of course, proper authentication and authorisation is required: to prove that you are entitled to use these API's you will need to present a short lived bearer token in the header of the API call. And that bearer token can only be obtained if you are the rightful owner of the account. As you are not the only Nest user in the world, you need to identify your Nest. This is done by obtaining a unique project id, which is part of the API path. More about that later.
But how do you get that bearer token? Well, you'll need your client-id, a client-secret and a refresh token. If you have all that, you can call the second API, which will return the bearer token. Which immediately begs the question: how do you obtain a client-id, a client-secret and a refresh token?
Well, that is the gist of the matter: you'll need to jump through some hoops first.
I assume you have hooked up your Nest thermostat to your smart phone using an active, existing Google account. Verify this by opening up the Nest app. Tap on the gear wheel and select "Account" to check which account is hooked up to your Nest.
You need access to the Smart Device Management API (aka SDM API). Google requires you to register for their "Device Access program" first. You have to pay them five bucks for the honour. Take it, or leave it. It's beyond me how it is that you have to pay for access to your data, coming from your device, but that's how it is.
Ensure you are logged in using the Google account you used to connect to your Nest. Go to https://console.nest.google.com/device-access, accept the Terms of Service (for the sandbox and the API's) and pay the five credits. If all goes well, you are presented with a screen on which you can create a project. Do not do that yet, you will return there later.
This step requires you to interact with the Google Cloud Console. Google uses the concept of "projects" (i.e. GCP projects) to allow you to group your credentials etc. - it has nothing to do with the device-access project you will create later.
You can use any Google account you want, as long as you have enabled Device Access on it (see the Pay up step above). Normally that would be the account you used to connect your smart phone to your Nest. Go to https://console.cloud.google.com/apis/credentials. Either set up a new project (again: do not confuse this with the device access project you will create later) or select an existing one.
If you create a project ensure it is in "production" state. If you leave it in test, the credentials you obtain to make your script run will be invalidated every seven days. Ensure NOT to provide an app logo in the Oauth consent screen setup (in the app information section). Having a logo there ensures you have to go through Google's verification procedure to get your project to "production" state. Which is totally unnecessary for this type of application in which you are both the developer and the user and merely try to access your own Nest's data. Removing the logo aferwards is almost impossible to do, so once you have set it, you're stuck with it.
Check if you have access to the SDM API's by clicking on "Enabled APIs and Services" in the left hand column. You should see a list of APIs and "Smart Device Management APIs" should be in that list (probably on top if you just activated it).
Next, select "Credentials" in the left hand column, then hit "+ Create Credentials" (in the middle top hand of the screen) and choose to create an "OAuth client id".
If you have not yet configured OAuth Consent you will be prompted to do so now. Fill out the proper fields, most of them are self-explanatory. Do not configure an App logo. The application home page can be any page but it is nice to have it have a relation with your app. A privacy policy example (in Dutch) can be found here https://www.msmog.nl/downloads/Privacyverklaring-10-mei-2018.pdf>. In the fields "Authorised domains" you fill out the domains of the systems you will use to process data. Also, add "google.com" there.
The application type to set is "Web application". Also fill in an "Authorised redirect URI", and set it to https://www.google.com. Then hit the big blue "Create" button at the end of the page. Your client ID and Client Secret will be calculated and you will get an option to download the JSON. You can also simply return to the page to copy them over manually.
Next, add the Google account you are using for device access (in principle the one you used to hook up your Nest to your smart phone, remember) as a test user. Click "OAuth Consent Screen" in the left hand corner, and search for the header "Test Users". Hit "+ Add Users" and add your Google account mail details there.
If you are presented with a "Choose an account" screen, select the Google account you used to hook up your smart phone to the Nest. Next a screen will be presented where you should toggle on the permissions for your home. Ensure you select at least the Nest thermostat. Then click Next. On the "Choose an account to continue to Project Name" screen (where Project Name is the name of your GCP project) select the Google account you wish to authorize for the SDM API (the one used to hook up Nest to your smart phone).
After choosing an account, you may get a warning screen that states Google hasn't verified this app. If so, to continue, click the "Advanced" option and then click "Go to Project Name (unsafe)". On the "Grant Project Name permission" screen, click "Allow" to give the project permission to access your Google account. On the "Confirm your choices" screen, make sure the permissions you wish to grant are checked and click "Allow" to confirm.
You should be redirected to https://www.google.com. An "Authorization Code" is returned as the code parameter in the URL, which should be in this format:
https://www.google.com?code=authorization-code&scope=https://www.googleapis.com/auth/sdm.service
See also: https://developers.google.com/nest/device-access/authorize.
The authorization code can be used to retrieve an access token, that you can use to call the SDM API. Open a terminal and run the following curl command, replacing: oauth2-client-id and oauth2-client-secret with the OAuth2 Client ID and Client Secret from your GCP Credentials, and authorization-code with the code you received in the previous step.
curl -d "" -L -X POST 'https://www.googleapis.com/oauth2/v4/token?client_id=oauth2-client-id&client_secret=oauth2-client-secret&code=authorization-code&grant_type=authorization_code&redirect_uri=https://www.google.com'
Google OAuth returns two tokens, an access token and a refresh token. Copy both these values. The access token is used to call the SDM API and the refresh token is used to get a new access token.
This step is required as part of authorization. Until this call is made, the authorized project will not appear on PCM and events can not be published.
Fill out the values you obtained in the demo script and run the script. If all went well, you will see some stats. This also finally registers your project and credentials.
You now have succesfully hooked up your Nest thermostat and can read and set values from it / to it using a bash script.