Siri unable to connect to HomeKit: “Hmm I don’t see anything connected”
I was recently given a 2015 i7 Macbook Pro, 16GB ram/500GB SSD, with a huge LCD leak covering almost the entire screen. A replacement screen is about £500 and a risky repair which may or may not work, plus I don’t need another laptop, so I decided to give it a whirl as an HA server.
My existing HA has been on a Pi 3B+ for about 2 years and is working fine, but a few times a day I see ‘Client unable to keep up with pending messages, stayed over 512 for 5 seconds’ websocket errors. It’s difficult to pin these down but the root cause seems to be too many connections overwhelming the Pi. This is one the reasons for the migration. I have a medium-sized install, about 20 lights, 30 switches/sockets, 40 physical sensors, 300 automations. Hue and Xiaomi hubs. Plus a few other devices (5 PCs, thermostat, front door lock, vacuum, tv, receiver, NAS, xb1, ps4 etc) and 8 phones/tablets.
Another reason for migration is reboots. They take a long time. A year ago they were around 3 minutes, now they’re over 5.
Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
This is one of the reasons for the migration. I get these errors several times a day. If I had to guess I’d say 10 times. If you get this error repeatedly it can totally freeze your Lovelace UI and automations will be very slow, I discovered this after installing the EdgeOS integration back when it first came out. It spewed out a ton of data every second or two.
So these days I am careful not to add sensors that spam too much data (be careful with outo-entities!). The error currently (on the Pi) doesn’t have any noticeable effect, other than annoying me. Maybe occasionally a light takes a half-second longer than normal to turn on. But I assume it’s going to get worse over time as I keep buying stuff to feed my habit.
Speed increases
In the past when investigating hardware upgrade options I’ve had difficulty finding actual numbers for the speed increases people saw when upgrading hardware, so for those of you who are interested:
‘Check config’ has gone from 20-30 seconds down to 2 seconds
Restarting HA has gone from 5 minutes 10 seconds to 50 seconds
Hitting ‘Refresh’ in the mobile app used to take 5-10 seconds. It now takes less than 1 second. Swiping between lovelace panels is also much more responsive.
When installing HA there is a stage where it says ‘downloading assets, this might take up to 20 minutes’. This took less than 5 seconds on the Macbook. I initially thought it had crapped out somehow. When I did this on the Pi a couple of years ago it took at least 10 minutes.
Install issues
I went with VirtualBox and supervised hass.io (not sure what it’s called anymore lol) because I wanted to use MacOS and I wanted no-hassle HA add-ons that I had on the Pi.
The first thing I discovered is do not rely on HA’s snapshot feature. The Pi and the MacBook were both on 0.118.2. I took a new snapshot on the Pi which came in at about 800MB, but after restoring it to the MacBook discovered half the files were missing, including all custom components, automation, scripts, packages, etc. This is apparently a known bug.
In the end I just restarted with a new image, installed all my integrations and add-ons manually, and then copied the contents of the /config folder on the Pi (including invisible files), to the MacBook and changed the internal IP in configuration.yaml. This worked pretty well straight off the bat. For a couple of days I had both the Pi and the MacBook Pro both running with almost identical configs. I was mildly surprised that the vast majority of things continued to work normally.
Do not try and use NAT in VirtualBox. It works but port forwarding is a hassle. Use bridge mode. I started down this NAT rabbit hole because bridge mode initially wasn’t working. I later discovered that bridge mode doesn’t work on wifi (my initial testing was on wifi, this laptop has no ethernet port, and I had to find some adapters). I’m now using a thunderbolt-ethernet adaptor on this laptop and bridge mode works as expected. I briefly used a cheap USB-ethernet adapter which also worked but swapped as I wanted to free up the USB port.
Bluetooth was slightly problematic. The Mac host doesn’t like to give control of the internal bluetooth to VirtualBox so you need a dongle. I couldn’t get one of those ‘tri-band’ wifi + BT4.0 dongles to work, or an ancient D-Link DBT-122 (possibly predates BLE which would explain it). I had success with a TP-Link UB4A dongle. It needs to be plugged in after the VM starts up to be captured properly, but according to various sources on the VirtualBox forums this is expected. I later moved the Bluetooth dongle to a USB hub without any problems. After that first capture it seems to survive reboots, you don’t need to unplug it when you reboot the VM.
Things that don’t work with two HA systems
LocalTuya doesn’t work with two HA instances. I have about 20 Tuya energy monitoring plugs. Some plugs will be picked up one instance, some by the other. No plugs ever had readings available on both instances. Removing the Tuya plugs from the Pi fixed it.
Homekit caused all sorts of problems to begin with. I now put it down to having Homekit setup on the Pi and the Macbook at the same time. This is supposedly possible by assigning each one a different name on each HA instance , but in my testing it would only last an hour or two before both sets of Homekit entities became unavailable.
Things that worked on the Pi but not on VirtualBox
For a long time I couldn’t get androidtv working. At first I thought it was due to this problem. The ADB server is installed and ADB server logs look okay, but I get constant ‘Could not connect to TV at 192.168.0.29:5555 using ADB server at 127.0.0.1:5037’ errors. In the end rebooting the Fire Stick fixed it. It has its own power supply so doesn’t turn off with the TV. It hadn’t been rebooted for quite a while.
I also couldn’t get Samba working. Not a big deal as I can use FTP or SSH instead.
I had a few hitches getting the nginx SSL proxy up and running, mainly because while trying to troubleshoot Homekit I had turned off my router firewall. If you’re in the UK and use Virgin cable don’t be like me and forget that the Superhub’s port forwarding rules do NOT work unless the firewall is active, and there is nothing in the GUI to indicate this.
Other notes
After using the VirtualBox .vdmk image for a while I was warned by HA that I was running out of space. Cue a load of Google about how to expand the image. After expanding the image it now boots to a UEFI shell. I don’t know why and couldn’t figure out how to make it stop. Cue a load more Google about how to write a UEFI startup script and what to put in it. Finally it boots to HA again, a few seconds slower than before, but now with 32GB instead of 6GB.
Keeping the MacBook awake isn’t as simple as it should be. You can avoid all these issues by simply keeping the screen open. If you want to keep the screen closed (which I do!) you need to keep a keyboard, mouse, and power cable attached. You also need software to prevent sleeping (e.g; Amphetamine app, Caffeinate CLI app) but in my testing so far they don’t seem to be 100% reliable. A few times I’ve had the MacBook go to sleep almost instantly after a reboot (presumably because the lid is closed) and for whatever reason Amphetamine didn’t stop it.
Auto-starting the VM when the Mac reboots is also unexpectedly difficult. It doesn’t seem to work as a launchAgent or a aliased Login item. I can double-click the alias and launch the VM, but it doesn’t work in Login items. Still testing, but not too important. I’m expecting this to run for months between reboots, and it has a battery in the event of a power cut.
Currently the Pi is still running with a stripped-down config with only Alexa/Google voice announcements and z-wave configured (restart time down to 2 minutes, wooo). I haven’t tried to move the z-wave Aeotec stick to the MacBook as I only have 1 z-wave device (front door lock) and the Pi is close to it and the MacBook isn’t. To be decided! I’ll either pass lock info from Pi to Macbook or move the MacBook.
While troubleshooting Homekit I noticed that my Tuya plugs were reporting power consumption figures as ‘state changed’ events multiple times per second. I only expose 4 of these switches (out of 20) to Homekit but it’s extra traffic for no reason. I don’t need power consumption in Homekit, so I created some template switches and exposed those instead of the actual Tuya switches.
If you restart a few dozen times in a few days like I’ve recently done you may find your Meross integration stops working with a message about requesting too many tokens. Logging in with the Meross app will fail with an obscure error message. Wait 5 hours and login again with the Meross app. It’ll make you reset your password, and then the integration will start working again.
I had my first ‘Client unable to keep up with pending messages, stayed over 512 for 5 seconds’ error message today. This is the first one I’ve seen in the days since migration, so an improvement over the ten or so I might see in one day on the Pi, but still a bit disappointing.
The last word
Overall it hasn’t been too bad. The initial migration was reasonably easy, fixing the few issues after that was a bit of a hassle. But it’s still been a fun few evenings with the exception of sorting out Homekit which was mucho frustration. But now that’s it done I can purge it from my memory, and enjoy the results. Having a much faster Home Assistant host also opens a few other doors. I’ve already started thinking about facial recognition.
Update: I’ve now moved the Aeotec z-wave stick to the Macbook. I expected this to be much more difficult than it was. After reading this it was simply a case of plugging it into the Macbook and it worked immediately. Didn’t even have to change the usb_path!
Update 2: I was mistaken, Samba does work! I eventually realised that my initial failures were when I was still mucking around with NAT. Now that the VM is in bridge mode Samba works normally, both via direct IP and homeassistant.local
Possibly related
