Philips Hue whole home installation with C4 physical switches/keypads and circadian warmth and brightness schedule

[ajmccaus]

After doing months of research, testing, and implementing my own DIY smart lighting system using Philips Hue and Apple's HomeKit, I decided to embark on my Control4 lighting journey this past September. This post is to summarize what I was aiming for, what I did, and the outcomes for the benefit of anyone that might want to do something similar. I'd also like to emphasize the limitations and issues with the current Control4 Free Philips Hue driver, those limitations being a significant impediment to reaching my goals with the system.

I'm new to the forum and Control4 and have read through much of the content available on implementing Philips Hue lighting in C4. Many of the issues I see people dealing with are related to the issues I've found with the C4 Hue driver, and could be fixed with a better implementation of the Hue API in C4. Although I'm not a C4 dealer, I am a fairly technical user with software and hardware design experience. My C4 system covers a home theatre, lighting, and security (Overhead garage door control, locks and doorbell camera). This post will be focused on specifically the Philips Hue lighting integration with C4, so on to that.

Requirements

Overall I wanted a lighting system with:

• Automatic time of day brightness and warmth (color temperature) control, so that the lights are less bright in the morning and at night, and also warmer when less bright
• full color lighting in some rooms, e.g. bathrooms and bedrooms have dim red light by default when activated late at night so as not to disturb sleep, and some rooms have full color scenes for relaxing ambiance and dance party purposes
• Control via any app (C4, Hue, HomeKit, etc) and physical switches to replace my existing switches (i.e. no hue dimmer switches fixed to the wall next to my dumb switches with tape or some barrier preventing the power being cut to the bulbs)
• A great user experience consistent with normal light switches, as no one is interested in having to read a manual prior to using the lights in my home
• A way to turn off the power to the bulbs when necessary, from the physical light switch
• Consistent aesthetics for the switches around the house, especially with multi-gang switch banks
• Occupancy on, occupancy off, and vacancy mode in most rooms, with intuitive automatic switching between the different modes
• Compatible with my existing home as a retrofit
• As cost effective as possible while nailing the above key features

I started building this system with HomeKit and Philips Hue and ended up with 1 Hue bridge, 45 hue bulbs, 3 Hue motion sensors, and 5 Hue dimmer switches.

The system worked reasonably well but I quickly ran into the limitations of HomeKit which prevented reaching my ultimate goal. Those limitations that C4 (at least on paper) addressed were:

• No way to backup the programming, so if anything went wrong with the AppleTV HomeKit Hub I'd have to start from scratch again
• No physical switches available that satisfied my requirements for aesthetics and function
• No customization of circadian (HomeKit's "Adaptive Lighting") schedule
• Limited capability for implementing advanced programming and variables (to some extent addressed by HomeBridge)
• Limited product integration support (although this is completely addressed by setting up a HomeBridge server for HomeKit)
• Random glitches that are intolerable for mission critical applications like door locks and primary lighting
• Me being solely responsible and able to fix the system when broken

Implementation

After looking at options for physical switches I decided on using Control4 essential switches which have the following advantages over other options out there:

• Can be set as keypads to control the Hue lights while maintaining power to the bulbs, so that I can control the bulbs via control4 and other means (e.g. Hue app, HomeKit)
• Can be set up with consistent functionality for single, double, and triple tap on the bottom and top switches, along with other non-hue lighting circuits
• Can also control the load to the hue bulbs so I can turn off the power to them completely if I want or need to
• Can be tied into the control4 system seamlessly to control or be controlled by other control4 devices
• Are aesthetically pleasing especially compared to putting hue dimmer switches beside the load switches or covering the load switches in some way
• Are a good price for the functionality they give, about half the cost of the regular C4 switches. Mainly the difference (for the switches, not dimmers) is the essential switches don't have advanced control of the LED lights on the switch itself like the regular switches do

For the circadian schedule I used a driver from Janus Technologies (Control4 Circadian Lighting (drivercentral.io)). I also installed ceiling motion sensors from Nyce which worked alongside my existing 3 Hue motion sensors. I recently learned about a Hue motion sensor driver from UnilogiQ (@Bogdy, Control4 drivers | UnilogIQ) and will update when I get that installed and configured.

Outcomes and Issues:

Initially the implementation was a disaster. The lights were unresponsive and unpredictable. When the switch was pressed to turn on the light, the light would usually turn on, but to a dim level for a brief moment and then to some other brightness. When the switch was pressed to turn off the light, the light would usually turn off, but sometimes it took a while, sometimes it would just go dim and not off, and sometimes it would go off but then come back on a few seconds or so later. When the switch was held to dim or brighten, it was laggy and would continue to change brightness for moments or even several seconds after the switch was released. One and two bulb groups were not too bad, but above that, and particularly with my 6 bulb group, it was very laggy and quite annoying. Not a good long term option. Overall the user experience sucked. It sucked to the point that my wife tried to convince me to ditch the C4 switches and go back to the old regular dumb switches. However, I knew my $30 Hue dimmer switch was doing a much better job so there was no excuse for the poor performance. There had to be a way to make it work.

After some troubleshooting I figured out the root cause was the way in which the free C4 hue driver implements the Hue API to communicate with the bridge. The driver was sending many individual requests when one or maybe a few requests would do it. In other words, there was no attempt by the driver to minimize the number of requests to the bridge, and with the size and scope of my installation that meant that C4 was flooding my bridge with requests and overflowing the buffer in many cases. That meant that sometimes requests were getting missed or were delayed in being executed, hence the intermittent and unpredictable results when pressing the buttons on the switches to turn on and off the lights. In addition to those problems, I found a bug in the C4 hue driver which caused the color temperature to lock at the default level when turned on, so that the circadian driver could not control the color temperature of the bulbs without first manually toggling the color temperature driver in Composer. 

With this root cause determined, I was abled to set up the required groups and scenes in the Hue app and implement custom programming on the C4 switch buttons to solve the problem. Turning on and off the lights was done with the  There is still room for improvement, and I'm missing the dimming functionality on the switches still, but the overall user experience is currently more or less on par with the similar C4 switches I'm using to control the load on the non-hue lighting circuits. In other words, everyone in my household is now satisfied with the user experience of the switches and feels it's an improvement over the regular dumb switches we had before. In addition, the way I implemented the programming solved the problem of the circadian driver being unable to control the color temperature, and it now works perfectly. And the circadian lighting schedule is awesome - I highly recommend the Janus driver (Control4 Circadian Lighting (drivercentral.io) when implemented in this way.

This post is already long enough so I won't get into the details of the programming here, but I'm happy to discuss further with anyone who is interested.

Next steps and Improvements:

Although the core functionality is there with the above, there are still issues that a significant update of C4's hue driver or another driver would solve. Those are:

1. dimming by holding the C4 switch buttons. This is done with a simple Hue API call and could be easily implemented in a driver or even custom Composer programming if there was a driver that enabled PUT requests to be sent from code in Composer.
2. The amount of custom programming required to implement my solution is excessive. Each button event needs to be programmed separately. A good driver that implements the above and that links to the switches would be a great improvement.
3. Turning the bulbs on. Currently this is implemented by activating Hue scenes using UnilogIQ's Hue Motion Sensor driver (@Bogdy, Control4 drivers | UnilogIQ), but a better way for just turning on the switch would be just to turn on the group using a simple API call ({"on":true}) so that the lights would turn on to the most recently used settings rather than go to a defined scene. This is especially important in a circadian application since the color temperature and brightness are constantly changing through the day.

Overall, with the improvements above this would be a great way to implement Philips Hue and/or circadian lighting for retrofits and smaller homes with lower budgets that don't allow for the fully wired solutions.

[ajmccaus]

UPDATE: Unilogiq now has functioning brightness control in their driver and I have implemented smooth dimming using C4 switches and custom programming with the Unilogiq Hue driver. In addition, they have implemented the Hue {"on":true} API call, so I've implemented that as well, which is a big improvement. Now when I press the switch or a motion sensor turns on the Hue bulbs, they go on to their last used state which is usually closer to the circadian default than activating a fixed scene. But the activate scene feature still has it's place and is very useful.

Code example:

[milehighguy]

This is great. I'm trying to implement this now in a single room, but am stuck on one point - how are you using the UnilogIQ driver to turn the lights on? That driver doesn't seem to expose a generic 'on' command - all I see are actions that activate a particular scene or turn bulbs off.

[ajmccaus]

Mostly I’m still using the activate scene command to turn on the lights from the UnilogiQ driver. I have programming for the switch to activate the scene when the switch is pressed.