why is Control4 controller sending packets to 169.254.169.254:80, and should I unblock them?

[Cranky Oldman]

Every few minutes my firewall blocks two packets that my Control4 controller sends to link-local address 169.254.169.254, port 80.  Inspecting the packets, they have a destination MAC address starting with 00:e0:67 which corresponds to "eac AUTOMATION-CONSULTING GmbH".  The packets tend to go out seven minutes apart, but sometimes the gap is shorter, as you can see from the timestamps in the attached screenshot.  You can also see that the controller is incrementing the source port by 2 each time it sends a packet.

My firewall (pfSense) blocks these packets per a default rule.  By blocking these packets, am I preventing the controller from completing some task?

My controller is a Core 5 running OS 3.4.1.701303

[Popolou]

If i had to guess, i suspect a driver is using that address for its telemetry comms. I'd not be overly concerned being that it is link-local, an address more commonly used in cloud-based instances and being APIPA, cannot traverse the gateway.

[Cranky Oldman]

Thanks Popolou for your comments.  Your reference to cloud-based instances appears to be spot-on.  I searched the Control4 controller logs for the link-local IP address, and found matching entries in the file "c4metricsd.log".  The timestamps in the sample below match the timestamps for packets my router blocked.

2024-03-25 14:51:08.520 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
2024-03-25 14:51:08.520 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [EC2MetadataClient] Http request to retrieve credentials failed
2024-03-25 14:51:09.522 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
2024-03-25 14:51:09.522 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [EC2MetadataClient] Http request to retrieve credentials failed
2024-03-25 14:51:09.522 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [EC2MetadataClient] Can not retrive resource from http://169.254.169.254/latest/meta-data/placement/availability-zone

We can see there is some kind of Curl command that is failing, apparently related to Amazon Web Services.   Google found some matches for the string "http://169.254.169.254latest/meta-data/placement/availability-zone" eg.  Getting Ec2 Instance Availability Zone With Curl or Powershell // ShellRunner  

So it looks the Control4 software may be designed to run in a virtual environment, in which case these packets might gather useful info.  Not sure why the controller still sends out these requests when running in a local hardware environment.  I suppose this could be a way of testing "am I in a virtual environment?"  In any event, other than triggering some red-herring paranoia about bots infiltrating my network, the behavior appears to be benign

[Andrew luecke]

Very little is gained by blocking Control4 honestly, and it's simply more likely to cause issues when troubleshooting

 

[Cranky Oldman]

39 minutes ago, Andrew luecke said:

Very little is gained by blocking Control4 honestly, and it's simply more likely to cause issues when troubleshooting

I'm not worried about legitimate Control4 activity, but 3 times in my working life I had linux-based devices from reputable companies compromised by malware and turned into zombies inside the company network.  So I'm always a bit paranoid when I see network traffic I don't understand coming out of a device like that.  Best to block it first and ask questions later!

[RyanE]

https://tonylixu.medium.com/linux-networking-what-is-ip-address-169-254-169-254-f9e23b7332fe

RyanE

[Popolou]

On 3/25/2024 at 9:22 PM, Cranky Oldman said:

Thanks Popolou for your comments.  Your reference to cloud-based instances appears to be spot-on.  I searched the Control4 controller logs for the link-local IP address, and found matching entries in the file "c4metricsd.log".  The timestamps in the sample below match the timestamps for packets my router blocked.

2024-03-25 14:51:08.520 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
2024-03-25 14:51:08.520 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [EC2MetadataClient] Http request to retrieve credentials failed
2024-03-25 14:51:09.522 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [CurlHttpClient] Curl returned error code 28 - Timeout was reached
2024-03-25 14:51:09.522 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [EC2MetadataClient] Http request to retrieve credentials failed
2024-03-25 14:51:09.522 -0500 core5-000FFF9CC615 [1312] ERROR: AWS: [EC2MetadataClient] Can not retrive resource from http://169.254.169.254/latest/meta-data/placement/availability-zone

We can see there is some kind of Curl command that is failing, apparently related to Amazon Web Services.   Google found some matches for the string "http://169.254.169.254latest/meta-data/placement/availability-zone" eg.  Getting Ec2 Instance Availability Zone With Curl or Powershell // ShellRunner  

So it looks the Control4 software may be designed to run in a virtual environment, in which case these packets might gather useful info.  Not sure why the controller still sends out these requests when running in a local hardware environment.  I suppose this could be a way of testing "am I in a virtual environment?"  In any event, other than triggering some red-herring paranoia about bots infiltrating my network, the behavior appears to be benign

Glad to help. What you'll come to find from the wealth of data that comes with a C4 and pfSense combo (and if you use pfblockerNG) is a large world where devs use many different cloud-based systems not just to collect metrics but also to communicate back with their drivers to either provide improvements or upgrades/unlocks that you may have purchased. If you go down the blocking path, you may find you'll need to investigate each service in operation if in case you end up blocking one that causes a driver to default to its basic function. Some C4 services are purely telemetry (like the one you pointed out) but others do considerably more which could cause problems (such as to split.io). It's just the current state of play in the dev world.

[Cranky Oldman]

1 hour ago, Popolou said:

Glad to help. What you'll come to find from the wealth of data that comes with a C4 and pfSense combo (and if you use pfblockerNG) is a large world where devs use many different cloud-based systems not just to collect metrics but also to communicate back with their drivers to either provide improvements or upgrades/unlocks that you may have purchased. If you go down the blocking path, you may find you'll need to investigate each service in operation if in case you end up blocking one that causes a driver to default to its basic function. Some C4 services are purely telemetry (like the one you pointed out) but others do considerably more which could cause problems (such as to split.io). It's just the current state of play in the dev world.

another timely comment!  I was just in the middle of figuring out why some of the functionality on the Control4 website (voice control) wasn't working and I tracked it down to pfBlockerNG DNSBL blocking the domains events.split.io, sdk.split.io, and auth.split.io.   Looks like I'm going to have to whitelist them, at least temporarily.

my Control4 system is in a new house and the original network installation was done during construction by the Control4 dealer using an Araknis router that didn't have the advanced features that I was used to with pfSense, so as soon as the dealer finished his work I upgraded to a pfSense box and carefully rebuilt the network with VLANs and lots of rules to segregate Control4 from other IoT devices and from my private LAN, while trying not to break any Control4 functionality or block the dealer's ability to provide remote support.   Generally that conversion was successful although I did manage to break some things along the way, earning appropriate feedback from my wife!  I've learned not to make changes while she's in front of the TV!

One security hole I'm wrestling with is the fact that my Control4 dealer, or any of his staff, currently have the ability to remotely connect to my system and turn off my security alarm, unlock my front door, and turn off my security cameras.   I'd like to figure out how to block that access by default, and only open it up when I have a specific need for support.  But I haven't spent any time yet figuring out if that is possible/practical to implement without breaking my own remote access to my system.   It's on my to-do list as I continue to climb the C4 learning curve!

[msgreenf]

Disable remote support on your customer.control4.com profile

[Cranky Oldman]

8 minutes ago, msgreenf said:

Disable remote support on your customer.control4.com profile

Thanks msgreenf!  that was too easy once you told me where to look.  Crossing that item off my to-do list!

[msgreenf]

now if they know your WiFi SSID/Passkey they can still get support by sitting outside; but don’t think that is a valid risk to worry about

[Cranky Oldman]

1 minute ago, msgreenf said:

now if they know your WiFi SSID/Passkey they can still get support by sitting outside; but don’t think that is a valid risk to worry about

You're right that the random-guy-in-the-driveway scenario is low risk, but it probably won't surprise you if you've read this far in the thread that I'm more paranoid than most people about these things!  I set up the VLAN for the Control4 network with a whitelist of MAC addresses that are permitted to use the network.  So a random device that shows up in the driveway with the SSID/passkey will get connected to the access point, but won't get network access.  It was a bit of a pain to set up the whitelist initially, but once in place it shouldn't require any maintenance unless I change hardware somewhere in my system.  If I need a house call from my dealer at some point, I'll have to turn off access control or add his phone/PC to the whitelist.   

 

[msgreenf]

Not worth it. Risk vs reward. Don't spend more $ protecting something then it's worth...

[msgreenf]

Do you drive? Or do you have a convey surround you so  no one can hit you?

[msgreenf]

Pay for a security guard?

[Popolou]

5 hours ago, Cranky Oldman said:

another timely comment!  I was just in the middle of figuring out why some of the functionality on the Control4 website (voice control) wasn't working and I tracked it down to pfBlockerNG DNSBL blocking the domains events.split.io, sdk.split.io, and auth.split.io.   Looks like I'm going to have to whitelist them, at least temporarily.

For my next trick, tomorrow night's lottery numbers...

Oh and you really only want to whitelist the TLD so that you capture any future changes should they roll out new services/servers.

1 hour ago, Cranky Oldman said:

You're right that the random-guy-in-the-driveway scenario is low risk, but it probably won't surprise you if you've read this far in the thread that I'm more paranoid than most people about these things!  I set up the VLAN for the Control4 network with a whitelist of MAC addresses that are permitted to use the network.  So a random device that shows up in the driveway with the SSID/passkey will get connected to the access point, but won't get network access.  It was a bit of a pain to set up the whitelist initially, but once in place it shouldn't require any maintenance unless I change hardware somewhere in my system.  If I need a house call from my dealer at some point, I'll have to turn off access control or add his phone/PC to the whitelist.   

 

You probably already have it installed but i use the arpwatch package for this . Whilst primarily monitoring the blackhole vlan across my sites, if a new device joins any point in the network, it will trigger a notification.

[Cranky Oldman]

3 hours ago, Popolou said:

For my next trick, tomorrow night's lottery numbers...

lol!

thanks for the arpwatch reference; I haven't used that yet.  will check it out

[Cranky Oldman]

3 hours ago, msgreenf said:

Not worth it. Risk vs reward. Don't spend more $ protecting something then it's worth...

one of the luxuries of being retired is I can spend almost unlimited amounts of time making things like this work the way I think they should work.  No more 80/20 rule for me!   A younger me would have thought it was crazy too!