Erratic/Unreliable C4 behavior

[TundraSonic]

Just now, chopedogg88 said:

I don't believe anyone is suggesting it's a network bandwidth issue. I'd have to see your system to be able to help much more otherwise just shooting in the dark. 

Thanks. Because of my wife's job there are only a very few people allowed network access.

[TundraSonic]

So, with a change in light or audio level. On T3 the user moves the slider from 0 to 20%. Presumably the T3 sends a state change packet to the EA5, change happens, ack, etc. Or change doesn't happen, nack, retries, etc.

So when the slider moves itself to 74% or 100% (instead of the 20% selected by the user) AND that adjustment happens in the system EA5, how could the network have caused that?

With turning off a session not working I can see where a network problem could cause that (though proper programming would issue retries and then provide an error message). With the slider problems I can't get to how any kind of network problem could cause it. Network problems could cause it to not execute or to execute late, but that I can figure out a network problem cannot (or certainly should not) cause a change to a state outside of the bounds of the original command.

Thanks,

[cdepaola]

If this is a new home and a new install then your dealer should be much more involved in fixing these issues as on-site work is obviously needed.  If they can't figure it out they can then get C4 involved.   I understand wanting to get this figured out and seeing this forum as a source however what you're describing needs someone on-site. 

 

 

[lippavisual]

Get your wife's IT people involved then, if they were the ones who set it up.

Clearly there is some misconfiguration in the network.  You may be better off separating your home network and your wife's.  It would be much easier for you or anyone else to diagnose.

[booch]

I’m having a similar (in some ways) problem so am curious to see what you find. Specifically, I have time-of-day-specific scenes tied to (the invoking of) bonded scenes on all of my keypads (each room’s on/off). About 1/15 times it doesn’t work, in a random room, which is odd. My keypads/lighting are all Zigbee, so I’d thought mesh quality might be the issue. Now I’m not so sure.

I do have UniFi APs (though running on Pakedge switches/router) and, although all of the C4 stuff should be on the hardwired network, just disabled broadcast filtering (thanks to/for the tip above.)

[mujtaba.khokhar]

Clearly you have some sort of networking knowledge here. 

So let me be the second person to tell you that Control4 communicates over a broadcast network, its not a task done a packet sent and then and another packet sent back. when you ececute a task in c4 the processor send it to everything even via ZigBee to your remotes. 

The issue you are seeing with your sliders, when you turn it up to say 70% from 20% and it goes back to 20%, means the light level sate did not get back to the processor and or C4 failed to execute a command from the load (Dimmer Module in your case) so the slider has reverted back to last known state, this is the same for volume. 

C4 in terms of network transmits and receives  a lot of data (Open up Wire Shark point it to the IP of your controller and you'll see how much info is being sent out and received) 

Reason UniFi is on the "Ban" list: 

Issues with SDDP communication across VLANs. Ubiquiti EdgeMax routers and switches do not support PIM multicasting. This can affect routing SDDP traffic across VLANs. 

Also see thread here

https://community.ui.com/questions/PIM-on-the-roadmap/b0e73936-9052-477f-890d-c5dc5fec3f7f

Of course everyones millage varies, everyones setups are different and here we do try and accommodate everyone, but your constant disregard of advice and what people have said to you countless times is just making people not want to reach out. 

if you really don't think its network then start simple, are your RJ45 ends made of correctly on each end? Have they been tested? Do they pass gigabit at the minimum?

Can you ping your touch screens using a simple ping? whats your latency? Are you getting packet drops? Run a nmap? run a trace route? 

and "rouge" devices that could be causing this? IOT? Snoff? anything?

Have you checked your firewall? Do you have 

All things to consider, it's not just a case of plugging a few things in and being like yeh it will be fine. C4 and many other home automation systems rely on a solid network / backbone. if it's not there, things will likely fail. 

sorry to be so blunt, but i don't like beating around the bush

A few more notes to check off:

Disable 

• Spanning Tree Protocol
• Flow Control
• QoS (Quality of Service)
• Wireless isolation (QoS setting)
• Privacy Separator
• PaGP (Port Aggregation Protocol)
• IGMP Snooping (or any other Multicast Filtration of any kind)

Enable 

• Multicast function on the router, managed switch and access point (if applicable).

 

Honestly not sure what more info I can give to you, other than use Mikrotik next time with Ruckus APs or just go full out Araknis lol 

Thanks 

Muj

[TundraSonic]

11 hours ago, mujtaba.khokhar said:

The issue you are seeing with your sliders, when you turn it up to say 70% from 20% and it goes back to 20%, means the light level sate did not get back to the processor and or C4 failed to execute a command from the load (Dimmer Module in your case) so the slider has reverted back to last known state, this is the same for volume. 

Yes, this could certainly be the case if it goes back to it's previous state (or possibly even if it goes to an intermediary state). However, if the previous state is 0, the user moves the slider to 20%, C4 moves briefly to 20% but then jumps to 74% or 100% - how can that network related?

[TundraSonic]

13 hours ago, mujtaba.khokhar said:

but your constant disregard of advice and what people have said to you countless times is just making people not want to reach out. 

I am not at all disregarding people's advice. But when the response is simply 'network', that is not helpful. Am I to pull out all of my Ubiquity system and replace it with Cisco? 

 

13 hours ago, mujtaba.khokhar said:

• Spanning Tree Protocol
• Flow Control
• QoS (Quality of Service)
• Wireless isolation (QoS setting)
• Privacy Separator
• PaGP (Port Aggregation Protocol)
• IGMP Snooping (or any other Multicast Filtration of any kind)

Thank you. All of these except port aggregation are disabled and have been. We do have a NAS and a couple of Mac Pro's that are port agg'd but these are on a separate switch/VLAN/Subnet so should not be interfering in any way w/ C4? They are also LACP, not PaGP.  

Multicast is ON.

[TundraSonic]

12 hours ago, mujtaba.khokhar said:

Issues with SDDP communication across VLANs. Ubiquiti EdgeMax routers and switches do not support PIM multicasting. This can affect routing SDDP traffic across VLANs. 

No C4 traffic s/b crossing any VLAN (or subnet) boundaries.  It's all on the same Switch/VLAN/Subnet. This should not be an issue?

Ethernet connections were all tested and met 5e.  Something could still have come loose but there is no indication of that.

Some quick pings. These were from 2 hops (physical switches) away. I can stick a machine directly on the C4 switch/VLAN if that would help:EA

EA5:

12 packets transmitted, 12 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.294/0.357/0.433/0.045 ms

 

8DIM:

12 packets transmitted, 12 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.714/1.066/1.817/0.298 ms

 

T3a:

12 packets transmitted, 12 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.359/0.548/0.601/0.065 ms

 

T3b:

13 packets transmitted, 13 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.404/0.511/0.709/0.072 ms

Edit: The 8DIMs are all quite slow responding and much slower than any other device on the network I tried. I'm assuming that this is an internal turnaround issue and not network related. 

Firewall should not be an issue except for traffic to/from public internet:

Cable Modems > Firewalls > UI UDMP > Switch Core > Switch C4  

 

[TundraSonic]

On 12/28/2020 at 10:53 AM, cdepaola said:

If this is a new home and a new install then your dealer should be much more involved in fixing these issues as on-site work is obviously needed.  If they can't figure it out they can then get C4 involved.   I understand wanting to get this figured out and seeing this forum as a source however what you're describing needs someone on-site. 

Yes and no. As people I like my dealer very much. From a technical standpoint they're in a bit over their heads on many things.

They were originally to have done the network but after the bid was awarded to them (and I think after they'd already started pre-wire) they pulled out stating that they'd decided it was too complicated for their level of knowledge but that they'd help us find someone else to do it. As it turned out all other integrators said that they would only do the network if they could do the entire job. The only people who would do network only were corporate folks who wanted 3x - 7x as much as was in our budget (which was based on bids from 3 integrators).

Sometimes you have to make compromises and given delays already caused by our integrator pulling out at the last minute I reluctantly agreed to do the equipment/config/mgmt if they'd do all of the wiring and other bits which they agreed to. My wife's work is involved primarily to insure that her stuff is secure. Fortunately they have also been good about answering questions and offering advice otherwise.

Our dealer (1 tech does all of the C4/Media/Security/Surveillance stuff while other guys do all of the wiring) are great guys and know a lot of things that I don't. OTOH, I've had to fix a number of things in C4 that they did incorrectly and there are other things that I can't fix that I'm still waiting on them to correct like garage lights coming on every time any door is opened (they're busy and I'm relatively patient so not a big deal). 

So in the end it is sometimes much easier, faster and more efficient to do things yourself. And perhaps more important, doing so also provides for a good learning experience. In this case in particular if it does turn out to be a networking problem since I have to manage the network. There have also been a few instances of our dealer not understanding something and fixing the wrong things. This is the most complicated project they've done and I suspect they didn't account for the extra after install support needed based simply on complexity. Finally, they're overloaded and like others have been dealing w/ a lot of Covid issues and to the extent I can I'd like to keep the burden off of them.

[TundraSonic]

And sorry for the late replies. I was at urgent care yesterday afternoon & night so trying to get caught up on a lot of things.

[mujtaba.khokhar]

1 hour ago, TundraSonic said:

Thank you. All of these except port aggregation are disabled and have been. We do have a NAS and a couple of Mac Pro's that are port agg'd but these are on a separate switch/VLAN/Subnet so should not be interfering in any way w/ C4?  

Multicast is ON.

It shouldn't be, are you routing traffic via firewall? or do you have NAT rules set for this? Are you allowing traffic to float between VLANS?

one thing if you could do and i know its a pain, could you disable port aggregation, obviously unplug everything that's not c4 related and take a backup of your switch config, see how things behave then. I'd just like to rule out that & to also see how C4 behaves.

51 minutes ago, TundraSonic said:

Some quick pings. These were from 2 hops (physical switches) away. I can stick a machine directly on the C4 switch/VLAN if that would help:EA

EA5:

12 packets transmitted, 12 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.294/0.357/0.433/0.045 ms

 

8DIM:

12 packets transmitted, 12 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.714/1.066/1.817/0.298 ms

 

T3a:

12 packets transmitted, 12 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.359/0.548/0.601/0.065 ms

 

T3b:

13 packets transmitted, 13 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.404/0.511/0.709/0.072 ms

 

To me these look pretty good, latency is low, no packet loss.. 

Are you able to run a traceroute? 

secondly run a continuous ping and then try and operate the C4, do you get any packet loss?

Thirdly, id love to log into your processor run some terminal commands see if things are running okay and the project is corrupt, also to run some diagnostic on it too, and check the MD5 checksum is all good. 

 

Thanks 

Muj 

[TundraSonic]

2 hours ago, mujtaba.khokhar said:

It shouldn't be, are you routing traffic via firewall? or do you have NAT rules set for this? Are you allowing traffic to float between VLANS?

one thing if you could do and i know its a pain, could you disable port aggregation, obviously unplug everything that's not c4 related and take a backup of your switch config, see how things behave then. I'd just like to rule out that & to also see how C4 behaves.

Thanks. Are you saying that I should not be allowing multicast? 

The only traffic via firewall is to/from public internet and there are NAT rules for this. External connection from C4 apps or website to our EA5 work well. 

VLANs vary based on VLAN. Surveillance VLAN (cams) is extremely segregated. Guest VLAN and an IOT VLAN for stuff that needs internet but not direct connectivity to us internally are just that - they can see the internet but nothing else except a printer. LIFX VLAN is semi segregated - it allows what is necessary for LIFX to function. C4 VLAN can see internet and has necessary routing to our personal devices VLAN. Wife's VLAN is tight.

Disco'ing link ag would be a good bit of work itself and disruptive to editing/rendering work. I cannot imagine any way whatsoever that it would be causing problems especially with all on a different Switch, VLAN and Subnet from any C4 or HA stuff.   

[TundraSonic]

I did a long ping to the EA5 (still from 2 hops away) and a few C4 commands on a T3 (two scene activations, six light level changes, one gain change). No packet loss. C4 commands all executed properly. Ping times were running in the 2 - 3 ms range though.

[OceanDad]

If I were you, I'd take Muj up on his offer to take a look.

[TundraSonic]

I've been running pings against the EA5 & T3's. Here are a few.

 

84 packets transmitted, 84 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.191/0.328/0.571/0.077 ms

 

181 packets transmitted, 181 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.171/0.327/0.521/0.067 ms

 

129 packets transmitted, 129 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.285/0.497/0.826/0.084 ms

 

3000 packets transmitted, 3000 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 0.253/0.483/1.082/0.090 ms

[msgreenf]

Ping is not a valid test of blocking of broadcast traffic. 

[C4 User]

Your issue is not C4. It’s something related to your setup. I say that because I’ve had C4 for over 12 years now in multiple setups and the system has been very stable (other than the occasional new firmware bug) for many years now. My entire network and all C4 controllers are on UPS’s and I can go a year without a reboot if no updates or changes are made to the system. So most likely a network setup issue. Maybe a C4 controller setup issue given all the EA5’s you have. Or if some of your devices are in fact zigbee and you do not have a strong enough mesh that could be a problem. If your wife’s job requires that level of security put in a second firewall and isolate her work network completely. But you’ve spent a lot of time and money. Get the help you need to resolve the issue. My two cents. 

[TundraSonic]

17 hours ago, msgreenf said:

Ping is not a valid test of blocking of broadcast traffic. 

Thanks. Agree. Muj was asking for it though.

Again though, ALL C4 stuff and everything it connects to are on the same Switch/VLAN/Subnet. I cannot find anywhere that any broadcast traffic is being blocked or where there would be any expectation of it being blocked. If it is network I would love to find the problem but I'm stumped by this. Where is it being blocked? 

And, how can a network problem cause a slider to move to 74 or 100 when the user moves it from 0 to 20?

[mujtaba.khokhar]

23 hours ago, msgreenf said:

Ping is not a valid test of blocking of broadcast traffic. 

Thanks for that... it does however show latency and if it's high you know there's something wrong, just a box I was trying to tick & it's usualy a good starting point. 

 

Also, i've just logged into they only client i have who has UniF*ck (everyone else I have swapped out over the years).

I have these settings turned on:

Turn on IGMP Snooping

Turn on the wireless side IGMPv3 on 

Turn on mDNS

Pretty much goes against everything i said before, again this is not a 100% fix and your mileage may vary. 

All i can say is give it a go, give your c4 a reboot and see how it is. 

Only difference is they are on a flat network, no Vlans no nothing and they are on 2.10.6. 

Thanks 

Muj

 

[booch]

18 hours ago, TundraSonic said:

Thanks. Agree. Muj was asking for it though.

Again though, ALL C4 stuff and everything it connects to are on the same Switch/VLAN/Subnet. I cannot find anywhere that any broadcast traffic is being blocked or where there would be any expectation of it being blocked. If it is network I would love to find the problem but I'm stumped by this. Where is it being blocked? 

And, how can a network problem cause a slider to move to 74 or 100 when the user moves it from 0 to 20?

Not trying to interrupt your dialogue, but to chime in again with info. on my (potentially) related problem: I notice this too and suspect it may be related to some kind of ‘cache’/sleep logic in the lighting hardware. For example, I updated some scenes tonight and when activating them saw on two occasions (via navigator on my phone, which admittedly could be inaccurate) what appeared to be the hardware trying to first seek a prior preset command (mine change 5x daily) before seeking the scene level. Again, in my case, this could be a Zigbee strength issue, but it’s something to consider. Based on my (admittedly limited) understanding, it’s not unheard of for IOT devices to behave like this.

[TundraSonic]

Thanks. I have an article due today and an editor who doesn't seem to care about my personal home automation or network problems so time is limited until I get my work done.

Also, I'm a bit rusty w/ network stuff. 25 years ago I was solid, having to brush cobwebs off today (and some of this didn't exist 25 years ago).

IGMP Snooping can reduce traffic through pruning but I assume C4 want it off because it can prune too much causing delays as they have to re-establish. However, I believe this should only apply to traffic hopping a VLAN or subnet boundary? So long as all C4 stuff is on the same VLAN and subnet (and it is in this case) IGMP snooping shouldn't make any difference? Turning it on could help if it reduces traffic but this network is so lightly loaded that I'm not sure there'd be any difference.  I may try it later today to see what happens.

I currently have IGMPv3 off. Similar to above, all of the C4 devices are on the same VLAN & subnet and all hardwired. None of this involves anything wireless that I know of. Possibly turning this on could reduce multicast traffic but this network is rarely even very busy, much less saturated.

mDNS is needed for a number of apps like Apple TV iPhone Remote, etc.  I could turn it off for bit but similar to above I'm struggling to see what affect that would have on the C4 stuff other than maybe reducing mDNS traffic on the C4 VLAN.

Is C4 using UDP for all of this? 

Thanks,

[TundraSonic]

There are a few things that somewhat consistently malfunction such as the recessed light for my wife's vanity (video from other thread), lights in scullery that take a long time to turn on (press button on C4 6 button remote, long delay of varying lengths, lights on C4 dimmers finally react).  I may try to do see if I can get them to malfunction with a packet capture later today.

[wnpublic]

Lots of diy verbiage here, I support. But to return to what others have said:  where is the dealer (or any dealer) in all of this?  You need to engage someone with experience to diagnose what’s going on, rather than running mad science experiments (what happens when I enable X?) on your own or based on Internet suggestions. 

[Neo1738]

17 hours ago, wnpublic said:

Lots of diy verbiage here, I support. But to return to what others have said:  where is the dealer (or any dealer) in all of this?  You need to engage someone with experience to diagnose what’s going on, rather than running mad science experiments (what happens when I enable X?) on your own or based on Internet suggestions. 

As he stated either (earlier or in another thread not sure) bc of his wife's job he can't give access to his Networking as much. And his dealer said it was beyond his capabilities to do the setup of the network. That he could run the wires but the programming and setup was on the client. Tundra had his wife's IT people do some of the setup with him. That is why his dealer doesn't want to deal w the networking issues.