Erratic/Unreliable C4 behavior

[C4 User]

Given the investment is seams @TundraSonic has in C4, and the fact that his high security environment may restrict good support to his C4 setup for many years to come, I will say it again that it might be best for him to invest in a second firewall and simply separate the work network for his wife. Doing so will enable him to easily get the combined network/C4 support he might need. That said, I have never given my C4 dealer access to my network, including firewall or switches. I do simple management of my network myself and I have professional services monitoring all aspect of my network. In over 12 years, there has never been a network issue that we could not identify, isolate and resolve through monitoring and packet capture. One does not need C4 experience to ensure the network itself is functioning properly and not unintentionally restricting, blocking, or holding up traffic. My dealer gets access to C4 via a computer in my rack that I have to personally give him access to via teamviewer. Has always been solid.

[TundraSonic]

One quick note - The C4 dimmers are all on the EA5 ethernet switch, not Unifi. I'm not sure what difference that makes as I can't see the switch config in HE. Is there a router element of any sort between the 'ethernet in' and that switch? Is this a purely dumb switch and 'ethernet in' is a peer?

What does this mean for erratic behavior involving only a 6-button keypad and C4 dimmers (so everything within the C4 ecosystem)? 

Thanks,

 

 

[msgreenf]

I would not use the switch ports on the ea5. I would put everything direct on the unifi switch

[TundraSonic]

1 hour ago, C4 User said:

Given the investment is seams @TundraSonic has in C4, and the fact that his high security environment may restrict good support to his C4 setup for many years to come, I will say it again that it might be best for him to invest in a second firewall and simply separate the work network for his wife. 

Possibly a good idea but could also be quite difficult and expensive. If her working was limited to a single laptop in her office then it'd be a better option but she works all over the house and outside so we'd need to replicate all or a good chunk of our 11 AP's. Not to mention other possible issues of making sure that she has full access to printers, TV's, etc (which I think could be done w/ dual logins but that can get messy and with spanning tree turned off for C4 could be messier).

It's 2021 and networks are fully capable of operating in an integrated fashion. There s/b no need to separate her out.

HOWEVER, I'd once before discussed w/ our integrator having a separate C4 network for stuff that talks ONLY to the core EA5 which I believe would be T3's, Dimmers, maybe audio matrix & amps?  EA1's and CA1's?  

[TundraSonic]

11 minutes ago, msgreenf said:

I would not use the switch ports on the ea5. I would put everything direct on the unifi switch

Thanks. Any reason why?

[msgreenf]

2 minutes ago, TundraSonic said:

Thanks. Any reason why?

that switch is really meant for a smaller system where that is the only switch.  the more traffic you can keep on a single switch the better you are going to perform.

[wnpublic]

One or more elements of this seem off - coated in BS. Have fun y’all

[cdepaola]

1 hour ago, TundraSonic said:

One quick note - The C4 dimmers are all on the EA5 ethernet switch, not Unifi. I'm not sure what difference that makes as I can't see the switch config in HE. Is there a router element of any sort between the 'ethernet in' and that switch? Is this a purely dumb switch and 'ethernet in' is a peer?

What does this mean for erratic behavior involving only a 6-button keypad and C4 dimmers (so everything within the C4 ecosystem)? 

Thanks,

 

 

Somewhat important piece of information and an example of why Trouble Shooting via a Forum is difficult at best and why on-site support in this case is needed. We don't know your full set-up, how its configured, etc.

As @msgreenfmentioned, move everything using those switch ports onto your Unifi switch. 

Also, your wife's situation is not unique, by any stretch of the imagination.  If she's using multiple computers and devices for her secure communication in the home then they obviously aren't that important.  Clients and friends I've got that need this level of security typically have a single computer on an isolated and wired network and a secondary mobile device just for their business communications.  That is a home computer and and work only computer, a primary mobile device and a secondary work only device.  

[Neo1738]

11 APs? how much square footage over 10k? Get a couple Rukus cut that down a bit. But seriously if all of this is that ridiculous just separate c4,  create a vpn for c4 so when home on your 11 ap you can use c4 app as if on same network. 

[DanITman]

I run all Unifi gear from Ubiquiti without issue.  I think the C4 warning is about their edge products which are older.

Sonos, which also runs on a broadcast network functions fine on Unifi gear.  

[TundraSonic]

Back on my feet again...

House is 9400 sq ft but there are also AP's in the studio (separate bldg which aren't in yet so not in the count) and three covering space outside (including one solar powered at the end of an airmax radio shot that works surprisingly well). Ruckus would have cost 3x as much (plus ongoing maintenance costs), use 2 more AP's according to the design and they were still planning to use Unifi for the radio and outdoor AP's.

As mentioned previously C4 is already on it's own Switch/VLAN/Subnet. I'm not sure what benefit a VPN for C4 would be???

I currently use the C4 app as if I'm on the same network. ???

Thanks @DanITman. At one point C4's concerns were PIM (which shouldn't be an issue unless crossing route boundaries) but then they removed that but left them on the DNU list.

@cdepaola, They're pretty cautious with her devices and the data on them and she wants to be able to use them anywhere in the house or outside anytime. The dedicated thing in her office lasted about 2 months before she informed the IT folks how it was going to work.  

[Neo1738]

4 hours ago, TundraSonic said:

Back on my feet again...

House is 9400 sq ft but there are also AP's in the studio (separate bldg which aren't in yet so not in the count) and three covering space outside (including one solar powered at the end of an airmax radio shot that works surprisingly well). Ruckus would have cost 3x as much (plus ongoing maintenance costs), use 2 more AP's according to the design and they were still planning to use Unifi for the radio and outdoor AP's.

As mentioned previously C4 is already on it's own Switch/VLAN/Subnet. I'm not sure what benefit a VPN for C4 would be???

I currently use the C4 app as if I'm on the same network. ???

Thanks @DanITman. At one point C4's concerns were PIM (which shouldn't be an issue unless crossing route boundaries) but then they removed that but left them on the DNU list.

@cdepaola, They're pretty cautious with her devices and the data on them and she wants to be able to use them anywhere in the house or outside anytime. The dedicated thing in her office lasted about 2 months before she informed the IT folks how it was going to work.  

My point was to completely separate C4 from wife's networking gear. The vpn would be because you couldn't use the C4 app as if you were in the home without it if you were on the wife's network. 

[Control4Savant]

 

On 1/12/2021 at 4:21 PM, TundraSonic said:

 

Thanks @DanITman. At one point C4's concerns were PIM (which shouldn't be an issue unless crossin route boundaries) but then they removed that but left them on the DNU list.

@cdepaola, They're pretty cautious with her devices and the data on them and she wants to be able to use them anywhere in the house or outside anytime. The dedicated thing in her office lasted about 2 months before she informed the IT folks how it was going to work.  

Ubiquity is on the DNU list because of its advanced setup. Control4 does not want to become general IT support. 

 If your wife wants to ignore (sorry..."inform") the IT department on how she wants to go about her day, everything you are discussing is ultimately pointless. As @cdepaola stated, people needing this level of security are on completely dedicated and isolated environments, they don't get to just wonder around by the pool on a sunny day. 

[ekohn00]

@TundraSonic Read through this for kicks. I too have an all unifi setup and secure things for work. Never once had a problem with the unifi network. Did have a Yale/zigbee issue...but a little fine tuning resolved that.

I've noticed some oddities when the WAN has problems, I'd recommend you setup as many C4 devices (especially the controllers) with static IP's. Also make sure they're using a DNS like 1.1.1.1 or 8.8.8.8 instead of your gateway.

For trouble-shooting on this one, I'd first back up your unifi config then see if it works without using VLANs in the network. Just one simple flat network. Because of some oddities with VLAN config in unifi, some things just don't get set up right with Vlans.  Hopefully that solves your problem.

you mentioned the switches are wired? I think someone said they're RS-485? How are they set up, could be there's a problem how that's set up??? Did they use CAT4/5 for RS-485? That could be a problem....

 

[mindedc1]

So some clarification and information for you to cut through the clutter. I'm a networking professional and have architected many very large high scale networks (500K user+ type environments). I was CCIE level 20 years ago, tier 3 tac level at a Cisco competitor, have moved on to other products but manage a networking team now. My guys install around 20K access points each year and thousands of switches, lots of large firewalls etc. I also do a little C4 programming as freelance for a dealer on the side, some C4 driver development as well as have a decently large install myself with lots of odd things. I run pro networking gear myself at home in my environment (Ruckus and Juniper switching, have use Mist, Aerohive, and Aruba APs with my current C4 setup as well).

All of the C4 restrictions around networking are because most dealers don't have an in-depth knowledge of networking. Let's discuss:

C4 uses broadcast - not true. Do a wireshark and set the display filter for eth.dst == ff:ff:ff:ff:ff:ff, you'll see lots of ARPs and probably netbios garbage but nothing important for C4 beyond ARP. C4 does rely heavily on multicast (wireshark display filter ip.addr >= 224.0.0.0). Ubiquiti is supposedly on the "no" list because of some UBNT hardware not forwarding multicast, specifically the routers as I recall. It works fine other than that rare case, no reason to change to Cisco or anything else.

Spanning Tree Protocol - Recommended against because it's a pain to configure properly and configured improperly you can have issues, I suppose they consider it better to allow a customer's network to get locked in a hard loop. It works fine, I've used MSTP, RPVST+ and some other flavors with C4 and it's fine if configured properly. Home grade switches may have poor implementations.

Flow Control - Properly implemented flow control is fine, the issue is most network hardware doesn't expose getting pause frames from devices that are overloaded..again instead of identifying the problem it turns into some sort of mystery that's resolved when the overloaded device is replaced randomly. This could be an issue in switch to switch communications or server to server where lots of disk IO is happening. If you're running pro gear and routinely check for pause frames as part of your troubleshooting not an issue.

QoS (Quality of Service) - I've also used switch level QoS with no issues, I would be cautitous about QoS or traffic shaping in home grade routers. Industry standard DiffServe on switches works fine. I would avoid punishing multicast or anything C4 is doing. I would also avoid voice prioritization for wireless unless you have someone who really knows what they're doing set it up. Lots of bugs in those features on most wireless products.

Wireless isolation (QoS setting) - This will cause problems RE C4 discovery works off of multicast.

Privacy Separator - presume this is same as client isolation, same deal

PaGP (Port Aggregation Protocol) - Works fine as well as LACP/802.3AD dynamic and static. If  you don't know how to configure these protocols you can of course create problems. There are special criteria in PIM environments with LAG/PAGP

IGMP Snooping (or any other Multicast Filtration of any kind) - Presuming from the bitching I hear from dealer friends about this, I presume that either the C4 gear doesn't prune/join properly or they've had networking folks not set this feature up. I run IGMP enabled on my Juniper switches but your mileage may vary. They left off storm control, DHCP snooping, Dynamic ARP inspection, etc here but again, I use it and it works fine.

So, the recipe for success, is keep all the C4 gear and all the clients that talk to it on the same VLAN, even over wireless. 

You should be able to have multiple VLANS working with PIM with a phone or touchpad on one side, however if any of the C4 devices don't handle IGMP properly you may still have issues. C4 doesn't really define which devices need which s, g publicly so any PIM troubleshooting would be a guessing game. You may also have RP overload in a home environment with a setup that large. We generally filter out SSDP, MDNS, Bonjour etc in corp environments as I've seen 50 misbehaving clients take down a 1 terabit line rate switch with enough control plane traffic thanks to MDNS. Though technically possible you would be in the minority with this type setup. The exception here would be things that do not need multicast discovery. For example I have all of my IP cameras on a firewall zone on an isolated switch (ethernet connection on the outside of my house, security problem!). The IPs are manually configured in C4 so no need for multicast, works great. If you put things that never need to talk to C4 on different VLANs, that's fine as well.

I saw you ran some pings as a test. 12 pings isn't enough and if you're running 1G or 10G you're not loading enough of the wire to see an issue. I would run 1000 pings and I would also increase the packet size to 1472, after applying the header that's the largest payload you can squeak through a 1500 byte mtu ethernet connection. If you run 100% clean with no loss I would say it's unlikely network is your issue. You will see loss over wireless, I would test from wired. You could still have multicast being blocked but that would be more difficult to test. You can use netcat if you have a unix system or if you have two laptops you can generate a multicast stream with VLC and some video file. Google will explain that process better that I am willing to here. I would also suspect that your issue with controls not working is not a multicast issue. Multicast is used by C4 for discovery. Once the remote device is discovered (Say an EA1) the communications are unicast for which ping is a good test. Anyone that doesn't believe me is free to use the above wireshark filters to find out for themselves.

I think it's unlikely you have a wired network issue.

I would look into the EA that runs director. I've had severe C4 latency issues with an IP driver (Yes, a Control 4 issued driver BTW) eating a lot of CPU and lagging out director. Identifying that driver and fixing an issue with it has completely resolved the issue. I spent a week on the phone with support on that one. Engaging your dealer is the best way to identify if something like that is causing the issue because they're going to have access to deeper troubleshooting tools that you don't.

You could also be having a spectrum issue with zigbee. It's sitting on the 2.4 band. Given that you have 11 APs, hopefully someone has told you or you know to turn off most or all of the 2.4ghz radios. There are only three non-overlapping channels in 2.4, given that there's better reach on 2.4 vs 5ghz, it would be difficult to not have channel overlap 3-4x in that setup. I would only use 5Ghz if possible and potentially only enable 2.4 on three APs at most. If you need more you should do a heatmap of your house and identify which APs have the most RF barrier between them and configure the overlapping channels there, using the attenuation of of your structure to your benefit, adjusting down power as needed. Just the SSID broadcast from two networks (user and guest) could eat something like 40% of the 2.4 ghz bandwidth if replicated on three APs overlapping each other...it would also trash zigbee pretty bad. If someone configured the ubnt gear for 40 or 80 mhz channels on 2.4 you will have similar issues as well. My recomendation would be to disable 2.4 altogether if possible and use strictly non-overlapping channels on 5ghz, I think with 40mhz channels you should be able to just make it.

Speaking of wireless, I would also get someone who is a drop dead expert on the ubiquiti gear or an expert with Airmagnet to look into the wireless, especially if the touchscreens are wireless. 802.11r/k are squirlley in a lot of client and AP chipsets, you may have stability issued on the T3s with roaming given the number of APs you have r/k/v enabled. Band steering can also create issues etc... There are other issues with fast roaming on cheap gear as it generally doesn't generate l2 forwarding table updates on a client roam. Controller based enterprise gear does this, UBNT might, however I don't know. My understanding is that the UBNT controller doesn't really handle traffic so I would expect not. Other solutions like Mist, Aerohive, and Ruckus unleashed have this sorted in a decentralized forwarding plane setup so perhaps UBNT has this as well. I'm also not sure how well autopower/auto channel works on ubnt, generally only Mist, Aruba, and Cisco have this actually working. You should probably use an analyzer to tune down the power on your APs to make a more microcell environment. I would get an expert on UBNT wireless as its very complicated and it's much more about the skeletons in the closet on each platform.

Actually, just looked before submitting and it seems the T3s are 2.4 ghz only. If you're running them on wifi, I would kill all but one of your APs and put a T3 right next to the active AP and see if that stabilizes the T3. If so you probably have channel overlap issues on 2.4ghz.

Good luck, hopefully this helps.

 

[TundraSonic]

An update...

First a big bunch of thanks to @mindedc1 for the thorough overview. It'd be much better of C4 would provide information like this rather than overeact w/ their DNU list.

The problem appears to be excessive CPU churn in the EA-5, possibly from a memory leak in code somewhere. CPU usage occasionally ramps up and when it does then the problems start until it's rebooted. Hopefully more info on this soon.

 

[TundraSonic]

On 1/14/2021 at 8:46 AM, AVNeeds said:

 

Ubiquity is on the DNU list because of its advanced setup. Control4 does not want to become general IT support. 

 If your wife wants to ignore (sorry..."inform") the IT department on how she wants to go about her day, everything you are discussing is ultimately pointless. As @cdepaola stated, people needing this level of security are on completely dedicated and isolated environments, they don't get to just wonder around by the pool on a sunny day. 

If that is indeed the reason for being on the DNU list, I understand their wanting to do that. HOWEVER, it would be much better if they would instead post a note that Ubiquity requires advanced configuration to work properly (which I assume is the same for Cisco and other systems) and then provide detailed notes similar to what @mindedc1 provided but perhaps expanded a bit. 

As to my wife... It is possible to provide her with a secure way to work from places other than her office. It is not necessary to tie her down to her office or even to a segregated system that requires her to have two of everything.