Sign in to follow this  
Followers 0
digitaltrader

System slow and Zigbee Issues

33 posts in this topic

I have been experiencing some issues with system slowness/stability for a little while that have been getting worse over the last few weeks. I have noticed that after a reboot of the controller that sometimes the zigbee mesh fails to come back online and that it sometimes requires a second reboot a half an hour later or so for the mesh to come back online.

Some symptoms I have been experiencing:

1) System Lag: I have been experiencing a problem recently where I am experiencing a system pause/lag. I will be changing channels on the TV and then the system (HC-800) director will stop responding to my SR250 remote commands for 10 seconds and then execute all delayed commands in a sudden burst. This occurs almost every 2 minutes or so.

In order to further diagnose the problem I have attempted to use a 7" TS connected via cat5, a wifi iPhone device and a zigbee 6 button keypad at separate times during this 10 second pause to see if the system would respond to their commands. All devices fail to respond during the lag.

2) Zigbee mesh fails to reload after a controller reboot: When I reboot my controller (HC-800 in the UEC) that is running both director and zigbee server it often fails to bring the zigbee mesh back online. Sometimes a second reboot of the controller will bring the mesh back online. I have used PuTTY to log into my control and look at the CPU load and have noticed that sometimes the zserver2 process is running at nearly 25% of my CPU load with sometimes director using 25% as well when the system is at idle. The Zserver2 process also will drop back down to 1% but often spikes to 25% when the system is idle.

I have my dealer looking into this but would love to hear any suggestions as well.

Share this post


Link to post
Share on other sites

Couple of quick questions for added info.

a) When you use a device directly controlled by the 800 (using it's own serial or IR ports) using a 250 remote, are these delayed as well.

B) What's the general network set-up (is everything home-run to a single big switch, is it several localized switches, that sort of thing) and what is/are your main switch/router

c) Is the 800 the ONLY ZAP or are there more ZAPS or servers. And is the 800 the only controller on the system.

Share this post


Link to post
Share on other sites
Couple of quick questions for added info.

a) When you use a device directly controlled by the 800 (using it's own serial or IR ports) using a 250 remote, are these delayed as well.

B) What's the general network set-up (is everything home-run to a single big switch, is it several localized switches, that sort of thing) and what is/are your main switch/router

c) Is the 800 the ONLY ZAP or are there more ZAPS or servers. And is the 800 the only controller on the system.

Thanks for the response.

a) There are no IR devices connected directly to the HC-800 Director only my Rhapsody audio as well as an RS232 for an HDMI matrix switch. Most of my devices IR are run off of a C4 I/O extender connected to the same switch as the HC800 or a HC200 connected to the same switch as well

B) The network is a Cisco 2821 Business Grade router, Dell PowerConnect 6248P 48 port POE switch that feeds several other rooms into Dell 16 port unmanaged PowerConnect switches.

c) The HC800 is running director and Zigbee server. There are no additional ZAPS but one additional Zigbee server located in my outdoor area run on a HC300C. I have a Theater and a Great Room that each have their own HC800 as well that run a localized navigator as well as the IR/RS232 devices for those zones.

Edited by digitaltrader

Share this post


Link to post
Share on other sites

Log into the system via SSH and run a top or run system diagnostics and look at cpu and memory usage.

Top will tell you what's bogging the system down, if anything.

Composer Pro and a little bit of *nix knowledge can help solve your issue, or at least locate the problem.

If your dealer doesn't know how to ssh into the controller and monitor top and system logs, make sure that he engages C4 for support.

Share this post


Link to post
Share on other sites
Log into the system via SSH and run a top or run system diagnostics and look at cpu and memory usage.

Top will tell you what's bogging the system down, if anything.

Composer Pro and a little bit of *nix knowledge can help solve your issue, or at least locate the problem.

If your dealer doesn't know how to ssh into the controller and monitor top and system logs, make sure that he engages C4 for support.

Actually I saw this suggestion on a previous post of yours and used PuTTY to SSH into the controller and Director was running at 25% when the system was at idle and I saw occasional Zserver2 spikes to 25% as well but all CPU processes were around 50%. After a reboot the CPU is under 10% but things eventually get much closer to 50% after a few days.

Share this post


Link to post
Share on other sites
a) There are no IR devices connected directly to the HC-800 Director only my Rhapsody audio as well as an RS232 for an HDMI matrix switch. Most of my devices IR are run off of a C4 I/O extender connected to the same switch as the HC800 or a HC200 connected to the same switch as well

B) The network is a Cisco 2821 Business Grade router, Dell PowerConnect 6248P 48 port POE switch that feeds several other rooms into Dell 16 port unmanaged PowerConnect switches.

c) The HC800 is running director and Zigbee server. There are no additional ZAPS but one additional Zigbee server located in my outdoor area run on a HC300C. I have a Theater and a Great Room that each have their own HC800 as well that run a localized navigator as well as the IR/RS232 devices for those zones.

Ok you pretty much eliminate network issues, but just to be sure : is that matrix switching as normal while the rest is in lag (use the remote, not a TS), did you apply any recent firmware updates on the router?

Sounds a lot like the root here is the 800 itself (hardware or software hard to say). But ....

The only other thing I can think of that may be causing an issue is either a programming loop or a 2-way communication device flooding the controller with info. Although much worse then you are describing I recently had a 2-button transmit 200 commands per second - every single second - to an 800. This caused wild delays, CPU spikes etc - much like you, except this one was far far worse.

Share this post


Link to post
Share on other sites

Well after alot of monitoring and help from my dealer we are still not sure as to what may be causing the problem. I have noticed that after about 2 days the Director CPU usage (HC-800) goes from about 2% to 26% and that is when the system starts to fail. C4 tech support looked at the logs and wasn't certain but thought the EV mymovies or Recluse email/pop3 drivers may be at fault which were added at the time I started experiencing troubles. I was hoping to post the logs and get any advice/opinions. Fortunately I have a great dealer who is persistent and is not giving up.

Share this post


Link to post
Share on other sites

Take out both of the drivers, reboot the controller, and wait a few days to see if the drivers are in fact causing the problem. After a few days if you don't have any problems, add one driver only, wait a few more days and monitor CPU usage. I know it's kind of a pain and time consuming but that way you should be able to narrow down which driver is causing the problem, if it is a driver.

Share this post


Link to post
Share on other sites

Your cpu load is way over 1 in top, which doesn't jive with the 26% cpu usage of director.

Something else is going on. Have you port forwarded your 800 to the internet?

I too have the mymovies log entries, but those debug messages are most likely not the issue, but to be safe, it can't hurt to start removing drivers and see where the issue stops.

What version are you on?

Share this post


Link to post
Share on other sites
Your cpu load is way over 1 in top, which doesn't jive with the 26% cpu usage of director.

Something else is going on. Have you port forwarded your 800 to the internet?

I too have the mymovies log entries, but those debug messages are most likely not the issue, but to be safe, it can't hurt to start removing drivers and see where the issue stops.

What version are you on?

Thanks for looking. My HC800 is not port forwarded to the internet, as I use a VPN. I loaded a version of my project without the Recluse pop3 driver and CPU usage is down to 2%. Ill let it run a couple of days and report back.

Thanks!

Share this post


Link to post
Share on other sites

I have the same issues on my 1000V3. No response then a burst of button pushes so I'd say it's very possibly the new firmware and/or the MyMovies Driver which were added at the same time? Also, noticing camera weirdness with no pictures appearing sometimes and being unresponsive. Then it's fine the next time I look at the camera's???

Edited by qVAMPIREp

Share this post


Link to post
Share on other sites

I was on the phone with a level 2 tech for quite some time about a similar issue on my HC800 yesterday, and he did say there was a bug with some extra vegetable drivers and Control4 that causes some weird zigbee issues and that would be fixed in 2.3. The fix, he said, would be to wait till 2.3 comes out and then update the EV driver when it is available. Which should be shortly.

Share this post


Link to post
Share on other sites
I was on the phone with a level 2 tech for quite some time about a similar issue on my HC800 yesterday, and he did say there was a bug with some extra vegetable drivers and Control4 that causes some weird zigbee issues and that would be fixed in 2.3. The fix, he said, would be to wait till 2.3 comes out and then update the EV driver when it is available. Which should be shortly.

Well I loaded up my project without the Recluse pop3 driver 1st and saw an immediate difference in the continued CP load but with still regular CPU spike to 25% occurring every 1 minute lasting 10-12 seconds. I retested without the EV mymovies driver and the regular CPU spike at 1 minute intervals disappeared as well as the issue causing my zigbee remote commands to execute in bursts. So in my case it looks like both driver share some blame.

The big question is does 2.3 solve these issues, and when is the release date?

Share this post


Link to post
Share on other sites

What happens when you readd our driver? My house is running the driver on gmail on a hc300 without exhibiting the same issue.

Share this post


Link to post
Share on other sites

Standard answer on 2.3. When it's ready. I too have both Alan's smtp driver and mymovies. Will see if I am able to reproduce.

Share this post


Link to post
Share on other sites
What happens when you readd our driver? My house is running the driver on gmail on a hc300 without exhibiting the same issue.

Its been over 24 hours and my CPU usage is still 1% and load is still running low. I will try having my dealer add the pop3 driver back in to see if it has any affect.

Share this post


Link to post
Share on other sites
Your cpu load is way over 1 in top, which doesn't jive with the 26% cpu usage of director.

The hc800 has a quad core processor. 26% CPU and a load avg of >1 jive completely. If director is single threaded, i think it is, and you have a quad core processor, you can only use up to 25% on a single process. I would guess 26% is just a rounding quirk from 25%.

if director is at 25% and the system is slowing down, then director is probably maxing out a core on a bug or programming loop. sounds like a bug, since programming loops usually immediately peg director.

Share this post


Link to post
Share on other sites

fwiw I've noticed similar issues and the CPU usage for director dropped from 25% immediately to 1% after deleting the POP3 driver and has been hovering around low single digits.

Share this post


Link to post
Share on other sites
Your cpu load is way over 1 in top' date=' which doesn't jive with the 26% cpu usage of director.[/quote']

The hc800 has a quad core processor. 26% CPU and a load avg of >1 jive completely. If director is single threaded, i think it is, and you have a quad core processor, you can only use up to 25% on a single process. I would guess 26% is just a rounding quirk from 25%.

if director is at 25% and the system is slowing down, then director is probably maxing out a core on a bug or programming loop. sounds like a bug, since programming loops usually immediately peg director.

"Jive"?? Suggest getting the Control4 Websters Dictionary app :)

Share this post


Link to post
Share on other sites
What happens when you readd our driver? My house is running the driver on gmail on a hc300 without exhibiting the same issue.

Alan, I had my dealer re-add the driver about 24 hours ago and I went from a 1-2% average CPU use to slowly ramping up to a steady 13% CPU use where it now sits steadily. The MyMovies EV driver has not been re-added. It looks like the pop3 definitely has some sort of memory leak/CPU drag. I will leave it running to determine if it gets right back to the 25% CPU use over the next 24 hours.

Share this post


Link to post
Share on other sites

Strange ill see if I can replicate it on my test bench. The only thing I can think of is the check in interval. If it is set to the minimal of 15 seconds it would probably be working all the time. Mine at my house is set to a minute.

Share this post


Link to post
Share on other sites
Strange ill see if I can replicate it on my test bench. The only thing I can think of is the check in interval. If it is set to the minimal of 15 seconds it would probably be working all the time. Mine at my house is set to a minute.

It was set to 15 seconds. When set to 2 minutes the load drops from 1 to 0.6 but CPU remains at 15%.

Share this post


Link to post
Share on other sites
Strange ill see if I can replicate it on my test bench. The only thing I can think of is the check in interval. If it is set to the minimal of 15 seconds it would probably be working all the time. Mine at my house is set to a minute.

Alan, the issue seems to be exacerbated by the 15 seconds check in but still occurs with 30 second check in. Also the version that was reinstalled on my system is the one that doesn't work with gmail.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0
    • 5 Posts
    • 83 Views
    • 7 Posts
    • 208 Views
    • cgl
    • cgl
    • 9 Posts
    • 164 Views
    • 1 Posts
    • 94 Views
    • 15 Posts
    • 342 Views
    • 3 Posts
    • 61 Views
    • 3 Posts
    • 78 Views