Jump to content

  • Log in with Facebook Log in with Twitter Log In with Google      Sign In   
  • Create Account

Welcome to the New c4Forums.com!

We will be enhancing the platform daily now that we are officially online. You can read and discuss the new platform here or for access issues please email c4forums@c4forums.com. Thanks!


Photo

System slow and Zigbee Issues


  • Please log in to reply
32 replies to this topic

#1 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 01 November 2012 - 03:37 PM

I have been experiencing some issues with system slowness/stability for a little while that have been getting worse over the last few weeks. I have noticed that after a reboot of the controller that sometimes the zigbee mesh fails to come back online and that it sometimes requires a second reboot a half an hour later or so for the mesh to come back online.

Some symptoms I have been experiencing:

1) System Lag: I have been experiencing a problem recently where I am experiencing a system pause/lag. I will be changing channels on the TV and then the system (HC-800) director will stop responding to my SR250 remote commands for 10 seconds and then execute all delayed commands in a sudden burst. This occurs almost every 2 minutes or so.
In order to further diagnose the problem I have attempted to use a 7" TS connected via cat5, a wifi iPhone device and a zigbee 6 button keypad at separate times during this 10 second pause to see if the system would respond to their commands. All devices fail to respond during the lag.

2) Zigbee mesh fails to reload after a controller reboot: When I reboot my controller (HC-800 in the UEC) that is running both director and zigbee server it often fails to bring the zigbee mesh back online. Sometimes a second reboot of the controller will bring the mesh back online. I have used PuTTY to log into my control and look at the CPU load and have noticed that sometimes the zserver2 process is running at nearly 25% of my CPU load with sometimes director using 25% as well when the system is at idle. The Zserver2 process also will drop back down to 1% but often spikes to 25% when the system is idle.

I have my dealer looking into this but would love to hear any suggestions as well.

#2 Cyknight

Cyknight

    C4 Guru

  • Control4 Dealer
  • 3,611 posts

Posted 01 November 2012 - 03:51 PM

Couple of quick questions for added info.
a) When you use a device directly controlled by the 800 (using it's own serial or IR ports) using a 250 remote, are these delayed as well.
B) What's the general network set-up (is everything home-run to a single big switch, is it several localized switches, that sort of thing) and what is/are your main switch/router
c) Is the 800 the ONLY ZAP or are there more ZAPS or servers. And is the 800 the only controller on the system.
Vae Victis

#3 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 01 November 2012 - 04:09 PM

Couple of quick questions for added info.
a) When you use a device directly controlled by the 800 (using it's own serial or IR ports) using a 250 remote, are these delayed as well.
B) What's the general network set-up (is everything home-run to a single big switch, is it several localized switches, that sort of thing) and what is/are your main switch/router
c) Is the 800 the ONLY ZAP or are there more ZAPS or servers. And is the 800 the only controller on the system.

Thanks for the response.

a) There are no IR devices connected directly to the HC-800 Director only my Rhapsody audio as well as an RS232 for an HDMI matrix switch. Most of my devices IR are run off of a C4 I/O extender connected to the same switch as the HC800 or a HC200 connected to the same switch as well

B) The network is a Cisco 2821 Business Grade router, Dell PowerConnect 6248P 48 port POE switch that feeds several other rooms into Dell 16 port unmanaged PowerConnect switches.

c) The HC800 is running director and Zigbee server. There are no additional ZAPS but one additional Zigbee server located in my outdoor area run on a HC300C. I have a Theater and a Great Room that each have their own HC800 as well that run a localized navigator as well as the IR/RS232 devices for those zones.

Edited by digitaltrader, 01 November 2012 - 04:10 PM.


#4 pstuart

pstuart

    C4 Guru

  • c4Forums Member
  • 911 posts

Posted 01 November 2012 - 04:11 PM

Log into the system via SSH and run a top or run system diagnostics and look at cpu and memory usage.

Top will tell you what's bogging the system down, if anything.

Composer Pro and a little bit of *nix knowledge can help solve your issue, or at least locate the problem.

If your dealer doesn't know how to ssh into the controller and monitor top and system logs, make sure that he engages C4 for support.

#5 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 01 November 2012 - 04:22 PM

Log into the system via SSH and run a top or run system diagnostics and look at cpu and memory usage.

Top will tell you what's bogging the system down, if anything.

Composer Pro and a little bit of *nix knowledge can help solve your issue, or at least locate the problem.

If your dealer doesn't know how to ssh into the controller and monitor top and system logs, make sure that he engages C4 for support.

Actually I saw this suggestion on a previous post of yours and used PuTTY to SSH into the controller and Director was running at 25% when the system was at idle and I saw occasional Zserver2 spikes to 25% as well but all CPU processes were around 50%. After a reboot the CPU is under 10% but things eventually get much closer to 50% after a few days.

#6 Cyknight

Cyknight

    C4 Guru

  • Control4 Dealer
  • 3,611 posts

Posted 01 November 2012 - 04:55 PM

a) There are no IR devices connected directly to the HC-800 Director only my Rhapsody audio as well as an RS232 for an HDMI matrix switch. Most of my devices IR are run off of a C4 I/O extender connected to the same switch as the HC800 or a HC200 connected to the same switch as well

B) The network is a Cisco 2821 Business Grade router, Dell PowerConnect 6248P 48 port POE switch that feeds several other rooms into Dell 16 port unmanaged PowerConnect switches.

c) The HC800 is running director and Zigbee server. There are no additional ZAPS but one additional Zigbee server located in my outdoor area run on a HC300C. I have a Theater and a Great Room that each have their own HC800 as well that run a localized navigator as well as the IR/RS232 devices for those zones.

Ok you pretty much eliminate network issues, but just to be sure : is that matrix switching as normal while the rest is in lag (use the remote, not a TS), did you apply any recent firmware updates on the router?

Sounds a lot like the root here is the 800 itself (hardware or software hard to say). But ....

The only other thing I can think of that may be causing an issue is either a programming loop or a 2-way communication device flooding the controller with info. Although much worse then you are describing I recently had a 2-button transmit 200 commands per second - every single second - to an 800. This caused wild delays, CPU spikes etc - much like you, except this one was far far worse.
Vae Victis

#7 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 11 November 2012 - 10:30 PM

Well after alot of monitoring and help from my dealer we are still not sure as to what may be causing the problem. I have noticed that after about 2 days the Director CPU usage (HC-800) goes from about 2% to 26% and that is when the system starts to fail. C4 tech support looked at the logs and wasn't certain but thought the EV mymovies or Recluse email/pop3 drivers may be at fault which were added at the time I started experiencing troubles. I was hoping to post the logs and get any advice/opinions. Fortunately I have a great dealer who is persistent and is not giving up.

#8 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 11 November 2012 - 10:31 PM

Here is a PuTTY screen shot showing the CPU usage

#9 xc420

xc420

    C4 Guru

  • c4Forums Member
  • 658 posts
  • LocationHouston, TX

Posted 13 November 2012 - 01:53 AM

Take out both of the drivers, reboot the controller, and wait a few days to see if the drivers are in fact causing the problem. After a few days if you don't have any problems, add one driver only, wait a few more days and monitor CPU usage. I know it's kind of a pain and time consuming but that way you should be able to narrow down which driver is causing the problem, if it is a driver.

#10 pstuart

pstuart

    C4 Guru

  • c4Forums Member
  • 911 posts

Posted 13 November 2012 - 09:18 AM

Your cpu load is way over 1 in top, which doesn't jive with the 26% cpu usage of director.

Something else is going on. Have you port forwarded your 800 to the internet?

I too have the mymovies log entries, but those debug messages are most likely not the issue, but to be safe, it can't hurt to start removing drivers and see where the issue stops.

What version are you on?

#11 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 13 November 2012 - 12:10 PM

Your cpu load is way over 1 in top, which doesn't jive with the 26% cpu usage of director.

Something else is going on. Have you port forwarded your 800 to the internet?

I too have the mymovies log entries, but those debug messages are most likely not the issue, but to be safe, it can't hurt to start removing drivers and see where the issue stops.

What version are you on?

Thanks for looking. My HC800 is not port forwarded to the internet, as I use a VPN. I loaded a version of my project without the Recluse pop3 driver and CPU usage is down to 2%. Ill let it run a couple of days and report back.

Thanks!

#12 qVAMPIREp

qVAMPIREp

    C4 Guru

  • c4Forums Member
  • 639 posts

Posted 13 November 2012 - 02:41 PM

I have the same issues on my 1000V3. No response then a burst of button pushes so I'd say it's very possibly the new firmware and/or the MyMovies Driver which were added at the same time? Also, noticing camera weirdness with no pictures appearing sometimes and being unresponsive. Then it's fine the next time I look at the camera's???

Edited by qVAMPIREp, 13 November 2012 - 02:41 PM.


#13 LSDave

LSDave

    C4 Guru

  • c4Forums Member
  • 480 posts
  • LocationWinnipeg

Posted 13 November 2012 - 04:10 PM

I was on the phone with a level 2 tech for quite some time about a similar issue on my HC800 yesterday, and he did say there was a bug with some extra vegetable drivers and Control4 that causes some weird zigbee issues and that would be fixed in 2.3. The fix, he said, would be to wait till 2.3 comes out and then update the EV driver when it is available. Which should be shortly.

#14 qVAMPIREp

qVAMPIREp

    C4 Guru

  • c4Forums Member
  • 639 posts

Posted 13 November 2012 - 04:13 PM

^ Thanks for that LSDave :)

#15 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 14 November 2012 - 01:13 AM

I was on the phone with a level 2 tech for quite some time about a similar issue on my HC800 yesterday, and he did say there was a bug with some extra vegetable drivers and Control4 that causes some weird zigbee issues and that would be fixed in 2.3. The fix, he said, would be to wait till 2.3 comes out and then update the EV driver when it is available. Which should be shortly.

Well I loaded up my project without the Recluse pop3 driver 1st and saw an immediate difference in the continued CP load but with still regular CPU spike to 25% occurring every 1 minute lasting 10-12 seconds. I retested without the EV mymovies driver and the regular CPU spike at 1 minute intervals disappeared as well as the issue causing my zigbee remote commands to execute in bursts. So in my case it looks like both driver share some blame.

The big question is does 2.3 solve these issues, and when is the release date?

#16 alanchow

alanchow

    Driverworks Nut

  • Control4 Dealer
  • 1,325 posts
  • LocationMelbourne, Victoria, Australia

Posted 14 November 2012 - 02:56 AM

What happens when you readd our driver? My house is running the driver on gmail on a hc300 without exhibiting the same issue.

Alan Chow | Co-Founder @ Chowmain | Twitter | LinkedIn

 

Chowmain%20Logo%20Transparent%20Medium.p


#17 pstuart

pstuart

    C4 Guru

  • c4Forums Member
  • 911 posts

Posted 14 November 2012 - 07:46 AM

Standard answer on 2.3. When it's ready. I too have both Alan's smtp driver and mymovies. Will see if I am able to reproduce.

#18 digitaltrader

digitaltrader

    C4 Guru

  • c4Forums Member
  • 672 posts

Posted 14 November 2012 - 12:14 PM

What happens when you readd our driver? My house is running the driver on gmail on a hc300 without exhibiting the same issue.

Its been over 24 hours and my CPU usage is still 1% and load is still running low. I will try having my dealer add the pop3 driver back in to see if it has any affect.

#19 thegreatheed

thegreatheed

    End User

  • c4Forums Member
  • 144 posts

Posted 14 November 2012 - 12:20 PM

Your cpu load is way over 1 in top, which doesn't jive with the 26% cpu usage of director.

The hc800 has a quad core processor. 26% CPU and a load avg of >1 jive completely. If director is single threaded, i think it is, and you have a quad core processor, you can only use up to 25% on a single process. I would guess 26% is just a rounding quirk from 25%.

if director is at 25% and the system is slowing down, then director is probably maxing out a core on a bug or programming loop. sounds like a bug, since programming loops usually immediately peg director.

#20 steve909

steve909

    End User

  • c4Forums Member
  • 25 posts

Posted 14 November 2012 - 11:26 PM

fwiw I've noticed similar issues and the CPU usage for director dropped from 25% immediately to 1% after deleting the POP3 driver and has been hovering around low single digits.




0 user(s) are reading this topic

0 members, 0 guests, 0 anonymous users