Multiple Yamaha receivers dropping off the network (was intermittent, now constant on 2.8)

[jfh]

 If you have multiple Yamaha receivers running IP drivers, this post should interest you.  I have spent the better part of the last few weeks trying to learn everything I could about this.

This problem will seem familiar to some of you - there was a patch for 2.5.3 that supposedly fixed a problem where a Yamaha receiver would randomly drop off the network if you were using the IP driver when there were multiple Yamaha receivers in the project.

In addition to the C4 patch, newer versions of the Yamaha drivers were required.  Some reported that using a different network port on the receiver instead of the default 50000 solved the problem.

Although there are some threads mentioning specific models of receivers, the actual model doesn't seem to matter.  Others with different models see the same issues I do.

The problems were intermittent until 2.8 and now 2 of the 3 receivers drop off the network at least once a day, sometimes every few hours, but will no discernible pattern.

All my receivers have the latest firmware and the project has the most current versions of the Yamaha drivers (July 2015 seems to be the latest).  My main controller is an HC-800.  Lowering the keep alive threshold on the properties page has no practical impact.

A while back, C4 posted a dealer memo that suggested changing the default network port of the receivers so that all the receivers have a unique port number.  

When this was done with the firmware/drivers at most current levels, this created/exposed another problem for the receivers not using the default of 50000 - in addition to trying communications on the new port (say 50001) which succeeded, it would also try on the default port (which obviously failed).  This caused C4 to see the receiver as off the network, even when it was still connected.  As long as the receiver was on the network, C4 would mark the device back online, but at the default driver settings, the receiver was unavailable to C4 for about 10 out of every 30 seconds.  Commands to the receiver were eventually processed, but resulted in a noticeable lag between when the user did something and it actually happened.  Restoring the network port to the default 50000 stopped issues with one receiver.

In addition, I found info in the logs that seems to expose a logic bug in the driver - when C4 can no longer communicate with the receiver, the driver flags the device as offline but continues to stack/send the keep alive requests required by the receiver.  I think the driver should stop sending the keep alive pings once the network connection drops (goes to false) and restart them when status goes true again.  

The driver seems to flood the receiver with all the "missed" requests once the receiver comes back online, causing the controller to back up.  (Not really sure if this is what's happening, just seems that way).  If you catch this quickly and power cycle the receiver, the queue will clear fairly fast and C4 control resumes.  But, if some time passes (an hour?), the entire system becomes sluggish (thousands of errors in the log) and the main controller needs to be rebooted.

Note that no other devices drop off the network.  There are three different non-managed switches (different makes/models).  If I power down the HC-800, the receivers always show on the network when using a network scanner or the Yamaha application.  When the receivers were Onkyo, Yamaha and Denon, the Yahama never dropped.  The network gear involved hasn't changed since the beginning of the project.

 

The Yamaha receivers:

Living Room RX-A750
   Newest addition.   Has been rock solid.  Only started dropping once default network port was changed.  Problem seems to have gone away once network port changed back to default 50000.  Early pictures of this receiver showed a serial port.  Mine doesn't have one. Driver is very new (July 2015)

 Theater RX-A3030.  
    When this was the only Yamaha in the project, it never dropped.  When the other receivers were added (prior to 2.8), it started dropping from time to time.  After 2.8, drops were far more frequent.   Using the RX-A3030 certified driver or the newer/similar RX-A3050 driver makes no difference in the results.  The newer driver works fine, as long as the receiver is on the network.  This has a serial port, so I plan to have this changed to serial control.

Master Bedroom RX-S600
  No model specific driver.  Was running older RX-A820 driver, now running same driver as LR receiver.  Driver works fine as long as the receiver stays on the network. Does not have a serial port (slim form factor).

 

Problem is almost certainly some combination of issues between the receiver firmware (all of the receivers are close enough where I suspect they use the same network code), the Yamaha IP drivers (ditto network code) and C4 2.8.  

I am using the Chowmain Ping driver to detect when C4 can't communicate with a receiver to alert/take corrective action (the EV Ping driver was too unreliable).  It would be a lot easier to react if the Yamaha driver exposed the network connectivity flag visible on the properties page so an event could be triggered when the state changes.

Control 4 support told my dealer:
   1) Read the bulletin 
   2) This shouldn't be happening.  
   3) Use serial control instead (completely ignoring that it's not an option for 2 of the 3 receivers)
   4) We won't work with the customer unless a dealer is onsite.  (With no indication of what would be done differently than looking at things remotely)

I have asked my dealer to request a better action plan from C4 support to address the root cause.  There is a C4 interface person at Yamaha (who I think I found), but if C4 doesn't drive this I suspect nothing gets fixed.

I know some with the problem have gone to serial control and/or swapped out receivers.  

If there is anyone that has multiple Yamaha receivers using the IP drivers, please post receiver models, drivers/levels, whether you see problems/patterns and what release of C4 you are running.

Any Yamaha and/or C4 dealers - if you can add anything to the above, please do so.  It would seem that both Yamaha and C4 would like to see this fixed.  I'm hoping to get them as much info as I can to try and make that happen.

[Matt Lowe]

i have one A820 at home and it has been rock solid. kind of an odd issue.

if you have just one of three online does there seem to be any issues? when you change the port are you doing this in the driver and on the receiver?

are these setup with DHCP reservations?

Have you tried static ip addresses?

what are you currently doing to fix the issue? reboot yamaha?

[jfh]

If I take two offline, the remaining one does not drop for 6-8 hours (longest I have ever tested).   I had the RX-A3030 as the only receiver for about a year and a half.  Never dropped.  

All three have fixed IP addresses.  Tried DHCP with reservations.  No change.  Went back to fixed IP.

Power cycling the receiver "fixes" the problem (not necessary to unplug the receiver) as the receiver reconnects to the network almost instantly (responds to ping, visible in the Yamaha ap).   Right now, I don't have the MBR receiver plugged into a controllable outlet.  Fortunately, it's the easiest one to access the power switch.    Generally, C4 control is restored right away.  If it has been too long between connectivity loss and power cycle, the HC-800 has to be rebooted.  For the Theater receiver, I am turning the outlet off/on when Ping doesn't see a connection.  I need to pause the room when that happens - right now the sound in the theater just "disappears", which doesn't go over well.

[Matt Lowe]

everything that i can find is pointing to a network or Yamaha related issue. Control4 has released new drivers and the latest i found as about the time you had said. Serial is basically what most everyone with the issue has resorted to. why can you not run serial?

 

if you want to continue trouble shoot disconnect everything from the network except the control4 and a yamaha. then add one component back until it become unstable again

 

[jfh]

Only one of the receivers has a serial port.  I am going to move that one to serial control as soon as I get my cable order.  But using serial doesn't fix the actual problem - it just avoids it.  

Even with moving one to serial control, to get to only one Yamaha receiver I would have to replace a perfectly good receiver with another brand.  I think if I had to tell my wife we had to give up the Yamaha that she really likes for one of the receivers we got rid of in the first place she would be even more upset with C4 than she is already.

I'm not sure what I would learn by disconnecting/reconnecting devices one by one - that isn't likely to show me any new info (and will substantially increase the errors reported by C4 when devices the project expects to be online aren't) and I already know that the only devices impacted are the receivers.  If I take down the HC-800, all three receivers seem to stay on the network without issue.   Sure, it could be some other network issue, but nothing points to anything other than C4/Yamaha.  If C4 came back and said "this only happens with Brand X of router or Brand Y of switch", I'd replace them.   But without any evidence to suspect the network gear, it's reasonable it's (a) Yamaha firmware (b) Yamaha driver (c) C4 OS or some combination.  I'll bet $ on (b).

The driver LUA output shows me when the connection drops, which matches the time reported in Director diags that the connection to the receiver network port is refused.   But I don't have a network sniffer to tell me if the receiver is going offline at the same time or some other event is involved, so I can only see the C4 side of things.

[IonaC4]

I HAD this issue several months ago.  I tried the port changes too with no luck.  I also tried to use MAC address instead of IP, no worky.  I had the issue with 2.7.x and 2.8

(3) A730's

(1) A1010

Knock on wood, I have not had this issue since I made a change outside of the C4 footprint.  I was using an Apple Airport extreme router.  I was told by my dealer that c4 and Apple routers do not play well together.  I changed my router to a commercial grade Luxul router and have not seen the issue since.  I have over 80 ip based devices on my network so it may have been a bit much for the apple router.  I also have two 48 port cisco managed switches in the mix.  Now I am back to the static IP using 50000.

Something to think about...

 

 

[Matt Lowe]

Networking as i suspected is likely the culprit fingers crossed

[jfh]

Open to suggestions on how to (reasonable) determine if there is a network problem unrelated to Yamaha firmware or drivers.

My router is known for being top-notch (ASUS RT-68P).   Handles a pretty large network with lots of traffic fine.  Not a single issue with anything other than the Yamahas and only then if more than one is online.

Given what seems to be a long history of Yamaha receiver networking issues (and Denon and others), I really think that, even if the switch or the router is knocking the receiver off the network, it's because the receiver isn't doing something right or not getting the proper commands.

And this doesn't seem to be load related - I have had receivers drop with almost no load on the network and have had perfect C4 control even when my network is significant amounts of computer data and multiple video and audio streams simultaneously.

The MBR receiver still drops whether the receiver is attached through the MBR switch (Netgear GS308) or direct to the router.  Have also tried Cisco, TrendNet and DLink switches in the loop.  I have tried running the MBR off the same switch as the one receiver that never drops.   Every combination I have tried ended with the same result.  

That's why I'm almost 99.9% sure this is a Yamaha firmware or driver issue.  I'd be happy for someone to prove me wrong if it leads to fixing the problem

 

 

 

[jreinardy]

3 hours ago, Robert Tafaro said:

I was told by my dealer that c4 and Apple routers do not play well together. 

 

I find this a bit funny since my C4 dealer recommended and ultimately installed an Apple Airport Extreme as an upgrade to my older Netgear.  This was done as part my initial C4 install last year.  I have had no issues so far, though I only have one IP controlled receiver, the others use IR flashers.  Having said that, I have no doubt that the Luxul is a more capable device and could have solved some volume related issues.

[mallom]

I don't understand why dealers choose to use a consumer grade $200 router/AP combo instead of a Ubiquiti ERLite 3 + Ubiquiti Unifi AP enterprise grade for the same price. Airport Extreme/Express should NOT be used with automation systems. It can work Ok with a single room project but I would not risk it. 

 

[jfh]

Bizzare problem seen when moving receiver from IP to serial:

just connected one of the problem receivers with serial cable and dealer put in serial driver and deleted the old IP driver from the project and we rebooted the controller.  As expected, everything worked fine with receiver.

However, when using digital audio through the HC-800 (My Music, native Pandora) or Sonos, music would play for between 5 seconds and a minute or so and then the audio stopped.    C4 would think current song was still playing, but if you were in a playlist, everything but the current song just disappeared from the navigator.

Disconnecting the network cable from the receiver, pulling plugs on both receiver and HC-800 seems to have fixed the disappearing audio problem.   Time will tell if there is any positive impact on the other two receivers connected by IP.

 

i feel bad for you dealers that have to try to explain stuff like this to customers who aren't technical ....

[Matt Lowe]

so this happening on control4 native muisic and sonos? have you replaced this router yet? any managed switches in use?

[jfh]

It was happening on both native C4 and Sonos (I think the Sonos is actually connected through the 800 though).   The disappearing audio problem went away when I disconnected the network cable from the receiver that is now connected by serial and hard power cycled both the receiver and the HC-800, so the problem seems to have been caused by some issue related to having a network and serial connection to the receiver at the same time (even though C4 shouldn't have been doing anything via IP any more since the driver had been removed).

I don't have any managed switches in my main (C4) network.  Haven't replaced the router (it is on the C4 recommended list).  I am looking into possibilities just because there has been a lot of improvement in the last few years, but am even more sure now this is a Yamaha driver issue (there is absolutely a logic bug that can result in a flood of command requests from C4 to receiver  that can never be completed)

 

Update on the original multiple receivers using IP:  Might be too soon to tell, but maybe the trigger is "more than 2".  Since the Theater receiver went to serial control, neither of the other receivers has dropped from the network.  

 

[jfh]

It was fun while it lasted - lost connectivity again tonight.   So much for "more than two"- confirmed as more than 1.

[Fling023]

I'm a dealer and we sell exclusively Yamaha receivers (had all sorts of issues with others int the past). I have a job coming up in about 2 weeks where I will be installing 2 RX-A750's in a house in 2 different rooms. I am very curious about this problem. I have had issues with Yamaha receivers falling off of the network before but it has been the controllers issue and nothing else. The connection between the controller and the receiver was UUID. The fix was to make the connection just a static IP. You have to set the static in the receiver, controller, and router (reservation). This has worked for me. I will be interested to see how this works in my upcoming job running 2.8.

[Matt Lowe]

oo i forgot bout that have you made the connections in composer a static ip not the sddp

[jfh]

The receivers are set to static IP (each has a router reservation just in case).

One of the receivers connected now by IP is an RX-A750 has been rock solid on both 2.7 and 2.8 - just whatever you do DO NOT change the default network port on it or the driver will flood Control4 with errors and you will eventually have to reboot the controller to recover.

The other receiver is still dropping off the network (far more than it ever did on 2.72)

The connections are supposed to be IP, as Control4 had recommended to use IP instead of SDDP, but I can't confirm that as I only have HE.  I know my dealer said one was IP only, but will ask him to check the other and change it to IP only if it isn't already and report back.

[Matt Lowe]

yea i dont doubt it is ip driver but the driver has a few options on how to specify which receiver to control. yes you would need pro in order to look if you want i can take a look ill  be at the office for a little longer

[jfh]

Interesting.  My dealer just responded to my email and said C4 support told him the opposite last week - use SDDP only, not IP.

He will change it for me to IP only this week and we'll see if that helps.

[Matt Lowe]

best of luck!

[Fling023]

I hope that takes care of your issue!

[jfh]

5 minutes ago, Fling023 said:

I hope that takes care of your issue!

 

I do too, but I think that was tried before and didn't fix it, so I'm not optimistic this is going to get fixed without a change to the driver and/or receiver firmware.  But there's still hope ...

[Fling023]

I cannot tell you how many service calls I have been on recently to take Yamaha receivers off of SDDP and go with a static IP due to loss of control. It even happened to our demo system within our office. 

I will hold out hope for ya!

[Cyknight]

Lately, I'm staying away from receivers and IP control. Serial never fails. This doesn't seem to be a problem with C4 mind you - 'manufacturer' apps also fail.

We do not use Yamaha regularly (nothing against them - we can simply provide a better deal on others we consider their equal or better) but I had one recently. After it completely failing on SDDP and acting inconsistent on IP I simply used a serial connection (same driver has all options) and it's been flawless.

[jfh]

Maybe there's an underlying problem in SDDP then, who knows?

I changed the one receiver that had a serial port to serial control and would gladly switch the other two, but they don't have serial ports.  The MBR receiver needs a small (height) form factor and I'm not aware of any mfg that has one with a serial port.   

If C4 is going to certify the drivers - and not have a caveat you can't use more than one in a project - then they have a responsibility to Make sure they work as expected.  A work around is fine in many cases, but not addressing the root cause of a problem leads to others.

I appreciate the comments here, but am really disappointed at the cavalier response from C4 support and seeming lack of interest they have in addressing this. They won't talk to me and apparently aren't going to do anything for my dealer (who has no experience with Yamaha) unless he is on site.  What the heck is going to be different on site other than "Yup, it's happening exactly as the customer says"?   When I was in tech support, if I had ever dismissed the concerns of a customer the way C4 is, I would have been fired ...