Jump to content
C4 Forums | Control4

Network has gone down twice in one week


pinkoos

Recommended Posts


28 minutes ago, LollerAgent said:

Please stop double-posting.  It's making it difficult for the people who are trying to help you.

I effectively ended the camera/POE injector thread and referred back to this thread once we figured out it's NOT a camera/POE injector issue

Link to comment
Share on other sites

My dealer dumped the logs after the last outage (yesterday) and got on a long call with Araknis

They think they found the problem...the SecuritySpy Mac NVR software I had installed back on September 4

Since I don't quite understand everything my dealer is saying, I'll just quote him in this post:

"Every time you see NSD FAIL WAN[1] in the logs, it denotes Router is not able to handle the traffic that’s originating from outside the house to talk to devices in the house. And every time that has occurred, we have noticed network outage.

What’s causing this flooding or incoming request from outside? These are 2 lines extracted from the Activity log on the Router.

Oct 1 15:15:09 2019 Kernel  upnpd[538]: AddPortMapping: RemoteHost:  Protocol: TCP ExternalPort: 8001 InternalClient: 192.168.1.175.8001

Oct 1 15:15:09 2019 Kernel  upnpd[538]: AddPortMapping: RemoteHost:  Protocol: TCP ExternalPort: 8000 InternalClient: 192.168.1.175.8000

Upon further checking, we found that the IP address 192.168.1.175 belong to the Mac. So, there are services running on the Mac that are being continuously called from the internet to establish a connection on ports 8000 and 8001 (sounds to me ports usually used by cameras). Ideally, we would require ports to be manually opened on the firewall to allow this incoming traffic. Like I mentioned in the email below, UPNP service on the router allows this to happen automatically. The idea is to allow user authorized devices on the network to establish its own communication path from the outside world.

But somehow the app / service on the Mac is not well designed and what is happening is the same set of ports are being asked to be opened on the router repeatedly. I see requests coming in every 60 seconds for the same port that’s already opened and mapped to the internal client on IP .175.

This flooding of request from external clients on it’s WAN port is causing issues with real-time network service detection (checks that are made to see if internet is available on WAN port) and it flags is as Failed NSD on WAN Port, causing the router to cease its function of passing internet service over its WAN port (from our Motorola Modem) and that requires i). a restart or ii). Disconnect of all clients on the internal network side, which happens when we restart the core switch."

His suggestion:

"Once we know the app and the ports it needs, we can assign a static IP (outside of the DHCP range) and manually open ports on the firewall. But that will keep us out of this issue only if the App / Service has an option to turn off UPNP. If the App/ Service still utilizes UPNP, then the service will continue to request for the same port mappings and cause the WAP port to go down again.

Other option is to turn off UPNP on the router but that will cause all products, (nest doorbell, arlo cameras, etc.) that rely on ports mappings automatically being available to them across the router thru UPNP, to stop working from their apps."

I just now received a reply from the NVR software developer and this is what he said:

"This feature is controlled by the "Allow access from the Internet" settings under Preferences -> Web. If you turn these options off, SecuritySpy will not issue UPnP packets. I'm surprised these packets are causing problems - SecuritySpy's UPnP implementation is fairly standard, and we have not have any reports from other users of any similar problems. There is no reason why UPnP packets should cause this. Please report back and let me know if turning off UPnP does indeed resolve this problem - if so, I would have to conclude that your router has a buggy UPnP implementation."

Link to comment
Share on other sites

2 minutes ago, msgreenf said:

so did u disable that in security spy?

Yes, I remoted in to my Mac and disabled that and then tested remote connectivity and everything still works, so we may not need to open any ports on the router after all

However, my dealer is going to reboot the router b/c he wants to make sure that the port mapping from previous UPNP requests persists all restarts

Link to comment
Share on other sites

Ugh, network just went down again even though I turned off automatic port forwarding (though we never had to set anything up manually after that as the remote server was still working)

Next step is to keep the NVR software turned off (ie, not running) and unplug the POE injector so that those two things are completely taken out of the equation and see what happens over the next couple of days

Link to comment
Share on other sites

One thing I noticed this morning when it went down was that the GUI on the Apple TV homescreen got very laggy and jerky

Swiping through icons with the Apple TV remote was very slow and irregular

Not sure how/why the GUI would be affected when the network is down

Link to comment
Share on other sites

My dealer sent me the router logs and also came up with a list of all the devices using UPNP and that got me thinking whether the Plex Media Server I'm running on my Mac could be the culprit - it's just a guess, I don't really have anything to base that on but then I started searching online and there are some (generally old) discussions of Plex bringing down people's networks/routers.  There's a lot of technical talk in these threads which I don't follow, but I've passed the info on to my dealer to see what he thinks

Maybe a recent PMS update has a bug or something that is causing this?  I've been running the PMS since probably last October or November with no issues, so it's got to be something with a recent update (if it is Plex)

Here's the logs right before the crash this morning:

Oct 4 10:56:18 2019  System Log  HTTP Basic authentication success for user: araknis 
Oct 4 07:17:37 2019 Kernel  upnpd[538]: DeletePortMap: Remote Host:  Proto:TCP Port:21376  
Oct 4 07:17:17 2019 Kernel  upnpd[538]: AddPortMapping: RemoteHost:  Protocol: TCP ExternalPort: 21376 InternalClient: 192.168.1.175.5900 
Oct 4 06:56:59 2019 Kernel  upnpd[538]: AddAnyPortMapping: RemoteHost:  Protocol: TCP ExternalPort: 1029 InternalClient: 192.168.1.175.32400 
Oct 4 06:56:02 2019  System Log   NSD SUCCESS WAN[1] 
Oct 4 06:55:37 2019  System Log  OVRC connect to server success!  
Oct 4 06:55:32 2019  System Log   NSD FAIL WAN[1] 
Oct 4 06:52:23 2019  System Log   NSD SUCCESS WAN[1] 
Oct 4 06:49:05 2019  System Log  OVRC connection disconnect!  
Oct 4 06:41:22 2019  System Log   NSD FAIL WAN[1] 
Oct 4 06:32:52 2019 Kernel  upnpd[538]: AddAnyPortMapping: RemoteHost:  Protocol: TCP ExternalPort: 1029 InternalClient: 192.168.1.175.32400 
Oct 4 06:17:17 2019 Kernel  upnpd[538]: AddPortMapping: RemoteHost:  Protocol: TCP ExternalPort: 21376 InternalClient: 192.168.1.175.5900 
Oct 4 05:32:52 2019 Kernel  upnpd[538]: AddAnyPortMapping: RemoteHost:  Protocol: TCP ExternalPort: 1029 InternalClient: 192.168.1.175.32400 

And here's the list my dealer came up with (the first two are not UPNP).  .175 is my Mac, but we don't know what .177 is quite yet.  I think .101 must be my son's PC - he recently installed Minecraft.  I have no idea what all those "icXXXXX" are

80	8080	192.168.1.201	Both	NVR1
80	8081	192.168.1.210	Both	Camera 1
32400	1032	192.168.1.175	TCP	Plex_Media_Server
5900	28296	192.168.1.175	TCP	iC28296
5900	28297	192.168.1.177	TCP	iC28297
32400	1024	192.168.1.177	TCP	Plex_Media_Server
32400	1025	192.168.1.175	TCP	Plex_Media_Server
5900	28298	192.168.1.175	TCP	iC28298
5900	28299	192.168.1.177	TCP	iC28299
32400	1026	192.168.1.177	TCP	Plex_Media_Server
5900	28300	192.168.1.175	TCP	iC28300
5900	28301	192.168.1.177	TCP	iC28301
5900	28302	192.168.1.175	TCP	iC28302
52293	52293	192.168.1.101	UDP	Minecraft
52104	52104	192.168.1.101	UDP	Minecraft
5900	25942	192.168.1.175	TCP	iC25942
5900	23111	192.168.1.175	TCP	iC23111
5900	22555	192.168.1.175	TCP	iC22555
5900	21756	192.168.1.175	TCP	iC21756
8000	8000	192.168.1.175	TCP	SecuritySpy-8000
8001	8001	192.168.1.175	TCP	SecuritySpy-8001
5900	22514	192.168.1.175	TCP	iC22514
6881	6881	192.168.1.175	TCP	libtorrent/1.2.1.0
6881	6881	192.168.1.175	UDP	libtorrent/1.2.1.0
32400	1029	192.168.1.175	TCP	Plex_Media_Server
Link to comment
Share on other sites

2 hours ago, Sukmano said:

ICxxxx port forwarding for External access to your PMS enabled via uPnP

Thank you, I can pass this info on to my dealer

I'm curious, why are there so many instances of PMS?

I run PMS on my Mac but then I thought all other devices are just accessing that - these devices include 4 Apple TVs, 4 iPhones and 1 iPad 

And, I also access the PMS on my Mac itself to playback on the Mac

Link to comment
Share on other sites

4 minutes ago, msgreenf said:

There aren't so many instances. It just opens many ports

Ok

I counted from that list my dealer gave me and there's like 17 ports being opened by Plex (things that actually say Plex and the icXXXX ports)

Is that normal?  Could that possibly be the problem?

And, I still don't understand why when the network crashes, it's a reboot of the switch that brings it back up and not a reboot of the router

 

Link to comment
Share on other sites

Ok
I counted from that list my dealer gave me and there's like 17 ports being opened by Plex (things that actually say Plex and the icXXXX ports)
Is that normal?  Could that possibly be the problem?
And, I still don't understand why when the network crashes, it's a reboot of the switch that brings it back up and not a reboot of the router
 
Pretty normal yes
Link to comment
Share on other sites

10 minutes ago, msgreenf said:
16 minutes ago, pinkoos said:
Basically only rebooting the switch will  get the network back up

You are overrunning the switch w traffic. The switch could be failing

So if we swap out the switch but keep the same amount of traffic going through, would this possibly just happen again?

I'm trying to read through some discussions on how to change some settings within PMS regarding UPNP, port forwarding, etc, but I don't understand how to do all that stuff, hopefully my dealer can help with that

But, I'm just trying to figure out (if Plex is the problem) whether we're better off trying to change the PMS settings rather than swap out the switch if the underlying problem is too much traffic

Or, do I need a larger switch?  This one is 24 ports but they're not all being used

Link to comment
Share on other sites

This thread is quite old. Please consider starting a new thread rather than reviving this one.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...

Important Information

By using this site, you agree to our Terms of Use.