Trying to find what is messing up my network (not directly C4 related)

[eggzlot]

So it has been a very annoying week with my network

2 switches in 1 rack (these are both plugged into a  Panamax SM3 and that is plugged into an APC UPS)

1 switch in another rack

Last Wednesday: woke up to notice some items not working, after close inspection, saw 1 bank (12 ports) of ports on Switch 1 (non Poe) out.  Moved devices in those ports to another switch near by (Switch 2, happens to be PoE).  Cisco replaced Switch 1 with an RMA - lovely.

Installed new switch Tuesday, moved things back into place.

This Wednesday - at 3:30 my network seems to go down.  after troubles hooting, it is ANYTHING plugged into either switch, but the switches had full power, all the green/orange lights blinking, but nothing was on the network.  we did notice a few sensors associated with an EA1 not working, and that EA1 is also in the SM3. Rebooted the switches and the EA1 via the SM3 and things seemed to work again

Today (Friday), I go to use a touch screen and it just is stuck on "connecting".  test out the other touch screens, they are down too as are a lot of devices (PoE and non PoE).  I noticed all the devices down (like my Blue Iris PC) are all in the PoE Switch (Switch 2)  They had the right date/time on the screen saver, but couldnt connect to C4.  reboot C4 EA5, nothing.  switches all look light up, etc.  So I unplugged the switches from the SM3 and went straight into the APC.  still nothing.  I then removed the APC And put it into a belkin surge bar that was NIB, still not working.  I then moved everything off the PoE switch into the switch that was just replaced (Switch 1) and luckily had enough injector ports on a recently purchased midspan injector to power up all the devices.  Now with the PoE switch out of the way, everything is working.  Still plugged into the Belkin power strip.  so now I plan to RMA the second Switch.

So could the SM3 or the APC have gone bad and maybe messed up the switches?  I find it very suspect two switches go bad within a week at the same time? the switches werenot purchased together, one in 2013 and one in 2017 or 2018.

Once I get a new RMA switch from Cisco should I try the APS UPC and the SM3 or could they be "dirty" and mess up the next switches?

Edited August 26, 2022 by eggzlot

[South Africa C4 user]

A loop in the network?

Sonos?

Flo?

All potential culprits

[eggzlot]

22 minutes ago, South Africa C4 user said:

A loop in the network?

Sonos?

Flo?

All potential culprits

No Sonos

No Flo

and if a loop, wouldn't I have put the loop into the other switch?  I still have 3 switches, so though I've decommissioned 1, 2 are still active.  so I don't think its a loop, but I am also not a network Engineer.

I do wonder, the new Cisco SG220 switch I installed, are there any guidelines to get it working properly with C4 with multicasting, etc?  Perhaps I had some stuff config'd on the old one and forgot to move it over.  But either way, today its the older PoE one giving me the issues, not the newly installed one

[lostndashuffle]

Unplug everything from said switch. Add 1 thing back at a time. 

Double check items that are wifi capable that are hardwired into switch, and make sure wifi is disabled.

[Cyknight]

Any wifi that CAN do mesh networking? If so I'd check there first, they can sometimes cause awkward loops when they lose networking, then it returns (and they don't switch off of mesh mode).

 

Based on description I'm tempted to lean to a loop, though difficult to be sure.

[mstafford388]

Any Sony TVs wired into the network?  They have a known issue where their TVs try to use IPV6 and it's crashing entire networks randomly.  Doesn't necessarily sound like the case based on your info but just want to add any potential options. 

[eggzlot]

10 hours ago, Cyknight said:

Any wifi that CAN do mesh networking? If so I'd check there first, they can sometimes cause awkward loops when they lose networking, then it returns (and they don't switch off of mesh mode).

 

Based on description I'm tempted to lean to a loop, though difficult to be sure.

no mesh - my Wifi is handled by Unifi APs

1 hour ago, mstafford388 said:

Any Sony TVs wired into the network?  They have a known issue where their TVs try to use IPV6 and it's crashing entire networks randomly.  Doesn't necessarily sound like the case based on your info but just want to add any potential options. 

I do have 2 Sony TV's hardwired into my network.....Anything to check in the menus?

 

as a small update, the 28 port PoE switch that caused the 2nd network drop - I moved everything off of it and my network is stable.  I've replugged in that switch, and now I can find it on my network.  I've yet to plug anything into it, maybe tonight I'll plug in 1-2 touch screens and let that sit for ~48 hours and see if it is stable.

I am wondering if I did cause a loop - could connecting a switch (accidentally) to two other switches cause a loop?  I have 3 switches, the 2 with issues is not my "main" switch, the main switch is my 3rd switch near my router.  And the 2 switches that have been giving me some issues are both directly wired into that main switch.  While I was taking out the switch with the dead ports I had to remove a few ethernet wires and wondering if I put something in to the wrong switch and maybe that PoE switch was plugged into both of the other switches when I put things back together (makes sense in my head, not sure if that is clear in my writing)

[mstafford388]

4 minutes ago, eggzlot said:

no mesh - my Wifi is handled by Unifi APs

I do have 2 Sony TV's hardwired into my network.....Anything to check in the menus?

 

 

If it is the sony TVs taking down the network the fix is actually to switch those TVs to wifi, there aren't any settings that would fix it. 

[msgreenf]

8 minutes ago, eggzlot said:

could connecting a switch (accidentally) to two other switches cause a loop?  I

yes

[eggzlot]

1 minute ago, mstafford388 said:

If it is the sony TVs taking down the network the fix is actually to switch those TVs to wifi, there aren't any settings that would fix it. 

Thanks but that isn't an option - I am using NetPlay video distribution and the Netplay app resides on the Android OS and to be able to receive the content VS recommends to have the TVs hardwired.  Is there a way to see if they are trying to use IPV6 even if there isnt a setting to change it?  I could go the route of getting a FireTV for each location and hardwiring that but that it kinda defeats the simplicity of Video Storm/Netplay and just an ethernet cable into my Sony TVs to get all of my content.

[eggzlot]

Just now, msgreenf said:

yes

go me!

I am wondering if that is the issue as I was removing 1 switch and having to move some wires around, maybe I plugged something into the other switch when I didn't mean to.  that is what happens when you have 100's of non labeled wires

[msgreenf]

3 minutes ago, eggzlot said:

go me!

I am wondering if that is the issue as I was removing 1 switch and having to move some wires around, maybe I plugged something into the other switch when I didn't mean to.  that is what happens when you have 100's of non labeled wires

that could 100% be it.  Pro Tip: use a different color cable for uplinks between switches

[eggzlot]

1 minute ago, msgreenf said:

that could 100% be it.  Pro Tip: use a different color cable for uplinks between switches

if there was a messy rack of the year award I wouldn't be eligible because I would have already won the Lifetime Achievement award for worst rack ever

Yes I should have a color for cameras, a color for data ports, a color for video distribution, a color for uplink, a color for touchscreens, etc.  I just have unorganized chaos

[msgreenf]

 

Well pick what's most important and start there. Rome wasn't built in a day 

[eggzlot]

52 minutes ago, msgreenf said:

 

Well pick what's most important and start there. Rome wasn't built in a day 

rome may not have been built in a day, but it may take me 3 days to just crimp 1 ethernet cable   I should buy some mixed packs of different colors and plan it out.  seems like a nice winter project when its cold outside.

[Cyknight]

14 hours ago, eggzlot said:

no mesh - my Wifi is handled by Unifi APs

Uhm, unifi are capable of mesh networking - in fact I've seen this exact issue with a unifi setup once, where the CAT had a bad wire - AP got powered just fine, but lost network connection regularly, reverted to wireless uplink (which was for some reason enabled) then reconnected hardwired causing a loop situation that would paralyze the network until it either lost hardwire again, or dropped the uplink. Was a fun day of tracing the root cause ...

 

14 hours ago, eggzlot said:

as I was removing 1 switch and having to move some wires around, maybe I plugged something into the other switch when I didn't mean to

Also very likely yes - it happens, trust me.

[eggzlot]

9 hours ago, Cyknight said:

Uhm, unifi are capable of mesh networking - in fact I've seen this exact issue with a unifi setup once, where the CAT had a bad wire - AP got powered just fine, but lost network connection regularly, reverted to wireless uplink (which was for some reason enabled) then reconnected hardwired causing a loop situation that would paralyze the network until it either lost hardwire again, or dropped the uplink. Was a fun day of tracing the root cause ...

 

Also very likely yes - it happens, trust me.

well to my novice mind mesh = Eero or something like that.  With this switch still out of commission, all of my Unifi AP's are working fine for the last 3-4 days.  I am going to assume I created a loop when removing 1 switch and accidently plugging the two switches together.

If my network stays stable all week on the two switches, this weekend I will try to re-introduce the 3rd switch back into the mix by just a cluster of devices at a time.