Thanks Jakelay! This is very helpful.
I have played two setups last a few weeks just to see the difference. I got exactly same conclusion as yours. What I end up doing is to Roku/ATV on amplifier directly and skip the matrix. Blu-ray player, security camera, etc. left on the matrix. My setup was pretty simple. I actually think if previous owner did not leave the matrix for me, I probably do not even need it because those sources are rarely used and I can treat them as local sources and leave them in one room.
Two other reasons that made me go with this setup were I notices the matrix seemed occasionally crash from time to time, like once per movie or something. When it crashes, the TVs shows snow signal. Another reason was I have two new TVs and one old TV. The old one has lower resolution. Using matrix is not easy to manage the EDID issue.
I also thought about using TV to streaming directly. In my wiring, on TV side, there are only two CAT cables, one for LAN, another one for video. The only way I can see is to use ARC to return audio to my amplifier. While looking at the price of the video extender with ARC feature, the cost actually more than ATV. So I did not go that route. If previous owner did not leave extenders, I would consider this route as well.