Yes, eventually they don't really want to be cooped up in one room, especially when they realize that there is a bigger world out there. Getting safety glasses is a good idea. They will protect your eyes and allow you to remain more relaxed in his presence, since the more relaxed you are, the more relaxed he will be.
When he starts staring at you intensely, try to verbally deescalate the situation, say soothing things. The sound of your voice should remind him of your close bond from when he was little and possibly break the primitive inclination towards aggression. Aggression on your part, verbal or physical, is likely to be met with more aggression on his part.
Try to figure out what is triggering his aggressive response. Often there is a pattern. For one thing, they can become upset when you clean up their pee and poop, because that is part of how they assert their territorial claim. So, they can regard removing it as an aggressive act, a denial of their territorial claim. Also, they are observant and don't like being tricked.
With the passage of time, when the aggressive instinct has subsided, I think they almost forget how ugly the situation became previously, though you will remember it intensely, which will make you tense which may trigger wariness on their part.
Engaging in allogrooming is a good way to build relationship capital. They are most receptive to allogrooming in the hour before sunset. You describe something like this in your subsequent post. If things go well, he will groom your hands, chewing off dead skin and licking your fingers.
I think the inclination towards territorial aggression increases with age and seasonally, during the winter months, when life become rough for squirrels and standing one's ground to retain good territory and a well protected and warm nesting location is critical to survival.