I expect they are somewhat oblivious to you, unless you are in their environment and they perceive you as a predator or other threat. You could ask them, but like the song says, “They won’t answer, they can’t talk.”
Sounds territorial. But there’s plenty of fishing in O’Neill, try the SH call numbers, Level 3, Shelf 128.