Randy,you are right to be leary of those critters. Most of them are shy enough to not bother you, but it only takes one big bull that's in a bad mood to ruin your day.
Lately, on the Columbia I've noticed that some of the seals have developed a new curiosity about boaters. The smaller seals used to go the other way when I got near them, but now, some of them come right at me. I think some boaters might be feeding them, or they've figured out that there might be a chance to pick up an easy fish. Some of the fishermen I've talked to have had seals remove a fish from their line, without so much as a thank you.
I always feel more comfortable when the sea lions are occupied somewhere other than where I am. As you indicated, it can cause you to almost spill your bladder when one of those buggers comes up next to you and thrashes the water, even if it is only because he was startled by your presence.