I been diving with both sea lions and seals it's not scary in general and usually a lot of fun. Harbor seals are very shy and usually only try to sneak up behind you. Sea lions are the opposite, they will get right in your face; doing acrobatics and swimming circles around you. Usually itís the females and juveniles who interact with divers; Iíve only been diving with a bull once.
Most marine life is not there to attack you. Sea lions will become aggressive when trying to steal your fish, if you get too close to their territory on shore, or juveniles swimming in the nursery, otherwise they are probably just curious. Of course, they are wild animals and you might run into one on a bad day. Remember kayakers are coming into close proximity with sea lions on a daily basis on the West Coast, how many attacks have been reported?