We've left from Toquart Bay heading to Barkley Sound a couple of times and not needed a 4X4 though good road clearance is a plus. The road is very rough so plan on driving very slow and have your boats properly secured and padded.
In the summer the end of the road turns into a motorhome campground so arriving early enough to depart that day is a plus. In the off season we've camped (been stormbound) at the end of the road for a couple of days and been all by ourselves.
A number of years back we had a great couple of days stay in Pipestem inlet waiting for the fog to clear in the islands. We had sunshine, there were very few people, lots of wildlife and some great freshwater swimming holes. It's also a short paddle from the end of the road which is how we ended up exploring that area. I'd check to make sure camping is allowed on Pipestem as that was a while ago.
Listen to your weather radio multiple times per day. Our last trip started with a light and variable winds forcast as we shoved off. Checking at lunch winds were predicted to reach 20mph that night (we started looking at where we could get protection) By the time we arrived at the campsite the park service was telling all boats to get off the water winds predicted to 50 by 11 that night we were in a real storm that lasted for over 24 hours. In the morning my new kayak, which had been securely tied above the high tide mark, was full of sand and gravel and looked well used from the beating it took from the large waves on the lee of the island. We've hit storms 3 out of 3 trips (April, August and Sept).
Sea Kayaker Magazine