As I'm involved both professionally and as a volunteer in the paddling community I've learned that two of the most aggravating words I hear are "experienced" and "Intermediate". And neither of these words mean that the user is knowledgeable or educated.
Regarding flotation, I agree that flotation is a necessary and required piece of safety equipment. But flotation only does you good, at least here in the cold water, if there is someone to haul you out of it. In fact, every year here in Seattle, on New Years Day, we have the Cold Water Rescue Seminar. At this free event we encourage paddlers to show up dressed in their kayaking finest (what they would typically wear to paddle in cold water) and just float out in the cold water until they're cold enough to haul themselves back onto the beach. Just floating and wearing typical gear under a drysuit people typically last 10 to 15 minutes before they get out. It's quite an eye opening experience for many. Especially since so many paddle around wearing less in the summer when the water is only a few degrees warmer, especially those pesky SUP folks.
The next part of that seminar is to help participants develop strategies to get the swimmer out of the water quickly, as in the first thing they do. The overall point of the whole deal is to emphasize that cold water kills and not through hypothermia.
To your point that you can still drown in the waves; well not much to be done about that except carry SCUBA gear. It also raises an entirely different topic revolving around trip planning and risk assessment/management. Perhaps you shouldn't have gone out that day?
At the end of the day, I'm happy if people just wear the bloody things rather than tell me they are strong swimmers or that they don't plan to tip over. Wear the PFD, wear the PFD, wear the PFD. If you need it and you're not wearing it, it's too late.