Google " sea kayak lonely island Georgian Bay ". Some guy from Chicago and a friend kayaked from Tobermory to Lonely Island and back. I believe they crossed from Cove Island to Yeo Island to Fitzwilliam Island then to Club Island and finally to Lonely Island. If you google it you'll find that.
My girlfriends dad has waterfront property on the southwest end of Manitoulin. His property is across from Burnt Island. I kayaked out to Western Duck Island. A 7 km open water crossing. My girlfriend doesn't like open water crossings so we came back with no further exploration. I plan on doing a multi day next time I am up on Manitoulin. I am gonna Circumnavigate all the Duck Islands.
Anyway check out that article. Sounds like a fun trip.