It was not all smooth sailing for competitors in the Rottnest Channel Swim yesterday, with rescuers bringing a team of four back to Fremantle after their boat sank.