As described in historical dictionaries

Sorry, we don't have any historical texts which describe the Sworm surname. That's probably because it was very rare in Great Britain more than a century ago.

BritishSurnames.uk is a Good Stuff website.