On January 21766 he married to Elizabeth Boscawen , daughter of Hon. Edward Boscawen, Admiral of the Blue, and sister to George Evelyn Boscawen, 3rd Viscount Falmouth . With her he had seven sons and two daughters:
Henry Charles Somerset, 6th Duke of Beaufort his heir and successor; and
Edward Arthur Somerset (February 21817 - March 121886), who gained the rank of Lieutenant-General and on September 131849 married to Agatha Miles (d. August 121912), daughter of Sir William Miles, 1st Baronet Miles of Leigh Court , with whom he had one child, a daughter:
Lord Arthur John Henry Somerset (February 121780 - April 181816) who married on June 231808 to Elizabeth Boscawen (bef. 1793 - March 21872), daughter of George Evelyn Boscawen, 3rd Viscount Falmouth , with whom he had no issue;
Lady Elizabeth Somerset (bef. 1781 - May 51836), who was amrried to Very Reverend Charles Talbot, Dean of Salibury , grandson of the 1st Baron Talbot of Hensol, on June 271796, with whom she had three sons and three daughters;
Lord William George Henry Somerset (September 21784 - January 141861), who gained the title of a reverend and held the office of Prebendary of Bristol, and who was married twice, on June 291813 to Elizabeth Molyneux , daughter of Lt.-Gen., Sir Thomas Molyneux, 5th Baronet , with whom he had one child, a son, and who died in 1843, and in 1844 to Frances Westby Brady , who died on August 311854, with whom he had no issue;
Henry Charles Capel Somerset (June 201816 - January 171905), who gained the rank of a Colonel and on September 101840 married to Alice Elizabeth O'Connell , daughter of Lt.-Gen., Sir Maurice Charles O'Connell, who died on August 291892, and with whom he had - besides a natural daughter - his only child, a son:
Blanche Isabella Somerset, who was married to George Henry Hopkinson, with whom she had only a son;
Lord John Thomas Henry Somerset (August 301787 - October 31846), who gained the rank of a Colonel, and who married to Catherine Annesley , daughter of Arthur Annesley, 1st Earl of Mountnorris on December 41814, with whom he had no issue;