The Gray family was that most intimately associated with Skibo Castle in its earlier days. Its progenitor was Sir William Gray, Chantor of Ross, whose two sons were legitimised in a precept granted at St. Andrews on 5th June, 1539. The older one, John, became hereditary constable of Skibo Castle, and its lands were feued to him. In 1565 Bishop Robert Stewart assigned the castle to John Gray. In 1570 John Gray of Skibo, Chamberlain to Robert, Bishop of Caithness, and his son, Gilbert “retired to St. Andrews there to wait until they might return to Skibo without danger." This was after the burning of Dornoch by the Mackays of Strathnaver. Gilbert was his son by his second wife, Elizabeth Barclay, daughter of the laird of Pitgarthie. He was first married to Janet Matheson, sister of Sir John Matheson, Chancellor of Caithness. He died at Skibo in 1586, and was buried at Dornoch. His son Gilbert, who succeeded him, was Chantor of Caithness, but resigned his office in 1583, and died at Skibo on October 3, 1624. He, too, was twice married, and had eight sons and one daughter, Janet, who married

William Cuthbert, Provost of Inverness. The most outstanding member of his family was John Gray, who became Dean of Caithness in 1608, tutored the young Earl of Sutherland, and was mainly responsible for the partial restoration of the Cathedral after its destruction by fire. Gilbert Gray was succeeded in 1624 by his oldest son, George, whose first wife was Jean, daughter of John Gordon of Embo, who died in childbed in 1612. Sir Robert Gordon pays her a tribute in which he says :—" Shoe wes in hospitality and sundrie other vertues nothing inferior to any in that cuntrey : as much lamented by her friends and all other cuntriemen as any gentlewoman that died in Southerland these many yeirs, and chieflie shoe wes exceidinglie bewailed and regrated by her husband, who through the love and affection which her did cary unto her and to the children which he had by her, did not marie agane for seaven yeirs after her death.”

He died on July 11th, 1629, " to the great regrait of all the inhabitants of that countrey and the adjacent shires wher he had bene a good patriot dureing his days,” as Sir Robert records.

Robert, his oldest son, succeeded him, and, as the nearest heir to his grandfather, he had sasine, dated at Dornoch, 19th April, 1634, from John, Earl of Sutherland, of the lands of Alustie, Ardalies, and others. He married Jean Seton, a niece of the Earl of Winton, by whom he had three sons and four daughters. It was she who entertained at Skibo, in 1651, the ill-fated Marquis of Montrose after his capture, and who insulted the officer in command of his guard by throwing a roasted leg of mutton at his head, as she exclaimed: " If ye dinna ken ye're manners an ye're place, I'll mak ye." Her husband was heavily fined for her offence. Gilbert Gray, his eldest son, was served heir to his father on 25th February, 1693. He made generous provision for his family of three sons and four daughters during his lifetime, and he died before 14th July, 1716. His oldest son and successor, George Gray, married three times. His second wife was a niece of Viscount Stair, and bore him two daughters, the younger of whom married John Mackay of Tordarroch. His third wife was Isobel, third daughter of the laird of Newmore, and their oldest son, Robert, succeeded his father.

He was served heir-male to his grandfather in Skibo Castle and Mains in 1737. His two half-sisters, Isobel and Jean, along with the latter's husband, raised an action against him in 1740, charging him to enter himself as heir to his father, whereupon he raised a process for the recovery of family documents and the banner of the hereditary constable of Skibo, which bore the armorial bearings of the family and the motto " Constant." The legal proceedings thus instituted cost him his inheritance, which he had to surrender in 1744, owing to bonds with which the property was burdened. He thereupon resumed his military career, and died without issue on 12th April, 1776. His family's long connection with Skibo ceased with him.