In 1331 the clan of O’Toole seized the castle, pillaged, and burned it. Next year the Crown took possession again, and three years later they thought it expedient to appoint Lord Gerald Rochford constable of Ferns Castle. He held office for ten years, and was summoned to Parliament as a Baron. In 1347 Bishop Esmond came into possession of the stronghold, and was succeeded by Bishops Charnels and Denn. The latter prelate was the last Bishop to reside in the castle, for from 1402 to 1530 the stronghold was in possession of the MacMurroughs. At the end of this period it was captured by Lord Deputy Grey. He was on his way from Kilkenny to Dublin, when he sent word from Leghlyn to Stephen FitzHenry at Kilkea, to meet him at Ferns Castle with his guns and men. Lord Grey marched by night, and arrived in the morning before the fortress. He demanded its surrender, which was refused by the garrison, “using very spiteful language.” The day was spent in preparing for the attack. He posted his men round the building in the ditches and other cover, so that none of the besieged should escape, and the troops broke down the gate leading to the drawbridge. A Mr. Thomas Allen, who was with the attacking party, noticed that one of the garrison kept watch every now and then from one particular place, and he sent a gunner to hide himself where he could cover the spot with his weapon. This was successful, and the man was shot as soon as he returned. He was the governor and chief gunner of the castle. The rest of the garrison then asked for a parley, which was granted. Lord Grey told them that unless they surrendered before the arrival of the ordnance, which was within a mile of the stronghold, he would not accept a surrender, but kill them all. They then agreed to give the castle up, and two of the English were stationed in it during the night. The next day the Lord Deputy appointed a garrison of the MacMurroughs to guard the stronghold, taking their chief with him to Dublin as a hostage. A writer of this time describes the castle as the old inheritance of the Earl of Shrewsbury or the Duke of Norfolk, and “oon of the auncientis and strongest castells within this lande.” In 1536 Cahir Maclnnycross Kavanagh, the MacMurrough, was appointed constable by the Crown, but two years later he was superseded by Sir Richard Butler. At the time great anxiety was felt for fear of an attack from the Kavanaghs; and in 1550, we learn from the State Papers that Cahir M’Arte Kavanagh had managed to get possession of the castle by treatment. The next year Richard Kettyng asked the Privy Council to confirm the King’s letter granting him the stronghold, but they refused. It was considered necessary to have English captains in the castles of the districts to hold the Kavanaghs in subjection, so that a list of constables to Ferns Castle is recorded, the most remarkable being the Mastersons, father and son, the former being accused of conspiring against the Queen in 1569.