The next possessor who left his mark upon the castle was George Luttrell, Sheriff of Somerset in 1593. He built the market-house in the town and the older part of the present dwelling-house, which bears date 1589, incorporating with it much of the curtain wall, towers, and walls of the older and more distinctly military building. The entrance to the ward seems to have remained as before, through the gateway between the flanking towers. During the wars between Charles and the Parliament, the Luttrells sided warmly with neither party, and were out of favour with both. Its owners at this time were Thomas Luttrell, who died 1644, and George, died 1655. In 1643 a Royalist garrison, under Colonel Wyndham, took possession, and the castle was visited by Prince Charles, whose chamber is still pointed out. In 1646 Blake laid siege to the castle for the Parliament, and battered it from the north-west, behind the Luttrell Arms. It was surrendered by Wyndham in April, 1646. A few iron cannon balls, memorials of this siege, have been found. The government, although they apologised for the military occupation of the castle, levied a local rate for pulling it down. Probably this referred only to the upper part of the curtain wall on either side of the gatehouse. It is said that the gatehouse was injured, but its present condition shows that the injury could not have been of a very serious character. A century later the accounts show that the Luttrells raised the surface of the lower ward, probably about fourteen feet, evidently with earth obtained by scraping the adjacent slope of the tor. This, which involved the closing up of the old gateway, was probably combined with the construction of a new approach, which passed below and outside of the gatehouse, wound round the castle and the tor, and entered the lower ward at the new level. Matters thus remained until the accession of Mr. George Luttrell in 1869, when, under the judicious advice of Mr. Salvin, a great addition was made to the Elizabethan house, a new tower was constructed on the west front, and the foundation and pavements of buildings along the north front, and connected with the entrance gate and the gatehouse were laid open, and the walls restored and rebuilt, and a terrace formed along a part of the curtain. The approach for carriages was also much improved, though, as before, at the cost of avoiding the great gatehouse.