His practice at the bar had been for some time falling off, and his health had already begun to fail, when, on 10th November, 1823, Clerk was appointed an Ordinary Lord of Session in the place of Lord Bannatyne. Assuming the title of Lord Eldin, he took his seat on the bench on 22nd November. As a judge he was not a success; his temperament was not a judicial one, and his faculties at the date of his elevation were seriously impaired. In consequence of the infirmities of age, after five years of judicial work, he resigned in 1828, and was succeeded by Lord Fullerton. He died unmarried at his house, No. 16 Picardy Place, Edinburgh, on 30th May, 1832, in the seventy-sixth year of his age. As a pleader Clerk was distinguished by strong sense, acuteness, and the most profound reasoning. Throughout his entire career at the bar he delighted in defying, ridiculing, and insulting the bench; and it is recorded that his whole session was one keen and truceless conflict with judicial authority. He was in the habit of saying whatever he liked to certain of the Outer House judges without reproof. Lord Craigie especially, it is said, suffered a species of torture from him that required great natural sweetness and kindness of disposition to endure. Clerk, however, did not come off so well with the Inner House judges. On one celebrated occasion, having used somewhat threatening language towards Lord Glenlee in the Second Division, he was reluctantly compelled by the Court to make an apology to the offended judge. An account of this remarkable scene will be found in the “Journal of Henry Cockburn” (1874, vol. ii., pp. 207-210). In politics Clerk was a zealous Whig. He had a considerable taste for fine arts, occasionally amused himself in drawing, painting, and modelling, and had such an attachment to cats that his house could always boast of half-a-dozen feline indwellers. It is recorded that at the sale of his collection of paintings and prints, which took place at his house in Picardy Place after his decease, the floor of the drawing-room gave way, and about eighty persons--one of whom was killed--“were precipitated into the room below, to the destruction also of much valuable china and numerous articles of vertu there displayed.” In appearance Clerk was singularly plain; he was also very lame, one of his legs being shorter than the other; and his inattention to dress was proverbial. It is related that when walking down the High Street one day from the Court he overheard a young lady saying to her companion rather loudly, “There goes Johnnie Clerk, the lame lawyer,” upon which he turned round and said, “Na, madam, I may be a lame man, but no’ a lame lawyer.”