Gillespie afterwards minister of Kells, and Robert Lundie the future minister of Kelso, he formed habits of particular intimacy. From the Presbytery of Dumfries, he obtained licence as a probationer in the spring of 1798, and he thereafter accepted the situation of tutor in the family of Colonel Erskine afterwards Earl of Mar, who then resided at Dalhonzie, near Crieff. In this post he distinguished himself by inducing the inhabitants of the district to take up arms in the defence of the country, during the excitement, which then prevailed respecting an invasion. In the spring of 1799, the parishes of Lochmaben and Ruthwell, both in the gift of the Earl of Mansfield, became simultaneously vacant, and the choice of them was accorded to Mr Duncan by the noble patron. He preferred Ruthwell, and was ordained to the charge of that parish, on the 19th September. In preferring the parish of Ruthwell to the better position and wider field of ministerial usefulness presented at Lochmaben, Mr Duncan was influenced by the consideration, that the population of the former parish was such as would enable him to extend the pastoral superintendence to every individual of his flock. In this respect he realised his wishes; but not content with efficiently discharging the more sacred duties of a parochial clergyman, he sought with devoted assiduity, the amelioration of the physical condition of his people. Relieving an immediate destitution in the parish, by a supply of Indian corn brought on his own adventure, he was led to devise means of preventing the recurrence of any similar period of depression. With this intention, he established two friendly societies in the place, and afterwards a local bank for the savings of the industrious. The latter proved the parent of those admirable institutions for the working classes, known as _Savings' Banks_, which have since become so numerous throughout Europe and the United States of America. The Ruthwell Savings' Bank was established in 1810. Numerous difficulties attended the early operation of the system, on its general adoption throughout the country, but these were obviated and removed by the skill and promptitude of the ingenious projector. At one period his correspondence on the subject cost him in postages an annual expenditure of one hundred pounds, a sum nearly equal to half the yearly emoluments of his parochial cure. The Act of Parliament establishing Savings' Banks in Scotland, which was passed in July 1819, was procured through his indomitable exertions, and likewise the Act of 1835, providing for the better regulation of these institutions.