The position of the Milnes in Yorkshire was almost unique. In Wakefield, during the flight of years, there sprang into prominence certain merchant princes whose names became household words throughout the county. The Milnes, Heywoods and Naylors, in turn, rose to affluence; but foremost and distinct among these remained the Milnes, who from 1670 owned the great cloth trade of the North, and who, towards the close of the eighteenth century, were represented by four brothers whose firm had secured a monopoly of that trade between England and Russia. These brothers, by reason of their wealth and influence, were received on terms of intimacy by the older county families. They built themselves each a substantial house in Wakefield, fashioned out of bricks which they manufactured and timber which they had imported from Russia, with which country they were naturally in constant communication in the course of their business. These houses, which stood close together, facing the main road through Wakefield, were handsome in construction and luxuriously furnished; but, by and by, two branches of the family migrated from the town of their birth; James Milnes built Thornes House, and Richard Slater Milnes purchased the estate of Fryston, where he took up his residence about 1790. His new possession was a larger and more comfortable home than the dwelling he had quitted, and although standing in the centre of the great West Riding industries, it was beautifully situated on the banks of the river Aire. Besides extensive gardens and shrubberies, it was surrounded by a fine park, while adjoining it were miles of beautiful larch and beech woods. On the death of Richard Slater Milnes it passed into the possession of his son, Robert Pemberton, who with his brother, Richard Rodes, were the only two sons in a family of nine children. The brothers, in some particulars, presented a marked contrast to each other, though both were fascinating and clever. Robert Pemberton was extremely eccentric, but brilliant. He was recognised to be full of promise, and it was anticipated that he would one day make a considerable stir in the political world. Writing of him many years later, John Stanhope mentioned the following anecdotes:-- "Mr Milnes of Fryston was one of my earliest friends. After a sharp contest with Mr Smyth of Heath he was returned for the Borough of Pontefract. His Maiden speech in Parliament produced a very great sensation; but a second speech which he made shortly after was considered as a failure, though Mr Plummer Ward, himself no bad judge, declared it was superior to the former and spoke highly of it. I rather think that Milnes terminated it abruptly and was considered to have broken down. He seems himself to have thought so for he made no further effort, and, soon after, abandoning all political views, turned his mind entirely to Agriculture.