In 1882 he accepted the Colonial Office from Mr. Gladstone, and held it until the fall of the Government in the summer of 1885. His ministry was not a very eventful one, and it was marked by that steady adherence to a middle line which had always characterised him. He congratulated the country that the indifference to our colonies which had prevailed during his youth had passed away, but he was by no means favourable to extensions of the Empire. 'We have quite black men enough,' he was accustomed to say; and he believed that any increase of our responsibilities was likely to endanger the Empire, and to divert the energies of politicians from pressing home questions. He did not condemn the policy which led to the occupation of Egypt by England, but he declared that even if it was inevitable it was a misfortune, and that we ought to 'see that we do not on any pretext, however plausible, get that Egyptian millstone tied permanently round our necks.' He was very sceptical about Imperial Federation, and entirely incredulous about the possibility of an Imperial Zollverein. He deplored the protectionism of the colonies, but was himself a strict free-trader of the school of Cobden, and utterly opposed to any attempt to negotiate treaties with the colonies on a basis of preferential tariffs. On the other hand, he showed himself quite ready to favour Confederation in Australia, and he accepted gratefully Australian help in the Soudan, but he was much alarmed by tendencies in some colonies which might lead to complications with foreign Powers, and he incurred considerable unpopularity in Australia by refusing to consent to the annexation by Queensland of New Guinea.