Although the general might now be reckoned among the exiles, he had not fallen into utter disgrace, for Juliana Maria afterwards took him under her protection, and tried to keep him, for the purpose of intimidating the violent ministerial opposition. For in July, 1774, he unexpectedly received an invitation from her to come at once to Fredensborg, where the court was residing at the time; but when the ministers heard of this, they were penetrated with fear, and induced the War College to intimate to the general that he was to remain at his post in Rendsburg, and send an apology to the queen. Although Banner found himself compelled to obey on this occasion, on receiving soon after another request from the queen to come across, he informed her that he would accede to her wishes, and appeared at Fredensborg in the beginning of August. As, however, he was a thorn in the eye of the ministers, he soon began quarrelling with them, the result of which was, that he was commanded by the Generalty, who would not listen to his plans and complaints, to return to his post. This induced him to send in his resignation; but it was not accepted. Hence he imagined himself indispensable, and took a step by which he hoped to overthrow his opponents in the War College. He sent into the privy council a rambling plan for a thorough reconstruction of the army; but as Eickstedt had anticipated him, and handed in a similar project, Banner's was sent back to him unheeded. Infuriated at this, he again forwarded his resignation, and dated his request on the eventful day, January 17th, 1775; but this artful trick did not avail him. On January 23rd, a royal cabinet letter was sent to the Generalty, to the effect that the king, in consideration of the proofs of fidelity, zeal, and devotedness, which Lieut.-General von Köller-Banner had furnished, felt himself induced to assent to his petition of January 17th in all points. He would, however, retain his former pay of 4,400 dollars, of which, 2,600 had been granted him for his meritorious services on January 17th, 1772. Furthermore, he would still remain in the king's service, and be always ready to act as a Danish general whenever the king thought proper, and as befitted a lieutenant-general; at the same time, he was permitted to visit other armies, and take part in foreign campaigns. Köller-Banner, after this, left the country and went to Vienna and the Austrian army, but returned at the beginning of 1777 to Copenhagen, where he was again most graciously received by the old queen. Soon afterwards, however, he was mixed up in a scandalous affair with the magistracy about a child an actress had given birth to. The excitement caused by this was so general, that he received his full discharge from the military service. But the protection which the hero of January 17th still enjoyed was so great, that his 4,400 dollars were left him as a life pension.