In 1804 Jackson resigned his judgeship in order to give exclusive attention again to his private affairs. He had fallen badly into debt, and his creditors were pressing him hard. One expedient after another failed, and finally Hunter's Hill had to be given up. He saved enough from the wreck, however, to purchase a small plantation eight miles from Nashville; and there, after several years of financial rehabilitation, he erected the handsome brick house which the country came subsequently to know as "The Hermitage." In partnership with two of his wife's relatives, Jackson had opened a store in which, even while still a member of the highest tribunal of the State, he not infrequently passed tea and salt and calico over the counter to his neighbors. In small trading, however, he was not adept, and the store failed. Nevertheless, from 1804 until 1813 he successfully combined with planting and the stock-raising business enterprises of a larger sort, especially slave and horse dealing. His debts paid off, he now became one of the most prosperous, as he already was one of the most influential, men of the Cumberland country. But it was not given to Andrew Jackson to be a mere money-maker or to dwell in quietness. In 1804 he was denied the governorship of the New Orleans Territory because he was described to Jefferson as "a man of violent passions, arbitrary in his disposition, and frequently engaged in broils and disputes." During the next decade he fully lived up to this description. He quarreled with Governor John Sevier, and only the intervention of friends prevented the two from doing each other violence. He broke off friendly relations with his old patron, Judge McNairy. In a duel he killed Charles Dickinson, who had spoken disparagingly of Mrs. Jackson, and he himself suffered a wound which weakened him for life. He publicly caned one Thomas Swann. In a rough-and-tumble encounter with Thomas Hart Benton and the latter's brother Jesse he was shot in the shoulder and one of his antagonists was stabbed. This list of quarrels, threats, fights, and other violent outbursts could be extended to an amazing length. "Yes, I had a fight with Jackson," Senator Benton admitted late in life; "a fellow was hardly in the fashion then who hadn't."