At this time and afterwards there was an overwhelming sentiment in favor of the Union in some parts of North Georgia. The people of that section had few slaves, and the arguments in favor of the protection of slavery in the Territories did not appeal to them: consequently they were opposed to secession. There was but one thing that prevented serious trouble between these Union men and the State government, and that was the fact that Joe Brown was governor. He knew the North Georgians thoroughly, and he knew precisely how to deal with them. General Harrison W. Riley, a leading citizen of Lumpkin County, declared that he intended to seize the mint at Dahlonega, and hold it for the United States. This threat was telegraphed to Governor Brown by some of the secession leaders in that part of the State, and they appealed to him to send troops to Dahlonega at once, and seize the mint by force. But the governor knew Riley and the people of North Georgia too well to make any show of force. He knew that any such demonstration would excite sympathy for Riley, and inflame the Union sentiment there.. So Governor Brown wrote to some of Riley's friends, telling them what he had heard, and saying that he had known General Riley too long, and had too high an opinion of his good sense and patriotism, to believe the report. At the same time the governor informed the superintendent of the mint that the State of Georgia now held that institution. The superintendent said he was willing to act under the orders of the governor. At Jasper, the county seat of Pickens County, the feeling of loyalty to the Union was very strong. The delegate from that county to the State convention had refused to sign his name to the ordinance of secession. Soon after the State had seceded, the citizens of Jasper planted a pole, and raised on it a United States flag, and kept it floating there for several weeks in open defiance of the Confederate and State authorities. This was an event to be delicately handled. The slightest mistake would have created a state of feeling in North Georgia that would have given no end of trouble during the whole war. But the Union flag floating in Pickens County irritated the rest of the State; and hundreds of appeals were made to Governor Brown to send troops to Jasper, and have the flag taken down by force. To these appeals he made but one response, and then turned a deaf ear to all criticism. "Let the flag float there," he said. "It floated over our fathers, and we all love the flag now. We have only been compelled to lay it aside by the injustice that has been practiced under its folds. If the people of Pickens desire to hang it out and keep it there, let them do so. I shall send no troops to interfere with it."