In 1857 Mr. Lee's health gave way, and his physician ordered him to relinquish all cares of business. Acting in accordance with this advice, he sold his interest to his partners for sixty-five thousand dollars, taking the notes of the firm for that amount. After a few months of travel in his own country, he sailed for Europe in June, 1858, in company with Willard Small, with the intention of spending five years on the continent. He proved to be a good traveler; his keen observation encompassed everything; his generous heart and the geniality of his nature won to him many friends. Ere many months had elapsed he had traversed England, France, Germany, Belgium, and Spain. While he was in Paris, an incident occured, the recollection of which has served to enliven many a social occasion. It was the exciting time succeeding the attempted assassination of Napoleon by Orsini. Mr. Lee always wore a long, sandy beard, and in his travels sported a soft, broad-brimmed hat. One day, while walking about the streets, he was arrested and taken to the Palais de Justice. Explanations and expostulations proved unavailing. The prisoner was declared to be a "red Republican," and, in those days, that was no joke. It was only after the production of a passport and the interference of the United States consul, that the authorities were induced to release their captive. Mr. Lee was in Paris, and was on the point of making a second journey into Spain, when the United States mail brought him a letter, conveying the tidings of the death of both Mr. Phillips and Mr. Sampson, and the failure of the house. The panic of 1857 had made sad havoc with the book trade generally, and those firms which weathered the storm were sorely pressed. Phillips and Sampson met with heavy losses, but struggled on in the hope of recovering lost ground. But, in 1859, the death of the senior members of the firm seemed to paralyze its prosperity, and the worst quickly followed. Mr. Lee had received no warning of the impending calamity, and for the time was much overcome by the announcement. He foresaw what it implied, however, and at once returned to Boston, to find himself a heavy loser by the financial disaster. Still undaunted, he gathered up what remained of his fortune and, in February, 1860, he became a member of the firm of Crosby, Nichols and Company, which had purchased many of the stereotype plates belonging to the late firm of Phillips, Sampson and Company, and which now took the name of Crosby, Nichols, Lee and Company. But the long stagnation of trade, succeeded by losses in the southern states, consequent upon the political troubles of those days, bore heavily upon the new firm; and, in the spring of 1861, Mr. Lee left the business and again trod the streets of Boston without a dollar that he could call his own! Thus, after twenty years of business activity, his fortune was gone, and nothing remained for him to do except to begin life over again.