Rosser owned land in Ramsey County, invested in the development of the town of St. Peter in Nicollet County, and was the Secretary of Minnesota Territory (akin to Lieutenant Governor) from 1853 to 1857.

Before 1853 he lived in Petersburg, Virginia. In 1850 he held three female slaves, ages 40, 30, and 13.