Remember when Abe was saying this--before John Brown's raid in 1859 and before he had met with Frederick Douglass. He had been raised in the back woods with wolves and coyotes and his sister. What experience had he had with blacks? Compare his experiences with those of privileged Wm Seward of Auburn, NY.
Also, compare this early statement with what Abe said at Cooper Union in Feb. 1860. Abe was not perfect, but he grew with his experience.