They are both correct. In "handwritten letters" (no hyphen in handwritten) the word "handwritten" is an adjective describing letters. In "handwriting letters" we can look at "handwriting" as an action verb. But now that you point it out, it would be much better to say "writing letters by hand" instead of "handwriting letters." I think it would be too confusing for new English speakers to leave it as "handwriting letters."