If my surname is OSEI-MENSAH, how do I enter the first 5 letters of my surname to retrieve my US Visa Application.

You should enter “OSEIM” as the first five letters, following the U.S. visa application formatting rules for hyphenated last names.

Enter “OSEI-” (including the hyphen) as the first five letters of your surname.

I would say enter OSEIM (the first 5 letters of your surname without the hyphen).