That's a massive yikes from me. This person is not your friend, with their inability to listen to the word 'no' and completely inappropriate physical touch it sounds like they want to take advantage of you in some way and it'll get worse if allowed to continue. I'd recommend minimizing or ceasing contact entirely, the fact you're concerned about how he'd take you speaking up about it says it all.