I used to be nervous about riding some roller coasters. Particularly those at Cedar Point. I was fine with some of them, but some of the more extreme ones, I didn't want to ride. And one day, I did. And it was mostly awesome. (Some rides weren't so awesome.) I was able to ride the rides for two reasons. One, I just kept telling myself "It's just a ride. It's been out for a number of years, it's been tested, and it's safe. What is there to be afraid of?" I also went to the park, that day, with someone who made me feel brave, and confident about myself. I don't know if you have someone like that in your life, but if you go with them, it may help.
I'm not sure that you have a true fear, as you said that you like other roller coasters. (Not that I'm trying to impose or put words in your mouth or anything.) In this case, it may be a simple case of mind over matter. Perhaps if you tell yourself that you will ride an upside-down roller coaster, and that you won't be afraid of it, and just get in line, without thinking about the aspect that's making you fearful, maybe you'll be able to get on the ride and not be worried about it. It might help if you have someone there, again, for moral support.
Or, maybe this just isn't for you. There's nothing wrong with not liking roller coasters. A number of people on this thread have stated that they hate roller coasters, and that's fine. As for you, you just don't like a certain kind of roller coaster, and that's fine too. Unless this is something that's become an inhibition in your daily life, there's nothing wrong with not wanting to ride a certain roller coaster, or any roller coasters at all. If you feel this is something you're willing to and can conquer, then by all means, go for it. If this is something that turns you into a puddle of jelly, though, and you thinking that by riding the ride, it would only make things worse, maybe it's not worth doing. I think you may want to weigh both sides before you attempt a ride. Best of luck, though, with whatever you decide.