I ride once a week. I'm pretty new to riding, but it's very enjoyable even though I prefer ATVs.
I've loved horses my entire life, way before the pony craze. Being a guy, I kept it to a secret until I moved into my own place and started lessons. My one complaint about horses in general is that they are very polarizing. They're both ugly/cute, annoying/enjoyable, easygoing/anal, blah blah, you get the picture. It also depends on the horse as well. Some are bombproof angels that will put up with the most clueless, inexperienced rider while others are like little balls of energy that wanna go all the time, sometimes at the expense of the expecting rider. One other complaint is if you don't know how to stay with the tempo of the horse's steps, trotting is a beating on your ass, unlike cantering which is much more smooth (unless you fall off tempo...).
One other thing, don't let horseback riding enthusiasts fool you into the "horseback riding is good exercise" trick. Go play an equivalent time of basketball if you want real exercise