I do agree that having genders in a language is a plus for communication overall, but it does make English easier to learn. I remember trying to learn German and having a hard time with genders as they're different from my native language, Portuguese. And, just like @Ittoni, I think having various ways of saying words we use a lot is also an asset.
The one thing I have a hard time with not being able to use is the word "saudade". Not only is it a symbol of my country and culture, it's a word we use a lot. "Saudade" is the feeling of longing for something or someone that you don't have currently (more often than not, it's longing for childhood memories/objects or people who died/who went away/who you've not seen in a long time). I find it difficult to tell my boyfriend how much I miss him without that word.