I think Rarity is the biggest reason for Spike's inability to physically grow. Spike has a crush on Rarity, and as long as he has that crush, he wouldn't typically make decisions that she wouldn't like. He became greedy once, then Rarity scolded him. (She didn't know it was him, but that doesn't make a difference.) So if Rarity was gone for any reason, Spike may grow. But then there is also Twilight Sparkle. Spike sees Twilight as more of a motherly figure, so he wouldn't want to get on her bad side, either. So that would make it hard for him to physically grow. Get Rarity and Twilight out of the picture, then there's a chance he would grow. Applejack, Fluttershy, Pinkie Pie, and Rainbow Dash may contribute to his lack of growing, but I think Rarity and Twilight do it the most.
Rarity represents the element of generosity. I don't think she would want anyone to be greedy. If dragons could grow without greed, I'm pretty sure Rarity wouldn't stop him. She doesn't want him to be greedy because it's wrong, not because he wouldn't be her little slave anymore. (he was never her slave in the first place.)
Twilight is more likely to not want Spike to grow, but that's because she may have the fear of losing him. I think Spike doesn't want to grow either for that same reason.
In fact, maybe Spike doesn't want to grow. He doesn't need to grow right now, either. He's still mentally a child. He's getting more mature, but he still has the mind of a child. It's best, right now, for him to remain the way he is. I doubt Spike would want to leave his friends. He prefers ponies to dragons. He also didn't like himself when he grew to a giant dragon.
So my final answer is: Rarity and Twilight are the ones who prevent Spike from growing the most, and I don't think Spike wants to be a big dragon.