Same here. I keep meaning to replay it as I've only beaten it once. The looming time-limit makes the game too stressful for me though, even with the half-speed song.
Yeah, I could never beat the game when I was younger when I got to that point. Even now it's frustrating. I think that since the overworld allows all of the worlds to be completed in any order, the difficulty stays stable for most of the game. It was pretty innovative for handheld platformers at that time, but it killed any sort of difficulty progression.