It will absolutely make a difference, even at lower volumes.
Even at low volumes, the music will be cleaner , punchier...just overall sweeter. Especially with an HK, it gives a warmer tone than other receivers in the price range.

I think that is a common misconception. I also used to think that I didn't need a better receiver since I never listen at higher volumes anyway. It's all about clarity and quailty of sound, not volume.
Clarity and quality can be heard at any volume.

Look at the HK 430 or 630.
You can also look into the NAD receivers. The T752 or T762 are both being sold a great prices since they are last year's models.