Yes, depending on spacing will avoid too few or too many on polos cuz the shirts don't look "right" to more or as nice... 3 seems to be the sweet spot IMO... 2 or 4 can work if spaced well... 5+ is often getting into crazy town. ;-)
2 or 3 is the permanent acceptable range (just like suit jackets). Fashion will push it back to 1 or up to 4 or 5, but those styles fade. I have a 4 button shirt right now, but it feels like too much. I kind of prefer 3 right now.
It's definitely something I notice, but not something I've given too much thought too. I normally notice it when I throw on a golf shirt that only has two button and I'm only closing one (I'm thinking Nike TW version of a year or so ago).