The only time I ever hear of upper class white men being referred to as "boy" is when someone is trying NOT to hold them accountable for some wrongdoing. The word is then used to symbolize their innocence.

And yeah, there were often Black grandfathers who got referred to as "boy".