I'm gonna be straight with you, (and as someone who has dealt with people like him before), the only thing you did wrong was to exist in that place at that time. He was looking for a confrontation and would have tried to make one regardless of who showed up.

You gave him the benefit of the doubt, as a lot of us would do, (except any woman walking alone, who would have simply ran away), when someone stops to talk to us from their car. You stand well back from the car, and answer the questions as quickly as possible, and keep moving.

So yeah, he did want something. He wanted to fight. But you're right though, walking away and saying nothing to him at all, is a good tactic. (I'm too terrified to think what would have happened if your husband had not been there though.)