That's the difference between himself and Goku. Vegeta was very aware of his own strength and knew what he could do. Humans were fragile and needed to be treated with care, and he wasn't someone to smack down one who wasn't an opponent. The things he did while under Frieza's command were different; he followed those orders to ensure that his life remained intact, and he fought Goku and the other Z Warriors because they started conflicts with him first. They saw him as a threat — as they well should — but all of their information on him came directly from how he behaved on the battlefield.

None of them knew how Vegeta was when he was simply existing. All but Bulma, that is.

His hand fell away. "Insurance. How degrading."