To retreat is to pull back: Pulling back from a push.
Retreat is also what happens after a failed attack.
How you see the thing you are retreating from informs how you feel about the need to retreat. What you say to yourself you were doing prior to the retreat has an effect.
Then, we turn it all on its head by chosing to "go on a retreat". A very forward action for a pulling back word.