When you let someone down: What is it that you picked up in the first place? What is it that you let down?
Being let down is a function of expectation. And an expectation should only exist when it is accepted, preferably explicitly.
If you too find yourself living in anxiety of constantly letting others down: Start choosing what expectations you are willing to pick up.