Goldman shares fell 3.4% Wednesday, the third-biggest decline in the Standard & Poor's 500 Financials Index.
Smith pointed to Chief Executive Officer Lloyd Blankfein and company president Gary D. Cohn as responsible for letting the firm falter.
"I truly believe that this decline in the firm's moral fiber represents the single most serious threat to its long-run survival," wrote Smith.
Smith painted a picture of executives with zero respect for their clients, saying it makes him "ill how callously people talk about ripping their clients off," and that in the past year he's heard five different managing directors refer to their clients as "muppets."
Even though Smith's Goldman letter weighed on the bank's share price Wednesday, shares had almost made up the losses by midday trading today (Thursday).
So just how much will the Goldman letter actually weigh on GS stock and popularity?