Loans Held for Sale - Narrative (Details) - USD ($) $ in Thousands		3 Months Ended
	May. 31, 2015	Mar. 31, 2016	Mar. 31, 2015
Accounts, Notes, Loans and Financing Receivable [Line Items]
Gain on loans held for sale, net		$ 15,572	$ 44,504
Lending [Member] \| Line of Credit [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Collateral		285,100
Servicing [Member] \| Line of Credit [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Collateral		37,100
Unrelated Party [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Gain on loans held for sale, net	$ 7,200		12,900
Unpaid principal balance	$ 33,000		42,700
MSRs Retained [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Gain on loans held for sale, net		6,500	8,500
Ginnie Mae Loans [Member]
Accounts, Notes, Loans and Financing Receivable [Line Items]
Gain on loans held for sale, net		5,000	4,300
Gain on sales of reverse mortgage loans		$ 8,000	$ 25,600