Fair Value of Financial Instruments - Market Valuation Adjustments, Net (Details) - USD ($) $ in Thousands	3 Months Ended
Mar. 31, 2017	Mar. 31, 2016
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income, net	$ 1,713	$ 6,281
Total Market Valuation Gains (Losses), Net	15,129	(15,848)
Mortgage Banking Activities, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	17,397	7,237
Mortgage Banking Activities, Net \| Residential Loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	8,532	5,439
Mortgage Banking Activities, Net \| Commercial Loans at Fair Value
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	0	433
Mortgage Banking Activities, Net \| Consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	0	1,484
Mortgage Banking Activities, Net \| Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(1,400)	(12,754)
Investment Fair Value Changes, Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	1,551	(19,538)
Investment Fair Value Changes, Net \| Consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(1,810)	(1,580)
Investment Fair Value Changes, Net \| Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(5,127)	(35,810)
Investment Fair Value Changes, Net \| Residential loans held-for-investment at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(2,333)	23,463
Investment Fair Value Changes, Net \| Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	11,143	(5,601)
Investment Fair Value Changes, Net \| Risk sharing investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(205)	(10)
Investment Fair Value Changes, Net \| Impairments on AFS securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Market valuations gains and losses, net	(117)	0
MSR Income (Loss), Net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income, net	(3,819)	(3,547)
MSR Income (Loss), Net \| Risk management derivatives, net
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income, net	(749)	41,057
MSR Income (Loss), Net \| MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Mortgage servicing rights income, net	$ (3,070)	$ (44,604)