Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	12 Months Ended
Dec. 31, 2019	Dec. 31, 2018	Dec. 31, 2017
Loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	$ (1,290)	$ (732)	$ (3,706)
Contingent Consideration
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	(3,217)	0	0
ABS issued
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 liability	(130,421)	(71,468)	(29,187)
Residential loans, held-for-investment \| Residential loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	67,470	(17,757)	523
Residential loans, held-for-investment \| Residential loans at consolidated Sequoia entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(10,062)	24,799	17,727
Residential loans, held-for-investment \| Freddie Mac SLST
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	63,583	21,295	0
Business purpose residential loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	14,603	445	0
Single-family rental loans \| CAFL
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(14,681)	0	0
Multifamily loans \| Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	130,083	46,555	0
Trading securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	18,865	(12,256)	28,612
AFS Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	0	(89)	(1,011)
Servicer advance investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	3,001	(702)	0
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(11,957)	1,942	1,277
Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	(3,260)	1,824	0
Shared home appreciation options
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	842	0	0
Loan purchase and interest rate lock commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	10,190	2,913	3,243
Loan forward sale commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	0	0	2,177
Other assets - Guarantee asset
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gain (loss) attributable to Level 3 asset	$ (932)	$ (251)	$ (1,223)