Fair Value of Financial Instruments - Portion of Net Gains (Losses) Attributable to Level 3 Assets and Liabilities Still Held Included in Net Income (Details) - USD ($) $ in Thousands	3 Months Ended
Mar. 31, 2024	Mar. 31, 2023
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net (loss) attributable to Level 3 liability still held included in net income	$ (3,420)	$ (6)
HEI Point Entities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	2,888	1,194
Held-for-investment at fair value \| Residential consumer loans at Redwood
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(1,497)	156
Held-for-investment at fair value \| Sequoia
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	6,910	2,349
Held-for-investment at fair value \| Freddie Mac SLST
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	3,367	8,759
Residential investor loans
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(1,397)	12,239
Consolidated Agency multifamily loans \| Freddie Mac K-Series
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	243	363
Residential Investor Term \| CAFL
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	10,251	(8,810)
Trading Securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	14,506	1,793
Available-for-sale securities
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	629	(28)
HEI
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	6,075	3,433
Servicer Advance Investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(634)	(1,352)
MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	2,723	(424)
Excess MSRs
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	(1,245)	(229)
Other investments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	0	(94)
Loan purchase commitments
Fair Value, Assets and Liabilities Measured on Recurring and Nonrecurring Basis [Line Items]
Net gains (losses) attributable to level 3 assets still held included in net income	$ 1,315	$ 353