Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details)
$ in Thousands

Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	3 Months Ended
	Mar. 31, 2021 USD ($) $ / loan	Dec. 31, 2020 USD ($)	Mar. 31, 2020
Assets
ABS issued, net, Fair value \| $	$ 6,475,472	$ 6,900,362
ABS Issued
Liabilities
ABS issued \| $	$ 1,325,920
ABS Issued \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input	0.08
ABS Issued \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input	0.54
ABS Issued \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input	0.31
ABS Issued \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.01
ABS Issued \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.19
ABS Issued \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
ABS Issued \| Default rate \| Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued \| Default rate \| Maximum
Liabilities
ABS issued, measurement input	0.31
ABS Issued \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.02
ABS Issued \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input	0.30
ABS Issued \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input	0.32
Freddie Mac SLST
Liabilities
ABS issued \| $	$ 1,725,235
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input	0.08
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.01
Freddie Mac SLST \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.07
Freddie Mac SLST \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
Freddie Mac SLST \| Default rate \| Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST \| Default rate \| Maximum
Liabilities
ABS issued, measurement input	0.12
Freddie Mac SLST \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.08
Freddie Mac SLST \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input	0.35
Freddie Mac K-Series
Liabilities
ABS issued \| $	$ 460,565
Freddie Mac K-Series \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.01
Freddie Mac K-Series \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.10
Freddie Mac K-Series \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.02
CAFL
Liabilities
ABS issued \| $	$ 2,963,752
CAFL \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input	0.03
CAFL \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input	0.03
CAFL \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
CAFL \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.01
CAFL \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.20
CAFL \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
CAFL \| Default rate \| Minimum
Liabilities
ABS issued, measurement input	0.03
CAFL \| Default rate \| Maximum
Liabilities
ABS issued, measurement input	0.17
CAFL \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.11
CAFL \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input	0.30
CAFL \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input	0.30
CAFL \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input	0.30
Jumbo fixed-rate loans
Assets
ABS issued, net, Fair value \| $	$ 422,819
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Minimum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Maximum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans \| Whole Loan Spread to TBA Price \| Minimum
Assets
Loans held-for-sale, measurement input	3.75
Jumbo fixed-rate loans \| Whole Loan Spread to TBA Price \| Maximum
Assets
Loans held-for-sale, measurement input	3.75
Jumbo fixed-rate loans \| Whole Loan Spread to TBA Price \| Weighted Average
Assets
Loans held-for-sale, measurement input	3.75
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Minimum
Assets
Loans held-for-sale, measurement input	0.0250
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Maximum
Assets
Loans held-for-sale, measurement input	0.0250
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Weighted Average
Assets
Loans held-for-sale, measurement input	0.0250
Jumbo loans committed to sell
Assets
Loans receivable, fair value \| $	$ 573,657
Jumbo loans committed to sell \| Whole loan committed sales price \| Minimum
Assets
Loans held-for-sale, measurement input	100.68
Jumbo loans committed to sell \| Whole loan committed sales price \| Maximum
Assets
Loans held-for-sale, measurement input	103.28
Jumbo loans committed to sell \| Whole loan committed sales price \| Weighted Average
Assets
Loans held-for-sale, measurement input	101.50
Legacy Sequoia
Assets
Loans receivable, fair value \| $	$ 274,861
Sequoia Choice
Assets
Loans receivable, fair value \| $	1,276,112
Liabilities
Fair value of securities owned \| $	226,000
Freddie Mac SLST
Assets
Loans receivable, fair value \| $	2,154,351
Liabilities
Fair value of securities owned \| $	431,000
Single-family rental loans
Assets
Loans receivable, fair value \| $	$ 333,110
Single-family rental loans \| Prepayment rate (annual CPR) \| Minimum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans \| Prepayment rate (annual CPR) \| Maximum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Loans receivable, measurement input	0.03
Single-family rental loans \| Senior credit spread \| Minimum
Assets
Loans receivable, measurement input	0.0075
Single-family rental loans \| Senior credit spread \| Maximum
Assets
Loans receivable, measurement input	0.0075
Single-family rental loans \| Senior credit spread \| Weighted Average
Assets
Loans receivable, measurement input	0.0075
Single-family rental loans \| Subordinate credit spread \| Minimum
Assets
Loans receivable, measurement input	0.0120
Single-family rental loans \| Subordinate credit spread \| Maximum
Assets
Loans receivable, measurement input	0.1598
Single-family rental loans \| Subordinate credit spread \| Weighted Average
Assets
Loans receivable, measurement input	0.0312
Single-family rental loans \| Senior credit support \| Minimum
Assets
Loans receivable, measurement input	0.33
Single-family rental loans \| Senior credit support \| Maximum
Assets
Loans receivable, measurement input	0.33
Single-family rental loans \| Senior credit support \| Weighted Average
Assets
Loans receivable, measurement input	0.33
Single-family rental loans \| IO discount rate \| Minimum
Assets
Loans receivable, measurement input	0.07
Single-family rental loans \| IO discount rate \| Maximum
Assets
Loans receivable, measurement input	0.10
Single-family rental loans \| IO discount rate \| Weighted Average
Assets
Loans receivable, measurement input	0.08
Single-family rental loans \| Non-securitizable loan dollar price \| Minimum
Assets
Loans receivable, measurement input	86
Single-family rental loans \| Non-securitizable loan dollar price \| Maximum
Assets
Loans receivable, measurement input	102
Single-family rental loans \| Non-securitizable loan dollar price \| Weighted Average
Assets
Loans receivable, measurement input	99
CAFL
Assets
Loans receivable, fair value \| $	$ 3,212,526
Liabilities
Fair value of securities owned \| $	261,000
Bridge loans
Assets
Loans receivable, fair value \| $	$ 626,484
Bridge loans \| Discount rate \| Minimum
Assets
Loans held-for-investment, measurement input	0.06
Bridge loans \| Discount rate \| Maximum
Assets
Loans held-for-investment, measurement input	0.15
Bridge loans \| Discount rate \| Weighted Average
Assets
Loans held-for-investment, measurement input	0.08
Freddie Mac K-Series
Assets
Loans receivable, fair value \| $	$ 489,545
Liabilities
Fair value of securities owned \| $	29,000
Trading and AFS securities
Assets
Trading and AFS securities \| $	$ 364,320
Trading and AFS securities \| Prepayment rate (annual CPR) \| Minimum
Assets
Trading and AFS securities	0.07
Trading and AFS securities \| Prepayment rate (annual CPR) \| Maximum
Assets
Trading and AFS securities	0.65
Trading and AFS securities \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Trading and AFS securities	0.28
Trading and AFS securities \| Discount rate \| Minimum
Assets
Trading and AFS securities	0.03
Trading and AFS securities \| Discount rate \| Maximum
Assets
Trading and AFS securities	0.27
Trading and AFS securities \| Discount rate \| Weighted Average
Assets
Trading and AFS securities	0.08
Trading and AFS securities \| Default rate \| Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities \| Default rate \| Maximum
Assets
Trading and AFS securities	0.25
Trading and AFS securities \| Default rate \| Weighted Average
Assets
Trading and AFS securities	0.04
Trading and AFS securities \| Loss severity \| Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities \| Loss severity \| Maximum
Assets
Trading and AFS securities	0.50
Trading and AFS securities \| Loss severity \| Weighted Average
Assets
Trading and AFS securities	0.20
Trading and AFS securities \| Measurement Input, CRT Dollar Price \| Minimum
Assets
Trading and AFS securities	90
Trading and AFS securities \| Measurement Input, CRT Dollar Price \| Maximum
Assets
Trading and AFS securities	112
Trading and AFS securities \| Measurement Input, CRT Dollar Price \| Weighted Average
Assets
Trading and AFS securities	98
Servicer advance investments
Assets
Servicing asset \| $	$ 206,525
Servicer advance investments \| Minimum
Assets
Expected remaining life	4 years
Servicer advance investments \| Maximum
Assets
Expected remaining life	4 years
Servicer advance investments \| Weighted Average
Assets
Expected remaining life	4 years
Servicer advance investments \| Prepayment rate (annual CPR) \| Minimum
Assets
Servicing asset, measurement input	0.20
Servicer advance investments \| Prepayment rate (annual CPR) \| Maximum
Assets
Servicing asset, measurement input	0.30
Servicer advance investments \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Servicing asset, measurement input	0.20
Servicer advance investments \| Discount rate \| Minimum
Assets
Servicing asset, measurement input	0.03
Servicer advance investments \| Discount rate \| Maximum
Assets
Servicing asset, measurement input	0.03
Servicer advance investments \| Discount rate \| Weighted Average
Assets
Servicing asset, measurement input	0.03
Residential loan purchase commitments, net
Liabilities
Loan purchase commitments, net \| $	$ 33,942
Residential loan purchase commitments, net \| Prepayment rate (annual CPR) \| Minimum
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net \| Prepayment rate (annual CPR) \| Maximum
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net \| Whole Loan Spread to TBA Price \| Minimum
Assets
Loans receivable, measurement input	3.75
Residential loan purchase commitments, net \| Whole Loan Spread to TBA Price \| Maximum
Assets
Loans receivable, measurement input	3.75
Residential loan purchase commitments, net \| Whole Loan Spread to TBA Price \| Weighted Average
Assets
Loans receivable, measurement input	3.75
Residential loan purchase commitments, net \| Whole loan spread to swap rate \| Minimum
Assets
Loans receivable, measurement input	0.0250
Residential loan purchase commitments, net \| Whole loan spread to swap rate \| Maximum
Assets
Loans receivable, measurement input	0.0250
Residential loan purchase commitments, net \| Whole loan spread to swap rate \| Weighted Average
Assets
Loans receivable, measurement input	0.0250
Residential loan purchase commitments, net \| Mortgage servicing income \| Minimum
Assets
Servicing asset, measurement input	0
Residential loan purchase commitments, net \| Mortgage servicing income \| Maximum
Assets
Servicing asset, measurement input	0.0014
Residential loan purchase commitments, net \| Mortgage servicing income \| Weighted Average
Assets
Servicing asset, measurement input	0.0008
Residential loan purchase commitments, net \| Committed sales price \| Minimum
Liabilities
Loan purchase commitments, measurement input	98.53
Residential loan purchase commitments, net \| Committed sales price \| Maximum
Liabilities
Loan purchase commitments, measurement input	102.15
Residential loan purchase commitments, net \| Committed sales price \| Weighted Average
Liabilities
Loan purchase commitments, measurement input	100.13
Residential loan purchase commitments, net \| Pull-through rate \| Minimum
Liabilities
Loan purchase commitments, measurement input	0.22
Residential loan purchase commitments, net \| Pull-through rate \| Maximum
Liabilities
Loan purchase commitments, measurement input	1
Residential loan purchase commitments, net \| Pull-through rate \| Weighted Average
Liabilities
Loan purchase commitments, measurement input	0.76
Residential loan purchase commitments, net \| MSR Multiple \| Minimum
Assets
Loans receivable, measurement input	0
Residential loan purchase commitments, net \| MSR Multiple \| Maximum
Assets
Loans receivable, measurement input	3.8
Residential loan purchase commitments, net \| MSR Multiple \| Weighted Average
Assets
Loans receivable, measurement input	2.8
MSRs
Assets
Servicing asset \| $	$ 7,945
MSRs \| Prepayment rate (annual CPR) \| Minimum
Assets
Servicing asset, measurement input	0.09
MSRs \| Prepayment rate (annual CPR) \| Maximum
Assets
Servicing asset, measurement input	1
MSRs \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Servicing asset, measurement input	0.39
MSRs \| Discount rate \| Minimum
Assets
Servicing asset, measurement input	0.12
MSRs \| Discount rate \| Maximum
Assets
Servicing asset, measurement input	0.12
MSRs \| Discount rate \| Weighted Average
Assets
Servicing asset, measurement input	0.12
MSRs \| Per loan annual cost to service \| Minimum
Assets
Servicing asset, measurement input	97
MSRs \| Per loan annual cost to service \| Maximum
Assets
Servicing asset, measurement input	97
MSRs \| Per loan annual cost to service \| Weighted Average
Assets
Servicing asset, measurement input	97
Excess MSRs
Assets
Servicing asset \| $	$ 32,465
Excess MSRs \| Prepayment rate (annual CPR) \| Minimum
Assets
Servicing asset, measurement input	0.20
Excess MSRs \| Prepayment rate (annual CPR) \| Maximum
Assets
Servicing asset, measurement input	0.28
Excess MSRs \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Servicing asset, measurement input	0.23
Excess MSRs \| Discount rate \| Minimum
Assets
Servicing asset, measurement input	0.14
Excess MSRs \| Discount rate \| Maximum
Assets
Servicing asset, measurement input	0.17
Excess MSRs \| Discount rate \| Weighted Average
Assets
Servicing asset, measurement input	0.16
Excess MSRs \| Excess mortgage servicing income \| Minimum
Assets
Servicing asset, measurement input	0.0008
Excess MSRs \| Excess mortgage servicing income \| Maximum
Assets
Servicing asset, measurement input	0.0017
Excess MSRs \| Excess mortgage servicing income \| Weighted Average
Assets
Servicing asset, measurement input	0.0011
Shared home appreciation options
Assets
Shared home appreciation options \| $	$ 45,823
Shared home appreciation options \| Prepayment rate (annual CPR) \| Minimum
Assets
Shared home appreciation options			0.14
Shared home appreciation options \| Prepayment rate (annual CPR) \| Maximum
Assets
Shared home appreciation options			0.24
Shared home appreciation options \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Shared home appreciation options			0.17
Shared home appreciation options \| Discount rate \| Minimum
Assets
Shared home appreciation options	0.13
Shared home appreciation options \| Discount rate \| Maximum
Assets
Shared home appreciation options	0.13
Shared home appreciation options \| Discount rate \| Weighted Average
Assets
Shared home appreciation options	0.13
Shared home appreciation options \| Home price appreciation \| Minimum
Assets
Shared home appreciation options			0.03
Shared home appreciation options \| Home price appreciation \| Maximum
Assets
Shared home appreciation options			0.04
Shared home appreciation options \| Home price appreciation \| Weighted Average
Assets
Shared home appreciation options			0.03
REO
Assets
REO \| $	$ 1,316
REO \| Loss severity \| Minimum
Assets
REO, measurement input	0.04
REO \| Loss severity \| Maximum
Assets
REO, measurement input	0.53
REO \| Loss severity \| Weighted Average
Assets
REO, measurement input	0.25