Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details)
$ in Thousands

Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	6 Months Ended
	Jun. 30, 2021 USD ($) $ / loan	Dec. 31, 2020 USD ($)	Jun. 30, 2020
Assets
ABS issued, net, Fair value \| $	$ 7,360,766	$ 6,900,362
ABS Issued
Liabilities
ABS issued \| $	$ 2,248,759
ABS Issued \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input	0.07
ABS Issued \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input	0.51
ABS Issued \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input	0.33
ABS Issued \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.01
ABS Issued \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.18
ABS Issued \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
ABS Issued \| Default rate \| Minimum
Liabilities
ABS issued, measurement input	0
ABS Issued \| Default rate \| Maximum
Liabilities
ABS issued, measurement input	0.39
ABS Issued \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.02
ABS Issued \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input	0.25
ABS Issued \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input	0.50
ABS Issued \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input	0.32
Freddie Mac SLST
Liabilities
ABS issued \| $	$ 1,650,087
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input	0.08
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input	0.06
Freddie Mac SLST \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.02
Freddie Mac SLST \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.07
Freddie Mac SLST \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
Freddie Mac SLST \| Default rate \| Minimum
Liabilities
ABS issued, measurement input	0.09
Freddie Mac SLST \| Default rate \| Maximum
Liabilities
ABS issued, measurement input	0.10
Freddie Mac SLST \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.09
Freddie Mac SLST \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input	0.35
Freddie Mac SLST \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input	0.35
Freddie Mac K-Series
Liabilities
ABS issued \| $	$ 454,324
Freddie Mac K-Series \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.01
Freddie Mac K-Series \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.09
Freddie Mac K-Series \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.02
CAFL
Liabilities
ABS issued \| $	$ 3,007,596
CAFL \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input	0.03
CAFL \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input	0.03
CAFL \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
CAFL \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input	0.01
CAFL \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input	0.13
CAFL \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.03
CAFL \| Default rate \| Minimum
Liabilities
ABS issued, measurement input	0.03
CAFL \| Default rate \| Maximum
Liabilities
ABS issued, measurement input	0.18
CAFL \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input	0.09
CAFL \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input	0.30
CAFL \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input	0.30
CAFL \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input	0.30
Single Family Rental Interest Rate Lock Commitments
Liabilities
Loan commitments \| $	$ 744
Single Family Rental Interest Rate Lock Commitments \| Prepayment rate (annual CPR) \| Minimum
Liabilities
Loan liabilities, measurement input	0.03
Single Family Rental Interest Rate Lock Commitments \| Prepayment rate (annual CPR) \| Maximum
Liabilities
Loan liabilities, measurement input	0.03
Single Family Rental Interest Rate Lock Commitments \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
Loan liabilities, measurement input	0.03
Single Family Rental Interest Rate Lock Commitments \| Senior credit spread \| Minimum
Liabilities
Loan liabilities, measurement input	0.0070
Single Family Rental Interest Rate Lock Commitments \| Senior credit spread \| Maximum
Liabilities
Loan liabilities, measurement input	0.0070
Single Family Rental Interest Rate Lock Commitments \| Senior credit spread \| Weighted Average
Liabilities
Loan liabilities, measurement input	0.0070
Single Family Rental Interest Rate Lock Commitments \| Subordinate credit spread \| Minimum
Liabilities
Loan liabilities, measurement input	0.0105
Single Family Rental Interest Rate Lock Commitments \| Subordinate credit spread \| Maximum
Liabilities
Loan liabilities, measurement input	0.1531
Single Family Rental Interest Rate Lock Commitments \| Subordinate credit spread \| Weighted Average
Liabilities
Loan liabilities, measurement input	0.0391
Single Family Rental Interest Rate Lock Commitments \| Senior credit support \| Minimum
Liabilities
Loan liabilities, measurement input	0.34
Single Family Rental Interest Rate Lock Commitments \| Senior credit support \| Maximum
Liabilities
Loan liabilities, measurement input	0.34
Single Family Rental Interest Rate Lock Commitments \| Senior credit support \| Weighted Average
Liabilities
Loan liabilities, measurement input	0.34
Single Family Rental Interest Rate Lock Commitments \| IO discount rate \| Minimum
Liabilities
Loan liabilities, measurement input	0.10
Single Family Rental Interest Rate Lock Commitments \| IO discount rate \| Maximum
Liabilities
Loan liabilities, measurement input	0.11
Single Family Rental Interest Rate Lock Commitments \| IO discount rate \| Weighted Average
Liabilities
Loan liabilities, measurement input	0.10
Single Family Rental Interest Rate Lock Commitments \| Pull-through rate \| Minimum
Liabilities
Loan liabilities, measurement input	1
Single Family Rental Interest Rate Lock Commitments \| Pull-through rate \| Maximum
Liabilities
Loan liabilities, measurement input	1
Single Family Rental Interest Rate Lock Commitments \| Pull-through rate \| Weighted Average
Liabilities
Loan liabilities, measurement input	1
Jumbo fixed-rate loans
Assets
ABS issued, net, Fair value \| $	$ 232,343
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Minimum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Maximum
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Loans held-for-sale, measurement input	0.20
Jumbo fixed-rate loans \| Whole Loan Spread to TBA Price \| Minimum
Assets
Loans held-for-sale, measurement input	2.00
Jumbo fixed-rate loans \| Whole Loan Spread to TBA Price \| Maximum
Assets
Loans held-for-sale, measurement input	2.00
Jumbo fixed-rate loans \| Whole Loan Spread to TBA Price \| Weighted Average
Assets
Loans held-for-sale, measurement input	2.00
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Minimum
Assets
Loans held-for-sale, measurement input	0.0215
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Maximum
Assets
Loans held-for-sale, measurement input	0.0215
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Weighted Average
Assets
Loans held-for-sale, measurement input	0.0215
Jumbo loans committed to sell
Assets
Loans receivable, fair value \| $	$ 928,170
Jumbo loans committed to sell \| Whole loan committed sales price \| Minimum
Assets
Loans held-for-sale, measurement input	100.88
Jumbo loans committed to sell \| Whole loan committed sales price \| Maximum
Assets
Loans held-for-sale, measurement input	103.03
Jumbo loans committed to sell \| Whole loan committed sales price \| Weighted Average
Assets
Loans held-for-sale, measurement input	102.38
Legacy Sequoia
Assets
Loans receivable, fair value \| $	$ 260,875
Sequoia
Assets
Loans receivable, fair value \| $	2,222,553
Liabilities
Fair value of securities owned \| $	235,000
Freddie Mac SLST
Assets
Loans receivable, fair value \| $	2,098,624
Liabilities
Fair value of securities owned \| $	450,000
Single-family rental loans
Assets
Loans receivable, fair value \| $	$ 418,442
Single-family rental loans \| Prepayment rate (annual CPR) \| Minimum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans \| Prepayment rate (annual CPR) \| Maximum
Assets
Loans receivable, measurement input	0.03
Single-family rental loans \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Loans receivable, measurement input	0.03
Single-family rental loans \| Senior credit spread \| Minimum
Assets
Loans receivable, measurement input	0.0070
Single-family rental loans \| Senior credit spread \| Maximum
Assets
Loans receivable, measurement input	0.0070
Single-family rental loans \| Senior credit spread \| Weighted Average
Assets
Loans receivable, measurement input	0.0070
Single-family rental loans \| Subordinate credit spread \| Minimum
Assets
Loans receivable, measurement input	0.0105
Single-family rental loans \| Subordinate credit spread \| Maximum
Assets
Loans receivable, measurement input	0.1531
Single-family rental loans \| Subordinate credit spread \| Weighted Average
Assets
Loans receivable, measurement input	0.0391
Single-family rental loans \| Senior credit support \| Minimum
Assets
Loans receivable, measurement input	0.34
Single-family rental loans \| Senior credit support \| Maximum
Assets
Loans receivable, measurement input	0.34
Single-family rental loans \| Senior credit support \| Weighted Average
Assets
Loans receivable, measurement input	0.34
Single-family rental loans \| IO discount rate \| Minimum
Assets
Loans receivable, measurement input	0.09
Single-family rental loans \| IO discount rate \| Maximum
Assets
Loans receivable, measurement input	0.09
Single-family rental loans \| IO discount rate \| Weighted Average
Assets
Loans receivable, measurement input	0.09
Single-family rental loans \| Non-securitizable loan dollar price \| Minimum
Assets
Loans receivable, measurement input	82
Single-family rental loans \| Non-securitizable loan dollar price \| Maximum
Assets
Loans receivable, measurement input	102
Single-family rental loans \| Non-securitizable loan dollar price \| Weighted Average
Assets
Loans receivable, measurement input	99
CAFL
Assets
Loans receivable, fair value \| $	$ 3,263,878
Liabilities
Fair value of securities owned \| $	268,000
Bridge loans
Assets
Loans receivable, fair value \| $	$ 726,569
Bridge loans \| Discount rate \| Minimum
Assets
Loans held-for-investment, measurement input	0.06
Bridge loans \| Discount rate \| Maximum
Assets
Loans held-for-investment, measurement input	0.15
Bridge loans \| Discount rate \| Weighted Average
Assets
Loans held-for-investment, measurement input	0.08
Freddie Mac K-Series
Assets
Loans receivable, fair value \| $	$ 485,157
Liabilities
Fair value of securities owned \| $	31,000
Trading and AFS securities
Assets
Trading and AFS securities \| $	$ 354,886
Trading and AFS securities \| Prepayment rate (annual CPR) \| Minimum
Assets
Trading and AFS securities	0.08
Trading and AFS securities \| Prepayment rate (annual CPR) \| Maximum
Assets
Trading and AFS securities	0.62
Trading and AFS securities \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Trading and AFS securities	0.28
Trading and AFS securities \| Discount rate \| Minimum
Assets
Trading and AFS securities	0.02
Trading and AFS securities \| Discount rate \| Maximum
Assets
Trading and AFS securities	0.31
Trading and AFS securities \| Discount rate \| Weighted Average
Assets
Trading and AFS securities	0.07
Trading and AFS securities \| Default rate \| Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities \| Default rate \| Maximum
Assets
Trading and AFS securities	0.25
Trading and AFS securities \| Default rate \| Weighted Average
Assets
Trading and AFS securities	0.04
Trading and AFS securities \| Loss severity \| Minimum
Assets
Trading and AFS securities	0
Trading and AFS securities \| Loss severity \| Maximum
Assets
Trading and AFS securities	0.50
Trading and AFS securities \| Loss severity \| Weighted Average
Assets
Trading and AFS securities	0.22
Trading and AFS securities \| Measurement Input, CRT Dollar Price \| Minimum
Assets
Trading and AFS securities	95
Trading and AFS securities \| Measurement Input, CRT Dollar Price \| Maximum
Assets
Trading and AFS securities	113
Trading and AFS securities \| Measurement Input, CRT Dollar Price \| Weighted Average
Assets
Trading and AFS securities	102
Servicer Advance Investments
Assets
Servicing asset \| $	$ 184,551
Servicer Advance Investments \| Minimum
Assets
Expected remaining life	4 years
Servicer Advance Investments \| Maximum
Assets
Expected remaining life	4 years
Servicer Advance Investments \| Weighted Average
Assets
Expected remaining life	4 years
Servicer Advance Investments \| Prepayment rate (annual CPR) \| Minimum
Assets
Servicing asset, measurement input	0.20
Servicer Advance Investments \| Prepayment rate (annual CPR) \| Maximum
Assets
Servicing asset, measurement input	0.30
Servicer Advance Investments \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Servicing asset, measurement input	0.21
Servicer Advance Investments \| Discount rate \| Minimum
Assets
Servicing asset, measurement input	0.03
Servicer Advance Investments \| Discount rate \| Maximum
Assets
Servicing asset, measurement input	0.03
Servicer Advance Investments \| Discount rate \| Weighted Average
Assets
Servicing asset, measurement input	0.03
Residential loan purchase commitments, net
Liabilities
Loan purchase commitments, net \| $	$ 12,725
Residential loan purchase commitments, net \| Prepayment rate (annual CPR) \| Minimum
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net \| Prepayment rate (annual CPR) \| Maximum
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Loans receivable, measurement input	0.20
Residential loan purchase commitments, net \| Whole Loan Spread to TBA Price \| Minimum
Assets
Loans receivable, measurement input	2.00
Residential loan purchase commitments, net \| Whole Loan Spread to TBA Price \| Maximum
Assets
Loans receivable, measurement input	2.00
Residential loan purchase commitments, net \| Whole Loan Spread to TBA Price \| Weighted Average
Assets
Loans receivable, measurement input	2.00
Residential loan purchase commitments, net \| Whole loan spread to swap rate \| Minimum
Assets
Loans receivable, measurement input	0.0191
Residential loan purchase commitments, net \| Whole loan spread to swap rate \| Maximum
Assets
Loans receivable, measurement input	0.0215
Residential loan purchase commitments, net \| Whole loan spread to swap rate \| Weighted Average
Assets
Loans receivable, measurement input	0.0201
Residential loan purchase commitments, net \| Mortgage servicing income \| Minimum
Assets
Servicing asset, measurement input	0
Residential loan purchase commitments, net \| Mortgage servicing income \| Maximum
Assets
Servicing asset, measurement input	0.0015
Residential loan purchase commitments, net \| Mortgage servicing income \| Weighted Average
Assets
Servicing asset, measurement input	0.0009
Residential loan purchase commitments, net \| Committed sales price \| Minimum
Liabilities
Loan liabilities, measurement input	102.00
Residential loan purchase commitments, net \| Committed sales price \| Maximum
Liabilities
Loan liabilities, measurement input	103.03
Residential loan purchase commitments, net \| Committed sales price \| Weighted Average
Liabilities
Loan liabilities, measurement input	102.64
Residential loan purchase commitments, net \| Pull-through rate \| Minimum
Liabilities
Loan liabilities, measurement input	0.21
Residential loan purchase commitments, net \| Pull-through rate \| Maximum
Liabilities
Loan liabilities, measurement input	1
Residential loan purchase commitments, net \| Pull-through rate \| Weighted Average
Liabilities
Loan liabilities, measurement input	0.72
MSRs
Assets
Servicing asset \| $	$ 8,721
MSRs \| Prepayment rate (annual CPR) \| Minimum
Assets
Servicing asset, measurement input	0.07
MSRs \| Prepayment rate (annual CPR) \| Maximum
Assets
Servicing asset, measurement input	0.84
MSRs \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Servicing asset, measurement input	0.36
MSRs \| Discount rate \| Minimum
Assets
Servicing asset, measurement input	0.12
MSRs \| Discount rate \| Maximum
Assets
Servicing asset, measurement input	0.12
MSRs \| Discount rate \| Weighted Average
Assets
Servicing asset, measurement input	0.12
MSRs \| Per loan annual cost to service \| Minimum
Assets
Servicing asset, measurement input	96
MSRs \| Per loan annual cost to service \| Maximum
Assets
Servicing asset, measurement input	96
MSRs \| Per loan annual cost to service \| Weighted Average
Assets
Servicing asset, measurement input	96
Excess MSRs
Assets
Servicing asset \| $	$ 29,988
Excess MSRs \| Prepayment rate (annual CPR) \| Minimum
Assets
Servicing asset, measurement input	0.21
Excess MSRs \| Prepayment rate (annual CPR) \| Maximum
Assets
Servicing asset, measurement input	0.30
Excess MSRs \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Servicing asset, measurement input	0.24
Excess MSRs \| Discount rate \| Minimum
Assets
Servicing asset, measurement input	0.13
Excess MSRs \| Discount rate \| Maximum
Assets
Servicing asset, measurement input	0.16
Excess MSRs \| Discount rate \| Weighted Average
Assets
Servicing asset, measurement input	0.15
Excess MSRs \| Excess mortgage servicing income \| Minimum
Assets
Servicing asset, measurement input	0.0008
Excess MSRs \| Excess mortgage servicing income \| Maximum
Assets
Servicing asset, measurement input	0.0017
Excess MSRs \| Excess mortgage servicing income \| Weighted Average
Assets
Servicing asset, measurement input	0.0011
Shared home appreciation options
Assets
Shared home appreciation options \| $	$ 44,319
Shared home appreciation options \| Prepayment rate (annual CPR) \| Minimum
Assets
Shared home appreciation options			0.16
Shared home appreciation options \| Prepayment rate (annual CPR) \| Maximum
Assets
Shared home appreciation options			0.24
Shared home appreciation options \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Shared home appreciation options			0.17
Shared home appreciation options \| Discount rate \| Minimum
Assets
Shared home appreciation options	0.13
Shared home appreciation options \| Discount rate \| Maximum
Assets
Shared home appreciation options	0.13
Shared home appreciation options \| Discount rate \| Weighted Average
Assets
Shared home appreciation options	0.13
Shared home appreciation options \| Home price appreciation \| Minimum
Assets
Shared home appreciation options			0.03
Shared home appreciation options \| Home price appreciation \| Maximum
Assets
Shared home appreciation options			0.04
Shared home appreciation options \| Home price appreciation \| Weighted Average
Assets
Shared home appreciation options			0.03
REO
Assets
REO \| $	$ 1,233
REO \| Loss severity \| Minimum
Assets
REO, measurement input	0.04
REO \| Loss severity \| Maximum
Assets
REO, measurement input	0.40
REO \| Loss severity \| Weighted Average
Assets
REO, measurement input	0.23