Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details)
$ in Thousands

Fair Value of Financial Instruments - Quantitative Information about Significant Unobservable Inputs Used in Valuation of Level 3 Assets and Liabilities Measured at Fair Value (Details) $ in Thousands	6 Months Ended
	Jun. 30, 2019 USD ($)	Jun. 30, 2019	Jun. 30, 2019 $ / loan	Dec. 31, 2018 USD ($)
Liabilities
Contingent consideration \| $	$ 24,932			$ 0
ABS Issued
Liabilities
ABS issued \| $	2,378,306
ABS Issued \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input		0.08
ABS Issued \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input		0.49
ABS Issued \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input		0.19
ABS Issued \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input		0.03
ABS Issued \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input		0.15
ABS Issued \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input		0.04
ABS Issued \| Default rate \| Minimum
Liabilities
ABS issued, measurement input		0
ABS Issued \| Default rate \| Maximum
Liabilities
ABS issued, measurement input		0.09
ABS Issued \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input		0.02
ABS Issued \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input		0.20
ABS Issued \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input		0.22
ABS Issued \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input		0.21
Freddie Mac SLST
Liabilities
ABS issued \| $	991,766
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input		0.06
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input		0.06
Freddie Mac SLST \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input		0.06
Freddie Mac SLST \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input		0.03
Freddie Mac SLST \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input		0.08
Freddie Mac SLST \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input		0.03
Freddie Mac SLST \| Default rate \| Minimum
Liabilities
ABS issued, measurement input		0.02
Freddie Mac SLST \| Default rate \| Maximum
Liabilities
ABS issued, measurement input		0.02
Freddie Mac SLST \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input		0.02
Freddie Mac SLST \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input		0.30
Freddie Mac SLST \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input		0.30
Freddie Mac SLST \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input		0.30
Freddie Mac K-Series
Liabilities
ABS issued \| $	3,543,057
Freddie Mac K-Series \| Prepayment rate (annual CPR) \| Minimum
Liabilities
ABS issued, measurement input		0
Freddie Mac K-Series \| Prepayment rate (annual CPR) \| Maximum
Liabilities
ABS issued, measurement input		0
Freddie Mac K-Series \| Prepayment rate (annual CPR) \| Weighted Average
Liabilities
ABS issued, measurement input		0
Freddie Mac K-Series \| Discount rate \| Minimum
Liabilities
ABS issued, measurement input		0.02
Freddie Mac K-Series \| Discount rate \| Maximum
Liabilities
ABS issued, measurement input		0.09
Freddie Mac K-Series \| Discount rate \| Weighted Average
Liabilities
ABS issued, measurement input		0.03
Freddie Mac K-Series \| Default rate \| Minimum
Liabilities
ABS issued, measurement input		0.01
Freddie Mac K-Series \| Default rate \| Maximum
Liabilities
ABS issued, measurement input		0.01
Freddie Mac K-Series \| Default rate \| Weighted Average
Liabilities
ABS issued, measurement input		0.01
Freddie Mac K-Series \| Loss severity \| Minimum
Liabilities
ABS issued, measurement input		0.20
Freddie Mac K-Series \| Loss severity \| Maximum
Liabilities
ABS issued, measurement input		0.20
Freddie Mac K-Series \| Loss severity \| Weighted Average
Liabilities
ABS issued, measurement input		0.20
Loan purchase commitments
Assets
Loan purchase commitments, net \| $	4,707
Loan purchase commitments \| Prepayment rate (annual CPR) \| Minimum
Assets
Loan purchase commitments, net, measurement input		0.15
Loan purchase commitments \| Prepayment rate (annual CPR) \| Maximum
Assets
Loan purchase commitments, net, measurement input		0.20
Loan purchase commitments \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Loan purchase commitments, net, measurement input		0.18
Loan purchase commitments \| Whole loan spread to TBA price \| Minimum
Assets
Loan purchase commitments, net, measurement input			0.61
Loan purchase commitments \| Whole loan spread to TBA price \| Maximum
Assets
Loan purchase commitments, net, measurement input			1.55
Loan purchase commitments \| Whole loan spread to TBA price \| Weighted Average
Assets
Loan purchase commitments, net, measurement input			1.54
Loan purchase commitments \| MSR multiple \| Minimum
Assets
Loan purchase commitments, net, measurement input		0.7
Loan purchase commitments \| MSR multiple \| Maximum
Assets
Loan purchase commitments, net, measurement input		4.6
Loan purchase commitments \| MSR multiple \| Weighted Average
Assets
Loan purchase commitments, net, measurement input		2.2
Loan purchase commitments \| Pull-through rate \| Minimum
Assets
Loan purchase commitments, net, measurement input		0.07
Loan purchase commitments \| Pull-through rate \| Maximum
Assets
Loan purchase commitments, net, measurement input		1
Loan purchase commitments \| Pull-through rate \| Weighted Average
Assets
Loan purchase commitments, net, measurement input		0.70
Loan purchase commitments \| Mortgage servicing amount \| Minimum
Assets
Loan purchase commitments, net, measurement input		0.0095	0.0006
Loan purchase commitments \| Mortgage servicing amount \| Maximum
Assets
Loan purchase commitments, net, measurement input		0.0365	0.0014
Loan purchase commitments \| Mortgage servicing amount \| Weighted Average
Assets
Loan purchase commitments, net, measurement input		0.0228	0.001
Loan purchase commitments \| Whole loan spread to swap - hybrid \| Minimum
Assets
Loan purchase commitments, net, measurement input		0.0085
Loan purchase commitments \| Whole loan spread to swap - hybrid \| Maximum
Assets
Loan purchase commitments, net, measurement input		0.0345
Loan purchase commitments \| Whole loan spread to swap - hybrid \| Weighted Average
Assets
Loan purchase commitments, net, measurement input		0.0129
Contingent consideration
Liabilities
Contingent consideration \| $	24,932
Contingent consideration \| Discount rate \| Minimum
Liabilities
Contingent consideration, measurement input			0.23
Contingent consideration \| Discount rate \| Maximum
Liabilities
Contingent consideration, measurement input			0.23
Contingent consideration \| Discount rate \| Weighted Average
Liabilities
Contingent consideration, measurement input			0.23
Contingent consideration \| Probability of outcomes \| Minimum
Liabilities
Contingent consideration, measurement input			0
Contingent consideration \| Probability of outcomes \| Maximum
Liabilities
Contingent consideration, measurement input			1
Contingent consideration \| Probability of outcomes \| Weighted Average
Liabilities
Contingent consideration, measurement input			0.90
Jumbo fixed-rate loans
Assets
Residential loans, at fair value \| $	2,724,429
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Minimum
Assets
Residential loans, measurement input		0.20
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Maximum
Assets
Residential loans, measurement input		0.20
Jumbo fixed-rate loans \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Residential loans, measurement input		0.20
Jumbo fixed-rate loans \| Whole loan spread to TBA price \| Minimum
Assets
Residential loans, measurement input			1.55
Jumbo fixed-rate loans \| Whole loan spread to TBA price \| Maximum
Assets
Residential loans, measurement input			1.55
Jumbo fixed-rate loans \| Whole loan spread to TBA price \| Weighted Average
Assets
Residential loans, measurement input			1.55
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Minimum
Assets
Residential loans, measurement input			0.0080
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Maximum
Assets
Residential loans, measurement input			0.0365
Jumbo fixed-rate loans \| Whole loan spread to swap rate \| Weighted Average
Assets
Residential loans, measurement input			0.0180
Jumbo hybrid loans
Assets
Residential loans, at fair value \| $	344,151
Jumbo hybrid loans \| Prepayment rate (annual CPR) \| Minimum
Assets
Residential loans, measurement input		0.15
Jumbo hybrid loans \| Prepayment rate (annual CPR) \| Maximum
Assets
Residential loans, measurement input		0.15
Jumbo hybrid loans \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Residential loans, measurement input		0.15
Jumbo hybrid loans \| Whole loan spread to swap rate \| Minimum
Assets
Residential loans, measurement input			0.0065
Jumbo hybrid loans \| Whole loan spread to swap rate \| Maximum
Assets
Residential loans, measurement input			0.0360
Jumbo hybrid loans \| Whole loan spread to swap rate \| Weighted Average
Assets
Residential loans, measurement input			0.0141
Jumbo loans committed to sell
Assets
Residential loans, at fair value \| $	374,481
Jumbo loans committed to sell \| Whole loan committed sales price \| Minimum
Assets
Residential loans, measurement input			101.84
Jumbo loans committed to sell \| Whole loan committed sales price \| Maximum
Assets
Residential loans, measurement input			103.08
Jumbo loans committed to sell \| Whole loan committed sales price \| Weighted Average
Assets
Residential loans, measurement input			102.12
Legacy Sequoia
Assets
Residential loans, at fair value \| $	457,750
Sequoia Choice
Assets
Residential loans, at fair value \| $	2,147,356
Freddie Mac SLST
Assets
Residential loans, at fair value \| $	1,235,089
Business purpose residential loans, at fair value
Assets
Residential loans, at fair value \| $	91,501
Business purpose residential loans, at fair value \| Prepayment rate (annual CPR) \| Minimum
Assets
Residential loans, measurement input		0.02
Business purpose residential loans, at fair value \| Prepayment rate (annual CPR) \| Maximum
Assets
Residential loans, measurement input		0.10
Business purpose residential loans, at fair value \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Residential loans, measurement input		0.05
Business purpose residential loans, at fair value \| Discount rate \| Minimum
Assets
Residential loans, measurement input		0.12
Business purpose residential loans, at fair value \| Discount rate \| Maximum
Assets
Residential loans, measurement input		0.12
Business purpose residential loans, at fair value \| Discount rate \| Weighted Average
Assets
Residential loans, measurement input		0.12
Business purpose residential loans, at fair value \| Senior credit spread \| Minimum
Assets
Residential loans, measurement input			0.0095
Business purpose residential loans, at fair value \| Senior credit spread \| Maximum
Assets
Residential loans, measurement input			0.0095
Business purpose residential loans, at fair value \| Senior credit spread \| Weighted Average
Assets
Residential loans, measurement input			0.0095
Business purpose residential loans, at fair value \| Subordinate credit spread \| Minimum
Assets
Residential loans, measurement input			0.0140
Business purpose residential loans, at fair value \| Subordinate credit spread \| Maximum
Assets
Residential loans, measurement input			0.1200
Business purpose residential loans, at fair value \| Subordinate credit spread \| Weighted Average
Assets
Residential loans, measurement input			0.0306
Business purpose residential loans, at fair value \| Senior credit support \| Minimum
Assets
MSR, measurement input		0.35
Business purpose residential loans, at fair value \| Senior credit support \| Maximum
Assets
MSR, measurement input		0.35
Business purpose residential loans, at fair value \| Senior credit support \| Weighted Average
Assets
MSR, measurement input		0.35
Residential bridge loans
Assets
Residential loans, at fair value \| $	159,353
Residential bridge loans \| Discount rate \| Minimum
Assets
Residential loans, measurement input		0.07
Residential bridge loans \| Discount rate \| Maximum
Assets
Residential loans, measurement input		0.08
Residential bridge loans \| Discount rate \| Weighted Average
Assets
Residential loans, measurement input		0.07
Freddie Mac K-Series
Assets
Residential loans, at fair value \| $	3,749,657
Trading and AFS securities
Assets
Trading and AFS securities \| $	1,477,486
Trading and AFS securities \| Prepayment rate (annual CPR) \| Minimum
Assets
Trading and AFS securities		0
Trading and AFS securities \| Prepayment rate (annual CPR) \| Maximum
Assets
Trading and AFS securities		0.60
Trading and AFS securities \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Trading and AFS securities		0.10
Trading and AFS securities \| Discount rate \| Minimum
Assets
Trading and AFS securities		0.03
Trading and AFS securities \| Discount rate \| Maximum
Assets
Trading and AFS securities		0.14
Trading and AFS securities \| Discount rate \| Weighted Average
Assets
Trading and AFS securities		0.05
Trading and AFS securities \| Default rate \| Minimum
Assets
Trading and AFS securities		0
Trading and AFS securities \| Default rate \| Maximum
Assets
Trading and AFS securities		0.20
Trading and AFS securities \| Default rate \| Weighted Average
Assets
Trading and AFS securities		0.02
Trading and AFS securities \| Loss severity \| Minimum
Assets
Trading and AFS securities		0
Trading and AFS securities \| Loss severity \| Maximum
Assets
Trading and AFS securities		0.40
Trading and AFS securities \| Loss severity \| Weighted Average
Assets
Trading and AFS securities		0.21
Servicer advance investments
Assets
Mortgage servicing rights \| $	$ 259,222
Servicer advance investments \| Minimum
Assets
Expected remaining life	2 years
Servicer advance investments \| Maximum
Assets
Expected remaining life	2 years
Servicer advance investments \| Weighted Average
Assets
Expected remaining life	2 years
Servicer advance investments \| Prepayment rate (annual CPR) \| Minimum
Assets
MSR, measurement input		0.08
Servicer advance investments \| Prepayment rate (annual CPR) \| Maximum
Assets
MSR, measurement input		0.15
Servicer advance investments \| Prepayment rate (annual CPR) \| Weighted Average
Assets
MSR, measurement input		0.14
Servicer advance investments \| Discount rate \| Minimum
Assets
MSR, measurement input		0.04
Servicer advance investments \| Discount rate \| Maximum
Assets
MSR, measurement input		0.04
Servicer advance investments \| Discount rate \| Weighted Average
Assets
MSR, measurement input		0.04
MSRs
Assets
Mortgage servicing rights \| $	$ 47,396
MSRs \| Prepayment rate (annual CPR) \| Minimum
Assets
MSR, measurement input		0.05
MSRs \| Prepayment rate (annual CPR) \| Maximum
Assets
MSR, measurement input		0.45
MSRs \| Prepayment rate (annual CPR) \| Weighted Average
Assets
MSR, measurement input		0.12
MSRs \| Discount rate \| Minimum
Assets
MSR, measurement input		0.11
MSRs \| Discount rate \| Maximum
Assets
MSR, measurement input		0.17
MSRs \| Discount rate \| Weighted Average
Assets
MSR, measurement input		0.11
MSRs \| Per loan annual cost to service \| Minimum
Assets
MSR, measurement input			82
MSRs \| Per loan annual cost to service \| Maximum
Assets
MSR, measurement input			82
MSRs \| Per loan annual cost to service \| Weighted Average
Assets
MSR, measurement input			82
Excess MSRs
Assets
Mortgage servicing rights \| $	33,620
Excess MSRs \| Prepayment rate (annual CPR) \| Minimum
Assets
MSR, measurement input		0.08
Excess MSRs \| Prepayment rate (annual CPR) \| Maximum
Assets
MSR, measurement input		0.14
Excess MSRs \| Prepayment rate (annual CPR) \| Weighted Average
Assets
MSR, measurement input		0.11
Excess MSRs \| Discount rate \| Minimum
Assets
MSR, measurement input		0.11
Excess MSRs \| Discount rate \| Maximum
Assets
MSR, measurement input		0.16
Excess MSRs \| Discount rate \| Weighted Average
Assets
MSR, measurement input		0.14
Excess MSRs \| Excess mortgage servicing income \| Minimum
Assets
MSR, measurement input			0.0007
Excess MSRs \| Excess mortgage servicing income \| Maximum
Assets
MSR, measurement input			0.0017
Excess MSRs \| Excess mortgage servicing income \| Weighted Average
Assets
MSR, measurement input			0.0012
Guarantee asset
Assets
Guarantee asset \| $	1,999
Guarantee asset \| Prepayment rate (annual CPR) \| Minimum
Assets
Guarantee asset		0.15
Guarantee asset \| Prepayment rate (annual CPR) \| Maximum
Assets
Guarantee asset		0.15
Guarantee asset \| Prepayment rate (annual CPR) \| Weighted Average
Assets
Guarantee asset		0.15
Guarantee asset \| Discount rate \| Minimum
Assets
Guarantee asset		0.11
Guarantee asset \| Discount rate \| Maximum
Assets
Guarantee asset		0.11
Guarantee asset \| Discount rate \| Weighted Average
Assets
Guarantee asset		0.11
REO
Assets
REO \| $	$ 5,732
REO \| Loss severity \| Minimum
Assets
REO, measurement input		0.03
REO \| Loss severity \| Maximum
Assets
REO, measurement input		0.52
REO \| Loss severity \| Weighted Average
Assets
REO, measurement input		0.07