Schedule IV - Mortgage Loans on Real Estate (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2020 USD ($) loans	Dec. 31, 2019 USD ($)	Dec. 31, 2018 USD ($)	Dec. 31, 2020 USD ($)
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Value	$ 3,049,166	$ 12,707,625	$ 10,157,126	$ 3,049,166
Principal Amount of Loans Subject to Delinquent Principal or Interest				406,293
Reconciliation of Balance Sheet Reported Amounts of Mortgage Loans on Real Estate
Beginning balance	20,780,548	12,707,625	10,157,126
Additions during period:
Purchases	569,557	8,762,553	2,983,295
Accretion of purchase discount	5,265	11,234	19,940
Change in realized and unrealized gains	101,957	638,557	4,096
Deductions during period:
Repayments of principal	(674,337)	(1,052,812)	(182,163)
Collection of interest	0	(11,429)	(21,754)
Transfer to investment securities available for sale (2)	(237,297)	0	0
Transfer to REO	(8,509)	(6,105)	(7,998)
Cost of loans sold	(17,478,478)	(213,871)	(109,000)
Provision for loan loss	0	2,780	(1,235)
Change in realized and unrealized losses	0	0	(85,115)
Amortization of premium	(15,352)	(57,984)	(49,567)
Balance at end of period	3,049,166	20,780,548	12,707,625
Cumulative Effect, Period of Adoption, Adjustment
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Value	5,812	0	0
Reconciliation of Balance Sheet Reported Amounts of Mortgage Loans on Real Estate
Beginning balance	$ 5,812	0	0
Deductions during period:
Balance at end of period		$ 5,812	$ 0
Residential Loans Held in Securitization Trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	1,204
Carrying Value	$ 64,247			64,247
Principal Amount of Loans Subject to Delinquent Principal or Interest				8,744
Deductions during period:
Balance at end of period	$ 64,247
Residential Loans Held in Securitization Trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.63%
Residential Loans Held in Securitization Trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	13.33%
Mortgage Loans Under $100,000 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	1,335
Carrying Value	$ 66,968			66,968
Principal Amount of Loans Subject to Delinquent Principal or Interest				6,285
Deductions during period:
Balance at end of period	$ 66,968
Mortgage Loans Under $100,000 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.38%
Mortgage Loans Under $100,000 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	14.99%
Mortgage Loans Under $100,000 \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	421
Carrying Value	$ 18,222			18,222
Principal Amount of Loans Subject to Delinquent Principal or Interest				579
Deductions during period:
Balance at end of period	$ 18,222
Mortgage Loans Under $100,000 \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.75%
Mortgage Loans Under $100,000 \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.00%
Mortgage Loans Under $100,000 \| Business Purpose Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	176
Carrying Value	$ 20,610			20,610
Principal Amount of Loans Subject to Delinquent Principal or Interest				160
Deductions during period:
Balance at end of period	$ 20,610
Mortgage Loans Under $100,000 \| Business Purpose Loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	7.75%
Mortgage Loans Under $100,000 \| Business Purpose Loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	15.00%
Mortgage Loans Under $100,000 \| Residential Loans Held in Securitization Trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	1,591
Carrying Value	$ 181,487			181,487
Principal Amount of Loans Subject to Delinquent Principal or Interest				21,816
Deductions during period:
Balance at end of period	$ 181,487
Mortgage Loans Under $100,000 \| Residential Loans Held in Securitization Trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.88%
Mortgage Loans Under $100,000 \| Residential Loans Held in Securitization Trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.80%
Mortgage Loans Between $100,000 - $199,999 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	1,323
Carrying Value	$ 163,575			163,575
Principal Amount of Loans Subject to Delinquent Principal or Interest				13,176
Deductions during period:
Balance at end of period	$ 163,575
Mortgage Loans Between $100,000 - $199,999 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Mortgage Loans Between $100,000 - $199,999 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	11.84%
Mortgage Loans Between $100,000 - $199,999 \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	50
Carrying Value	$ 6,212			6,212
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 6,212
Mortgage Loans Between $100,000 - $199,999 \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%
Mortgage Loans Between $100,000 - $199,999 \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.13%
Mortgage Loans Between $100,000 - $199,999 \| Business Purpose Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	290
Carrying Value	$ 53,684			53,684
Principal Amount of Loans Subject to Delinquent Principal or Interest				1,218
Deductions during period:
Balance at end of period	$ 53,684
Mortgage Loans Between $100,000 - $199,999 \| Business Purpose Loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	7.85%
Mortgage Loans Between $100,000 - $199,999 \| Business Purpose Loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	14.50%
Mortgage Loans Between $100,000 - $199,999 \| Residential Loans Held in Securitization Trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	782
Carrying Value	$ 150,240			150,240
Principal Amount of Loans Subject to Delinquent Principal or Interest				20,409
Deductions during period:
Balance at end of period	$ 150,240
Mortgage Loans Between $100,000 - $199,999 \| Residential Loans Held in Securitization Trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.75%
Mortgage Loans Between $100,000 - $199,999 \| Residential Loans Held in Securitization Trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	11.44%
Mortgage Loans Between $200,000 - $299,999 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	616
Carrying Value	$ 134,048			134,048
Principal Amount of Loans Subject to Delinquent Principal or Interest				12,380
Deductions during period:
Balance at end of period	$ 134,048
Mortgage Loans Between $200,000 - $299,999 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.00%
Mortgage Loans Between $200,000 - $299,999 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	11.38%
Mortgage Loans Between $200,000 - $299,999 \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	18
Carrying Value	$ 3,953			3,953
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 3,953
Mortgage Loans Between $200,000 - $299,999 \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%
Mortgage Loans Between $200,000 - $299,999 \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	8.63%
Mortgage Loans Between $200,000 - $299,999 \| Business Purpose Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	175
Carrying Value	$ 46,937			46,937
Principal Amount of Loans Subject to Delinquent Principal or Interest				2,187
Deductions during period:
Balance at end of period	$ 46,937
Mortgage Loans Between $200,000 - $299,999 \| Business Purpose Loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	7.85%
Mortgage Loans Between $200,000 - $299,999 \| Business Purpose Loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.50%
Mortgage Loans Between $200,000 - $299,999 \| Residential Loans Held in Securitization Trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	832
Carrying Value	$ 295,477			295,477
Principal Amount of Loans Subject to Delinquent Principal or Interest				52,553
Deductions during period:
Balance at end of period	$ 295,477
Mortgage Loans Between $200,000 - $299,999 \| Residential Loans Held in Securitization Trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.38%
Mortgage Loans Between $200,000 - $299,999 \| Residential Loans Held in Securitization Trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.79%
Mortgage Loans Over $299,999 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	753
Carrying Value	$ 326,591			326,591
Principal Amount of Loans Subject to Delinquent Principal or Interest				25,432
Deductions during period:
Balance at end of period	$ 326,591
Mortgage Loans Over $299,999 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.88%
Mortgage Loans Over $299,999 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.63%
Mortgage Loans Over $299,999 \| Business Purpose Loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	339
Carrying Value	$ 250,130			250,130
Principal Amount of Loans Subject to Delinquent Principal or Interest				4,615
Deductions during period:
Balance at end of period	$ 250,130
Mortgage Loans Over $299,999 \| Business Purpose Loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	7.00%
Mortgage Loans Over $299,999 \| Business Purpose Loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.99%
First mortgage loans \| Consolidated SLST
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loans	7,645
Carrying Value	$ 1,266,785			1,266,785
Deductions during period:
Balance at end of period	$ 1,266,785
First mortgage loans \| Consolidated SLST \| Financial Asset, Equal to or Greater than 90 Days Past Due
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Principal Amount of Loans Subject to Delinquent Principal or Interest				$ 236,739
First mortgage loans \| Consolidated SLST \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.38%
First mortgage loans \| Consolidated SLST \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	10.50%