Schedule IV - Mortgage Loans on Real Estate (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($)	Dec. 31, 2017 USD ($)	Dec. 31, 2019 USD ($)
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Value	$ 12,707,625	$ 10,157,126	$ 10,157,126	$ 20,780,548
Principal Amount of Loans Subject to Delinquent Principal or Interest				224,706
Reconciliation of Balance Sheet Reported Amounts of Mortgage Loans on Real Estate
Beginning balance	12,707,625	10,157,126	7,565,459
Additions during period:
Purchases	8,762,553	2,983,295	2,987,775
Accretion of purchase discount	11,234	19,940	19,686
Change in realized and unrealized gains (losses)	638,557	4,096	10,214
Deductions during period:
Repayments of principal	(1,052,812)	(182,163)	(175,664)
Collection of interest	(11,429)	(21,754)	(26,081)
Transfer to REO	(6,105)	(7,998)	(7,228)
Cost of mortgages sold	(213,871)	(109,000)	(176,470)
Provision for loan loss	2,780	(1,235)	1,739
Change in realized and unrealized gains (losses)	0	(85,115)	(270)
Amortization of premium	(57,984)	(49,567)	(42,034)
Balance at end of period	$ 20,780,548	$ 12,707,625	$ 10,157,126
Original loan amount $0 - $99,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1,355
Carrying Value	$ 62,833			62,833
Principal Amount of Loans Subject to Delinquent Principal or Interest				12,942
Deductions during period:
Balance at end of period	$ 62,833
Original loan amount $0 - $99,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.99%
Original loan amount $0 - $99,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	14.99%
Original loan amount $0 - $99,999 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1,482
Carrying Value	$ 84,154			84,154
Principal Amount of Loans Subject to Delinquent Principal or Interest				6,901
Deductions during period:
Balance at end of period	$ 84,154
Original loan amount $0 - $99,999 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.38%
Original loan amount $0 - $99,999 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	13.50%
Original loan amount $0 - $99,999 \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	701
Carrying Value	$ 32,827			32,827
Principal Amount of Loans Subject to Delinquent Principal or Interest				595
Deductions during period:
Balance at end of period	$ 32,827
Original loan amount $0 - $99,999 \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.75%
Original loan amount $0 - $99,999 \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.00%
Original loan amount $100,000 - $199,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	577
Carrying Value	$ 62,926			62,926
Principal Amount of Loans Subject to Delinquent Principal or Interest				13,302
Deductions during period:
Balance at end of period	$ 62,926
Original loan amount $100,000 - $199,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Original loan amount $100,000 - $199,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.48%
Original loan amount $100,000 - $199,999 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	2,769
Carrying Value	$ 328,284			328,284
Principal Amount of Loans Subject to Delinquent Principal or Interest				29,491
Deductions during period:
Balance at end of period	$ 328,284
Original loan amount $100,000 - $199,999 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.50%
Original loan amount $100,000 - $199,999 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	13.38%
Original loan amount $100,000 - $199,999 \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	74
Carrying Value	$ 9,549			9,549
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 9,549
Original loan amount $100,000 - $199,999 \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%
Original loan amount $100,000 - $199,999 \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.13%
Original loan amount $200,000 - $299,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	139
Carrying Value	$ 27,775			27,775
Principal Amount of Loans Subject to Delinquent Principal or Interest				7,914
Deductions during period:
Balance at end of period	$ 27,775
Original loan amount $200,000 - $299,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.00%
Original loan amount $200,000 - $299,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	11.44%
Original loan amount $200,000 - $299,999 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1,484
Carrying Value	$ 301,835			301,835
Principal Amount of Loans Subject to Delinquent Principal or Interest				29,612
Deductions during period:
Balance at end of period	$ 301,835
Original loan amount $200,000 - $299,999 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.25%
Original loan amount $200,000 - $299,999 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.00%
Original loan amount $200,000 - $299,999 \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	18
Carrying Value	$ 4,137			4,137
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 4,137
Original loan amount $200,000 - $299,999 \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%
Original loan amount $200,000 - $299,999 \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	8.63%
Original loan amount over $299,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	112
Carrying Value	$ 49,222			49,222
Principal Amount of Loans Subject to Delinquent Principal or Interest				16,477
Deductions during period:
Balance at end of period	$ 49,222
Original loan amount over $299,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Original loan amount over $299,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.75%
Original loan amount over $299,999 \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1,701
Carrying Value	$ 668,673			668,673
Principal Amount of Loans Subject to Delinquent Principal or Interest				56,318
Deductions during period:
Balance at end of period	$ 668,673
Original loan amount over $299,999 \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Original loan amount over $299,999 \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.00%
Original loan amount over $299,999 \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1
Carrying Value	$ 295			295
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 295
Original loan amount over $299,999 \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.88%
Original loan amount over $299,999 \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.88%
Residential loans and loans held for sale \| Other mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	14
Carrying Value	$ 2,406			2,406
Principal Amount of Loans Subject to Delinquent Principal or Interest				413
Deductions during period:
Balance at end of period	$ 2,406
Residential loans and loans held for sale \| Other mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.36%
Residential loans and loans held for sale \| Other mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.63%
First mortgage loans \| Residential mortgage loans held in securitization trust, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	8,103
Carrying Value	$ 1,328,886			1,328,886
Deductions during period:
Balance at end of period	$ 1,328,886
First mortgage loans \| Residential mortgage loans held in securitization trust, at fair value \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.38%
First mortgage loans \| Residential mortgage loans held in securitization trust, at fair value \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	10.50%
First mortgage loans \| Multifamily \| Multi-family loans held in securitization trusts, at fair value
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	828
Carrying Value	$ 17,816,746			17,816,746
Principal Amount of Loans Subject to Delinquent Principal or Interest				$ 0
Deductions during period:
Balance at end of period	$ 17,816,746
First mortgage loans \| Multifamily \| Multi-family loans held in securitization trusts, at fair value \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.04%
First mortgage loans \| Multifamily \| Multi-family loans held in securitization trusts, at fair value \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.76%