Schedule IV - Mortgage Loans on Real Estate (Details) $ in Thousands	12 Months Ended
	Dec. 31, 2018 USD ($) loan	Dec. 31, 2017 USD ($)	Dec. 31, 2016 USD ($)	Dec. 31, 2018 USD ($)	Dec. 31, 2013 USD ($)
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Carrying Value	$ 10,157,126	$ 10,157,126	$ 7,565,459	$ 12,707,625	$ 7,792,422
Principal Amount of Loans Subject to Delinquent Principal or Interest				151,591
Reconciliation of Balance Sheet Reported Amounts of Mortgage Loans on Real Estate
Beginning balance	10,157,126		7,565,459
Additions during period:
Purchases	2,983,295	2,987,775	82,167
Accretion of purchase discount	19,940	19,686	32,688
Deconsolidation	0	0	0
Change in realized and unrealized gains (losses)	4,096	10,214	10,794
Deductions during period:
Repayments of principal	(182,163)	(175,664)	(175,216)
Collection of interest	(21,754)	(26,081)	(32,928)
Transfer to REO	(7,998)	(7,228)	(8,892)
Cost of mortgages sold	(109,000)	(176,470)	(96,344)
Provision for loan loss	(1,235)	1,739	847
Change in realized and unrealized gains (losses)	(85,115)	(270)	0
Amortization of premium	(49,567)	(42,034)	$ (40,079)
Balance at end of period	$ 12,707,625	$ 10,157,126
First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	20
Carrying Value	$ 4,994			4,994
Principal Amount of Loans Subject to Delinquent Principal or Interest				2,177
Deductions during period:
Balance at end of period	$ 4,994
First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.98%
First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	15.00%
Multifamily \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	606
Carrying Value	$ 11,679,847			11,679,847
Principal Amount of Loans Subject to Delinquent Principal or Interest				16,872
Deductions during period:
Balance at end of period	$ 11,679,847
Multifamily \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.04%
Multifamily \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.18%
Original loan amount $0 - $99,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1,670
Carrying Value	$ 78,252			78,252
Principal Amount of Loans Subject to Delinquent Principal or Interest				12,140
Deductions during period:
Balance at end of period	$ 78,252
Original loan amount $0 - $99,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.99%
Original loan amount $0 - $99,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	14.99%
Original loan amount $0 - $99,999 \| Residential mortgage loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	27
Carrying Value	$ 1,803			1,803
Principal Amount of Loans Subject to Delinquent Principal or Interest				92
Deductions during period:
Balance at end of period	$ 1,803
Original loan amount $0 - $99,999 \| Residential mortgage loans held in securitization trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	4.00%
Original loan amount $0 - $99,999 \| Residential mortgage loans held in securitization trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.63%
Original loan amount $0 - $99,999 \| Single Family \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	772
Carrying Value	$ 41,547			41,547
Principal Amount of Loans Subject to Delinquent Principal or Interest				4,015
Deductions during period:
Balance at end of period	$ 41,547
Original loan amount $0 - $99,999 \| Single Family \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.50%
Original loan amount $0 - $99,999 \| Single Family \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	14.59%
Original loan amount $0 - $99,999 \| Single Family \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	963
Carrying Value	$ 46,529			46,529
Principal Amount of Loans Subject to Delinquent Principal or Interest				434
Deductions during period:
Balance at end of period	$ 46,529
Original loan amount $0 - $99,999 \| Single Family \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.75%
Original loan amount $0 - $99,999 \| Single Family \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.13%
Original loan amount $100,000 - $199,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	745
Carrying Value	$ 78,666			78,666
Principal Amount of Loans Subject to Delinquent Principal or Interest				13,049
Deductions during period:
Balance at end of period	$ 78,666
Original loan amount $100,000 - $199,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.98%
Original loan amount $100,000 - $199,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.48%
Original loan amount $100,000 - $199,999 \| Residential mortgage loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	69
Carrying Value	$ 10,338			10,338
Principal Amount of Loans Subject to Delinquent Principal or Interest				1,029
Deductions during period:
Balance at end of period	$ 10,338
Original loan amount $100,000 - $199,999 \| Residential mortgage loans held in securitization trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.50%
Original loan amount $100,000 - $199,999 \| Residential mortgage loans held in securitization trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.63%
Original loan amount $100,000 - $199,999 \| Single Family \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1,424
Carrying Value	$ 161,571			161,571
Principal Amount of Loans Subject to Delinquent Principal or Interest				17,303
Deductions during period:
Balance at end of period	$ 161,571
Original loan amount $100,000 - $199,999 \| Single Family \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	1.88%
Original loan amount $100,000 - $199,999 \| Single Family \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	13.13%
Original loan amount $100,000 - $199,999 \| Single Family \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	113
Carrying Value	$ 14,978			14,978
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 14,978
Original loan amount $100,000 - $199,999 \| Single Family \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.00%
Original loan amount $100,000 - $199,999 \| Single Family \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.00%
Original loan amount $200,000 - $299,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	177
Carrying Value	$ 32,813			32,813
Principal Amount of Loans Subject to Delinquent Principal or Interest				9,091
Deductions during period:
Balance at end of period	$ 32,813
Original loan amount $200,000 - $299,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	0.00%
Original loan amount $200,000 - $299,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	12.04%
Original loan amount $200,000 - $299,999 \| Residential mortgage loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	45
Carrying Value	$ 10,772			10,772
Principal Amount of Loans Subject to Delinquent Principal or Interest				764
Deductions during period:
Balance at end of period	$ 10,772
Original loan amount $200,000 - $299,999 \| Residential mortgage loans held in securitization trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.38%
Original loan amount $200,000 - $299,999 \| Residential mortgage loans held in securitization trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.38%
Original loan amount $200,000 - $299,999 \| Single Family \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	706
Carrying Value	$ 135,073			135,073
Principal Amount of Loans Subject to Delinquent Principal or Interest				15,906
Deductions during period:
Balance at end of period	$ 135,073
Original loan amount $200,000 - $299,999 \| Single Family \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Original loan amount $200,000 - $299,999 \| Single Family \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	11.38%
Original loan amount $200,000 - $299,999 \| Single Family \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	24
Carrying Value	$ 5,548			5,548
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 5,548
Original loan amount $200,000 - $299,999 \| Single Family \| Second mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	6.25%
Original loan amount $200,000 - $299,999 \| Single Family \| Second mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.00%
Original loan amount over $299,999 \| Distressed residential mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	110
Carrying Value	$ 38,735			38,735
Principal Amount of Loans Subject to Delinquent Principal or Interest				12,170
Deductions during period:
Balance at end of period	$ 38,735
Original loan amount over $299,999 \| Distressed residential mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Original loan amount over $299,999 \| Distressed residential mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	9.40%
Original loan amount over $299,999 \| Single Family \| First mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	888
Carrying Value	$ 331,980			331,980
Principal Amount of Loans Subject to Delinquent Principal or Interest				37,509
Deductions during period:
Balance at end of period	$ 331,980
Original loan amount over $299,999 \| Single Family \| First mortgage loans \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	2.00%
Original loan amount over $299,999 \| Single Family \| First mortgage loans \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	10.89%
Original loan amount over $299,999 \| Single Family \| Second mortgage loans
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	1
Interest Rate	6.88%
Carrying Value	$ 297			297
Principal Amount of Loans Subject to Delinquent Principal or Interest				0
Deductions during period:
Balance at end of period	$ 297
Original loan amount $300,000 - $399,999 \| Residential mortgage loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	22
Carrying Value	$ 7,252			7,252
Principal Amount of Loans Subject to Delinquent Principal or Interest				1,556
Deductions during period:
Balance at end of period	$ 7,252
Original loan amount $300,000 - $399,999 \| Residential mortgage loans held in securitization trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	4.00%
Original loan amount $300,000 - $399,999 \| Residential mortgage loans held in securitization trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.23%
Original loan amount $400,000 - $499,999 \| Residential mortgage loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	9
Carrying Value	$ 3,762			3,762
Principal Amount of Loans Subject to Delinquent Principal or Interest				456
Deductions during period:
Balance at end of period	$ 3,762
Original loan amount $400,000 - $499,999 \| Residential mortgage loans held in securitization trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	4.75%
Original loan amount $400,000 - $499,999 \| Residential mortgage loans held in securitization trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.13%
Original loan amount over $499,999 \| Residential mortgage loans held in securitization trusts
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Number of Loans \| loan	24
Carrying Value	$ 22,868			22,868
Principal Amount of Loans Subject to Delinquent Principal or Interest				$ 7,028
Deductions during period:
Balance at end of period	$ 22,868
Original loan amount over $499,999 \| Residential mortgage loans held in securitization trusts \| Minimum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	3.00%
Original loan amount over $499,999 \| Residential mortgage loans held in securitization trusts \| Maximum
SEC Schedule, 12-29, Real Estate Companies, Investment in Mortgage Loans on Real Estate [Line Items]
Interest Rate	5.13%