Schedule IV - Mortgage Loans on Real Estate (Details) - USD ($) $ in Thousands	12 Months Ended
Dec. 31, 2016	Dec. 31, 2015
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	$ 2,720,344
Carrying amount of loans	2,693,092
Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	1,659,115	$ 999,583
Carrying amount of loans	1,641,856	994,301
Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	1,061,229	939,006
Carrying amount of loans	1,051,236	931,351
Condominium [Member] \| Condominium - Bethesda, Maryland [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	51,695
Carrying amount of loans	$ 52,476
Vacation Home Portfolio [Member] \| Vacation Home Portfolio - Various [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	7.50%
Face amount of loans	$ 88,876	94,147
Carrying amount of loans	$ 88,195	93,277
Vacation Home Portfolio [Member] \| Vacation Home Portfolio - Various, United States [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	7.00%
Face amount of loans	$ 59,500	50,000
Carrying amount of loans	59,193	49,595
Hotel [Member] \| Hotel - Philadelphia, PA [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	34,000	34,000
Carrying amount of loans	33,987	33,994
Hotel [Member] \| Hotel, New York, NY, September 2020 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	108,857
Carrying amount of loans	108,768
Hotel [Member] \| Hotel, New York, NY, September 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	7,573
Carrying amount of loans	7,513
Hotel [Member] \| Hotel - U.S. Virgin Islands [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	42,000	42,000
Carrying amount of loans	$ 41,795	41,600
Hotel [Member] \| Hotel Portfolio, Missouri [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	11.00%
Face amount of loans	$ 23,863
Carrying amount of loans	$ 23,863
Hotel [Member] \| Hotel - Burbank, CA [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	10.40%
Face amount of loans	$ 20,000
Carrying amount of loans	$ 20,000
Hotel [Member] \| Hotel - Phoenix, AZ [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	11.50%
Face amount of loans	$ 25,000
Carrying amount of loans	25,000
Hotel [Member] \| Hotel - Washington D.C. [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	20,000
Carrying amount of loans	19,978
Hotel [Member] \| Hotel, New York, NY, March 2017 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	50,000
Carrying amount of loans	50,228
Condo Pre-development [Member] \| Condo Pre-development, United Kingdom [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	123,400
Carrying amount of loans	123,400
Mixed Use [Member] \| Mixed Use - Cincinnati, OH [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	165,000	165,000
Carrying amount of loans	163,987	163,173
Mixed Use [Member] \| Mixed Use - Brooklyn, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	85,770	85,770
Carrying amount of loans	86,142	85,658
Mixed Use [Member] \| Mixed Use - New York, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	45,789
Carrying amount of loans	45,541
Mixed Use [Member] \| Mixed Use, Chicago, IL [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	129,397
Carrying amount of loans	$ 128,271
Mixed Use [Member] \| Mixed Use - North Carolina [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	11.10%
Face amount of loans	$ 6,525
Carrying amount of loans	6,525
Mixed Use [Member] \| Mixed Use, New York, NY, December 2019 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	99,581
Carrying amount of loans	99,660
Mixed Use [Member] \| Mixed Use Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	45,000
Carrying amount of loans	45,174
Mixed Use [Member] \| Mixed Use, New York, NY, October 2019 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	30,000
Carrying amount of loans	29,926
Mixed Use [Member] \| Mixed Use, New York, NY, October 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	30,175
Carrying amount of loans	29,924
Multifamily [Member] \| Multifamily, New York, New York [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	55,000
Carrying amount of loans	54,775
Multifamily [Member] \| Multifamily - Williston, ND [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	49,706	49,691
Carrying amount of loans	39,713	49,665
Multifamily [Member] \| Multifamily - Williston, ND [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	5,000
Carrying amount of loans	0
Retail Redevelopment [Member] \| Retail Redevelopment - Miami, FL [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	45,000	45,000
Carrying amount of loans	45,415	44,925
Retail Redevelopment [Member] \| Retail Redevelopment - Miami, FL (2) [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	220,000	33,000
Carrying amount of loans	218,771	32,804
Retail [Member] \| Retail, Brooklyn, NY, March 2017 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	23,000	1,653
Carrying amount of loans	23,012	1,636
Retail [Member] \| Retail - Brooklyn, NY, March 2017 - 2 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	5,910	5,910
Carrying amount of loans	5,921	5,858
Retail [Member] \| Retail, Brooklyn, NY, March 2017-3 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	7,500
Carrying amount of loans	7,489
Office [Member] \| Office - Richmond, VA [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	54,000	54,000
Carrying amount of loans	53,849	$ 53,475
Office [Member] \| Office - Boston, Massachusetts [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	28,659
Carrying amount of loans	28,543
Office [Member] \| Office, New York, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	105,000
Carrying amount of loans	$ 104,005
Office [Member] \| Office Complex - Missouri [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	11.75%
Face amount of loans	$ 9,414
Carrying amount of loans	9,414
Condo Conversion [Member] \| Condo Conversion - Brooklyn, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	41,016
Carrying amount of loans	40,825
Condo Conversion [Member] \| Condo Conversion, New York, NY, July 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	59,636
Carrying amount of loans	59,442
Condo Conversion [Member] \| Healthcare Portfolio - Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	48,944
Carrying amount of loans	48,399
Healthcare [Member] \| Healthcare Portfolio - Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	130,000
Carrying amount of loans	$ 128,856
Data Center [Member] \| Data Center - Manassas, VA [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	8.00%
Face amount of loans	$ 80,000
Carrying amount of loans	$ 79,634
Warehouse [Member] \| Warehouse Portfolio - Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	11.50%
Face amount of loans	$ 32,000
Carrying amount of loans	$ 32,000
Office Condo [Member] \| Office Condo - NY, NY [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	11.25%
Face amount of loans	$ 14,000
Carrying amount of loans	$ 13,679
Ski Resort [Member] \| Ski Resort Montana [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Interest rate	14.00%
Face amount of loans	$ 15,000
Carrying amount of loans	14,773
Senior Housing Facility [Member] \| Senior Housing - United Kingdom [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	41,693
Carrying amount of loans	41,693
Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	22,000
Carrying amount of loans	21,967
Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida (2) [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	15,500
Carrying amount of loans	15,477
Condo Development [Member] \| Condo Development, New York, NY, July 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	56,925
Carrying amount of loans	56,693
Destination Resort [Member] \| Destination Resort, Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Face amount of loans	75,000
Carrying amount of loans	$ 72,877
London Interbank Offered Rate (LIBOR) [Member] \| Condo Construction [Member] \| Condominium - Bethesda, Maryland [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	15.00%
London Interbank Offered Rate (LIBOR) [Member] \| Hotel [Member] \| Hotel - Philadelphia, PA [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	4.55%
London Interbank Offered Rate (LIBOR) [Member] \| Hotel [Member] \| Hotel, New York, NY, September 2020 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	5.25%
London Interbank Offered Rate (LIBOR) [Member] \| Hotel [Member] \| Hotel, New York, NY, September 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	5.25%
London Interbank Offered Rate (LIBOR) [Member] \| Hotel [Member] \| Hotel - U.S. Virgin Islands [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	4.95%
London Interbank Offered Rate (LIBOR) [Member] \| Hotel [Member] \| Hotel, New York, NY, August 2021 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	6.30%
London Interbank Offered Rate (LIBOR) [Member] \| Hotel [Member] \| Hotel - Washington D.C. [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	9.60%
London Interbank Offered Rate (LIBOR) [Member] \| Hotel [Member] \| Hotel, New York, NY, March 2017 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	12.00%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use - Cincinnati, OH [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	9.00%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use - Brooklyn, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	7.00%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use - New York, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	6.00%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use, Chicago, IL [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	5.50%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use, New York, NY, December 2019 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	10.50%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	9.05%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use, New York, NY, October 2019 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	10.00%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use [Member] \| Mixed Use, New York, NY, October 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	12.00%
London Interbank Offered Rate (LIBOR) [Member] \| Multifamily [Member] \| Multifamily, New York, New York [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	10.50%
London Interbank Offered Rate (LIBOR) [Member] \| Multifamily [Member] \| Multifamily - Williston, ND [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	5.90%
London Interbank Offered Rate (LIBOR) [Member] \| Multifamily [Member] \| Multifamily - Williston, ND [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	5.90%
London Interbank Offered Rate (LIBOR) [Member] \| Retail Redevelopment [Member] \| Retail Redevelopment - Miami, FL [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	6.00%
London Interbank Offered Rate (LIBOR) [Member] \| Retail Redevelopment [Member] \| Retail Redevelopment - Miami, FL (2) [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	6.25%
London Interbank Offered Rate (LIBOR) [Member] \| Retail [Member] \| Retail, Brooklyn, NY, March 2017 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	7.00%
London Interbank Offered Rate (LIBOR) [Member] \| Retail [Member] \| Retail - Brooklyn, NY, March 2017 - 2 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	7.00%
London Interbank Offered Rate (LIBOR) [Member] \| Retail [Member] \| Retail, Brooklyn, NY, March 2017-3 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	7.00%
London Interbank Offered Rate (LIBOR) [Member] \| Retail [Member] \| Retail, New York, NY, December 2018 [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	5.75%
London Interbank Offered Rate (LIBOR) [Member] \| Office [Member] \| Office - Richmond, VA [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	5.15%
London Interbank Offered Rate (LIBOR) [Member] \| Office [Member] \| Office - Boston, Massachusetts [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	6.25%
London Interbank Offered Rate (LIBOR) [Member] \| Office [Member] \| Office, New York, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	4.70%
London Interbank Offered Rate (LIBOR) [Member] \| Condo Conversion [Member] \| Condo Conversion - Brooklyn, NY [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	6.50%
London Interbank Offered Rate (LIBOR) [Member] \| Condo Conversion [Member] \| Condo Conversion, New York, NY, July 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	12.25%
London Interbank Offered Rate (LIBOR) [Member] \| Condo Conversion [Member] \| Healthcare Portfolio - Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	10.75%
London Interbank Offered Rate (LIBOR) [Member] \| Healthcare [Member] \| Healthcare Portfolio - Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	10.00%
London Interbank Offered Rate (LIBOR) [Member] \| Mixed Use Pre-development [Member] \| Mixed Use Pre-development [Member] \| Commercial Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	6.75%
London Interbank Offered Rate (LIBOR) [Member] \| Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	11.50%
London Interbank Offered Rate (LIBOR) [Member] \| Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida (2) [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	11.50%
London Interbank Offered Rate (LIBOR) [Member] \| Condo Development [Member] \| Condo Development, New York, NY, July 2020 [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	12.75%
London Interbank Offered Rate (LIBOR) [Member] \| Destination Resort [Member] \| Destination Resort, Various [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	8.90%
Three Month LIBOR [Member] \| Condo Pre-development [Member] \| Condo Pre-development, United Kingdom [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	12.50%
Three Month LIBOR [Member] \| Senior Housing Facility [Member] \| Senior Housing - United Kingdom [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Basis spread on variable interest rate	8.25%

Schedule IV - Mortgage Loans on Real Estate (Details) - USD ($)
$ in Thousands

12 Months Ended

Dec. 31, 2016

Dec. 31, 2015

Mortgage Loans on Real Estate [Line Items]

Face amount of loans

$ 2,720,344

Carrying amount of loans

2,693,092

Commercial Mortgage Portfolio Segment [Member]

Mortgage Loans on Real Estate [Line Items]

Face amount of loans