Subordinate Loan Portfolio (Details) - USD ($) $ in Thousands	Dec. 31, 2016	Dec. 31, 2015
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	$ 2,720,344
Carrying amount of loans	2,693,092
Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	1,061,229	$ 939,006
Carrying amount of loans	1,051,236	931,351
Unfunded Commitment	55,387	54,423
Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel - Burbank, CA [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	20,000
Carrying amount of loans	20,000
Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel - Phoenix, AZ [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	25,000
Carrying amount of loans	25,000
Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel - Washington D.C. [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	20,000
Carrying amount of loans	19,978
Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel, New York, NY, March 2017 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	50,000
Carrying amount of loans	50,228
Subordinate Mortgage Portfolio Segment [Member] \| Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	22,000
Carrying amount of loans	21,967
Subordinate Mortgage Portfolio Segment [Member] \| Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida (2) [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	15,500
Carrying amount of loans	15,477
Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use - North Carolina [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	6,525
Carrying amount of loans	6,525
Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use, New York, NY, October 2019 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	30,000
Carrying amount of loans	29,926
Subordinate Mortgage Portfolio Segment [Member] \| Office [Member] \| Office Complex - Missouri [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	9,414
Carrying amount of loans	9,414
Subordinate Mortgage Portfolio Segment [Member] \| Warehouse [Member] \| Warehouse Portfolio - Various [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	32,000
Carrying amount of loans	32,000
Subordinate Mortgage Portfolio Segment [Member] \| Ski Resort [Member] \| Ski Resort Montana [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	15,000
Carrying amount of loans	14,773
Subordinate Mortgage Portfolio Segment [Member] \| Senior Housing Facility [Member] \| Senior Housing - United Kingdom [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	41,693
Carrying amount of loans	41,693
Subordinate Mortgage Portfolio Segment [Member] \| Destination Resort [Member] \| Destination Resort, Various [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	75,000
Carrying amount of loans	72,877
Subordinate Mortgage Portfolio Segment [Member] \| Condo Pre-development [Member] \| Condo Pre-development, United Kingdom [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	123,400
Carrying amount of loans	123,400
Subordinate Mortgage Portfolio Segment [Member] \| Healthcare [Member] \| Healthcare Portfolio - Various [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	130,000
Carrying amount of loans	128,856
Subordinate to Company's Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	12,573	26,328
Carrying amount of loans	7,513	26,095
Unfunded Commitment	7,501	12,478
Subordinate to Company's Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel, New York, NY, September 2018 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	7,573	2,595
Carrying amount of loans	7,513	2,458
Unfunded Commitment	7,501	12,478
Subordinate to Company's Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Multifamily Portfolio [Member] \| Multifamily - Williston, ND [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	5,000	5,000
Carrying amount of loans	0	5,000
Unfunded Commitment	0	0
Subordinate to Company's Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use - Brooklyn, NY [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		12,347
Carrying amount of loans		12,222
Unfunded Commitment		0
Subordinate to Company's Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Condo Conversion [Member] \| Condo Conversion, New York, NY, September 2016 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		6,386
Carrying amount of loans		6,415
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	1,048,656	912,678
Carrying amount of loans	1,043,723	905,256
Unfunded Commitment	47,886	41,945
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		25,000
Carrying amount of loans		25,000
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel Portfolio - Rochester, MN [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	23,863	24,182
Carrying amount of loans	23,863	24,182
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel - Burbank, CA [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	20,000	20,000
Carrying amount of loans	20,000	20,000
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel - Phoenix, AZ [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	25,000
Carrying amount of loans	25,000
Unfunded Commitment	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel - Washington D.C. [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	20,000	20,000
Carrying amount of loans	19,978	19,934
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Hotel [Member] \| Hotel, New York, NY, March 2017 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	50,000	50,000
Carrying amount of loans	50,228	49,522
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	22,000	22,000
Carrying amount of loans	21,967	21,895
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Multifamily Portfolio [Member] \| Multifamily Portfolio - Florida (2) [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	15,500	15,500
Carrying amount of loans	15,477	15,426
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Multifamily Portfolio [Member] \| Multifamily, New York, New York [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	55,000	55,000
Carrying amount of loans	54,775	54,558
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use - North Carolina [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	6,525	6,525
Carrying amount of loans	6,525	6,525
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use - New York, NY [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	99,581
Carrying amount of loans	99,660
Unfunded Commitment	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use - Various, May 2017 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	45,000	45,000
Carrying amount of loans	45,174	44,854
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use, New York, NY, October 2018 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	30,000	30,000
Carrying amount of loans	29,926	29,785
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use, New York, NY, October 2019 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	30,175
Carrying amount of loans	29,924
Unfunded Commitment	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use Various, December 2016 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		19,500
Carrying amount of loans		19,377
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use - London, England [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		50,676
Carrying amount of loans		50,676
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Mixed Use [Member] \| Mixed Use - New York, NY, December 2017 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		88,368
Carrying amount of loans		87,818
Unfunded Commitment		785
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Office [Member] \| Office Complex - Missouri [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	9,414	9,566
Carrying amount of loans	9,414	9,566
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Office [Member] \| Office Condo New York, NY [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	14,000	14,000
Carrying amount of loans	13,679	13,631
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Office [Member] \| Office - Michigan [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		8,753
Carrying amount of loans		8,753
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Warehouse [Member] \| Warehouse Portfolio - Various [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	32,000	32,000
Carrying amount of loans	32,000	32,000
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Ski Resort [Member] \| Ski Resort Montana [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	15,000
Carrying amount of loans	14,773
Unfunded Commitment	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Ski Resort [Member] \| Ski Resort, Big Sky, MT [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		15,000
Carrying amount of loans		14,878
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Senior Housing Facility [Member] \| Senior Housing - United Kingdom [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	41,693	79,735
Carrying amount of loans	41,693	79,735
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Condo Development [Member] \| Condo Development - New York, NY [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	56,925	34,184
Carrying amount of loans	56,693	33,567
Unfunded Commitment	19,830	41,160
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Condo Conversion [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	59,636
Carrying amount of loans	59,442
Unfunded Commitment	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Condo Conversion [Member] \| Condo Conversion, New York, NY, August 2018 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		52,418
Carrying amount of loans		51,941
Unfunded Commitment		0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Condo Conversion [Member] \| Condo Conversion, New York, NY, July 2019 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	48,944
Carrying amount of loans	48,399
Unfunded Commitment	28,056
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Destination Resort [Member] \| Destination Resort, Various [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	75,000	75,000
Carrying amount of loans	72,877	71,362
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Condo Pre-development [Member] \| Condo Pre-development, United Kingdom [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	123,400	81,048
Carrying amount of loans	123,400	81,048
Unfunded Commitment	0	0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Healthcare [Member] \| Condo Conversion, New York, NY, August 2018 [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance	130,000
Carrying amount of loans	128,856
Unfunded Commitment	$ 0
Subordinate to Third Party Commercial Mortgage Loans [Member] \| Subordinate Mortgage Portfolio Segment [Member] \| Healthcare [Member] \| Healthcare Portfolio - Various [Member]
Mortgage Loans on Real Estate [Line Items]
Current Principal Balance		39,223
Carrying amount of loans		39,223
Unfunded Commitment		$ 0

Subordinate Loan Portfolio (Details) - USD ($)
$ in Thousands

Dec. 31, 2016

Dec. 31, 2015

Mortgage Loans on Real Estate [Line Items]

Current Principal Balance

$ 2,720,344

Carrying amount of loans

2,693,092

Subordinate Mortgage Portfolio Segment [Member]

Mortgage Loans on Real Estate [Line Items]

Current Principal Balance

1,061,229