Business Purpose Residential Loans - Product Types and Characteristics of Business Purpose Loans (Details) $ in Thousands			12 Months Ended
	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan	Dec. 31, 2019 USD ($) loan	Dec. 31, 2018 USD ($) loan
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of Loans \| loan			669	1,484
Total Principal	$ 524,928	$ 1,034,284	$ 524,928	$ 1,034,284
30-89 Days DQ	1,650	559	1,650	559
90 Days DQ	$ 747	$ 747	$ 747	$ 747
Single-family rental loans
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of Loans \| loan	201	11	201	11
Total Principal	$ 321,636	$ 27,566	$ 321,636	$ 27,566
30-89 Days DQ	0	0	0	0
90 Days DQ	1,818	0	1,818	0
Single-family rental loans \| $0 to $250
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	0		0
Loan Balance, maximum	250		$ 250
Number of Loans \| loan			85
Total Principal	10,506		$ 10,506
30-89 Days DQ	0		0
90 Days DQ	130		$ 130
Single-family rental loans \| $0 to $250 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			5.50%
Single-family rental loans \| $0 to $250 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.63%
Single-family rental loans \| $251 to $500
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	251	251	$ 251	251
Loan Balance, maximum	500	500	$ 500	$ 500
Number of Loans \| loan			9	2
Total Principal	3,708	787	$ 3,708	$ 787
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	$ 0	$ 0
Single-family rental loans \| $251 to $500 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.94%	6.79%
Single-family rental loans \| $251 to $500 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			6.00%	7.47%
Single-family rental loans \| $501 to $750
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	501	501	$ 501	$ 501
Loan Balance, maximum	750	750	$ 750	$ 750
Number of Loans \| loan			21	2
Total Principal	13,335	1,252	$ 13,335	$ 1,252
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	$ 0	$ 0
Single-family rental loans \| $501 to $750 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.55%	6.12%
Single-family rental loans \| $501 to $750 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			5.96%	7.25%
Single-family rental loans \| $751 to $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	751	751	$ 751	$ 751
Loan Balance, maximum	1,000	1,000	$ 1,000	$ 1,000
Number of Loans \| loan			13	3
Total Principal	11,676	2,488	$ 11,676	$ 2,488
30-89 Days DQ	0	0	0	0
90 Days DQ	0	0	$ 0	$ 0
Single-family rental loans \| $751 to $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			5.00%	5.91%
Single-family rental loans \| $751 to $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			5.93%	6.58%
Single-family rental loans \| Over $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, maximum	1,000	1,000	$ 1,000	$ 1,000
Number of Loans \| loan			73	4
Total Principal	282,411	23,039	$ 282,411	$ 23,039
30-89 Days DQ	0	0	0	0
90 Days DQ	$ 1,688	$ 0	$ 1,688	$ 0
Single-family rental loans \| Over $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.35%	5.62%
Single-family rental loans \| Over $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			6.28%	6.94%
Residential bridge loans held-for-investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of Loans \| loan	2,653	157	2,653	157
Total Principal	$ 741,232	$ 112,360	$ 741,232	$ 112,360
30-89 Days DQ	39,553	4,991	39,553	4,991
90 Days DQ	8,989	12,134	8,989	12,134
Residential bridge loans held-for-investment \| $0 to $250
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	0	0	0	0
Loan Balance, maximum	250	250	$ 250	$ 250
Number of Loans \| loan			2,207	50
Total Principal	197,449	7,941	$ 197,449	$ 7,941
30-89 Days DQ	1,447	262	1,447	262
90 Days DQ	369	695	$ 369	$ 695
Residential bridge loans held-for-investment \| $0 to $250 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			6.53%	8.00%
Residential bridge loans held-for-investment \| $0 to $250 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			12.00%	12.00%
Residential bridge loans held-for-investment \| $251 to $500
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	251	251	$ 251	$ 251
Loan Balance, maximum	500	500	$ 500	$ 500
Number of Loans \| loan			198	38
Total Principal	71,361	13,297	$ 71,361	$ 13,297
30-89 Days DQ	2,811	636	2,811	636
90 Days DQ	675	469	$ 675	$ 469
Residential bridge loans held-for-investment \| $251 to $500 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			6.99%	8.00%
Residential bridge loans held-for-investment \| $251 to $500 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			13.00%	10.00%
Residential bridge loans held-for-investment \| $501 to $750
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	501	501	$ 501	$ 501
Loan Balance, maximum	750	750	$ 750	$ 750
Number of Loans \| loan			71	21
Total Principal	42,862	12,410	$ 42,862	$ 12,410
30-89 Days DQ	2,072	1,769	2,072	1,769
90 Days DQ	508	0	$ 508	$ 0
Residential bridge loans held-for-investment \| $501 to $750 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			6.99%	7.50%
Residential bridge loans held-for-investment \| $501 to $750 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			9.99%	10.00%
Residential bridge loans held-for-investment \| $751 to $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	751	751	$ 751	$ 751
Loan Balance, maximum	1,000	1,000	$ 1,000	$ 1,000
Number of Loans \| loan			40	19
Total Principal	34,646	16,937	$ 34,646	$ 16,937
30-89 Days DQ	1,771	840	1,771	840
90 Days DQ	2,443	0	$ 2,443	$ 0
Residential bridge loans held-for-investment \| $751 to $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.28%	7.50%
Residential bridge loans held-for-investment \| $751 to $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			10.00%	10.00%
Residential bridge loans held-for-investment \| Over $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, maximum	1,000	1,000	$ 1,000	$ 1,000
Number of Loans \| loan			137	29
Total Principal	394,914	61,775	$ 394,914	$ 61,775
30-89 Days DQ	31,452	1,484	31,452	1,484
90 Days DQ	$ 4,994	$ 10,970	$ 4,994	$ 10,970
Residential bridge loans held-for-investment \| Over $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			5.79%	8.00%
Residential bridge loans held-for-investment \| Over $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			10.25%	10.00%
Redwood \| Single-family rental loans held-for-investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of Loans \| loan	107		107
Total Principal	$ 231,211		$ 231,211
30-89 Days DQ	0		0
90 Days DQ	0		0
Redwood \| Single-family rental loans held-for-investment \| $251 to $500
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	251		251
Loan Balance, maximum	500		$ 500
Number of Loans \| loan			20
Total Principal	7,925		$ 7,925
30-89 Days DQ	0		0
90 Days DQ	0		$ 0
Redwood \| Single-family rental loans held-for-investment \| $251 to $500 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.88%
Redwood \| Single-family rental loans held-for-investment \| $251 to $500 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.47%
Redwood \| Single-family rental loans held-for-investment \| $501 to $750
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	501		$ 501
Loan Balance, maximum	750		$ 750
Number of Loans \| loan			26
Total Principal	15,620		$ 15,620
30-89 Days DQ	0		0
90 Days DQ	0		$ 0
Redwood \| Single-family rental loans held-for-investment \| $501 to $750 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.45%
Redwood \| Single-family rental loans held-for-investment \| $501 to $750 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.25%
Redwood \| Single-family rental loans held-for-investment \| $751 to $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	751		$ 751
Loan Balance, maximum	1,000		$ 1,000
Number of Loans \| loan			16
Total Principal	13,616		$ 13,616
30-89 Days DQ	0		0
90 Days DQ	0		$ 0
Redwood \| Single-family rental loans held-for-investment \| $751 to $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.91%
Redwood \| Single-family rental loans held-for-investment \| $751 to $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			6.58%
Redwood \| Single-family rental loans held-for-investment \| Over $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, maximum	1,000		$ 1,000
Number of Loans \| loan			45
Total Principal	194,050		$ 194,050
30-89 Days DQ	0		0
90 Days DQ	$ 0		$ 0
Redwood \| Single-family rental loans held-for-investment \| Over $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			3.93%
Redwood \| Single-family rental loans held-for-investment \| Over $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			6.94%
CAFL \| Single-family rental loans held-for-investment
Accounts, Notes, Loans and Financing Receivable [Line Items]
Number of Loans \| loan	783		783
Total Principal	$ 2,078,214		$ 2,078,214
30-89 Days DQ	22,771		22,771
90 Days DQ	29,039		29,039
CAFL \| Single-family rental loans held-for-investment \| $0 to $250
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	0		0
Loan Balance, maximum	250		$ 250
Number of Loans \| loan			2
Total Principal	398		$ 398
30-89 Days DQ	0		0
90 Days DQ	0		$ 0
CAFL \| Single-family rental loans held-for-investment \| $0 to $250 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			5.46%
CAFL \| Single-family rental loans held-for-investment \| $0 to $250 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			5.80%
CAFL \| Single-family rental loans held-for-investment \| $251 to $500
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	251		$ 251
Loan Balance, maximum	500		$ 500
Number of Loans \| loan			56
Total Principal	25,643		$ 25,643
30-89 Days DQ	1,306		1,306
90 Days DQ	0		$ 0
CAFL \| Single-family rental loans held-for-investment \| $251 to $500 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.92%
CAFL \| Single-family rental loans held-for-investment \| $251 to $500 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.05%
CAFL \| Single-family rental loans held-for-investment \| $501 to $750
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	501		$ 501
Loan Balance, maximum	750		$ 750
Number of Loans \| loan			148
Total Principal	91,414		$ 91,414
30-89 Days DQ	1,259		1,259
90 Days DQ	1,990		$ 1,990
CAFL \| Single-family rental loans held-for-investment \| $501 to $750 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.75%
CAFL \| Single-family rental loans held-for-investment \| $501 to $750 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.31%
CAFL \| Single-family rental loans held-for-investment \| $751 to $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, minimum	751		$ 751
Loan Balance, maximum	1,000		$ 1,000
Number of Loans \| loan			98
Total Principal	85,472		$ 85,472
30-89 Days DQ	1,639		1,639
90 Days DQ	879		$ 879
CAFL \| Single-family rental loans held-for-investment \| $751 to $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.62%
CAFL \| Single-family rental loans held-for-investment \| $751 to $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.23%
CAFL \| Single-family rental loans held-for-investment \| Over $1,000
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loan Balance, maximum	1,000		$ 1,000
Number of Loans \| loan			479
Total Principal	1,875,287		$ 1,875,287
30-89 Days DQ	18,567		18,567
90 Days DQ	$ 26,170		$ 26,170
CAFL \| Single-family rental loans held-for-investment \| Over $1,000 \| Minimum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			4.31%
CAFL \| Single-family rental loans held-for-investment \| Over $1,000 \| Maximum
Accounts, Notes, Loans and Financing Receivable [Line Items]
Interest Rate			7.57%