Investment Fair Value Changes, Net - Components of Investment Activities (Details) - Investment Fair Value Changes, Net - USD ($) $ in Thousands	3 Months Ended		6 Months Ended
	Jun. 30, 2017	Jun. 30, 2016	Jun. 30, 2017	Jun. 30, 2016
Investment Holdings [Line Items]
Changes in fair value of assets	$ 8,115	$ (11,066)	$ 9,666	$ (30,604)
Residential loans held-for-investment, at Redwood
Investment Holdings [Line Items]
Changes in fair value of assets	8,354	(647)	6,021	22,816
Trading securities
Investment Holdings [Line Items]
Changes in fair value of assets	18,926	431	30,069	(5,170)
Net investments in consolidated Sequoia entities
Investment Holdings [Line Items]
Changes in fair value of assets	(987)	(251)	(2,797)	(1,831)
Risk sharing investments
Investment Holdings [Line Items]
Changes in fair value of assets	(513)	(694)	(718)	(704)
Risk management derivatives, net
Investment Holdings [Line Items]
Changes in fair value of assets	(17,838)	(9,600)	(22,965)	(45,410)
Valuation adjustments on commercial loans held-for-sale
Investment Holdings [Line Items]
Changes in fair value of assets	300	0	300	0
Impairments on AFS securities
Investment Holdings [Line Items]
Changes in fair value of assets	$ (127)	$ (305)	$ (244)	$ (305)