Financing Agreements - Pledged Collateral (Details) - USD ($) $ in Thousands	Jun. 30, 2023	Dec. 31, 2022
Debt Instrument [Line Items]
Accrued interest on assets	$ 31,700	$ 30,900
Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	8,748,019	8,057,175
Non-Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,187,094	1,331,386
Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	2,741,539	2,654,442
Securitized \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	4,819,386	4,071,347
Residential whole loans, at carrying value \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,663,035	1,814,780
Residential whole loans, at carrying value \| Non-Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	65,527	215,993
Residential whole loans, at carrying value \| Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	364,419	284,683
Residential whole loans, at carrying value \| Securitized \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,233,089	1,314,104
Residential whole loans, at fair value \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	6,458,834	5,980,471
Residential whole loans, at fair value \| Non-Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,121,567	1,095,556
Residential whole loans, at fair value \| Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	1,781,344	2,164,158
Residential whole loans, at fair value \| Securitized \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	3,555,923	2,720,757
Securities, at fair value \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	543,400	177,111
Securities, at fair value \| Non-Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	0	0
Securities, at fair value \| Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	543,400	177,111
Securities, at fair value \| Securitized \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	0	0
Other assets: REO \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	82,750	84,813
Other assets: REO \| Non-Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	0	19,837
Other assets: REO \| Mark-to-Market \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	52,376	28,490
Other assets: REO \| Securitized \| Asset Pledged as Collateral with Right
Debt Instrument [Line Items]
Fair value of securities pledged as collateral under financing agreements	$ 30,374	$ 36,486