Other Assets - Schedule of Other Assets (Details) - USD ($) $ in Thousands	Mar. 31, 2021	Dec. 31, 2020
Other Assets [Abstract]
Contingent loan repurchase asset	$ 399,126	$ 480,221
Prepaid expenses	26,725	21,176
Derivatives, at fair value	15,483	23,246
Prepaid representation, warranty and indemnification claims - Agency MSR sale	15,173	15,173
REO	8,827	7,771
Prepaid lender fees, net	7,071	9,556
Deferred tax asset, net	3,634	3,543
Security deposits	2,072	2,222
Mortgage backed securities, at fair value	1,613	2,019
Other	5,147	6,556
Other assets	$ 484,871	$ 571,483

Amount, after accumulated amortization, of debt issuance costs. Includes, but is not limited to, legal, accounting, underwriting, printing, and registration costs.

Name:	us-gaap_DeferredFinanceCostsNet
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Amount, after allocation of valuation allowances and deferred tax liability, of deferred tax asset attributable to deductible differences and carryforwards, with jurisdictional netting.

+ References

Reference 1: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 740
-SubTopic 10
-Section 45
-Paragraph 6
-URI http://asc.fasb.org/extlink&oid=120406818&loc=d3e31931-109318

Reference 2: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 740
-SubTopic 10
-Section 45
-Paragraph 4
-URI http://asc.fasb.org/extlink&oid=120406818&loc=d3e31917-109318

+ Details

Name:	us-gaap_DeferredIncomeTaxAssetsNet
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Carrying value of amounts transferred to third parties for security purposes that are expected to be returned or applied towards payment in the future.

+ References

+ Details

Name:	us-gaap_DepositsAssets
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Fair value of the assets less the liabilities of a derivative or group of derivatives.

+ References

Reference 1: http://www.xbrl.org/2003/role/disclosureRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 825
-SubTopic 10
-Section 50
-Paragraph 10
-URI http://asc.fasb.org/extlink&oid=121572278&loc=d3e13433-108611

+ Details

Name:	us-gaap_DerivativeFairValueOfDerivativeNet
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Fair value of securitized, pay-through debt securities collateralized by real estate mortgage loans classified as held-to-maturity.

+ References

No definition available.

+ Details

Name:	us-gaap_MortgageBackedSecuritiesHeldToMaturityFairValueDisclosure
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Amount of assets classified as other.

+ References

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 210
-SubTopic 10
-Section S99
-Paragraph 1
-Subparagraph (SX 210.5-02(17))
-URI http://asc.fasb.org/extlink&oid=120391452&loc=d3e13212-122682

Reference 2: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 942
-SubTopic 210
-Section S99
-Paragraph 1
-Subparagraph (SX 210.9-03(10))
-URI http://asc.fasb.org/extlink&oid=120398452&loc=d3e534808-122878

Reference 3: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 944
-SubTopic 210
-Section S99
-Paragraph 1
-Subparagraph (SX 210.7-03(a)(10))
-URI http://asc.fasb.org/extlink&oid=120400017&loc=d3e572229-122910

+ Details

Name:	us-gaap_OtherAssets
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- References

No definition available.

+ Details

Name:	us-gaap_OtherAssetsAbstract
Namespace Prefix:	us-gaap_
Data Type:	xbrli:stringItemType
Balance Type:	na
Period Type:	duration

- Definition

Carrying amount of other real estate not separately disclosed, including, but not limited to, real estate investments, real estate loans that qualify as investments in real estate, and premises that are no longer used in operations may also be included in real estate owned. This does not include real estate assets taken in settlement of troubled loans through surrender or foreclosure.

+ References

+ Details

Name:	us-gaap_OtherRealEstate
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

The fair value, as of the date of each statement of financial position presented, of loans that are restricted as collateral for the payment of repurchase agreements, and that are reclassified and separately reported in the statement of financial position because the transferee has the right by contract or custom to sell or re-pledge them.

+ References

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 860
-SubTopic 30
-Section 45
-Paragraph 1
-URI http://asc.fasb.org/extlink&oid=66007379&loc=d3e113888-111728

+ Details

Name:	us-gaap_PledgedAssetsSeparatelyReportedLoansPledgedForRepurchaseAgreementsAtFairValue
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Amount of asset related to consideration paid in advance for costs that provide economic benefits in future periods, and amount of other assets.

+ References

No definition available.

+ Details

Name:	us-gaap_PrepaidExpenseAndOtherAssets
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Carrying amount as of the balance sheet date of expenditures made in advance of when the economic benefit of the cost will be realized, and which will be expensed in future periods with the passage of time or when a triggering event occurs.

+ References

Reference 1: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 942
-SubTopic 210
-Section S99
-Paragraph 1
-Subparagraph (SX 210.9-03.10)
-URI http://asc.fasb.org/extlink&oid=120398452&loc=d3e534808-122878

Reference 2: http://fasb.org/us-gaap/role/ref/legacyRef
-Publisher FASB
-Name Accounting Standards Codification
-Topic 944
-SubTopic 210
-Section S99
-Paragraph 1
-Subparagraph (SX 210.7-03.10)
-URI http://asc.fasb.org/extlink&oid=120400017&loc=d3e572229-122910

+ Details

Name:	us-gaap_PrepaidExpenseCurrentAndNoncurrent
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant

- Definition

Carrying amount as of the balance sheet date of amounts paid in advance which will be charged against earnings in periods after one year or beyond the operating cycle, if longer.

+ References

+ Details

Name:	us-gaap_PrepaidExpenseOtherNoncurrent
Namespace Prefix:	us-gaap_
Data Type:	xbrli:monetaryItemType
Balance Type:	debit
Period Type:	instant