Loans Held for Investment - Schedule of Past Due Status of Loans Held for Investment, Gross (Details) - USD ($) $ in Thousands	9 Months Ended	12 Months Ended
Mar. 31, 2021	Jun. 30, 2020	Mar. 31, 2020
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	90 days	90 days
Current	$ 831,272	$ 898,947
30-89 Days Past Due	0	219
Non-Accrual	10,721	5,300
Total Loans Held for Investment, Gross	841,993	904,466	$ 915,288
Mortgage loans, Single-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	244,798	293,326
30-89 Days Past Due	0	219
Non-Accrual	9,595	5,265
Total Loans Held for Investment, Gross	254,393	298,810	326,686
Mortgage Loans, Multi Family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	482,157	491,903
30-89 Days Past Due	0	0
Non-Accrual	1,126	0
Total Loans Held for Investment, Gross	483,283	491,903	475,941
Mortgage loans, Commercial real estate
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	99,722	105,235
30-89 Days Past Due	0	0
Non-Accrual	0	0
Total Loans Held for Investment, Gross	99,722	105,235	105,691
Construction
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	3,508	7,801
30-89 Days Past Due	0	0
Non-Accrual	0	0
Total Loans Held for Investment, Gross	3,508	7,801	6,346
Mortgage Loans Other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	140	143
30-89 Days Past Due	0	0
Non-Accrual	0	0
Total Loans Held for Investment, Gross	140	143	502
Commercial business loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	851	445
30-89 Days Past Due	0	0
Non-Accrual	0	35
Total Loans Held for Investment, Gross	851	480	502
Consumer loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	96	94
30-89 Days Past Due	0	0
Non-Accrual	0	0
Total Loans Held for Investment, Gross	$ 96	$ 94	$ 122
Bankruptcy [Member]
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	60 days
Troubled Debt Restructurings [Member]
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	90 days
Commercial Real Estate Or Second Mortgage [Member]
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	120 days