Loans Held for Investment - Schedule of Past Due Status of Loans Held for Investment, Gross (Details) - USD ($) $ in Thousands	12 Months Ended
Jun. 30, 2020	Jun. 30, 2019
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	90 days
Current	$ 898,947	$ 874,032
30-89 Days Past Due	219	665
Non-Accrual	5,300	6,641
Total Loans Held for Investment, Gross	904,466	881,338
Mortgage loans, Single-family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	293,326	318,671
30-89 Days Past Due	219	660
Non-Accrual	5,265	5,621
Total Loans Held for Investment, Gross	298,810	324,952
Mortgage Loans, Multi Family
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	491,903	439,041
30-89 Days Past Due	0	0
Non-Accrual	0	0
Total Loans Held for Investment, Gross	491,903	439,041
Mortgage loans, Commercial real estate
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	105,235	111,928
30-89 Days Past Due	0	0
Non-Accrual	0	0
Total Loans Held for Investment, Gross	105,235	111,928
Construction
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	7,801	3,667
30-89 Days Past Due	0	0
Non-Accrual	0	971
Total Loans Held for Investment, Gross	7,801	4,638
Mortgage Loans Other
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	143	167
30-89 Days Past Due	0	0
Non-Accrual	0	0
Total Loans Held for Investment, Gross	143	167
Commercial business loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	445	429
30-89 Days Past Due	0	0
Non-Accrual	35	49
Total Loans Held for Investment, Gross	480	478
Consumer loans
Financing Receivable, Recorded Investment, Past Due [Line Items]
Current	94	129
30-89 Days Past Due	0	5
Non-Accrual	0	0
Total Loans Held for Investment, Gross	$ 94	$ 134
Bankruptcy [Member]
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	60 days
Troubled Debt Restructurings [Member]
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	90 days
Commercial Real Estate Or Second Mortgage [Member]
Financing Receivable, Recorded Investment, Past Due [Line Items]
Loans deemed uncollectible, period of delinquency	120 days