Organization and Summary of Significant Accounting Policies - Loans originated and held for sale (Details) - USD ($)	12 Months Ended
	Jun. 30, 2020	Jun. 30, 2019	Jun. 30, 2020	Jun. 30, 2019
Accounts, Notes, Loans and Financing Receivable [Line Items]
Loans serviced under MPF program			$ 7,400,000	$ 9,700,000
Recourse Liability	$ 270,000	$ 250,000	270,000	250,000
Recourse Liability [Roll Forward]
Balance, beginning of the period	250,000	283,000
Recourse recovery	20,000	(33,000)
Balance, end of the period	270,000	250,000
Mortgage Partnership Finance (MPF) Program
Accounts, Notes, Loans and Financing Receivable [Line Items]
Recourse Liability	70,000	50,000	70,000	50,000
Recourse Liability [Roll Forward]
Balance, beginning of the period	50,000
Balance, end of the period	70,000	50,000
Other Investors
Accounts, Notes, Loans and Financing Receivable [Line Items]
Recourse Liability	200,000	200,000	$ 200,000	200,000
Recourse Liability [Roll Forward]
Balance, beginning of the period	200,000
Balance, end of the period	200,000	200,000
Provident Bank Mortgage (PBM)
Accounts, Notes, Loans and Financing Receivable [Line Items]
Recourse Liability	25,000	25,000		$ 25,000
Recourse Liability [Roll Forward]
Balance, beginning of the period	25,000
Balance, end of the period		25,000
Loan sale premium refunds	78,000	96,000
Provident Bank Mortgage (PBM) \| Mortgage Partnership Finance (MPF) Program
Accounts, Notes, Loans and Financing Receivable [Line Items]
Repurchases of single-family loans	$ 1,100,000	$ 948,000