Loans Receivable and Allowance for Loan Losses - Past Due Financing Receivables (Details) - USD ($) $ in Thousands	Jun. 30, 2019	Dec. 31, 2018
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	$ 64,934	$ 53,286
Accruing loans past due 90 or more days	353	1,529
Guaranteed portion of delinquent SBA loans excluded from Nonaccrual loans	32,100	29,200
Past due loans	65,093	56,936
30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	20,501	28,467
60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	3,552	1,407
90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	41,040	27,062
Legacy Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	50,249	42,248
Accruing loans past due 90 or more days	353	243
Past due loans	51,006	47,005
Legacy Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	18,195	27,137
Legacy Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	2,685	575
Legacy Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	30,126	19,293
Acquired Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	14,685	11,038
Accruing loans past due 90 or more days	0	1,286
Past due loans	14,087	9,931
Acquired Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	2,306	1,330
Acquired Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	867	832
Acquired Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	10,914	7,769
Residential \| Legacy Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	0	0
Accruing loans past due 90 or more days	0	0
Residential \| Acquired Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	0	0
Accruing loans past due 90 or more days	0	0
Commercial \| Legacy Loans \| Retail
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	4,682	5,153
Accruing loans past due 90 or more days	0	0
Commercial \| Legacy Loans \| Hotel & motel
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	6,503	7,325
Accruing loans past due 90 or more days	0	0
Commercial \| Legacy Loans \| Gas station & car wash
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	31	31
Accruing loans past due 90 or more days	0	0
Commercial \| Legacy Loans \| Mixed use
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	717	749
Accruing loans past due 90 or more days	0	0
Commercial \| Legacy Loans \| Industrial & warehouse
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	7,278	6,111
Accruing loans past due 90 or more days	0	0
Commercial \| Legacy Loans \| Other
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	10,794	5,940
Accruing loans past due 90 or more days	0	0
Commercial \| Acquired Loans \| Retail
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	5,350	829
Accruing loans past due 90 or more days	0	0
Commercial \| Acquired Loans \| Hotel & motel
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	5,423	5,500
Accruing loans past due 90 or more days	0	1,286
Commercial \| Acquired Loans \| Gas station & car wash
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	245	247
Accruing loans past due 90 or more days	0	0
Commercial \| Acquired Loans \| Mixed use
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	370	1,224
Accruing loans past due 90 or more days	0	0
Commercial \| Acquired Loans \| Industrial & warehouse
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	328	349
Accruing loans past due 90 or more days	0	0
Commercial \| Acquired Loans \| Other
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	1,201	259
Accruing loans past due 90 or more days	0	0
Construction \| Legacy Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	0	0
Accruing loans past due 90 or more days	0	0
Construction \| Acquired Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	0	0
Accruing loans past due 90 or more days	0	0
Commercial business \| Legacy Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	15,318	14,837
Accruing loans past due 90 or more days	0	0
Past due loans	12,242	12,938
Commercial business \| Legacy Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	1,450	5,500
Commercial business \| Legacy Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	121	435
Commercial business \| Legacy Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	10,671	7,003
Commercial business \| Acquired Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	958	1,632
Accruing loans past due 90 or more days	0	0
Past due loans	321	1,263
Commercial business \| Acquired Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	84	397
Commercial business \| Acquired Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	613
Commercial business \| Acquired Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	237	253
Trade finance \| Legacy Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	2,721	1,661
Accruing loans past due 90 or more days	0	0
Past due loans	2,721	2,697
Trade finance \| Legacy Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	1,036
Trade finance \| Legacy Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Trade finance \| Legacy Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	2,721	1,661
Trade finance \| Acquired Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	0	0
Accruing loans past due 90 or more days	0	0
Past due loans	0	0
Trade finance \| Acquired Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Trade finance \| Acquired Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Trade finance \| Acquired Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Consumer and other \| Legacy Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	2,205	441
Accruing loans past due 90 or more days	353	243
Past due loans	7,767	16,800
Consumer and other \| Legacy Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	5,721	16,413
Consumer and other \| Legacy Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	250	140
Consumer and other \| Legacy Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	1,796	247
Consumer and other \| Acquired Loans
Financing Receivable, Past Due [Line Items]
Nonaccrual loans	810	998
Accruing loans past due 90 or more days	0	0
Past due loans	2,314	334
Consumer and other \| Acquired Loans \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	1,337	0
Consumer and other \| Acquired Loans \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	709	0
Consumer and other \| Acquired Loans \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	268	334
Real estate \| Legacy Loans \| Residential
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Residential \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Residential \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Residential \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Construction
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Construction \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Construction \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Construction \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Retail \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	1,108	1,542
Real estate \| Legacy Loans \| Retail \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	733
Real estate \| Legacy Loans \| Retail \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	811	0
Real estate \| Legacy Loans \| Retail \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	297	809
Real estate \| Legacy Loans \| Hotel & motel \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	5,743	5,368
Real estate \| Legacy Loans \| Hotel & motel \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	1,716	153
Real estate \| Legacy Loans \| Hotel & motel \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	347	0
Real estate \| Legacy Loans \| Hotel & motel \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	3,680	5,215
Real estate \| Legacy Loans \| Gas station & car wash \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	689	31
Real estate \| Legacy Loans \| Gas station & car wash \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	658	0
Real estate \| Legacy Loans \| Gas station & car wash \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Gas station & car wash \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	31	31
Real estate \| Legacy Loans \| Mixed use \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	358	0
Real estate \| Legacy Loans \| Mixed use \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Legacy Loans \| Mixed use \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	174	0
Real estate \| Legacy Loans \| Mixed use \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	184	0
Real estate \| Legacy Loans \| Industrial & warehouse \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	4,310	3,387
Real estate \| Legacy Loans \| Industrial & warehouse \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	74	1,465
Real estate \| Legacy Loans \| Industrial & warehouse \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	849	0
Real estate \| Legacy Loans \| Industrial & warehouse \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	3,387	1,922
Real estate \| Legacy Loans \| Other \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	16,068	4,242
Real estate \| Legacy Loans \| Other \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	8,576	1,837
Real estate \| Legacy Loans \| Other \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	133	0
Real estate \| Legacy Loans \| Other \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	7,359	2,405
Real estate \| Acquired Loans \| Residential
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Residential \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Residential \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Residential \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Construction
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Construction \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Construction \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Construction \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Retail \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	5,039	949
Real estate \| Acquired Loans \| Retail \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	347
Real estate \| Acquired Loans \| Retail \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	38	0
Real estate \| Acquired Loans \| Retail \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	5,001	602
Real estate \| Acquired Loans \| Hotel & motel \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	4,462	5,206
Real estate \| Acquired Loans \| Hotel & motel \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Hotel & motel \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Hotel & motel \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	4,462	5,206
Real estate \| Acquired Loans \| Gas station & car wash \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	221	375
Real estate \| Acquired Loans \| Gas station & car wash \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	154
Real estate \| Acquired Loans \| Gas station & car wash \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Gas station & car wash \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	221	221
Real estate \| Acquired Loans \| Mixed use \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	194	1,141
Real estate \| Acquired Loans \| Mixed use \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	107
Real estate \| Acquired Loans \| Mixed use \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	0
Real estate \| Acquired Loans \| Mixed use \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	194	1,034
Real estate \| Acquired Loans \| Industrial & warehouse \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	542	261
Real estate \| Acquired Loans \| Industrial & warehouse \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	94	142
Real estate \| Acquired Loans \| Industrial & warehouse \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	120	0
Real estate \| Acquired Loans \| Industrial & warehouse \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	328	119
Real estate \| Acquired Loans \| Other \| Commercial
Financing Receivable, Past Due [Line Items]
Past due loans	994	402
Real estate \| Acquired Loans \| Other \| Commercial \| 30 to 59 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	791	183
Real estate \| Acquired Loans \| Other \| Commercial \| 60 to 89 Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	0	219
Real estate \| Acquired Loans \| Other \| Commercial \| 90 or More Days Past Due
Financing Receivable, Past Due [Line Items]
Past due loans	$ 203	$ 0

Loans Receivable and Allowance for Loan Losses - Past Due Financing Receivables (Details) - USD ($)
$ in Thousands

Jun. 30, 2019

Dec. 31, 2018

Financing Receivable, Past Due [Line Items]

Nonaccrual loans

$ 64,934

$ 53,286

Accruing loans past due 90 or more days

353

1,529

Guaranteed portion of delinquent SBA loans excluded from Nonaccrual loans

32,100