Exhibit 23.1

Consent of Independent Registered Public Accounting Firm

The Board of Directors

R.G. Barry Corporation:

We consent to the incorporation by reference in the Registration Statements (Nos. 33-23567, 33-67596, 33-83252, 333-06875, 333-28671, 333-81105, 333-90544, 333-111100, 333-131672 and 333-168828) on Form S-8 of R.G. Barry Corporation of our reports dated September 12, 2012, with respect to the consolidated balance sheets of R.G. Barry Corporation and subsidiaries as of June 30, 2012 and July 2, 2011 and the related consolidated statements of income, shareholders’ equity and comprehensive income, and cash flows for the fiscal years ended June 30, 2012, July 2, 2011 and July 3, 2010 and the effectiveness of internal control over financial reporting as of June 30, 2012, which reports appear in the Annual Report on Form 10-K of R.G. Barry Corporation for the fiscal year ended June 30, 2012.

/s/ KPMG LLP

Columbus, Ohio

September 12, 2012