
<PAGE>   1
                                                                   EXHIBIT 10.23


                                 AMENDMENT NO. 1
                                     TO THE
                             R. G. BARRY CORPORATION
                           DEFERRED COMPENSATION PLAN
                         (EFFECTIVE AS OF MARCH 1, 1997)


         WHEREAS, R. G. Barry Corporation ("Company") maintains the "R. G. Barry
Corporation Deferred Compensation Plan", effective as of September 1, 1995, and
as may be subsequently amended (hereinafter referred to as the "Plan"), for the
benefit of its Eligible Employees and the Eligible Employees of any
participating Affiliate;

         WHEREAS, the Company desires to amend the provisions of the Plan to
increase the maximum Matching Amount; and

         WHEREAS, Section 9.1 of the Plan provides that the Board of Directors
of the Company may amend the Plan from time to time with respect to all
participating Employers under the Plan;

         NOW, THEREFORE, in accordance with the provisions of Section 9.1 of the
Plan, the following actions are hereby taken and the Plan is hereby amended in
the following respect:

                  Section 4.4, Matching Amount, of the Plan shall be deleted in
                  its entirety, and the following new Section 4.4 shall be
                  substituted therefor:

                  4.4 MATCHING AMOUNT

                  For each Plan Year, each Employer shall allocate a Matching
                  Amount on behalf of each Participant who has elected to defer
                  a Salary Deferral Amount or Bonus Deferral Amount for the Plan
                  Year. The Matching Amount shall equal 50 percent of the Salary
                  Deferral Amount deferred by a Participant for each payroll
                  period; provided, however, that the maximum Matching Amount
                  shall not exceed one and one-half percent of the Participant's
                  base salary for each payroll period. In addition, the Matching
                  Amount shall equal 50 percent of the Bonus Deferral Amount
                  deferred by a Participant; provided, however, that the maximum
                  Matching Amount for a Plan Year, including the Matching Amount
                  attributable to the Salary Deferral Amount, shall not exceed
                  one and one-half percent of the Participant's annual base
                  salary. Notwithstanding the preceding sentence, the Maximum
                  Matching Amount for the Plan Year
<PAGE>   2
                  ending December 31, 1997, including the Matching Amount
                  attributable to the Salary Deferral Amount, shall not exceed
                  the sum of (a) one percent of the Participant's annual base
                  salary for January and February 1997 and (b) one and one-half
                  percent of the Participant's annual base salary for March
                  through December 1997. Matching Amounts to be allocated on
                  behalf of a participant for any Plan Year shall be determined
                  in reference to each payroll period within such Plan Year and
                  to the date when the Bonus earned during the Plan Year would
                  be paid if no deferral election applied.

                                * * * * * * * * *

         IN WITNESS WHEREOF, R. G. Barry Corporation has caused this instrument
to be executed on this 1st day of March, 1997, by its duly authorized officers
effective as provided above.


                                    COMPANY:
                                    R. G. BARRY CORPORATION

                                    By:  /s/ Harry Miller
                                        ----------------------------------------
                                         Vice President of Human Resources


                                    By:  /s/ Richard L. Burrell
                                       -----------------------------------------
                                         Senior Vice President of Finance and
                                         Treasurer


                                    By:  /s/ Michael Krasnoff
                                       -----------------------------------------
                                         Vice President of Finance and Assistant
                                         Treasurer

                                      -2-
