
<PAGE>
 
                                                                      Exhibit 23


               Consent of Ernst & Young LLP, Independent Auditors


    We consent to the incorporation by reference in this Annual Report (Form
10-K) of Polymer Group, Inc. of our report dated January 23, 1997, included in
the 1996 Annual Report to Shareholders of Polymer Group, Inc.

    Our audits also included the financial statement schedule of Polymer Group,
Inc. listed in Item 14(a). This schedule is the responsibility of the Company's
management. Our responsibility is to express an opinion based on our audits. In
our opinion, the financial statement schedule referred to above, when considered
in relation to the basic financial statements taken as a whole, presents fairly
in all material respects the information set forth therein.

    We also consent to the incorporation by reference in the Registration
Statement (Form S-8 No. 33-04969) pertaining to the 1996 Key Employee Stock
Option Plan of Polymer Group, Inc. of our report dated January 23, 1997 with
respect to the consolidated financial statements incorporated herein by
reference, and our report included in the preceeding paragraph with respect to
the financial statement schedule included in this Annual Report (Form 10-K) of
Polymer Group, Inc.






                                             /s/ Ernst & Young LLP



Greenville, South Carolina
January 23, 1997


                                      30
