
<PAGE>
 
                                                                   EXHIBIT 23.1
 
              CONSENT OF ERNST & YOUNG LLP, INDEPENDENT AUDITORS
   
  We consent to the reference to our firm under the caption "Experts" in
Amendment No. 1 to the Registration Statement (Form S-4) and related
Prospectus of Polymer Group, Inc. for the registration of $400 million in 9%
senior subordinated notes due 2007 and to the inclusion therein and to the
incorporation by reference therein of our report dated January 23, 1997, with
respect to the consolidated financial statements and to the incorporation by
reference therein of our report dated January 23, 1997, with respect to the
financial statement schedule of Polymer Group, Inc. included in its Annual
Report (Form 10-K) for the year ended December 28, 1996, filed with the
Securities and Exchange Commission.     
 
                                          /s/ Ernst & Young LLP
 
Greenville, South Carolina
   
September 3, 1997     
