
<PAGE>
 
                                                                   Exhibit 10.32

                                VOTING AGREEMENT
                                ----------------


          THIS AGREEMENT is made as of May 15, 1996, by and among Polymer Group,
Inc., a Delaware corporation (the "Company"), The InterTech Group, Inc., a South
Carolina corporation ("InterTech"), Golder, Thoma, Cressey Fund III Limited
Partnership, an Illinois limited partnership ("GTC"), Jerry Zucker ("Zucker"),
James G. Boyd ("Boyd"), FTG, Inc., a South Carolina corporation ("FTG"), Chase
Manhattan Investment Holdings, Inc., a Delaware corporation ("Chase") and Leeway
& Co. ("Leeway").  InterTech, GTC, Zucker, Boyd, FTG, Chase and Leeway are
sometimes collectively referred to herein as the "Stockholders" and individually
as a "Stockholder."  Certain capitalized terms used herein are defined in
paragraph 4 hereof.

          The Company and the Stockholders desire to enter into this Agreement
for the purpose of establishing the composition of the Company's board of
directors (the "Board").

          NOW, THEREFORE, in consideration of the mutual covenants contained
herein and other good and valuable consideration, the receipt and sufficiency of
which are hereby acknowledged, the parties to this Agreement hereby agree as
follows:

          1.  Board of Directors.

          (a) From and after the date hereof and until the provisions of this
paragraph 1 cease to be effective, each Stockholder shall vote all of his or its
Stockholder Shares and any other voting securities of the Company over which
such Stockholder has voting control and shall take all other necessary or
desirable actions within his or its control (whether in the capacity as a
stockholder, director, member of a board committee or officer of the Company or
otherwise, and including, without limitation, attendance at meetings in person
or by proxy for purposes of obtaining a quorum and execution of written consents
in lieu of meetings), and the Company shall take all necessary and desirable
actions within its control (including, without limitation, calling special board
and stockholder meetings), so that:

               (i)   the authorized number of directors on the Board shall be
     established at seven directors;

               (ii)  the following persons shall be elected to the Board:

                     (A) the Chief Executive Officer of the Company;

                     (B) the Executive Vice President of the Company;
                 
                     (C) two representatives designated by GTC (the "GTC 
          Directors"); and
<PAGE>
 
                     (D) two representatives jointly selected by GTC and the ZB
          Group (based upon a vote of the holders of a majority of the Company's
          voting stock held by the ZB Group (the "Independent Directors")),
          provided that no Independent Director shall be (x) a member of the
          Company's management or an employee or officer of the Company or any
          of its Subsidiaries or (y) an officer, stockholder, general partner or
          employee of GTC, any member of the ZB Group or any of their
          Affiliates.

               (iii) any committees of the Board are to be created only upon
     the approval of a majority of the members of the Board;

               (iv)  the removal from the Board (with or without cause) of any
     representative designated hereunder pursuant to (ii)(C) and (ii)(D) above
     shall be at the written request of GTC and of GTC and the holders of a
     majority of the Stockholder Shares held by the ZB Group, respectively, but
     only upon such written request and under no other circumstances; and

               (v) in the event that any representative designated pursuant to
     (ii)(C) and (ii)(D) above for any reason ceases to serve as a member of the
     Board during his term of office, the resulting vacancy on the Board shall
     be filled by a representative designated by GTC and by GTC and the holders
     of a majority of the Stockholder Shares held by the ZB Group, respectively,
     as provided hereunder.

          (b) The Company shall pay the reasonable out-of-pocket expenses
incurred by each director in connection with attending the meetings of the Board
and any committee thereof. In addition, the Company shall pay to each GTC
Director and each Independent Director an annual fee of $5,000, and $500 for any
committee meeting attended on a day other than a day of a Board meeting; which
amounts shall be subject to periodic review and increase by the Board. So long
as any GTC Director or Independent Director serves on the Board and for five
years thereafter, the Company shall maintain directors and officers indemnity
insurance coverage satisfactory to GTC, and the Company's certificate of
incorporation and bylaws shall provide for indemnification and exculpation of
directors to the fullest extent permitted under applicable law.

          (c) The rights of GTC and the ZB Group, respectively, under this
paragraph 1 shall terminate at such time as any such Person (together with its
Permitted Transferees) holds in the aggregate less than 10% of the Common Stock
on a fully diluted basis (assuming the exercise of all outstanding options,
warrants and other securities convertible into or exchangeable for Common
Stock).

          (d) If any party fails to designate a representative to fill a
directorship pursuant to the terms of this paragraph 1, the election of a person
to such directorship shall be accomplished in accordance with the Company's or
any Subsidiary's by-laws and applicable law, as appropriate.

          2. Legend.  Each certificate evidencing voting capital stock of the
Company owned by a party hereto and each certificate issued in exchange for or
upon the transfer of any such
               
                                      -2-
<PAGE>
 
securities (if such shares remain subject hereto after such transfer) shall be
stamped or otherwise imprinted with a legend in substantially the following
form:

     "The securities represented by this certificate are subject to a Voting
     Agreement dated as of May 15, 1996, among the issuer of such securities
     (the "Company") and certain of the Company's stockholders.  A copy of such
     Voting Agreement will be furnished without charge by the Company to the
     holder hereof upon written request."

The Company shall imprint such legend on certificates evidencing Stockholder
Shares outstanding prior to the date hereof.  The legend set forth above shall
be removed from the certificates evidencing any shares which cease to be
Stockholder Shares in accordance with the terms hereof.

          3. Transfer.  Prior to transferring any Stockholder Shares (other than
in a Public Sale or upon the Sale of the Company to any Person), the
transferring Stockholder shall cause the prospective transferee to execute and
deliver to the Company and the other Stockholders a counterpart of this
Agreement.

          4. Definitions.

          "Affiliate" means any Person which controls, is controlled by or is
under common control with another Person, any partner of any Person which is a
partnership and Persons which have received distributions of securities from a
partnership holding such securities.

          "Certificate of Incorporation" means the Company's amended and
restated certificate of incorporation of the Company as filed with the Delaware
Secretary of State on May 15, 1996.

          "Common Stock" means the Company's Common Stock, par value $.01 per
share.

          "Family Group" means an Executive's spouse and descendants (whether
natural or adopted) and any trust solely for the benefit of the Executive and/or
the Executive's spouse and/or descendants.

          "Permitted Transferees" means (i) in the case of a Stockholder who is
a natural person, such person's spouse, descendants (whether natural or adopted)
and any trust solely for the benefit of such person and/or such person's spouse
and/or descendants, and (ii) in the case of any other Stockholder, any Affiliate
of such Stockholder.

          "Person" means an individual, a partnership, a corporation, an
association, a joint stock company, a trust, a joint venture, an unincorporated
organization and a governmental entity or any department, agency or political
subdivision thereof.

          "Public Sale" means any sale of Stockholder Shares to the public
pursuant to an offering registered under the Securities Act or to the public
through a broker, dealer or market maker pursuant to the provisions of Rule 144
adopted under the Securities Act.

                                      -3-
<PAGE>
 
          "Recapitalization Agreement" means the Recapitalization Agreement of
even date herewith among the Company and the Stockholders.

          "Securities Act" means the Securities Act of 1933, as amended from
time to time.

          "Stockholder Shares" means (i) any Common Stock owned by any
Stockholder, whether acquired pursuant to the Recapitalization Agreement or
otherwise, and (ii) any equity securities issued or issuable directly or
indirectly with respect to the Securities referred to in clause (i) above by way
of stock dividend or stock split or in connection with a combination of shares,
recapitalization, merger, consolidation or other reorganization. As to any
particular shares constituting Stockholder Shares, such shares will cease to be
Stockholder Shares when they have been disposed of in a Public Sale.

          "Subsidiary" means, with respect to any Person, any corporation,
partnership, association or other business entity of which (i) if a corporation,
a majority of the total voting power of shares of stock entitled (without regard
to the occurrence of any contingency) to vote in the election of directors,
managers or trustees thereof is at the time owned or controlled, directly or
indirectly, by that Person or one or more of the other Subsidiaries of that
Person or a combination thereof, or (ii) if a partnership, association or other
business entity, a majority of the partnership or other similar ownership
interest thereof is at the time owned or controlled, directly or indirectly, by
any Person or one or more Subsidiaries of that Person or a combination thereof.
For purposes hereof, a Person or Persons shall be deemed to have a majority
ownership interest in a partnership, association or other business entity if
such Person or Persons shall be allocated a majority of partnership, association
or other business entity gains or losses or shall be or control the managing
director or general partner of such partnership, association or other business
entity.

          "ZB Group" means Zucker, Boyd, InterTech, and FTG.

          5. Amendment and Waiver.  Except as otherwise provided herein, no
modification, amendment or waiver of any provision of this Agreement shall be
effective against the Company or the Stockholders unless such modification,
amendment or waiver is approved in writing by the Company, GTC and the ZB Group
(determined by a vote of the holders of a majority of the Stockholder Shares
held by the ZB Group).  The failure of any party to enforce any of the
provisions of this Agreement shall in no way be construed as a waiver of such
provisions and shall not affect the right of such party thereafter to enforce
each and every provision of this Agreement in accordance with its terms.

          6. Severability.  Whenever possible, each provision of this Agreement
shall be interpreted in such manner as to be effective and valid under
applicable law, but if any provision of this Agreement is held to be invalid,
illegal or unenforceable in any respect under any applicable law or rule in any
jurisdiction, such invalidity, illegality or unenforceability shall not affect
any other provision or any other jurisdiction, but this Agreement shall be
reformed, construed and enforced in such jurisdiction as if such invalid,
illegal or unenforceable provision had never been contained herein.

                                      -4-
<PAGE>
 
          7.  Entire Agreement.  Except as otherwise expressly set forth herein,
this document embodies the complete agreement and understanding among the
parties hereto with respect to the subject matter hereof and supersedes and
preempts any prior understandings, agreements or representations by or among the
parties, written or oral.

          8. Successors and Assigns.  Except as otherwise provided herein, this
Agreement shall bind the Stockholders and the respective successors and assigns
of each of them.

          9. Counterparts.  This Agreement may be executed in separate
counterparts each of which shall be an original and all of which taken together
shall constitute one and the same agreement.

          10.  Remedies.  The Company and the Stockholders shall be entitled to
enforce their rights under this Agreement specifically to recover damages by
reason of any breach of any provision of this Agreement and to exercise all
other rights existing in their favor.  The parties hereto agree and acknowledge
that money damages may not be an adequate remedy for any breach of the
provisions of the Agreement and that the Company and the Stockholders in its
sole discretion apply to any court of law or equity of competent jurisdiction
for specific performance and/or injunctive relief (without posting a bond or
other security) in order to enforce or prevent any violation of the provisions
of this Agreement.

          11.  Notices.  Any notice provided for in this Agreement shall be in
writing and shall be either personally delivered, or mailed first class mail
(postage prepaid) or sent by reputable overnight courier service (charges
prepaid) to the Company at the address set forth below and to any other
recipient at the address indicated on the signature page hereto and to any
subsequent holder of Stockholder Shares subject to this Agreement at such
address as indicated by the Company's records, or at such address or to the
attention of such other person as the recipient party has specified by prior
written notice to the sending party.  Notices will be deemed to have been given
hereunder when delivered personally, three days after deposit in the U.S. mail
and one day after deposit with a reputable overnight courier service.  The
Company's address is:

                        Polymer Group, Inc.
                        4838 Jenkins Avenue
                        North Charleston, South Carolina  29406
                        Attention:  President

          12.  Governing Law.  The corporate law of Delaware shall govern all
issues concerning the relative rights of the Company and its stockholders.  All
other questions concerning the construction, validity and interpretation of this
Agreement shall be governed by the internal law, and not the law of conflicts,
of New York.

          13.  Descriptive Headings.  The descriptive headings of this Agreement
are inserted for convenience only and do not constitute a part of this
Agreement.

                           *     *     *     *     *

                                      -5-
<PAGE>
 
          IN WITNESS WHEREOF, the parties hereto have executed this Stockholder
Agreement on the day and year first above written.

 
                                        POLYMER GROUP, INC.
 
                                        By:  /s/ Jerry Zucker
                                             ---------------------------  
                                        Its: Chairman, President & CEO
                                             ---------------------------


Address:                                THE INTERTECH GROUP, INC.
 4838 Jenkins Avenue
 North Charleston, SC 29406             By:  /s/ Jerry Zucker
                                             ---------------------------
                                        Its: Chairman, President & CEO
                                             ---------------------------

Address:                                By:  /s/ Bruce V. Rauner
 6100 Sears Tower                            --------------------------- 
 Chicago, IL 60606-6402                 Its: General Partner
                                             --------------------------- 

Address:                                /s/ Jerry Zucker
 c/o The InterTech Group, Inc.          --------------------------------
 4838 Jenkins Avenue                    Jerry Zucker
 North Charleston, SC 29406
 
                                        /s/ James G. Boyd
Address:                                --------------------------------
 c/o The InterTech Group, Inc.          James G. Boyd
 4838 Jenkins Avenue
 North Charleston, SC 29406

                                      -6-
<PAGE>


Address:                            FTG, INC.
 4838 Jenkins Avenue
 North Charleston, SC 29406         By:  /s/  Jerry Zucker
                                         ---------------------------------- 
                                    Its: Chairman, President & CEO
                                         ----------------------------------

Address:                            CHASE MANHATTAN INVESTMENT
 c/o Chase Capital Partners         HOLDINGS, INC.
 380 Madison Avenue, 12th Floor
 New York, NY  10017                By: /s/ Donna L. Carter
 Attention: Robert Ruggiero             -----------------------------------

                                    Its: Senior Vice President & Treasurer
                                         ----------------------------------
 
                                    LEEWAY & CO.

                                    By:  State Street Bank & Trust Company,
Address:                                 Partner
 c/o State Street Bank and
   Trust Company                    By:  /s/ John Muir
 Master Trust Division-Q4W               ----------------------------------
 P.O. Box 1992                      
 Boston, MA 02101                   Its: Assistant Vice President          
                                          --------------------------------- 

                                      -7-
