<Page>

                                                                    EXHIBIT 10.4

                          [MATLINPATTERSON LETTERHEAD]

April 11, 2003

Polymer Group, Inc.
P.O. Box 5069
North Charleston, SC 29405

Ladies and Gentlemen:

          MatlinPatterson Global Opportunities Partners L.P. (the "Fund")
currently beneficially owns and controls $38.0 million aggregate principal
amount of the 10% Convertible Subordinated Notes due 2007 (the "Junior Notes")
issued by Polymer Group, Inc. (the "Company") pursuant to the indenture dated as
of March 5, 2003 (the "Indenture"), between the Company and Wilmington Trust
Company, as trustee. The Junior Notes are convertible into shares of the
Company's Class A common stock at the option of the holder at any time and
currently bear cash interest at the rate of 10% per annum (12% in the case of a
default under the Junior Notes). The Fund has also caused to be issued an
irrevocable letter of credit dated as of March 5, 2003 (the "Letter of Credit"),
in favor of JPMorgan Chase Bank, as administrative agent for the lenders under
the Company's Third Amended, Restated and Consolidated Credit Agreement dated as
of March 5, 2003 (the "Credit Agreement"). The Fund and the Company are party to
the Senior Subordinated Note Purchase Agreement dated as of March 5, 2003 (the
"Senior Note Agreement"), which provided for the issuance to the Fund of a
Senior Subordinated Note dated as of March 5, 2003 (the "Senior Note"), to
evidence future drawings under the Letter of Credit. The Senior Note bears
interest on a cash basis at the rate of 10% per annum (12% in the case of a
default under the Senior Note) on any outstanding amounts.

          The Company has obtained the consent of the lenders under the Credit
Agreement to enter into Amendment No. 1 thereto substantially in the form of
Exhibit A attached hereto ("Amendment No. 1 "), and the Company and the lenders
have entered into Amendment No. 1. Solely for the purpose of assisting the
Company in maintaining compliance with the financial covenants during the period
beginning on March 6, 2003 and ending on January 4, 2004 (the "Period"), the
Fund hereby agrees with the Company to take the following steps as follows:

1.   The Fund will amend the Senior Note and such of the Junior
     Notes it holds or controls so that interest that accrues on
     such Senior Note from March 6, 2003 until January 31, 2005
     and interest that accrues on such Junior Notes from March 6,
     2003 until January 5, 2004, will not be required to be paid
     by the Company in cash, but rather will, at the sole option
     of the Company, be payable by having the Company issue an
     additional principal amount of Senior Note and Junior Notes,
     respectively ("PIK Notes"), which PIK Notes shall also
     provide that interest is payable in additional principal
     amount of Senior Note and Junior Notes, respectively, in
     full and complete satisfaction of any and all accrued and
     unpaid interest on such Senior Note and Junior Notes. The
     Fund will also consent to the necessary changes to the
     Senior Note Agreement and the Indenture to allow for the
     issuance of PIK Notes.

<Page>

2.   In the event that on or before May 15, 2004, the Company is
     unable to meet the financial covenants contained in Section
     9.10 (b) of the Credit Agreement (the "Senior Leverage
     Covenant"), Section 9.10 (c) of the Credit Agreement (the
     "Interest Covenant"), or Section 9.10 (d) of the Credit
     Agreement (the "Adjusted Interest Covenant"), the Fund will
     allow the Company to instruct the Agent to (and if the
     Company fails to so instruct the Agent, the Agent may) draw
     under the Letter of Credit and use the proceeds from such
     drawing to make a corresponding dollar-for-dollar reduction
     of an amount outstanding under the Credit Agreement, up to
     the full amount of the remaining balance of the Letter of
     Credit, to be in compliance with such Covenant.

3.   In the event that the Company is unable to meet the
     financial covenant contained in Section 9.10 (a) of the
     Credit Agreement (the "Leverage Covenant"), including in the
     circumstance resulting from a drawing under the Letter of
     Credit to comply with the Senior Leverage Covenant and
     corresponding increase in the amount outstanding under the
     Senior Note, at the request of the Company at any time on or
     before May 15, 2004, the Fund will convert such of its
     Junior Notes into shares of Class A common stock to reduce
     total outstanding indebtedness of the Company such that the
     Company will, following such conversion, be in compliance
     with the Leverage Covenant.

4.   In the event that the Company has undertaken the actions set
     forth in paragraphs 1, 2 and 3 above and, after taking such
     actions the Company is unable to meet the Interest Covenant
     solely for the 12-month period ending on the last day of the
     Company's third fiscal quarter of 2003, the Fund will
     purchase additional securities from the Company in an
     aggregate principal amount of up to $10 million (the "New
     Securities"), such that following any such purchase, the
     Company will be in compliance with the Interest Covenant.
     The New Securities will be in the form of senior
     subordinated debt or equity securities, as may be agreed
     between the Fund and the Company. In the event the New
     Securities are senior subordinated debt securities, any
     interest payments required to be made during the Period
     shall only be made by issuing additional principal amount of
     New Securities, and no cash payment shall be made in respect
     of interest payments on the New Securities during the
     Period.

All of the obligations of the Fund contained in this agreement are subject to
the continuing condition that Amendment No. 1 shall be in full force and effect.
This agreement is for the sole benefit of the Company, is only enforceable by
the Company, and may not be relied upon by any third party. No party shall
become a third party beneficiary as a result of this agreement.

                                        2
<Page>

          If you are in agreement with the above terms, please indicate your
acceptance by signing and returning a copy to us, following which this shall
become a binding agreement.


                                        MatlinPatterson Global Opportunities
                                        Partners L.P.

                                        By: MatlinPatterson Global Advisors
                                            LLC, its investment advisor

                                        By:
                                           -----------------------------------
                                           Name:
                                           Title:


Confirmed and agreed:
Polymer Group, Inc.


By:
    ----------------------
    Name:
    Title:

                                        3

