The U.S. Consumer Financial Protection Bureau has formally begun a rulemaking process that could result in new regulations limiting what credit card companies can charge customers in late fees, the agency said Wednesday.

The CFPB estimates that such fees cost Americans $12 billion annually, with the average late fee costing $26. Consumers with the lowest credit scores, who typically live in underprivileged zip codes, pay on average $300 per year in late fees alone, the agency said in a report issued in March.

“Credit cards should be a safe, secure and fair source of small dollar credit, in part because Congress has codified a lot of safeguards into federal law that give people confidence,” CFPB Director Rohit Chopra told reporters at a press conference Wednesday. “Our effort is particularly timely given that the [current rules] allow credit card companies to hike late fees by the rate of inflation.”

He noted that the 2009 CARD act and the 2010 law that created the CFPB give the agency authority to regulate late fees so that they are imposed only to recoup the cost of late payments to issuers. The Federal Reserve, which previously had authority to enforce the law, also included a provision in its rule making that gave credit card companies immunity from enforcement even if these fees generated excess profits.

The CFPB on Wednesday took the first step in amending these rules by publishing an advance notice of proposed rule making, asking that credit card issuers, consumer groups and the public weigh in to help the agency understand how late fees are set, whether they in fact deter late payments and if issuers are generating excess profits from them.

The rulemaking process will likely stretch into next year, but could ultimately save Americans billions of dollars in late fee payments, senior CFPB officials said.

The CFPB says that credit card issuers earn $120 billion per year from credit card fees and interest, with 10% of that revenue coming from late fees.

Credit cards are often issued by large banks like Chase   JPM, -0.08% and Wells Fargo WFC, -0.59%  in partnership with companies like Visa  V, +0.93% and Mastercard  MA, +0.18% that maintain the payment networks. Many community banks, credit unions and retailers are also involved in credit-card issuance.