At the height of the pandemic, the richest New Yorkers left in droves.

The trend led to months of hand-wringing, with both Mayor Eric Adams and Governor Kathy Hochul spurning proposals to raise taxes on the rich for fear of driving more of them to low-tax states.

Now, a new report based on the latest census and state tax filing data has found a reversal: The ranks of millionaires have come surging back, while lower- and middle-income New Yorkers are heading for the exits, according to the study, published Tuesday by the Fiscal Policy Institute, a nonprofit policy group.

The people leaving New York at the fastest rate last year were families making between $32,000 and $65,000. A disproportionately high share of these movers were Black and Hispanic. They were followed by people earning $104,000 to $172,000 a year, an above-average income in many parts of the country but a more modest one in New York City.

Continuing to lose these residents, who form the backbone of many essential services and white-collar industries, could jeopardize the city’s uneven recovery, said Andrew Beveridge, the president of Social Explorer, a demographic firm that reviewed the new data.

“If you want a subway system, an office sector, a restaurant industry, you need these people,” he said.

The report also found that affluent residents who left New York did not appear to have been driven away by recent tax increases. More than three-quarters of rich people who left during the pandemic moved to other high-tax states, including Connecticut, New Jersey and California. The report defines this group as the top 1 percent of income-earners, making more than $815,000 a year.

The findings come at a time when the city is preparing to slash the budgets of public services including police, sanitation and schools — cuts that could push more working-class residents out of the state, said Nathan Gusdorf, the director of the Fiscal Policy Institute.

“The main priority for policymakers should be retaining the middle- and working-class populations of New York, by making it affordable and livable,” he added.

New York lost 431,000 residents from July 2020 to July 2022, wiping out half of the state’s population gains through the last decade, according to the report. Since 2020, nearly 94 percent of the state’s population loss was from New York City.

At first, the exodus was led by the wealthiest New Yorkers, who were more likely to work remotely, and had the means to move, Mr. Gusdorf said.

But the trend has turned around. While the state lost 2,400 millionaire households from 2020 to 2022, there was a net gain of 15,100 in the same period, because of strong financial markets that boosted earnings, and the return of some families. In 2022, the most recent year data was available, the richest New Yorkers left the state at far lower rates than all other income groups, in line with prepandemic norms.

Instead, lower-income families have been moving away at higher rates. On net, more than 65,000 residents who made $32,000 to $65,000 left the state last year, or 2 percent of that population. That is the most of any income group, and three times the rate of outbound migration for the wealthiest 1 percent of New Yorkers.

The next largest group of people who left the state last year was those who made between $104,000 and $172,000, with a net loss of 58,000 people. The median income in New York City was about $75,000 last year.

Among the lower-income group, Black and Hispanic New Yorkers were at least twice as likely to move out of the state as white residents, according to the Fiscal Policy Institute.

For many working-class New Yorkers, it was the high cost of living, not taxes, that drove them away.

Danna Dennis, 40, who was raised in Queens and Brooklyn, moved to Newark, N.J., in 2019, when she was pregnant with her first child. Ms. Dennis, a community organizer for a transit nonprofit group, was making about $50,000 a year, and had been paying around $600 a month to rent a room in East Flatbush, Brooklyn.

Child care blew up her budget. To move to a two-bedroom apartment, she would have had to pay around $2,500 a month in Brooklyn. The lowest quote she received for day care was $2,700 a month. After being turned down for a below-market-rate apartment in the Bronx, she had a breakdown.

“I cried the whole way home on the 2 train,” she recalled. “I said, ‘That’s it, I give up.’”

Now she rents a three-bedroom apartment with her husband, Ifeanyi Njoku, a group-home caregiver, and their two children in East Orange, N.J., for $2,800 a month, although she still works in New York. The couple now has a combined annual income of $130,000, but it’s still not enough to comfortably afford rising rent and child care costs, including medical expenses for their older son, who has special needs.

“I call us the ‘make too much, but not enough’ demographic,” she said. “You either have to be all the way on the top, or you have to be way on the bottom.”

Raising taxes on the wealthiest New Yorkers could help fund public services like free prekindergarten and housing subsidies that could help reduce those burdens — with little impact on the city’s millionaires, Mr. Gusdorf said.

The group found that there was no meaningful increase in the number of millionaires who left the state after tax increases on the richest New Yorkers in 2017 and 2021.

A spokesman for the governor referred to a recent statement in which Ms. Hochul said she would not raise taxes this year. “Taxes are high enough in the state of New York and we have to live within our means,” she said.

The mayor’s office said that amid fiscal challenges, “New Yorkers cannot be asked to shoulder the burden” for the failings of the state and federal governments.

Wealthy New Yorkers are crucial to the city and state’s economy, and any discussion of raising their taxes becomes delicate. Millionaires who pay taxes contributed 45 percent of New York State’s total income tax revenue in 2021, the highest share since 2015, according to an analysis of state data by the Fiscal Policy Institute.

E.J. McMahon, the founding senior fellow at the Empire Center for Public Policy, a fiscally conservative research group, said New York City should “absolutely not” raise taxes on the rich, because it already has the highest combined personal-income-tax rate in the country.

For many working-class New Yorkers who have left, it’s getting harder to justify returning.

Risalat Zakaria, 37, a film and television editor, was living in a 950-square-foot co-op in the South Bronx with his wife and son when they decided in 2021 to move to St. Louis to be closer to his wife’s law school.

The couple makes around $110,000 a year — less than they did in New York — but their new cost of living is significantly lower. For full-day child care during the workweek, they pay $900 a month — a third of what they might pay for similar care in New York, Mr. Zakaria said.

For all the changes, Mr. Zakaria, who grew up in Corona, Queens, isn’t feeling homesick. He has been back to the city just a handful of times, he said.

“Each time is more miserable than the last,” he said.