Seven years later—with more than 1 million former students defaulting on government-backed loans every year and with his former agency pulling the plug on a partnership with the consumer bureau tasked with protecting them—his fears could soon be realized.

To many close observers, a move to cripple the bureau has seemed inevitable. Republican members of Congress for years have tried, without success, to restrict its authority and called for the firing of its director, Richard Cordray. “Think about it: Director Cordray has served longer under Donald Trump than Steve Bannon has,” said Isaac Boltansky, an analyst at Compass Point Research & Trading, referring to the president’s former adviser. The latest move by Education Secretary Betsy DeVos is simply “some of the political appointees of the Trump administration fighting back accordingly.”

Congress and the Obama administration created the CFPB in 2010, after the financial crisis, in an effort to crack down on dodgy loan companies and to supervise businesses that had historically operated with little oversight, such as payday lenders and student loan servicers. Although polls suggest few Americans know about the CFPB, since 2013 they’ve filed more than 15,000 complaints with it about their federal student loans, its public database shows, accusing companies of everything from abusive collection calls to trying to stop them from enrolling in government-promoted, income-based repayment plans.

The consumer bureau has tried to curb loan practices it thinks are hurting borrowers, including actions carried out on behalf of the federal government. It has issued public warnings about potentially deceptive acts and underused debt-relief programs, referred cases to the Department of Justice, and pushed companies to stop practices it says are unfair, such as steering troubled borrowers into short-term plans that postpone their payments but still let interest accrue. Along with two states’ attorneys general, it has sued one of the Education Department’s largest loan contractors, Navient Corp., accusing it of illegally cheating hundreds of thousands of student debtors out of their right to lower their payments. Navient has denied wrongdoing, saying it followed department guidelines.

The Education Department, meanwhile, hasn’t been penalizing its contracted loan servicers for engaging in the practices the CFPB has tried to change. The department has been consistently criticized over the years by government watchdogs—both its own inspector general and the Government Accountability Office—for what they’ve called its lackluster oversight of its contractors. One audit found the agency didn’t even keep track of borrowers’ complaints.

But to DeVos’s agency, the CFPB’s involvement has only complicated the lives of borrowers and the government contractors paid to counsel them and collect on their debts. The two agencies’ directives to loan companies sometimes conflict, which causes confusion, DeVos deputies Kathleen Smith and A. Wayne Johnson told Cordray in the letter last week terminating their partnership. The department, they wrote, is shutting out the bureau to “enhance the efficiencies of our servicers.” Elizabeth Hill, a department spokeswoman, didn’t elaborate.

According to Sarah Bloom Raskin, who worked on student loan policy as deputy Treasury secretary in the Obama administration, the Education Department’s move probably has more to do with protecting its contracted loan servicers from new costs than with helping student debtors.

For instance, student debtors for years have been able to make monthly payments based on what they earn rather than what they owe—but the paperwork required to enroll them can be “very expensive” for loan servicers, Navient’s chief executive, Jack Remondi, said back in 2013, when the company was part of Sallie Mae. Millions of young Americans were eligible for those payment plans but didn’t know about them. The Education Department and its Federal Student Aid office, which manages the government’s student loan portfolio, were content with low enrollment—until around 2012, when the White House and the CFPB began to prod them to direct their contractors to inform eligible borrowers. Since then, enrollment has soared.

The Federal Student Aid office, which Johnson now leads, was created by Congress in 1998 and directed to improve customer service for students. “But at the end of the day, what FSA is judged by is not customer service, but how much money is returned to the Treasury,” Bergeron said. “Their default option is to think of servicers first and borrowers as an afterthought,” Raskin said. And the government’s loan contractors put its interests first; their ultimate customer, after all, is the agency that hired it to collect on debt. Navient told a federal court in March that “there is no expectation that the servicer will act in the interest of the consumer.”

Hill, the department spokeswoman, said her agency is “committed to putting students’ needs first, easing the burdens on student loan borrowers, and enhancing efficiencies for servicers.”

Now the department “has made its position clear that there is no room for CFPB involvement here,” said Brian Slagle, an attorney at Ballard Spahr LLP who represents financial firms before the bureau.

Wall Street has already taken notice. This week, Michael Tarkan, director of research at Compass Point, recommended that clients buy Navient stock. The agency’s decision, he said, is an “unambiguous signal” that the Trump administration will take a more lenient approach to policing student loan companies.