As investigators struggled for weeks to find who might have committed the brutal stabbings of four University of Idaho students in the fall of 2022, they were focused on a key piece of evidence: DNA on a knife sheath that was found at the scene of the crime.

At first they tried checking the DNA with law enforcement databases, but that did not provide a hit. They turned next to the more expansive DNA profiles available in some consumer databases in which users had consented to law enforcement possibly using their information, but that also did not lead to answers.

F.B.I. investigators then went a step further, according to newly released testimony, comparing the DNA profile from the knife sheath with two databases that law enforcement officials are not supposed to tap: GEDmatch and MyHeritage.

It was a decision that appears to have violated key parameters of a Justice Department policy that calls for investigators to operate only in DNA databases “that provide explicit notice to their service users and the public that law enforcement may use their service sites.”

It also seems to have produced results: Days after the F.B.I.’s investigative genetic genealogy team began working with the DNA profiles, it landed on someone who had not been on anyone’s radar: Bryan Kohberger, a Ph.D. student in criminology who has now been charged with the murders.

The case has shown both the promise and the unregulated power of genetic technology in an era in which millions of people willingly contribute their DNA profiles to recreational databases, often to hunt for relatives. In the past, law enforcement officials would need to find a direct match between DNA at the crime scene and that of a specific suspect. Now, investigators can use consumer DNA data to build family trees that can zero in on a person of interest — within certain policy limits.

While some companies have allowed users to choose whether their DNA information may be used to help criminal investigations, the decision by the authorities to skirt those limits could mean that the companies’ privacy assurances are essentially meaningless.

Erin Murphy, a law professor at New York University who focuses on DNA and new policing methods, said she was surprised that the F.B.I. might have violated rules that the federal government had spent so much time working to establish. She was also concerned that investigators seemingly had no repercussions for doing so.

“I think what we are teaching law enforcement is that the rules have no meaning,” she said.

Steve Kramer, a former F.B.I. lawyer who has specialized in genetic genealogy investigations, said the rules were designed as a framework, not a legal limitation. They can help guide typical investigative work, he said, but when it comes to a serious case where other investigative options are limited, such as the Idaho case, investigators may need to take additional steps.

“We’ll never know, thank God, what Bryan Kohberger would have done had he not been caught,” he said.

On the morning of Nov. 13, 2022, four University of Idaho students — Madison Mogen, 21; Kaylee Goncalves, 21; Xana Kernodle, 20; and Ethan Chapin, 20 — were found dead in an off-campus home, the victims of a vicious stabbing spree. The police spent weeks looking for a suspect as residents of the small college town of Moscow, Idaho, waited in fear.

Behind the scenes, investigators were examining a wide range of people: classmates, people charged with prior assaults and some people with the thinnest connections to Idaho. On a knife sheath found next to two of the victims, investigators found DNA. But when they put the sample into the federal law enforcement database CODIS, there was no match.

In recent testimony from a closed-door court hearing, Idaho officials described how on Nov. 22, investigators brought the DNA sample to Othram, a company near Houston that specializes in genetic genealogy, often helping law enforcement solve decades-old cold cases by taking a modern analysis of the DNA profile. Othram began doing genetic genealogy and building a family tree, apparently following the protocols of Justice Department policy.

Mr. Kramer said the policy had been put into place after he and other investigators used genetic genealogy to solve the Golden State Killer case in 2018. In that case, investigators had used online services to identify a new suspect. Those services were uncomfortable with the intrusion, but some, such as GEDmatch, have since allowed users to consent to having their data included as part of a DNA analysis tool that can be used by law enforcement.

While the Justice Department policy prohibits use of DNA genetic databases that object to law enforcement access, the policy offers a broad footnote, saying that it “does not impose any legal limitations on otherwise lawful investigative” techniques.

In the Idaho case, according to the recent testimony, a preliminary report produced by Othram said the closest match it was working with shared 70.7 centimorgans of DNA with the crime scene sample. Mr. Kramer said that was a low match, typically representing two people who would perhaps share a great-great grandparent.

To enhance the family tree, Othram hoped to expand the DNA data available by approaching someone on the tree — one of four brothers — to see if they would be willing to contribute their DNA to help determine whether the team was on the right path. The identity of the brothers was not disclosed in the publicly available records, though their last name was not Kohberger.

Matthew Gamette, the director of forensic services for the Idaho State Police, testified that the brother was not interested in participating and had asked not to be contacted again. Othram was eventually asked to stop its work on Dec. 10, with the F.B.I. expected to take over the genealogy search.

In a memo, according to the testimony, the F.B.I. acknowledged turning to MyHeritage and a broader version of GEDmatch that includes people who have not opted in to law enforcement searches.

The two companies did not respond to messages seeking comment.

Leah Larkin, a genealogist working with Mr. Kohberger’s defense team, testified that investigators discovered a match of 250 centimorgans somewhere on the family tree, a level of comparison that offers much more potential to uncover a final match. Photos showed that investigators had built a family tree on a white board, Ms. Larkin said, with handwritten notes mapping out lines of relatives.

What they found appears to have led to Mr. Kohberger on Dec. 19. Days later, investigators went to his family home in Pennsylvania, where he was staying with his parents during the winter holidays, and collected trash from the home that better connected the crime scene DNA to him.

Mr. Kohberger’s defense team has challenged much of the state’s evidence and sought to undermine the DNA evidence by arguing that the authorities violated his constitutional rights by failing to obtain warrants before searching the DNA data. But a judge in the case has rejected those arguments as the case moves toward trial this summer.

Mr. Kramer, the former F.B.I. lawyer, said that the use of genetic genealogy could prevent other kinds of law enforcement intrusions by helping narrow the scope of an investigation. In the Kohberger case, he noted, a white Hyundai Elantra had been seen driving near the victims’ home, and that kind of evidence could lead detectives to examine the lives of many Elantra drivers who had done nothing wrong.

But Ms. Murphy, the law professor, noted that many people’s DNA could be shed at what might later become the scene of a crime, and that the widening tools had the potential to bring innocent people under extensive scrutiny or false charges, without clear rules.

She says there are growing calls for legislation and legal review to establish mandatory parameters for the use of genetic genealogy.

“If this is a method that we want to use as a society, we should be able to come up with rules we can agree on, then expect people to follow them,” she said.