One day recently, on a table in Jean Oh’s lab in the Squirrel Hill neighborhood of Pittsburgh, a robot arm was busy at a canvas. Slowly, as if the air were viscous, it dipped a brush into a pool of light gray paint on a palette, swung around and stroked the canvas, leaving an inch-long mark amid a cluster of other brushstrokes. Then it pulled back and paused, as if to assess its work.
The strokes, mostly different shades of gray, suggested something abstract — an anthill, maybe. Dr. Oh, the head of the roBot Intelligence Group at Carnegie Mellon University, dressed in a sweatshirt bearing the words “There Are Artists Among Us,” looked on with approval. Her doctoral student, Peter Schaldenbrand, stood alongside.
Dr. Oh’s work, which includes robot vision and topics in autonomous aviation, often touches on what is known as the sim-to-real gap: how machines trained in a simulated environment can act in the real world. In recent years, Mr. Schaldenbrand has led an effort to bridge the sim-to-real gap between sophisticated image-generation programs like Stable Diffusion and physical works of art like drawings and paintings. This has mainly been manifest in the project known as FRIDA, the latest iteration of which was rhythmically whirring away in a corner of the lab. (FRIDA is an acronym for Framework and Robotics Initiative for Developing Arts, although the researchers chose the acronym, inspired by Frida Kahlo, before deciding what it stood for.)
The process of moving from language prompts to pixelated images to brushstrokes can be complicated, as the robot must account for “the noise of the real world,” Dr. Oh said. But she, Mr. Schaldenbrand and Jim McCann, a roboticist at Carnegie Mellon who also helped develop FRIDA, believe that the research is worth pursuing for two reasons: It could improve the interface between humans and machines, and it could, through art, help connect people to one another.