American Keys, 29, was forced to stop with the score locked at 6-3 6-7 (6-8) 5-5 on Court One.
She appeared to pick up the injury when she hit a forehand while serving for victory at 5-2 in the final set.
Keys, 29, took a medical timeout at the change of ends when leading 5-4 and was treated before leaving the court.
The 12th seed reappeared with strapping high on her thigh and was clearly hampered by the injury.
Paolini, 28, broke again to level at 5-5 before Keys decided she could no longer continue.
“Right now I’m so sorry for her,” said Paolini. “To end the match like this is bad. What can I say?
“I think we played a really good match. It was tough, a lot of ups and downs. I’m feeling a bit happy but also sad for her. It’s not easy to win like that.”
French Open finalist Paolini had never won a match in the Wimbledon main draw before this year’s tournament.
After winning the first set, she trailed in the second 5-1 but rallied brilliantly to force a tie-break – which Keys eventually took on her fourth set point.
Keys established a big lead again in the decider before her injury dashed her dreams.
Elsewhere, Croatian Donna Vekic progressed to the last eight by overcoming Spaniard Paula Badosa in three sets.
The world number 37 won 6-2 1-6 6-4 in a rain-interrupted game on court two.
Vekic will play New Zealand’s Lulu Sun in the quarter-finals after her win over Briton Emma Raducanu.