Thousands of devotees had packed into the Puttingal Devi temple in the coastal district of Kollam. The fireworks display began at midnight and went on for hours.
The fire started when one of the crackers fell onto a shed where the fireworks were stored, said residents near the temple site, about 70 km from the state capital Thiruvananthapuram.
Prime Minister Narendra Modi flew to Kollam with a team of doctors to help state authorities cope with the large number of injured people, moving swiftly to pre-empt criticism over lack of public safety.
"The fire at the temple in Kollam is heart-rending and shocking beyond words," he said in a Twitter post. "My thoughts are with families of the deceased and prayers with the injured."
Modi has faced public criticism in the past for failing to respond quickly to disasters such as the floods in Chennai late last year. Large parts of the metropolis were under water for days before government help arrived.
Kerala's Chief Minister Oommen Chandy said 97 people were dead in the temple fire, many of whom were trapped by flames in the temple building. He said 245 were injured.
Television images showed people, some clutching children with burn injuries, being taken to hospitals.
The federal government also ordered the military to help the Kerala state authorities with two navy ships sailing to Kollam with medical supplies.
With Kerala in the midst of an election for a new state assembly, the temple fire quickly turned into a political issue as local leaders demanded an investigation into the fireworks display.
In Kerala, Modi's Bharatiya Janata Party trails its rivals the Congress party and the communists. The BJP has focused on building up grassroots strength in the state for decades.
Kerala's Chief Minister Chandy said the Kollam district administration had denied the temple authorities permission to carry out the fireworks display, and that the government will investigate how they went ahead.
"There was no permission to even store the fireworks," he said.
Police has registered a case against the temple authorities for negligence. The temple trust was not immediately available for comment.
Kerala is studded with temples, managed by rich and powerful trusts that often flout local regulations. Each year temples carry out fireworks displays, often competing to stage the most spectacular ones. There are judges who decide the winners.
Kollam district magistrate A. Shainamol said people living in the neighborhood of the temple had complained about the danger of these fireworks in the past.
The explosion from the fireworks was so strong that some parts of the temple roof caved in. Local media showed bulldozers trying to clear the area of the debris.
The Puttingal temple is one of the oldest in the state. It was built on the site of an ant hill where locals believe a goddess appeared centuries ago.