This article is more than
7 year oldAid workers are among the dead, with the Red Cross saying six of its employees were killed.
The MSF aid agency said it was treating 120 injured people and appealed for help with medical evacuations.
President Muhammadu Buhari, whose army is fighting Boko Haram militants, expressed dismay and urged calm.
The attack took place near the border with Cameroon where the military is engaged in what it calls its final push against Boko Haram.
It is thought to be the first time Nigeria's military has admitted to making such a mistake.
An official from the Borno state government, who was helping to co-ordinate the evacuation of the injured by helicopter, told the Associated Press that 100 people had been killed.
MSF told the BBC that, although it could not confirm the figure, up to 95 people had died.
"This large-scale attack on vulnerable people who have already fled from extreme violence is shocking and unacceptable," said Dr Jean-Clement Cabrol, MSF director of operations.
"The safety of civilians must be respected. We are urgently calling on all parties to ensure the facilitation of medical evacuations by air or road for survivors who are in need of emergency care."
MSF said it had teams in Cameroon and Chad ready to treat wounded patients.
Many of the casualties, it said, were believed to be displaced people who had fled from areas where Boko Haram had carried out attacks.
"We are in close contact with our teams, who are in shock following the event," MSF spokesman Etienne l'Hermitte said.
Red Cross spokesman Jason Straziuso was quoted by AP as saying the agency's dead employees had been "part of a team that had brought in desperately needed food for over 25,000 displaced persons".
Gen Rabe Abubakar, a spokesman for the Nigerian military, said that some "remnants" of Boko Haram had been detected outside Rann and the military had acted to eliminate them.
After the military realised its mistake, they were "all in pain", he said.
"However, in a military operation such as this, from time to time these things do occur," he added. "Even though it was highly regretful, it was never intended."
Gen Lucky Irabor, who commands counter-insurgency operations in the north-east, said there would be an investigation.
A spokesman for the Nigerian president said the administration would offer help to the government of Borno state "in attending to this regrettable operational mistake".
Boko Haram has caused havoc in Africa's most populous country through a wave of bombings, assassinations and abductions.
It is fighting to overthrow the government and create an Islamic state.
The group has stepped up attacks in the past few weeks as the end of the rainy season enabled its fighters to move more easily in the bush.
Last month, the UN launched a $1bn (£825,000) appeal for those facing hunger and starvation in the region.
It said nearly 5.1 million people in three north-eastern states were expected to face serious food shortages as for a third year in a row farmers had been unable to plant, fearing unexploded improvised devices left behind by militants.
Urgent aid was needed for some 100,000 people, mostly children, at risk of dying of starvation.