Garry Rodrigues tapped into an empty net in the 92nd minute after a mix-up in the Black Stars defence.
Jamiro Monteiro had put the islanders ahead andcthen had a goal ruled out for offside before half-time.
Alexander Djiku did equalise with a stooping header, but Ghana keeper Richard Ofori made two big saves before Rodrigues pounced to grab victory.
Cape Verde move two points clear in Group B after Egypt drew 2-2 with Mozambique earlier on Sunday.
Yet what may previously have been a shock defeat for four-time Afcon winners Ghana may not be viewed as a major upset given the mere 12 places separating the sides in the world rankings.
The West Africans suffered a group stage exit at the 2021 finals, while Cape Verde reached the last 16 before falling to eventual champions Senegal, and recent performances and results from Chris Hughton's Black Stars have been underwhelming.
Ofori had stuck out his left foot to deny the lively Jovane Cabral and also parried away a shot from Rodrigues before the substitute's goal sparked passionate scenes as the Blue Sharks players celebrated with supporters in the stands.
Cape Verde had taken a 17th-minute lead with the first real action of the game, as Ofori palmed away Cabral's well struck low effort and Monteiro was on hand to tuck the loose ball home.
Ghana thought they had levelled nine minutes before the break, but Majeed Ashimeru's stunning long-range strike was ruled out for offside following a lengthy Video Assistant Referee review.
After consulting his screen on the touchline referee Jean-Jacques Ndala Ngambo eventually concluded that Ransford Konigsdorffer, who had hit the post moments earlier, was blocking the eyeline of Blue Sharks keeper Vozinha inside the area.
Djiku levelled proceedings in the 56th minute, holding off his marker six yards out and guiding Jordan Ayew's corner inside the left-hand upright, but that goal proved to be Ghana's only effort on target during the match.
Ghana missed influential forward Mohammed Kudus, with the West Ham man absent with a hamstring niggle, and centre-back Djiku made a crucial acrobatic clearance to divert Bebe's effort over the bar with 18 minutes remaining.
There was a moment of history for Andre Ayew, introduced to captain the Black Stars in the 62nd minute, who became the third man to appear at eight Afcon finals - matching the record held by former Cameroon defender Rigobert Song and ex-Egypt midfielder Ahmed Hassan.
Having previously had spells at Swansea, West Ham and Nottingham Forest, the 34-year-old could yet set a new record of scoring at seven different editions of the tournament.
But, following Rodrigues' late heroics, Ghana face a difficult assignment against Egypt in their second Group B outing on Thursday (19:00 GMT) before Cape Verde take on Mozambique the following day.
BBC Sport Africa