Background: This study is part of a broader phenomenological study on the experiences of family caregivers and their care recipients. There is a general paucity of research on the experiences of primary and secondary caregivers, and the negative impact of elderly care on caregivers in the urban poor settings in Ghana. This study explored primary and secondary caregivers’ challenges and coping strategies in the urban poor context in Accra, Ghana. Methods: This study was conducted in Ga Mashie. Thirty-one caregivers were interviewed. A phenomenological analysis was conducted using NVivo 10. Findings: Primary and secondary caregivers experienced economic, physical, social, and psychological burdens. Also, caregivers used spirituality and perseverance to cope with their challenges. Conclusions: The findings demonstrate that caregivers’ challenges varied by type of caregiver. Researchers and policymakers should consider the type of caregiver when designing interventions to mitigate the negative impacts of family caregiving on caregivers.