Background: In Japan, there is a high incidence of family members caring for their elderly. To facilitate this, caregivers often quit their jobs, work reduced hours, and forfeit leisure activities. Objectives: This study examined the relationship between the mental health of the caregivers and the sacrifices and adjustments they make to care for the elderly. Methods: A cross-sectional survey was conducted with responses from 171 caregivers. Referencing Pearlin’s stress process model, the relationship among five types of work change, four types of leisure activity quitting, caregivers’ subjective care burden, and depression were analyzed using t-test and multiple regression analysis. Methods: Caregivers who quit their work or other home activities had significantly more daily living care responsibilities than those who did not. Moreover, caregivers who gave up leisure activities had a greater sense of subjective care burden than those who did not. The experience of giving up peer activities and taking leave of absence from work was significantly associated with increased depressive symptoms. Being a part-timer or financially prosperous was associated with good mental health. Conclusions: To support family caregivers, it is essential to reduce the burden of long-term care and provide financial help and an environment where they interact with their peers, and their moods can be enhanced.