Background: Heart failure (HF) causes high rates of hospital admissions. It is known that disease progression impacts the health-related quality of life (HRQoL) of both patients and caregivers, yet to date, this finding is based on cross-sectional studies with limited samples. Objectives: The study aim is to analyze the relationship between HF patients’ use of hospital services (a proxy for disease progression) and the HRQoL of their family caregivers. Methods: This work is a multicenter nested case-control study on a population of patients admitted to hospitals in southern Spain due to heart failure. The sample comprised 530 patient-caregiver dyads. Hospital admission data were retrospectively collected for the 5 years prior to inclusion in the study. Bivariate analyses and multivariate logistic regression were used to determine associations between patient deterioration and caregivers’ quality of life. Results: Patients’ use of hospital services was associated with worsened quality of life for family caregivers, with an overall OR of 1.48 (95% CI: 1.23-1.79). A positive correlation was found between patients’ perceptions of their physical health and the perceived mental health of caregivers (r = 0.127, p = 0.004) and between the perceived mental health of both (r = 0.291; p <0.0001). Conclusions: Greater use of hospital services by patients with HF is an independent predictor of deterioration of family caregivers’ HRQoL. The physical and mental components of patients’ and their family caregivers’ HRQoL interact and influence each other. Additional factors, such as the nature and intensity of care provided, also determine the worsening of a family caregiver’s HRQoL. Clinical Relevance: These results can be used to identify family caregivers of people with heart failure at risk of suffering a deterioration in their health-related quality of life. Increased use of hospital services is an independent predictor of the deterioration of the family caregivers’ health-related quality of life. Since clinical nurses are the main provider who gives support and education to family caregivers, they should be alert to this situation and individualize interventions to prevent this deterioration.