Background: Physical activity (PA) has been positively associated with health-related quality of life (HRQoL) among cancer patients and family caregivers. However, there has been no relevant research for patient-caregiver dyads. Methods: Path analysis, based on the actor–partner interdependence model (APIM), was used to examine the relationship between physical activity and health-related quality of life and explore the mediating role of emotional distress in 233 dyads. Results: In both patients and caregivers, physical activity had a direct positive effect on physical quality of life (QoL) but not on mental. There was a significant indirect effect of physical activity on health-related quality of life via emotional distress for both dyad members. Patients’ and caregivers’ confidence in fighting cancer was negatively associated with their own emotional distress. Caregivers’ confidence in fighting cancer was positively associated with their physical activity and also negatively associated with patients’ emotional distress. Conclusions: Physical activity may be considered as a possible behavioral and rehabilitation strategy for improving health-related quality of life in patient-caregiver dyads and reducing negative symptoms. Future research and intervention may consider cancer patient-family caregiver dyad as a unit of care.