Purpose: We aimed to assess the influence of anxiety and depression on the physical and mental quality of life (QoL) in patient with chronic obstructive pulmonary disease (COPD) and caregiver dyads, detect the simultaneous effect of anxiety and depression of each partner on the other’s QoL and determine the dyadic patterns. Methods: A cross-sectional descriptive design was used. The actor–partner interdependence model estimated by structural equation modeling was used for the dyadic analysis. Patient Health Questionnaire-9 (PHQ-9), Generalized Anxiety Disorder-7 (GAD-7) and 12-Item Short-Form Health Survey (SF-12) were used to measure depression, anxiety and QoL, respectively. Results: Eighty COPD dyads were enrolled in the study. Patients presented higher depression symptoms and poorer physical and mental QoL than their caregivers, whereas comparable levels of anxiety were found in patients and caregivers. The model exploring the effects of depression and anxiety on mental QoL found that patients’ depressive symptoms negatively influence their mental QoL, and caregivers’ anxiety and depression symptoms negatively impact their mental QoL. The model exploring the effects of anxiety and depression on physical QoL detected one statistically significant actor effect with patients’ depressive symptoms negatively influencing their physical QoL, and two partner effects with caregivers’ anxiety worsening patients’ physical QoL and caregivers’ depression improving patients’ physical QoL. Conclusions: The results suggest that caregivers’ psychological distress influences caregivers’ mental QoL and patients’ physical QoL. Therefore, health-care professionals should assess and treat anxiety and depression in both members of the COPD dyad to improve their QoL.