Background and objectives: Social factors have demonstrated to affect pain intensity and quality of life of pain patients, such as social support or the attitudes and responses of the main informal caregiver. Similarly, pain has negative consequences on the patient’s social environment. However, it is still rare to include social factors in pain research and treatment. This study compares patient and caregivers’ accuracy, as well as explores personality and health correlates of empathic accuracy in patients and caregivers. Materials and Methods: The study comprised 292 chronic pain patients from the Pain Clinic of the Vall d’Hebron Hospital in Spain (main age = 59.4 years; 66.8% females) and their main informal caregivers (main age = 53.5 years; 51.0% females; 68.5% couples). Results: Patients were relatively inaccurate at estimating the interference of pain on their counterparts (t = 2.16; p = 0.032), while informal caregivers estimated well the patient’s status (all differences p > 0.05). Empathic accuracy on patient and caregiver status did not differ across types of relationship (i.e., couple or other; all differences p > 0.05). Sex differences in estimation only occurred for disagreement in pain severity, with female caregivers showing higher overestimation (t = 2.18; p = 0.030). Patients’ health status and caregivers’ personality were significant correlates of empathic accuracy. Overall, estimation was poorer when patients presented higher physical functioning. Similarly, caregiver had more difficulties in estimating the patient’s pain interference as patient general and mental health increased (r = 0.16, p = 0.008, and r = 0.15, p = 0.009, respectively). Caregiver openness was linked to a more accurate estimation of a patient’s status (r = 0.20, p < 0.001), while caregiver agreeableness was related to a patient’s greater accuracy of their caregivers’ pain interference (r = 0.15, p = 0.009). Conclusions: Patients poorly estimate the impact of their illness compared to caregivers, regardless of their relationship. Some personality characteristics in the caregiver and health outcomes in the patient are associated with empathic inaccuracy, which should guide clinicians when selecting who requires more active training on empathy in pain settings.