Background: Informal caregivers may experience a significant burden while caring for cancer patients. Little is known about how caregiver burden varies across different palliative cancer care settings and the factors influencing it. Objectives: We compared the severity of caregiver subjective stress burden (emotional impact) among caregivers of patients seen in the outpatient supportive care center (SCC) with those being cared for in the acute palliative care unit (PCU). Secondary aims were to compare other caregiver burden dimensions, quality of life, and any association of caregiver subjective stress burden to various patient and caregiver factors. Methods: Eligible patients and their informal caregivers in the SCC or PCU at a comprehensive cancer center in the USA were approached and enrolled. The Montgomery-Borgatta Caregiver Burden Scale and the Short-form 36 were used to measure burden and quality of life. Multivariate general linear regression was employed to evaluate the effect of covariates on subjective stress burden. Results: Ninety-eight dyads in the SCC and 74 dyads in the PCU were enrolled. PCU caregivers reported worse subjective stress burden (p = 0.0029) and mental health (p = 0.0299). Multivariate analysis showed correlations between subjective stress burden and caregivers' objective burden (p = 0.0136), subjective demand burden (p ≤ 0.0001), mental health (p = 0.0074), duration of caregiving (p = 0.0680), education (p = 0.0192) and with patients' anxiety (p = 0.0003) and current/recent cancer treatment (p = 0.0579). Conclusion: PCU caregivers demonstrated worse emotional burden and mental health than those in the SCC. More research is needed to tailor interventions for various caregiver burden dimensions. NCI Clinical Trial Registration Number ID: NCI-2019-01197