This study examines differences in the mental and physical health of the U.S. population during the early stages of the COVID-19 pandemic among 3 groups: noncaregivers, short-term caregivers (1 year or less), and long-term caregivers (greater than 1 year). Data from the Understanding America Study are used to describe group differences in reports of psychological distress and somatic symptoms. Logistic and negative binomial regression models are used to examine whether these differences persist after adjusting for demographic, socioeconomic, and prepandemic health conditions. To understand within-group differences in caregiving demands, the intensity of care provided by short-term and long-term caregivers, as well as selected patients’ health conditions are summarized. Adults’ mental and physical health varied substantially by caregiver status. Caregivers continued to fare worse than noncaregivers in terms of mental health and fatigue, and long-term caregivers were more likely to report headache, body aches, and abdominal discomfort than both short-term caregivers and noncaregivers, net of controls. The nature of caregiving differed between short-term and long-term caregivers, with the latter more likely to provide greater hours of care, and to be looking after patients with permanent medical conditions.Efforts to understand and mitigate the impact of the pandemic on population health should include caregivers, whose mental and physical health were already vulnerable before COVID-19.