Objectives: This study aimed at revealing the caregiving challenges of the caregivers of people with dementia (PwD) during the COVID-19 pandemic when daycare service was stopped as an infection control measure, and discussed ways to help PwD and their family caregivers to maintain their well-being in the era of the pandemic. Methods: Between April and May 2020, a cross-sectional survey was conducted in 152 family caregivers of PwD who were clients of daycare service prior to the pandemic. The survey examined caregivers’ stress and challenges faced during daycare service cessation, their perceived needs for continuation of daycare service, and observed changes in functional status of PwD. Regression analyses were performed to explore the associated factors of caregiving stress and preference for continuation of daycare service. Results: Family caregivers of PwD experienced greater caregiving stress after cessation of daycare service. Infection was their main challenge in caregiving, and their physical and emotional health was adversely affected by the longer time commitment with PwD under the stay-home policy. Older age of caregivers, greater emotional and communication problems of PwD, and more time spent with PwD were associated with greater caregiving stress. More than one-third of the participants preferred the continuation of daycare service during the pandemic. Conclusions: Policy makers should consider the well-being of PwD and their caregivers when planning infection control measures. Daycare service with enhanced infection controlled measures should remain available to PwD during the COVID-19 pandemic.