Objectives: Patients’ irritability and aggression have been linked to caregiver depression, but the behaviors that most burden caregivers are not yet definitively identified. This study examines the connection between behavioral and psychological symptoms of dementia (BPSD) and the burnout of caregivers caring for home-dwelling elders with dementia symptoms in Japan. Methods: 80 Japanese rural and urban family caregivers completed detailed questionnaires about their experiences in caring for demented family members. We statistically analyzed the results for correlations between types of dementia, Pines Burnout, and Caregiver Distress. Results: BPSD symptom severity significantly correlated with caregiver distress. The dementia symptoms most strongly correlated with caregiver burnout were: aggression, irritability, abnormal motor behavior, and hallucinations. Conclusions: Among the commonest symptoms, apathy, anxiety, and depression did not seriously aggravate caregiver burnout. Caregivers displayed higher burnout facing agitation/aggression, irritability, aberrant motor behavior, and hallucinations. Caregivers’ reported distress was surprisingly dissimilar to their burnout scores; patients’ delusions and anxiety led to higher distress reporting but not to burnout. Clinical Implications: Advance diagnosis of BPSD symptoms should be helpful to support nurses and caregivers of dementia patients. Particular support should be considered for caregivers and nurses of patients expressing aggression, irritability, abnormal motor behavior, and hallucination.