Background: Depression among family caregivers is becoming an increasingly important issue due to a growing elderly population. This study aimed to examine the association of living with a patient with dementia and family caregivers’ depressive symptoms, among Korean adults. Methods: This study used the data of 371,287 participants after excluding those who indicated having dementia themselves from the Korea Community Health Survey of 2018–2019. Depressive symptoms were measured using the Patient Health Questionnaire-9. Data were analyzed using multiple logistic regression. Results: The rates of spouse caregivers having depressive symptoms were 9.4% and 10.8% among men and women, respectively. The odds ratio for risk of depressive symptoms among male and female spouse caregivers in comparison to non-caregivers was 2.65 and 2.28, respectively. In the subgroup analysis, the highest income group was associated with risk of depressive symptoms, with an odds ratio of 4.28 for men, and 3.02 for women. Conclusion: Having a patient with dementia in the family was significantly associated with family caregivers’ depressive symptoms. In particular, when the patient with dementia was a spouse, both women and men were likely to have depressive symptoms. To reduce the burden of caregivers, we need management policies and interventions for family caregivers.