Background: As the population is ageing, the need for informal caregivers increases, and thus we need to know more about the effects on caregivers. This study aims to determine both cross-sectional and longitudinal associations between perceived limitation of informal caregiving and mental health of caregivers. Methods: This population-based cohort study was based on the Swedish Psykisk hälsa, Arbete och RelaTioner (PART) study, and 9346 individuals aged 18–65 were included. Data were collected through questionnaires, interviews and Swedish registers. Informal care was defined as care given to a family member. Self-reported and diagnosed depression and anxiety were included as outcomes. Covariates included sex, age, social support and socio-economic position. Ordinal logistic regression and Cox regression were performed to determine the associations between caregiving and anxiety or depression. Results: Self-reported depression and anxiety was only increased among those experiencing limitations (adjusted odds ratios [aOR] 2.00, 95% confidence intervals [CI] 1.63–2.47 for depression; aOR 2.07, 95% CI 1.57–2.74 for anxiety) compared to those not giving care, respectively. The adjusted hazard ratio (aHR) were increased for diagnosed depression (aHR 1.97, 95% CI 1.27–3.05) and for diagnosed anxiety (aHR 1.86, 95% CI 1.06–3.25) among those giving care and experiencing limitations, compared to those not giving care. No significant associations were found in caregivers without limitations. Conclusion: Caregivers experiencing limitations showed a significant association with short- and long-term anxiety and depression. This study implies the importance of exploring the degree to which informal caregiving can be provided without adding burden to caregivers.