Background: Outpatient medical follow-up post-stroke is not only crucial for secondary prevention but is also associated with a reduced risk of rehospitalization. However, being voluntary and non-urgent, it is potentially determined by both healthcare needs and the socio-demographic context of stroke survivor-caregiver dyads. Therefore, we aimed to examine the role of caregiver factors in outpatient medical follow-up (primary care (PC) and specialist outpatient care (SOC)) post-stroke. Method: Stroke survivors and caregivers from the Singapore Stroke Study, a prospective, yearlong, observational study, contributed to the study sample. Participants were interviewed 3-monthly for data collection. Counts of PC and SOC visits were extracted from the National Claims Database. Poisson modelling was used to explore the association of caregiver (and patient) factors with PC/SOC visits over 0–3 months (early) and 4–12 months (late) post-stroke. Results: For the current analysis, 256 stroke survivors and caregivers were included. While caregiver-reported memory problems of a stroke survivor (IRR: 0.954; 95% CI: 0.919, 0.990) and caregiver burden (IRR: 0.976; 95% CI: 0.959, 0.993) were significantly associated with lower early post-stroke PC visits, co-residing caregiver (IRR: 1.576; 95% CI: 1.040, 2.389) and negative care management strategies (IRR: 1.033; 95% CI: 1.005, 1.061) were significantly associated with higher late post-stroke SOC visits. Conclusion: We demonstrated that the association of caregiver factors with outpatient medical follow-up varied by the type of service (i.e., PC versus SOC) and temporally. Our results support family-centred care provision by family physicians viewing caregivers not only as facilitators of care in the community but also as active members of the care team and as clients requiring care and regular assessments.