Cardiac surgery (CSX) can be lifesaving in elderly patients (age ≥ 80 years) but may still be associated with complications and functional decline. Frailty represents a determinant to outcomes in critically ill patients, but little is known about its influence on elderly CSX-patients. This is a secondary exploratory analysis of a multi-center, prospective observational cohort study of 610 elderly patients admitted to the ICU and followed for one year to document long-term outcomes. CSX-ICU-patients (n = 49) were compared to surgical ICU patients (n = 184) with regard to demographics, frailty, and outcomes. Of all surgical patients, 102 (43%) were considered vulnerable or frail. The subdistribution hazard ratio (SHR) of time to discharge home (TTDH) for vulnerable/frail vs. fit/well patients was 0.54 (95% confidence interval (CI), 0.34, 0.86, p = 0.007). The p-value for effect modification between surgery group (CSX vs. surgical ICU patients) and Clinical Frailty Scale (CFS) group was not significant (p = 0.37) suggesting that the observed difference in the CFS effect between the CSX and surgical ICU patients is consistent with random error. A further subgroup analysis shows that among surgical ICU patients, the SHR of time to discharge home (TTDH) for vulnerable/frail vs. fit/well patients was 0.49 (95% CI, 0.29, 0.83) while the corresponding SHR for CSX patients was 0.77 (0.32–1.88). In conclusion, preoperative frailty reduced the rate of discharge to home in both surgical and CSX patients, but a larger sample of CSX patients is needed to adequately address this question in this patient group.