Algorithms and features in medical studies contain many “knobs” that govern the learning process from a high-level perspective: they are called hyperparameters, and investigators typically tune them by hand. In this case study, we present three mathematically grounded techniques to automatically optimize hyperparameters, and demonstrate their use in the problem of outcome prediction for ICU patients who suffer from sepsis.