Epistasis shapes evolutionary outcomes during protein adaptation. In particular, when the effects of single mutations or mutational interactions are idiosyncratic, that is, unique to a genetic background, the predictability of protein evolution becomes greatly impaired. Here, we unveil a quantitative picture of the prevalence and role of idiosyncrasy in protein evolution by analysing 45 protein fitness landscapes, generated from seven enzymes. We found that mutational effects and epistasis are highly idiosyncratic across the landscapes. Idiosyncrasy obscured functional predictions of mutated proteins when using limited mutational data, and often continued to impair prediction upon incorporation of epistatic information. We show that idiosyncrasy stems from higher-order epistasis, and highlight examples where it permits, or restricts, evolutionary accessibility of certain genotypes. Our work suggests that idiosyncrasy deeply confounds predictions in protein evolution necessitating its incorporation into predictive models and in-depth exploration of its underlying molecular mechanisms.