Multiplexed assays of variant effects (MAVEs) make it possible to measure the functional impact of all possible single amino acid residue substitutions in a protein in a single experiment. Combination of variant effect data from several such experiments provides the opportunity to conduct large-scale analyses of variant effect scores measured across proteins, but can be complicated by variations in the phenotypes that are probed across experiments. Thus, using variant effect datasets obtained with similar MAVE techniques can help reveal general rules governing the effects of amino acid variation for a single molecular phenotype. In this work, we accordingly combined data from six individual variant abundance by massively parallel sequencing (VAMP-seq) experiments and analysed a total of 31,614 variant effect scores reporting solely on the impact of single amino acid residue substitutions on the cellular abundance of proteins. Using our combined variant effect dataset, we derived and analysed a collection of amino acid substitution matrices describing the average impact on cellular abundance of all residue substitution types in different structural environments. We found that the substitution matrices predict the cellular abundance of protein variants with surprisingly high accuracy when given structural information only in the form of whether a residue is buried or exposed. We thus propose our substitution matrix-based predictions as strong baselines for future abundance model development.