Non-invasive biomarkers may predict adverse events such as acute rejection after kidney transplantation. It is uncertain how these biomarkers, often derived from a single study, perform across different cohorts of kidney transplant recipients. This study aimed to develop a cross-validation framework that evaluates the performance of biomarkers associated with acute rejection, and an integrated gene signature set for defining acute rejection in kidney transplant recipients. We included publicly available datasets that reported acute rejection episodes after kidney transplantation. We tested the predictive probability for acute rejection using a unique set of gene signatures within the individual dataset. Validation of this set of gene signatures in other datasets was conducted using a cross-validation framework. We identified eight studies (1454 participants), published between 2010 and 2019. Of these, seven out of eight sets of gene signatures had good predictive probabilities (80-95%) for acute rejection within their own cohorts, but the predictability reduced to less than 50% when tested in other independent datasets. By integrating biomarkers gene signatures sets with high specificity scores across all studies, a set of 150 genes (included CXCL6, CXCL11, OLFM4 and PSG9) which are known to be associated with immune responses, had good test accuracy (varied between 70-90%) for acute rejection. A biomarker set of gene signatures for acute rejection determined within a specific cohort of kidney transplant recipients may not provide highly accurate prediction in an independent cohort of transplant recipients. However, integration of biomarkers gene signatures sets with high specificity scores may improve the prediction performance of these markers.