Breast Cancer is the most frequently diagnosed malignancy and the leading cause of cancer death among women worldwide, the main strategy to reduce mortality is early detection and treatment. Histopathology biopsy imaging is currently the clinically practiced gold standard for diagnosing breast cancer since it captures the effect of the disease on the tissues with a comprehensive view. Pathologists examine the images at various magnification factors to identify the type of the tumor, because if just one magnification is taken into account, the decision may not be accurate, especially if there are variances in the patient’s score at each magnification level. Thus, machine learning models must process the image at various magnification factors for an accurate classification of breast cancer tumors. This work explores the performance of transfer learning and late fusion to construct a multi-modality ensemble that fuses different magnification-specific deep learning models in order to predict the class of the breast cancer histopathological slides.