Background: Over-treatment of estrogen receptor positive (ER+), lymph node-negative (LNN) breast cancer patients with chemotherapy is a pressing clinical problem that can be addressed by improving techniques to predict tumor metastatic potential. Here we demonstrate that analysis of second harmonic generation (SHG) emission direction in primary tumor biopsies can provide prognostic information about the metastatic outcome of ER+, LNN breast cancer, as well as stage 1 colorectal adenocarcinoma. Methods: SHG is an optical signal produced by fibrillar collagen. The ratio of the forward-to-backward emitted SHG signals (F/B) is sensitive to changes in structure of individual collagen fibers. F/B from excised primary tumor tissue was measured in a retrospective study of LNN breast cancer patients who had received no adjuvant systemic therapy and related to metastasis-free survival (MFS) and overall survival (OS) rates. In addition, F/B was studied for its association with the length of progression-free survival (PFS) in a subgroup of ER+ patients who received tamoxifen as first-line treatment for recurrent disease, and for its relation with OS in stage I colorectal and stage 1 lung adenocarcinoma patients. Results: In 125 ER+, but not in 96 ER-negative (ER-), LNN breast cancer patients an increased F/B was significantly associated with a favorable MFS and OS (log rank trend for MFS: p = 0.004 and for OS: p = 0.03). On the other hand, an increased F/B was associated with shorter PFS in 60 ER+ recurrent breast cancer patients treated with tamoxifen (log rank trend p = 0.02). In stage I colorectal adenocarcinoma, an increased F/B was significantly related to poor OS (log rank trend p = 0.03), however this relationship was not statistically significant in stage I lung adenocarcinoma. Conclusion: Within ER+, LNN breast cancer specimens the F/B can stratify patients based upon their potential for tumor aggressiveness. This offers a "matrix-focused" method to predict metastatic outcome that is complementary to genomic "cell-focused" methods. In combination, this and other methods may contribute to improved metastatic prediction, and hence may help to reduce patient over-treatment.