While larger tumors are often associated with a worse prognosis, other factors like the type of cancer, its location, and whether it has spread to other parts of the body (metastasized) also play significant roles. For example, a small but highly aggressive tumor can be more dangerous than a larger, slow-growing one.