The residual sum of squares (RSS) is a measure of how much variation in $y$ is left unexplained by the model. $RSS = \sum_{i=1}^n(y_i - \hat Y_i)^2$ where $\hat Y_i = \hat \beta_0 + \hat \beta_i X_i$. RSS can be written in matrix-vector notation as $RSS = ||\vec Y - X \vec \beta||^2 = (Y - X \beta)^T(Y - X \beta)$