[pystatsmodels] Why is StatsModels showing log likelihood function decreases as the fit gets better?

[kay bhide]

I was going through
https://www.statsmodels.org/stable/examples/notebooks/generated/ols.html

and as you see from the example (the output of sm.OLS) Log-Likelihood
increases as R-squared decreases.

Shouldn't Log-Likelihood and R-squared increase together?

Shouldn't Log-Likelihood increase as your model fits better to the data?

Or am I wrong in thinking Log-Likelihood should increase as the model fit
gets better.

[j***@gmail.com]

Post by kay bhide
I was going through
https://www.statsmodels.org/stable/examples/notebooks/generated/ols.html
and as you see from the example (the output of sm.OLS) Log-Likelihood
increases as R-squared decreases.
Shouldn't Log-Likelihood and R-squared increase together?
Shouldn't Log-Likelihood increase as your model fits better to the data?
Or am I wrong in thinking Log-Likelihood should increase as the model fit
gets better.
The maximized loglikelihood value is similar to residual sum of squares

(ssr), although with opposite sign.

Rsquare is a relative measure, that is scaled the represent the proportion
of the total sum of square explained by the model.
ssr and llf are not scaled and differ across datasets, e.g. larger datasets
have larger expected ssr and llf.
Consequently we cannot compare loglikelihood or ssr across datasets.

In the notebook most models use different generated datasets, AFAICS.

In contrast we can compare loglikelihood values for the same dataset across
models.
This is used in likelihood ratio test for comparing a full model with a
model nested in it.
But it is also used for non-nested models when we use, for example,
information criteria like AIC or BIC.
Models with larger likelihood fit better in those cases.

Another related problem: We cannot directly compare likelihood of a model
with a model that has transformed response variable.
For example we cannot compare OLS between OLS(y, x) and OLS(log(y, x)).
(However, we could compare to loglikelihood that is specific for the same y
= X b versus y = exp(X b), e.g. using GLM with different link functions)

(related aside: there is a likelihood ratio test for non-nested models
estimated on the same dataset which just measure or tests which model is
closer to the data. Vuong test)

Josef