Link and Family: One Picks Your Effect Measure, the Other Your Standard Error

อ่านฉบับภาษาไทย (Thai version)

Abstract

With a binary outcome, choose the effect measure before the model: risks of 10% and 20% give a risk difference of 10%, a risk ratio of 2.0 and an odds ratio of 2.25. The link sets what a coefficient means: the logit link gives an odds ratio and the log link a risk ratio, whichever family is used. The family sets the variance: a binary outcome has variance p(1 − p), while the Poisson assumes p, so modified Poisson borrows the Poisson fit and replaces its standard error with a robust sandwich one. Log-binomial regression has the right variance but must keep every fitted risk, e raised to the linear predictor, at or below 1, so its optimizer searches against a wall and may fail to converge. Logistic regression never meets that wall, and modified Poisson removes it. This article follows each step with worked numbers, from the linear predictor to the optimizer's path along the wall.

🔍 Click to enlarge Click anywhere to close

Three questions hiding inside one word

A perinatal team is analysing neonatal death in a birth cohort. The outcome is 0 or 1: the baby died in the neonatal period or did not. The exposure is antenatal care (ANC), and the analysis plan lists logistic regression, log-binomial regression, modified Poisson regression and robust standard errors. It reads like four competing models, and it is hard to see why one study needs so many.

The confusion comes from the word model, which here mixes three separate questions:

1. Which effect do I want to report?
2. Which link do I use?
3. Which variance do I assume?

Separate these three, and everything becomes much simpler.

Choose the effect measure first

The deepest point is that there are several models because they answer different effect measures, or use different ways of computing one. Suppose the outcome is coded 0 or 1, for example neonatal death:

• unexposed group: risk = 10%;
• exposed group: risk = 20%.

From the same data we can answer in several ways.

Same data, three answers

Logistic regression gives an odds ratio

Logistic regression models the logit, the logarithm of the odds:

Here $p$ is the risk of the outcome, $X$ the exposure (1 if exposed, 0 if not), and $\beta_0$ and $\beta_1$ the coefficients the regression estimates. Therefore

This is the one and only reason logistic regression gives an odds ratio. It is not wrong; it answers a different question.

Suppose a logistic model gives $OR = 3.9$. Do not write that the risk is 3.9 times higher. It means that the odds are 3.9 times higher. Especially when the outcome is common, the odds ratio can sit far from the risk ratio.

Why both log-binomial and modified Poisson?

Because both use the same log link.

A log-binomial model assumes the outcome is binomial and uses the log link [1]:

Here $X\beta$ is short for the whole right-hand side, $\beta_0 + \beta_1 X$ plus any other terms.

Modified Poisson regression uses a Poisson working model, a distribution borrowed for the fitting rather than believed [2]:

but it still uses $\log(p) = X\beta$, exactly as before. So in both models the coefficient is read the same way:

This is the point that makes people ask: if both give a risk ratio, why build two models? The answer:

Log-binomial is built to fit a binary outcome directly, but it is harder to compute.

Modified Poisson accepts a variance model that does not match a binary outcome so that fitting becomes easier, then uses a robust standard error to repair the variance.

Separate the link from the family

This is the root of the whole topic. Think of a model as two pieces:

The link function connects the risk to the right-hand side of the equation, and it decides what the coefficient means [3, 4]:

• logit link, $\log\left(\frac{p}{1-p}\right) = X\beta$, so the coefficient gives an odds ratio;
• log link, $\log(p) = X\beta$, so the coefficient gives a risk ratio.

The family is the distribution assumed for the outcome. Its job here is to say how the variance is thought of.

What the robust standard error is for

First separate it from the coefficient. The estimate $\hat\beta$ and its standard error $SE(\hat\beta)$ are two different things.

• The coefficient tells the effect, for example $e^{\beta_1} = 1.89$.
• The standard error tells the uncertainty around the coefficient. It is what builds the confidence interval and the P value.

In modified Poisson the Poisson variance is wrong for a binary outcome, so the ordinary standard error may be wrong too. The model therefore uses a

so that inference stays valid even though the assumed variance function is wrong. A robust (sandwich) standard error estimates the variance from the observed residuals rather than from the assumed variance [2]. What it can and cannot repair is the subject of part 4 of this series.

Why log-binomial needs no modifying

A binary outcome really is binomial, so the binomial variance, $\operatorname{Var}(Y) = p(1-p)$, is already right. Conceptually the model is very natural:

The problem is the constraint $0 \le p \le 1$. Log-binomial sets

and if $X\beta > 0$, then $e^{X\beta} > 1$, a probability that cannot exist.

The fitting algorithm, the optimizer that searches for the coefficients, must therefore look for the answer inside a narrow space where every observation has $p_i \le 1$. In some datasets this ends in a convergence failure: the algorithm never settles on stable estimates [5].

Why modified Poisson avoids this

A Poisson model does not force the fitted mean to stay at or below 1, so it accepts, for example,

Read as a real probability, that is impossible. But the optimization becomes much easier. This is the trade-off [6].

Log-binomial versus modified Poisson

Why adjusted models may not match exactly

Add covariates:

where ANC is antenatal care and SES socioeconomic status. Now each person $i$ has a risk $p_i$ of their own. The binomial and the Poisson families use different variances, and so different weights, when they estimate $\beta$. The two models may then give

a small difference.

A large gap is not automatically interaction

A common rule says: if the two adjusted methods differ clearly, suspect interaction. That is too strong. Interaction is one possible cause, not the only one. When log-binomial and modified Poisson differ clearly, the cause may be any of several kinds of model misspecification:

• an interaction that was not included;
• a continuous covariate that is really nonlinear but was entered as linear;
• a wrong functional form;
• omitted covariates;
• a model that forces one common risk ratio when the risk ratio really differs across covariates;
• extreme fitted risks.

One more caution: modified Poisson generally converges very easily, but it should not be said to converge 100% of the time.

Everything in one tree

With a binary outcome, start with one question: what do I want to report?

                   Binary outcome
                         │
             What effect do I want?
                         │
       ┌─────────────────┼─────────────────┐
       │                 │                 │
      OR                RR                RD
       │                 │                 │
   logit link         log link       identity link
       │                 │
   Logistic        ┌──────┴──────┐
 regression        │             │
                   │             │
             Log-binomial   Modified Poisson
                   │             │
             Binomial       Poisson working
              family           family
                   │             │
              RR = exp(b)   RR = exp(b)
                                 │
                            Robust SE

So there are not many models to memorise. There are only two questions:

1. Which effect do I want: OR, RR or RD?
2. If I want RR, how will I estimate it: log-binomial, or modified Poisson with a robust standard error?

A real Poisson model has μ, not p

Writing $\log(p) = X\beta$ for modified Poisson looks odd, because an ordinary Poisson regression has no $p$. A real Poisson regression looks like this:

with the log link

Here $\mu_i$ is the expected count, the mean count, not a probability. For example, take the number of hospital admissions in one year, $Y_i = 0, 1, 2, 3, \ldots$, and the model

Then

which is a count ratio or a rate ratio, depending on the setting.

Three models side by side

The most important picture

Put the mean model at the top, and the three models fall into place:

                         Mean model
                            │
                       log(mean)
                            │
              ┌─────────────┴─────────────┐
              │                           │
        Count outcome                 Binary outcome
              │                           │
       E(Y) = μ count                E(Y) = p
              │                           │
        log(μ) = Xβ                  log(p) = Xβ
              │                           │
       Poisson regression        ┌────────┴────────┐
                                 │                 │
                          Log-binomial      Modified Poisson
                                 │                 │
                         Binomial variance   Poisson variance
                           p(1-p)                 p
                                                   │
                                             wrong variance
                                                   │
                                              robust SE

The most accurate sentence is this one:

A real Poisson regression is written $\log(\mu) = X\beta$. In modified Poisson, a binary $Y$ has $E(Y) = p$, so $\mu$ equals $p$ in the mean model and the equation can be written $\log(p) = X\beta$, even though the Poisson variance assumption is wrong.

And this is why modified Poisson gives a risk ratio although it is called Poisson: we use the Poisson mean structure but do not believe the Poisson variance, so we use a robust standard error instead.

A probability above 1 never comes from the data

In reality a probability can never exceed 1. A value of $p > 1$ does not come from the real data. It comes from a temporary value the model tries while it searches for $\beta$. So keep the real data and the estimating equation apart.

The real data are fine. In the unexposed group, 20/100 people have the outcome, so $p_0 = 0.20$. In the exposed group, 40/100 do, so $p_1 = 0.40$. Everything lies between 0 and 1, as it should.

Why the log link allows such numbers

Write $\eta = X\beta$ for the whole right-hand side of the equation, the linear predictor (the end of this article looks at it closely). Under the log link

and its range is

Look at a few values:

But $\eta = 0.1$ gives $p = 1.105$, and $\eta = 1$ gives $p = 2.718$.

The exponential function itself does not know that we are using it for a probability. All it knows is that $e^x > 0$. To make it a probability we have to add a constraint:

because if $X_i\beta \le 0$, then $e^{X_i\beta} \le 1$.

This is why logistic regression fits more easily

Look at the inverse of the logit link:

Whatever $X\beta$ is, the result is a valid probability. If $X\beta = -10$, then $p \approx 0$. If $X\beta = 0$, then $p = 0.5$. If $X\beta = 10$, then $p \approx 1$, but it never goes above 1:

This is a huge advantage of the logistic link.

The log link and the logit link, value by value

Compare the two. Under log-binomial, $p = e^{\eta}$:

η = -3   → p = 0.05   valid
η = -1   → p = 0.37   valid
η =  0   → p = 1.00   valid (the edge)
η =  0.2 → p = 1.22   invalid
η =  1   → p = 2.72   invalid

so it has to force $\eta \le 0$. Under logistic, $p = e^{\eta}/(1 + e^{\eta})$:

η = -3   → p = 0.047
η = -1   → p = 0.269
η =  0   → p = 0.500
η =  1   → p = 0.731
η =  3   → p = 0.953
η = 100  → p ≈ 1

All of these are valid. This is the deep difference between the log link and the logit link.

How a log-binomial fit has to search

The real parameter space of a log-binomial model is

and the computer must find $\beta$ only inside this space.

The trouble grows with the number of predictors. With

the model needs $\beta_0 + \beta_1 X_{i1} + \beta_2 X_{i2} + \cdots \le 0$ for every observation. One person may be old, exposed and severely ill, so that the linear predictor sits very close to zero:

The optimizer wants to move a coefficient a little further to fit everyone else better. But moving to $X_i\beta = +0.001$ would give $p_i = 1.001$, which is forbidden. The optimizer is walking right along the wall:

                INVALID
                 p > 1
                   │
                   │  ← the wall, p = 1
───────────────────┤
                 ● │ ← the MLE may sit very close to here
              ↗    │
          optimizer│
        ↗          │
   valid region    │
     p < 1         │

The MLE, the maximum likelihood estimate, is the set of coefficients the fit is looking for.

Where modified Poisson fixes it

Modified Poisson says: I will not enforce the ceiling $p \le 1$ during the optimization. It uses

and the Poisson allows $\mu = 0.2,\ 1,\ 2,\ 10,\ \ldots$, because for a count a mean above 1 is not strange at all. The optimizer runs more freely, so it converges more easily.

Used with a binary $Y$, $E(Y) = p$, so the coefficient estimates the risk ratio. But because we are deliberately using the Poisson on a binary outcome, the variance is wrong, and a

is required.

The Poisson stretches the search space, not the probability

The picture to see is that the Poisson does not stretch probability. It stretches the space in which the optimizer is allowed to search for coefficients.

Let $\eta = X\beta$. Log-binomial and Poisson both use $\mu = e^{\eta}$, the exponential curve $y = e^x$, and the line $y = 1$ is the ceiling of a probability. Log-binomial reads $y$ as $p$, so it can use only the left side, $\eta \le 0$:

but $\eta = 0.2 \Rightarrow p = 1.22$ is no longer allowed. So log-binomial has only this much room to move:

η

<-----------------------------|----------------------->
                              0

      VALID                   |        INVALID
      p < 1                   |         p > 1

  -3    -2    -1    -0.5      0       +0.5      +1
   ↓     ↓     ↓      ↓       ↓         ↓         ↓
 0.05  0.14  0.37   0.61    1.00      1.65      2.72

                              ↑
                           the wall

-0.5 → -0.3 → -0.1 → +0.1

What the Poisson does differently

The Poisson does not say that $\mu$ is a probability. It says $\mu$ is a mean count. So $\mu = e^{\eta}$ may be 0.5, 1, 2, 10 or 100, and there is no ceiling at 1. The optimizer's space becomes:

η

<------------------------------------------------------>

 -3      -2      -1       0       +1       +2
  ↓       ↓       ↓       ↓        ↓        ↓
 0.05    0.14    0.37     1       2.72     7.39

        all usable for the Poisson

If the optimizer walks $-0.5 \to -0.3 \to -0.1 \to +0.1 \to +0.2$ and then back to $-0.05$, the Poisson says: walk on, I can still compute the likelihood and the estimating equation at every point. It can pass through the region that, read as a probability, would exceed 1 on its way to the best coefficients.

That is what “fits more easily” means. It does not mean that $p > 1$ has become real. It means:

Real data have many people, so the problem grows

Suppose there are 4 people, with

Log-binomial needs $\eta_i \le 0$ for every one of them. The optimizer wants to raise $\beta_1$ by 0.1 to fit people 1 to 3 better.

One person can block the whole step

Modified Poisson takes the wall away

For log-binomial, the allowed coefficients form a closed region, and the best point may sit right at its edge:

        feasible coefficients
              ______
             /      \
            /        \
           /     ●    | ← the optimum sits at the edge
          /          /
         /__________/

moving outside = probability > 1 = forbidden

For modified Poisson there is no such boundary:

no boundary p ≤ 1

──────────────────────────────────
              ↗
         ↗  ●
    ↗
the optimizer moves freely in every direction
──────────────────────────────────

So numerical algorithms such as Newton-Raphson or iteratively reweighted least squares (IRLS) do not have to keep asking whether a step will push someone's predicted risk above 1, and they converge much more easily.

It can still be used for a binary outcome because, for $Y = 0, 1$, $E(Y) = p$. Modified Poisson uses the mean equation $\log E(Y \mid X) = X\beta$, which is $\log p = X\beta$, so $e^{\beta_1} = RR$. What it borrows from the Poisson is a way of estimating the mean that has no boundary. The Poisson variance, $\operatorname{Var}(Y) = \mu$, is wrong for a binary outcome, whose real variance is $\operatorname{Var}(Y) = p(1-p)$, so it is corrected with a robust standard error.

What η is: a name for the right-hand side

$\eta$ (the Greek letter eta) is not a new variable from the data. It is only a short name for what sits on the right-hand side of the regression equation. Take

To avoid writing it out every time, we name it

so the equation shortens to $\log(p) = \eta$, which solves back to $p = e^{\eta}$. $\eta$ is called the linear predictor.

η can be any number; the link converts it

With several variables it works the same way, for example

$\eta$ is the total score the regression computes for each person, before it is turned back into a probability or risk.

The key points: $\eta$ is not a probability; it may be negative, positive or any number at all; and it is the link function that converts $\eta$ into the quantity we want:

• logistic: $\eta = \log\dfrac{p}{1-p}$;
• log-binomial: $\eta = \log(p)$;
• Poisson: $\eta = \log(\mu)$.

So $\eta$ really is just

shorthand for the intercept plus each coefficient times its predictor.

Related in the wiki: [[risk-regression-models-epidemiology]] [[glm-families-effect-measures]]