Missing observations#
Decision. Any subset of a period’s observations may be missing (NaN). The filter uses the observed elements (DK §4.10). For a missing element of \(y_t\), the smoothed disturbance and its variance are the moments of that element of \(\varepsilon_t\) given the observed data,
which are not zero and \(H_t\) when \(H_t\) has correlations. The simulation smoothers draw missing elements from the same conditional distribution. \(F_t\) covers all elements, missing or not; \(F_t^{-1}\) covers the observed ones, zero-padded.
Why. EM (DK §7.3.4) and the analytic score (DK §7.3.3) take expectations over the disturbances given the data; with correlated H, the observed elements carry information about the missing ones, and the correct moments are needed for both to be right. Zero-padding \(F_t^{-1}\) keeps the full-size smoother formulas exact without selecting rows of Z at each step.
Alternatives. statsmodels reports 0 and \(H_t\) for missing elements and draws them unconditionally in its simulation smoother. The two agree on observed elements, which the tests compare.
Standardized residuals are NaN for missing elements and in the diffuse periods, where statsmodels reports 0, so that tests on the residuals skip them rather than count zeros.
Where. observed_inverse in statespace_filter; the smoother’s
disturbance step.