Many physics analyses in Higgs, top and electroweak physics improve the kinematic reconstruction of the final state by constrained fits. This is a particularly powerful tool at $e^+e^-$ colliders, where the initial state four-momentum is known and can be employed to constrain the final state. A crucial ingredient to kinematic fitting is an accurate estimate of the measurement uncertainties, in particular for composing objects like jets. This contribution will show how the particle flow concept, which is a design-driver for most detectors proposed for future Higgs factories, can -- in addition to an excellent jet energy measurement -- provide detailed estimates of the covariance matrices for each individual particle-flow object and each individual jet. Combined with information about leptons and secondary vertices in the jets, the kinematic fit enables to correct $b$- and $c$-jets for missing momentum from neutrinos from semi-leptonic heavy quark decays. The impact on the reconstruction of invariant di-jet masses and the resulting improvement in $ZH$ vs $ZZ$ separation will be presented, using as an example the full simulation of the ILD detector concept. As an outlook, the expected benefit for the Higgs self-coupling measurement from double Higgs production will be discussed.