Not all are children of Abraham because they are his offspring, but “Through Isaac shall your offspring be named.”
Romans 9:7 ESV

Not all of Abraham's or Isaac's children inherited the Covenant. Although all of Jacob's children inherited the covenant, the precedent has been established and some of his more remote descendants can be cut off.

The true children of Abraham are by promise and divine intervention, but that doesn’t mean it doesn’t include his natural children. Isaac was the natural son of Abraham and Sarah, even if his conception was miraculous. Salvation and judgment both are to the Jew first, and only then to the gentile.