Abraham considered Ishmael his legitimate son and God agreed. He wasn't illegitimate; he just wasn't the son of Promise. He had to be separated from Isaac and Isaac's inheritance, just as we need to make a separation between clean and unclean in order to preserve the clean.
Genesis 21