But Abram said to Sarai, “Behold, your servant is in your power; do to her as you please.” Then Sarai dealt harshly with her, and she fled from her.
Genesis 16:6

In God's economy, the slave-master relationship takes precedence over wife-husband. (Why? I don't know, exactly, but see Exodus 21:1-6.) This is why a handmaid's children belonged to her mistress as much as they belonged to her, and why Hagar would still be under Sarai's authority despite also being Abram's concubine.
#lekhlekha