...they did not obey your voice or walk in your law. They did nothing of all you commanded them to do. Therefore you have made all this disaster come upon them.
Jeremiah 32:23

We aren't kicked out of God's house for every incidental sin or moment of weakness (not even for habitual weakness) but we still suffer the consequences.