Like the man who carries out the ashes of the red heifer, the man who drives the Yom Kippur goat into the wilderness is not necessarily a priest or Levite.
Leviticus 16:21

The act of driving the goat into the wilderness makes this man unclean. Like David driving the Philistines and other nations out of the land of Israel. It had to be done, and David was the man to do it, but it made him unfit to build a house for God because of the blood he had shed.
Leviticus 16:28