In scripture, stones often represent people. Something borne on the shoulders is responsibility and authority. Remember the two onyx stones on the High Priest's shoulders, inscribed with the names of the tribes.

The stones that the men in Joshua 4 carried on their shoulders also represented the burden of their leadership over their respective tribes.