Barney is proud of the attention he gives to his official appearance. When on duty, he's always dressed in his full uniform - including cap, tie and firearm. And well polished shoes (including the backs).
Where Barney carries his bullet is another question. It's supposed to be in his shirt pocket (usually the left one), but every once in a while he'll sneak it into his gun, where it never seems to stay very long.