A process that is carried out at the cellular level which results in the assembly, arrangement of constituent parts, or disassembly of polar granules, cytoplasmic, non-membranous RNA/protein complex aggregates in the primordial germ cells of many higher eukaryotes.