Let \(r=|z|<\sqrt{3}-1\). Using triangle inequality, \[ |z^{2}+2z\cos\alpha|\le |z|^{2}+2|z||\cos\alpha|\le r^{2}+2r. \] Since \(r<\sqrt{3}-1\), \[ r^{2}+2r<(\sqrt{3}-1)^{2}+2(\sqrt{3}-1)= (3-2\sqrt{3}+1)+2\sqrt{3}-2=2. \] Hence, \[ \boxed{|z^{2}+2z\cos\alpha|<2}. \]