When selecting the object (dimension, ratings, etc.), click on the icon to apply a color and then check "Default". If it doesn't, it stays on the original color regardless of the layer.
For annotations, it's "Color" in the "Formatting" bar and for geometry it's "Line color" in the "Line format" bar (for both, it's possible to apply the default color):