Instead of doing a matter removal, you do a sweep by creating a body that is not merged with the 1st that you then subtract with the Combine function. Sometimes this method is faster.
Your idea is good but it seems to me that the problem could rather be at the level of the perfect perpendicularity of the groove with respect to the central axis of the cam. It's almost the same problem as the laser cutting on tubes that @ac cobra explained to us a few months ago.
A possible track with 3 sketches: a trajectory (pink spline) and a guide curve (blue spline) to constrain the orientation of the section (orange sketch).
And test the options of the Scan Material Removal function... In this case, set the profile twist to "Follow the path and the first guide curve". Result: