While I've stripped 124 starter motors completely for inspection, I don't think I've ever removed only the solenoid with the remainder of the starter intact.
However, iirc, to remove the solenoid plunger, requires manually pushing the starter pinion fully forward i.e. to what would be it's engaged position when starter motor is operated - this causes the top of the lever that connects the solenoid plunger to the starter pinion to move rearwards, which should allow the plunger to be released from the fork on the end of the lever.
If this ^^ doesn't work, you might have to remove the pivot pin from the operating lever to allow more movement.
If this ^^ doesn't work, you'll probably have to release the front housing from the starter motor (2 long through-bolts that go through the rear housing), and if you do have to do it this way, note carefully where/how item 9 (rubber lever buffer) fits, then fit the solenoid assembly to the front housing before refitting the front housing.
To identify the parts mentioned above :- See 1st image below, Item 7 is the starter pinion; Item 8 is the operating lever; Item 10 is the front housing.
View attachment 444147
Red arrow is pointing at how the solenoid plunger is attached to the operating lever - with the pinion manually pushed forward (i.e. to the left in image) upper end of operating lever moves rearwards (i.e. to the right in image) which exposes the open end of lever fork and should allow the solenoid plunger to be released (might need a bit of wriggling, iirc, the spring on the solenoid plunger hinders things a little).
View attachment 444149