I drove a Z4 3ltr back when they were launched, a 3400S again on launch and a 3.0si Coupe a few year later.
Coming from a 911 background, I really found all three disappointing in terms of steering feel, traction, throttle response, chassis stiffness. Basically all three had nice engines and were great to cruise around in. But when pressing on, lacked involvement, as can be seen from the reviews at the time. Way before the 911, I had an MX5 which was just fantastic fun, road or track.
It was berluddy disappointing to be honest and so I had zero interest in driving an M. I drove an early 330i E90 which had excellent steering, handling and an acceptable ride, so bought that as a stop gap.
I got bored one day and drove a Z4M that was on display. What a revelation. Decent steering, stiff chassis, beautiful vocal engine, good throttle reponse (for DBW), great to drift around. Lovely. Loads of character. Five years later I'm still loving it. It's far from faultless, but every drive is an event.
I might switch it in a couple of years for a manual 997.2 C2, but otherwise, I can't think of much to replace it.