It really does depend what you want from your car.
I love the E46 M3, they are just so usable, yet still feel proper raw for what is, let's be honest, a luxury German coupe.
I actually preferred my SMG E46 M3 to my M3 CSL as an overall package. I flashed the CSL SMG software onto it, which was the one area I felt the regular car needed sorting, and I would have loved the steering rack from the CSL too, but over all I did find the regular M3 just a class package all round.
The problem for me with the E86, either 3.0 or M, is they got the suspension so wrong, more so at the rear, the front is not too bad, but the rear needs more bite and definitely less bounce!
Jump into an E46 M3 and that becomes so, so obvious.
Jump into a M Coupe from the 3.0si and the shortfalls are still there, doesn't matter what engine you have, there are other things that you notice well before you can truly appreciate what the S54 brings to the table, that is not something that can be said about the M3, with that everything is just better.