I did exactly the same swap back in January 2013 and I regretted it but I used my Z4M as a weekend motor and ran a 520d as a daily and hoped the M3 would be the complete all round single car for me but I was left disappointed by the M3, I did 10k miles in 10 months in it but with hindsight I wished I had stuck with my 520d/Z4M combo.
In saying that I seem to be the only one that was disappointed with the V8 M3 ownership experience as everyone that I read who has one loves them to bits so as they say you only live once so if its a scratch you want to itch then go for it
My collection day.