I would like the idea to have a private messaging system where users can speak directly to admins or mods, but not user to user. The private conversations with mods could be about disputes, as Curious_101 mentioned, where users can contact a mod for private concerns. I'm guessing the flag/report function works similar this way, but having a response back from the mods would show response and action.
The reason I'd vote no, is because user to user private conversations would lessen the activity in this forum. I've read several intros where users read many posts before deciding to join the site, and because of that activity, it helps people get a better understanding of what this site shares. Plus, Rina and Renita are right about a possible wave of harassments popping up, the moment trolls are free to send messages.
Good luck finding the middle ground where everyone can agree on something. This site is going to be completely different one day.