I was cleaning some pistols last night and I remembered that I never really tried out all of the grip inserts on my M&P 2.0. Well, none of them felt bad or uncomfortable, in fact they all felt pretty good. I just had to randomly pick one. I've had the same dilemma with other pistols that have grip inserts, especially the VP9. Everytime I thought I had found my favorite combination I started second guessing because there are so many combinations.
This is where Sig really shines with the P320. The medium compact grip feels just right. Having other inserts would just interfere with that simple good fit. I suppose were spoiled when people complain about too many choices.

Your point is very well taken. The P320 medium full/carry size fits perfect for me.

And even if it wasn’t, grips are cheap and can transform the P320 far more that a palm swell. I made my FFL try my P320 Legion with the Wilson grip on it and he couldn’t believe it was the same gun.

Try them out while shooting. Some point different for me even if they feel good. If that also doesn’t make a difference, I usually go with the smallest grips for better control.