Force equals mass times acceleration squared, so acceleration is more important than mass for the force. It depends how much the larger object slows you down if it ends up with more power when it hits, which would be more of a factor the more at you're limit you are and if it's you having to physically move the object. If there's a little rock and a medium rock and both you can sling forward with relative ease, the medium would indeed tend to always do more damage. If you can barely toss a huge rock forward it could still hurt if it falls and crushes your foot, but getting hit with a small pebble would sting more on the initial impact then the large rock getting tossed forward into you, more like a bump. Gravity is supplying the acceleration needed for the heavier object to do more damage in that case as it falls down onto you, otherwise it would be very weak.