If we take the Balearic slingers, they used projectiles weighing roughly 1 mina - that's 270 grams. Should we multiply 30-36 joules (24 grams) by 10? That would make it about 300-360 joules at least.
My physics are rather shoddy, so I'm sure that number would be a little too high.. but you get the idea. Although, the Balearic slingers didn't use lead bullets, but stones. So the coefficiency of drag will be different (different flight characteristics, and different projected flight path).
The armor would be too close to the skin to protect the body. It's like the newton balls.
Imagine the first ball is the stone, the second ball is the armor, the 3rd is the human body. The kinetic energy is transferred easily. From the slinging stone, to the armor, to the skin. It would cause internal bleeding. It's called "blunt trauma" in the medical field - and it can indeed be fatal. Only way to protect yourself is to have padding inside - and even that would be of limited efficacy.