JHD throwing down some nice animation tests lately. He talked a little of accomplishing subtle acting in the comments:
I shot reference and acted out the line over and over so I had a lot of material. There's a lot of garbage but every now and then there were moments that I liked so I started to pick out those moments. I had a general idea of the beats and what the acting choices were from the get go, I just wanted to get the subtleties from the reference.
But there are many moments that needed pushing and simplifying. After I had the main blocking done I got in and worked chunk by chunk until it felt right.
ran across Brendan Body's blog, he had some good thoughts on subtety: subtlety 1 subtlety 2 an interesting point he makes, if you're character is thinking exactly what they're saying, what's the point, big deal. Don't act it "on the nose" make them deeper.
here's the clip he's talking about in the 1st one, his dialogue starts around :53
a note from me on acting things out. Draw a face on a paper plate or something for you to focus on if your character is talking to someone, otherwise you're spending mental energy on trying to hold onto someone imaginary, energy you could be putting into being more into your performance. Also if your character has a prop, a hat, a cane, a bow tie, try and get one also, it will help you transition into being the character instead of just yourself. And act it out more then 10 times so you stop feeling self conscious and you stop just doing your cliche'd first idea.
thanks to David for find that post by Brendan