By default JPEG images uploaded as custom avatars are resized at 75 / 100 quality. Sometimes this would result in bad-looking images so forum administrators may want to increase it. To do this:

Admin CP -> Settings -> Options -> Message Attachment Options -> Thumbnail Quality

That's it, just set it 95.

Now you will need to re-generate all the thumbnail images by the new quality index:

Maintenance -> General Update Tools -> Rebuild Custom Avatar Thumbnails, Rebuild Avatar Thumbnails

Done!