Observing another’s affective state and adjusting one’s behavior to respond to it, is the basic functionality of empathy. To enable robots to do this, they need a mechanism to learn how to provide the most appropriate empathic behavior through continuous interaction with humans. To this end, we propose a reinforcement learning based framework for cognitive empathy, which uses reinforcement learning to learn the most appropriate empathic behavior for different emotional states during human-robot interactions. To verify the proposed framework, an experiment is conducted with the humanoid robot Pepper over 28 participants, where their facial emotion expression is tracked continuously and used to select appropriate empathic behaviors. The obtained results show the proposed reinforcement learning model converges to the optimal empathic behaviors for all emotions that were expressed a sufficient number of times, which helps the participants feel more positive emotions like happiness.