What If LLMs Learn Better from Language Than from Rewards?