Reinforcement Learning (RL) has shown remarkable capabilities in solving complex decision-making problems, yet it often suffers from slow convergence and high computational demands. This study investigates the potential of using pre-trained Large Language Models (LLMs) as external tutors to accelerate RL convergence. A novel student-teacher architecture is proposed, where RL agents receive structured guidance from LLMs, including an advice reusing mechanism that stores and re-applies previously suggested actions. The effectiveness of this approach is evaluated across 49 experimental configurations, incorporating three RL algorithms (DQN, PPO, A2C), three environments (Blackjack, Connect Four, Snake), and three open-source LLMs (LLaMA3.1, Vicuna, DeepSeek-R1). Results demonstrate that LLM tutoring accelerates convergence without degrading the agents' final performance, with further improvements observed when advice reuse is employed. DeepSeek-R1, the largest tested model, achieved the most significant impact. These findings suggest a promising pathway for leveraging LLMs in RL training, highlighting opportunities for more sample-efficient, scalable, and explainable learning frameworks.