Large language models (LLM), such as chatgpt, have gained significant popularity and media attention. However, their development is mainly dominated by several well -financed technological giants due to the excessive costs associated with the preference of these models, estimated at at least $ 10 million, but probably much higher.
The factor has limited access to LLM for smaller organizations and academic groups, but a team of scientists at Stanford University is to change this. Leaded by a graduate Hong Liu, they developed an innovative approach called Sophia, which can shorten the time of claim by half.
The key to optimizing Sophia is two innovative techniques developed by Stanford. The first technique, known as an estimate of the curvature, consists in improving the efficiency of estimating the curvature of LLM parameters. To illustrate this, Liu compares the LLM preferring process with the mounting line in the factory. Like the factory manager, he tries to optimize the steps required to transform the raw materials into a finished product, the preferring LLM consists in optimizing the progress of millions or billions of parameters towards the final goal. The crooked of these parameters represents their maximum reaching speed, analogous to the factory employees.
While the estimation of the curvature was difficult and expensive, Stanford researchers found a way to increase performance. They noticed that earlier methods updated the estimation of curvature at every stage of optimization, which leads to potential ineffectiveness. In Sophia, they reduced the frequency of the curvature of the curvature to about 10 steps, which gives significant profits from performance.
The second technique used by Sophia is called clipping. It aims to overcome the problem with the inaccurate estimation of the curvature. Setting the maximum estimate of curvature, Sophia prevents overloading LLM parameters. The team compares this to imposing work with factory employees or navigation in the optimization landscape aimed at reaching the lowest valley, avoiding saddle points.
The Stanford team tested Sophia, subjecting to a relatively small LLM using the same size and configuration of the model as GPT-2 OpenAI. By combining the estimation and cut of the curvature of Sophia, it reached a 50% reduction in the number of optimization steps and the required time compared to the widely used Adam optimizer.
One noteworthy advantage of Sophia is its adaptation, enabling her to manage parameters of various curves more effectively than Adam. In addition, this breakthrough means the first significant improvement in relation to Adam in the language model for nine years. Liu believes that Sophia can significantly reduce the costs of training large large models, with even greater benefits as the models are scaled.
Looking to the future, Liu and his colleagues plan to apply Sophia to larger LLM and examine its potential in other domains, such as computer vision models and multimodal models. Although Sophia's transition to new areas will require time and resources, its Nature Open Source allows a wider community to contribute and adapt it to various fields.
To sum up, Sophia is a significant progress in accelerating a large language model, democratizing access to these models and potentially revolutionizing various fields of machine learning.
