In this blog post I dive into the plans for Legally Speaking AI to finetune our own proprietary model based on the open sourced weights available today.

After the publication of the weights of Meta's large scale language model (LLM) LLaMA, there has been a surge in the capabilities of cheap finetuning. Through 🤗 HuggingFace's PEFT library, users have been enabled to finetune LLMs with consumer hardware. The results have been staggering, for specific tasks these simpler LLMs outperform the likes of GPT4 when tuned on custom datasets.

Part of Legally Speaking AI's mote is the large Swedish database of law-related queries. We can leverage this to finetune an LLM. I did this in a Google Colab Notebook, though the entire dataset can't be incorporated on the GPU's provided in Colab. However, there are currently no commercially licensed LLMs with Swedish knowledge. Once this is provided, or I assemble a team that can train our own pre-trained model from scratch, I must rely on the leaked weights of LLaMA, which I cannot deploy commercially and have bad knowledge of Swedish.

Law GPT opportunities

There is a large opportunity to create a pipeline, leveraging commercially licensed models as they are published, to capture the non-english speaking legal services market. Though, we stand at a crossroads, either we train our own model from scratch or we wait for the open-sourced LLMs to improve and deploy finetuning.

• Low hardware requirement to deploy strong finetuned model.
• Commercial mote through access to Swedish legal data and queries.
• In need of commercially licensed model, currently waiting for launch.
• Meanwhile, one can finetune OpenAI's models, but to quite a high cost.

With this said, I believe we can have the joint strength to capture this LLM opportunity. Do not hesitate to reach out to us and join the journey!