Retrieval-Augmented Generation (RAG) offers a great way to provide data to Large Language Models (LLMs) and generate responses based on your own content. This approach is particularly beneficial when dealing with sensitive data that should remain within controlled environments. In my current projects as Private Equity Tech Advisor, I often handle such data. For that purpose, I have been using Meta’s LLama 3.2 model running locally rather than in the cloud.
But what if your use case doesn’t involve sensitive data? Could using a publicly accessible model like OpenAI’s GPT improve the outcomes? Let’s compare!
This is post is part of a series of posts on LLMs and RAGs. Check out the other articles as well:
LLama 32 versus OpenAI 3.2
I used the same prompt and the same document as input. Let’s compare the responses.
My prompt was:
DOCUMENT:
{context}
QUESTION:
{query}
INSTRUCTIONS:
Answer the user's QUESTION using the DOCUMENT text above.
Keep your answer grounded in the facts of the DOCUMENT.
If the DOCUMENT doesn’t contain the facts to answer the QUESTION then please say so.
The document (context) was based on an article I wrote some time ago on QA.
Meta’s Llama 3.2 model
Here’s the answer by Meta’s LLama 3.2 model
Can you tell me how modern QA should look like?
According to the document, a modern QA approach should be the following:
- Go without a separate QA team by default
- Hire engineers that write tests on the proper level and own the quality aspect
- Use metrics, quality KPIs, and post-mortems to oversee the quality of software in the department
- Have support teams when large features go live and exploratory testing
- Automation for few critical paths
In other words, a modern QA approach focuses on ensuring that engineers write high-quality tests and own the quality aspect, rather than relying solely on a separate QA team. It also emphasizes the importance of metrics, post-mortems, and automation to improve software quality.
Nice and short…
OpenAI 4o
And here is the response of OpenAI’s gpt-4o model:
Modern QA, based on the context provided, should integrate closely with the engineering process > rather than function as a separate team. Here are the key aspects of how modern QA should look:
Embedded Within Engineering: QA should not be a distinct team but rather integrated within > engineering. Engineers should be responsible for writing tests and ensuring quality from the outset.
Focus on Metrics and Quality KPIs: QA’s role shifts towards overseeing the quality of > software by monitoring metrics and conducting post-mortems when necessary. This involves tracking > bugs, customer support requests, and other quality indicators.
Support for Large Features and Exploratory Testing: While engineers and product managers > handle most testing and feature releases, QA can provide additional support for large features that > impact multiple areas of an application. This includes exploratory testing to ensure comprehensive > coverage.
Selective Automation: Instead of maintaining a large, unmanageable e2e test suite, QA should > focus on automating a few critical paths in the application. This ensures that crucial functions > like login/logout are tested reliably without overburdening the system with false positives.
Continuous Collaboration with Product Managers: Product managers should handle ticket > acceptance based on defined criteria and user experience expectations. QA can assist but should not > replace the product manager’s role in this process.
In summary, modern QA is about integrating quality assurance into the engineering workflow, > focusing on strategic automation, quality metrics, and providing support for complex features > rather than acting as a separate gatekeeping entity.
Pretty cool answer as well. Very educated. And different to what Llama 3.2 delivers.
Results
- OpenAI’s GPT-4o: The response was more comprehensive and aligned with my expectations. It provided a nuanced and detailed explanation of modern QA practices, emphasizing integration and strategic focus.
- Meta’s LLama 3.2: While the answer was accurate and useful, it was less detailed compared to GPT-4o’s response.
Additionally, OpenAI’s GPT-4o was significantly faster, delivering an almost instant response. In contrast, running LLama 3.2 locally took about a minute, highlighting the efficiency of cloud-based models for production-ready solutions (Yes - I know - apples to peaches comparison).
Summary
Choosing between LLama and OpenAI depends largely on your use case:
- For Sensitive Data: Opt for running LLama locally on a well-configured machine with a GPU. This setup makes sure that your data remains isolated from spying eyes.
- For Non-Sensitive Data: Starting a project with OpenAI is often more advantageous. It accelerates time to market with its speed and quality of results. If cost or sensitivity of data becomes a concern, you can always pivot back to a locally running model like LLama 3.2.
Ultimately, RAGs empower you to leverage the strengths of both local and cloud-based models, ensuring flexibility and performance tailored to your specific needs. Use them when you can!
