{"id":1978,"date":"2024-04-26T10:30:00","date_gmt":"2024-04-26T00:30:00","guid":{"rendered":"https:\/\/digitalbbq.au\/?p=1978"},"modified":"2024-04-25T12:03:42","modified_gmt":"2024-04-25T02:03:42","slug":"a-no-code-guide-to-local-large-language-models","status":"publish","type":"post","link":"https:\/\/digitalbbq.au\/index.php\/2024\/04\/26\/a-no-code-guide-to-local-large-language-models\/","title":{"rendered":"A No-Code Guide to Local Large Language Models"},"content":{"rendered":"\n

Artificial Intelligence (AI) has exploded since the public release of ChatGPT, bringing the power of AI to the masses at a low cost has changed the way that many of us work and interact with the world. However, there are concerns around the privacy and security of using online, privately owned large language models (LLMs). These concerns include data privacy and security risks, the potential for LLMs to memorize sensitive information, ethical considerations in data usage, regulatory compliance challenges, and the need for informed consent and transparency.<\/p>\n\n\n\n

The alternative to using online services like OpenAI’s ChatGPT and Google’s Gemini is to run a large language model locally, but for the average person, this is a non-starter as there is a considerable technical barrier to entry.<\/p>\n\n\n\n

Well, not anymore.<\/p>\n\n\n\n

Today we’ll explore how to easily set up a highly capable, locally running, “talk to anything with any LLM” application with Retreival Augmented Generation (RAG) capability using just your laptop or desktop. By leveraging two powerful tools, LMStudio and AnythingLLM, you can create a comprehensive LLM experience without any coding required.<\/p>\n\n\n\n

Disclaimer: I am not affiliated with either LMStudio or AnythingLLM, and the information provided in this blog post is for informational purposes only. Please assess whether this guide suits your needs before proceeding.<\/pre>\n\n\n\n
LMStudio<\/h2>\n\n\n\n
LMStudio<\/a> is a user-friendly desktop application that allows users to experiment with local and open-source large language models (LLMs) and allows individuals to leverage the power of LLMs on their own machines without relying on cloud-based services.<\/p>\n\n\n\n
One of the key advantages of LMStudio<\/a> is its simplicity, making it accessible to users with varying levels of technical expertise, its a one-stop-shop for downloading, configuring, and running a wide range of ggml-compatible models from Hugging Face.<\/p>\n\n\n\n
Because your data remains local and under your control, LMStudio is an ideal choice for users who value data privacy and want to build and experiment with AI models without exposing sensitive information to third-party services.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
To install LMStudio, simply head over to lmstudio.ai<\/a> and download the right version for your operating system and hardware. If your system has an RX6000 or newer AMD GPU and you want to utilise GPU acceleration, at this time, you will need to download the technology preview version of lmstudio that supports AMD ROCm<\/a>.<\/p>\n\n\n\n
LMStudio doesn’t come pre-packaged with a language model, but once LMStudio is installed, the home screen presents you with an interface to download a language model from Hugging Face. You can choose one of the well-known models such as Llama 2 or the new Llama 3 model, Mistral, Google Gemma, Microsoft Phi2, Alibaba’s Qwen and much more. Which model you download depends on what you want to achieve, but for basic chat, any of the popular chat models that are able to run on your local hardware will suit your needs.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Once you have a model downloaded, click on the chat icon in the menubar on the left and load the model. In this example, I’m running the Meta Llama 3 Instruct model with 7 billion parameters and 8bit quantization – I’ll explain all of this later.<\/p>\n\n\n\n
To load the model, simply click the big blue Select a model to load<\/em> button at the top of the window, and you’re good to go. Performance will depend on your hardware, but it’s exciting having a local chat at your fingertips all the same.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
But what if you want to chat with documents? That’s where AnythingLLM comes in.<\/p>\n\n\n\n
AnythingLLM<\/h2>\n\n\n\n
AnythingLLM<\/a> is a desktop application that allows you to chat with anything. It can connect to various LLMs such as ChatGPT, Anthropic, Mistral and most importantly for our local setup, various local LLM platforms such as LMStudio.<\/p>\n\n\n\n
When you first run AnythingLLM, you will be asked to select your LLM, your embeddings and your vector database. To keep things incredibly simple, all you need to do here is select LMStudio as your LLM, and leave the embeddings and vector database using the defaults.<\/p>\n\n\n\n
When selecting LMStudio as your LLM, you will be prompted to input the URL and the token context window for LMStudio.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You can find this information in LMStudio, firstly, click on the folder icon in the menubar to browse your downloaded models and open the model inspector, this will show information about the model you want to use. In this case, our Llama 3 model has a context window of 8192.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Next, to find the server URL, start the server within LMStudio<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Once the server loads, it will display the URL in the server log which is http:\/\/localhost:1234<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Once you have those values configured in AnythingLLM, you’ll be asked to configure a workspace. The workspace is where you can upload documents, or connect to webpages and github repositories. It’s also where you will be chatting with your documents while LMStudio is in server mode.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
From here it’s time to start chatting. I have found that the quality of responses is highly dependent on the quality and clarity of the document that you’re referencing. There are also minor adjustments you can make to how AnythingLLM ingests the dataset to fine-tune text matching capabilities.<\/p>\n\n\n\n
The way the document retrieval works is that the text in the document is chunked into smaller portions and stored as a number in a vector database. When you ask the LLM a question, information stored in the vector database is checked for similarity to your question, and then used as context in the LLM response.<\/p>\n\n\n\n
As an example, if I load The Toyota Way<\/em> by Jeffrey Liker into AnythingLLM and ask questions about lean management concepts, the responses generated as a result are comprehensive and accurate.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
On the other hand, if we load a document comprised of short clauses and a large number of cross-references the retrieval ability and resulting accuracy of the responses with the default settings are hit and miss.<\/p>\n\n\n\n
I have also been finding an increasing number of documents are oddly formatted when printed to PDF, such as this example from the Plumbing and Drainage standard, AS3500.1:2021<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
I’m not sure as to the cause of these formatting anomalies, but as humans, we are able to make sense of the jumbled letters extracted from the PDF; for our vector database and our language model to reference, the pdf context becomes an irrelevant mess.<\/p>\n\n\n\n
To improve the responses, consider fine-tuning the following:<\/p>\n\n\n\n
    \n
  • Remove superfluous information from the document and re-upload. Remove contents pages, bibliographies and other texts that do not contain relevant content and improve the PDF formatting and compatibility where possible.<\/li>\n\n\n\n
  • Adjust the text chunking and chunk overlaps. Text chunking is a method used in natural language processing that divides text into smaller segments, usually based on grammatical meanings and phrases. Adjusting these values can improve your responses<\/li>\n\n\n\n
  • Consider adjusting the number of context snippets and the similarity thresholds of the vector database in your workspace.<\/li>\n\n\n\n
  • Change your prompt. As simple as it sounds, adjusting your prompt and being more direct can result in more accurate responses.<\/li>\n<\/ul>\n\n\n\n
    This is of course not the only method for configuring a local LLM, but it is arguably the quickest. Including downloading of a language model, you can create a powerful, private, and customizable LLM experience in under an hour. The future of local LLM usage is becoming more accessible thanks to tools like LMStudio, ollama, Jan and Local AI. Paired with a user-friendly interface like AnythingLLM, you can build out powerful local LLMs without relying on paid services. The model you choose will ultimately determine your chatting experience, so be sure to select one that aligns with your needs.<\/p>\n","protected":false},"excerpt":{"rendered":"
    Artificial Intelligence (AI) has exploded since the public release of ChatGPT, bringing the power of AI to the masses at a low cost has changed the way that many of us work and interact with the world. However, there are concerns around the privacy and security of using online, privately owned large language models (LLMs). These concerns include data privacy…<\/p>\n","protected":false},"author":1,"featured_media":1979,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"advanced_seo_description":"Discover how to set up a local LLM using LMStudio and AnythingLLM without coding. Enjoy privacy, customization, and cost-effectiveness in this comprehensive guide.","jetpack_seo_html_title":"","jetpack_seo_noindex":false,"jetpack_seo_schema_type":"","_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Learn how to set up a local LLM using LMStudio and AnythingLLM in this no-code guide. Enjoy privacy, customization, and cost-effectiveness while exploring the world of AI. #LocalLLM #AI","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[1],"tags":[82,77],"class_list":["post-1978","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-digital","tag-ai","tag-automation"],"jetpack_publicize_connections":[],"jetpack-related-posts":[{"id":2208,"url":"https:\/\/digitalbbq.au\/index.php\/2024\/10\/17\/structuring-your-thoughts-and-general-assistance-with-ai\/","url_meta":{"origin":1978,"position":0},"title":"Structuring Your Thoughts and General Assistance with AI","author":"Ryan Lenihan","date":"17 October 2024","format":false,"excerpt":"In the AEC industry, staying ahead relies on efficient communication, planning, and the ability to turn complex, fragmented information into actionable insights. One practical way AI is helping with this is through Large Language Models (LLMs), like GPT. Think of these models as a \"calculator for words\" that helps you\u2026","rel":"","context":"In "Digital"","block_context":{"text":"Digital","link":"https:\/\/digitalbbq.au\/index.php\/category\/digital\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/2024-09-29_15-29-59.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/2024-09-29_15-29-59.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/2024-09-29_15-29-59.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/2024-09-29_15-29-59.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/2024-09-29_15-29-59.jpg?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":2224,"url":"https:\/\/digitalbbq.au\/index.php\/2024\/11\/21\/automating-project-requirements-analysis-for-faster-bim-planning\/","url_meta":{"origin":1978,"position":1},"title":"Automating Project Requirements Analysis for Faster BIM Planning","author":"Ryan Lenihan","date":"21 November 2024","format":false,"excerpt":"As architects, engineers, and construction professionals, we all know how critical it is to manage project requirements efficiently to ensure we develop comprehensive BIM Execution plans for our projects. Artificial intelligence, particularly Large Language Models (LLMs), can now help streamline the traditionally manual and time-consuming process of analyzing these requirements.\u2026","rel":"","context":"In "Digital"","block_context":{"text":"Digital","link":"https:\/\/digitalbbq.au\/index.php\/category\/digital\/"},"img":{"alt_text":"Business women creating a plan in coworking space","src":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/business-women-creating-a-plan-in-coworking-space.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/business-women-creating-a-plan-in-coworking-space.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/business-women-creating-a-plan-in-coworking-space.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/business-women-creating-a-plan-in-coworking-space.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/business-women-creating-a-plan-in-coworking-space.jpg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/business-women-creating-a-plan-in-coworking-space.jpg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":2174,"url":"https:\/\/digitalbbq.au\/index.php\/2024\/09\/26\/the-rapid-advancement-of-ai-and-understanding-its-capabilities-and-limitations\/","url_meta":{"origin":1978,"position":2},"title":"The Rapid Advancement of AI and Understanding Its Capabilities and Limitations","author":"Ryan Lenihan","date":"26 September 2024","format":false,"excerpt":"Welcome to the first installment of our series based on my presentation at BiLT in Melbourne, titled \"Top 10 Practical AI Workflows to Elevate Your Practice Today.\" In this series, I plan to explore each topic covered in more detail, starting with a fundamental understanding of Artificial Intelligence (AI). Whether\u2026","rel":"","context":"In "Digital"","block_context":{"text":"Digital","link":"https:\/\/digitalbbq.au\/index.php\/category\/digital\/"},"img":{"alt_text":"Yellow Toy Robot Manning AI Chat Stand","src":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/yellow-toy-robot-manning-ai-chat-stand.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/yellow-toy-robot-manning-ai-chat-stand.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/yellow-toy-robot-manning-ai-chat-stand.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/yellow-toy-robot-manning-ai-chat-stand.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/yellow-toy-robot-manning-ai-chat-stand.jpg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/yellow-toy-robot-manning-ai-chat-stand.jpg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":2631,"url":"https:\/\/digitalbbq.au\/index.php\/2026\/05\/07\/ai-overconfidence-and-the-dunning-kruger-effect-in-the-workplace\/","url_meta":{"origin":1978,"position":3},"title":"AI, Overconfidence, and the Dunning\u2013Kruger Effect in the Workplace","author":"Ryan Lenihan","date":"7 May 2026","format":false,"excerpt":"Artificial intelligence (AI) tools like large language models (LLMs) can act as a double-edged sword in professional settings. They can be powerful \u201cforce multipliers\u201d that enhance expert productivity but also risk amplifying the Dunning\u2013Kruger effect among less experienced users. When novices lean on AI-generated outputs without sufficient knowledge to validate\u2026","rel":"","context":"In "Team Building"","block_context":{"text":"Team Building","link":"https:\/\/digitalbbq.au\/index.php\/category\/team-building\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2026\/05\/corporate_success_dke_sm.png?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2026\/05\/corporate_success_dke_sm.png?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2026\/05\/corporate_success_dke_sm.png?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2026\/05\/corporate_success_dke_sm.png?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2026\/05\/corporate_success_dke_sm.png?resize=1050%2C600&ssl=1 3x"},"classes":[]},{"id":2106,"url":"https:\/\/digitalbbq.au\/index.php\/2024\/08\/01\/giving-excel-an-ai-brain-with-office-scripts\/","url_meta":{"origin":1978,"position":4},"title":"Giving Excel an AI Brain with Office Scripts","author":"Ryan Lenihan","date":"1 August 2024","format":false,"excerpt":"As an Excel user, I'm always looking for ways to automate repetitive tasks and make my workflow more efficient. That's where Office Scripts come in. The way office scripts are advertised out of the box is that they're a feature in Excel that enables you to automate tasks by recording\u2026","rel":"","context":"In "Digital"","block_context":{"text":"Digital","link":"https:\/\/digitalbbq.au\/index.php\/category\/digital\/"},"img":{"alt_text":"Spreadsheet table paper with pencil with math number, education concept.","src":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/07\/spreadsheet-table-paper-with-pencil-with-math-number-education-concept-.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/07\/spreadsheet-table-paper-with-pencil-with-math-number-education-concept-.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/07\/spreadsheet-table-paper-with-pencil-with-math-number-education-concept-.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/07\/spreadsheet-table-paper-with-pencil-with-math-number-education-concept-.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/07\/spreadsheet-table-paper-with-pencil-with-math-number-education-concept-.jpg?resize=1050%2C600&ssl=1 3x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/07\/spreadsheet-table-paper-with-pencil-with-math-number-education-concept-.jpg?resize=1400%2C800&ssl=1 4x"},"classes":[]},{"id":2218,"url":"https:\/\/digitalbbq.au\/index.php\/2024\/10\/24\/email-productivity-with-ai-and-robotic-process-automation\/","url_meta":{"origin":1978,"position":5},"title":"Email Productivity with AI and Robotic Process Automation","author":"Ryan Lenihan","date":"24 October 2024","format":false,"excerpt":"If you're like me, some days you might receive a hundred or more emails on any given day, and managing that influx of emails can be tedious. Between coordinating with clients, handling project updates, and scheduling site visits, email can quickly become overwhelming. That\u2019s where email automation using LLMs like\u2026","rel":"","context":"In "Digital"","block_context":{"text":"Digital","link":"https:\/\/digitalbbq.au\/index.php\/category\/digital\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/b7004048-a68a-443b-a18d-64d7b4669509.jpg?resize=350%2C200&ssl=1","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/b7004048-a68a-443b-a18d-64d7b4669509.jpg?resize=350%2C200&ssl=1 1x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/b7004048-a68a-443b-a18d-64d7b4669509.jpg?resize=525%2C300&ssl=1 1.5x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/b7004048-a68a-443b-a18d-64d7b4669509.jpg?resize=700%2C400&ssl=1 2x, https:\/\/i0.wp.com\/digitalbbq.au\/wp-content\/uploads\/2024\/09\/b7004048-a68a-443b-a18d-64d7b4669509.jpg?resize=1050%2C600&ssl=1 3x"},"classes":[]}],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/digitalbbq.au\/wp-content\/uploads\/2024\/04\/artificial-neuron-concept.jpg","_links":{"self":[{"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/posts\/1978","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/comments?post=1978"}],"version-history":[{"count":5,"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/posts\/1978\/revisions"}],"predecessor-version":[{"id":1999,"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/posts\/1978\/revisions\/1999"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/media\/1979"}],"wp:attachment":[{"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/media?parent=1978"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/categories?post=1978"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/digitalbbq.au\/index.php\/wp-json\/wp\/v2\/tags?post=1978"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}