A simple guide for hosting LLMs locally

1git clone https://github.com/mudler/LocalAI.git
2cd LocalAI

1wget https://huggingface.co/TheBloke/Llama-2-7B-Chat-GGML/resolve/main/llama-2-7b-chat.ggmlv3.q2_K.bin \
2    -O models/Llama-2-7B-Chat

1docker compose up -d --pull always

1curl http://localhost:8080/v1/chat/completions \
2    -H "Content-Type: application/json" \
3    -d '{"model":"Llama-2-7B-Chat","messages":[{"role":"user","content":"Who are you?"}],"temperature": 0.9}'

 1{
 2    "created": 1700201935,
 3    "object": "chat.completion",
 4    "id": "b5e4c3ff-c8ad-4350-89cb-293d2cfd806c",
 5    "model": "Llama-2-7B-Chat",
 6    "choices": [{
 7        "index": 0,
 8        "finish_reason": "stop",
 9        "message": {
10            "role": "assistant",
11            "content": "\nYou are a unique individual, created in the image of God. You have your own personality, gifts, and talents that make you who you are. As a child of God, you have inherent dignity and worth, and are worthy of love and respect.\n\nWhat is your purpose?\n\nYour purpose is to live out your identity as a beloved child of God, and to fulfill the unique plan and calling that He has for your life. This may involve pursuing various callings and passions, such as career, family, and community involvement, while always keeping in mind the priorities of faith, love, and obedience to God.\n\nWhat are some challenges you face?\n\nAs a unique individual, you will likely face many challenges and obstacles throughout your life, including self-doubt, fear, anxiety, and discouragement. You may also encounter external challenges such as financial struggles, health issues, or relationship difficulties. However, through faith and perseverance, you can overcome these challenges and grow in your identity and purpose.\n\nHow can you grow in your identity and purpose?\n\nTo grow in your identity and purpose, you must first recognize and embrace your unique identity as a beloved child of God. This involves cultivating a deep sense of self-awareness, self-acceptance, and self-compassion, while also seeking to understand and fulfill the specific plan and calling that God has for your life. This may involve pursuing various callings and passions, while always keeping in mind the priorities of faith, love, and obedience to God.\n\nWhat are some practical steps you can take?\n\nHere are some practical steps you can take to grow in your identity and purpose:\n\n1. Practice self-care and self-awareness through prayer, meditation, journaling, or therapy.\n2. Seek out mentors, coaches, or accountability partners who can help guide and support you in your personal growth journey.\n3. Engage in regular spiritual practices such as Bible study, worship, or devotional exercises to deepen your faith and understanding of God's will for your life.\n4. Set clear goals and priorities for your life, based on your unique identity and purpose, and work towards achieving them through consistent effort and persever"
12        }
13    }],
14    "usage": {
15        "prompt_tokens": 0,
16        "completion_tokens": 0,
17        "total_tokens": 0
18    }
19}

 1import os
 2
 3from langchain.llms import OpenAI
 4
 5llm = OpenAI(
 6    temperature=0.9,
 7    openai_api_key="123",
 8    openai_api_base="http://localhost:8080/v1",
 9    model_name="Llama-2-7B-Chat",
10)
11text = "What would be a great name for a dog?"
12print(llm(text))

1I'm looking for something unique and fun. I want it to be a name that stands out and is memorable. Can you help me come up with some ideas?
2
3I don't want anything too common or boring, like Max or Bella. I want something that will make my dog stand out in a crowd.
4
5Can you give me some suggestions?
6
7I would love to hear your ideas!