An introduction to Guidance: BabyAGI without OpenAI

• Twitter: theemozilla
• Jupyter Notebook: babyagi-without-openai-guidance.ipynb

Guidance¶

If you’re already convinced of how awesome Guidance is, you can skip this section

Guidance is a new template language from Microsoft that allows you to guide the output of lanugage models. This may not sound revolutionary on the surface, but if you’ve ever tried to compose together ouputs from a LM in a non-trivial manner and spent hours re-processing the same prompt or writing tons of boilerplate chaining code, Guidance really can be a next-level unlock.

One of the biggest frustrations when it comes to chaining LM outputs is trying to “force” the model to generate in a specific way, for example following certain steps or in a structure that’s easily programatically interpreted. Indeed, the classic chain-of-thought prompt (i.e. “let’s think about it step-by-step”) is merely a prompting technique to guide the model to reason about the task in a desireable way. But what if we could just like… make the model do what we want? That’s where Guidance comes in.

Guidance is essentially Handlebars templates for language models, where a magic gen command invokes the model in the context of the evaluated template up to this point. As developers, we write the template structure of what we want the output of the model to be, and Guidance manages having the model “fill in the blanks”. By way of illustration, consider:

Here, the text on the white background is static, the blue background text represents variables passed in at runtime, and the green background text are the outputs generated by the language model. The Guidance code for this is:

The following is a character profile for an RPG game in JSON format.
```json
{
    "id": "{{id}}",
    "description": "{{description}}",
    "name": "{{gen 'name'}}",
    "age": {{gen 'age' pattern='[0-9]+' stop=','}},
    "armor": "{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{/select}}",
    "weapon": "{{select 'weapon' options=valid_weapons}}",
    "class": "{{gen 'class'}}",
    "mantra": "{{gen 'mantra' temperature=0.7}}",
    "strength": {{gen 'strength' pattern='[0-9]+' stop=','}},
    "items": [{{#geneach 'items' num_iterations=5 join=', '}}"{{gen 'this' temperature=0.7}}"{{/geneach}}]
}```

In Guidance we specify how we want our final text to look, using control statements such as gen to specify where the model should be used to generate specific pieces. There are tons of different control statements — in the above example we use the select statement to choose between only a few options (underneath Guidance compares the log proabilities of the options and chooses the most likely one). We can also specify regular expressions that define the valid form of an output (making strength be numeric). The Guidance repo has tons of example notebooks that show off its various capabilities. One thing I will say is that, as of this writing (May 2023) there is almost no formalized documentation as Guidance is still very much a work-in-progess. I did all of my learning by looking through the example notebooks.

Local models in Guidance¶

One of the best things about Guidance is its first-class support of running on local models via transformers. I would go so far as to say that in Guidance local models are strictly better than using the OpenAI API (although it does support this as well). Perhaps its greatest feature from a quality-of-life perspective when using local models is “acceleration”.

Guidance acceleration works by caching the intermediate states of the model up to the boundary points within the template. When the the template is re-evaluated, Guidance can “skip ahead” to the point where the prompt actually changes. This greatly speeds up the core write-run-evaluate core developer loop when designing templates. If you’ve ever sat there re-running the same prompt for the ten thousandth time just to get to where you’ve made a change, then acceleration is truly a life saver.

BabyAGI¶

BabyAGI is a proof-of-concept “AGI”. In reality, it’s a task management system that uses a vector database and instructs the language model to plan out and exceute tasks aimed at completing a specific objective (for example, “paperclip the universe”).

The reference BabyAGI code uses the OpenAI APIs. I will refrain from editorializing here, but let’s just say I’d prefer to be able to do this using open source models 🙂.

BabyAGI in Guidance¶

First, we need to install the prerequisites. We’ll be using transformers and accelerate to run our local models and langchain, faiss-cpu, and sentence_transformers for embeddings and vector database.

We’ll be using Vicuna as our language model. Vicuna has shown to be an excellent open-source competitor to GPT-3.5. It is a 13B parameter model finetuned from Llama.

Loading a 13B parameter model takes 52 GB of RAM when loaded at full precision or 26 GB at half precision. transformers has the helpful load_in_8bit parameter that reduces this to 13 GB, but unfortunately Guidance doesn’t support this yet (until PR #8 merged). I have a fork of Guidance that adds in support — install this if you want to use 8-bit quantization. Otherwise, the regular guidance package suffices.

Next, it’s time to load our model!

We’ll use LangChain to help us with embeddings and vector databases. The HuggingFaceEmbeddings class uses all-mpnet-base-v2 to generate embeddings, and we’ll use the simple FAISS vector database from Meta.

Okay, let’s write some Guidance. BabyAGI has three core prompts: task execution, task creation, and task prioritization. We’ll start with the task execution prompt. The reference implementation can be found here.

Many models are “chat” trained. That is to say, they’ve been finetuned to act as a chatbot. To get the best performance out of these models, you need to prompt them in the same way that they were trained. Unfortunately, each model has its own idiosyncrasies for this prompting. Normally, this would mean you would need different prompts for each model. However, Guidance solves this by the special #system, #user, and #assistant commands. Several popular models are supported (Vicuna, StableLM, MPT) and adding support for a new model is a simple subclass. Once supported, the same prompt can be used across different chat trained models.

In this prompt, objective, context, and task are variables we’ll pass in at runtime. We’ll generate the result variable, which will be accessible as a property of the returned object when we call execution_prompt.

When executing in a Jupyter notebook, Guidance automatically presents the output in an easily digestable format. We can see the seperate system, user, and assistant sections and see where runtime varabiles were inserted (blue) ana the model generated new text (green). Every variable that is created via gen is available in the result object directly — already parsed!

Next up is the task creation prompt. A large part of the reference implementation is concerned with cleaning up the response from the model, and hoping (praying) it will follow instructions regarding how to structure the output so it can be programatically iterepreted. But, this is just the task the Guidance excels at!

Our goal is to have the model take whatever the output of the previous task was and create a list of tasks to carry out the objective. This is done iteratively, so we pass in any previously incomplete tasks, and ask the model for new tasks. While we politely ask the model to output the tasks as an array, with Guidance we can actually make it happen.

The geneach command tells Guidance to do a looped generation, creating a list of tasks. By placing geneach inside of [""] we can enforce a JSON array structure (the unless command is used to add neccessary commas).

Our tasks correctly generated and parsed! We then place these in a dictionary under the key task_name for later use in the full BabyAGI procedure.

The last prompt to write is the task prioritization prompt.

In the task creation prompt we asked the model to output the tasks as a JSON array. Now we’ll try a harder use case: an ordered numerical list starting with some arbitrary number.

Based on experience, the model often hallucinates here and invents new tasks or repeats them. However, we want it to simply re-order the exisiting tasks. While this could be solved with a redesigned prompt or a more aligned model, we can also use the if command to control the number of tasks that can be outputted and stop after a predetermined number.

Here, we compare the special @index of the current task being generated (a magic number updated by Guidance after each loop) and break out if it’s equal to the number of tasks we know we passed in.

Now it’s time for the full BabyAGI procedure. It may be helpful to refer back to the diagram that shows the flow of logic, but essentially it’s our three prompts being run in succession.

And there you have it, BabyAGI in Guidance using only local, open source models!

Hopefully this was a helpful introduction. Likely this will quickly get outdated as Guidance is under very active development.