- \n
- Twitter: theemozilla<\/a><\/em><\/li>\n
- Jupyter Notebook: babyagi-without-openai-guidance.ipynb<\/a><\/em><\/li>\n<\/ul>\n
Guidance\u00b6<\/a><\/h3>
If you’re already convinced of how awesome Guidance is, you can skip this section<\/em><\/p>\n
Guidance<\/a> is a new template language from Microsoft that allows you to guide the output of lanugage models.\nThis may not sound revolutionary on the surface, but if you’ve ever tried to compose together ouputs from a LM in a non-trivial manner and spent hours re-processing the same prompt or writing tons of boilerplate chaining code, Guidance really can be a next-level unlock.<\/p>\n
One of the biggest frustrations when it comes to chaining LM outputs is trying to “force” the model to generate in a specific way, for example following certain steps or in a structure that’s easily programatically interpreted.\nIndeed, the classic chain-of-thought prompt (i.e. “let’s think about it step-by-step”) is merely a prompting technique to guide the model to reason about the task in a desireable way.\nBut what if we could just like… make the model do what we want?\nThat’s where Guidance comes in.<\/p>\n
Guidance is essentially Handlebars<\/a> templates for language models, where a magic
gen<\/code> command invokes the model in the context of the evaluated template up to this point.\nAs developers, we write the template structure of what we want the output of the model to be, and Guidance manages having the model “fill in the blanks”.\nBy way of illustration, consider:<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n<\/div>\n\nIn [ ]:<\/div>\n\n\n<\/span>from<\/span> IPython.display<\/span> import<\/span> Image<\/span>\nfrom<\/span> IPython.core.display<\/span> import<\/span> HTML<\/span> \nImage<\/span>(<\/span>url<\/span>=<\/span>\"https:\/\/raw.githubusercontent.com\/microsoft\/guidance\/main\/docs\/figures\/json_animation.gif\"<\/span>,<\/span> width<\/span>=<\/span>376<\/span>,<\/span> height<\/span>=<\/span>234<\/span>)<\/span>\n<\/pre><\/div>\n\n <\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<\/div>\n\n\n\n\n \nOut[ ]:<\/div>\n\n\n\n\n![](\"https:\/\/raw.githubusercontent.com\/microsoft\/guidance\/main\/docs\/figures\/json_animation.gif\")
\n<\/div>\n\n<\/div>\n\n<\/div>\n\n<\/div>\n\n<\/div>\n\n\n\n<\/div>\n\n<\/div>\nHere, the text on the white background is static, the blue background text represents variables passed in at runtime, and the green background text are the outputs generated by the language model.\nThe Guidance code for this is:<\/p>\n
The following is a character profile for an RPG game in JSON format.\n```json\n{\n \"id\": \"{{id}}\",\n \"description\": \"{{description}}\",\n \"name\": \"{{gen 'name'}}\",\n \"age\": {{gen 'age' pattern='[0-9]+' stop=','}},\n \"armor\": \"{{#select 'armor'}}leather{{or}}chainmail{{or}}plate{{\/select}}\",\n \"weapon\": \"{{select 'weapon' options=valid_weapons}}\",\n \"class\": \"{{gen 'class'}}\",\n \"mantra\": \"{{gen 'mantra' temperature=0.7}}\",\n \"strength\": {{gen 'strength' pattern='[0-9]+' stop=','}},\n \"items\": [{{#geneach 'items' num_iterations=5 join=', '}}\"{{gen 'this' temperature=0.7}}\"{{\/geneach}}]\n}```\n<\/code><\/pre>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<\/div>\n\n<\/div>\nIn Guidance we specify how we want our final text to look, using control statements such as
gen<\/code> to specify where the model should be used to generate specific pieces.\nThere are tons of different control statements — in the above example we use theselect<\/code> statement to choose between only a few options (underneath Guidance compares the log proabilities of the options and chooses the most likely one).\nWe can also specify regular expressions that define the valid form of an output (makingstrength<\/code> be numeric).\nThe Guidance repo has tons of example notebooks<\/a> that show off its various capabilities.\nOne thing I will say is that, as of this writing (May 2023) there is almost no formalized documentation as Guidance is still very<\/strong> much a work-in-progess.\nI did all of my learning by looking through the example notebooks.<\/p>\nLocal models in Guidance\u00b6<\/a><\/h4>
One of the best things about Guidance is its first-class support of running on local models via transformers<\/a>.\nI would go so far as to say that in Guidance local models are strictly better than using the OpenAI API (although it does support this as well).\nPerhaps its greatest feature from a quality-of-life perspective when using local models is “acceleration”.<\/p>\n
Guidance acceleration works by caching the intermediate states of the model up to the boundary points within the template.\nWhen the the template is re-evaluated, Guidance can “skip ahead” to the point where the prompt actually changes.\nThis greatly speeds up the core write-run-evaluate core developer loop when designing templates.\nIf you’ve ever sat there re-running the same prompt for the ten thousandth time just to get to where you’ve made a change, then acceleration is truly a life saver.<\/p>\n
BabyAGI\u00b6<\/a><\/h3>
BabyAGI<\/a> is a proof-of-concept “AGI”. In reality, it’s a task management system that uses a vector database and instructs the language model to plan out and exceute tasks aimed at completing a specific objective (for example, “paperclip the universe”).<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>
\n\n\n<\/div>\n\nIn [ ]:<\/div>\n\n\n<\/span>Image<\/span>(<\/span>url<\/span>=<\/span>\"https:\/\/user-images.githubusercontent.com\/21254008\/235015461-543a897f-70cc-4b63-941a-2ae3c9172b11.png\"<\/span>,<\/span> width<\/span>=<\/span>496<\/span>,<\/span> height<\/span>=<\/span>367<\/span>)<\/span>\n<\/pre><\/div>\n\n <\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n\n<\/div>\n\n\n\n\n \nOut[ ]:<\/div>\n\n\n\n\n![](\"https:\/\/user-images.githubusercontent.com\/21254008\/235015461-543a897f-70cc-4b63-941a-2ae3c9172b11.png\")
\n<\/div>\n\n<\/div>\n\n<\/div>\n\n<\/div>\n\n<\/div>\n\n\n\n<\/div>\n\n<\/div>\nThe reference BabyAGI code<\/a> uses the OpenAI APIs.\nI will refrain from editorializing here, but let’s just say I’d prefer to be able to do this using open source models 🙂.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n\n\n<\/div>\n\n<\/div>\nBabyAGI in Guidance\u00b6<\/a><\/h3>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
\n\n\n<\/div>\n\n<\/div>\nFirst, we need to install the prerequisites.\nWe’ll be using
transformers<\/code> andaccelerate<\/code> to run our local models andlangchain<\/code>,faiss-cpu<\/code>, andsentence_transformers<\/code> for embeddings and vector database.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n<\/div>\n\nIn [ ]:<\/div>\n\n\n<\/span>!<\/span>pip <\/span>install <\/span>transformers <\/span>accelerate <\/span>langchain <\/span>faiss-cpu <\/span>sentence_transformers <\/span>ipywidgets\n<\/pre><\/div>\n\n <\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<\/div>\n\n\n\n<\/div>\n\n<\/div>\nWe’ll be using Vicuna<\/a> as our language model.\nVicuna has shown to be an excellent open-source competitor to GPT-3.5.\nIt is a 13B parameter model finetuned from Llama.<\/p>\n
Loading a 13B parameter model takes 52 GB of RAM when loaded at full precision or 26 GB at half precision.\n
transformers<\/code> has the helpfulload_in_8bit<\/code> parameter that reduces this to 13 GB, but unfortunately Guidance doesn’t support this yet (until PR #8<\/a> merged).\nI have a fork of Guidance that adds in support — install this if you want to use 8-bit quantization. Otherwise, the regularguidance<\/code> package suffices.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n<\/div>\n\nIn [ ]:<\/div>\n\n\n<\/span># 8-bit quantization support<\/span>\n!<\/span>pip <\/span>install <\/span>git+https:\/\/github.com\/jquesnelle\/guidance@transformers-quantization-parameters <\/span>bitsandbytes\n\n# or, regular guidance<\/span>\n# !pip install guidance<\/span>\n<\/pre><\/div>\n\n <\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<\/div>\n\n\n\n<\/div>\n\n<\/div>\nNext, it’s time to load our model!<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>
\n\n\n<\/div>\n\nIn [ ]:<\/div>\n\n\n<\/span>import<\/span> guidance<\/span>\n\nllm<\/span> =<\/span> guidance<\/span>.<\/span>llms<\/span>.<\/span>transformers<\/span>.<\/span>Vicuna<\/span>(<\/span>\n model<\/span>=<\/span>\"eachadea\/vicuna-13b-1.1\"<\/span>,<\/span>\n device_map<\/span>=<\/span>\"auto\"<\/span>,<\/span>\n load_in_8bit<\/span>=<\/span>True<\/span>\n)<\/span>\n<\/pre><\/div>\n\n <\/div>\n<\/div>\n<\/div>\n<\/div>\n\n<\/div>\n\n\n\n<\/div>\n\n<\/div>\nWe’ll use LangChain<\/a> to help us with embeddings and vector databases.\nThe
HuggingFaceEmbeddings<\/code> class uses all-mpnet-base-v2<\/a> to generate embeddings, and we’ll use the simple FAISS vector database<\/a> from Meta.<\/p>\n\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n\n\n<\/div>\n\nIn [ ]:<\/div>\n\n\n<\/span> - Jupyter Notebook: babyagi-without-openai-guidance.ipynb<\/a><\/em><\/li>\n<\/ul>\n