When you get started with Cody, it is possible that you may feel disappointed or disheartened about why Cody is unable to generate the expected responses. In this short blog, we won’t delve deep into how you should be using Cody, but we will give you a rough idea of how Cody uses your documents to generate responses so that you can better understand the generation process and experiment with it.
Two main factors mainly affect response generation using your documents:
Chunking
Context Window
Both of these terminologies, chunking and the context window, are interrelated. A simple analogy can be drawn to compare response generation with cooking food. Chunks can be seen as the individual pieces of vegetables that you cut, while the context window represents the size of the cooking utensil. It is important to cut the vegetables into optimal-sized chunks to enhance the overall taste, and a larger utensil allows for more vegetable pieces to be added.
What is Chunking?
In simple terms, chunking is the act of breaking down content into manageable pieces for efficient use of memory. If you have read our blogs, you may be aware that models such as GPT require significant resources, and to tackle the constraints of the context window, we employ multiple processes like chunking.
Chunking is a process performed after you upload the documents to Cody. It divides or segments the document into multiple chunks, with each chunk containing relevant surrounding context. These chunks are then assigned numerical tags for easier computation, which is known as embedding. It is important to find the optimal chunk size. A smaller chunk size reduces context relevance, while a larger chunk size introduces more noise. Cody’s chunking algorithm dynamically adjusts the chunk size based on the token distribution set by the user.
How does the Context Window affect the Bot responses?
Various factors such as personality prompt, relevance score, etc., influence the quality of bot responses. The context window of the model also plays a significant role in determining the quality. The context window refers to the amount of text that an LLM (Language Model) can process in a single call. Since Cody utilizes embeddings and context injection to generate answers using OpenAI models, a larger context window allows for more data ingestion by the model in each query.
💡 Each Query (≤ Context Window) = Bot Personality + Knowledge Chunks + History + User Input + Response
Context windows of different models:
GPT-3.5: 4096 Tokens (≈3500 words)
GPT-3.5 16K: 16000 Tokens (≈13000 words)
GPT-4: 8000 Tokens (≈7000 words)
When the context window is larger, it enables a greater proportion of each parameter, including Personality, Chunks, History, Input, and Response. This expanded context empowers the bot to generate responses that are more relevant, coherent, and creative in nature.
Cody’s latest addition enables users to check document citations by clicking on the document name at the end of the responses. These citations correspond to the chunks obtained through semantic search. Cody determines the chunk threshold for context based on the relevance score set by the user. If the user sets a high relevance score, Cody only uses chunks that surpass a predefined threshold as context for generating the answer.
Example
Assuming a predefined threshold limit value of 90% for a high relevance score, Cody discards any chunks with a relevance score lower than 90%. We recommend new users start with a lower relevance score (low or balanced), specifically when using uploaded documents (PDFs, Powerpoints, Word, etc) or websites. Uploaded documents or websites may encounter formatting and readability issues during pre-processing, which can result in lower relevance scores. Formatting the document using our built-in text editor instead of uploading raw documents will ensure the highest accuracy and trust score.
If you found this blog interesting and wish to delve deeper into the concepts of context window and chunking, we highly recommend reading this blog written by Kristian from All About AI. For more resources, you can also check out our Help Center and join our Discord community.
A step-by-step guide to building your own AI to meet your business needs.
The applications of bots are vast, and with the rise in bot utilization of generative models, it can be tempting to jump on the bandwagon and build your very own ChatGPT. With the launch of Cody, building AI chatbots has never been easier before. The model-agnostic and no-code interface of Cody tries to make personal AI accessible to every individual and business domain that exists. However, although Cody strives to make the entire process as straightforward as possible, it can be a little intimidating for someone who is new to the world of AI. That’s why, in this blog, we provide an end-to-end explanation of how to build your own ChatGPT using Cody. We will walk you through a use case, helping you kickstart your Cody journey with confidence.
The Use Case
We will be building an AI chatbot using Cody that will act as a tour guide. This will be a valuable addition for tourism-related businesses, as it can greatly improve the customer experience. But, the scope of building a chatbot with Cody is not limited to just one business domain. Users can experiment with different prompts to build a bot catering to their own business and also share it with their customers after reading this blog.
Requirements
Your business-related content (PDF, Word, Website or Text)
Account on Cody (Recommended Plan: Premium)
15 minutes
Step 1: Adding data to the knowledge base
To add documents to train Cody for your business needs, follow these steps:
Go to the Content section of Cody’s interface.
Create a new folder to organize your documents.
There are three methods available to add data to the knowledge base:
Write: Utilize the built-in text editor to directly input content.
Upload: Upload PDF, Word, or PowerPoint files containing relevant information.
Import Website: This feature allows you to crawl your business website and automatically collect data. (Note: This feature is available on Premium and Advanced Plans only.)
For this specific tutorial, we will use a Boston guidebook that includes information about various tourist attractions, their descriptions, hours of operation, addresses and frequently asked questions (FAQs).
After uploading the necessary documents, you will be able to track their status, which can be either “learning” or “learned.” The status indicates whether Cody is currently in the process of learning from the documents or if it has successfully completed the learning process. Once Cody has learned the uploaded documents, you can proceed to the next step, which involves building the actual chatbot.
Note: It is crucial to ensure that the documents you provide follow a specific format that enhances the quality of responses generated by Cody. To learn more about the recommended document formatting, we recommend reading the documentation provided on formatting guidelines.
Step 2: Creating the Bot’s Purpose
The purpose is further divided into 3 parts:
Personality
The Personality acts as the brain of the bot and plays an essential role in generating effective and useful responses. It is made up of multiple parameters such as Prompt, Relevance Score, Token Distribution, Prompt Persistence and Reverse Vector Search. For a detailed explanation of these terminologies, we recommend referring to our previous blog, which provides comprehensive insights for those new to Cody. Cody provides two options to the user for creating the personality: Template Mode and Advanced Mode.
Template Mode provides users with pre-defined prompts, offering a convenient plug-and-play solution. For this tutorial and use-case, we will be limiting ourselves to using the Advanced Mode since the use-case is not currently covered in the Template Mode. In the future, we plan to create a library of prompts with their specific use cases, providing even more options for users and also adding more presets to the Template Mode.
Personality for the AI tour guide:
Prompt
You are Cody Travel Guide, an AI Assistant dedicated to providing accurate information on Boston. Your primary task is to assist me by providing me with reliable and clear responses to my questions, based on the information available in the knowledge base as your only source. Refrain from mentioning ‘knowledge base’ or file names during the conversation. You are reluctant to make any claims unless they are stated or supported by the knowledge base. When a definitive answer is unavailable, acknowledge your inability to answer and inform me that you cannot respond. Instead of saying ‘the information is based on my knowledge base’ just say ‘Excuse me if I missed out anything’.
Try to use the following format to display the information:
<h2> Name of Place </h2>
<strong> Description: </strong> <br>
<strong> Address: </strong> <br>
<strong> Working Hours: </strong> <br>
<strong> Additional Information: </strong> <br>
Go to the <a href=’Insert link’>website</a> to know more.
If asked for custom tour plan or itinerary, create one using the timings and description provided in this format:
💡 If you want to format your bot responses in a certain way, Cody supports Markdown and HTML tags, where you can define a formatting template for your bot.
Every personality prompt must follow a specific structure to get quality responses. Important tips for engineering an effective prompt with examples:
→ Define the fallbacks:
“Instead of saying ‘that the information is based on my knowledge base’ just say ‘Excuse me if I missed out anything’.”
“If you cannot find relevant information in the knowledge base, acknowledge your inability and suggest contacting [Enter your website URL/contact no.]”
→ Delimit the sources:
“Use the knowledge base as your only source.”
“Your response must be in the same language as my request, and should not mention the word knowledge base during conversation.”
→ Define the goal:
“Your goal is to provide me with assistance and answer my questions about [Enter your business domain].”
“Avoid unrelated activities or engaging in non-knowledge base related discussions or creative writing.”
Sample boilerplate prompt for customer support:
📌 You are Cody, a Website Support AI Assistant who works for Villa Homes. Your goal is to provide me with support and answering my questions about Villa Homes, using the knowledge base as your only source.
Avoiding unrelated activities or engaging in non-knowledge base related discussions or creative writing. If you cannot find relevant information in the knowledge base or if the user asks non-related questions that are not part of the knowledge base, acknowledge your inability and suggest that I contact Villa Homes’s team at +40XXXXXXX
Your response must be in the same language as my request, and should not mention the word knowledge base during conversation.
To know more about engineering better prompts, read this blog where we explain all the fundamentals of Cody in detail.
General
In this section, you will be required to define the name and description of your bot, as well as select the model you want to use with Cody. You have three different OpenAI models to choose from:
GPT-3.5
GPT-3.5 16K
GPT-4
With the Premium Plan, you have access to all three models, providing you with the freedom to choose the most suitable one for your specific needs. It offers a practical choice, allowing you to leverage the full range of capabilities offered by Cody.
We will be proceeding with GPT-4 since it is the most advanced model currently available and perfectly suited for the use-case we will be demonstrating due to its enhanced creativity.
Knowledge
The folder created in the first step should be linked to your specific bot in this section. If you have created multiple folders that you want to link, simply select all the folders you would like to train Cody on.
It is important to understand that Cody is not a black box that has a mind of its own. It is a model that simply predicts the next word in the sentence. As the famous saying in data science goes, “Garbage In, Garbage Out.” The better you set the personality, define the rules of the bot, and clean the data you provide, the better responses it will generate. With newer additions such as Template Mode with various presets, we are trying to make the personality prompting process a lot easier for users.
Step 3: Testing and Sharing the Bot
Now comes the most exciting part! Trying out and testing the bot you’ve built yourself can give you a great sense of achievement. There are two ways to test and share your bot: creating a chat or using the newly launched customizable widget.
To create a chat widget in just three simple steps:
Go to the Bots section of Cody’s interface.
Select the bot you created and click on the three dots ‘⋮’ for more options.
Click on ‘Embed‘.
And voila! You have successfully created a customizable Cody Widget.
When using a Cody Widget, you have two options to share it with others: Share Link or Embed. If you don’t have a website for your business or if you’re not comfortable with modifying your website code, you can easily share the bot with your customers using the provided link.
There are two types of embeddable widgets available:
Inline Embed: This type of widget will be embedded within a specified space on your website. It will occupy the necessary space as required by the widget within the defined main element.
Pop-Up Embed: This type of widget appears as a floating icon on your website. When a visitor clicks on the icon, the widget expands and opens up, allowing for interaction with the chatbot.
Both types of embeddable widgets provide different ways to integrate the Cody bot into your website, offering flexibility in terms of space utilization and user experience. You can choose the one that best suits your website design and user interaction preferences.
Customizing the widget
Our latest feature release offers users complete freedom and customization options for widgets. This means you can tailor the widget’s appearance, behavior, and overall user experience to align with your specific requirements and branding.
Customization options include:
Header
Change the layout of header (Left or Center).
Add your business logo
Color
Title
Subtitle
Chat
Message size (Chat bubble size)
Background color of the chat
Bot
Initial Messages
Message background color
Bot Avatar
Bot Colour
Human
Suggested questions
Message background color
Composer
Placeholder message
Send button icon
Cody branding (Can be removed only in Premium and Advanced Plans)
Launcher
Size
Screen Position
Background Color (Floating button color)
Icon
Close Icon
All these customization options should make Cody more personalized and in line with your business aesthetics resulting in a much improved user-experience for your customers.
That’s about it!
Building and sharing a bot using Cody is now easier than ever, requiring just three simple steps. With the recent introduction of widget customizability options, there’s never been a better time to create your own bot using Cody. The added flexibility in customizing the widget allows you to tailor the bot’s appearance and behavior to match your specific business needs and branding.
We are constantly working on developing and introducing more exciting features to enhance the Cody experience. Our team is dedicated to making the entire process of building and deploying bots even more seamless and user-friendly. Stay tuned for upcoming updates as we continue to improve and refine the Cody platform. For more resources, you can also check out our Help Center and join our Discord community.
Should you upgrade to the paid version of Cody? Here’s why you might want to.
A few days ago, we released a newer model for all our paid users right after the OpenAI release: GPT-3.5 16k. As intimidating as it might sound, it could be a game-changer for your business. In this blog, we will delve into the use cases of GPT-3.5 16k, explore its advantages, and highlight how it differs from the existing GPT-3.5 model and the latest higher-end GPT-4.
What is GPT-3.5 16K?
If you have used Cody’s free version before, you might already be familiar with the vanilla GPT-3.5 model, which utilizes OpenAI’s gpt-3.5-turbo model. This model is the popular choice for many users as it offers affordability, speed and reliability in most cases. On the other hand, the GPT-3.5-16k utilizes OpenAI’s gpt-3.5-turbo-16k model, which is an extension of the gpt-3.5-turbo. The significant difference lies in the ’16k’ aspect.
What is 16K?
The ’16K’ suffix indicates that the model has a context window of 16,000 tokens, a significant increase from the existing 4,096 tokens. In our previous blog, we explained what tokens are in detail. A smaller context window in models can result in several limitations, including:
Lack of relevance: With a limited context window, the model may struggle to capture and maintain relevance to the broader context of a conversation or task.
Inability to maintain context: A smaller context window can make it challenging for the model to remember and reference information from earlier parts of a conversation, leading to inconsistencies and difficulties in maintaining a coherent dialogue.
Constraints on input-query lengths: Shorter context windows impose constraints on the length of input queries, making it difficult to provide comprehensive information or ask complex questions.
Knowledge-base context limitations: A smaller context window may face limitations in incorporating knowledge from relevant documents from the knowledge due to the data ingestion limit.
Advantages of a bigger context window
A question might arise in the minds of a few people: How is GPT-3.5 able to process over 1000 web pages and documents on Cody, despite its capacity of only 4096 tokens? With advancements in the field of generative AI, providing context does not mean simply feeding the entire document as is to language models like GPT-3.5 Turbo. Backend processes such as chunking, embeddings, and vector databases are utilized to preprocess the data, maintaining relevance within the chunks and allowing the model to navigate the predefined context window.
In the current scenario, a larger context window would enhance the AI’s overall performance by accommodating larger and more complex inputs, while reducing the number of vector-storage transactions required to generate a response. Since the context window encompasses both the input and output, a larger window would also enable the model to provide elaborate and coherent responses while maintaining the conversational context.
A larger context window would also help mitigate any hallucinations that may occur when the token limit in a conversation is exceeded.
GPT-3.5 Turbo 16K v/s GPT-4
Although gpt-3.5-turbo-16k is the latest release from OpenAI, gpt-4 still outshines it in various aspects such as understanding visual context, improved creativity, coherence, and multilingual performance. The only area where GPT-3.5-16k excels is the context window, as GPT-4 is currently available in the 8k variant, with the 32k variant still being gradually rolled out.
In the meantime, until the 32k version of gpt-4 becomes widely accessible, the GPT-3.5-16k stands out with its larger context window. If you’re specifically looking for a model that offers a more extensive context window, the GPT-3.5-16k is the ideal choice.
Use cases of a bigger context window
Customer Support: A larger context window enhances the short-term memory of the model, making it well-suited for applications involving customer support, form-filling, and user data collection. It enables the model to maintain context over a longer period, leading to more relevant responses to user inputs such as names, customer IDs, complaints, and feedback.
Employee Training: Leveraging Cody for employee training purposes proves highly effective. Employee training often involves extensive data related to business activities, steps, and processes. To maintain contextual relevance throughout the training program, it becomes necessary to incorporate the entire conversation history of the trainee. A larger context window allows for the inclusion of more historical information, facilitating a more comprehensive and effective training experience.
Data Analysis: Tasks involving financial analysis and statistical inference often require processing large volumes of data to derive meaningful insights. With a bigger context window, the model can hold more relevant information during computation, resulting in a more coherent and accurate analysis. For instance, comparing balance sheets and the overall performance of a company year over year can be executed more effectively with a larger context window.
Comparison between GPT-3.5 4K v/s 16K
To demonstrate the improvements on the 16K model, we queried a .csv file of the Periodic Table containing 118 elements and their features.
From the comparison, it is visible that GPT-3.5 4K was unable to produce all the radioactive elements in its response and skipped some elements. In contrast, GPT-3.5 16K produced almost all the radioactive elements present in the provided table. This demonstrates the improved elaboration of responses due to the bigger context window. This was just a small glimpse into the potential the 16k context window holds with there being endless applications and implementations of the same. With GPT-4 32K in the pipeline, the 16K model can facilitate a smoother transition to a bigger context window.
Should you upgrade?
The larger context window is undoubtedly a significant update and not merely a gimmick. Improved context understanding plays a crucial role in enhancing the quality of responses, and a larger context window unlocks substantial potential for these Language Model Models (LLMs). By allowing for a more extensive grasp of the conversation history and contextual cues, LLMs can deliver more accurate and contextually appropriate outputs.
As mentioned earlier, the 16K variant of GPT-3.5 is available to all users starting from the Basic Plan. If you have been using the Personal (free) plan for some time, you would have already experienced the capabilities offered by Cody. The Basic Plan provides excellent value for money, particularly if you don’t require the additional features of GPT-4. It is suitable for individuals who are building a bot as a project or a prototype for their business with the added model choice of GPT-3.5 16K. In the future, when we release the GPT-4 32K variant, you can always upgrade to the premium plan when the need arises for more tokens.
For larger enterprises, the Advanced Plan is the most powerful option, catering to resource-intensive and high-volume usage requirements. It offers comprehensive capabilities to meet the demands of large-scale operations.
A beginner’s guide to understanding Cody Terminologies in a simplified manner.
Before you get started with building your own GPT-based bot for your business using Cody, it is essential that you understand the core terminologies so that the flexibility our bot-builder offers does not intimidate you. You do not need to be an AI expert or have prior knowledge in this domain to understand these terminologies, as they will be explained in the most simplified way. Think of this as a beginner’s guide or ‘Cody For Dummies‘.
This blog will not cover setting up Cody or importing data into Cody since we have already posted a few blogs that will help you with the setup. In the upcoming blogs, we will delve deeper into those subjects. Without any further ado, let’s understand the different terminologies you will come across while building your bot.
Personality
Cody offers two ways to define your bot’s personality: Template Mode and Advanced Mode. The personality of your bot establishes the context and behaviour of your bot to ensure it generates helpful responses that align with your expectations and prevent any unforeseen hallucinations.
❓Hallucination refers to the phenomenon where a language model generates responses that may be imaginative, speculative, or not grounded in factual information.
Template Mode
This mode is useful for those businesses who want a plug-and-play solution to building a bot. Our developers have created bot templates catering to different business domains such as Support, Training, HR and Marketing.
Select any domain from the following and Cody will handle the rest. For added flexibility, Cody also provides settings that can be used to customize the personality to some extent.
For eg. If you choose to go ahead with IT Support as your bot personality, you can change your bot’s nickname, and decide how it will be handling unanswerable queries (a default case) and the languages you need the bot to converse in.
This mode is still under development and will be soon getting updated to be able to cater to multiple other business domains with a host of customization features.
Need more flexibility? Don’t worry, we have got you covered with our Advanced Mode.
Advanced Mode
The Advanced Bot Builder gives you the freedom to customize multiple parameters and lets you build the perfect bot that suits your use case. Currently, you can customize the following parameters:
Prompt
Relevance Score
Token Distribution
Persistent Prompt
Reverse Vector Search
Prompt
The prompt will define the personality of your bot. To simplify the prompting process, consider the bot as an employee at your business. Although there is no specific structure for writing a personality prompt, we have prepared a list of parameters for your reference.
Parameter
Example
AI’s Nickname
“You are Cody.”
Job Profile
“You work as an IT Support Engineer at my company.”
Tone
“Maintain a professional and friendly demeanour throughout all interactions, ensuring that users feel comfortable and supported.”
Domain
“Technology, Support, Technology Consultant”
Expertise Level
“Remember to convey a sense of expertise and confidence in your responses.”
Ethical Guidelines
“Do not make up your own responses or use data which does not exist in the training data. Refrain from mentioning ‘unstructured knowledge base’ or file names during the conversation.”
Additional Services (optional)
“Try to promote sales of XYZ wherever possible.”
Default Case (For unanswerable queries)
“In instances where a definitive answer is unavailable, acknowledge your inability to answer and inform the user that you cannot respond.”
A sample prompt:
“You are the waiter at a pizza joint. Maintain a professional and friendly demeanor throughout all interactions, ensuring that users feel comfortable and supported. Remember to convey a sense of expertise and confidence in your responses. Additionally, I encourage you to actively promote our premium pizzas whenever appropriate. Do not refer to any menu sources other than the ones provided in the knowledge base. While recommending pizzas, state their prices and any offers that are applicable for them, too.”
Relevance Score
The Relevance Score reflects the degree of similarity between the user’s query and Cody’s response. Using semantic search, Cody compares the user’s query with the data present in the knowledge base. A higher relevance score will result in a precise answer but will compromise on understanding the overall context of the query and vice versa. In simple words, the relevance score is the degree to which the AI fears making mistakes and taking risks while responding.
Token Distribution
The token is the computational currency for large language models like the GPT family. The query (input statement) asked by the user is broken down into blocks of characters known as ‘tokens’. As AI models are really resource-intensive, in order to address the computational constraints and memory limitations, these models have a certain limit to the input data that can be processed and generated. This limit is the ‘context window’.
Cody uses the GPT family of models and the number of tokens available are limited. The token distribution feature helps in micro-managing the usage of tokens for different purposes.
They are mainly divided into Context, History and Response Generation.
Context: The tokens required for understanding the user query and knowledge base context.
History: The tokens required for adding context to the user query using the chat history.
Response Generation: The tokens required for assessing the coherence, grammar, and semantic validity of the generated text.
For the highest accuracy is important that the context makes up a large portion of the Token Distribution.
Persistent Prompt
By continuously reinforcing the prompt (personality of the bot), you create a form of conversational context and constraint that keeps the AI on track and helps maintain compliance with the desired outcomes. It acts as a reminder for the AI to stay within the predefined boundaries and provide responses that are relevant, accurate, and aligned with your goals.
Reverse Vector Search
Reverse Vector Search is a feature specifically designed for businesses that utilize Cody for employee training purposes. It seamlessly integrates previously generated AI responses with user queries, creating a comprehensive contextual understanding that enhances the quality and relevance of subsequent responses. It is especially valuable when using Cody for step-by-step guides and instructional purposes.
To Wrap It Up
When you have a good grasp of these terminologies, you’ll be able to refine your bot’s responses and make them even more personalized. Stay tuned for upcoming blogs offering further guidance, updates, and additional information to unleash your Cody AI bot’s full potential. For more resources, you can also check out our Help Center and join our Discord community.
