Before you start training Cody on your company’s Excel data, it is necessary to clarify a few concepts to ensure the best responses from your bot. GPT, or Generative Pre-Trained Transformers, are language models trained on extensive datasets to predict the next word in a sentence or phrase in order to complete it. They are specifically trained on natural language datasets comprising large samples of unstructured conversational or literal data. Unlike statistical models such as Linear Regression, GPTs are not proficient in predicting numbers using logical training data. For example, if you train GPT on a dataset that claims 2+2=5, it will respond by stating that 2+2=5 without attempting to understand the logical inconsistency (this is just an example; OpenAI does handle such queries with accurate responses). This, coupled with another limitation of LLMs, which is hallucinations, creates an environment that is not well-suited for mathematical calculations.

Now that you understand the limitations of GPT, let us guide you through a process of training GPT on Excel data for free. We have developed a method to add Excel or CSV data to your Cody knowledge base. As mentioned earlier, GPT excels at understanding natural language, so we will convert the Excel data into a readable format that can be easily consumed by the language model.

## Step 1: Transforming the Excel Data

Grab the CSV or Excel Data that you want to train your Bot on and convert them into a text file using this utility created by us. The utility converts the excel data into a text-file by annotating the data with their corresponding headers. By annotating the cell items with headers, lets the language model comprehend the context better since there is a high probability of the headers getting skipped due to document segmentation in the pre-processing stage.

Eg.

Excel Data:

Text Data:

`{The Name is ‘John’. The Age is ‘16’.}, {The Name is ‘Marie’. The Age is ‘18’.}`

The generated text file follows a format similar to JSON but with a more literary style to provide a more human-like feel. Although this solution is currently in an experimental stage and not yet integrated into the Cody app, it works well with all three GPT models but we are continuously exploring better solutions for this purpose.

Utility Interface:

Sample CSV data:

After uploading the CSV or Excel data to the utility, you can preview the data before generating the GPT-compatible text file.

Rows Per Part: For larger datasets, it is advisable to divide the dataset into multiple parts. This division improves semantic search and enhances the quality of responses.

Include Cell References: If you want the text file to include Excel cell references, you can select this option. The bot can then refer to these cell references when creating step-by-step guides for actions that can be performed in Excel. For example, it can generate a formula to find the median.

A compressed zip folder will be generating that will contain all the parts of your excel data in .txt format.

To add the transformed data to the Cody Knowledge Base, follow these steps:

1. Go to the Cody application and navigate to the “Content” section.
2. Create a new folder within the knowledge base where you want to store the data.
3. Once the folder is created, navigate inside it.
5. Select all the transformed data files from your local storage that you want to add to the knowledge base.
6. Confirm the selection and initiate the upload process.
7. The transformed data files will be uploaded and added to the Cody Knowledge Base, stored within the folder you created. After the documents have been successfully learned, the document status will be displayed as ‘learned’.

## Step 3: Setting up the Bot Personality

As this is still in an experimental stage, we are working on improving the prompt before we add it to the template mode.

Prompt:

You are Data Cody, an AI Data Analyst for my company. Your primary objective is to generate inferences from the Excel data provided to you. The Excel Cell references may be given in the form of \$Cell. Do not mention the cell reference in responses. The information contained within ‘{}’ is one record. If asked for the details of a specific record, list them out in pointers.

System Prompt:

This process works well with all three GPT models, so even if you are on the free plan, you can give it a try. However, it’s worth noting that GPT-3.5 16K and GPT-4 models tend to comprehend the data better. If you’re satisfied with the answers you receive on the free plan but want more flexibility in formatting the responses and the ability to compare multiple records, upgrading to GPT-3.5 16K or GPT-4 can be beneficial. The additional context window provided by these models allows for more comprehensive analysis and manipulation of the data.

## Demo

Reference for first query:

Reference for second query:

## Limitations

The ability to upload Excel or CSV files to Cody does not make it a direct alternative to spreadsheet tools like Google Sheets or Microsoft Excel. There are several limitations to consider when working with structured data in Cody:

1. Hallucinations during Analytical Tasks: Tasks involving statistical or analytical calculations, such as asking Cody for averages, medians, or min/max values, may yield incorrect responses. Cody does not perform real-time calculations and can provide inaccurate results. OpenAI’s recent updates, like the Code Interpreter and function calling, may improve this in the future.
2. Error While Comparing Records: In certain cases, Cody may encounter difficulties fetching data from different segments of the document, resulting in responses indicating that the information is unavailable. This scenario is more likely with the GPT-3.5 model available in the free plan. Upgrading to the Basic or Premium plans allows you to use the GPT-3.5 16K model or the GPT-4 model. Both of these models have larger context windows and can potentially address this limitation.

## Conclusion

Despite these limitations, this process is particularly useful for scenarios where your business FAQ data or other literal data, such as employee training data, is stored in Excel or CSV format. Cody can be trained on this data without requiring any modifications. Cody also performs well when fetching details of a single record, describing the data, or providing suggestions based on inferred insights from numerical datasets like balance sheets or sales figures.

Om Kamath

## More From Our Blog

### Anthropic's Claude 3.5 Sonnet Released: Better Than GPT-4o?

Claude 3.5 Sonnet is the latest model in the Claude 3.5 family of large language models (LLMs). Introduced by Anthropic in March 2024, it marks a significant leap forward. This model surpasses its predecessors and notable competitors like GPT-4o and ...