How To Train GPT On Excel Data For Free? (Beta)

A guide to adding Excel data to your Cody knowledge base and training ChatGPT for free.

Before you start training Cody on your company’s Excel data, it is necessary to clarify a few concepts to ensure the best responses from your bot. GPT, or Generative Pre-Trained Transformers, are language models trained on extensive datasets to predict the next word in a sentence or phrase in order to complete it. They are specifically trained on natural language datasets comprising large samples of unstructured conversational or literal data. Unlike statistical models such as Linear Regression, GPTs are not proficient in predicting numbers using logical training data. For example, if you train GPT on a dataset that claims 2+2=5, it will respond by stating that 2+2=5 without attempting to understand the logical inconsistency (this is just an example; OpenAI does handle such queries with accurate responses). This, coupled with another limitation of LLMs, which is hallucinations, creates an environment that is not well-suited for mathematical calculations.

Step 1: Transforming the Excel Data

Grab the CSV or Excel Data that you want to train your Bot on and convert them into a text file using this utility created by us. The utility converts the excel data into a text-file by annotating the data with their corresponding headers. By annotating the cell items with headers, lets the language model comprehend the context better since there is a high probability of the headers getting skipped due to document segmentation in the pre-processing stage.

Eg.

Excel Data:

Text Data:

{The Name is ‘John’. The Age is ‘16’.}, {The Name is ‘Marie’. The Age is ‘18’.}

The generated text file follows a format similar to JSON but with a more literary style to provide a more human-like feel. Although this solution is currently in an experimental stage and not yet integrated into the Cody app, it works well with all three GPT models but we are continuously exploring better solutions for this purpose.

Utility Interface:

Sample CSV data:

It is recommended that you clean the data before transformation to get the best quality of responses from your bot.

After uploading the CSV or Excel data to the utility, you can preview the data before generating the GPT-compatible text file.

Rows Per Part: For larger datasets, it is advisable to divide the dataset into multiple parts. This division improves semantic search and enhances the quality of responses.

Include Cell References: If you want the text file to include Excel cell references, you can select this option. The bot can then refer to these cell references when creating step-by-step guides for actions that can be performed in Excel. For example, it can generate a formula to find the median.

A compressed zip folder will be generating that will contain all the parts of your excel data in .txt format.

Step 2: Adding the Data to your Cody Knowledge Base

To add the transformed data to the Cody Knowledge Base, follow these steps:

1. Go to the Cody application and navigate to the “Content” section.
2. Create a new folder within the knowledge base where you want to store the data.
3. Once the folder is created, navigate inside it.
4. Click on the “Upload” button to upload the transformed data.
5. Select all the transformed data files from your local storage that you want to add to the knowledge base.
6. Confirm the selection and initiate the upload process.
7. The transformed data files will be uploaded and added to the Cody Knowledge Base, stored within the folder you created. After the documents have been successfully learned, the document status will be displayed as ‘learned’.

Step 3: Setting up the Bot Personality

As this is still in an experimental stage, we are working on improving the prompt before we add it to the template mode.

Prompt:

You are Data Cody, an AI Data Analyst for my company. Your primary objective is to generate inferences from the Excel data provided to you. The Excel Cell references may be given in the form of $Cell. Do not mention the cell reference in responses. The information contained within ‘{}’ is one record. If asked for the details of a specific record, list them out in pointers.

System Prompt:

Try to respond in a human-like way when asked about any detail. Don’t justify your answers.

This process works well with all three GPT models, so even if you are on the free plan, you can give it a try. However, it’s worth noting that GPT-3.5 16K and GPT-4 models tend to comprehend the data better. If you’re satisfied with the answers you receive on the free plan but want more flexibility in formatting the responses and the ability to compare multiple records, upgrading to GPT-3.5 16K or GPT-4 can be beneficial. The additional context window provided by these models allows for more comprehensive analysis and manipulation of the data.

Demo

Reference for first query:

Reference for second query:

Limitations

The ability to upload Excel or CSV files to Cody does not make it a direct alternative to spreadsheet tools like Google Sheets or Microsoft Excel. There are several limitations to consider when working with structured data in Cody:

1. Hallucinations during Analytical Tasks: Tasks involving statistical or analytical calculations, such as asking Cody for averages, medians, or min/max values, may yield incorrect responses. Cody does not perform real-time calculations and can provide inaccurate results. OpenAI’s recent updates, like the Code Interpreter and function calling, may improve this in the future.
2. Error While Comparing Records: In certain cases, Cody may encounter difficulties fetching data from different segments of the document, resulting in responses indicating that the information is unavailable. This scenario is more likely with the GPT-3.5 model available in the free plan. Upgrading to the Basic or Premium plans allows you to use the GPT-3.5 16K model or the GPT-4 model. Both of these models have larger context windows and can potentially address this limitation.

Conclusion

Despite these limitations, this process is particularly useful for scenarios where your business FAQ data or other literal data, such as employee training data, is stored in Excel or CSV format. Cody can be trained on this data without requiring any modifications. Cody also performs well when fetching details of a single record, describing the data, or providing suggestions based on inferred insights from numerical datasets like balance sheets or sales figures.

As an interim solution for training Cody on Excel or CSV data, we greatly appreciate your feedback on this approach. We value your input and encourage you to share your thoughts with us on our Discord Server or by reaching out to us through the Get Help feature. We are eager to hear about your experience and learn more from your feedback. Hope you liked our approach of training GPT on Excel data for free. Check out our blogs to know more about Cody.