openai_fine_tuning fix for asyncio (#12667)

* Add files via upload Adding nest_asyncio.apply() to resolve RuntimeError: asyncio.run() cannot be called from a running event loop * Delete experimental/openai_fine_tuning/openai_fine_tuning-2.ipynb * Update openai_fine_tuning.ipynb with asyncio to fix runtime error

openai_fine_tuning fix for asyncio (#12667)
ca0d87a2 · Eleni Verteouri · GitHub · 029f7433 · ca0d87a2
Unverified Commit ca0d87a2 authored 11 months ago by Eleni Verteouri Committed by GitHub 11 months ago
--- a/experimental/openai_fine_tuning/openai_fine_tuning.ipynb
+++ b/experimental/openai_fine_tuning/openai_fine_tuning.ipynb
@@ -131,6 +131,17 @@
    ")"
   ]
  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import nest_asyncio\n",
+    "\n",
+    "nest_asyncio.apply()"
+   ]
+  },
  {
   "cell_type": "code",
   "execution_count": null,

 %% Cell type:markdown id: tags:

 # Fine Tuning GPT-3.5-Turbo

 In this notebook, we walk through an example of fine-tuning gpt-3.5-turbo.

 Specifically, we attempt to distill GPT-4's knowledge, by generating training data with GPT-4 to then fine-tune GPT-3.5.

 All training data is generated using two different sections of our index data, creating both a training and evalution set.

 Evaluation is done using the `ragas` library, which we will detail later on.

 %% Cell type:code id: tags:

 ``` python
 # !pip install llama-index pypdf sentence-transformers ragas
 ```

 %% Cell type:code id: tags:

 ``` python
 import os
 import openai

 os.environ["OPENAI_API_KEY"] = "sk-..."
 openai.api_key = os.environ["OPENAI_API_KEY"]
 ```

 %% Cell type:markdown id: tags:

 ## Data Setup

 Here, we first down load the PDF that we will use to generate training data.

 %% Cell type:code id: tags:

 ``` python
 !curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf
 ```

 %% Output

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 20.7M  100 20.7M    0     0  16.7M      0  0:00:01  0:00:01 --:--:-- 16.8M

 %% Cell type:markdown id: tags:

 The next step is generating a training and eval dataset.

 We will generate 40 questions on different sections of the PDF we downloaded.

 We can use GPT-3.5 on the eval questions to get our baseline performance.

 Then, we will use GPT-4 on the train questions to generate our training data. The training data will be collected with out `OpenAIFineTuningHandler`.

 This step is entirely optional if you don't want to spend the time/tokens -- the eval and training questions are also provided in this folder, as well as the training data!

 %% Cell type:markdown id: tags:

 ### Train Generation

 %% Cell type:code id: tags:

 ``` python
 from llama_index.core import SimpleDirectoryReader
 from llama_index.llms.openai import OpenAI
 from llama_index.core.evaluation import DatasetGenerator

 documents = SimpleDirectoryReader(
    input_files=["IPCC_AR6_WGII_Chapter03.pdf"]
 ).load_data()

 # Shuffle the documents
 import random

 random.seed(42)
 random.shuffle(documents)

 gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)
 ```

 %% Cell type:code id: tags:

 ``` python
 question_gen_query = (
    "You are a Teacher/ Professor. Your task is to setup "
    "a quiz/examination. Using the provided context from a "
    "report on climate change and the oceans, formulate "
    "a single question that captures an important fact from the "
    "context. Restrict the question to the context information provided."
 )

 dataset_generator = DatasetGenerator.from_documents(
    documents[:50],
    question_gen_query=question_gen_query,
    llm=gpt_35_llm,
 )
 ```

 %% Cell type:code id: tags:

 ``` python
+import nest_asyncio
+
+nest_asyncio.apply()
+```
+
+%% Cell type:code id: tags:
+
+``` python
 # NOTE: this may take some time. Go grab a coffee!
 questions = dataset_generator.generate_questions_from_nodes(num=40)
 print("Generated ", len(questions), " questions")
 ```

 %% Output

    Generated  40  questions

 %% Cell type:code id: tags:

 ``` python
 with open("train_questions.txt", "w") as f:
    for question in questions:
        f.write(question + "\n")
 ```

 %% Cell type:markdown id: tags:

 ### Eval Generation

 Now, lets generate questions on a completely different set of documents, in order to create our eval dataset.

 %% Cell type:code id: tags:

 ``` python
 dataset_generator = DatasetGenerator.from_documents(
    documents[
        50:
    ],  # since we generated ~1 question for 40 documents, we can skip the first 40
    question_gen_query=question_gen_query,
    llm=gpt_35_llm,
 )
 ```

 %% Cell type:code id: tags:

 ``` python
 # NOTE: this may take some time. Go grab a coffee!
 questions = dataset_generator.generate_questions_from_nodes(num=40)
 print("Generated ", len(questions), " questions")
 ```

 %% Output

    Generated  40  questions

 %% Cell type:code id: tags:

 ``` python
 with open("eval_questions.txt", "w") as f:
    for question in questions:
        f.write(question + "\n")
 ```

 %% Cell type:markdown id: tags:

 ## Initial Eval with GPT-3.5-Turbo Query Engine

 For this eval, we will be using the [`ragas` evaluation library](https://github.com/explodinggradients/ragas).

 Ragas has a ton of evaluation metrics for RAG pipelines, and you can read about them [here](https://github.com/explodinggradients/ragas/blob/main/docs/metrics.md).

 For this notebook, we will be using the following two metrics

 - `answer_relevancy` - This measures how relevant is the generated answer to the prompt. If the generated answer is incomplete or contains redundant information the score will be low. This is quantified by working out the chance of an LLM generating the given question using the generated answer. Values range (0,1), higher the better.
 - `faithfulness` - This measures the factual consistency of the generated answer against the given context. This is done using a multi step paradigm that includes creation of statements from the generated answer followed by verifying each of these statements against the context. The answer is scaled to (0,1) range. Higher the better.

 %% Cell type:code id: tags:

 ``` python
 questions = []
 with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())
 ```

 %% Cell type:code id: tags:

 ``` python
 from llama_index.core import VectorStoreIndex, Settings

 # limit the context window to 2048 tokens so that refine is used
 Settings.context_window = 2048

 gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)


 index = VectorStoreIndex.from_documents(documents)

 query_engine = index.as_query_engine(similarity_top_k=2, llm=gpt_35_llm)
 ```

 %% Cell type:code id: tags:

 ``` python
 contexts = []
 answers = []

 for question in questions:
    response = query_engine.query(question)
    contexts.append([x.node.get_content() for x in response.source_nodes])
    answers.append(str(response))
 ```

 %% Cell type:code id: tags:

 ``` python
 from datasets import Dataset
 from ragas import evaluate
 from ragas.metrics import answer_relevancy, faithfulness

 ds = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
    }
 )

 result = evaluate(ds, [answer_relevancy, faithfulness])
 print(result)
 ```

 %% Output

    /Users/loganmarkewich/llama_index/llama-index/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm

    evaluating with [answer_relevancy]

      0%|          | 0/3 [00:00<?, ?it/s]Connection error caused failure to post http://localhost:1984/runs  in LangSmith API. Please confirm your LANGCHAIN_ENDPOINT.
     67%|██████▋   | 2/3 [00:43<00:21, 21.71s/it]Connection error caused failure to patch http://localhost:1984/runs/11ab063b-c643-4ebc-b717-9e756d55b7c0  in LangSmith API. Please confirm your LANGCHAIN_ENDPOINT.
    100%|██████████| 3/3 [00:55<00:00, 18.63s/it]

    evaluating with [faithfulness]

    100%|██████████| 3/3 [03:31<00:00, 70.48s/it]

    {'ragas_score': 0.8576, 'answer_relevancy': 0.9778, 'faithfulness': 0.7638}

 %% Cell type:markdown id: tags:

 ## GPT-4 to Collect Training Data

 Here, we use GPT-4 and the `OpenAIFineTuningHandler` to collect data that we want to train on.

 %% Cell type:code id: tags:

 ``` python
 from llama_index.llms.openai import OpenAI
 from llama_index.core.callbacks import OpenAIFineTuningHandler
 from llama_index.core.callbacks import CallbackManager

 finetuning_handler = OpenAIFineTuningHandler()
 callback_manager = CallbackManager([finetuning_handler])

 llm = OpenAI(model="gpt-4", temperature=0.3)

 Settings.callback_manager = (callback_manager,)
 ```

 %% Cell type:code id: tags:

 ``` python
 questions = []
 with open("train_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())
 ```

 %% Cell type:code id: tags:

 ``` python
 from llama_index.core import VectorStoreIndex

 index = VectorStoreIndex.from_documents(documents)

 query_engine = index.as_query_engine(similarity_top_k=2, llm=llm)
 ```

 %% Cell type:code id: tags:

 ``` python
 for question in questions:
    response = query_engine.query(question)
 ```

 %% Cell type:markdown id: tags:

 ## Create Fine-Tuning Data

 Fine-Tuning data must be written as a list of messages in a `.jsonl` file. Using the finetuning-handler, we can easily write the messages to a `.jsonl` file.

 %% Cell type:code id: tags:

 ``` python
 finetuning_handler.save_finetuning_events("finetuning_events.jsonl")
 ```

 %% Output

    Wrote 61 examples to finetuning_events.jsonl

 %% Cell type:markdown id: tags:

 ## Launch Fine-Tuning Job

 %% Cell type:code id: tags:

 ``` python
 !python ./launch_training.py ./finetuning_events.jsonl
 ```

 %% Output

    Num examples: 61
    First example:
    {'role': 'system', 'content': "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines."}
    {'role': 'user', 'content': 'Context information is below.\n---------------------\npage_label: 410\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\nIt is challenging to apply this experimental approach to communities or ecosystems (see Figure \nBox\xa03.1.1).To date, most research on community or ecosystem response to climate-induced drivers has been in large-volume (>10,000 l) \nmesocosms (Riebesell and Gattuso, 2014), or at natural analogues such as CO 2 seeps, in which only one driver (ocean acidification) is \naltered (see (4) in Figure Box\xa03.1.1).Only very recently have two drivers been incorporated into climate-change manipulation studies \nexamining responses of primary producers to secondary consumers (see (5) in Figure Box\xa03.1.1a; Nagelkerken et\xa0al., 2020).Therefore, \n‘natural experiments’ from the geological past (Reddin et\xa0al., 2020) provide insights into how food webs and their constituents respond to \ncomplex change involving multiple drivers.Contemporary observations are occasionally long enough (>50\xa0years) to capture community \nresponses to complex climate change.For example, Brun et\xa0al.(2019) reported a shift in zooplankton community structure in the North \nAtlantic (1960–2014), with major biogeochemical ramifications.Conducting sufficiently long manipulation experiments to study the effect of adaptation on organisms is equally difficult (see Figure \nBox\xa03.1.1b), with much research restricted to multi-year studies of the microevolution of fast-growing (more than one division per day) \nphytoplankton species responding to single drivers (Lohbeck et\xa0al., 2012; Schaum et\xa0al., 2016).In a few experimental evolution studies \n(see (7) in Figure Box\xa03.1.1a; Brennan et\xa0al., 2017), multiple drivers have been used, but none have used communities or ecosystems (see \nFigure Box\xa03.1.1b).Nevertheless, the fossil record provides limited evidence of adaptations to less rapid (relative to present day) climate \nchange (Jackson et\xa0al., 2018).Despite the need to explore ecological or biogeochemical responses to projected future ocean conditions, \nlogistical challenges require that assessments of climate-change impacts at scales larger than mesocosms use large-scale, long-term in \nsitu observational studies (as documented in Section\xa03.4).\n\npage_label: 409\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\n3\n409Oceans and Coastal Ecosystems and Their Services  Chapter 3\nunderlies inhibited thermal adaptation under nitrogen-limited \nconditions (low confidence) (Aranguren-Gassis et\xa0 al., 2019).When \nselection is strong due to unfavourable environmental conditions, \nmicrobial populations can encounter functional and evolutionary \ntrade-offs evidenced by reducing growth rates while increasing \ntolerance and metabolism of reactive oxygen species (Lindberg and \nCollins, 2020).Other trade-offs can be observed in offspring quality \nand number (Lindberg and Collins, 2020).These findings contribute \ntowards a mechanistic framework describing the range of evolutionary \nstrategies in response to multiple drivers (Collins et\xa0al., 2020), but other \nhazards, such as extreme events (e.g., MHWs), still need to be included \nbecause their characteristics may alter the potential for adaptation of \nspecies and populations to climate change (Gruber et\xa0al., 2021).3.3.5 Ecological Response to Multiple Drivers\nAssessing ecological responses to multiple climate-induced drivers \nrequires a combination of approaches, including laboratory- and \nfield-based experiments, field observations (e.g., natural gradients, \nclimate analogues), study of paleo-analogues and the development \nof mechanistic and empirical models (Clapham, 2019; Gissi et\xa0 al., \n2021).Experimental studies of food-web responses are often limited \nto an individual driver, although recent manipulations have used a \nmatrix of >1000-l mesocosms to explore ecological responses to both \nwarming and acidification (see Box\xa0 3.1; Nagelkerken et\xa0 al., 2020).Hence, complementary approaches are needed to indirectly explore \nthe mechanisms underlying ecosystem responses to global climate \nchange (Parmesan et\xa0al., 2013).Observations from time series longer \nthan modes of natural variability (i.e., decades) are essential for \nrevealing and attributing ecological responses to climate change (e.g., \nSection\xa03.4; Barton et\xa0al., 2015b; Brun et\xa0al., 2019).Also, paleorecords \nprovide insights into the influence of multiple drivers on marine \nbiota (Cross-Chapter Box\xa0 PALEO in Chapter\xa0 1; Reddin et\xa0 al., 2020).Specifically, associations between vulnerabilities and traits of marine \nectotherms in laboratory experiments correspond with organismal \nresponses to ancient hyperthermal events (medium confidence) \n(Reddin et\xa0 al., 2020).This corroboration suggests that responses to \nmultiple drivers inferred from the fossil record can help provide insights \ninto the future status of functional groups, and hence food webs, under \nrapid climate change.Multi-species and integrated end-to-end ecosystem models are \npowerful tools to explore and project outcomes to the often-interacting \ncumulative effects of climate change and other anthropogenic drivers \n(Section\xa03.1; Kaplan and Marshall, 2016; Koenigstein et\xa0al., 2016; Peck \nand Pinnegar, 2018; Tittensor et\xa0 al., 2018; Gissi et\xa0 al., 2021).These \nmodels can integrate some aspects of the knowledge accrued from \nmanipulation experiments, paleo- and contemporary observations, help \ntest the relative importance of specific drivers and driver combinations, \nand identify synergistic or antagonistic responses (Koenigstein et\xa0al., \n2016; Payne et\xa0al., 2016; Skogen et\xa0al., 2018; Tittensor et\xa0al., 2018).As these models are associated with wide-ranging uncertainties \n(SM3.2.2; Payne et\xa0 al., 2016; Trolle et\xa0 al., 2019; Heneghan et\xa0 al., \n2021), they cannot be expected to accurately project the trajectories \nof complex marine ecosystems under climate change; hence, they are \nmost useful for assessing overall trends and in particular for providing a plausible envelope of trajectories across a range of assumptions \n(Fulton et\xa0al., 2018; Peck et\xa0al., 2018; Tittensor et\xa0al., 2018).\n---------------------\nGiven the context information and not prior knowledge, answer the query.\nQuery: What are some approaches used to assess ecological responses to multiple climate-induced drivers in the context of climate change and the oceans?\nAnswer: '}
    {'role': 'assistant', 'content': 'Several approaches are used to assess ecological responses to multiple climate-induced drivers. These include laboratory- and field-based experiments, field observations such as natural gradients and climate analogues, the study of paleo-analogues, and the development of mechanistic and empirical models. Experimental studies often focus on individual drivers, but recent manipulations have used large-volume mesocosms to explore ecological responses to both warming and acidification. Observations from time series longer than modes of natural variability are essential for revealing and attributing ecological responses to climate change. Paleorecords also provide insights into the influence of multiple drivers on marine biota. Multi-species and integrated end-to-end ecosystem models are powerful tools to explore and project outcomes to the often-interacting cumulative effects of climate change and other anthropogenic drivers. These models can integrate some aspects of the knowledge accrued from manipulation experiments, paleo- and contemporary observations, help test the relative importance of specific drivers and driver combinations, and identify synergistic or antagonistic responses.'}
    No errors found
    Num examples missing system message: 21
    Num examples missing user message: 0
    
    #### Distribution of num_messages_per_example:
    min / max: 2, 3
    mean / median: 2.6557377049180326, 3.0
    p5 / p95: 2.0, 3.0
    
    #### Distribution of num_total_tokens_per_example:
    min / max: 229, 2011
    mean / median: 1274.27868852459, 1385.0
    p5 / p95: 533.0, 1848.0
    
    #### Distribution of num_assistant_tokens_per_example:
    min / max: 11, 334
    mean / median: 72.36065573770492, 37.0
    p5 / p95: 23.0, 193.0
    
    0 examples may be over the 4096 token limit, they will be truncated during fine-tuning
    Dataset has ~77731 tokens that will be charged for during training
    By default, you'll train for 3 epochs on this dataset
    By default, you'll be charged for ~233193 tokens
    As of Augest 22, 2023, fine-tuning gpt-3.5-turbo is $0.008 / 1K Tokens.
    This means your total cost for training will be $0.621848 per epoch.
    File uploaded...
    Waiting for file to be ready...
    Training job launched. You will be emailed when it's complete.

 %% Cell type:markdown id: tags:

 ## Evaluation

 After some time, your model will be done training!

 The next step is running our fine-tuned model on our eval dataset again to measure any performance increase.

 %% Cell type:code id: tags:

 ``` python
 ft_model_name = "ft:gpt-3.5-turbo-0613:..."
 ```

 %% Cell type:code id: tags:

 ``` python
 from llama_index.llms.openai import OpenAI


 ft_llm = OpenAI(model=ft_model_name, temperature=0.3)
 ```

 %% Cell type:code id: tags:

 ``` python
 questions = []
 with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())
 ```

 %% Cell type:code id: tags:

 ``` python
 from llama_index import VectorStoreIndex

 index = VectorStoreIndex.from_documents(documents)

 query_engine = index.as_query_engine(similarity_top_k=2, llm=ft_llm)
 ```

 %% Cell type:code id: tags:

 ``` python
 contexts = []
 answers = []

 for question in questions:
    response = query_engine.query(question)
    contexts.append([x.node.get_content() for x in response.source_nodes])
    answers.append(str(response))
 ```

 %% Cell type:code id: tags:

 ``` python
 from datasets import Dataset
 from ragas import evaluate
 from ragas.metrics import answer_relevancy, faithfulness

 ds = Dataset.from_dict(
    {
        "question": questions,
        "answer": answers,
        "contexts": contexts,
    }
 )

 result = evaluate(ds, [answer_relevancy, faithfulness])
 print(result)
 ```

 %% Output

    evaluating with [answer_relevancy]

    100%|██████████| 3/3 [00:50<00:00, 16.92s/it]

    evaluating with [faithfulness]

    100%|██████████| 3/3 [03:15<00:00, 65.20s/it]

    {'ragas_score': 0.8845, 'answer_relevancy': 0.9758, 'faithfulness': 0.8088}

 %% Cell type:markdown id: tags:

 ## Exploring Differences

 Let's quickly compare the differences in responses, to demonstrate that fine tuning did indeed change something.

 %% Cell type:code id: tags:

 ``` python
 from llama_index.core import VectorStoreIndex

 index = VectorStoreIndex.from_documents(documents)
 ```

 %% Cell type:code id: tags:

 ``` python
 questions = []
 with open("eval_questions.txt", "r") as f:
    for line in f:
        questions.append(line.strip())
 ```

 %% Cell type:code id: tags:

 ``` python
 print(questions[12])
 ```

 %% Output

    What is a key barrier globally for ocean health, governance, and adaptation to climate change according to the report?

 %% Cell type:markdown id: tags:

 ### Original

 %% Cell type:code id: tags:

 ``` python
 from llama_index.core.response.notebook_utils import display_response
 from llama_index.llms.openai import OpenAI


 gpt_35_llm = OpenAI(model="gpt-3.5-turbo", temperature=0.3)
 ```

 %% Cell type:code id: tags:

 ``` python
 query_engine = index.as_query_engine(llm=gpt_35_llm)

 response = query_engine.query(questions[12])

 display_response(response)
 ```

 %% Output

    **`Final Response:`** According to the report, a key barrier globally for ocean health, governance, and adaptation to climate change is the availability of technology, knowledge, and financial support, as well as existing governance structures.

 %% Cell type:markdown id: tags:

 ### Fine-Tuned

 %% Cell type:code id: tags:

 ``` python
 from llama_index.llms.openai import OpenAI


 ft_llm = OpenAI(model=ft_model_name, temperature=0.3)
 ```

 %% Cell type:code id: tags:

 ``` python
 query_engine = index.as_query_engine(llm=ft_llm)

 response = query_engine.query(questions[12])

 display_response(response)
 ```

 %% Output

    **`Final Response:`** The report identifies a broad range of barriers and limits for adaptation to climate change in ecosystems and human systems. These limitations include the availability of technology, knowledge, and financial support, as well as existing governance structures. Existing ocean-governance structures are already facing multi-dimensional, scale-related challenges because of climate change.

 %% Cell type:markdown id: tags:

 As we can see, the fine-tuned model provides a more thorough response! This lines up with the increased faithfullness score from ragas, since the answer is more representative of the retrieved context.

 %% Cell type:markdown id: tags:

 ## Conclusion

 So, in conclusion, finetuning with only ~61 questions actually helped improve our eval scores!

 **answer_relevancy: 0.9778 -> 0.9758**

 The answer relenvancy appears to be basically unchanged, between models.

 **faithfulness: 0.7638 -> 0.8088**

 The faithfulness appears to have been improved! This mains the anwers given better fuffil the original question that was asked.