Skip to content
Snippets Groups Projects
Unverified Commit 76cb6036 authored by Thierry Moreau's avatar Thierry Moreau Committed by GitHub
Browse files

Merge branch 'main' into tmoreau89/android

parents 1bf33ef5 c82bedda
No related branches found
No related tags found
No related merge requests found
Expand all Hide whitespace changes
Inline Side-by-side
recipes/use_cases/agents/langchain/README.md
+ 34
− 25
View file @ 76cb6036
# LangChain <> Llama3 Cookbooks
LLM agents use [planning, memory, and tools](https://lilianweng.github.io/posts/2023-06-23-agent/) to accomplish tasks.
LLM agents use [planning, memory, and tools](https://lilianweng.github.io/posts/2023-06-23-agent/) to accomplish tasks. Agents can empower Llama 3 with important new capabilities. Here, we will show how to give Llama 3 the ability to perform web search, as well as multi-modality: image generation (text-to-image), image analysis (image-to-text), and voice (text-to-speech) tools!
LangChain offers several different ways to implement agents.
LangChain offers several different ways to implement agents with Llama 3:
(1) Use [AgentExecutor](https://python.langchain.com/docs/modules/agents/quick_start/) with [tool-calling](https://python.langchain.com/docs/integrations/chat/) versions of Llama 3.
(1) `ReAct agent` - Uses [AgentExecutor](https://python.langchain.com/docs/modules/agents/quick_start/) with [tool-calling](https://python.langchain.com/docs/integrations/chat/) versions of Llama 3.
(2) Use [LangGraph](https://python.langchain.com/docs/langgraph), a library from LangChain that can be used to build reliable agents with Llama 3.
(2) `LangGraph tool calling agent` - Uses [LangGraph](https://python.langchain.com/docs/langgraph) with [tool-calling](https://python.langchain.com/docs/integrations/chat/) versions of Llama 3.
(3) `LangGraph custom agent` - Uses [LangGraph](https://python.langchain.com/docs/langgraph) with **any** version of Llama 3 (so long as it supports structured output).
As we move from option (1) to (3) the degree of customization and flexibility increases:
(1) `ReAct agent` using AgentExecutor is a great for getting started quickly with minimal code, but requires a version of Llama 3 with reliable tool-calling, is the least customizable, and uses higher-level AgentExecutor abstraction.
(2) `LangGraph tool calling agent` is more customizable than (1) because the LLM assistant (planning) and tool call (action) nodes are defined by the user, but it still requires a version of Llama 3 with reliable tool-calling.
(3) `LangGraph custom agent` does not require a version of Llama 3 with reliable tool-calling and is the most customizable, but requires the most work to implement.
![langgraph_agent_architectures](https://github.com/rlancemartin/llama-recipes/assets/122662504/5ed2bef0-ae11-4efa-9e88-ab560a4d0022)
---
### AgentExecutor Agent
### `ReAct agent`
AgentExecutor is the runtime for an agent. AgentExecutor calls the agent, executes the actions it chooses, passes the action outputs back to the agent, and repeats.
The AgentExecutor manages the loop of planning, executing tool calls, and processing outputs until an AgentFinish signal is generated, indicating task completion.
Our first notebook, `tool-calling-agent`, shows how to build a [tool calling agent](https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/) with AgentExecutor and Llama 3.
This shows how to build an agent that uses web search, text2image, image2text, and text2speech tools.
---
### LangGraph Agent
### `LangGraph tool calling agent`
[LangGraph](https://python.langchain.com/docs/langgraph) is a library from LangChain that can be used to build reliable agents.
LangGraph can be used to build agents with a few pieces:
- **Planning:** Define a control flow of steps that you want the agent to take (a graph)
- **Memory:** Persist information (graph state) across these steps
- **Tool use:** Modify state at any step
Our second notebook, `langgraph-tool-calling-agent`, shows an alternative to AgentExecutor for building a Llama 3 powered agent.
---
Our second notebook, `langgraph-agent`, shows an alternative way to AgentExecutor to build a Llama 3 powered agent in LangGraph.
### `LangGraph custom agent`
It discusses some of the trade-offs between AgentExecutor and LangGraph.
Our third notebook, `langgraph-custom-agent`, shows how to build a Llama 3 powered agent without reliance on tool-calling.
---
### LangGraph RAG Agent
### `LangGraph RAG Agent`
Our third notebook, `langgraph-rag-agent`, shows how to apply LangGraph to build advanced Llama 3 powered RAG agents that use ideas from 3 papers:
Our fourth notebook, `langgraph-rag-agent`, shows how to apply LangGraph to build a custom Llama 3 powered RAG agent that use ideas from 3 papers:
* Corrective-RAG (CRAG) [paper](https://arxiv.org/pdf/2401.15884.pdf) uses self-grading on retrieved documents and web-search fallback if documents are not relevant.
* Self-RAG [paper](https://arxiv.org/abs/2310.11511) adds self-grading on generations for hallucinations and for ability to answer the question.
* Adaptive RAG [paper](https://arxiv.org/abs/2403.14403) routes queries between different RAG approaches based on their complexity.
We implement each approach as a control flow in LangGraph:
- **Planning:** The sequence of RAG steps (e.g., retrieval, grading, and generation) that we want the agent to take
- **Memory:** All the RAG-related information (input question, retrieved documents, etc) that we want to pass between steps
- **Tool use:** All the tools needed for RAG (e.g., decide web search or vectorstore retrieval based on the question)
- **Planning:** The sequence of RAG steps (e.g., retrieval, grading, and generation) that we want the agent to take.
- **Memory:** All the RAG-related information (input question, retrieved documents, etc) that we want to pass between steps.
- **Tool use:** All the tools needed for RAG (e.g., decide web search or vectorstore retrieval based on the question).
We will build from CRAG (blue, below) to Self-RAG (green) and finally to Adaptive RAG (red):
![Screenshot 2024-05-03 at 10 50 02 AM](https://github.com/rlancemartin/llama-recipes/assets/122662504/ec4aa1cd-3c7e-4cd1-a1e7-7deddc4033a8)
![langgraph_rag_agent_](https://github.com/rlancemartin/llama-recipes/assets/122662504/ec4aa1cd-3c7e-4cd1-a1e7-7deddc4033a8)
---
### `Local LangGraph RAG Agent`
### Local LangGraph RAG Agent
Our fourth notebook, `langgraph-rag-agent-local`, shows how to apply LangGraph to build advanced RAG agents using Llama 3 that run locally and reliably.
Our fifth notebook, `langgraph-rag-agent-local`, shows how to apply LangGraph to build advanced RAG agents using Llama 3 that run locally and reliably.
See this [video overview](https://www.youtube.com/watch?v=sgnrL7yo1TE) for more detail.
See this [video overview](https://www.youtube.com/watch?v=sgnrL7yo1TE) for more detail on the design of this agent.
recipes/use_cases/agents/langchain/langgraph-agent.ipynb → recipes/use_cases/agents/langchain/langgraph-custom-agent.ipynb
+ 54
− 43
View file @ 76cb6036
Source diff could not be displayed: it is too large. Options to address this: view the blob.
recipes/use_cases/agents/langchain/langgraph-tool-calling-agent.ipynb 0 → 100644
+ 831
− 0
View file @ 76cb6036
This diff is collapsed.
recipes/use_cases/agents/langchain/tool-calling-agent.ipynb
+ 26
− 49
View file @ 76cb6036
%% Cell type:markdown id:9856250d-51bb-49d3-962d-483178cfafab tags:
 
<a href="https://colab.research.google.com/github/meta-llama/llama-recipes/blob/main/recipes/use_cases/agents/langchain/tool-calling-agent.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
 
%% Cell type:code id:158a3f92-139a-4e92-9273-e018b027fc54 tags:
 
``` python
! pip install -U langchain_groq langchain tavily-python replicate
```
 
%% Cell type:markdown id:745f7d9f-15c4-41c8-94f8-09e1426581cc tags:
%% Cell type:markdown id:3fcd455a-6255-4c7c-9e8e-2aaf1b323c05 tags:
 
# Tool calling agent with Llama 3
 
[Tool calling](https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/) allows an LLM to detect when one or more tools should be called.
It will then respond with the inputs that should be passed to those tools.
LangChain has a general agent that works with tool-calling LLMs. In this notebook we'll use the following tools with Agent Executor:
LLM-powered agents combine planning, memory, and tool-use (see [here](https://lilianweng.github.io/posts/2023-06-23-agent/), [here](https://www.deeplearning.ai/the-batch/how-agents-can-improve-llm-performance/)).
 
LangChain's [agent executor](https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/) offers a simple way to quickly get started with agents.
 
%% Cell type:markdown id:3fcd455a-6255-4c7c-9e8e-2aaf1b323c05 tags:
Here, we will show how to augment a tool-calling version of Llama 3 with various multi-modal capabilities using an agent.
 
![image.png](attachment:6af2e76d-6f3d-44c9-a128-c19e70a400e9.png)
 
%% Cell type:markdown id:8baac811-22b0-4bcb-a7d4-6eb87a1b61d3 tags:
 
## Tools
## Introduce the tools that we want our agent to use
 
### 1. Custom Function
 
We'll define a few tools that our agent will use.
This is a custom function that we want our agent to utilize.
 
%% Cell type:code id:674be83d-d392-41f0-8b7f-a461ebac89f4 tags:
 
``` python
def magic_function(input: int) -> int:
"""Applies a magic function to an input."""
return input + 2
 
magic_function(3)
```
 
%% Output
 
5
 
%% Cell type:code id:07dc6530-abb0-4bda-a071-1163239ab2c6 tags:
%% Cell type:code id:75c46af4-f03e-4448-a49e-8bfe0e644c2a tags:
 
``` python
from langchain.agents import tool
 
@tool
def magic_function(input: int) -> int:
"""Applies a magic function to an input."""
return input + 2
```
 
%% Cell type:markdown id:51448223-8e7f-4fcc-a0f9-9d2f56e15dde tags:
 
### 2. Web Search
 
Let's use [Tavily](https://tavily.com/#api) for web search.
We'll use [Tavily](https://tavily.com/#api) for web search.
 
%% Cell type:code id:16569e69-890c-4e0c-947b-35ed2e0cf0c7 tags:
%% Cell type:code id:4c6be142-757c-4ee4-983e-0b5a54fafff9 tags:
 
``` python
# Ensure API key is set
import os
from getpass import getpass
TAVILY_API_KEY = getpass()
os.environ["TAVILY_API_KEY"] = TAVILY_API_KEY
```
 
%% Output
········
%% Cell type:code id:8a4d9feb-80b7-4355-9cba-34e816400aa5 tags:
 
``` python
from langchain_community.tools.tavily_search import TavilySearchResults
web_search_tool = TavilySearchResults()
```
 
%% Cell type:markdown id:42215e7b-3170-4311-8438-5a7b385ebb64 tags:
 
### 3. Text-2-Image
 
We'll use Replicate, which [hosts an open DALL-E model](https://replicate.com/lucataco/open-dalle-v1.1/versions/1c7d4c8dec39c7306df7794b28419078cb9d18b9213ab1c21fdc46a1deca0144).
We'll use [Replicate](https://replicate.com/), which offers free to try API key and hosts an [open DALL-E model](https://replicate.com/lucataco/open-dalle-v1.1/versions/1c7d4c8dec39c7306df7794b28419078cb9d18b9213ab1c21fdc46a1deca0144).
 
Test the code before converting it to a tool (this may take 1-2 minutes to run):
Test the code (this may take 1-2 minutes to run):
 
%% Cell type:code id:c130481d-dc6f-48e0-b795-7e3a4438fb6a tags:
%% Cell type:code id:2fb8879e-24e2-4370-b89e-10644ce1ddfc tags:
 
``` python
# Ensure API key is set
REPLICATE_API_TOKEN = getpass()
os.environ["REPLICATE_API_TOKEN"] = REPLICATE_API_TOKEN
```
 
%% Output
········
%% Cell type:code id:873cad84-8624-408d-935d-6ce9560c941c tags:
 
``` python
import replicate
import requests
from PIL import Image
from io import BytesIO
import matplotlib.pyplot as plt
 
def display_image(image_url):
"""Display generated image"""
response = requests.get(image_url)
img = Image.open(BytesIO(response.content))
plt.imshow(img)
plt.axis('off')
plt.show()
 
def text2image(text: str) -> str:
"""generate an image based on a text."""
output = replicate.run(
"lucataco/open-dalle-v1.1:1c7d4c8dec39c7306df7794b28419078cb9d18b9213ab1c21fdc46a1deca0144",
input={
"width": 1024,
"height": 1024,
"prompt": text, #"a yellow lab puppy running free with wild flowers in the mountain behind",
"scheduler": "KarrasDPM",
"num_outputs": 1,
"guidance_scale": 7.5,
"apply_watermark": True,
"negative_prompt": "worst quality, low quality",
"prompt_strength": 0.8,
"num_inference_steps": 60
}
)
print(output)
return output[0]
 
output = text2image("a yellow lab puppy running free with wild flowers in the mountain behind")
display_image(output)
```
 
%% Output
 
['https://replicate.delivery/pbxt/zdOZ0CkeysXSdy3fbtMioeAIEp7sYfDGsq0echfIHMXqTGwsE/out-0.png']
 
 
%% Cell type:markdown id:6771c9ee-8d90-4cd0-bd40-797baf838dc0 tags:
 
Wrap the code above as a custom tool:
 
%% Cell type:code id:d426794d-c915-45a0-b0d3-f5c71eda0c81 tags:
 
``` python
from langchain.agents import tool
 
@tool
def text2image(text: str) -> int:
"""generate an image based on a text."""
output = replicate.run(
"lucataco/open-dalle-v1.1:1c7d4c8dec39c7306df7794b28419078cb9d18b9213ab1c21fdc46a1deca0144",
input={
"width": 1024,
"height": 1024,
"prompt": text, #"a yellow lab puppy running free with wild flowers in the mountain behind",
"scheduler": "KarrasDPM",
"num_outputs": 1,
"guidance_scale": 7.5,
"apply_watermark": True,
"negative_prompt": "worst quality, low quality",
"prompt_strength": 0.8,
"num_inference_steps": 60
}
)
print(output)
return output[0]
```
 
%% Cell type:markdown id:a1f43ae8-826b-4703-a938-7599213546a7 tags:
 
### 4. Image-2-Text
 
We'll use Replicate, which [hosts llava-13b](https://replicate.com/yorickvp/llava-13b).
We'll use Replicate, which hosts [llava-13b](https://replicate.com/yorickvp/llava-13b).
 
Test the code before converting it to a tool:
 
%% Cell type:code id:b20fdbfb-f2d3-4982-bbcf-b1a4805c298f tags:
 
``` python
def image2text(image_url: str, prompt: str) -> int:
"""generate a text on image_url based on prompt."""
input = {
"image": image_url,
"prompt": prompt
}
 
output = replicate.run(
"yorickvp/llava-13b:b5f6212d032508382d61ff00469ddda3e32fd8a0e75dc39d8a4191bb742157fb",
input=input
)
 
return "".join(output)
 
text = image2text(output, "tell me a bedtime story about the image")
text
```
 
%% Output
 
'Once upon a time, in a beautiful mountain valley, there was a small, happy puppy named Max. Max loved to explore the outdoors and play in the wildflowers that grew on the mountainside. One day, Max decided to take a leap of faith and jump off a cliff, soaring through the air with excitement. As he landed, he found himself in a field of flowers, surrounded by the vibrant colors of yellow, purple, and white. Max was so delighted by his adventure that he started panting with joy and playfully barked at the flowers, as if they were his new friends. From that day on, Max continued to explore the mountain valley, always finding new ways to enjoy the beauty of nature and the thrill of adventure.'
 
%% Cell type:markdown id:c339ee7b-9c4a-4973-8fde-89b81a72dec7 tags:
 
Wrap the code above as another custom tool:
 
%% Cell type:code id:46aa60e5-6841-4292-b79e-c4f43c1220a5 tags:
 
``` python
from langchain.agents import tool
 
@tool
def image2text(image_url: str, prompt: str) -> int:
"""generate a text on image_url based on prompt."""
input = {
"image": image_url,
"prompt": prompt
}
 
output = replicate.run(
"yorickvp/llava-13b:b5f6212d032508382d61ff00469ddda3e32fd8a0e75dc39d8a4191bb742157fb",
input=input
)
 
return "".join(output)
```
 
%% Cell type:markdown id:03a65ae4-d18e-42f0-b821-884a42a72209 tags:
 
### 5. Text-2-Speech
 
We'll use Replicate, which [hosts text-2-speech](https://replicate.com/cjwbw/seamless_communication).
We'll use Replicate, which hosts [text-2-speech](https://replicate.com/cjwbw/seamless_communication).
 
Test the code before creating yet another custom tool (this may take a couple of minutes to run):
 
%% Cell type:code id:36957c72-1530-4369-be8c-7c397b30e4ac tags:
 
``` python
from IPython.display import Audio
 
def play_audio(output_url):
return Audio(url=output_url, autoplay=False)
 
def text2speech(text: str) -> int:
"""convert a text to a speech."""
output = replicate.run(
"cjwbw/seamless_communication:668a4fec05a887143e5fe8d45df25ec4c794dd43169b9a11562309b2d45873b0",
input={
"task_name": "T2ST (Text to Speech translation)",
"input_text": text,
"input_text_language": "English",
"max_input_audio_length": 60,
"target_language_text_only": "English",
"target_language_with_speech": "English"
}
)
return output['audio_output']
 
output_url = text2speech(text)
play_audio(output_url)
```
 
%% Output
 
<IPython.lib.display.Audio object>
 
%% Cell type:markdown id:7d5dfff0-f7be-4ca9-b6dc-72c332f66231 tags:
 
Wrap the code above to a new custom tool:
 
%% Cell type:code id:2a9b450b-0174-4079-9ff3-4b9907fb6f48 tags:
 
``` python
from langchain.agents import tool
 
@tool
def text2speech(text: str) -> int:
"""convert a text to a speech."""
output = replicate.run(
"cjwbw/seamless_communication:668a4fec05a887143e5fe8d45df25ec4c794dd43169b9a11562309b2d45873b0",
input={
"task_name": "T2ST (Text to Speech translation)",
"input_text": text,
"input_text_language": "English",
"max_input_audio_length": 60,
"target_language_text_only": "English",
"target_language_with_speech": "English"
}
)
return output['audio_output']
```
 
%% Cell type:markdown id:51d348c5-e828-4d93-8265-503cce967940 tags:
 
Collect all tools created to a list:
 
%% Cell type:code id:bfc0cfbe-d5ce-4c26-859f-5d29be121bef tags:
 
``` python
tools = [magic_function, web_search_tool, text2image, image2text, text2speech]
 
tools
```
 
%% Output
 
[StructuredTool(name='magic_function', description='magic_function(input: int) -> int - Applies a magic function to an input.', args_schema=<class 'pydantic.v1.main.magic_functionSchema'>, func=<function magic_function at 0x105a6f9c0>),
TavilySearchResults(),
StructuredTool(name='text2image', description='text2image(text: str) -> int - generate an image based on a text.', args_schema=<class 'pydantic.v1.main.text2imageSchema'>, func=<function text2image at 0x10f1d6de0>),
StructuredTool(name='image2text', description='image2text(image_url: str, prompt: str) -> int - generate a text on image_url based on prompt.', args_schema=<class 'pydantic.v1.main.image2textSchema'>, func=<function image2text at 0x10f1d7ba0>),
StructuredTool(name='text2speech', description='text2speech(text: str) -> int - convert a text to a speech.', args_schema=<class 'pydantic.v1.main.text2speechSchema'>, func=<function text2speech at 0x10f1ab4c0>)]
 
%% Cell type:markdown id:e88c2e1d-1503-4659-be4d-98900a69253f tags:
 
## LLM
 
Here, we need a Llama 3 model that supports tool use.
 
This can be accomplished via prompt engineering (e.g., see [here](https://replicate.com/hamelsmu/llama-3-70b-instruct-awq-with-tools)) or fine-tuning (e.g., see [here](https://huggingface.co/mzbac/llama-3-8B-Instruct-function-calling) and [here](https://huggingface.co/mzbac/llama-3-8B-Instruct-function-calling)).
 
We can review LLMs that support tool calling [here](https://python.langchain.com/docs/integrations/chat/) and Groq is included.
We can review LangChain LLM integrations that support tool calling [here](https://python.langchain.com/docs/integrations/chat/) and Groq is included.
 
[Here](https://github.com/groq/groq-api-cookbook/blob/main/llama3-stock-market-function-calling/llama3-stock-market-function-calling.ipynb) is a notebook by Groq on function calling with Llama 3 and LangChain.
 
%% Cell type:code id:dc534b40-e2a6-41b6-9f0c-13139c54f6fa tags:
 
``` python
GROQ_API_KEY = getpass()
 
os.environ["GROQ_API_KEY"] = GROQ_API_KEY
```
 
%% Output
 
········
 
%% Cell type:code id:99c919d2-198d-4c3b-85ba-0772bf7db383 tags:
 
``` python
from langchain_groq import ChatGroq
llm = ChatGroq(temperature=0, model="llama3-70b-8192")
```
 
%% Cell type:markdown id:695ffb74-c278-4420-b10b-b18210d824eb tags:
 
## Agent
 
We use LangChain [tool calling agent](https://python.langchain.com/docs/modules/agents/agent_types/tool_calling/).
 
%% Cell type:code id:fae083a8-864c-4394-93e5-36d22aaa5fe3 tags:
 
``` python
# Prompt
from langchain_core.prompts import ChatPromptTemplate, MessagesPlaceholder
prompt = ChatPromptTemplate.from_messages(
[
("system", "You are a helpful assistant"),
("human", "{input}"),
MessagesPlaceholder("agent_scratchpad"),
]
)
prompt.pretty_print()
```
 
%% Output
 
================================ System Message ================================
You are a helpful assistant
================================ Human Message =================================
{input}
============================= Messages Placeholder =============================
{agent_scratchpad}
 
%% Cell type:code id:421f9565-bc1a-4141-aae7-c6bcae2c63fc tags:
 
``` python
# create an agent using the llm, tools and prompt created above
from langchain.agents import AgentExecutor, create_tool_calling_agent, tool
agent = create_tool_calling_agent(llm, tools, prompt)
agent_executor = AgentExecutor(agent=agent, tools=tools, return_intermediate_steps=True)
```
 
%% Cell type:code id:229372f4-abb3-4418-9444-cadc548a8155 tags:
 
``` python
# this will invoke the magic_function tool
agent_executor.invoke({"input": "what is the value of magic_function(3)?"})
```
 
%% Output
 
{'input': 'what is the value of magic_function(3)?',
'output': 'The result of `magic_function(3)` is indeed 5.',
'intermediate_steps': [(ToolAgentAction(tool='magic_function', tool_input={'input': 3}, log="\nInvoking: `magic_function` with `{'input': 3}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'id': 'call_s7m7', 'function': {'arguments': '{"input":3}', 'name': 'magic_function'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls', 'logprobs': None}, id='run-c7d9aa30-c924-4f98-9ee7-811161a6ee1a', tool_calls=[{'name': 'magic_function', 'args': {'input': 3}, 'id': 'call_s7m7'}], tool_call_chunks=[{'name': 'magic_function', 'args': '{"input":3}', 'id': 'call_s7m7', 'index': None}])], tool_call_id='call_s7m7'),
5)]}
 
%% Cell type:markdown id:7a0d9d62-ce4e-497b-b0be-ec8d8f1e1b80 tags:
 
Note that the `ToolAgentAction` in the returned `intermediate_steps` shows the tool called with input.
 
%% Cell type:code id:6914b16c-be7a-4838-b080-b6af6b6e1417 tags:
 
``` python
# this will invoke the tavily_search_results_json tool
agent_executor.invoke({"input": "whats the weather in sf?"})
```
 
%% Output
 
{'input': 'whats the weather in sf?',
'output': 'According to the tool call result, the current weather in San Francisco is sunny with a temperature of 64.9°F (18.3°C) and a humidity of 68%. The wind is blowing at 3.8 mph (6.1 kph) from the ENE direction.',
'intermediate_steps': [(ToolAgentAction(tool='tavily_search_results_json', tool_input={'query': 'weather in san francisco'}, log="\nInvoking: `tavily_search_results_json` with `{'query': 'weather in san francisco'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'id': 'call_mzrb', 'function': {'arguments': '{"query":"weather in san francisco"}', 'name': 'tavily_search_results_json'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls', 'logprobs': None}, id='run-709eea16-89af-4f04-80b7-14bbe331d3f6', tool_calls=[{'name': 'tavily_search_results_json', 'args': {'query': 'weather in san francisco'}, 'id': 'call_mzrb'}], tool_call_chunks=[{'name': 'tavily_search_results_json', 'args': '{"query":"weather in san francisco"}', 'id': 'call_mzrb', 'index': None}])], tool_call_id='call_mzrb'),
[{'url': 'https://www.weatherapi.com/',
'content': "{'location': {'name': 'San Francisco', 'region': 'California', 'country': 'United States of America', 'lat': 37.78, 'lon': -122.42, 'tz_id': 'America/Los_Angeles', 'localtime_epoch': 1715360953, 'localtime': '2024-05-10 10:09'}, 'current': {'last_updated_epoch': 1715360400, 'last_updated': '2024-05-10 10:00', 'temp_c': 18.3, 'temp_f': 64.9, 'is_day': 1, 'condition': {'text': 'Sunny', 'icon': '//cdn.weatherapi.com/weather/64x64/day/113.png', 'code': 1000}, 'wind_mph': 3.8, 'wind_kph': 6.1, 'wind_degree': 60, 'wind_dir': 'ENE', 'pressure_mb': 1015.0, 'pressure_in': 29.96, 'precip_mm': 0.0, 'precip_in': 0.0, 'humidity': 68, 'cloud': 0, 'feelslike_c': 18.3, 'feelslike_f': 64.9, 'vis_km': 16.0, 'vis_miles': 9.0, 'uv': 5.0, 'gust_mph': 8.3, 'gust_kph': 13.3}}"},
{'url': 'https://www.accuweather.com/en/us/san-francisco/94103/weather-forecast/347629',
'content': 'San Francisco, CA Weather Forecast, with current conditions, wind, air quality, and what to expect for the next 3 days.'},
{'url': 'https://www.timeanddate.com/weather/usa/san-francisco/hourly',
'content': 'Sun & Moon. Weather Today Weather Hourly 14 Day Forecast Yesterday/Past Weather Climate (Averages) Currently: 55 °F. Passing clouds. (Weather station: San Francisco International Airport, USA). See more current weather.'},
{'url': 'https://www.accuweather.com/en/us/san-francisco/94103/current-weather/347629',
'content': 'Current weather in San Francisco, CA. Check current conditions in San Francisco, CA with radar, hourly, and more.'},
{'url': 'https://www.wunderground.com/hourly/us/ca/san-francisco/94130/date/2024-05-10',
'content': 'San Francisco Weather Forecasts. Weather Underground provides local & long-range weather forecasts, weatherreports, maps & tropical weather conditions for the San Francisco area. ... Friday 05/10 ...'}])]}
 
%% Cell type:code id:8ce1b38c-a22a-4035-a9a1-2ea0da419ade tags:
 
``` python
# this will invoke the text2image tool
output = agent_executor.invoke({"input": "generate an image based on this text: a yellow lab puppy running free with wild flowers in the mountain behind"})
```
 
%% Output
 
['https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png']
 
%% Cell type:code id:fa945d8b-f5f2-444f-87cc-1458301d521f tags:
 
``` python
output, output['intermediate_steps'][0][-1]
```
 
%% Output
 
({'input': 'generate an image based on this text: a yellow lab puppy running free with wild flowers in the mountain behind',
'output': 'Here is the result of the tool call:\n\nThe generated image is available at the provided URL: https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png',
'intermediate_steps': [(ToolAgentAction(tool='text2image', tool_input={'text': 'a yellow lab puppy running free with wild flowers in the mountain behind'}, log="\nInvoking: `text2image` with `{'text': 'a yellow lab puppy running free with wild flowers in the mountain behind'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'id': 'call_6hjz', 'function': {'arguments': '{"text":"a yellow lab puppy running free with wild flowers in the mountain behind"}', 'name': 'text2image'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls', 'logprobs': None}, id='run-4b677104-9fa9-4e1b-8fed-2b898f7860eb', tool_calls=[{'name': 'text2image', 'args': {'text': 'a yellow lab puppy running free with wild flowers in the mountain behind'}, 'id': 'call_6hjz'}], tool_call_chunks=[{'name': 'text2image', 'args': '{"text":"a yellow lab puppy running free with wild flowers in the mountain behind"}', 'id': 'call_6hjz', 'index': None}])], tool_call_id='call_6hjz'),
'https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png')]},
'https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png')
 
%% Cell type:code id:6ca33feb-7eb5-4712-8ec5-b4ef1ff61056 tags:
 
``` python
image_url = output['intermediate_steps'][0][-1]
display_image(image_url)
```
 
%% Output
 
 
%% Cell type:code id:459720a1-a73c-4c52-82f6-132be9d427da tags:
 
``` python
# this will invoke the image2text tool
output = agent_executor.invoke({"input": f"convert this image url to a bedtime story: {image_url}"})
```
 
%% Cell type:code id:55e7518c-e7d8-4ce7-9a4a-7909fb3a8b88 tags:
 
``` python
output, output['intermediate_steps'][0][-1]
```
 
%% Output
 
({'input': 'convert this image url to a bedtime story: https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png',
'output': "What a lovely bedtime story! It sounds like Max had a wonderful adventure in the mountain valley, chasing butterflies and exploring the beautiful scenery. I'm glad he got to rest and appreciate the beauty around him. Would you like me to generate another story or assist you with something else?",
'intermediate_steps': [(ToolAgentAction(tool='image2text', tool_input={'image_url': 'https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png', 'prompt': 'bedtime story'}, log="\nInvoking: `image2text` with `{'image_url': 'https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png', 'prompt': 'bedtime story'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'id': 'call_pc72', 'function': {'arguments': '{"image_url":"https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png","prompt":"bedtime story"}', 'name': 'image2text'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls', 'logprobs': None}, id='run-8370c0cf-378b-4d4d-b8aa-efcae40c1df9', tool_calls=[{'name': 'image2text', 'args': {'image_url': 'https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png', 'prompt': 'bedtime story'}, 'id': 'call_pc72'}], tool_call_chunks=[{'name': 'image2text', 'args': '{"image_url":"https://replicate.delivery/pbxt/En1lI29WUnpdANbpMKk2abYKxIvzK7WN7YTl4115S1hKHwsE/out-0.png","prompt":"bedtime story"}', 'id': 'call_pc72', 'index': None}])], tool_call_id='call_pc72'),
'Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind.\n\nAs he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase.\n\nAfter a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again.')]},
'Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind.\n\nAs he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase.\n\nAfter a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again.')
 
%% Cell type:code id:1f44ba91-e9d7-4fc9-92c0-ba8a408a7172 tags:
 
``` python
# this will invoke the text2speech tool
story = output['intermediate_steps'][0][-1]
output = agent_executor.invoke({"input": f"convert the text to speech: {story}"})
```
 
%% Cell type:code id:e033017d-8eba-4afa-b23b-c95688c527cd tags:
 
``` python
output, output['intermediate_steps'][0][-1]
```
 
%% Output
 
({'input': 'convert the text to speech: Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind.\n\nAs he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase.\n\nAfter a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again.',
'output': 'Here is the result of the tool call:\n\nhttps://replicate.delivery/pbxt/i53EMcpHlFaGNBUyD9bTIbZNdWwyH7cL5qdeT3eeo2BlABmlA/out.wav',
'intermediate_steps': [(ToolAgentAction(tool='text2speech', tool_input={'text': 'Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind. As he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase. After a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again.'}, log="\nInvoking: `text2speech` with `{'text': 'Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind. As he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase. After a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again.'}`\n\n\n", message_log=[AIMessageChunk(content='', additional_kwargs={'tool_calls': [{'id': 'call_z2yg', 'function': {'arguments': '{"text":"Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind. As he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase. After a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again."}', 'name': 'text2speech'}, 'type': 'function'}]}, response_metadata={'finish_reason': 'tool_calls', 'logprobs': None}, id='run-caee5588-4d51-4e3c-b638-fcac91cef73c', tool_calls=[{'name': 'text2speech', 'args': {'text': 'Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind. As he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase. After a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again.'}, 'id': 'call_z2yg'}], tool_call_chunks=[{'name': 'text2speech', 'args': '{"text":"Once upon a time, in a beautiful mountain valley, there was a small white dog named Max. Max loved to explore the outdoors and was always eager to find new adventures. One day, he decided to take a walk along a trail that led to a stunning view of the mountains. As he ran along the trail, he came across a field filled with colorful flowers, including yellow ones. Max was so excited to see the flowers that he started running towards them, with his tongue hanging out, and his ears flapping in the wind. As he ran, he noticed a group of butterflies flying around the flowers, adding to the beauty of the scene. Max was so captivated by the butterflies that he stopped running and started chasing them, trying to catch them in his mouth. The butterflies were playful and kept flying away, leading Max on a fun and exciting chase. After a while, Max realized that he was getting tired, so he decided to take a break and rest on the grass. As he lay down, he looked up at the mountains and the flowers, feeling grateful for the wonderful adventure he had experienced. He knew that he would always cherish this memory and that he would come back to this beautiful place again and again."}', 'id': 'call_z2yg', 'index': None}])], tool_call_id='call_z2yg'),
'https://replicate.delivery/pbxt/i53EMcpHlFaGNBUyD9bTIbZNdWwyH7cL5qdeT3eeo2BlABmlA/out.wav')]},
'https://replicate.delivery/pbxt/i53EMcpHlFaGNBUyD9bTIbZNdWwyH7cL5qdeT3eeo2BlABmlA/out.wav')
 
%% Cell type:code id:c742da14-fae6-4ab9-8cfd-17df8a2835e7 tags:
 
``` python
Audio(url=output['intermediate_steps'][0][-1], autoplay=False)
```
 
%% Output
 
<IPython.lib.display.Audio object>
 
%% Cell type:markdown id:6b97040f-5bac-4fe6-893e-e7deef8ab246 tags:
 
We can see that the agent correctly decides which tool to call for different query using AgentExecutor.
 
In the next [notebook](langgraph-agent.ipynb), we will show an alternative way to implement this agent using LangGraph.
In the next [notebook](langgraph-tool-calling-agent.ipynb), we will show an alternative way to implement this agent using LangGraph.
......
scripts/spellcheck_conf/wordlist.txt
+ 4
− 1
View file @ 76cb6036
......@@ -1342,9 +1342,12 @@ sha
tmoreau
toolchain
wifi
AgentFinish
ReAct
customizable
Kaggle
SalesBot
Weaviate
MediaGen
SDXL
SVD
SVD
\ No newline at end of file
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment

:feet::heart: Strive to be the person your dogs believe you are :heart::feet: