Skip to content
Snippets Groups Projects
Unverified Commit e9aa2a85 authored by Logan's avatar Logan Committed by GitHub
Browse files

fix null model meta references (#18109)

* fix null model meta references

* notebook
parent 6ba3f55c
No related branches found
No related tags found
No related merge requests found
%% Cell type:markdown id: tags:
<a href="https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/docs/examples/llm/gemini.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>
%% Cell type:markdown id: tags:
# Google GenAI
%% Cell type:markdown id: tags:
In this notebook, we show how to use the the `google-genai` Python SDK with LlamaIndex to interact with Google GenAI models.
If you're opening this Notebook on colab, you will need to install LlamaIndex 🦙 and the `google-genai` Python SDK.
%% Cell type:code id: tags:
``` python
%pip install llama-index-llms-google-genai llama-index
```
%% Cell type:markdown id: tags:
## Basic Usage
You will need to get an API key from [Google AI Studio](https://makersuite.google.com/app/apikey). Once you have one, you can either pass it explicity to the model, or use the `GOOGLE_API_KEY` environment variable.
%% Cell type:code id: tags:
``` python
import os
os.environ["GOOGLE_API_KEY"] = "..."
```
%% Cell type:markdown id: tags:
## Basic Usage
You can call `complete` with a prompt:
%% Cell type:code id: tags:
``` python
from llama_index.llms.google_genai import GoogleGenAI
llm = GoogleGenAI(
model="gemini-2.0-flash",
# api_key="some key", # uses GOOGLE_API_KEY env var by default
)
resp = llm.complete("Who is Paul Graham?")
print(resp)
```
%% Output
Paul Graham is a prominent figure in the world of computer programming, entrepreneurship, and essay writing. He's best known as:
* **Co-founder of Y Combinator:** This is arguably his biggest claim to fame. Y Combinator (YC) is a highly influential startup accelerator that has funded companies like Airbnb, Dropbox, Reddit, Stripe, and many others. Graham's insights and mentorship at YC have shaped the landscape of the tech startup world.
* **Author and Essayist:** Graham is a prolific writer, known for his insightful essays on topics ranging from technology and startups to programming, design, and even art. His essays are widely read and often spark debate within the tech community. Some popular essays include "Hackers & Painters," "Beating the Averages," and "Do Things That Don't Scale."
* **Computer Programmer and Hacker:** Graham is a skilled programmer, particularly known for his work in Lisp. He co-founded Viaweb, a startup that created software for building online stores, which was later acquired by Yahoo! and became Yahoo! Store.
* **Venture Capitalist:** Through Y Combinator, Graham and his partners have invested in and mentored a large number of startups, playing a significant role in the venture capital ecosystem.
In summary, Paul Graham is a multifaceted individual who has made significant contributions to the tech world as an entrepreneur, programmer, writer, and investor. His influence is particularly strong within the startup community.
%% Cell type:markdown id: tags:
You can also call `chat` with a list of chat messages:
%% Cell type:code id: tags:
``` python
from llama_index.core.llms import ChatMessage
from llama_index.llms.google_genai import GoogleGenAI
messages = [
ChatMessage(
role="system", content="You are a pirate with a colorful personality"
),
ChatMessage(role="user", content="Tell me a story"),
]
llm = GoogleGenAI(model="gemini-2.0-flash")
resp = llm.chat(messages)
print(resp)
```
%% Output
assistant: Ahoy there, matey! Gather 'round, you landlubbers, and let ol' One-Eyed Jack spin ye a yarn that'll curl yer barnacles and raise yer timbers! This ain't no tale for the faint of heart, mind ye, it's the true story of the Devil's Kiss, the most beautiful, and most cursed, ruby in all the seven seas!
Now, I acquired this… knowledge, shall we say, back when I was a mere swab, scrubbing decks for Captain "Rotten" Roger Reddington. Rotten Roger, bless his scurvy-infested soul, was obsessed with finding the lost treasure of Bartholomew Blackheart, a pirate so vile, he used to floss with sharks' teeth!
For years, Rotten Roger chased whispers and rumors, from the soggy jungles of Tortuga to the frozen fjords of the North. Then, one day, we landed on a forsaken isle, a place called Isle de la Llorona, or Weeping Woman's Isle, for those who can't mangle a good foreign tongue. Locals warned us the island was haunted, cursed by the spirit of a maiden who Blackheart had scorned and left to die.
Rotten Roger, being about as superstitious as a sea slug, scoffed. "Ghosts? Bah! I'll face a Kraken with a spoon before I fear a woman's tears!" He charged into the jungle, me and the rest of the crew trailing behind like so many sea dogs.
We stumbled upon a crumbling temple, covered in vines and reeking of decay. Inside, amidst the cobwebs and bat droppings, we found it! Blackheart's tomb! Rotten Roger let out a whoop that would wake the dead, and started tearing through the sarcophagus.
And there it was! Resting on a cushion of faded velvet, the Devil's Kiss. A ruby so red, it looked like it held a drop of hellfire itself. It pulsed with a wicked light, hypnotizing even the most grizzled pirates.
Rotten Roger grabbed the ruby, cackling like a hyena. That's when the wind started howling, and the ground began to shake. The air grew thick with the scent of brine and sorrow.
Then we saw her. The Weeping Woman. A spectral figure, pale as moonlight, with tears that flowed like rivers. She let out a wail that could shatter glass, and pointed a skeletal finger at Rotten Roger.
"You dare disturb his rest?" she shrieked. "You shall pay the price!"
Now, most of the crew, being the cowardly lot they were, hightailed it back to the ship. But Rotten Roger, blinded by greed, stood his ground. He clutched the Devil's Kiss and roared, "I ain't afraid of no ghost! This ruby is mine!"
That's when the curse truly began. One by one, the crew started to meet unfortunate ends. One swallowed a live lobster whole, another was struck by lightning on a clear day, and a third… well, let's just say he tried to shave his beard with a cutlass and wasn't very good at it.
Rotten Roger, meanwhile, became increasingly paranoid and erratic. He wouldn't sleep, he barely ate, and he started seeing the Weeping Woman everywhere he looked. He blamed the ruby, but he couldn't bring himself to let it go.
Finally, in a fit of madness, he decided to bury the ruby back on the Isle de la Llorona, hoping to appease the ghost. But as he dug the hole, the Weeping Woman appeared again. She grabbed him, and with a bloodcurdling scream, pulled him into the earth!
The next morning, the few remaining crew members, myself included, abandoned the island. We never spoke of the Devil's Kiss again.
So, that's the tale, mateys. A cautionary one, indeed. The Devil's Kiss is still out there, buried with Rotten Roger on the Isle de la Llorona. A beautiful, cursed gem that brings nothing but misfortune. Take it from ol' One-Eyed Jack, some treasures are best left buried. Now, who's buying the next round of grog? This old pirate's throat is drier than a desert rat's sock!
%% Cell type:markdown id: tags:
## Streaming Support
Every method supports streaming through the `stream_` prefix.
%% Cell type:code id: tags:
``` python
from llama_index.llms.google_genai import GoogleGenAI
llm = GoogleGenAI(model="gemini-2.0-flash")
resp = llm.stream_complete("Who is Paul Graham?")
for r in resp:
print(r.delta, end="")
```
%% Output
Paul Graham is a prominent figure in the technology and startup world. Here's a summary of who he is and why he's known:
* **Computer Programmer and Essayist:** Graham is a skilled programmer, known for his work in Lisp, and he's a prolific and influential essayist on topics ranging from programming and startups to art and philosophy.
* **Co-founder of Viaweb (later Yahoo! Store):** He co-founded Viaweb, which was one of the first companies to offer a web-based application for building and hosting online stores. Yahoo! acquired Viaweb in 1998, and it became Yahoo! Store.
* **Founder of Y Combinator (YC):** He's most famous for co-founding Y Combinator (YC) in 2005. YC is a highly successful startup accelerator that provides seed funding, mentorship, and network connections to early-stage startups. It has funded many well-known companies, including Airbnb, Dropbox, Reddit, Stripe, and many others.
* **Influential Thinker on Startups:** Through Y Combinator and his essays, Graham has profoundly shaped the way people think about startups. He's known for his practical advice, focusing on building things that people want, iterating quickly, and achieving rapid growth. His essays are widely read and discussed in the startup community.
* **Author:** He's the author of several books, including "On Lisp," "Hackers & Painters," and "ANSI Common Lisp."
In summary, Paul Graham is a successful entrepreneur, programmer, essayist, and investor who has played a significant role in shaping the modern startup ecosystem through Y Combinator and his influential writings.
%% Cell type:code id: tags:
``` python
from llama_index.core.llms import ChatMessage
messages = [
ChatMessage(role="user", content="Who is Paul Graham?"),
]
resp = llm.stream_chat(messages)
for r in resp:
print(r.delta, end="")
```
%% Output
Paul Graham is a prominent figure in the technology and startup world, best known for several key accomplishments:
* **Co-founder of Y Combinator (YC):** This is arguably his most significant achievement. Y Combinator is a highly influential startup accelerator that has funded and mentored numerous successful companies, including Airbnb, Dropbox, Reddit, Stripe, and many more. Graham's experience and insights shaped YC's unique approach to early-stage startup investing and guidance.
* **Author and Essayist:** Graham is a prolific writer, known for his thought-provoking essays on topics ranging from programming and startups to art, education, and societal trends. His essays are widely read and discussed within the tech community. Some popular essays include "Hackers and Painters," "Do Things That Don't Scale," and "How to Start a Startup." His writing style is known for its clarity, directness, and often contrarian viewpoints.
* **Programmer and Hacker:** He has a strong background in computer science and programming. He holds a Ph.D. in computer science from Harvard.
* **Viaweb/Yahoo! Store:** Before Y Combinator, Graham co-founded Viaweb, one of the first software as a service (SaaS) companies, which provided tools for building online stores. Viaweb was later acquired by Yahoo! and became Yahoo! Store.
**In summary, Paul Graham is a highly influential figure in the startup world because of his role in creating and shaping Y Combinator, his insightful essays, and his background as a programmer and entrepreneur.** He is often sought after for his advice and opinions on startups, technology, and innovation.
%% Cell type:markdown id: tags:
## Async Usage
Every synchronous method has an async counterpart.
%% Cell type:code id: tags:
``` python
from llama_index.llms.google_genai import GoogleGenAI
llm = GoogleGenAI(model="gemini-2.0-flash")
resp = await llm.astream_complete("Who is Paul Graham?")
async for r in resp:
print(r.delta, end="")
```
%% Output
Paul Graham is a prominent figure in the tech world, known for several key accomplishments:
* **Co-founder of Y Combinator:** This is arguably his most well-known role. Y Combinator (YC) is a highly influential startup accelerator that has funded and mentored companies like Airbnb, Dropbox, Stripe, Reddit, and many others. Graham's vision and approach to early-stage startup funding have significantly shaped the startup ecosystem.
* **Programmer and Essayist:** Before Y Combinator, Graham was a respected programmer. He co-founded Viaweb, an early e-commerce platform that was acquired by Yahoo! and became Yahoo! Store. He's also a prolific essayist, writing on topics ranging from technology, startups, programming, art, and society. His essays are known for their clear writing style, insightful observations, and sometimes contrarian viewpoints. They are widely read within the startup community.
* **Lisp Expert:** Graham is a strong proponent of the Lisp programming language. He wrote the book "On Lisp" and has advocated for its use in various applications.
* **Investor and Advisor:** Through Y Combinator and personally, Graham has invested in and advised numerous startups. His experience and perspective are highly sought after by entrepreneurs.
In short, Paul Graham is a highly influential programmer, entrepreneur, investor, and essayist who has played a significant role in shaping the modern tech startup landscape, particularly through his work with Y Combinator.
%% Cell type:code id: tags:
``` python
messages = [
ChatMessage(role="user", content="Who is Paul Graham?"),
]
resp = await llm.achat(messages)
print(resp)
```
%% Output
assistant: Paul Graham is a prominent figure in the world of computer science, venture capital, and writing. Here's a breakdown of who he is and why he's significant:
**Key Highlights:**
* **Computer Scientist:** He holds a Ph.D. in computer science from Harvard and is known for his work in Lisp programming. He developed Viaweb, one of the first software companies to provide an online application.
* **Entrepreneur:** He co-founded Viaweb, which was later acquired by Yahoo! and became Yahoo! Store.
* **Venture Capitalist:** He co-founded Y Combinator (YC), one of the most successful startup accelerators in the world. YC has funded companies like Airbnb, Dropbox, Stripe, Reddit, Coinbase, and many others. He stepped down from managing YC in 2014, but remains involved.
* **Essayist:** Graham is a prolific and influential essayist. He writes about a wide range of topics, including startups, technology, cities, writing, and philosophy. His essays are widely read in the tech community and beyond.
* **Influencer:** Because of his success with Y Combinator and his insightful essays, Graham is considered a major thought leader in the startup and technology world. His ideas have shaped the way many startups are built and funded.
**In more detail:**
* **Education and Early Career:** Graham studied at Cornell University and Harvard University, where he earned his Ph.D. in computer science. His academic background is in artificial intelligence and Lisp programming.
* **Viaweb (Later Yahoo! Store):** He co-founded Viaweb, which provided tools for businesses to create online stores. It was a pioneer in e-commerce and one of the first SaaS (Software as a Service) companies. Yahoo! acquired Viaweb in 1998 and renamed it Yahoo! Store.
* **Y Combinator (YC):** In 2005, Graham, along with Jessica Livingston, Robert Morris, and Trevor Blackwell, founded Y Combinator. YC provides seed funding, mentorship, and networking opportunities to early-stage startups. It runs programs twice a year, investing in a large number of startups each cycle.
* **Essays:** Graham's essays are a cornerstone of his influence. They are known for their clarity, insightful observations, and contrarian viewpoints. Some of his most popular essays include "How to Start a Startup," "Do Things That Don't Scale," "Cities and Ambition," and "Hackers and Painters." These essays often offer practical advice and thought-provoking perspectives on entrepreneurship, technology, and life. They are available on his website, paulgraham.com.
* **Programming Language: Lisp:** Graham is a strong advocate for Lisp and has written extensively about its power and elegance. He believes that Lisp's features make it particularly well-suited for prototyping and exploring new ideas.
In summary, Paul Graham is a multifaceted figure who has made significant contributions to computer science, entrepreneurship, and venture capital. His work with Y Combinator has helped launch numerous successful startups, and his essays have inspired and informed countless people in the tech industry and beyond.
%% Cell type:markdown id: tags:
## Vertex AI Support
By providing the `region` and `project_id` parameters (either through environment variables or directly), you can use an Anthropic model through Vertex AI.
%% Cell type:code id: tags:
``` python
# Set environment variables
!export GOOGLE_GENAI_USE_VERTEXAI=true
!export GOOGLE_CLOUD_PROJECT='your-project-id'
!export GOOGLE_CLOUD_LOCATION='us-central1'
```
%% Cell type:code id: tags:
``` python
from llama_index.llms.google_genai import GoogleGenAI
# or set the parameters directly
llm = GoogleGenAI(
model="gemini-2.0-flash",
vertexai_config={"project": "your-project-id", "location": "us-central1"},
# you should set the context window to the max input tokens for the model
context_window=200000,
max_tokens=512,
)
```
%% Output
Paul Graham is a prominent figure in the tech and startup world, best known for his roles as:
* **Co-founder of Y Combinator (YC):** This is arguably his most influential role. YC is a highly successful startup accelerator that has funded companies like Airbnb, Dropbox, Stripe, Reddit, and many others. Graham's approach to funding and mentoring startups has significantly shaped the startup ecosystem.
* **Essayist and Programmer:** Before YC, Graham was a programmer and essayist. He's known for his insightful and often contrarian essays on a wide range of topics, including programming, startups, design, and societal trends. His essays are widely read and discussed in the tech community.
* **Founder of Viaweb (later Yahoo! Store):** Graham founded Viaweb, one of the first application service providers, which allowed users to build and manage online stores. It was acquired by Yahoo! in 1998 and became Yahoo! Store.
In summary, Paul Graham is a highly influential figure in the startup world, known for his role in creating Y Combinator, his insightful essays, and his earlier success as a programmer and entrepreneur.
%% Cell type:markdown id: tags:
## Multi-Modal Support
Using `ChatMessage` objects, you can pass in images and text to the LLM.
%% Cell type:code id: tags:
``` python
!wget https://cdn.pixabay.com/photo/2021/12/12/20/00/play-6865967_640.jpg -O image.jpg
```
%% Output
--2025-03-09 20:36:14-- https://cdn.pixabay.com/photo/2021/12/12/20/00/play-6865967_640.jpg
Resolving cdn.pixabay.com (cdn.pixabay.com)... 172.64.147.160, 104.18.40.96, 2606:4700:4400::6812:2860, ...
Connecting to cdn.pixabay.com (cdn.pixabay.com)|172.64.147.160|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 71557 (70K) [binary/octet-stream]
Saving to: ‘image.jpg’
image.jpg 100%[===================>] 69.88K --.-KB/s in 0.02s
2025-03-09 20:36:14 (3.08 MB/s) - ‘image.jpg’ saved [71557/71557]
%% Cell type:code id: tags:
``` python
from llama_index.core.llms import ChatMessage, TextBlock, ImageBlock
from llama_index.llms.google_genai import GoogleGenAI
llm = GoogleGenAI(model="gemini-2.0-flash")
messages = [
ChatMessage(
role="user",
blocks=[
ImageBlock(path="image.jpg"),
TextBlock(text="What is in this image?"),
],
)
]
resp = llm.chat(messages)
print(resp)
```
%% Output
assistant: The image contains four wooden dice on a dark fabric surface. All the dice are showing the side with six dots.
%% Cell type:markdown id: tags:
## Structured Prediction
LlamaIndex provides an intuitive interface for converting any Anthropic LLMs into a structured LLM through `structured_predict` - simply define the target Pydantic class (can be nested), and given a prompt, we extract out the desired object.
%% Cell type:code id: tags:
``` python
from llama_index.llms.google_genai import GoogleGenAI
from llama_index.core.prompts import PromptTemplate
from llama_index.core.bridge.pydantic import BaseModel
from typing import List
class MenuItem(BaseModel):
"""A menu item in a restaurant."""
course_name: str
is_vegetarian: bool
class Restaurant(BaseModel):
"""A restaurant with name, city, and cuisine."""
name: str
city: str
cuisine: str
menu_items: List[MenuItem]
llm = GoogleGenAI(model="gemini-2.0-flash")
prompt_tmpl = PromptTemplate(
"Generate a restaurant in a given city {city_name}"
)
# Option 1: Use `as_structured_llm`
restaurant_obj = (
llm.as_structured_llm(Restaurant)
.complete(prompt_tmpl.format(city_name="Miami"))
.raw
)
# Option 2: Use `structured_predict`
# restaurant_obj = llm.structured_predict(Restaurant, prompt_tmpl, city_name="Miami")
```
%% Output
Warning: there are non-text parts in the response: ['function_call'],returning concatenated text from text parts,check out the non text parts for full response from model.
%% Cell type:code id: tags:
``` python
print(restaurant_obj)
```
%% Output
name='Mamma Mia' city='Miami' cuisine='Italian' menu_items=[MenuItem(course_name='pasta', is_vegetarian=False)]
%% Cell type:markdown id: tags:
#### Structured Prediction with Streaming
Any LLM wrapped with `as_structured_llm` supports streaming through `stream_chat`.
%% Cell type:code id: tags:
``` python
from llama_index.core.llms import ChatMessage
from IPython.display import clear_output
from pprint import pprint
input_msg = ChatMessage.from_str("Generate a restaurant in San Francisco")
sllm = llm.as_structured_llm(Restaurant)
stream_output = sllm.stream_chat([input_msg])
for partial_output in stream_output:
clear_output(wait=True)
pprint(partial_output.raw.dict())
restaurant_obj = partial_output.raw
restaurant_obj
```
%% Output
{'city': 'San Francisco',
'cuisine': 'Italian',
'menu_items': [{'course_name': 'pizza', 'is_vegetarian': False}],
'name': "Luigi's"}
/tmp/ipykernel_43030/1885953561.py:11: PydanticDeprecatedSince20: The `dict` method is deprecated; use `model_dump` instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.10/migration/
pprint(partial_output.raw.dict())
Restaurant(name="Luigi's", city='San Francisco', cuisine='Italian', menu_items=[MenuItem(course_name='pizza', is_vegetarian=False)])
%% Cell type:markdown id: tags:
## Tool/Function Calling
Google GenAI supports direct tool/function calling through the API. Using LlamaIndex, we can implement some core agentic tool calling patterns.
%% Cell type:code id: tags:
``` python
from llama_index.core.tools import FunctionTool
from llama_index.core.llms import ChatMessage
from llama_index.llms.google_genai import GoogleGenAI
from datetime import datetime
llm = GoogleGenAI(model="gemini-2.0-flash")
def get_current_time(timezone: str) -> dict:
"""Get the current time"""
return {
"time": datetime.now().strftime("%Y-%m-%d %H:%M:%S"),
"timezone": timezone,
}
# uses the tool name, any type annotations, and docstring to describe the tool
tool = FunctionTool.from_defaults(fn=get_current_time)
```
%% Cell type:markdown id: tags:
We can simply do a single pass to call the tool and get the result:
%% Cell type:code id: tags:
``` python
resp = llm.predict_and_call([tool], "What is the current time in New York?")
print(resp)
```
%% Output
Warning: there are non-text parts in the response: ['function_call'],returning concatenated text from text parts,check out the non text parts for full response from model.
{'time': '2025-03-09 20:36:30', 'timezone': 'America/New_York'}
%% Cell type:markdown id: tags:
We can also use lower-level APIs to implement an agentic tool-calling loop!
%% Cell type:code id: tags:
``` python
chat_history = [
ChatMessage(role="user", content="What is the current time in New York?")
]
tools_by_name = {t.metadata.name: t for t in [tool]}
resp = llm.chat_with_tools([tool], chat_history=chat_history)
tool_calls = llm.get_tool_calls_from_response(
resp, error_on_no_tool_call=False
)
if not tool_calls:
print(resp)
else:
while tool_calls:
# add the LLM's response to the chat history
chat_history.append(resp.message)
for tool_call in tool_calls:
tool_name = tool_call.tool_name
tool_kwargs = tool_call.tool_kwargs
print(f"Calling {tool_name} with {tool_kwargs}")
tool_output = tool.call(**tool_kwargs)
print("Tool output: ", tool_output)
chat_history.append(
ChatMessage(
role="tool",
content=str(tool_output),
# most LLMs like Anthropic, OpenAI, etc. need to know the tool call id
additional_kwargs={"tool_call_id": tool_call.tool_id},
)
)
resp = llm.chat_with_tools([tool], chat_history=chat_history)
tool_calls = llm.get_tool_calls_from_response(
resp, error_on_no_tool_call=False
)
print("Final response: ", resp.message.content)
```
%% Output
Warning: there are non-text parts in the response: ['function_call'],returning concatenated text from text parts,check out the non text parts for full response from model.
Calling get_current_time with {'timezone': 'America/New_York'}
Tool output: {'time': '2025-03-09 20:36:32', 'timezone': 'America/New_York'}
Final response: The current time in New York is 2025-03-09 20:36:32.
......
......@@ -33,7 +33,7 @@ from llama_index.core.base.llms.types import (
)
from llama_index.core.bridge.pydantic import BaseModel, Field, PrivateAttr
from llama_index.core.callbacks import CallbackManager
from llama_index.core.constants import DEFAULT_TEMPERATURE
from llama_index.core.constants import DEFAULT_TEMPERATURE, DEFAULT_NUM_OUTPUTS
from llama_index.core.llms.callbacks import llm_chat_callback, llm_completion_callback
from llama_index.core.llms.function_calling import FunctionCallingLLM
from llama_index.core.llms.llm import ToolSelection
......@@ -88,8 +88,9 @@ class GoogleGenAI(FunctionCallingLLM):
ge=0.0,
le=2.0,
)
generate_kwargs: dict = Field(
default_factory=dict, description="Kwargs for generation."
context_window: Optional[int] = Field(
default=None,
description="The context window of the model. If not provided, the default context window 200000 will be used.",
)
is_function_calling_model: bool = Field(
default=True, description="Whether the model is a function calling model."
......@@ -106,13 +107,14 @@ class GoogleGenAI(FunctionCallingLLM):
api_key: Optional[str] = None,
temperature: float = DEFAULT_TEMPERATURE,
max_tokens: Optional[int] = None,
context_window: Optional[int] = None,
vertexai_config: Optional[VertexAIConfig] = None,
http_options: Optional[types.HttpOptions] = None,
debug_config: Optional[google.genai.client.DebugConfig] = None,
generation_config: Optional[types.GenerateContentConfig] = None,
callback_manager: Optional[CallbackManager] = None,
is_function_calling_model: bool = True,
**generate_kwargs: Any,
**kwargs: Any,
):
# API keys are optional. The API can be authorised via OAuth (detected
# environmentally) or by the GOOGLE_API_KEY environment variable.
......@@ -149,18 +151,14 @@ class GoogleGenAI(FunctionCallingLLM):
client = google.genai.Client(**config_params)
model_meta = client.models.get(model=model)
if not max_tokens:
max_tokens = model_meta.output_token_limit
else:
max_tokens = min(max_tokens, model_meta.output_token_limit)
super().__init__(
model=model,
temperature=temperature,
max_tokens=max_tokens,
generate_kwargs=generate_kwargs,
context_window=context_window,
callback_manager=callback_manager,
is_function_calling_model=is_function_calling_model,
**kwargs,
)
self.model = model
......@@ -169,9 +167,16 @@ class GoogleGenAI(FunctionCallingLLM):
# store this as a dict and not as a pydantic model so we can more easily
# merge it later
self._generation_config = (
generation_config.model_dump() if generation_config else {}
generation_config.model_dump()
if generation_config
else types.GenerateContentConfig(
temperature=temperature,
max_output_tokens=max_tokens,
).model_dump()
)
self._max_tokens = (
max_tokens or model_meta.output_token_limit or DEFAULT_NUM_OUTPUTS
)
self._max_tokens = max_tokens
@classmethod
def class_name(cls) -> str:
......@@ -179,7 +184,12 @@ class GoogleGenAI(FunctionCallingLLM):
@property
def metadata(self) -> LLMMetadata:
total_tokens = (self._model_meta.input_token_limit or 0) + self._max_tokens
if self.context_window is None:
base = self._model_meta.input_token_limit or 200000
total_tokens = base + self._max_tokens
else:
total_tokens = self.context_window
return LLMMetadata(
context_window=total_tokens,
num_output=self._max_tokens,
......
......@@ -27,7 +27,7 @@ exclude = ["**/BUILD"]
license = "MIT"
name = "llama-index-llms-google-genai"
readme = "README.md"
version = "0.1.2"
version = "0.1.3"
[tool.poetry.dependencies]
python = ">=3.9,<4.0"
......
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment