use new anthropic token counting api (#16909)

29c9035c · Logan · GitHub · 35906a1b · 29c9035c · 29c9035c
Unverified Commit 29c9035c authored 4 months ago by Logan Committed by GitHub 4 months ago
--- a/docs/docs/examples/llm/anthropic.ipynb
+++ b/docs/docs/examples/llm/anthropic.ipynb
@@ -59,7 +59,7 @@
    "\n",
    "First we want to set the tokenizer, which is slightly different than TikToken.\n",
    "\n",
-    "**NOTE**: The Claude 3 tokenizer has not been updated yet; using the existing Anthropic tokenizer leads to context overflow errors for 200k tokens. We've temporarily set the max tokens for Claude 3 to 180k."
+    "**NOTE**: Anthropic recently updated their token counting API. Older models like claude-2.1 are no longer supported for token counting in the latest versions of the Anthropic python client."
   ]
  },
  {

 %% Cell type:markdown id:6453d3d5 tags:

 <a href="https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/docs/examples/llm/anthropic.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

 %% Cell type:markdown id:72ed6f61-28a7-4f90-8a45-e3f452f95dbd tags:

 # Anthropic

 Anthropic has recently released its latest models: `Claude 3 Opus`, `Claude 3 Sonnet`, and `Claude 3 Haiku` (which will be available soon). By default, the `claude-2.1 model` is used. This notebook provides guidance on how to utilize these new models.

 1. Claude 3 Opus - claude-3-opus-20240229
 2. Claude 3 Sonnet	- claude-3-sonnet-20240229
 3. Claude 3 Haiku - claude-3-haiku-20240307

 %% Cell type:markdown id:c78b172f tags:

 If you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.

 %% Cell type:code id:8e31874a tags:

 ``` python
 %pip install llama-index-llms-anthropic
 ```

 %% Cell type:code id:50fc1a30 tags:

 ``` python
 !pip install llama-index
 ```

 %% Cell type:markdown id:3cbf8694-ad53-459a-84c1-64de2dadeaf5 tags:

 #### Set Tokenizer

 First we want to set the tokenizer, which is slightly different than TikToken.

-**NOTE**: The Claude 3 tokenizer has not been updated yet; using the existing Anthropic tokenizer leads to context overflow errors for 200k tokens. We've temporarily set the max tokens for Claude 3 to 180k.
+**NOTE**: Anthropic recently updated their token counting API. Older models like claude-2.1 are no longer supported for token counting in the latest versions of the Anthropic python client.

 %% Cell type:code id:c6ac37cb-b588-44c7-8fd9-8eab454900a5 tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic
 from llama_index.core import Settings

 tokenizer = Anthropic().tokenizer
 Settings.tokenizer = tokenizer
 ```

 %% Cell type:markdown id:b81a3ef6-2ee5-460d-9aa4-f73708774014 tags:

 #### Call `complete` with a prompt

 %% Cell type:code id:85fbba23 tags:

 ``` python
 import os

 os.environ["ANTHROPIC_API_KEY"] = "YOUR ANTHROPIC API KEY"
 ```

 %% Cell type:code id:910b50ad-c55e-487e-8808-5905dfaa78b4 tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic

 # To customize your API key, do this
 # otherwise it will lookup ANTHROPIC_API_KEY from your env variable
 # llm = Anthropic(api_key="<api_key>")
 llm = Anthropic(model="claude-3-opus-20240229")

 resp = llm.complete("Paul Graham is ")
 ```

 %% Cell type:code id:dfda925e-89c5-47a6-9311-16916ab08b66 tags:

 ``` python
 print(resp)
 ```

 %% Output

    Paul Graham is a well-known entrepreneur, programmer, venture capitalist, and essayist. He is best known for co-founding Viaweb, one of the first web application companies, which was later sold to Yahoo! in 1998 and became Yahoo! Store. Graham is also the co-founder of Y Combinator, a highly successful startup accelerator that has helped launch numerous successful companies, such as Dropbox, Airbnb, and Reddit.
    
    Some key points about Paul Graham:
    
    1. Programming: Graham is a skilled programmer and has written extensively on the subject, including his book "Hackers & Painters: Big Ideas from the Computer Age."
    
    2. Essays: He is a prolific essayist, writing on various topics related to technology, startups, and entrepreneurship. His essays have been influential in the tech startup community.
    
    3. Lisp: Graham is an advocate for the Lisp programming language and has written several essays on its advantages.
    
    4. Y Combinator: As a co-founder of Y Combinator, Graham has played a significant role in shaping the startup ecosystem and has mentored and invested in numerous successful companies.
    
    5. Wealth and inequality: In recent years, Graham has written about income inequality and the concentration of wealth, sparking discussions and debates within the tech community.
    
    Overall, Paul Graham is a significant figure in the technology and startup world, known for his contributions as a programmer, investor, and thought leader.

 %% Cell type:markdown id:f25ccf92 tags:

 #### You can also use an anthropic model through Vertex AI

 %% Cell type:code id:c4a9db35 tags:

 ``` python
 import os

 os.environ["ANTHROPIC_PROJECT_ID"] = "YOUR PROJECT ID HERE"
 os.environ["ANTHROPIC_REGION"] = "YOUR PROJECT REGION HERE"
 ```

 %% Cell type:markdown id:ed545c13 tags:

 ##### Do keep in mind that setting region and project_id here will make Anthropic use the Vertex AI client

 %% Cell type:code id:bd125110 tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic

 llm = Anthropic(
    model="claude-3-5-sonnet@20240620",
    region=os.getenv("ANTHROPIC_REGION"),
    project_id=os.getenv("ANTHROPIC_PROJECT_ID"),
 )

 resp = llm.complete("Paul Graham is ")
 ```

 %% Cell type:code id:de92ce84 tags:

 ``` python
 print(resp)
 ```

 %% Output

    Paul Graham is a well-known computer programmer, entrepreneur, venture capitalist, and essayist. Here are some key points about him:
    
    1. Co-founder of Y Combinator: Graham is best known for co-founding Y Combinator, one of the most successful startup accelerators in the world.
    
    2. Programming language creator: He created the programming language Arc, a dialect of Lisp.
    
    3. Entrepreneur: Before Y Combinator, he co-founded Viaweb, one of the first web-based application companies, which was later acquired by Yahoo!.
    
    4. Author: Graham has written several books on programming and startups, including "Hackers & Painters" and "On Lisp."
    
    5. Essayist: He is known for his insightful essays on technology, startups, and society, which are widely read in the tech community.
    
    6. Investor: Through Y Combinator and personally, he has invested in numerous successful startups, including Dropbox, Airbnb, and Reddit.
    
    7. Advocate for startups: He has been a strong proponent of entrepreneurship and has helped shape the modern startup ecosystem.
    
    8. Computer Science background: Graham holds a Ph.D. in Computer Science from Harvard University.
    
    9. Influential thinker: His ideas on startups, technology, and society have had a significant impact on Silicon Valley and the broader tech industry.
    
    Paul Graham is widely respected for his contributions to the tech industry, his insights on startups and innovation, and his role in shaping the modern entrepreneurial landscape.

 %% Cell type:markdown id:d6ad0a98-92dd-48fd-9823-175d701c1ab2 tags:

 #### Call `chat` with a list of messages

 %% Cell type:code id:5fd3137b-05ce-40a5-bdb0-5ce048f5ca25 tags:

 ``` python
 from llama_index.core.llms import ChatMessage
 from llama_index.llms.anthropic import Anthropic

 messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="Tell me a story"),
 ]
 resp = Anthropic(model="claude-3-opus-20240229").chat(messages)
 ```

 %% Cell type:code id:0d38e262-c98d-4780-aef0-efc00c251da6 tags:

 ``` python
 print(resp)
 ```

 %% Output

    assistant: *clears throat and speaks in a pirate accent* Aye, gather 'round me hearties and I'll spin ye a yarn of adventure on the high seas!
    
    T'was a dark and stormy night when the Black Pearl set sail from Tortuga. The salty sea spray stung me eyes as I stood at the helm, guidin' me beloved ship through the roilin' waves. Me loyal crew scurried about, securin' the riggin' and battening down the hatches.
    
    Suddenly, the lookout cried "Ship ahoy!" and pointed off the starboard bow. I raised me spyglass and spied a Spanish galleon, her decks heavily laden with treasure. The crew gave a hearty cheer - we'd be feastin' and drinkin' well tonight!
    
    I ordered the crew to ready the cannons as we drew alongside the galleon. "Fire all!" I bellowed and the Pearl shook as the guns unleashed a barrage. The Spaniards returned fire but they were no match for me skilled gunners.
    
    We boarded the galleon, swords flashin' and pistols blazin'. The fight was fast and bloody but in the end, the Pearl was victorious! We claimed the treasure as our own - mountains of gold and jewels glintin' in the moonlight.
    
    As we sailed away, I couldn't help but grin. T'was a fine night of piratin' and I knew many more adventures lay ahead for me and me crew. No matter the danger, the Black Pearl would always prevail! Yo ho ho!
    
    *laughs heartily* And that, me friends, is a taste of the pirate's life. May yer sails always be full and yer horizons bright. Fare thee well!

 %% Cell type:markdown id:56a55ce6-08e3-4534-9bae-345686308b3e tags:

 ## Streaming

 %% Cell type:markdown id:57901d1c-d1d4-442e-bb91-cd8f054ae2fd tags:

 Using `stream_complete` endpoint

 %% Cell type:code id:cd9e2b22-7e62-4f50-a9af-84453aeda071 tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic

 llm = Anthropic(model="claude-3-opus-20240229", max_tokens=100)
 resp = llm.stream_complete("Paul Graham is ")
 ```

 %% Cell type:code id:65d68dfc-a97e-4a69-935a-e675fb7b4ed0 tags:

 ``` python
 for r in resp:
    print(r.delta, end="")
 ```

 %% Output

    Paul Graham is a well-known entrepreneur, programmer, venture capitalist, and essayist. He is best known for co-founding Viaweb, one of the first web application companies, which was later sold to Yahoo! in 1998 and became Yahoo! Store.
    
    After the sale of Viaweb, Graham and his wife Jessica Livingston co-founded Y Combinator in 2005, a highly successful startup accelerator that has helped launch

 %% Cell type:code id:10b63238-8d01-48f7-b2ec-a56d23fec172 tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic

 llm = Anthropic(model="claude-3-opus-20240229")
 messages = [
    ChatMessage(
        role="system", content="You are a pirate with a colorful personality"
    ),
    ChatMessage(role="user", content="Tell me a story"),
 ]
 resp = llm.stream_chat(messages)
 ```

 %% Cell type:code id:d90ec6f2-8f49-4f96-9290-c7ed9bb8ba45 tags:

 ``` python
 for r in resp:
    print(r.delta, end="")
 ```

 %% Output

    *clears throat and speaks in a gruff, piratey voice*
    
    Aye, gather 'round me hearties and I'll spin ye a yarn of adventure on the high seas!
    
    'Twas a dark and stormy night, the kind where the wind howls like a banshee and the waves crash over the deck. Me and me crew were sailin' the Caribbean, searchin' for treasure and glory.
    
    Suddenly, the lookout cried "Ship ahoy!" and sure enough, a Spanish galleon was bearin' down on us, her decks bristlin' with cannons. The scurvy dogs wanted our gold, but I'd sooner walk the plank than surrender!
    
    "All hands to battle stations!" I bellowed. "Ready the cannons and prepare to board!"
    
    A mighty battle erupted, cannons boomin' and swords clashin'. We swung over on ropes and fought the Spaniards hand-to-hand on the pitchin' and rollin' deck. Me cutlass was a blur as I dueled their captain, a big brute with a wicked scar.
    
    Finally, I drove me blade into that bilge rat's black heart and he fell dead at me feet. His crew surrendered and we took their ship as a prize. In the hold, we found chests overflowing with gold doubloons and jewels - a king's ransom!
    
    We sailed off into the sunset, our pirate flag snappin' in the breeze, flush with coin and the thrill of victory. And that, me buckos, is a taste of the pirate life! Now who wants some grog?
    
    *laughs heartily*

 %% Cell type:markdown id:d4b6ea50-d777-4174-a326-6e4e57b9ea8b tags:

 ## Configure Model

 %% Cell type:code id:1ce3de8d-287e-402d-936f-64a106c8fac2 tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic

 llm = Anthropic(model="claude-3-sonnet-20240229")
 ```

 %% Cell type:code id:f1727a8f-7653-42e9-a27b-4826e93ddfe5 tags:

 ``` python
 resp = llm.stream_complete("Paul Graham is ")
 ```

 %% Cell type:code id:8b53c20f-bb17-4265-8fd0-8b5921a16495 tags:

 ``` python
 for r in resp:
    print(r.delta, end="")
 ```

 %% Output

    Paul Graham is a computer scientist, entrepreneur, venture capitalist, and author. He is best known for the following:
    
    1. Co-founding Y Combinator: Y Combinator is a prominent startup accelerator based in Silicon Valley. It has funded and helped launch thousands of startups, including Airbnb, Dropbox, Stripe, and Reddit.
    
    2. Writing essays on startups and technology: Graham has written numerous influential essays on topics related to startups, programming, and entrepreneurship. His essays are widely read and have helped shape the thinking of many entrepreneurs and technologists.
    
    3. Developing the programming language Arc: In the early 2000s, Graham developed a new programming language called Arc, which was designed to be a more powerful and expressive dialect of Lisp.
    
    4. Advocating for the use of Lisp and functional programming: Graham is a strong proponent of the Lisp programming language and functional programming paradigms. He has written extensively about the benefits of these approaches and has influenced many programmers to explore them.
    
    5. Authoring books: Graham has authored several books, including "Hackers & Painters: Big Ideas from the Computer Age" (2004), "On Lisp" (1993), and "ANSI Common Lisp" (1995).
    
    6. Investing in startups: Through Y Combinator and his own investments, Graham has invested in and advised numerous successful startups, helping to shape the technology industry.
    
    Overall, Paul Graham is widely respected in the technology and startup communities for his contributions as a programmer, writer, investor, and advocate for innovative ideas and approaches.

 %% Cell type:markdown id:5152a2b4-78e6-47a5-933d-f5186ec0f775 tags:

 ### Async

 %% Cell type:code id:7df04c0e-43ee-4176-9aad-94781d0ed36d tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic

 llm = Anthropic("claude-3-sonnet-20240229")
 resp = await llm.acomplete("Paul Graham is ")
 ```

 %% Cell type:code id:be09c52b-604a-4f05-8f93-36e6ea882ff5 tags:

 ``` python
 print(resp)
 ```

 %% Output

    Paul Graham is a computer scientist, entrepreneur, venture capitalist, and author. He is best known for the following:
    
    1. Co-founding Y Combinator: Y Combinator is a prominent startup accelerator based in Silicon Valley. It has funded and helped launch many successful startups, including Airbnb, Dropbox, Stripe, and Reddit.
    
    2. Writing essays on startups and technology: Graham has written numerous influential essays on topics related to startups, programming, and entrepreneurship. His essays are widely read and have helped shape the thinking of many entrepreneurs and technologists.
    
    3. Developing the programming language Arc: Graham designed and developed the programming language Arc, which was intended to be a more powerful and expressive dialect of Lisp.
    
    4. Authoring books: He has written several books, including "Hackers & Painters: Big Ideas from the Computer Age," "ANSI Common Lisp," and "On Lisp."
    
    5. Founding Viaweb: In the 1990s, Graham co-founded Viaweb, one of the earliest web-based application software companies. Viaweb was later acquired by Yahoo! in 1998.
    
    Graham is widely respected in the technology and startup communities for his insights, writings, and contributions to the field of computer science and entrepreneurship.

 %% Cell type:markdown id:e33bfa1f-589e-475a-8eb3-fa37d95759a7 tags:

 ## Structured Prediction

 LlamaIndex provides an intuitive interface for converting any Anthropic LLMs into a structured LLM through `structured_predict` - simply define the target Pydantic class (can be nested), and given a prompt, we extract out the desired object.

 %% Cell type:code id:d8b622d1-0c91-4cde-ab4c-3f83e0127a4b tags:

 ``` python
 from llama_index.llms.anthropic import Anthropic
 from llama_index.core.prompts import PromptTemplate
 from llama_index.core.bridge.pydantic import BaseModel
 from typing import List


 class MenuItem(BaseModel):
    """A menu item in a restaurant."""

    course_name: str
    is_vegetarian: bool


 class Restaurant(BaseModel):
    """A restaurant with name, city, and cuisine."""

    name: str
    city: str
    cuisine: str
    menu_items: List[MenuItem]


 llm = Anthropic("claude-3-5-sonnet-20240620")
 prompt_tmpl = PromptTemplate(
    "Generate a restaurant in a given city {city_name}"
 )

 # Option 1: Use `as_structured_llm`
 restaurant_obj = (
    llm.as_structured_llm(Restaurant)
    .complete(prompt_tmpl.format(city_name="Miami"))
    .raw
 )
 # Option 2: Use `structured_predict`
 # restaurant_obj = llm.structured_predict(Restaurant, prompt_tmpl, city_name="Miami")
 ```

 %% Cell type:code id:ec769384-d3fe-4761-99e3-bdddcb9c7e4d tags:

 ``` python
 restaurant_obj
 ```

 %% Output

    Restaurant(name='Ocean Breeze Bistro', city='Miami', cuisine='Seafood', menu_items=[MenuItem(course_name='Grilled Mahi-Mahi', is_vegetarian=False), MenuItem(course_name='Coconut Shrimp', is_vegetarian=False), MenuItem(course_name='Key Lime Pie', is_vegetarian=True), MenuItem(course_name='Vegetable Paella', is_vegetarian=True)])

 %% Cell type:markdown id:c63fb270-3d7d-4b9c-9cda-aaa42004c7dd tags:

 #### Structured Prediction with Streaming

 Any LLM wrapped with `as_structured_llm` supports streaming through `stream_chat`.

 %% Cell type:code id:e6e91261-e4ef-4706-8325-65d782f9a87b tags:

 ``` python
 from llama_index.core.llms import ChatMessage
 from IPython.display import clear_output
 from pprint import pprint

 input_msg = ChatMessage.from_str("Generate a restaurant in San Francisco")

 sllm = llm.as_structured_llm(Restaurant)
 stream_output = sllm.stream_chat([input_msg])
 for partial_output in stream_output:
    clear_output(wait=True)
    pprint(partial_output.raw.dict())
    restaurant_obj = partial_output.raw

 restaurant_obj
 ```

 %% Output

    {'city': 'San Francisco',
     'cuisine': 'California Fusion',
     'menu_items': [{'course_name': 'Sourdough Avocado Toast',
                     'is_vegetarian': True},
                    {'course_name': 'Dungeness Crab Cioppino',
                     'is_vegetarian': False},
                    {'course_name': 'Mission-style Veggie Burrito',
                     'is_vegetarian': True},
                    {'course_name': 'Grilled Napa Valley Lamb Chops',
                     'is_vegetarian': False},
                    {'course_name': 'Vegan Ghirardelli Chocolate Mousse',
                     'is_vegetarian': True}],
     'name': 'Golden Gate Grill'}

    Restaurant(name='Golden Gate Grill', city='San Francisco', cuisine='California Fusion', menu_items=[MenuItem(course_name='Sourdough Avocado Toast', is_vegetarian=True), MenuItem(course_name='Dungeness Crab Cioppino', is_vegetarian=False), MenuItem(course_name='Mission-style Veggie Burrito', is_vegetarian=True), MenuItem(course_name='Grilled Napa Valley Lamb Chops', is_vegetarian=False), MenuItem(course_name='Vegan Ghirardelli Chocolate Mousse', is_vegetarian=True)])

--- a/llama-index-integrations/llms/llama-index-llms-anthropic/llama_index/llms/anthropic/base.py
+++ b/llama-index-integrations/llms/llama-index-llms-anthropic/llama_index/llms/anthropic/base.py
@@ -210,7 +210,13 @@ class Anthropic(FunctionCallingLLM):

    @property
    def tokenizer(self) -> Tokenizer:
-        return self._client.get_tokenizer()
+        def _count_tokens(text: str) -> int:
+            return self._client.beta.messages.count_tokens(
+                messages=[{"role": "user", "content": text}],
+                model=self.model,
+            ).input_tokens
+
+        return _count_tokens

    @property
    def _model_kwargs(self) -> Dict[str, Any]:

--- a/llama-index-integrations/llms/llama-index-llms-anthropic/pyproject.toml
+++ b/llama-index-integrations/llms/llama-index-llms-anthropic/pyproject.toml
@@ -27,11 +27,11 @@ exclude = ["**/BUILD"]
 license = "MIT"
 name = "llama-index-llms-anthropic"
 readme = "README.md"
-version = "0.3.9"
+version = "0.4.0"

 [tool.poetry.dependencies]
 python = ">=3.8.1,<4.0"
-anthropic = {extras = ["bedrock", "vertex"], version = ">=0.34.2"}
+anthropic = {extras = ["bedrock", "vertex"], version = ">=0.39.0"}
 llama-index-core = "^0.11.0"

 [tool.poetry.group.dev.dependencies]