diff --git a/README.md b/README.md
index 8a8a28b206416dd9433cd1375021c4860b0a10ef..6a2db40f1f831f5d1eab3eb5bfc161b3f77402c4 100644
--- a/README.md
+++ b/README.md
@@ -10,8 +10,8 @@ A tree-based index containing text data that is created using GPT-3 and can be t
- The ability to feed "knowledge" to GPT-3 is restricted to this limited prompt size and model weights.
- **Thought**: What if GPT-3 can have access to potentially a much larger database of knowledge without retraining/finetuning?
-#### Solution
-That's where the **GPT Tree Index** comes in. Instead of relying on world knowledge encoded in the model weights, the GPT Tree Index does the following:
+#### Proposed Solution [WIP]
+That's where the **GPT Tree Index** comes in (if we can resolve some kinks!). Instead of relying on world knowledge encoded in the model weights, the GPT Tree Index does the following:
- Uses a pre-trained GPT-3 model primarily for *reasoning*/*summarization* instead of prior knowledge
- Takes as input a large corpus of text data, uses GPT-3 to build a tree-structured index over it
- Also use GPT-3 to traverse the tree index that it created in order to answer a query
diff --git a/examples/paul_graham_essay/TestEssay.ipynb b/examples/paul_graham_essay/TestEssay.ipynb
index c6b7baa57dbc5c9677ebf244bdb9aa9947163308..bc0dd8e8dae44ad700804e79cf040d3c796a6757 100644
--- a/examples/paul_graham_essay/TestEssay.ipynb
+++ b/examples/paul_graham_essay/TestEssay.ipynb
@@ -65,82 +65,82 @@
},
{
"cell_type": "code",
- "execution_count": 3,
- "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
- "metadata": {
- "tags": []
- },
+ "execution_count": 4,
+ "id": "46714db4-9592-4c55-9ca7-916758f2ce68",
+ "metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "> Starting query: What did the author do after his time at Y Combinator?\n",
- "> Answer: \n",
- "ANSWER: 3, 4, 6\n",
+ "> Starting query: What did the author do growing up?\n",
+ "> Selected node: \n",
+ "ANSWER: 1\n",
"\n",
- "These summaries were selected because they discuss what the author did after their time at Y Combinator. Summary 3 discusses how the author sold their company to Yahoo and then started painting. Summary 4 discusses how the author started writing essays and then eventually started their own investment firm. Summary 6 discusses how the author started working on a new programming language called Bel.\n",
- "> Node Summary text: The individual describes their experience working on a new website generator in Lisp with a partner. They explain how they came up with the idea to run the software on a server and allow users to control it through clicking links in a browser. They recount how they started a new company, Viaweb, with the goal of being easy and inexpensive to use. They describe how they recruited additional programmers and opened for business in January 1996. They explain how they realized that growth rate is the most important metric for a startup and recount how they reached breakeven a few years later. They describe how they sold the company to Yahoo in 1998 and how they used the money to start painting. They explain how they returned to New York a year later and had the idea to build a web app for making web apps.\n",
- "> Answer: \n",
- "ANSWER: 4, 5, 8\n",
+ "This summary was selected because it provides the most information about the author's background and what they did growing up. It discusses their experience with writing and programming before college, their studies in college, and their switch to Lisp programming. It also mentions their interest in art and their application to art school.\n",
+ "> Node Summary text: The individual worked on writing and programming before college. They attempted to write essays and stories, but they were not very good. In college, they studied philosophy but found it unfulfilling. They then switched to AI but realized that it was not going to work. They decided to focus on Lisp instead and wrote a book about it. They then took up art and applied to art school. They were accepted to RISD and then later to the Accademia di Belli Arti in Florence. They took the entrance exam and was accepted. They then had to learn Italian.\n",
+ "> Selected node: \n",
+ "ANSWER: 1\n",
"\n",
- "The author left Yahoo after his options vested and he made $2 million, as he had originally planned to do in order to paint. He returned to New York and picked up his old life, except now he was rich. He eventually had an idea for a new startup.\n",
- "> Node Summary text: opened for business, with 6 stores, in January 1996. It was just as well we waited a few months, because although we worried we were late, we were actually almost fatally early. There was a lot of talk in the press then about ecommerce, but not many people actually wanted online stores. [8] There were three main parts to the software: the editor, which people used to build sites and which I wrote, the shopping cart, which Robert wrote, and the manager, which kept track of orders and statistics, and which Trevor wrote. In its time, the editor was one of the best general-purpose site builders. I kept the code tight and didn't have to integrate with any other software except Robert's and Trevor's, so it was quite fun to work on. If all I'd had to do was work on this software, the next 3 years would have been the easiest of my life. Unfortunately I had to do a lot more, all of it stuff I was worse at than programming, and the next 3 years were instead the most stressful. There were a lot of startups making ecommerce software in the second half of the 90s. We were determined to be the Microsoft Word, not the Interleaf. Which meant being easy to use and inexpensive. It was lucky for us that we were poor, because that caused us to make Viaweb even more inexpensive than we realized. We charged $100 a month for a small store and $300 a month for a big one. This low price was a big attraction, and a constant thorn in the sides of competitors, but\n"
+ "The first summary provides the most information relevant to the question. It describes the author's work outside of school, which consisted of writing and programming.\n",
+ "> Node Summary text: \t\t What I Worked On February 2021 Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep. The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights. The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer. I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched\n"
]
},
{
"data": {
"text/plain": [
- "'The author went on to work on a variety of other projects.'"
+ "'The author grew up writing short stories and programming on an IBM 1401.'"
]
},
- "execution_count": 3,
+ "execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# try verbose=True for more detailed outputs\n",
- "new_index.query(\"What did the author do after his time at Y Combinator?\")"
+ "new_index.query(\"What did the author do growing up?\")"
]
},
{
"cell_type": "code",
- "execution_count": 4,
- "id": "46714db4-9592-4c55-9ca7-916758f2ce68",
- "metadata": {},
+ "execution_count": 3,
+ "id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
+ "metadata": {
+ "tags": []
+ },
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "> Starting query: What did the author do growing up?\n",
- "> Answer: \n",
- "ANSWER: 1, 2, 3\n",
+ "> Starting query: What did the author do after his time at Y Combinator?\n",
+ "> Selected node: \n",
+ "ANSWER: 3\n",
"\n",
- "These summaries were selected because they discuss the author's experience growing up and how they chose what to work on in the past.\n",
- "> Node Summary text: The individual worked on writing and programming before college. They attempted to write essays and stories, but they were not very good. In college, they studied philosophy but found it unfulfilling. They then switched to AI but realized that it was not going to work. They decided to focus on Lisp instead and wrote a book about it. They then took up art and applied to art school. They were accepted to RISD and then later to the Accademia di Belli Arti in Florence. They took the entrance exam and was accepted. They then had to learn Italian.\n",
- "> Answer: \n",
- "ANSWER: 1, 2, 3\n",
+ "This summary was selected because it provides the most information about the author's post-Y Combinator activities. It describes how the author started a new company and wrote a book on Lisp programming.\n",
+ "> Node Summary text: The individual describes their experience working on a new website generator in Lisp with a partner. They explain how they came up with the idea to run the software on a server and allow users to control it through clicking links in a browser. They recount how they started a new company, Viaweb, with the goal of being easy and inexpensive to use. They describe how they recruited additional programmers and opened for business in January 1996. They explain how they realized that growth rate is the most important metric for a startup and recount how they reached breakeven a few years later. They describe how they sold the company to Yahoo in 1998 and how they used the money to start painting. They explain how they returned to New York a year later and had the idea to build a web app for making web apps.\n",
+ "> Selected node: \n",
+ "ANSWER: 8\n",
"\n",
- "These summaries were selected because they describe the author's work and hobbies outside of school, which is what the question asks for.\n",
- "> Node Summary text: \t\t What I Worked On February 2021 Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep. The first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights. The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer. I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched\n"
+ "This summary was selected because it mentions that the author \"left [Y Combinator] in the summer of 1999\" which suggests that the author's time at Y Combinator had ended. Additionally, the summary mentions that the author \"went back to New York\" which suggests that the author was no longer in California.\n",
+ "> Node Summary text: it felt disconcertingly like working at Interleaf. Yahoo had given us a lot of options when they bought us. At the time I thought Yahoo was so overvalued that they'd never be worth anything, but to my astonishment the stock went up 5x in the next year. I hung on till the first chunk of options vested, then in the summer of 1999 I left. It had been so long since I'd painted anything that I'd half forgotten why I was doing this. My brain had been entirely full of software and men's shirts for 4 years. But I had done this to get rich so I could paint, I reminded myself, and now I was rich, so I should go paint. When I said I was leaving, my boss at Yahoo had a long conversation with me about my plans. I told him all about the kinds of pictures I wanted to paint. At the time I was touched that he took such an interest in me. Now I realize it was because he thought I was lying. My options at that point were worth about $2 million a month. If I was leaving that kind of money on the table, it could only be to go and start some new startup, and if I did, I might take people with me. This was the height of the Internet Bubble, and Yahoo was ground zero of it. My boss was at that moment a billionaire. Leaving then to start a new startup must have seemed to him an insanely, and yet also plausibly, ambitious plan. But I really was quitting to paint,\n"
]
},
{
"data": {
"text/plain": [
- "'The author grew up writing short stories and programming on an IBM 1401.'"
+ "'The author left Yahoo to start a new startup.'"
]
},
- "execution_count": 4,
+ "execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# try verbose=True for more detailed outputs\n",
- "new_index.query(\"What did the author do growing up?\")"
+ "new_index.query(\"What did the author do after his time at Y Combinator?\")"
]
},
{
diff --git a/examples/test_wiki/TestNYC.ipynb b/examples/test_wiki/TestNYC.ipynb
index a22c7f2c13a29071d126e2e44f6349bc735a18a0..c568e6a9d2270da7955bce8ffe636fd535482ee0 100644
--- a/examples/test_wiki/TestNYC.ipynb
+++ b/examples/test_wiki/TestNYC.ipynb
@@ -98,7 +98,7 @@
},
{
"cell_type": "code",
- "execution_count": 3,
+ "execution_count": 4,
"id": "68c9ebfe-b1b6-4f4e-9278-174346de8c90",
"metadata": {
"tags": []
@@ -109,12 +109,12 @@
"output_type": "stream",
"text": [
"> Starting query: What is the name of the professional women's basketball team in New York City?\n",
- "> Answer: \n",
+ "> Selected node: \n",
"ANSWER: 8\n",
"\n",
"This summary was selected because it mentions that New York City is home to the headquarters of the National Basketball Association.\n",
"> Node Summary text: New York City is home to the headquarters of the National Football League, Major League Baseball, the National Basketball Association, the National Hockey League, and Major League Soccer. The New York metropolitan area hosts the most sports teams in the first four major North American professional sports leagues with nine, one more than Los Angeles, and has 11 top-level professional sports teams if Major League Soccer is included, also one more than Los Angeles. The city has played host to more than forty major professional teams in the five sports and their respective competing leagues. Four of the ten most expensive stadiums ever built worldwide (MetLife Stadium, the new Yankee Stadium, Madison Square Garden, and Citi Field) are located in the New York metropolitan area. New York City is also known for its high rate of public transit use, more than 200,000 daily cyclists as of 2014, and many pedestrian commuters, making it the most energy-efficient major city in the United States. The city government was a petitioner in the landmark Massachusetts v. Environmental Protection Agency Supreme Court case forcing the EPA to regulate greenhouse gases as pollutants. In 2018, New York City announced a $1 billion investment to protect the integrity of its water system and to maintain the purity of its unfiltered water supply.\n",
- "> Answer: \n",
+ "> Selected node: \n",
"ANSWER: 4\n",
"\n",
"This summary was selected because it mentions the New York Liberty, the professional women's basketball team in New York City.\n",
@@ -127,19 +127,20 @@
"\"The professional women's basketball team in New York City is the New York Liberty.\""
]
},
- "execution_count": 3,
+ "execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
+ "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
"# try verbose=True for more detailed outputs\n",
"new_index.query(\"What is the name of the professional women's basketball team in New York City?\")"
]
},
{
"cell_type": "code",
- "execution_count": 4,
+ "execution_count": 6,
"id": "4fc3f18a-0ef9-453c-acf8-7aedd784cdcf",
"metadata": {},
"outputs": [
@@ -148,12 +149,12 @@
"output_type": "stream",
"text": [
"> Starting query: What battles took place in New York City in the American Revolution?\n",
- "> Answer: \n",
+ "> Selected node: \n",
"ANSWER: 2\n",
"\n",
"The second summary mentions the Battle of Long Island, which was fought in Brooklyn during the American Revolution. This is the most relevant summary to the question.\n",
"> Node Summary text: The American South. During construction in Foley Square in the 1990s, the African Burying Ground was discovered; the cemetery included 10,000 to 20,000 of graves of colonial-era Africans, some enslaved and some free.The 1735 trial and acquittal in Manhattan of John Peter Zenger, who had been accused of seditious libel after criticizing colonial governor William Cosby, helped to establish the freedom of the press in North America. In 1754, Columbia University was founded under charter by King George II as King's College in Lower Manhattan. The Stamp Act Congress met in New York in October 1765, as the Sons of Liberty, organized in the city, skirmished over the next ten years with British troops stationed there. The Battle of Long Island, the largest battle of the American Revolutionary War, was fought in August 1776 within the modern-day borough of Brooklyn. After the battle, in which the Americans were defeated, the British made the city their military and political base of operations in North America. The city was a haven for Loyalist refugees and escaped slaves who joined the British lines for freedom newly promised by the Crown for all fighters. As many as 10,000 escaped slaves crowded into the city during the British occupation. When the British forces\n",
- "> Answer: \n",
+ "> Selected node: \n",
"ANSWER: 3 The American Revolution is mentioned specifically in the third summary, which mentions the Battle of Long Island taking place in Brooklyn. This is the most relevant summary to the question.\n",
"> Node Summary text: act of 1799, children of slave mothers were to be eventually liberated but to be held in indentured servitude until their mid-to-late twenties. Together with slaves freed by their masters after the Revolutionary War and escaped slaves, a significant free-Black population gradually developed in Manhattan. Under such influential United States founders as Alexander Hamilton and John Jay, the New York Manumission Society worked for abolition and established the African Free School to educate Black children. It was not until 1827 that slavery was completely abolished in the state, and free Blacks struggled afterward with discrimination. New York interracial abolitionist activism continued; among its leaders were graduates of the African Free School. New York city's population jumped from 123,706 in 1820 to 312,710 by 1840, 16,000 of whom were Black.In the 19th century, the city was transformed by both commercial and residential development relating to its status as a national and international trading center, as well as by European immigration, respectively. The city adopted the Commissioners' Plan of 1811, which expanded the city street grid to encompass almost all of Manhattan. The 1825 completion of the Erie Canal through central New York connected the Atlantic port to the agricultural markets and commodities of the North American interior via the Hudson River and the Great Lakes. Local politics became dominated by Tammany Hall, a political machine supported by Irish and German immigrants. Several prominent American literary figures lived in New York during the 1830s and 1840s, including\n"
]
@@ -164,21 +165,62 @@
"'The Battle of Brooklyn and the Battle of White Plains.'"
]
},
- "execution_count": 4,
+ "execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
+ "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
"# try verbose=True for more detailed outputs\n",
"new_index.query(\"What battles took place in New York City in the American Revolution?\")"
]
},
{
"cell_type": "code",
- "execution_count": null,
+ "execution_count": 7,
"id": "97f3ddf1-8dc2-4fb8-831f-2c06649e0955",
"metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "> Starting query: What are the airports in New York City?\n",
+ "> Selected node: \n",
+ "ANSWER: 8\n",
+ "\n",
+ "This summary was selected because it mentions that New York City has \"several commercial airports.\"\n",
+ "> Node Summary text: New York City is home to the headquarters of the National Football League, Major League Baseball, the National Basketball Association, the National Hockey League, and Major League Soccer. The New York metropolitan area hosts the most sports teams in the first four major North American professional sports leagues with nine, one more than Los Angeles, and has 11 top-level professional sports teams if Major League Soccer is included, also one more than Los Angeles. The city has played host to more than forty major professional teams in the five sports and their respective competing leagues. Four of the ten most expensive stadiums ever built worldwide (MetLife Stadium, the new Yankee Stadium, Madison Square Garden, and Citi Field) are located in the New York metropolitan area. New York City is also known for its high rate of public transit use, more than 200,000 daily cyclists as of 2014, and many pedestrian commuters, making it the most energy-efficient major city in the United States. The city government was a petitioner in the landmark Massachusetts v. Environmental Protection Agency Supreme Court case forcing the EPA to regulate greenhouse gases as pollutants. In 2018, New York City announced a $1 billion investment to protect the integrity of its water system and to maintain the purity of its unfiltered water supply.\n",
+ "> Selected node: \n",
+ "ANSWER: 4\n",
+ "\n",
+ "This summary was selected because it mentions LaGuardia Airport and John F. Kennedy International Airport, which are both airports in New York City.\n",
+ "> Node Summary text: There have been 35 Major League Baseball World Series and 73 pennants won by New York teams. It is one of only five metro areas (Los Angeles, Chicago, Baltimore–Washington, and the San Francisco Bay Area being the others) to have two baseball teams. Additionally, there have been 14 World Series in which two New York City teams played each other, known as a Subway Series and occurring most recently in 2000. No other metropolitan area has had this happen more than once (Chicago in 1906, St. Louis in 1944, and the San Francisco Bay Area in 1989). The city's two Major League Baseball teams are the New York Mets, who play at Citi Field in Queens, and the New York Yankees, who play at Yankee Stadium in the Bronx. These teams compete in six games of interleague play every regular season that has also come to be called the Subway Series. The Yankees have won a record 27 championships, while the Mets have won the World Series twice. The city also was once home to the Brooklyn Dodgers (now the Los Angeles Dodgers), who won the World Series once, and the New York Giants (now the San Francisco Giants), who won the World Series five times. Both teams moved to California in 1958. There is also one Minor League Baseball team in the city, the Mets-affiliated Brooklyn Cyclones, and the city will gain a club in the independent Atlantic League when the Staten Island FerryHawks begin play in 2022.The city is represented in the National Football League by the New\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'The airports in New York City are John F. Kennedy International Airport, LaGuardia Airport, and Newark Liberty International Airport.'"
+ ]
+ },
+ "execution_count": 7,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# GPT doesn't find the corresponding evidence in the leaf node, but still gives the correct answer\n",
+ "# try verbose=True for more detailed outputs\n",
+ "new_index.query(\"What are the airports in New York City?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "53265fd4-da98-4cf9-abfb-3f76105fd2ff",
+ "metadata": {},
"outputs": [],
"source": []
}
diff --git a/gpt_index/index.py b/gpt_index/index.py
index 5ee67f1a8804ba24ecaeaecf71762db027052397..5ce559655f664f5a8b8850404bc3f29c355746c4 100644
--- a/gpt_index/index.py
+++ b/gpt_index/index.py
@@ -175,7 +175,7 @@ class GPTIndex(DataClassJsonMixin):
# number is 1-indexed, so subtract 1
selected_node = cur_node_list[number-1]
- print(f"> Answer: {response}")
+ print(f"> Selected node: {response}")
print(f"> Node Summary text: {' '.join(selected_node.text.splitlines())}")
if len(selected_node.child_indices) == 0: