[ad_1]
OpenAI has built a version of GPT-4, its latest text generation model, that can “remember” up to 50 pages of content thanks to a significantly larger context window.
That may not be significant. But vanilla GPT-4 has five times the capacity in “note” and eight times more than GPT-3.
“The model can use long documents dynamically,” OpenAI co-founder and president Greg Brockman said in a live demo this afternoon. “What kind of applications do we want to see? [this enables]He said.
In the case of text generation AI, the context window indicates the text the model is looking at before generating more text. While models like GPT-4 “learn” to write by training on billions of examples of text, they can only consider a small portion of that text at a time – determined primarily by the size of the context window.
Models with small context windows “forget” the content of even the most recent conversations, causing them to go off-topic. After a few thousand words, they also forget their initial instructions, instead deriving their behavior from the last information in the context window rather than from the original question.
Alan Pike, a former software engineer at Apple, famously described it this way.
“[The model] He forgets everything you are trying to teach him. It will forget you live in Canada. He forgets that you have children. You forget that you hate to book things on Wednesdays and please stop suggesting things on Wednesdays, you idiot. If neither of you mentions your name for the moment, he will forget that too. Contact a [GPT-powered] A character for a while, and you can feel like friendship with him, you will find a very cool place. Sometimes it gets a little confusing, but that happens to people too. Eventually, however, it becomes clear that he has no medium-term memory, and the illusion is shattered.
We haven’t been able to get our hands on the GPT-4 version of the extended context window gpt-4-32k yet. (OpenAI says it’s handling requests for high- and low-context GPT-4 models at “different rates based on capacity”. Gen. Model.
On a big “note” GPT-4 should be able to talk for hours – days, even – as opposed to minutes. And perhaps most importantly, it should be less likely to go off the rails. As Pike notes, one of the reasons chatbots like Bing Chat behave badly is that their initial instructions—to be a helpful chatbot, respond politely, and so on—are quickly pushed out of context by more questions and responses.
It may be a little more exaggerated than that. But the context window plays an important role in grounding the models. Without a doubt. Over time, we’ll see what real difference it makes.
[ad_2]
Source link
