Google Gemini: AlphaGo-GPT?

In a somewhat provocative new interview with Wired Magazine, Demis Hassabis, head of Google DeepMind, is quoted as saying that Gemini, which could be released as soon as this winter, will be more capable than OpenAI's ChatGPT.

He reveals that they are attempting to combine some of the strengths of AlphaGo type systems with the amazing language capabilities of large models.

Before we look into how that might work, here is the context of the Gemini announcement from Sundar Pichai.

They are focused on building more capable systems safely and responsibly.

This includes our next generation foundation model, Gemini, which is still in training.

While still early, we are already seeing impressive multimodal capabilities not seen in prior models.

Hassabis promises that we also have some new innovations that are going to be pretty interesting.

And I know many people will dismiss this as all talk, but remember DeepMind was behind not just AlphaGo, but also AlphaZero, which can play any two-player full information game from scratch.

They were also behind Alpha Star, which conquered Starcraft 2 with quote, long-term planning.

And let's remember that for later.

And most famously, perhaps, Asaba's led them to the incredible breakthrough of Alpha Fold and Alpha Fold 2, which are already impacting the fight against plastic pollution and antibiotic resistance.

So let's not underestimate DeepMind.

But back to Gemini, we hear from the information recently that the multimodality of Gemini will be helped in part by training on YouTube videos.

And apparently YouTube was also mined by OpenAI.

Of course, that's not just the text transcripts, but also the audio, imagery, and probably comments.

I wonder if Google DeepMind might one day use YouTube for more than that.

A few days ago, they released this paper on RoboCat, which they call a self-improving foundation agent for robotic manipulation.

And the paper says that with RoboCat, we demonstrate the ability to generalize to new tasks and robots, both zero-shot as well as through adaptation using only 100 to 1,000 examples for the target task.

We also show how a trained model itself can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop.

Notice that part about using the model itself to generate data.

That reminded me of a conversation I had with one of the authors of the textbooks are all you need paper, Ronan Eldan from Microsoft.

I'm making a video on their new Phi-1 model for coding.

We had a really great chat and we were discussing at one point AGI timelines.

And I said this, when you get elite math papers with proofs and elite scientific research, if you train on much more of those for way more epochs, I don't think we're that far away from AGI.

I personally can't see any barrier within the next five years.