Google announces new features for Circle to Search and Gemini on Android

At its I/O conference today, Google has unveiled a new feature for Circle to Search. It can now help students with homework. When a student circles a prompt they're stuck on, they will receive step by step instructions to solve "a range of physics and math word problems without leaving their digital info sheet or syllabus".

And "later this year" Circle to Search will be able to help solve more complex problems involving symbolic formulas, diagrams, and graphs. This is all made possible by Google's new LearnLM, an effort to enhance its products and models for learning.

Google says Circle to Search is already available on more than 100 million devices today, and it will land on 100 million more by the end of the year.

Gemini on Android will "soon" understand what app you're in and what you're doing inside it in order to be more helpful. So if you're watching a YouTube video, you'll be able to ask Gemini about that video. It will also let you drag and drop generated images into Gmail, Google Messages, and "other places".

If you have Gemini Advanced, then you will also be able to ask it things about PDF files you're viewing, without having to scroll through multiple pages. This will roll out to "hundreds of millions of devices" over the next few months.

Gemini Nano, the on-device foundation model, is getting an update "later this year" on Pixel devices, as it becomes Gemini Nano with Multimodality. So it won't just process text input locally on-device, but it will also understand more information in context like sights, sounds, and spoken language.

Gemini Nano's multimodal capabilities are coming to TalkBack "later this year", helping blind or low vision people to "get richer and clearer descriptions of what's happening in an image". And since Gemini Nano is on-device, the descriptions happen quickly and work even when there's no network connection.

Finally, Google says it's testing a new feature that uses Gemini Nano to provide real-time alerts during a call if it detects "conversation patterns commonly associated with scams". The example Google gives is if a "bank representative" asks you to urgently transfer funds, that sort of thing.

The protection all happens on-device, so the conversation "stays private to you". This will be an opt-in feature and Google promises it will tell us more "later this year".

Source