In the past, to enable multimodal models to understand a video, the common approach was to extract frames at fixed intervals and then feed the video footage, audio, and subtitles together into the model. While this method was simple, it had a clear drawback: regardless of whether the issue arose at the tenth second or two hours into the video, the system had to pay for the entire piece of content first. On September 1st, Google launched a proxy-based video understanding approach with Gemini, aiming to reverse this process—first, the model determines where to search, and then it performs encrypted sampling on the target segment and conducts repeated checks.
This capability has already been used for uploading videos and YouTube videos, covering Gemini 3.7, Flash, 3.6, Flash, and 3.5 Flash-Lite. It can be invoked through Gemini API, Google AI Studio, and Gemini Enterprise Agent Platform. Google also sets clear boundaries: the widespread adoption in Gemini applications will “soon” begin, while the Ask feature in YouTube will have to wait for the next few months. Therefore, it is not a unified upgrade that all consumers can use directly today.
Official tests indicate that, compared to the default processing method of one frame per second, the new approach can reduce video Token by up to 88%, lower costs by 66%, and in some evaluations, improve response quality by up to 7%. These figures represent results under the selected tasks and settings of Google; they cannot be directly extrapolated to mean that the same proportion of savings can be achieved for every video. What is truly noteworthy is that video understanding now resembles a search-driven approach rather than a mechanical scan from beginning to end.
The model first searches, then locates, and finally reviews; there's no need to expand the entire long video.
The core of proxy-based processing is to allow the model to autonomously choose tools within a cycle of “search—scan—check.” When a user asks when a certain action occurred, the model can first read subtitles, audio clues, and sparse footage to narrow down the time range; after finding candidate segments, it increases the frame extraction density to confirm the relationships between objects, characters, and actions frame by frame. If there is insufficient evidence, it can continue searching for another time period instead of immediately answering based on the first sample.
This difference is particularly evident in long videos. In a two-hour meeting recording, users may only want to know when a speaker promised to deliver something; in sports recordings, the question might be what happened before a foul; in industrial inspection videos, it could be necessary to identify a few frames of abnormal sparks. Fixed-frame sampling would encode a large amount of irrelevant footage, while low-frequency sampling might just miss the key moments. Dynamic strategies focus computing power on the “worth watching” parts, which not only reduces context consumption but also gives the model the opportunity to revisit the original evidence for verification.
The applicable tasks listed in Google include sub-second moment retrieval, "needle-in-a-haystack" searches within long segments, anomaly detection, and counting. It should not be mistakenly assumed that "able to search" means "never miss a detection." Video compression, subtitle errors, camera cuts, occlusions, and events that are too brief can still result in incorrect candidates in the initial search. The proxy will use more steps, which also means that the delay and the process of calling trajectories are more complex than a single inference. What is mainly saved is irrelevant video Token; however, this does not mean that each request will be faster.
For developers, another change is observability. In the past, failures were often merely indicated by incorrect answers; it was difficult to determine whether the model failed to recognize key frames or misunderstood them despite recognizing them. With tools in place, processes can include steps for searching for relevant areas, resampling data, and reviewing results. As a result, teams can break down evaluations into two parts: whether the correct information was found and whether the model understood it correctly. If Google continues to provide sufficient process records in production interfaces, debugging video applications will be more controllable than with purely black-box question-and-answer systems.
A decrease in costs will open up new opportunities, but permissions, evidence, and evaluation standards must not be relaxed accordingly.
Google states that this capability adopts the standard Token pricing and there are no additional functional fees. For teams that need to process a large volume of material, this is even more important than a single demonstration. Customer service can locate the moment of operational failure in the recorded screen, media organizations can search for a particular speech segment in the authorized material library, and manufacturing companies can first let the system filter out a small number of suspicious periods before submitting them for manual review. Tasks that were previously not economically feasible due to the high cost of full encoding may now become viable.
However, cost reduction can also exacerbate governance issues. Videos often contain multiple elements such as faces, voices, locations, screen content, and information about bystanders. An improved search efficiency does not automatically grant the authority to process this data. Enterprises must clearly define which materials are allowed to be uploaded, for how long they can be stored, who is permitted to access them, and whether the results can be used for personnel, security, or compliance assessments. This is especially true in monitoring and medical scenarios, where the time points indicated by the models should be considered as potential evidence, rather than unreviewed factual conclusions.
The evaluation method also needs to be changed. Simply counting the final answer accuracy rate will overlook cases where the system happens to guess correctly but identifies the wrong segment. A more reliable test should check at least the positioning interval, key frame coverage, basis for answers, and expression of uncertainty separately. Additionally, videos without target events should be included to observe whether the model will fabricate a time point in order to complete the task. For counting tasks,遮挡, repeated occurrences, and camera transitions should be tested; for anomaly detection, stability under low light conditions, noise, and with different cameras should be considered.
This update did not require Gemini to “watch all videos,” but instead taught it to watch selectively. For the AI product, this represents a more realistic path for expansion: rather than continuously expanding the context and stuffing all materials into the model at once, the model is first guided to establish a strategy for extracting information, and then expensive visual computing resources are used in only a few key areas. Saving Token is just a superficial benefit; the real breakthrough is that video understanding has evolved from a single inference process to a workflow that allows for checking, iteration, and refinement.
