ScreenSense

AI model · Computer vision · Image → Text by Whatfix

AI that understands the application context

ScreenSense is Whatfix's proprietary AI context engine that interprets real-time application context and user intent. It powers Whatfix AI by analyzing on-screen actions across web, desktop, mobile, and VDI environments to deliver relevant in-flow guidance.

Model specs

The full spec card (context, pricing per token, modalities and more) on AntSpace. Sign up free to see the full spec card.

Languages
English

Category: Computer Vision

License
Proprietary
Architecture
Proprietary context engine for LLM augmentation
Supported Platforms
Web, Desktop, Mobile, Citrix, VDI
Primary Function
Real-time application context and user intent interpretation
  • Interprets what users see and do on-screen
  • Powers real-time context for AI agents
  • Supports Web, Desktop, Mobile, and VDI
  • Proprietary context engine for LLM augmentation

Overview

ScreenSense is Whatfix's proprietary AI context engine designed to understand and interpret real-time application context and user intent. It serves as the foundational technology for Whatfix AI, enabling intelligent actions across enterprise application landscapes by analyzing what users see and do on their screens. The engine supports a wide range of environments, including web, desktop, mobile, and virtual desktop interfaces like Citrix and Azure Virtual Desktop (VDI). By augmenting large language models with this proprietary context, ScreenSense allows for the delivery of highly relevant, in-flow guidance and automated workflow execution. It is primarily used by enterprise teams to power digital adoption strategies and improve user productivity within complex software ecosystems.

Core Capabilities

  • Application Context Interpretation: The engine interprets what users see and do on their screens to provide real-time relevance for guidance and automation.
  • User Intent Recognition: It identifies user intent based on live behavior and application state to trigger appropriate assistance.
  • In-Flow Action Enablement: The model enables intelligent, in-flow actions across diverse enterprise application landscapes without manual configuration.
  • Contextual Guidance Support: It powers the ability to summarize knowledge and guide users step-by-step through complex tasks based on the current screen state.

Platform Coverage

  • Web Application Support: The engine functions across web-based enterprise applications to provide contextual awareness.
  • Desktop Application Support: It provides context awareness for native desktop software environments.
  • Mobile Application Support: ScreenSense interprets interfaces on mobile devices for consistent cross-platform adoption.
  • Virtual Desktop Support: The technology extends to virtualized environments including Citrix and Azure Virtual Desktop (VDI).

Use cases

  • Enterprise teams use ScreenSense to provide real-time, contextual guidance to employees working in complex software like Salesforce or SAP.
  • Support teams leverage the engine to automatically surface relevant knowledge base articles based on the user's current screen state.
  • Digital adoption managers use it to trigger automated workflow steps that help users complete unfamiliar back-office tasks accurately.

Specifications

Tasks
Application context interpretation, user intent recognition, screen analysis
License
Proprietary
Architecture
Proprietary context engine for LLM augmentation
Supported Platforms
Web, Desktop, Mobile, Citrix, VDI
Primary Function
Real-time application context and user intent interpretation

How it works

  1. Analyze Screen Context: ScreenSense continuously analyzes the user's screen state and actions within enterprise applications.
  2. Interpret User Intent: The engine interprets the application state and user behavior to identify the specific task or intent.
  3. Augment AI Models: This proprietary context is used to augment large language models, ensuring AI responses are grounded in the current application state.
  4. Deliver In-Flow Guidance: The system triggers relevant in-app guidance, summaries, or automated steps directly in the user's workflow.

FAQ

What is ScreenSense?

ScreenSense is Whatfix's proprietary AI engine that interprets what users see and do on their screens to provide real-time application context.

Which platforms does ScreenSense support?

It supports a wide range of environments including web, desktop, mobile, and virtual desktop interfaces like Citrix and Azure Virtual Desktop (VDI).

How does ScreenSense power Whatfix AI?

It enables Whatfix's AI agents to deliver highly relevant, in-flow guidance and automated workflow execution by augmenting large language models with real-time application context.

Added by AntSpace from public sources (whatfix.com).

More AI models by Whatfix