Context Graph

Brian Takita
Authors:Brian Takita
Posted on:May 24, 2024

Exploration on the utility of Context Graphs. The term Context Graph emerged in a conversation between Eric & me. To describe how LLM Context Prompts can be represented as a graph.

This post is a deeper dive into the concept & utility of Context Graph. Along with:

  • prompt + response nodes
  • related & adjacent graph nodes
  • representations of data

Exploring “Context Graph”

Prior Art

Mozilla Context Graph

Mozilla had a pilot project called Context Graph. Based on the manifesto by Mozilla Engineer Nick Nguyen.

From a Prescience conversation.

You: What happened to Mozilla's "Context Graph" project?

Presence: Mozilla's "Context Graph" project was an initiative aimed at helping users discover and contextualize information more effectively while browsing the web. The project sought to provide smart recommendations and contextual links to enrich the user's web experience by leveraging data from Mozilla’s broader ecosystem. As with many experimental projects, the Context Graph project was intended to enhance user experience through innovative technologies.

The project has since been discontinued. Here are some key points regarding its history and eventual discontinuation:

  1. Origins and Goals:
  • The Context Graph project was initiated around 2015 with the aim of creating a knowledge graph-like architecture to surface relevant information and provide a more intuitive browsing experience.
  • It included features like smart recommendations, contextual links, and enhanced search capabilities.
  1. Development and Features:
  • Mozilla experimented with various features and integration points for the Context Graph within its Firefox browser.
  • The project included tools like "Activity Stream," "Contextual Suggestions," and other recommendation engines that aimed to predict and suggest content based on user behavior.
  1. Challenges and Discontinuation: Despite initial enthusiasm, Mozilla faced several challenges, including technical hurdles, user privacy concerns, and the need for substantial data to train recommendation algorithms effectively. By mid-2017, Mozilla decided to wind down the Context Graph project. The official reason cited was a shift in focus toward other priorities and projects that Mozilla felt would have a greater impact on the web ecosystem.
  2. Legacy and Impact: Some of the project's concepts and partially developed features influenced other Mozilla initiatives and products. While the Context Graph itself was discontinued, its ideas lived on through various Firefox enhancements and contributions to open-source projects.

Mozilla continues to innovate and explore ways to improve the browsing experience with a strong focus on user privacy and security. The Context Graph project is a testament to their commitment to experimentation and user-centric design, even if not all projects make it to long-term implementation.

Attention

No One Knows what Attention Is

The article No One Knows what Attention[1] claims[2]:

Multiple Processes

Attention encompasses multiple processes. Making it difficult to define as a single system.

Compartmentalization Issues

Compartmentalizing attention & searching for an "attentional system" hinders understanding of human behavior. Ignoring the integrated, parallel, & reciprocal relationships among sensory, cognitive, & action processes.

Comparison with Memory

Similar arguments have been made about memory. Suggesting that various types of memory may not reflect separable systems. But rather different byproducts of cognitive systems.

Dynamic & Continuous Nature

Attention, selection, & intention are dynamic & continuous. Embedded within a densely interconnected, parallel processing system.

Conceptual Challenges

Using everyday language terms like "attention" & "intention" in scientific contexts leads to misconceptions. As these terms do not map neatly onto identifiable brain mechanisms.

Synthetic Approach

A synthetic approach, focusing on the interactions between sensory, motor, & cognitive phenomena. Is proposed to understand selective behavior & the mechanisms underlying it.

Reward & Selection History

The synthetic approach integrates reward & selection history. Explaining how different stimuli & contexts influence behavior.

Modeling Attention

Attention Is All You Need

Brain vs Mind or Emergent Phenomena

Consciousness as an Emergent Phenomenon: A Tale of Different Levels of Description

Salience

Relevance

Relevance Model

Authoring a Relevance Model

Iterative Feedback Loop

Revisions

Forking

Intention

Expressing Intention

Deriving Intention

[1]:

No one knows what attention is

[2]:

From No one knows what attention is

We are not the first to raise concerns about problems with the term “attention.” Multiple authors have highlighted the tendency to reify attention, creating circular explanations for empirical results

multiple processes underlie what is typically labeled as “attention”

We reaffirm these positions and further suggest that compartmentalizing “attention” and then searching for the “attentional system” hinders the development of a comprehensive understanding of human behavior because it ignores integrated, parallel, and reciprocal relationships among sensory, cognitive, and action processes.

very similar arguments to those that we present in the following have been put forward to question the concept of memory. Decades of research on human memory have seen an ever-increasing number of memory systems that were thought to represent separable aspects of memory performance, which then were thought to be explained by the existence of corresponding memory systems, with rather limited contributions to a mechanistic understanding of the underlying processes (Bechtel, 2008). As recent considerations suggest, however, the various types of memory may not at all reflect the operations of separable dedicated systems, but rather stand for different byproducts of normally functioning cognitive systems (Buckner & Schacter, 2004), and emerged at different times during the evolution of our species (Murray, Wise, & Graham, 2017). Similar arguments have been put forward for the concept of emotion (Barrett, 2017; Hommel, 2019b) and may be developed for other concepts as well, including “cognition” itself (Cisek, 2019). We focus here on attention because we believe that at least some of the related phenomena are best understood in terms of the kinds of interactions between sensory, motor, and cognitive phenomena that are the focus of this special issue.

the characteristics of the physically executed action actually reflect the “attentional” state of the target and non-target stimuli. Collectively, these data indicate that attention, selection, and intention are not readily separated in a set of discrete serial processes, but are more dynamic and continuous in nature and embedded within a densely interconnected, parallel processing system.

In an analytic approach to science, one runs the risk of becoming a slave to the concepts that have been generated. Many researchers have taken terms like “attention,” “intention,” and “decision making” from everyday language and expect this linguistic categorization to somehow map to identifiable mechanisms in the brain or functions. Of course, when one starts to peer into actual neural functions, there is no clear delineation, only a set of processes that interact to create selectivity in the end. These processes interact not because they belong to a dedicated system, but because the human brain and body evolved this way and selectivity was a necessary feature to achieve efficient behavior. Further, everything an individual does throughout their life (distant and recent past) creates, reinforces, and shapes selection: Turning to the left makes us ignore stimuli on the right, picking one apple makes us overlook the others, saying one word prevents us from uttering any other. And each of the different selections results in all ranges of rewards, from positive gains to negative losses. Selection and reward are thus inherent ingredients of all our lives and the way we lead them (Allport, 1987).

To produce selective behavior, multiple, inter-related processes integrate numerous sources of information. One of the challenges is that these processes unfold over different timeframes (e.g., Chapman et al., 2015b; Welsh, Neyedli, & Tremblay, 2013). Therefore, in a laboratory setting, if these processes are only observed during one point or snapshot during the selection process, the observation could appear to reflect “attention” or “intention” or “decision making and reward.” The synthetic approach proposed here also rectifies and makes explicit that reward and selection history are intertwined subjects, but likely reflect multiple processes that contribute to goal-oriented behavior. For example, the synthetic approach can account for harm avoidance. Specifically, harmful stimuli should receive priority processing for detection, yet the organism should move away from these stimuli. The primitive neural circuits for reward/approach and harm/avoid processes diverge early in evolutionary history, providing a process-based account for divergent findings regarding positive and negative value-based stimuli. Likewise, the synthetic approach explains why sensitivity to different features of objects depends on the action context (Bekkering & Neggers, 2002; Craighero et al., 1999; Fagioli et al., 2007; Welsh & Pratt, 2008) – because the context determines which action-centered parietal stream, with its idiosyncratic representation of the external world, is being selectively invigorated at a given time.

One of the great conundrums in experimental psychology and neuroscience is exactly how all of these streams of information diverge from initial sensory areas and then converge to produce action. Working backwards from what researchers observe in behavior, it is known that generally only one goal-directed movement is performed at a time, though more than one might be simultaneously represented (e.g., Cisek & Kalaska, 2005). As discussed in the section on evolutionary adaptations, we advocate for a parallel competitive structure with winner-take-all dynamics resolving to produce a single action for each action system (e.g., hand and eye). Of course, much of the detail about how this occurs is an open question and beyond the scope of this article. What we hope to emphasize here is the synthetic approach to understanding how complex sensory information is transformed into action. The corollary argument is that progress is hindered when we appeal to or attempt to apply catch-all terms like “attention.” Thus, rather than saying that an individual “pays more attention to a physically salient stimulus,” one should make an attempt to understand the mechanism by which physical salience translates to more efficient processing and behavior. Instead of arguing that rewarding stimuli “demand more attention,” provide a description of how a particular reward is associated with a particular target, and how, perhaps even more astoundingly, the cognitive system/brain then recalls this association in a fraction of a second to guide behavior on a subsequent trial. Experiment to figure out how and why visual information presented at a location selected for action is amplified, rather than passing the finding off as “just attention.” Hence, turn to the mechanisms that we understand and try to re-create the behavior that cognitive and neural scientists are interested in. If that approach turns out to be successful, there will be no need for undefinable concepts like attention, either in describing the explanandum or in describing the explanans.