Connectors Framework

@memofs/connectors is the local data ingestion framework for MemoFS. It connects external systems — such as GitHub issues/PRs/discussions, Notion workspaces/databases, Linear tickets, or custom internal APIs — directly into local MemoFS memory (.memofs/).

Following MemoFS's file-first and single-writer architecture, connectors execute strictly on the local machine. The host application provides its existing MemoFS instance, and the framework safely persists external items as durable markdown notes with content-derived identifiers and provenance metadata. Only the resulting memory files replicate to the cloud — authentication tokens, API keys, and raw secrets never touch the file replica.

Subpath Exports

@memofs/connectors is packaged with a single unified entry point:

Installation

Install @memofs/connectors alongside @memofs/core:

npm install @memofs/connectors @memofs/core

Quick Start

The host application initializes a MemoFS client (maintaining the single-writer contract per .memofs/ root) and passes it to runConnectors:

import { createNodeMemoFs } from "@memofs/core/node-fs";
import { runConnectors, EnvSecretResolver } from "@memofs/connectors";

const rootDir = ".";
const memo = createNodeMemoFs({ rootDir });

const result = await runConnectors({
  rootDir,
  memo,
  secretResolver: new EnvSecretResolver({ rootDir }),
});

console.log("Connectors executed:", result.ran);
console.log("Newly written note IDs:", result.written);
console.log("Skipped (already ingested):", result.skipped);
if (result.errors.length > 0) {
  console.warn("Recoverable errors encountered:", result.errors);
}

import { MemoFS } from "@memofs/core";
import { createNodeFsMemoryStore } from "@memofs/core/node-fs";
import { runConnectors, EnvSecretResolver } from "@memofs/connectors";

const rootDir = ".";
const store = createNodeFsMemoryStore({ rootDir });
const memo = new MemoFS({ store, projectId: "my-project", mode: "local" });

const result = await runConnectors({
  rootDir,
  memo,
  secretResolver: new EnvSecretResolver({ rootDir }),
});

console.log("Connectors executed:", result.ran);
console.log("Newly written note IDs:", result.written);

How It Works

1. Configuration (.memofs/connectors.json): The 11th canonical sync unit in MemoFS (CONNECTORS_PATH = ".memofs/connectors.json"). It defines which connectors to run and what to fetch. Each connector row references credentials through an opaque secretRef, never an inline token.
2. Secret Resolution: At runtime, the runner passes each secretRef to an injected SecretResolver (e.g. EnvSecretResolver in local development or CloudSecretResolver in production). The resolved token resides in memory only, is never logged, and is never written to disk.
3. Ingestion & Normalization: The runner invokes each enabled connector's ingest(ctx) method. The connector queries its remote API and normalizes items into standard ConnectorRecord objects without writing files directly.
4. Deduplication & Write Discipline: The runner calculates a deterministic note ID for each record (conn_<sha256[:16]>). It checks against previously written note IDs in the project's event history. Unchanged items are skipped; new or updated items are written through memo.writeMemory() with source: "connector".
5. Sync & Recall: The newly written markdown notes, events, and derived index chunks replicate to the cloud and become immediately available to agent queries via hybrid recall (memo.context(), memo.recall(), or MCP tools).

The Connector-Write Discipline

Every connector-emitted note written to MemoFS adheres to four strict guarantees:

Idempotency & Safe Retries

Because note IDs are derived from externalId + content, running connectors repeatedly or resuming after a partial network failure is 100% idempotent:

• Items whose external content has not changed produce identical IDs and are skipped during the deduplication check (result.skipped).
• Items whose content has changed produce a new deterministic ID and are ingested as an updated note.
• Retrying an aborted run writes only the items that were not completed, never creating duplicates.

Deduplication Mechanics

When runConnectors executes, it builds a deduplication index of existing note IDs via loadExistingNoteIds:

1. Primary path (Complete Event Stream): Reads all historical events from .memofs/events/memory-events.jsonl using readMemoryEvents(memo.store, { malformedLineMode: "skip" }) and extracts every recorded note ID (event.metadata.id).
2. Fallback path (Recent Memories): If the store does not expose an event log directly, falls back to memo.listRecentMemories({ limit: 500 }).
3. In-Run Tracking: When a record is written during the active pass, its ID is immediately added to the in-memory set to prevent duplicate writes within the same batch.

Workspace Configuration (connectors.json)

Connectors are configured in .memofs/connectors.json:

{
  "$schema": "../node_modules/@memofs/cli/schema/connectors.json",
  "connectors": [
    {
      "id": "github-main",
      "type": "github",
      "enabled": true,
      "schedule": "@hourly",
      "sourceMapping": {
        "repository": "owner/repo",
        "kinds": ["issues", "prs", "discussions"],
        "limit": 50
      },
      "secretRef": "gh_pat"
    },
    {
      "id": "team-notion",
      "type": "notion",
      "enabled": true,
      "sourceMapping": {
        "databaseId": "12345678-1234-1234-1234-123456789abc",
        "limit": 50
      },
      "secretRef": "notion_key"
    }
  ]
}

Configuration Schema (ConnectorConfig)

Config Validation Guardrails

To ensure security across replicated repositories, validateConnectorsFile() enforces strict validation on the entire .memofs/connectors.json payload, recursively inspecting every key and value at all depths:

1. Forbidden Top-Level Field Names: Any connector row containing token, secret, apiKey, apikey, or access_token is immediately rejected.
2. Recursive Key Substring Check: Every key across all nesting levels (including inside sourceMapping) is checked case-insensitively. If a key contains the substring token, secret, apikey, api_key, or password (and does not lowercase to exactly secretref), validation fails.
3. Secret Value Pattern Detection: Every string value anywhere in the file is scanned against known credential patterns:

Secret Resolution

Secrets are resolved dynamically at runtime through implementations of the SecretResolver interface:

export interface SecretResolver {
  resolve(secretRef: string): Promise<string>;
}

1. EnvSecretResolver (Local Development)

Reads a { "secretRef": "token" } map from the local, gitignored file .memofs/secrets.json:

{
  "gh_pat": "ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "notion_key": "secret_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}

import { EnvSecretResolver } from "@memofs/connectors";

const secretResolver = new EnvSecretResolver({ rootDir: "." });

2. CloudSecretResolver (Production & Hosted CLI)

Fetches decrypted credentials on demand from MemoFS Cloud using the project's Bearer API key:

import { CloudSecretResolver } from "@memofs/connectors";

const secretResolver = new CloudSecretResolver({
  projectId: "proj_abc123",
  apiKey: "tm_live_xxxxxxxxxxxx",
  cloudBaseUrl: "https://memofs.dev/api/v1",
});

Calls GET {cloudBaseUrl}/projects/:projectId/connectors/secret?ref=:secretRef with Authorization: Bearer <apiKey> and Accept: application/json.

3. StaticSecretResolver (Testing & Embedding)

Uses an in-memory dictionary. Ideal for automated testing or environments where secrets are pre-loaded in memory:

import { StaticSecretResolver } from "@memofs/connectors";

const secretResolver = new StaticSecretResolver({
  gh_pat: "ghp_mock_token_for_tests",
});

4. Custom Secret Resolver

You can implement SecretResolver to integrate with enterprise secret managers (HashiCorp Vault, AWS Secrets Manager, 1Password CLI, etc.):

import type { SecretResolver } from "@memofs/connectors";
import { ConnectorSecretError } from "@memofs/connectors";

class VaultSecretResolver implements SecretResolver {
  async resolve(secretRef: string): Promise<string> {
    const token = await fetchFromVault(secretRef);
    if (!token) {
      throw new ConnectorSecretError(secretRef, `Secret "${secretRef}" not found in Vault.`);
    }
    return token;
  }
}

Connector Registry

The ConnectorRegistry maps string type identifiers to Connector instances:

import {
  ConnectorRegistry,
  createConnectorRegistry,
  GitHubConnector,
  NotionConnector,
} from "@memofs/connectors";

// 1. Create a registry seeded with built-in connectors (GitHub + Notion)
const registry = createConnectorRegistry();

// 2. Register custom connectors
registry.register(new CustomJiraConnector());

// 3. Inspect registered connectors
console.log("Supported types:", registry.types()); // ["github", "notion", "jira"]
console.log("Has notion:", registry.has("notion")); // true

Runner Execution & Lifecycle

When runConnectors(options) is called:

interface RunConnectorsOptions {
  rootDir: string;
  memo: MemoFS;
  secretResolver: SecretResolver;
  connectorRegistry?: ConnectorRegistry;
  onlyType?: string;
  signal?: AbortSignal;
}

The runner performs the following steps:

1. Loads and validates .memofs/connectors.json via readConnectorsFile(rootDir).
2. Filters connectors using selectConnectors(file, { type: onlyType }).
3. Pre-loads existing note IDs from the event log for deduplication.
4. For each selected connector:
   • Resolves the token via secretResolver.resolve(config.secretRef).
   • Calls connector.ingest(ctx).
   • Computes deterministic note IDs for each returned ConnectorRecord.
   • Writes new/changed records to MemoFS memory.
   • Collects per-item and per-connector errors into result.errors without halting other connectors.
5. Returns RunConnectorsResult:

interface RunConnectorsResult {
  readonly written: readonly string[];          // Note IDs created (e.g. ["conn_a1b2c3d4e5f60718"])
  readonly skipped: readonly string[];          // External IDs skipped (e.g. ["issues:42"])
  readonly errors: readonly ConnectorIngestError[]; // Recoverable errors
  readonly ran: readonly string[];              // Connector IDs attempted (e.g. ["github-main"])
}

Error Handling

All connector framework errors inherit from ConnectorError and expose a stable machine-readable .code property:

import {
  ConnectorError,
  ConnectorConfigError,
  ConnectorSecretError,
} from "@memofs/connectors";

try {
  await runConnectors({ rootDir, memo, secretResolver });
} catch (error) {
  if (error instanceof ConnectorConfigError) {
    console.error("Config error code:", error.code); // "CONNECTOR_CONFIG_ERROR"
  } else if (error instanceof ConnectorSecretError) {
    console.error(`Failed resolving secret ref "${error.secretRef}":`, error.message);
  }
}

See Also

• Built-In Connectors (GitHub & Notion)
• Writing Custom Connectors
• @memofs/connectors API Reference
• CLI Connectors Commands
• Core Memory Runtime & Concepts