shiftlabs Start free
Menu Runners

Run commands in your network

A runner is a container you run in your own network. It takes commands from your workflows' step.exec steps, runs them next to your clusters and servers, and sends back the exit code and output. It only makes outbound HTTPS requests, so it needs no inbound port.

What a runner runs

A runner runs whatever its allowed workflows send. Give it only the credentials those workflows need, and limit it to named workflows.

  • Outbound only

    The runner calls https://api.shiftlabs.org and waits there for work. Nothing connects to it, so it needs no inbound port, load balancer or public address.

  • Your credentials stay with you

    Commands use the credentials you give the runner, such as a kubeconfig or a cloud role, and the readable secrets a step passes in. shiftlabs never sees the runner's own credentials.

  • Limited to named workflows

    A runner token can name the workflows it serves. A runner then takes jobs from those workflows only. Anyone who can deploy one of them can run commands on it.

  • One workspace

    A token belongs to one workspace. Its runners take jobs from that workspace only.

  • Give each runner the least it needs: a read-only role for a runner that only reads, a separate runner with its own token for deploys.
  • Commands run as the user runner (uid 1000), each in a folder of its own that is removed after the command.
  • Commands on one runner share its user, home folder and network. Files one command leaves in the home folder, such as .gitconfig, .ssh/config or .aws/config, change what later commands' tools do. With SHIFTLABS_RUNNER_JOBS above 1, commands running at the same time can also read each other's folders and secret files.
  • Only separate runners, with separate tokens, keep workflows' secrets apart. Running one command at a time, the default, doesn't. Give workflows that use different secrets runners of their own, and limit each token to those workflows.
  • When no command is running, the runner kills every process earlier commands left behind, so none lives on into the next command. With SHIFTLABS_RUNNER_JOBS above 1, that happens only once all its commands have ended.
  • Commands can't read the runner token: the runner hides its own environment, and it refuses to start when another process shows the token. The token and the keys made from it are always replaced by [redacted] in output.
  • A shell you open in the container with docker exec or kubectl exec gets the token from the container's environment, and a command could read it from that shell. Open one without it: docker exec -it shiftlabs-runner env -u SHIFTLABS_RUNNER_TOKEN bash, kubectl exec -it <pod> -- env -u SHIFTLABS_RUNNER_TOKEN bash, or use kubectl debug.
  • Developers who can deploy an allowed workflow decide what the runner runs. Use scoped API keys and approvals to control who deploys and when a run goes ahead.

Runner tokens

A runner signs in with a runner token. An admin creates one in Settings → Runners, or with POST /v1/runners/tokens.

shellcurl
curl -X POST https://api.shiftlabs.org/v1/runners/tokens \
  -H "authorization: Bearer $SHIFTLABS_API_KEY" -H "content-type: application/json" \
  -d '{"name": "prod-cluster", "pool": "prod", "workflows": ["restart-app", "rotate-certs"]}'

# 201 {"id": "rtk_…", "token": "slr_…", "name": "prod-cluster", "pool": "prod",
#      "workflows": ["restart-app", "rotate-certs"], "prefix": "slr_…", "createdAt": …}
  • name

    1 to 64 characters, to tell tokens apart.

  • pool

    Optional. The pool the token's runners serve. Left out, each runner picks its pool.

  • workflows

    Optional. 1 to 50 workflow names. Left out, the runners take jobs from every workflow in the workspace.

  • The token is shown once. We keep only values derived from it, so it can't be shown again. Store it as a secret where the runner runs.
  • Revoke a token in Settings → Runners or with DELETE /v1/runners/tokens/:id. Its runners stop at once, and the commands they are running fail. A revoked runner exits with code 2.
  • GET /v1/runners lists the tokens, never their values, and up to 500 runners seen in the last 7 days, most recently seen first: their name, pool, version, when they were last seen, whether they are online and how many commands they are running. Viewers can read it.
  • Creating and revoking tokens needs an admin. A workspace that has all its tokens gets 403 {"error": "quota exceeded", "limit": "runnerTokens", "max"}. See Limits.
Routes
RouteMethodMinimum role
/v1/runners {image, tokens, runners}: tokens without their values, and the runners seen in the last 7 daysGETviewer
/v1/runners/tokens {"name", "pool"?, "workflows"?}; 201 with the token, shown oncePOSTadmin
/v1/runners/tokens/:id {revoked}; its runners stop and their running commands failDELETEadmin

Run it with Docker

The image is shiftlabsorg/runner:1.0.0 on Docker Hub, for amd64 and arm64. Set the token and start it.

shelldocker
export SHIFTLABS_RUNNER_TOKEN=slr_…   # from Settings → Runners

# Check the token: prints the pool and the workflows it serves, then exits.
docker run --rm -e SHIFTLABS_RUNNER_TOKEN shiftlabsorg/runner:1.0.0 check

docker run -d --name shiftlabs-runner --restart unless-stopped \
  -e SHIFTLABS_RUNNER_TOKEN \
  -e SHIFTLABS_RUNNER_NAME=build-host-1 \
  -v "$PWD/runner-kubeconfig:/etc/shiftlabs/kubeconfig:ro" -e KUBECONFIG=/etc/shiftlabs/kubeconfig \
  shiftlabsorg/runner:1.0.0
  • -e SHIFTLABS_RUNNER_TOKEN without a value passes the variable from your shell, so the token stays out of the command line and docker inspect shows only the name. --env-file works too.
  • runner-kubeconfig is a kubeconfig made for the runner, with only the rights its workflows need. Don't mount your own ~/.kube/config: it is usually an admin's, and every allowed workflow could use it.
  • A file you mount must be readable by uid 1000.
  • docker stop lets running commands finish their stop: see When a runner stops.

Run it on Kubernetes

A Deployment of runners, with a ServiceAccount that may only list pods in one namespace. Commands then run kubectl against the cluster the runner is in, with that ServiceAccount's rights.

yamlrunner.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: shiftlabs-runner   # optional: any namespace works
---
apiVersion: v1
kind: Secret
metadata:
  name: shiftlabs-runner
  namespace: shiftlabs-runner
type: Opaque
stringData:
  token: slr_…   # from Settings → Runners
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: shiftlabs-runner
  namespace: shiftlabs-runner
---
# An example: read-only access to pods in the namespace apps, enough for kubectl get pods -n apps.
# Never let this ServiceAccount read Secrets in shiftlabs-runner: the token Secret is there.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: shiftlabs-runner-read-pods
  namespace: apps
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: shiftlabs-runner-read-pods
  namespace: apps
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: shiftlabs-runner-read-pods
subjects:
  - kind: ServiceAccount
    name: shiftlabs-runner
    namespace: shiftlabs-runner
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: shiftlabs-runner
  namespace: shiftlabs-runner
spec:
  replicas: 2
  selector:
    matchLabels:
      app: shiftlabs-runner
  template:
    metadata:
      labels:
        app: shiftlabs-runner
    spec:
      serviceAccountName: shiftlabs-runner
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 1000
        seccompProfile:
          type: RuntimeDefault
      volumes:
        - name: work
          emptyDir: {}
        - name: tmp
          emptyDir: {}
        - name: home
          emptyDir: {}
      containers:
        - name: runner
          image: shiftlabsorg/runner:1.0.0
          env:
            - name: SHIFTLABS_RUNNER_TOKEN
              valueFrom:
                secretKeyRef:
                  name: shiftlabs-runner
                  key: token
            - name: SHIFTLABS_RUNNER_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            # The runner makes this folder for itself: it refuses a work folder that others can write to.
            - name: SHIFTLABS_WORK_DIR
              value: /work/jobs
          volumeMounts:
            - name: work
              mountPath: /work
            - name: tmp
              mountPath: /tmp
            - name: home
              mountPath: /home/runner
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 1Gi
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ["ALL"]
  • kubectl apply -f runner.yaml starts two runners. Each pod is one runner, named after the pod. Raise replicas or SHIFTLABS_RUNNER_JOBS to run more commands at once.
  • terminationGracePeriodSeconds: 30 gives a stopping runner time to stop its commands, which get 10 seconds, and report them.
  • The image's files are read-only. Commands can write only to /work, /tmp and the home folder /home/runner, which start empty with each pod.
  • Give the Role only what your workflows need. For another cluster, mount a kubeconfig from a Secret and set KUBECONFIG instead.
  • The runner's ServiceAccount must not be able to read Secrets in the runner's own namespace, which holds the token. A command that could read that Secret would have the token.
  • A pod whose token was revoked exits with code 2 and restarts until you delete the Deployment.

Settings

The runner reads its settings from environment variables.

Runner settings
VariableWhat it sets
SHIFTLABS_RUNNER_TOKEN requiredThe runner token.
SHIFTLABS_RUNNER_NAMEThe runner's name in Settings → Runners and in results. Default: the host name.
SHIFTLABS_RUNNER_POOLThe pool it serves when its token names none. Default: default.
SHIFTLABS_RUNNER_JOBSHow many commands it runs at once, 1 to 16. Default: 1.
SHIFTLABS_REDACTNames of the runner's own environment variables whose values are replaced by [redacted] in output, comma-separated. See Output.
SHIFTLABS_WORK_DIRWhere each command's folder is made. Default: /work in the image, else $HOME/.shiftlabs-runner/work. It must be owned by the runner's user and not writable by others, or the runner exits with code 2. Each runner needs a work folder of its own: a second runner on the same folder exits with code 2. Command folders left from an earlier start are deleted when the runner starts.
SHIFTLABS_API_URLDefault: https://api.shiftlabs.org.
HTTPS_PROXY · SSL_CERT_FILEA proxy and a certificate file. See Proxies.
  • shiftlabs-runner check signs in, prints the pool and the workflows the token serves, and exits. shiftlabs-runner version prints the version.
  • The runner logs JSON lines on stderr: when it starts, and when each command starts and ends, with its run, workflow, step, exit code and duration. Logs never hold values, secrets or output.
  • Exit codes: 0 when stopped. 2 when the token is wrong or revoked or names another pool, a setting is wrong, the work folder is shared with others, a parent process shows the token in its environment, or the runner can't hide its own environment from commands. 3 when the runner is too old, with the tag to pull in its log.
  • The runner is the container's first process, with no init in front of it. It passes signals on, and it adopts the processes commands leave behind and kills them once no command is running. Don't start it with docker run --init or from a script that doesn't exec it: the process in front would keep the token readable, and the runner exits with code 2.

Credentials

A command gets credentials in two ways: from the runner itself, or from a readable secret that the step passes in.

  • The runner's own

    Commands get the runner's environment, without the SHIFTLABS_ variables. Mount a kubeconfig and set KUBECONFIG, use the pod's ServiceAccount in its cluster, set cloud variables such as AWS_PROFILE, or use workload identity: IRSA or EKS Pod Identity on AWS, Workload Identity on Google Cloud, workload identity on Azure. shiftlabs never sees these credentials. List any that commands might print in SHIFTLABS_REDACT.

  • Readable secrets, per command

    A value { secret: "NAME" } in env or files passes a secret marked readable to that one command. Only readable secrets can be passed. The value travels encrypted for the runner, each pass is audited, and the value is replaced by [redacted] in the output. A file holding a secret is readable only by the runner's user, and is removed after the command. Other commands running on the same runner at that time can read it: see What a runner runs.

  • Prefer the runner's own credentials for long-lived access to your systems, and secrets for values your workflows already keep in shiftlabs.
  • Each command also gets SHIFTLABS_RUN_ID, SHIFTLABS_WORKFLOW, SHIFTLABS_STEP, SHIFTLABS_ATTEMPT (the try, from 1), SHIFTLABS_JOB_ID and SHIFTLABS_IDEMPOTENCY_KEY. Your own env names can't start with SHIFTLABS_.
  • SHIFTLABS_IDEMPOTENCY_KEY is the step's idempotency key: the same on every try of the command, and for every call with the same step name in the run.

Run a command

step.exec(name, options) sends a command to a runner and waits for it to end. Give run, a bash script, or command, a program and its arguments with no shell.

restart-app
import { workflow } from "shiftlabs";

export default workflow<{ app: string }>(async ({ payload, step }) => {
  const pods = await step.exec("list pods", {
    command: ["kubectl", "get", "pods", "-n", "apps", "-o", "name"],
  });

  // Values go in env. A value pasted into the script would run as code.
  await step.exec("restart", {
    run: 'kubectl rollout restart "deployment/$APP" -n apps\nkubectl rollout status "deployment/$APP" -n apps --timeout=5m',
    env: { APP: payload.app },
    timeout: "10 minutes",
  });

  return { pods: pods.stdout.split("\n").filter(Boolean) };
});
deploy-web
import { ExecError, workflow } from "shiftlabs";

export default workflow<{ version: string }>(async ({ payload, step }) => {
  // kubectl exits with 1 when the deployment doesn't exist yet.
  const found = await step.exec("find web", {
    command: ["kubectl", "--kubeconfig", "kubeconfig", "get", "deployment", "web", "-n", "apps"],
    files: { kubeconfig: { secret: "STAGING_KUBECONFIG" } },
    okExitCodes: [0, 1],
  });

  try {
    await step.exec("deploy", {
      run: 'helm upgrade --install web oci://registry.example.com/charts/web --version "$VERSION" --kubeconfig kubeconfig --wait',
      env: { VERSION: payload.version },
      files: { kubeconfig: { secret: "STAGING_KUBECONFIG" } },
      runner: "staging",
      timeout: "20 minutes",
      retries: { limit: 2, delay: "1 minute" },
    });
  } catch (err) {
    if (err instanceof ExecError) return { deployed: false, exitCode: err.result.exitCode, stderr: err.result.stderr };
    throw err;
  }
  return { deployed: true, firstInstall: found.exitCode === 1 };
});
  • run

    A bash script, run with bash --noprofile --norc -eo pipefail, so it stops at the first failing command. Up to 64 KiB.

  • command

    The program and its arguments, run without a shell: 1 to 256 strings, up to 64 KiB in all. Give run or command, not both.

  • env

    Up to 100 environment variables. A value is text or { secret: "NAME" }, a readable secret.

  • files

    Up to 50 files written in the command's folder before it starts, by relative path such as config/app.yaml. Values as in env. Text up to 512 KiB in all.

  • stdin

    Text sent to the command's standard input, up to 256 KiB.

  • timeout

    How long the command may run. Default "10 minutes", and at most the plan's longest command.

  • runner

    The pool that runs it. Default "default".

  • okExitCodes

    The exit codes that count as success. Default [0]. "any" accepts every exit code. In Python, ok_exit_codes.

  • retries

    { limit, delay, backoff }: how many more times to try after a failure, 0 to 10, the wait between tries, and "constant", "linear" or "exponential" (the default). No retries by default.

  • Each command runs in a new folder, which is also its working directory. The folder and its files are removed when the command ends, so keep nothing there between steps.
  • Each attempt takes two steps: one sends the job and one reads its result. Waiting for a runner and for the command is not a step, but counts toward the run's lifetime.
  • A resumed run doesn't run the command again: it gets the stored result.
  • Options are checked before any step, as the platform checks them. A mistake throws TypeError with a message that starts with step.exec:.
  • In tests, runWorkflow's execs answers each command by step name. See Test a workflow.

Results and errors

step.exec returns the result when the exit code is in okExitCodes. In Python it is a dict with the same keys in snake_case.

  • exitCode

    The command's exit code. 124 when it timed out, 128 plus the signal number when a signal killed it, 127 when the program wasn't found and 126 when it couldn't start.

  • stdout · stderr

    The last 256 KiB of each stream, with secrets redacted.

  • stdoutBytes · stderrBytes · truncated

    How many bytes each stream wrote in all, and whether stdout or stderr holds only the end.

  • timedOut · durationMs

    Whether the timeout stopped it, and how long it ran.

  • runner · attempt

    The name of the runner that ran it, and which try this was, from 1.

  • Any other exit code throws ExecError with the message exec <name>: exited with code N, or exec <name>: timed out after <timeout>. Its result holds the whole result. Catch it to read the output or go on.
  • A command that never ran, or whose runner stopped answering, throws a plain error that says why: no runner in the pool took it within 1 hour, the runner stopped responding, its token was revoked, or the run ended.
  • With retries, either kind of failure waits delay and runs the command again as a new job, maybe on another runner. Only the last try's result or error reaches your code.
  • Some failures come at once, with no retries: no token of the pool allows this workflow, the queue is full, or the timeout is over the plan's longest command.

Timeouts

When a command runs past its timeout, the runner sends SIGTERM to the command and every process it started, then SIGKILL 10 seconds later. The result has timedOut: true and exit code 124, so step.exec throws ExecError unless okExitCodes takes 124.

  • A timeout is at least 1 second, and at most the plan's longest command: 30 minutes on Free, 12 hours on Pro.
  • The timeout counts from when the command starts. Waiting for a runner doesn't count; a job waits up to 1 hour.
  • A process that starts its own session, with setsid or as a daemon, leaves the command's process group, so the timeout doesn't stop it. The runner kills it once no command is running on the runner.

Output

The run's logs show the command's output as it runs. The result keeps the last 256 KiB of each stream.

  • Up to 2 MiB of each command's output reaches the run's logs, as whole lines of up to 4,000 characters. After that the logs stop, and the result still holds the end of each stream.
  • Before output leaves the runner, the values of the secrets passed to the command, and of the runner's variables listed in SHIFTLABS_REDACT, are replaced by [redacted]. For a secret of several lines, each line of 8 characters or more is replaced too. Values shorter than 8 bytes are not.
  • The runner token, and the keys the runner makes from it, are always replaced by [redacted].
  • shiftlabs then replaces every readable secret of the workspace in the output before storing it, as it does in other logs.
  • Output that isn't valid UTF-8 is stored with replacement characters.
  • Set SHIFTLABS_REDACT to the names of any credentials in the runner's environment that a command might print, for example SHIFTLABS_REDACT=AWS_SECRET_ACCESS_KEY,GITHUB_TOKEN.

Pools

A pool is a group of runners that take the same commands. step.exec's runner option picks the pool, and any online runner in it takes the command. Without it, the pool is default.

  • Token's pool

    A token created with pool serves that pool only. A runner that sets another SHIFTLABS_RUNNER_POOL is refused and exits with code 2.

  • Runner's pool

    A token without a pool lets each runner pick one with SHIFTLABS_RUNNER_POOL, or default.

  • Names

    Lowercase letters, digits and -, up to 63 characters, starting with a letter or digit.

  • Use pools to keep places apart: prod runners in production with production credentials, staging runners with staging ones.
  • Commands wait for a runner of their pool. When none takes a command within 1 hour, it fails.

When a runner stops

A runner that takes a command holds it for 60 seconds at a time, and renews that every 15 seconds while the command runs.

  • Stopped

    On SIGTERM, such as docker stop or a pod being replaced, the runner takes no more commands, stops the ones running (SIGTERM, then SIGKILL after 10 seconds), and reports them as failed with the runner was stopped before the command finished. The step fails at once.

  • Gone

    A runner that dies or loses its network stops renewing. 60 seconds later the command fails with the runner stopped responding.

  • Retried

    With retries, the command runs again as a new job, on any runner of the pool.

A command that failed this way may have done part of its work. Write commands that can run twice, and pass SHIFTLABS_IDEMPOTENCY_KEY, which stays the same on every try, to APIs that take an idempotency key.

Limits

Runner limits per workspace
LimitFreePro
Runner tokens220
Longest command, the most a timeout may be30 minutes12 hours
Commands waiting for a runner20500
How long a command waits for a runner1 hour1 hour
Runners online with one token100100
Commands at once with one token, its online runners' SHIFTLABS_RUNNER_JOBS added up250250
  • Commands run on your machines, so they use no CPU time or outbound requests of your plan. Their steps count as other steps do.
  • A command over the waiting limit fails at once.
  • A runner that would take a token past either limit gets 429, with the reason in its log, and tries again every minute. Lower SHIFTLABS_RUNNER_JOBS, stop a runner, or give it a token of its own.
  • Runner calls from one IP address are limited to 3,000 a minute, enough for several hundred runners behind one NAT.
  • While an operator restores a workspace's data, its runners get no jobs until the restore has finished.

Your own image

The image is Debian 13 slim with bash, coreutils, curl, jq, yq, git, an SSH client, unzip, Python 3 with pip and venv, kubectl, helm, kustomize, OpenTofu, the AWS CLI, gcloud, az and ansible-core. Add your own tools with FROM, or copy the runner into an image you already use.

dockerfileDockerfile
FROM shiftlabsorg/runner:1.0.0
USER root
RUN apt-get update && apt-get install -y --no-install-recommends postgresql-client \
  && rm -rf /var/lib/apt/lists/*
USER 1000
dockerfileDockerfile
FROM your-registry.example.com/ops-tools:2026.10

COPY --from=shiftlabsorg/runner:1.0.0 /usr/local/bin/shiftlabs-runner /usr/local/bin/
USER 1000
ENTRYPOINT ["/usr/local/bin/shiftlabs-runner"]
CMD ["run"]
  • The runner is one static binary for Linux on amd64 and arm64. It needs bash for run scripts, and CA certificates.
  • Run it as a user other than root. In the image it runs as runner, uid 1000, as the container's first process, with no init in front of it.

Proxies and certificates

  • Set HTTPS_PROXY to send the runner's requests through a proxy. Commands get the variable too.
  • For a proxy or network that uses your own certificate authority, mount its certificate bundle and set SSL_CERT_FILE to it. The runner always checks certificates; there is no setting to turn that off.
  • The runner needs to reach api.shiftlabs.org on port 443, nothing else.
  • Leave api.shiftlabs.org out of TLS inspection. A proxy that opens the runner's TLS sees each command and its output, and can take jobs in the runner's place. Those jobs then never run, so it can stop your commands from running.
  • Such a proxy can't read the values of secrets passed to commands, which are encrypted for the runner. It can't forge results, and it can't make your runner run anything your workflows didn't send. See The runner protocol.

The runner protocol

For security reviews: how a runner and shiftlabs talk. All of it is HTTPS from the runner to https://api.shiftlabs.org/runner/v1/.

  • The token never travels

    The runner derives three keys from its token with HMAC-SHA256: a bearer key it signs in with, a signing key and an encryption key. Only the bearer key is ever sent. shiftlabs keeps a hash of the bearer key, and the other two keys encrypted, not the token.

  • Long polling

    The runner registers, then asks for work. Each request waits up to 25 seconds for a job. A running command's output and lease renewal go up every 15 seconds, and the result when it ends.

  • Signed jobs

    Each job is signed with the signing key. The runner refuses a job with a bad signature, one meant for another runner, one that doesn't carry the random value of the request it answers, one issued too long ago, and any job it has seen before.

  • Signed results

    Output, lease renewals and results are signed with the signing key too, with a number that grows with each call. shiftlabs refuses any that don't match the runner and the job, and any replayed call.

  • Signed answers

    shiftlabs signs its answer to each lease renewal. A proxy can't keep a command running after shiftlabs failed it: without a signed answer, the runner stops the command when its lease runs out. A lease is never renewed past the command's timeout plus 2 minutes.

  • Encrypted secrets

    Readable secrets are added to a job when a runner takes it, never stored with the job, and each is audited as a reveal. Their values are encrypted with the encryption key, for that job only.

  • What TLS inspection exposes

    A proxy that opens TLS sees the bearer key, the commands and the output. With the bearer key it can take jobs, which then don't run. It can't decrypt secret values, sign results, or send your runner jobs of its own.

  • Audit and revocation

    Creating and revoking tokens, and each command a runner takes, are in the audit log. Revoking a token ends its runners' requests within seconds. Their running commands are stopped, and the steps fail.

Versions

The examples on this page pin shiftlabsorg/runner:1.0.0. That tag never changes. shiftlabsorg/runner:1 moves to each new 1.x release, so a runner that pulls it again gets fixes without a change on your side. A new major version is a new tag, announced in the changelog.

  • Pin a full version to decide when the runner changes, and move to the next release when it is announced.
  • A runner too old for the platform exits with code 3 and logs the tag to pull.