Contents

After opening an issue, I would forget to add priority and area labels. After this happened a few times, unclassified issues started piling up.

I manage to-dos with GitHub Issues for a dashboard app I develop on my own. Issue templates automatically label the type of change, but I was manually selecting the priority and the work areas, such as backend or frontend. So I decided to have GitHub Actions send each newly created issue to Jev and apply labels based on its classification.

Classification Options in Jev#

Jev is a model offered by TypeSafe that specializes in making judgments. Rather than generating prose in response to a question, it answers in a predefined format. The formats used here are Choice and Noul.

Choice is used for classifications that select one option from a fixed set. It also returns the probability of each option, so you can check both the selected answer and the model’s confidence in that judgment. It suits the task of choosing a priority from critical, high, medium, and low.

Noul returns the probability that the answer to a question is “yes,” as a value from 0 to 1. Asking “Does resolving this issue require changes in this area?” for each area lets you handle issues that span multiple areas. If both frontend and backend work are required, each question receives a yes.

Another format, Score, returns probabilities for each level and an expected value. I also used it to evaluate priority classification this time. Detailed specifications for each format are available in TypeSafe’s documentation.

Test Environment and Operating Rules#

I used the following environment to verify the setup:

  • Model: jev-1.13
  • Python SDK: typesafe-sdk 0.7.1
  • Python: 3.13
  • GitHub Actions: ubuntu-latest

I established two rules for labeling. If an issue already has priority or area labels, that category is left unchanged. Labels are also skipped when the model’s confidence is low. Specifically, priority labels are skipped if Choice’s confidence is below 0.5, and area labels are skipped if the probability of yes is below 0.5.

This avoids forcing a classification onto issues where the model’s judgment is uncertain. A person can review unlabeled issues as needed.

Prepare Questions and Descriptions in English#

TypeSafe’s documentation explains that Jev was trained primarily in English and achieves its highest accuracy in English. It can also handle Chinese, Japanese, and Korean text, but accuracy may differ from English.

I therefore wrote the questions and option descriptions in English, while passing issue titles and bodies in their original Japanese. For area descriptions, I translated the Japanese definitions used in the repository into English.

How well this setup classifies issues needs to be checked against actual issues. For this evaluation, I used issues I had previously labeled myself as ground truth and compared the results while varying the questions and classification methods.

For Areas, Ask Whether Changes Are Required, Not Whether They Are Mentioned#

Asking “Is this issue related to the frontend?” for each area made the scope too broad. For example, even if an error appears on screen, frontend code changes may be unnecessary if the cause is in the backend. Yet a question about whether the issue is “related” can still produce a yes for the frontend.

I changed the question to ask whether the area is “one of the main areas that must be changed to resolve this issue.” I also added conditions for no: merely being mentioned in the body, being indirectly affected, or being the place where the problem appears does not qualify.

snippet.py
Noul(
    instructions=f"Is {description} one of the main areas that must be changed to resolve this issue?",
    criteria={
        "true": "Substantial work in this area is required. A person doing this work would edit files or configuration belonging to this area.",
        "false": "This area is only mentioned, only indirectly affected, or merely the context in which the problem appears. No substantial work happens here.",
    },
)

Changing the wording of a question also changes the classification criteria. It seems useful to specify exactly what qualifies, then check the results against existing issues.

Use Choice to Determine Priority#

I tried both Choice and Score for priority classification. The results using past issues were as follows:

  • Choice, no confidence cutoff: 100% labeling rate, 67% exact agreement with ground truth, 95% within ±1 level
  • Choice, confidence ≥ 0.5: 81% labeling rate, 73% exact agreement with ground truth, 95% within ±1 level
  • Choice, confidence ≥ 0.7: 59% labeling rate, 76% exact agreement with ground truth, 98% within ±1 level
  • Score, confidence ≥ 0.5: 92% labeling rate, 56% exact agreement with ground truth, 94% within ±1 level

On this dataset, Choice had a higher agreement rate than Score. Score can return values between levels, which means you need to decide which level a value such as 2.5 belongs to. I rounded to the nearest integer for this evaluation, rounding halves up. With Choice, you can simply use the option with the highest probability.

Restricting Choice to confidence values of at least 0.5 produced an 81% labeling rate and 73% agreement with ground truth. Raising the threshold to 0.7 improved agreement, but reduced the proportion of issues receiving labels to 59%. Taking the labeling rate into account, I chose a threshold of 0.5.

The question is “How urgently should this issue be addressed?” Each option receives an English translation of its label definition. For example, critical covers issues requiring immediate attention, such as production outages, severe vulnerabilities, and data corruption. Low covers small improvements and cleanup work to tackle when time permits.

Run It from GitHub Actions#

The workflow runs on the issue opened event and also supports manual execution. For manual runs, you enter the issue number. Keep permissions limited to what is needed to read and write issues, and register the TypeSafe API key in the repository’s Secrets as TYPESAFE_API_KEY.

snippet.txt
name: Issue Triage

on:
  issues:
    types: [opened]
  workflow_dispatch:
    inputs:
      issue_number:
        description: 分類する issue 番号
        required: true

permissions:
  contents: read
  issues: write

jobs:
  triage:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
        with:
          python-version: "3.13"
          cache: pip
          cache-dependency-path: scripts/requirements.txt
      - run: pip install -r scripts/requirements.txt
      - env:
          TYPESAFE_API_KEY: ${{ secrets.TYPESAFE_API_KEY }}
          GH_TOKEN: ${{ github.token }}
          ISSUE_NUMBER: ${{ github.event.issue.number || inputs.issue_number }}
        run: python scripts/issue_triage.py "$ISSUE_NUMBER"

Both checkout and setup-python are pinned to commit SHAs rather than tags. This is one way to pin the exact revisions of actions used in a workflow. Add typesafe-sdk~=0.7.1 to requirements.txt as a dependency.

Only the issue number is passed to the script. The title, body, and existing labels are retrieved with gh issue view. The script checks existing labels first and skips the Choice question if a priority label is already present. If any area labels are present, it skips the Noul questions. It asks Jev only about missing categories, then adds labels only for results that meet the configured thresholds.

snippet.py
issue = json.loads(
    subprocess.run(
        ["gh", "issue", "view", issue_number, "--json", "title,body,labels"],
        check=True,
        capture_output=True,
        text=True,
    ).stdout
)
existing_labels = [label["name"] for label in issue["labels"]]

if not any(label.startswith("priority:") for label in existing_labels):
    questions["priority"] = Choice(
        instructions="How urgently should this issue be addressed?",
        criteria=PRIORITIES,
    )

if not any(label.startswith("area:") for label in existing_labels):
    for name, description in AREAS.items():
        questions[f"area_{name}"] = Noul(
            instructions=f"Is {description} one of the main areas that must be changed to resolve this issue?",
            criteria={
                "true": "Substantial work in this area is required.",
                "false": "This area is only mentioned, indirectly affected, or the context where the problem appears.",
            },
        )

This excerpt covers only the construction of the questions. Add the API call, probability extraction, and label application logic to the script according to the SDK’s specifications. You will also need to verify the label names and API return values for your actual repository.

Start by Setting Thresholds Using Past Issues#

These settings come from testing against existing issues in my personal app project. Different issue content or label definitions may produce different agreement rates.

Preparing a few past issues as ground truth and comparing results with different question wording and confidence thresholds makes it easier to choose settings that suit your workflow. If you adopt this approach, start by checking classifications against past issues and deciding on criteria that fit your repository.

Contents