Label Management

​

Overview

Project areas and tags are workspace-defined labels that help triage feedback snippets. When auto-triage is enabled for a particular label, Monterey will go ahead and auto assign a project area label when the feedback is ingested or the labels are backfilled, provided our AI models are confident in that prediction.

While project area labels and tag labels work functionally similar in downstream uses (filtering, chat, reports, etc.), they have one key difference that necessitates the separation of the two.

For a particular snippet when auto-triage is enabled:

• 0 or 1 project areas can be auto-assigned
• 0 or many tag areas can be auto-assigned

Thus, it is often the case that there are many more tags applied since one snippet may have multiple. We suggest keeping this distinction in mind when building out labels for each of these.

Note: If a feedback item is split into multiple snippets, where each snippet has a different project area, you will see the parent feedback with multiple project areas. Additionally, you can also always manually add more than one project area label.

—

Auto-triage performance will be influenced by a combination of clear label names, detailed label descriptions, and human-labeled examples that are captured throughout the workspace. A detailed description that will help create a successful auto-triaged label could include keywords associated with the label, plain english descriptions of the situations where a label should be assigned, and a minimization of company/workspace jargon that may be difficult for a language model to understand.

Human-labeled examples can be very helpful in tuning a set of labels to a specific workspace. For example, we might expect multiple workspaces to have a label for “Design”, but the exact definition/criteria for what feedback get’s that label may differ in separate workspaces. Beyond a difference in label descriptions, the human-labeled examples are supplied to our triage models to further personalize their performance for each workspace!

​

Creating and Updating Labels

To create or update labels, navigate to the Labels tab in the “Project Areas” or “Tags” page, where you will see a list of all existing labels.

When creating or updating a new label, you will be provided with a two page modal:

• First, a form input for entering the labels name, description and auto-triage enablement.
• Second, a select few snippets to optionally give positive or negative label assignments too.

​

Labeling Feedback Snippet Examples

The second tab in the “Project Areas” or “Tags” page will be labeled Collect Label Examples. This tab is a dedicated view for collecting human labels on snippets that were not able to be auto assigned a label upon ingestion or backfill.

The list of snippets is ordered such that the snippets that would provide the most value to our model as examples are placed at the top.

When selecting a new label or set of labels for a snippet, you may see some label options include an “AI suggestion” indicator. This indicator states that while we were not confident enough to auto-assign this label, we think it’s still pretty likely this is the correct label. As these suggestions are confirmed or rejected with human labels, are models will become more confident in it’s predictions over time.

Note: Human-labeled examples in this view are only saved when the “Apply Label Updates” is clicked, which will save all labels that have been human-labeled.

​

Managing Existing Human Labels

The third and final label management tab is the Manage Existing Labels tab. This tab is dedicated view for viewing and updating all existing human-applied label assignments.

These are helpful for seeing how existing snippets have been labeled, which will affect downstream auto-triage. Additionally, if a label is changed substantively, it would be prudent to find all human labeled examples for that label and update as needed.

​

Label Backfill

The final action you will find in the label management pages is the Backfill Labels button in the top-right. Clicking this button will run labeling auto-triage across all unlabeled snippets in your workspace. You may want to run backfill if:

• You have added and/or removed labels, or substantively updated label titles/descriptions
• Applied, removed or updated any human labels