LogoAI Work Hub
Back to Guides
Updated 2026

Aether Task Guide: How to Write Bulletproof Justifications and Keep the Project

The key to maintaining high scores on Aether: the 3-part justification formula, timer management strategy and a calibration method for recovering from low ratings.

Some annotators work on Aether for months. Others disappear after two weeks. The difference almost never comes down to choosing the right response — most candidates who make it through the qualification test are already at an acceptable level on that. The difference is in the written justification.

Aether’s quality system isn’t automated: a human reviewer reads what you wrote. They assess whether your reasoning is traceable, whether your citations are real, whether your conclusions follow from your arguments. An annotator who always picks the right response but can’t explain why gets removed. An annotator who occasionally picks the slightly wrong one but produces solid reasoning gets feedback, improves, and stays.


Why Justifications Are 90% of Your Score

Choosing between Response A and Response B is binary — you’re either right or wrong, and even that isn’t always clear-cut. The justification, on the other hand, is a piece of text that a reviewer evaluates on a nuanced scale — one that can compensate for or destroy the score on a single choice.

Here’s what a senior reviewer is concretely looking at when they read your justification:

1. Traceability of reasoning — Can they follow you from premise to conclusion without logical jumps? If you write “Response B is better because it’s more precise,” the reviewer can’t follow your reasoning because you haven’t defined what you mean by “precise” or indicated where this precision actually appears.

2. Specificity of evidence — Did you cite real portions of the responses, or did you describe a vague general impression? “Model A used an inappropriate tone” is vague. “Model A responded with ‘obviously’ in the third sentence, a word that comes across as condescending given the user’s genuine question” is specific.

3. Alignment with the rubric — Does your justification address the criteria that matter on Aether (factuality, instruction following, safety) or does it address criteria that aren’t in the rubric (creativity, originality, stylistic elegance)? Citing unofficial criteria doesn’t help — it can actually signal that you haven’t read the guidelines.

4. Proportionality of judgment — Did you use “Much Better” for a minor difference? Did you use “Slightly Better” for a substantial one? The degree of preference must correspond to the magnitude of the documented difference.


The 3-Part Formula for a Strong Justification

There’s no rigid template that works for every task. There is, however, a logical structure that produces robust justifications in the vast majority of cases.

Part 1 — The Premise: the main constraint of the prompt

Start by identifying and declaring the most relevant criterion for that specific task. This does two things: it demonstrates that you read and understood the prompt, and it gives the reviewer the reference frame for reading everything that follows.

“The prompt requested a technical concept explained for a non-expert audience, within a 200-word limit and without use of specialized terminology.”

The premise doesn’t need to be long — one or two sentences. Its purpose is to anchor the justification to the specific task, not to summarize the entire prompt.

Part 2 — The Clear Comparison: where one failed and the other delivered

This is the most important part and the one most often done poorly. You must cite specific portions of the responses — not describe general impressions.

The quotation rule: use quotation marks whenever you’re referencing something the response literally says. If you’re paraphrasing or describing, specify that you’re paraphrasing. The distinction matters because a reviewer can go and verify your citation.

The internal structure of the comparison follows this pattern:

  • What the losing response got wrong or missed (with specific evidence)
  • What the winning response did right on the same parameter (with specific evidence)
  • If relevant: other secondary differences (but no more than one or two — you don’t want to dilute the main point)

“Response A exceeds the word limit at approximately 280 words instead of the required 200, and includes technical terms like ‘hyperparameter’ and ‘activation function’ that violate the plain language constraint. Response B stays within 195 words and replaces technical terminology with accessible analogies (‘the model learns like a child receiving corrections’), respecting both prompt constraints.”

Part 3 — The Concluding Verdict: preference level and logic

Close with a sentence that states the preference level and connects it to the evidence you’ve already presented. Don’t introduce new arguments here — you’re concluding, not opening.

The preference scale and when to use each level:

Level When to use
Slightly Better A minor difference on a single criterion; both responses are substantially valid
Better A clear difference on one or more important criteria; the losing response has real flaws
Much Better A substantial difference: serious constraint violation, significant factual error, safety issue
Significantly Better Used rarely; reserved for cases where the losing response is clearly harmful or entirely off-target

“For these reasons, Response B is preferred at the ‘Better’ level: it respects both explicit prompt constraints while Response A violates two simultaneously.”


Practical Before/After Example

The prompt: “Explain what inflation is to a 10-year-old. Maximum 100 words. Do not use the word ‘economy’.”

Response A (112 words, contains “economy”): “Inflation is a phenomenon of the economy in which the prices of things increase over time. When there is inflation, with the same amount of money you can buy fewer things than before. For example, if an ice cream costs $1 today and $1.20 next year, there has been 20% inflation. Governments try to control inflation through the monetary policies of the central bank…”

Response B (87 words, does not contain “economy”): “Imagine that with $5 you can buy 5 candies today. Next year, those same $5 only buy 4. Your money has ‘lost power.’ This happens because things cost more — it’s called inflation. It’s like your money shrinks a little every year. That’s why your parents sometimes say that ‘things used to be cheaper.’”


❌ Rejected justification

“Model A is longer and answers the question better by giving more information. Model B is too simplistic.”

Why it gets rejected:

  • Cites no prompt constraints
  • “Answers better” isn’t supported with evidence
  • “Too simplistic” ignores that simplicity was an explicit requirement
  • Chose the wrong response based on a non-existent criterion (length = quality)

✅ Justification that passes review

“The prompt requested an explanation suitable for a 10-year-old within a 100-word limit and without using the word ‘economy.’

Response A violates both constraints: it exceeds the limit at approximately 112 words and includes the word ‘economy’ in the first sentence (‘phenomenon of the economy’). It also references ‘central bank’ and ‘monetary policies,’ which are not appropriate for the target cognitive level.

Response B stays within 87 words, never uses the word ‘economy,’ and employs a concrete analogy (candies, money that ‘shrinks’) accessible to a child. It maintains the required register from start to finish.

Preference: Much Better for Response B. Response A violates two explicit constraints simultaneously, while Response B respects both while maintaining communicative quality appropriate for the target audience.“

Why this passes review:

  • Opens by identifying the relevant constraints
  • Cites literal text with quotation marks (“phenomenon of the economy”)
  • Counts words rather than estimating
  • Uses “Much Better” proportionally given the severity of the violation (two simultaneous explicit constraint violations)
  • Closes with a verdict that summarizes the logic without adding new elements

Managing the Task Timer

Every Aether task has a visible timer. Its function isn’t just organizational — it’s also a quality signal. Anomalous times in either direction trigger flags in the system.

The rushing problem (submitting too quickly)

Completing a task in under 2–3 minutes is statistically impossible if the annotator has genuinely read everything, checked constraints, and written an articulated justification. The system knows this. Tasks submitted in very short times are automatically flagged for priority review — and when a reviewer opens an already-flagged task, they examine it with a more critical eye.

This doesn’t mean you should deliberately waste time. It means that if you’ve genuinely finished in 5 minutes, it’s worth rereading everything carefully before submitting. Almost always you’ll find something to improve.

The prolonged inactivity problem (timeout)

Tasks have a completion window. If you leave a task open without activity for an extended period — the exact limit varies but indicatively beyond 30–45 minutes of inactivity — the task may return to the queue or expire. An expired task generates no score and in some cases is logged as a missed completion.

The right approach for complex tasks requiring external research:

  1. Open the task and do a full first read
  2. Run external verifications in separate windows (the timer keeps running, but you’re “active” on the page)
  3. Return to the page, write the justification, submit

If you realize you’ve opened a task at a moment when you don’t have time to complete it, some projects allow you to “release” the task — returning it to the queue without penalty. Check if this option is available in the interface before letting it expire.

Indicative time distribution for a sustainable pace without flags:

Task type Reasonable minimum Efficient maximum
Simple task (short conversation) 8 minutes 15 minutes
Medium task (complex reasoning or instructions) 12 minutes 22 minutes
Advanced task (coding or intensive fact-checking) 18 minutes 35 minutes

What to Do When You Receive a Low Rating

Low ratings happen. Even to experienced annotators, even on tasks that seemed clear. How you react to negative feedback is often more important than the task itself.

How to read reviewer feedback

Aether feedback typically indicates:

  • The criterion on which your justification was deemed insufficient
  • Sometimes a reviewer note explaining the issue
  • The task score (which contributes to your aggregate Quality Score)

Read the feedback without defensiveness. The question to ask isn’t “is the reviewer right?” but “what could I have written that would have avoided this rating?” These are two different questions. The second one is the useful one.

Guiding questions for analyzing negative feedback:

  • Did I state the main constraint of the prompt?
  • Did I cite real text from the responses or describe impressions?
  • Was the preference level I used proportional?
  • Did I address criteria not present in Aether’s rubric?

When to open a dispute ticket

Disputing a low rating is possible, but should be used with judgment. Systematically opening tickets for every negative piece of feedback is recorded and interpreted as an inability to accept criticism — which, paradoxically, lowers reviewers’ confidence in your work.

Open a dispute ticket only if:

  1. The reviewer feedback cites a criterion not present in the project’s official guidelines
  2. The reviewer negatively rated one of your citations that is literally present in the response text (verification error)
  3. Your task received a rating that explicitly contradicts feedback on a previously identical task

In all other cases — including those where you think the reviewer is wrong — the most productive thing is to apply the feedback to the next task and measure whether the score improves.

How to course-correct after feedback

The most effective method is progressive calibration: don’t change everything after a single piece of feedback, but isolate the element the reviewer flagged and modify only that in the next task. If you changed three elements simultaneously and the score improves, you don’t know which of the three made the difference.

4-step calibration framework:

  1. Identify: which specific part of your justification generated the negative feedback?
  2. Hypothesize: what should you have written instead?
  3. Apply: use that modified structure in the next task
  4. Verify: does the feedback improve? If yes, you’ve calibrated correctly. If not, return to step 1.

A Quality Score that drops below 75% on Aether typically triggers a profile review. This isn’t necessarily an immediate removal, but it’s the point at which you should stop working “at volume” and focus on single tasks done with maximum care — until you bring the score back above 80%.

Ready to test your skills?

Explore our independently reviewed AI platforms and apply directly.

View Top Platforms