I for illustration to cook, and location on the measurement that turned into an obsession pinch high-end chef's knives. So I scrape the Reddit threads wherever group reason astir them and propulsion retired each brand, exemplary and alloy they mention, to spot what is getting bought and based on about.
Picking merchandise names retired of matter is simply a occupation called named-entity recognition, and mini models person done it for a decade. I was doing it pinch Gemini 3.1 Pro, 1 paid API telephone per comment. Overkill, but it worked: from "picked up a Mazaki successful achromatic #2, measurement amended than my aged Fibrox" it returned Mazaki arsenic a brand, Fibrox arsenic a exemplary and achromatic #2 arsenic a steel, and thing else. But the scraper pulls each caller comment, truthful the measure grew pinch really overmuch group posted, and the only measurement to headdress it was to skip comments.
The evident replacement, an unfastened NER exemplary called GLiNER tally zero-shot, trim the costs to thing and the accuracy to astir 0.65 F1 against Gemini's answers. That spread is what the remainder of this is about: could Gemini explanation 4,290 comments erstwhile and thatch GLiNER to adjacent it?
- What: Fine-tuned GLiNER ample v2.5 (459M) to tag brands, models and materials successful Reddit comments, connected labels Gemini 3.1 Pro wrote once.
- Why: Zero-shot GLiNER scored astir 0.65 F1 (est.). Gemini scored good and billed each remark for arsenic agelong arsenic the scraper ran.
- Approach: Ask Gemini for strings, not offsets. Compute offsets successful code. Add comments pinch nary products successful them arsenic negatives. Lock a 225-comment validation group earlier the 2nd run.
- Problems: Five of 10 runs produced nary usable model. Three grounded connected configuration. Two grounded connected a tensor called words_mask that I filled the measurement you capable an attraction mask.
- Result: 0.83 F1 against Gemini's labels aft 24 minutes connected a Tesla T4. $9 of labels, astir $2.50 of GPU time, and days of debugging.
What I group retired to do
The scheme had 3 steps. Have Gemini explanation a fewer 1000 Reddit comments once, marking each brand, exemplary and steel. Train GLiNER connected those labels. Then tally GLiNER connected my ain instrumentality for each remark aft that, and extremity calling Gemini.
Gemini branded 4,290 comments for $9, aliases $0.0021 a comment. That intends the trained exemplary pays for itself astatine astir remark 4,291, arsenic agelong arsenic later comments are astir the aforesaid magnitude and it runs connected a GPU I already own. The trial of occurrence was simple: connected 225 comments the exemplary had ne'er seen, really often does it tag the aforesaid words Gemini tagged? One drawback to support successful mind for each people successful this article. Nobody checked Gemini's labels by hand, truthful the exemplary is graded against Gemini, not against the truth. Where Gemini was wrong, the exemplary gets marked correct for copying the correction and incorrect for fixing it.
The approach
Gemini branded the comments done OpenRouter astatine somesthesia 0 successful 25 minutes. The punctual determination that mattered astir was to ne'er inquire the exemplary for characteristic offsets. It counts characters severely and returns spans disconnected by 2 aliases 3 positions. The punctual asks for the nonstop substring and a label, and TypeScript finds the offsets. If the drawstring is not successful the comment, the entity is dropped and logged.
// The exemplary returns strings. Code computes the offsets. { "entities": [ { "text": "Benchmade", "label": "knife brand" }, { "text": "940", "label": "knife model" }, { "text": "S30V", "label": "knife steel" } ] }Product names are afloat of punctuation a generic tokenizer splits, truthful a regex keeps VG-10, CPM-154 and 1.4116 full and emits each different non-space characteristic arsenic its ain token. Spans that still miss a token bound are dropped alternatively than guessed. About 30% of the training group is comments that incorporate a known false-positive trigger (gyuto, carbon, handle, patina) and nary product, branded arsenic empty. Before the 2nd tally I group speech 225 comments arsenic a validation group and ne'er touched them again. Training ran connected a Tesla T4 connected Modal pinch the HF Trainer.
per_device_train_batch_size = 2 gradient_accumulation_steps = 8 learning_rate = 1e-5 threshold = 0.45What went wrong
For 5 runs the exemplary learned nothing. The first 3 grounded connected configuration, and anyone utilizing the HF Trainer pinch GLiNER will deed them successful an afternoon.
| 1 | GLiNER's default max_steps=10000 overrode num_train_epochs=3; trained 39 epochs | Set max_steps explicitly |
| 2 | load_best_model_at_end without eval_strategy throws | Set eval_strategy="steps" |
| 3 | Trainer saved state-dict keys without the "model." prefix GLiNER's loader expects | Put the prefix backmost connected save |
| 4 | ner_labels missing connected antagonistic examples | Set the explanation database connected each example |
| 4–5 | words_mask built arsenic binary; nonaccomplishment level astatine 70–130 | Emit incremental connection indices |
Runs 4 and 5 were the costly ones. GLiNER's tokenize_inputs collapsed connected surgery Reddit emoji, truthful I had patched it, and the spot has to capable a tensor called words_mask. It sits adjacent to attention_mask, has the aforesaid shape, and each attraction disguise I person ever built is ones for existent tokens and zeros for padding. I built it that way. Nothing astir the sanction aliases the style says it is thing different than an attraction mask.
Training ran to completion. Loss started astir 130, drifted to astir 70 and stayed there. No crash, nary warning, nary NaN, gradients of mean size, checkpoints saved connected schedule, eval F1 adjacent zero. I blamed the explanation database first, because tally 4 besides had negatives pinch nary labels set. Fixing that and rerunning gave the aforesaid level loss. The only point incorrect pinch tally 5 was words_mask, a tensor I had ne'er looked at.
How I fixed it
I publication GLiNER's training loop alternatively of its docstrings. words_mask is not a mask. It is simply a connection index: 0 for special, punctual and padding tokens, past 1, 2, 3 for the first sub-token of each existent word. The span-scoring caput uses it to excavation sub-tokens backmost into words. Filled pinch ones it says the full remark is simply a azygous word, truthful the exemplary is asked to find marque and worldly spans wrong 1 tremendous token. It cannot, truthful the nonaccomplishment stays flat, and a level nonaccomplishment does not show which input is wrong.
# what I wrote # what GLiNER expects words_mask = [1,1,1,1,1] words_mask = [0,1,2,2,3] # [CLS] Mazaki wh ##ite #2With the scale fixed, tally 6 learned connected the first try. The remainder was tuning against the locked set. The 209M mean exemplary reached 0.800; the 459M ample model, which fits a T4 only pinch gradient accumulation, reached 0.83. Ten times much adversarial negatives (510 alternatively of 51) dropped F1 to 0.799, truthful tally 10 went backmost to 51. One period per people alternatively of a world cutoff took worldly callback from 0.787 to 0.911, because alloy names for illustration MagnaCut, S35VN and HAP40 people little assurance than brands and a azygous cutoff dropped them. Every ample tally bottoms retired astatine epoch 2 and overfits after; pinch 2,000 examples that is simply a dataset-size problem, and early stopping is the fix.
What I learned
It worked. The exemplary runs locally, matches Gemini's labels astatine 0.83 F1 connected comments it ne'er saw, and knows that "carbon steel" is simply a class alternatively than a alloy and that PM2 sometimes intends the Spyderco Paramilitary 2 and sometimes is conscionable letters. One earlier tally scored 0.879 connected a random split. I do not count it arsenic the result: connected random splits, 2 drops successful F1 that I had blamed connected my changes turned retired to travel from which comments landed successful the validation set.
On insubstantial the task costs little than lunch: $9 of labels, $2.50 of GPU. What it costs maine was the days spent connected a tensor that passed each cheque the codification had and was still wrong. I deliberation the mislaid days are the normal lawsuit for mini fine-tuning jobs. The exemplary and the information are seldom the problem; the codification betwixt them fails, and a level nonaccomplishment from a incorrect input tensor looks the aforesaid arsenic a level nonaccomplishment from difficult data. If I had to take betwixt a amended explanation group and an assertion connected each tensor I hand-build, I would return the assertion.
This exemplary powers New Knife Day, which tracks what weapon group connected Reddit are buying and arguing about. The knife-side write-up location has the afloat tally log. Both are linked below.
At a glance
ProblemFind the brand, exemplary and worldly names successful Reddit comments, and skip the generic words astir them, without paying an LLM per comment
ApproachHave Gemini explanation the comments once, past fine-tune GLiNER ample v2.5 (a DeBERTa-v3-large encoder) connected those labels and tally it locally
Result0.83 F1 connected a 225-comment validation group fixed earlier training; worldly callback 0.911 pinch a per-class threshold
Cost$9 successful Gemini labels positive astir $2.50 of T4 clip crossed 10 runs
StackTypeScript and MongoDB for the scraper and labels, Python and PyTorch for training connected Modal, FastAPI to service the model
English (US) · 
Indonesian (ID) · 
