Automatically naming the zones of any web page,
straight from its DOM.
Zones with a business meaning are flagged so they can be monitored: each gets its own metrics, and any automatic analysis can use them as page context.
| Name | Type |
|---|---|
| search-field__input | search_bar |
The name is free text chosen by whoever flags the zone. The type comes from a shared taxonomy.
Zone Taxonomy predicts the type of every node of a page: the labels on the left.
Today in the zoning editor (sketch): open the page, click each element ①, give it a name in the zones list ②. Repeat for every zone, on every page.
Creating zonings is:
Too manual: every zone is clicked and named by a person.
Cumbersome and time-consuming, page after page.
A human bottleneck for every feature that depends on zonings.
GoalFor a given web page, spot and classify the interesting zones, automatically.
The zones you expect on a checkout page are not the ones you expect on a product page. The label space is defined per page category.
page categories
distinct zone labels
(page category, zone) pairs
Zones defined per page category
Any node that is not a zone. Every page category's label list starts with it.
Internal analysts annotated first, to kick-start the taxonomy and the first models.
A labelling company then annotated at scale, same taxonomy.
Rule: annotators tag the biggest node that contains only the zone. Each page is captured with its DOM (HTML, visibility, boxes) and a screenshot.
annotated pages
website domains (final)
pages per domain
DOM nodes
labelled zones (4%)
nodes per page
Pages per page category · labelled zones
Chart and node/zone counts: raw m2 extraction (327 domains), before training filters. Final dataset: 320 domains. Dataset versions: v0 Oct 2022, v1 Jan 2024.
Pages of one site share its templates: they differ only locally (content, a block or two). The big differences are between sites.
is the real size of the training set: ≈ 23 pages per domain
Variability lives between websites. Markup, CSS naming, language and layout are set per site.
More pages of a known site add little. More data points, but not much new variability.
Split by domain. If a site were in both train and validation, the model would be scored on templates it has memorised (leakage). Domains are assigned so each zone type keeps a ~75/25 train/validation balance.
So validation measures the real task: zoning websites the model has never seen.
Only ~4% of nodes carry a zone label: a heavily imbalanced problem.
Each site has its own markup, CSS naming and language. Pages of one site vary only locally.
On average; training pages have 20 to 5,000 visible nodes, serving accepts up to 10,000.
One visual zone spans several nested nodes. Which one "is" the zone?
All three nodes show the same price. The rule tags the biggest node holding only the zone, yet a model may answer one level in. This is the granularity problem; it comes back in the deep dive.
1 · What should the model read?
✓ simple, interpretable
✗ tied to the label set, slow to explore
✗ the problem outgrows the features
✓ strong pretrained backbones
✓ detection / segmentation models
✗ rendering cost; zone boxes are already known
✓ independent of the label set
✓ no rendering needed
✗ a model to craft from scratch
2 · What should it predict?
✓ simpler modelling and training
✗ no access to parents or neighbours
✗ which snippet: with or without children? Labels differ from user annotations
✓ sees parents and children; each node keeps only its own HTML
✓ training labels = user annotations
✗ page batching, dynamic tree graphs, long gradient paths
Input: the page's DOM tree (each node's own HTML + visibility) and its page category. Output: one label per node: a zone, or others.
Each node's own HTML snippet (without its children) is split into five signals.
| Signal | What is extracted | Tokens | Length |
|---|---|---|---|
| Markup | <a class="btn add-to-cart" id="atc-btn" … role="button"></a> | characters | 400 |
| id | atc-btn | characters | 20 |
| class | btn add-to-cart | characters | 80 |
| Attributes | Add to cart|button (title, alt, role, type, itemprop, name, label, placeholder, aria-label) | characters | 25 |
| Displayed text | Add to cart → multilingual BERT word-piece embeddings (frozen), text flavour | word pieces | 20 × 768 |
Characters come from a 102-symbol vocabulary (printable ASCII + padding + unknown). Lengths were fixed from the observed length distributions.
Same recipe for every signal. Markup and attributes use k = 3, 5, 7, 9; id, class and text use k = 3, 5. Text starts from 768-d BERT word-piece vectors instead of 8-d character embeddings.
Based on the Child-Sum Tree-LSTM: K. S. Tai, R. Socher, C. D. Manning, Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks, ACL 2015, arXiv:1503.00075.
Child-sum TreeLSTM: each node combines its own local vector with the summed states of its children, using forget gates per child.
Upward pass computes nodes level by level by height: all leaves at once, then their parents, … up to the root.
Downward pass runs a second TreeLSTM from the root, fed with the upward states, with a residual memory link.
A price node now knows it sits under a product block, next to a title and a button.
First version. Replaced by masked tree attention: next slide.
The TreeLSTM runs one step per tree level, twice. Masked attention updates every node at once: parallel and much faster on GPU. Context is still taken in two directions, as two attention tracks:
Branch track (vertical). A node attends to its ancestors and descendants within a hop limit: 6 hops in the real model (1 in this illustration). Farther ones are masked; they don't help the decision.
Sibling track (horizontal). A node attends to the other children of its parent: what sits next to it.
Real model: two stacked layers bring in first-degree cousins: in layer 1 the grandparent gathers its descendants, in layer 2 the node reads its grandparent. Heavily inspired by TrENC, not identical (±6-hop branch, 2 layers).
Based on TrENC: Li et al., Extracting Shopping Interest-Related Product Types from the Web, arXiv:2305.14549 (2023): path + sibling attention on DOM trees.
Each head is small: Linear → sigmoid → Linear → softmax over that page category's zone list.
The page category is an input: it selects which head answers, so a product page can never be given a checkout-only zone.
All page categories share the encoder and the tree-context block, so knowledge transfers across page types.
In training, the loss is masked per head: each node only trains the head of its own page category.
Easy, confidently-correct nodes (most "others") contribute little; the hard ones drive learning.
Output biases are initialised so the untrained model already predicts "others" at its true frequency, so no wasted early epochs.
Page types with fewer training nodes weigh more in the loss.
Fully invisible subtrees are removed. Remaining invisible nodes still pass through the tree model but are masked out of the loss and metrics.
Adam (lr 3·10⁻³), learning rate halved on plateau, early stopping on validation F0.5.
One price on screen, three nested nodes, one annotation: by rule, the biggest node holding only the zone. If the model answers one level in, it still found the zone.
The near-zone idea is used in three places:
Don't punish the model hard for the levels next to a zone; teach how likely each one is to be the zone.
Score the zone as found when the prediction lands one level off, and count each zone only once.
Return one node per nested cluster, with the others listed as synonyms.
Around every labelled node, its ancestors and descendants that are labelled "others" get a soft target and a loss weight, both functions of the tree distance d:
| Node | Soft target | Loss weight |
|---|---|---|
| near-certitude likely the same zone | zone: 0.25(1−s)+0.75 99% at d=1 · 80% at d=6 | 1 − s |
| incertitude ambiguous | others: 0.25·s+0.75 76% at d=1 · 95% at d=6 | s |
| labelled node · far nodes | one-hot | 1 |
Split by website. No domain is in both train and validation (no template leakage), so the score is about unseen websites. Each (page, zone) pair keeps a ≈75/25 split (±15%).
Group predictions like serving does: one node per same-label nested cluster (≤ 8 hops, ≤ 2 extra branches).
Near-zone swap. A zone predicted "others" whose neighbour (≤ 3 hops) was predicted with that zone swaps labels with it: the zone counts as found once.
Score F0.5 per zone (precision-weighted), averaged over the zones of each page category, then over page categories. This score selects checkpoints and stops training.
custom macro F0.5 on validation websites (96% on training pages)
custom macro precision
First dataset: ~400 pages · ~200 websites · 5 page categories · 29 zone types.
F0.5 varies a lot across the 29 zone types
Zc = zone types of page category c, others excluded, with support > 0. β = 0.5 weighs precision more than recall.
Drop fully invisible subtrees; keep each node's own HTML.
Vectorize, run the model, take the page category's head.
Group same-label nested clusters: keep the most confident node; the others become its prediction_synonyms.
Return the labelled nodes with their confidence, keyed by the caller's node ids (cs_path or xpath).
In the labeling Chrome extension
pick the page type, launch, display labels
predicted zones drawn on the live page, with confidence ("Product picture 94.8%") · sketch
Character CNNs over markup, id, class, attributes, plus text.
Masked tree attention along branches and siblings: what's inside, where it sits.
One head per page category, one shared backbone.
Soft labels to train, tolerant scoring to evaluate, grouping to serve.
Trained on 320 websites, measured on sites it has never seen: DOM tree in, one confident answer per zone out.
Slim Frikha · Deep Learning Paris Meetup (Paris Machine Learning Applications Group), 5 July 2022
youtube.com/watch?v=QIfmJHyQIlY
ContentSquare SAS · inventors Slim Frikha, Mohamed-Amine Baatout · filed 29 Jul 2022, granted 29 Aug 2023
patents.google.com/patent/US11741186
Tree-LSTM (first tree-context version): K. S. Tai, R. Socher, C. D. Manning, Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks, ACL 2015, arXiv:1503.00075.
Tree attention design: Y. Li, C. Lockard, P. Shiralkar, C. Zhang, Extracting Shopping Interest-Related Product Types from the Web, arXiv:2305.14549 (2023).