Proposing a website's page groups, their rules
and their page categories, straight from its URLs.
Each page group gathers similar pages, is built with rules on the URL, and is assigned a page category.
contains, does not contain, exactly matches, begins with, ends with, regex… on the path, the query or custom variables, combined with AND / OR.
Why it matters: a mapping is the logical view of a site that analyses build on: journey analysis, and any automatic analysis that needs page context.
Today (sketch): pick a URL part ①, an operator ②, a value ③, check the matched URLs, name the page group ④ and pick its category ⑤. Repeat for every group, on every website.
Creating a mapping is:
Too manual: every rule is written by a person.
Cumbersome and hard to learn.
Time-consuming, group after group.
A human bottleneck for every feature that depends on mappings.
Task 1Create meaningful page groups (clusters)
Task 2Determine the conditions that build each group
Task 3Determine each group's page category
websites with a reference mapping
page categories
URLs sampled per website, half weighted by traffic
curated important terms
One URL, four parts, and its terms
Each URL comes with its views, its reference page group, that group's rules and page category. Mappings are kept when they are clean: enough URLs and categories, few URLs matching two groups, most traffic mapped.
Variability lives between websites. URL design, naming, language and rule style are set per site.
More URLs of a known site add little. They mostly repeat its templates with new IDs: 1,000 URLs per site are enough.
Split by website. Train, validation and test websites are kept apart, so scores measure sites the system has never seen.
Progress comes from new websites. Each newly mapped site brings new terms and conventions.
Group similar URLs without labels, then write rules for each group. The output has the shape of a mapping: groups, rules, categories.
A first attempt classified single URLs into page categories with a character-level CNN over path, query and custom variables. It performed poorly on several categories, and a per-URL label gives neither groups nor rules.
Clustering becomes a pairwise classification: for each pair of URLs, are they in the same reference page group, and in the same proposed group?
Validation websites are never used to build anything, so the score measures websites the system has never seen.
Sample 1,000 URLs per website.
Distance, field by field: same host? same value at each path level? how many shared query and custom-variable keys, with equal values? Normalized over levels, keys and URL parts, bounded in [0, 1]. Every term weighs the same.
Hierarchical clustering builds a tree (a dendrogram) over the URLs.
Cut the tree: try every number of clusters between 10 and 40, and keep the one with the best silhouette score.
Write rules for each cluster.
± 10.8% across websites
Validation websites
Illustrative tree: the cut gives product groups split by locale, and groups mixing page types.
A match on an ID or a locale counts as much as a match on product. For example, two product URLs are 0.21 apart, and a product and a cart only 0.43.
The number of clusters comes from the silhouette score, a geometric criterion. The splits that matter, like product versus cart, can sit anywhere in the tree, even at the very bottom: groups come out fragmented, or mixing page types.
Both need to know which terms matter: the second version adds a term library.
Unsupervised: no labels at prediction time, one run per website.
Guided by the term library and by the person's choice of terms.
Explainable: every group comes with rules and the terms behind it.
Most frequent path terms in the library
| term | websites | valid | page category |
|---|---|---|---|
| search | 151 | yes | search |
| checkout | 138 | yes | checkout |
| login | 116 | yes | form |
| cart | 97 | yes | cart |
| account | 86 | yes | my_account |
| contact | 72 | yes | help_support |
| faq | 53 | yes | help_support |
Three lists: path (302 terms), query (240), custom variables (109).
Build the library
Extract the terms used in the rules of reference mappings, from training websites only.
Annotate their importance by hand, with a page category when it is clear. The flywheel keeps adding terms (slide 17).
For a new website
Extract its distinct terms, with their share of URLs and traffic.
A term is important if it is very similar to a library term (Levenshtein ratio > 0.8). It inherits that term's category.
Terms on more than 95% of URLs lose their importance: they can't tell groups apart.
Field by field: same host? same value at each path level? how many shared keys, with equal values?
Term weights: an important term weighs 0.9, any other 0.1, so matching on product counts 9× more than matching on an ID.
Part weights come from the traffic of each part's important terms. The host gets half of the smallest.
Bounded in [0, 1]. The full N × N matrix is computed in Rust.
URLs like shop.example/products/1234?some_key=toto||c_key<c_value~page_category<product; important terms: path products, cart; custom variable page_category.
Without weighting · first version
With weighting · second version
Merged last. Two URLs that differ on an important term are far apart, so they merge late: the split on that term sits near the root.
Met first. Walking down from the root, the cut reaches nodes defined by an important term before any noise split, and stops there.
Not guaranteed before. Without weighting, such nodes can sit at the bottom of the tree, and the groups come out fragmented or meaningless.
1 · Flag nodes whose rules contain an important term with a single value, e.g. level 1 = cart.
2 · Walk down from the root and stop at important nodes. Branches with none collapse to their highest node.
3 · Select the stopping nodes as page groups: two important ones and one catch-all.
| URLs in the group | /product/12345 · /product/98761 · /product/55002 |
| Look-alike outside | /product/compare?ids=1,2 |
| positive | what every URL shares: host = shop.example, level 1 = product, level 2 = any |
| negative | what excludes the look-alikes: level 2 ≠ compare |
Main idea: if a field takes few distinct values in the group, those values become a condition; if it takes many, it becomes a wildcard. An in-house library, autoconda, does this.
A translation layer turns the conditions into the product's operators. Illustrative example.
Collect the categories of the important terms in the group's rules. One category: the group gets it. Several or none: left for the person to choose.
| First version | Second version | |
|---|---|---|
| Term importance | none | inherited from the curated library |
| Distance | every term weighs the same | important terms 9×, part weights from traffic |
| Cut | number of clusters by silhouette | walk down to nodes defined by an important term |
± 10.8%
first version
± 12.2%
second version
Using term importance, in the distance and in the cut, fixes both failures of the first version (slide 8): the distance and the tree now follow business meaning.
Come mostly from imperfect importance inheritance, which the person fixes in the Term Builder.
Pairwise macro F1 · validation websites · ± across websites.
Term Builder: important terms are pre-checked; the person keeps or changes them. Values of a key can be listed too.
Clusters: proposed groups with their rules (important parts highlighted), traffic and URL share. Groups without an important rule are folded away.
Auto-mapping proposes a mapping for a new website.
A person refines it and saves it: a new, qualitative mapping.
The terms used in its rules are parsed and added to the library, keeping the train / validation / test split by website.
New websites bring new terms and conventions, so the next proposals need less correction.
Variability comes from new websites, so the system improves by being used on more of them.
Validate the payload and parse each URL into host, path, query and custom variables (urldata).
Score terms against the library; take the person's important terms.
Distance matrix in Rust, Ward clustering in SciPy, important-term cut.
Rules from autoconda, category vote; return each group with its rules, traffic and URL share, a sample URL and its category.
The "model" is code plus the curated term library. FastAPI service in one Docker image; Python 3.11 with a Rust extension.
Two-stage image build (Rust wheel, then Python) → Harbor registry → version set in the infra repo → ArgoCD.
A curated library says which URL terms carry business meaning.
Field by field, with important terms weighing 9× more.
Weighting puts important-term nodes near the root, so walking down from the root stops on them.
Readable conditions per group, translated into product rules.
And a data flywheel: every refined mapping adds terms, and every new website makes the next proposal better.
Slim Frikha · Paris NLP Meetup, 22 June 2022
youtube.com/watch?v=TRqIAQnG5JM
ContentSquare SAS · inventors Slim Frikha, Michael Snellman · filed 29 Jul 2022, granted 12 Dec 2023
patents.google.com/patent/US11841891
ContentSquare SAS · inventor Slim Frikha · filed 29 Jul 2022, granted 20 Jun 2023
patents.google.com/patent/US11681771
ContentSquare SAS · application US 18/378,812 · claims priority to US 11,841,891
patents.google.com/patent/US12346362