Instance view of the static supplierName (keeps this.supplierName working).
Instance view of the static baseURL (keeps this.baseURL working).
PrivatesupplierThis supplier's own class, typed for static metadata access. this.constructor
is otherwise typed Function (no custom statics), so the instance getters below
read the concrete class's static fields through this narrowed view.
ProtectedsupportsInstance view of the static supportsCAS flag (keeps this.supportsCAS working).
ProtectedsupportsInstance view of the static supportsFormula flag.
ProtectedsupportsInstance view of the static supportsSMILES flag.
Instance view of the static shipping scope (keeps this.shipping working).
Instance view of the static country (keeps this.country working).
Instance view of the static paymentMethods (keeps this.paymentMethods working).
ProtectedshipsInstance view of the static shipsTo allowlist (keeps this.shipsTo working).
ProtectedapiInstance view of the static apiURL (keeps this.apiURL working).
StaticrequiredAll host origin patterns required for this supplier to function.
Automatically includes baseURL and, if defined, apiURL. Used by the
factory to check chrome permissions before querying.
Instance view of the static requiredHosts.
ProtectedeffectiveThe single query term this supplier should search by. Normally the raw query, but when the query is a CAS/formula/SMILES identifier the supplier can't search for itself (per supportsCAS/supportsFormula/supportsSMILES) and the factory resolved it, it's the broadest resolved name — the one likely to yield the most store results — so a name-only supplier can answer "Na6O18P6" by searching "Sodium hexametaphosphate". Matching then scores results against every candidate (see effectiveQueryCandidates/fuzzyScore). Falls back to the raw query when the type is supported, the query is a plain name, or nothing was resolved.
The effective search query.
// this.query === "10124-56-8", supportsCAS === false, resolved names available
this.effectiveQuery; // "Sodium hexametaphosphate"
Creates a new instance of the supplier base class. Initializes the supplier with query parameters, request limits, and abort controller. Sets up logging and default product values.
The search term to query products for
The maximum number of results to return (default: 5)
Optionalcontroller: AbortControllerAbortController instance for managing request cancellation
// Create a supplier with default limit
const supplier = new MySupplier("sodium chloride", undefined, new AbortController());
// Create a supplier with custom limit
const supplier = new MySupplier("acetone", 10, new AbortController());
// Create a supplier and handle cancellation
const controller = new AbortController();
const supplier = new MySupplier("ethanol", 5, controller);
// Later, to cancel all pending requests:
controller.abort();
Determines whether this supplier ships to the given country. Prefers the
explicit shipsTo allowlist when the supplier declares one; otherwise falls
back to the coarse shipping scope — "worldwide"/"international" ship
anywhere, while "domestic"/"local" ship only within the supplier's own
country.
The user's location as an ISO 3166-1 alpha-2 country code.
True if the supplier ships to location, false otherwise.
// Supplier with shipsTo = ["US", "CA"]:
supplier.shipsToCountry("US"); // true
supplier.shipsToCountry("DE"); // false
// Domestic US supplier (no shipsTo):
supplier.shipsToCountry("US"); // true
supplier.shipsToCountry("DE"); // false
// Worldwide supplier (no shipsTo):
supplier.shipsToCountry("DE"); // true
public shipsToCountry(location: CountryCode): boolean {
// Pass an explicit meta object rather than `this.constructor` so the
// (protected) `shipsTo` static doesn't clash with SupplierStaticMeta's public
// shape; the getters read the concrete class's static values.
return SupplierBase.shipsToCountryStatic(
{ shipping: this.shipping, country: this.country, shipsTo: this.shipsTo },
location,
);
}
StaticshipsWhether a supplier with the given static shipping metadata ships to
location. Shared by the instance shipsToCountry and SupplierFactory
so the UI can test shipping compatibility from a supplier's static fields
without instantiating it.
The supplier's static shipping/country/shipsTo metadata.
Destination country (ISO 3166-1 alpha-2).
True when the supplier ships to location.
public static shipsToCountryStatic(meta: SupplierStaticMeta, location: CountryCode): boolean {
if (meta.shipsTo) {
return meta.shipsTo.includes(location);
}
switch (meta.shipping) {
case 'worldwide':
case 'international':
return true;
case 'domestic':
case 'local':
return meta.country === location;
default:
return true;
}
}
Initializes the cache for the supplier. This is called after construction to ensure supplierName is set.
The cache is initialized with the supplier's name and is used to store both query results and product data. This method should be called after the supplier's name is set to ensure proper cache key generation.
class MySupplier extends SupplierBase<Product> {
constructor() {
super("acetone", 5);
// supplierName is set here
this.initCache(); // Initialize cache after supplierName is set
}
}
public initCache(
enabled: boolean = true,
doNotCacheEmptyResults: boolean = false,
cacheTtlMinutes: number = 0,
noCacheStatusCodes: number[] = [429],
): void {
this.cache = new SupplierCache(
this.supplierName,
this.constructor.name,
enabled,
doNotCacheEmptyResults,
cacheTtlMinutes,
);
// Stored on the supplier (not the cache): the decision is made at cache-write time in
// getProductData(WithCache), where the per-product fetch status is known.
this.noCacheStatusCodes = noCacheStatusCodes ?? [429];
}
Applies (or clears) a runtime override for the fuzz scorer. Driven by
userSettings.fuzzScorerOverride — when the user picks a scorer in the
Advanced drawer section, SupplierFactory calls this on each instance
so the choice takes effect uniformly across every supplier.
Silently ignores unknown names so an outdated / corrupted setting can't blow up the search flow — callers fall back to the subclass default.
Name of a scorer from FUZZ_SCORERS, or undefined to
clear the override and use the subclass default.
const supplier = new MySupplier("acetone", 5, controller);
supplier.setFuzzScorerOverride("token_set_ratio");
// fuzzyFilter now uses token_set_ratio regardless of MySupplier's default
supplier.setFuzzScorerOverride(undefined);
// back to MySupplier's default
public setFuzzScorerOverride(name: string | undefined): void {
console.debug('setFuzzScorerOverride', { name });
if (isFuzzScorerName(name)) {
this.fuzzScorerOverride = FUZZ_SCORERS[name];
} else {
this.fuzzScorerOverride = undefined;
}
}
Applies a runtime override for maxAllowableSearchTimeSec, driven by
userSettings.maxAllowableSearchTimeSec (set in the Advanced settings section). Accepts the raw
setting value (which may arrive as a string from the number input). An absent, empty, or
invalid value is ignored so the supplier keeps its class default; a valid non-negative number
(including 0 to disable the limit) replaces it.
The override in seconds, or any value (invalid/empty input is ignored)
supplier.setMaxAllowableSearchTimeSec(60); // cap searches at 60s
supplier.setMaxAllowableSearchTimeSec(""); // no-op, keep the per-supplier default
public setMaxAllowableSearchTimeSec(value: unknown): void {
if (value === undefined || value === null || value === '') {
return;
}
const seconds = Number(value);
if (!Number.isNaN(seconds) && seconds >= 0) {
this.maxAllowableSearchTimeSec = seconds;
}
}
Sets the parsed advanced-search query for this instance. Called by
SupplierFactory once per search so every supplier shares the same parse of
the user's input.
The parsed query, or undefined to clear it.
supplier.setParsedQuery(parseSearchQuery("Sodium OR Potassium"));
public setParsedQuery(parsed: ParsedSearchQuery | undefined): void {
this.parsedQuery = parsed;
}
Applies a runtime override for fuzzyFilteringDisabled, driven by
userSettings.fuzzyFilteringDisabled. When true, fuzzball scoring is skipped
and only the boolean predicate (substring matching) is applied.
True to disable fuzzy filtering, false (default) to keep it.
supplier.setFuzzyFilteringDisabled(true); // show raw/boolean-only results
public setFuzzyFilteringDisabled(value: boolean): void {
this.fuzzyFilteringDisabled = value === true;
}
Sets the map of resolved structure terms for this instance. Called by
SupplierFactory once per search so every supplier shares one resolution of
any SMILES/structure terms instead of each hitting the network.
Map of raw search term → resolved structure, or undefined when none.
supplier.setResolvedStructures(new Map([["CCO", { name: "ethanol", cas: ["64-17-5"] }]]));
public setResolvedStructures(resolved: ReadonlyMap<string, ResolvedStructure> | undefined): void {
this.resolvedStructures = resolved;
this.effectiveQueryCandidatesCache = undefined;
}
ProtectedeffectiveThe candidate queries this supplier matches against. Normally just
[this.query], but when the query is a CAS/formula/SMILES identifier the
supplier can't search itself (per supportsCAS/supportsFormula/
supportsSMILES), it's the factory-resolved chemical-name candidates —
the compound Title plus cleaned synonyms — so a product can be matched by
whichever name scores best (see fuzzyScore). Falls back to
[this.query] when the type is supported, the query is a plain name, or
nothing was resolved. Memoized.
The candidate queries, best first (never empty).
// this.query === "10124-56-8", supportsCAS === false, resolution available
this.effectiveQueryCandidates(); // ["Hexasodium hexametaphosphate", "Sodium hexametaphosphate", …]
protected effectiveQueryCandidates(): string[] {
if (this.effectiveQueryCandidatesCache === undefined) {
this.effectiveQueryCandidatesCache = this.resolveEffectiveQueryCandidates();
}
return this.effectiveQueryCandidatesCache;
}
ProtectedgetReturns the parsed query for this instance. Uses the effectiveQuery, so
an identifier query the supplier can't search is parsed as its resolved name.
Otherwise lazily parses this.query when SupplierFactory did not set a
parsed query (e.g. in unit tests that construct a supplier directly).
The parsed search query.
const { isAdvanced, ast } = this.getAst();
protected getAst(): ParsedSearchQuery {
const effective = this.effectiveQuery;
if (effective !== this.query) {
return parseSearchQuery(effective);
}
return this.parsedQuery ?? parseSearchQuery(this.query);
}
ProtectedsetupPlaceholder for any setup that needs to be done before the query is made. Override this in subclasses if you need to perform setup (e.g., authentication, token fetching).
A promise that resolves when the setup is complete.
await supplier.setup();
protected async setup(): Promise<void> {}
ProtectedhttpRetrieves HTTP headers from a URL using a HEAD request. Useful for checking content types, caching headers, and other metadata without downloading the full response.
The URL to fetch headers from
Promise resolving to the response headers or void if request fails
// Basic usage
const headers = await supplier.httpGetHeaders('https://example.com/product/123');
if (headers) {
console.log('Content-Type:', headers['content-type']);
}
// With error handling
try {
const headers = await supplier.httpGetHeaders('https://example.com/product/123');
if (headers) {
console.log('Headers:', headers);
}
} catch (err) {
console.error('Failed to fetch headers:', err);
}
protected async httpGetHeaders(url: string | URL): Promise<Maybe<HeadersInit>> {
const requestObj = new Request(this.href(url), {
signal: this.controller.signal,
headers: new Headers(this.headers),
referrer: this.baseURL,
referrerPolicy: 'strict-origin-when-cross-origin',
body: null,
method: 'HEAD',
mode: 'cors',
credentials: 'include',
});
try {
const httpResponse = await this.fetch(requestObj);
return Object.fromEntries(httpResponse.headers.entries()) satisfies HeadersInit;
} catch (error: unknown) {
if (error instanceof Error && error.name === 'AbortError') {
this.logger.warn('Request was aborted', { error, signal: this.controller.signal });
this.controller.abort('Abort signal detected');
} else {
this.logger.error('Error received during fetch:', {
error,
signal: this.controller.signal,
});
}
return;
}
}
ProtectedhttpSends a POST request to the given URL with the given body and headers. Handles request setup, error handling, and response caching.
The request configuration options
Promise resolving to the Response object or void if request fails
// Basic POST request
const response = await supplier.httpPost({
path: '/api/v1/products',
body: { name: 'Test Chemical' }
});
// POST with custom headers
const response = await supplier.httpPost({
path: '/api/v1/products',
body: { name: 'Test Chemical' },
headers: {
'Authorization': 'Bearer token123',
'Content-Type': 'application/json'
}
});
// POST with custom host and params
const response = await supplier.httpPost({
path: '/api/v1/products',
host: 'api.example.com',
body: { name: 'Test Chemical' },
params: { version: '2' }
});
// Error handling
try {
const response = await supplier.httpPost({ path: '/api/v1/products', body: { name: 'Test' } });
if (response && response.ok) {
const data = await response.json();
console.log('Created:', data);
}
} catch (err) {
console.error('POST failed:', err);
}
protected async httpPost({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<Response>> {
this.logger.log('httpPost| Requesting:', {
path,
host,
body,
params,
headers,
});
const method = 'POST';
const mode = 'cors';
const referrer = this.baseURL;
const referrerPolicy = 'strict-origin-when-cross-origin';
const signal = this.controller.signal;
const headersObj = new Headers({
...this.headers,
...headers,
});
let bodyStr = null;
if (body instanceof FormData) {
headersObj.set('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8');
bodyStr = body;
} else if (typeof body === 'string') {
bodyStr = body;
} else if (typeof body === 'object' && body !== null) {
bodyStr = JSON.stringify(body);
}
const url = this.href(path, params, host);
const requestObj = new Request(url, {
signal,
headers: headersObj,
referrer,
referrerPolicy,
body: bodyStr,
method,
mode,
credentials: 'include',
});
// Fetch the goods
const httpResponse = await this.fetch(requestObj);
if (!isHttpResponse(httpResponse) || !httpResponse.ok) {
const badResponse = await httpResponse.text();
this.logger.error('Invalid POST response: ', badResponse);
throw new TypeError(`Invalid POST response: ${String(httpResponse)}`);
}
return httpResponse;
}
ProtectedhttpSends a POST request with the body encoded as multipart/form-data.
Converts the given object into a FormData instance (one field per
key/value pair) before delegating to httpPost.
The request configuration options. body must be a non-null object.
Promise resolving to the Response object or void if the request fails
TypeError - If body is not an object, or the response is not a valid HTTP response
const response = await this.httpPostFormData({
path: "/api/v1/cart",
body: { productId: "123", quantity: "2" },
});
protected async httpPostFormData({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<Response>> {
if (typeof body !== 'object' || body === null) {
throw new TypeError('httpPostFormData| Body must be an object');
}
headers = {
...headers,
'Content-Type': 'application/x-www-form-urlencoded',
};
const formData = new FormData();
for (const [key, value] of Object.entries(body)) {
formData.append(key, value);
}
const httpResponse = await this.httpPost({ path, host, body: formData, params, headers });
if (!isHttpResponse(httpResponse) || !httpResponse.ok) {
const badResponse = await httpResponse?.text();
this.logger.error('Invalid POST response: ', badResponse);
throw new TypeError(`Invalid POST response: ${String(httpResponse)}`);
}
this.logger.log('httpPostFormData| Successfully sent POST request to:', path);
return httpResponse;
}
ProtectedhttpSends a POST request and returns the response as a JSON object.
The parameters for the POST request.
The response from the POST request as a JSON object.
// Basic usage
const data = await supplier.httpPostJson({
path: '/api/v1/products',
body: { name: 'John' }
});
// With custom headers and error handling
try {
const data = await supplier.httpPostJson({
path: '/api/v1/products',
body: { name: 'John' },
headers: { 'Authorization': 'Bearer token123' }
});
if (data) {
console.log('Created:', data);
}
} catch (err) {
console.error('POST JSON failed:', err);
}
protected async httpPostJson({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<JsonValue>> {
const httpResponse = await this.httpPost({ path, host, body, params, headers });
if (!isJsonResponse(httpResponse) || !httpResponse.ok) {
this.logger.error('httpPostJson| Invalid POST response: ', {
httpResponse,
path,
host,
body,
params,
headers,
});
throw new TypeError(`httpPostJson| Invalid POST response: ${httpResponse}`);
}
return await httpResponse.json();
}
ProtectedhttpSends a POST request and returns the response as a HTML string.
The request configuration options
Promise resolving to the HTML response as a string or void if request fails
TypeError - If the response is not valid HTML content
// Basic usage
const html = await supplier.httpPostHtml({
path: '/api/v1/products',
body: { name: 'John' }
});
protected async httpPostHtml({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<string>> {
const httpResponse = await this.httpPost({ path, host, body, params, headers });
if (!isHtmlResponse(httpResponse)) {
throw new TypeError(`httpPostHtml| Invalid POST response: ${httpResponse}`);
}
return await httpResponse.text();
}
ProtectedhttpSends a GET request to the given URL with the specified options. Handles request setup, error handling, and response caching.
The request configuration options
Promise resolving to the Response object or void if request fails
// Basic GET request
const response = await supplier.httpGet({
path: '/products/search',
params: { query: 'sodium chloride' }
});
// GET with custom headers
const response = await supplier.httpGet({
path: '/products/search',
headers: { 'Accept': 'application/json' }
});
// GET with custom host
const response = await supplier.httpGet({
path: '/products/search',
host: 'api.example.com',
params: { category: 'chemicals' }
});
// Error handling
try {
const response = await supplier.httpGet({ path: '/products/search' });
if (response && response.ok) {
const data = await response.json();
console.log('Products:', data);
}
} catch (err) {
console.error('GET failed:', err);
}
protected async httpGet({
path,
params,
headers,
host,
rethrowErrors,
}: RequestOptions): Promise<Maybe<Response>> {
// Check if the request has been aborted before proceeding
if (this.controller.signal.aborted) {
this.logger.warn('Request was aborted before fetch', {
signal: this.controller.signal,
});
return;
}
const headersRaw = { ...this.headers };
Object.assign(headersRaw, {
accept: [
'text/html',
'application/xhtml+xml',
'application/xml;q=0.9',
'image/avif',
'image/webp',
'image/apng',
'*/*;q=0.8',
].join(','),
...(headers ?? {}),
});
const requestObj = new Request(this.href(path, params, host), {
signal: this.controller.signal,
headers: new Headers(headersRaw),
referrer: this.baseURL,
referrerPolicy: 'no-referrer',
body: null,
method: 'GET',
mode: 'cors',
credentials: 'include',
redirect: 'follow',
});
try {
// Fetch the goods
const httpResponse = await this.fetch(requestObj.url, requestObj);
const responseHeaders = Object.fromEntries(
httpResponse.headers.entries(),
) satisfies HeadersInit;
this.logger.debug('responseHeaders:', responseHeaders);
this.logger.debug('responseHeaders.location:', responseHeaders.location);
return httpResponse;
} catch (error: unknown) {
if (error instanceof Error && error.name === 'AbortError') {
this.logger.warn('Request was aborted', { error, signal: this.controller.signal });
this.controller.abort('Abort signal detected');
return;
}
this.logger.error('Error received during fetch:', {
error,
signal: this.controller.signal,
});
// Opt-in: surface the failure (e.g. an HttpError 429) so the caller can apply
// status-aware retry/backoff. Default behavior remains swallow-and-return-undefined.
if (rethrowErrors) {
throw error;
}
return;
}
}
ProtectedfuzzyScores a single string against the current search query (this.query)
using the active fuzz scorer — the user's fuzzScorerOverride when set,
otherwise the supplier's fuzzScorer. Returns a 0–100 similarity score
(higher is closer). Useful for suppliers that can only fuzz-match after a
secondary request reveals the real product name, e.g. when the search
index only exposes coarse category breadcrumbs.
The text to score against this.query.
A similarity score from 0 (no match) to 100 (identical).
// this.query === "sodium borohydride"
this.fuzzyScore("Sodium borohydride, min 95%"); // ~90
this.fuzzyScore("Acetone"); // ~10
protected fuzzyScore(text: string): number {
const activeScorer = this.fuzzScorerOverride ?? this.fuzzScorer;
// Score against every effective-query candidate and keep the best — for an
// identifier query these are the resolved name + synonyms, so a product
// matches whichever name it names. A plain query has a single candidate.
let best = 0;
for (const candidate of this.effectiveQueryCandidates()) {
const score = activeScorer(candidate, text);
if (score > best) {
best = score;
}
}
return best;
}
ProtectedfuzzyFilters an array of data using fuzzy string matching to find items that closely match a query string. Uses the WRatio algorithm from fuzzball for string similarity comparison.
The search string to match against
Array of data objects to search through
Minimum match percentage (0-100) for a match to be included (default: 55)
Array of matching data objects with added fuzzy match metadata
// Example with simple string array
const products = [
{ title: "Sodium Chloride", price: 29.99 },
{ title: "Sodium Hydroxide", price: 39.99 },
{ title: "Potassium Chloride", price: 19.99 }
];
const matches = this.fuzzyFilter("sodium chloride", products);
// Returns: [
// {
// title: "Sodium Chloride",
// price: 29.99,
// _fuzz: { score: 100, idx: 0 }
// },
// {
// title: "Sodium Hydroxide",
// price: 39.99,
// _fuzz: { score: 85, idx: 1 }
// }
// ]
// Example with custom minMatchPercentage
const strictMatches = this.fuzzyFilter("sodium chloride", products, 90);
// Returns only exact matches with score >= 90
// Example with different data structure
const chemicals = [
{ name: "NaCl", formula: "Sodium Chloride" },
{ name: "NaOH", formula: "Sodium Hydroxide" }
];
// Override titleSelector to use formula field
this.titleSelector = (data) => data.formula;
const formulaMatches = this.fuzzyFilter("sodium chloride", chemicals);
protected fuzzyFilter<X>(
query: string,
data: X[],
minMatchPercentage: number = this.minMatchPercentage,
): X[] {
// User's Advanced-settings override wins over the subclass default.
const activeScorer = this.fuzzScorerOverride ?? this.fuzzScorer;
// console.log(
// `[fuzzyFilter] ${this.supplierName} query="${query}" — scorer comparison (cutoff=${minMatchPercentage})`,
// );
if (IS_DEV_BUILD) {
this.showFuzzScorerComparisonTable(query, data);
}
if (this.fuzzyFilterRankOnly) {
// Rank every candidate by score (no cutoff) and return them in score order; the
// caller slices the top N. Avoids dropping clear matches whose ratio-style score
// falls under minMatchPercentage purely because the title dwarfs the query.
return extract(query, data, {
scorer: activeScorer,
processor: this.titleSelector,
sortBySimilarity: true,
}).map(([obj, score, idx]) => this.attachFuzz(obj, score, idx));
}
const results = extract(query, data, {
scorer: activeScorer,
processor: this.titleSelector,
cutoff: minMatchPercentage,
sortBySimilarity: true,
}).reduce<FuzzyMatchResult<X>[]>((acc, [obj, score, idx]) => {
if (score < minMatchPercentage) {
this.logger.debug('fuzzyFilter: score below minimum match percentage, excluding product', {
product: obj,
score,
idx,
minMatchPercentage: minMatchPercentage,
});
return acc;
}
acc[idx] = Object.assign(obj, { _fuzz: { score, idx }, matchPercentage: score });
return acc;
}, []);
this.logger.debug('[fuzzyFilter]', {
supplierName: this.supplierName,
query,
minMatchPercentage,
activeScorer,
results,
});
// Get rid of any empty items that didn't match closely enough
return results.filter((item) => !!item);
}
ProtectedfuzzyAdvanced-search-aware companion to fuzzyFilter. Filters data using
the parsed query (getAst) so boolean operators (AND/OR/NOT) and
nesting are honored, and respects the fuzzyFilteringDisabled toggle.
Behavior matrix:
data unchanged (raw supplier results).Array of raw search-result objects to filter.
Minimum leaf match score when fuzzing is on.
The filtered (and, when fuzzing, ranked) subset, each item tagged
with _fuzz/matchPercentage like fuzzyFilter.
// this.query === "Sodium OR Potassium"
const matches = this.fuzzyFilterAst(products);
protected fuzzyFilterAst<X>(
data: X[],
minMatchPercentage: number = this.minMatchPercentage,
): X[] {
const parsed = this.getAst();
if (!parsed.isAdvanced) {
// Plain query: no filtering when disabled; the multi-candidate path for a
// resolved identifier query (several candidate names); else the legacy
// single-query fuzzy path.
if (this.fuzzyFilteringDisabled) {
return data;
}
if (this.effectiveQueryCandidates().length > 1) {
return this.fuzzyFilterCandidates(data, minMatchPercentage);
}
return this.fuzzyFilter(parsed.raw.trim(), data, minMatchPercentage);
}
const scorer = this.fuzzScorerOverride ?? this.fuzzScorer;
// Rank-only floors the leaf score at 0 so predicate-matching items are never dropped
// for a low fuzz score; the sort below still ranks them. Otherwise enforce the cutoff.
const threshold = this.fuzzyFilteringDisabled
? 1
: this.fuzzyFilterRankOnly
? 0
: minMatchPercentage;
const fuzzyWords = !this.fuzzyFilteringDisabled;
const matched = data.reduce<FuzzyMatchResult<X>[]>((acc, obj, idx) => {
const title = this.titleSelector(obj) ?? '';
const score = scoreAstMatch(title, parsed.ast, { scorer, threshold, fuzzyWords });
if (score === null) {
return acc;
}
acc.push(this.attachFuzz(obj, score, idx));
return acc;
}, []);
// Rank by relevance when fuzzing; preserve backend order when disabled.
if (!this.fuzzyFilteringDisabled) {
matched.sort((a, b) => (b.matchPercentage ?? 0) - (a.matchPercentage ?? 0));
}
return matched;
}
ProtectedfuzzyAdvanced-search-aware, keep-or-drop companion to fuzzyScore for
suppliers that re-filter a single title after a detail fetch (e.g. LiMac).
Returns the score to keep the item, or null to drop it, honoring both the
parsed query and the fuzzyFilteringDisabled toggle:
minMatchPercentage;
advanced query keeps items satisfying the predicate with fuzzy leaf scores.The text (e.g. a detail-page product name) to score.
A 0–100 score to keep the item, or null to drop it.
// this.query === "acid AND NOT boric"
this.fuzzyScoreAst("Sulfuric acid"); // a number
this.fuzzyScoreAst("Boric acid"); // null
protected fuzzyScoreAst(text: string): number | null {
const parsed = this.getAst();
if (!parsed.isAdvanced) {
if (this.fuzzyFilteringDisabled) {
return 100;
}
const score = this.fuzzyScore(text);
return score >= this.minMatchPercentage ? score : null;
}
const scorer = this.fuzzScorerOverride ?? this.fuzzScorer;
const threshold = this.fuzzyFilteringDisabled ? 1 : this.minMatchPercentage;
return scoreAstMatch(text, parsed.ast, {
scorer,
threshold,
fuzzyWords: !this.fuzzyFilteringDisabled,
});
}
ProtectedderiveDerives the backend search terms for a keyword-only supplier from an advanced query: one representative term per positive OR-group (the longest — most selective — token of each AND-group), de-duplicated and capped at maxFallbackQueries. Returns an empty array when there are no positive terms (e.g. a purely negative query), in which case the caller falls back to a single raw search.
The de-duplicated, capped list of backend search terms.
// this.query === "(Sodium OR Potassium) AND Hydroxide"
this.deriveFallbackTerms(); // ["Hydroxide", "Hydroxide"] -> ["Hydroxide"] (deduped)
protected deriveFallbackTerms(): string[] {
const groups = extractOrGroups(this.getAst().ast);
// How many AND-groups each term appears in — a term shared across groups (a
// common factor, e.g. "Hydroxide" in "(Sodium OR Potassium) AND Hydroxide")
// covers more of the query in fewer requests, so prefer it; break ties by
// length (more selective). Picking one representative term per group keeps
// each request's result set a superset of that group's matches.
const frequency = new Map<string, number>();
for (const group of groups) {
for (const term of new Set(group)) {
frequency.set(term, (frequency.get(term) ?? 0) + 1);
}
}
const terms = groups
.map(
(group) =>
group.slice().sort((a, b) => {
const byFrequency = (frequency.get(b) ?? 0) - (frequency.get(a) ?? 0);
return byFrequency !== 0 ? byFrequency : b.length - a.length;
})[0],
)
.filter((term): term is string => Boolean(term));
return [...new Set(terms)].slice(0, this.maxFallbackQueries);
}
ProtectedhttpMakes an HTTP GET request and returns the response as a string. Handles request configuration, error handling, and HTML parsing.
The request configuration options
Promise resolving to the HTML response as a string or void if request fails
TypeError - If the response is not valid HTML content
// Basic GET request
const html = await this.httpGetHtml({
path: "/api/products",
params: { search: "sodium" }
});
// GET request with custom headers
const html = await this.httpGetHtml({
path: "/api/products",
headers: {
"Authorization": "Bearer token123",
"Accept": "text/html"
}
});
// GET request with custom host
const html = await this.httpGetHtml({
path: "/products",
host: "api.supplier.com",
params: { limit: 10 }
});
protected async httpGetHtml({
path,
params,
headers,
host,
}: RequestOptions): Promise<Maybe<string>> {
const httpResponse = await this.httpGet({ path, params, headers, host });
if (!isHtmlResponse(httpResponse)) {
throw new TypeError(`httpGetHtml| Invalid GET response: ${httpResponse}`);
}
return await httpResponse.text();
}
ProtectedhttpMakes an HTTP GET request and returns the response as parsed JSON. Handles request configuration, error handling, and JSON parsing.
The request configuration options
Promise resolving to the parsed JSON response or void if request fails
TypeError - If the response is not valid JSON content
// Basic GET request
const data = await supplier.httpGetJson({ path: '/api/products', params: { search: 'sodium' } });
// GET request with custom headers
const data = await supplier.httpGetJson({
path: '/api/products',
headers: {
'Authorization': 'Bearer token123',
'Accept': 'application/json'
}
});
// GET request with custom host
const data = await supplier.httpGetJson({
path: '/products',
host: 'api.supplier.com',
params: { limit: 10 }
});
// Error handling
try {
const data = await supplier.httpGetJson({ path: '/api/products' });
if (data) {
console.log('Products:', data);
}
} catch (error) {
console.error('Failed to fetch products:', error);
}
protected async httpGetJson({
path,
params,
headers = {},
host,
}: RequestOptions): Promise<Maybe<JsonValue>> {
if (
typeof headers?.accept === 'undefined' ||
!Array.isArray(headers?.accept) ||
!headers?.accept.includes('application/json')
) {
headers.accept = ['application/json', 'text/plain', '*/*'].join(',');
}
const httpRequest = await this.httpGet({ path, params, headers, host });
if (!isJsonResponse(httpRequest)) {
const badResponse = isHttpResponse(httpRequest) ? await httpRequest.text() : undefined;
this.logger.error('Invalid HTTP GET JSON response:', {
badResponse,
httpRequest,
path,
params,
headers,
host,
});
return;
}
return await httpRequest.json();
}
ProtectedqueryExecutes a product search query with caching support. First checks the cache for existing results, then falls back to the actual query if needed. The limit parameter is only used for the actual query and doesn't affect caching.
The search term to query products for
The maximum number of results to return (defaults to instance limit)
Promise resolving to array of product builders or void if search fails
// Basic usage with default limit
const results = await supplier.queryProductsWithCache("acetone");
if (results) {
console.log(`Found ${results.length} products`);
}
// With custom limit
const results = await supplier.queryProductsWithCache("acetone", 10);
if (results) {
for (const builder of results) {
const product = await builder.build();
console.log(product.title, product.price);
}
}
protected async queryProductsWithCache(
query: string,
limit: number = this.limit,
): Promise<ProductBuilder<T>[] | void> {
// Check cache first (processed product data)
this.logger.debug(
'queryProductsWithCache: called for',
this.supplierName,
'query:',
query,
'limit:',
limit,
);
const key = this.cache.generateCacheKey(query);
const cached = await this.cache.getCachedQueryEntry(key);
this.logger.debug('queryProductsWithCache: cache hit:', !!cached, 'key:', key);
if (cached) {
const cachedLimit = cached.__cacheMetadata.limit;
const insufficientLimit = typeof cachedLimit === 'number' && cachedLimit < limit;
if (!insufficientLimit) {
this.logger.debug('Returning cached query results');
// Re-initialize product builders from cached processed data
return ProductBuilder.createFromCache<T>(this.baseURL, cached.data.slice(0, limit));
}
// Cached entry was built with a smaller limit than requested — drop it and re-query below.
this.logger.debug('Invalidating query cache due to insufficient limit', {
cachedLimit,
requestedLimit: limit,
});
await deleteSupplierQueryCacheEntry(key);
}
// If not in cache, perform the actual query. Run setup first so any
// subclass state it mutates (headers, localStorage, tokens, etc.) is
// in place before `queryProducts` reads it. Memoized, so this is cheap
// on repeat calls within the same supplier instance.
await this.ensureSetup();
const results = await this.queryProductsResolved(query, limit);
if (results) {
// Store processed results in cache (dumped/serialized form) and the limit used
await this.cache.cacheQueryResults(
query,
results.map((b) => b.dump()),
limit,
);
}
return results;
}
ProtectedqueryResolves the supplier's queryProducts for the current search, applying the
keyword-only advanced-search fallback when needed. For a plain query, or for
a supplier that handles boolean queries natively
(supportsNativeAdvancedSearch), this is a single queryProducts
call. For an advanced query on a keyword-only backend, it issues one search
per derived OR-group term (deriveFallbackTerms) and unions the
results, deduping by product URL/ID. Each queryProducts batch already
enforces the full boolean predicate via fuzzyFilterAst, so the union
is the set of products matching the whole query. Honors
httpRequestHardLimit; the fetches run inside execute()'s existing
maxAllowableSearchTimeSec race so an aborted controller cancels them.
The raw search query.
The per-supplier result limit.
The (possibly unioned) product builders, or void.
protected async queryProductsResolved(
query: string,
limit: number,
): Promise<ProductBuilder<T>[] | void> {
const parsed = this.getAst();
if (!parsed.isAdvanced || this.supportsNativeAdvancedSearch) {
return this.queryProducts(query, limit);
}
const terms = this.deriveFallbackTerms();
if (terms.length <= 1) {
// Nothing to union (single positive term, or a purely negative query):
// run the raw query once and let fuzzyFilterAst enforce the predicate.
return this.queryProducts(terms[0] ?? query, limit);
}
const seen = new Set<string>();
const union: ProductBuilder<T>[] = [];
for (const term of terms) {
if (this.requestCount >= this.httpRequestHardLimit) {
this.logger.warn('queryProductsResolved: httpRequestHardLimit reached, stopping fallback', {
term,
requestCount: this.requestCount,
});
break;
}
const batch = await this.queryProducts(term, limit);
if (!batch) {
continue;
}
for (const builder of batch) {
const key = String(builder.get('url') ?? builder.get('id') ?? builder.get('title') ?? '');
if (key === '' || seen.has(key)) {
continue;
}
seen.add(key);
union.push(builder);
}
}
return union.length > 0 ? union : undefined;
}
Executes the supplier's search query and returns the results. This method will execute all results concurrently (to the limits set in the supplier class), and resolve to an array of product objects.
Promise resolving to an array of products
This method is used to execute the supplier's search query and return the results.
public async *execute(): AsyncGenerator<T, void, undefined> {
// setup() is not called eagerly here — it's run lazily from the
// phase-boundary gates inside `queryProductsWithCache` and
// `getProductData` / `getProductDataWithCache`. A fully cached search
// never reaches those gates, so setup's token/cookie/permission
// requests are skipped entirely.
// Snapshot the user's ignore list once per search. Any product whose
// exclusion key matches an entry here is dropped before the detail phase
// runs (see the filter after queryProductsWithCache below).
this.excludedProductKeys = await loadExcludedProductKeys();
// Over-fetch by the number of previously-ignored products belonging to
// this supplier so that, in the worst case where every ignored product
// appears in the top of the query result set, we still end up with
// `this.limit` survivors after filtering. The queryProductsWithCache
// cache invalidates itself when the requested limit exceeds the cached
// limit, so this is safe.
const excludedForSupplier = await countExcludedProductsForSupplier(this.supplierName);
const fetchLimit = this.limit + excludedForSupplier;
incrementSearchQueryCount(this.supplierName);
// Optional per-supplier search-time budget; see armSearchTimeout. When it elapses the
// race below wins via SEARCH_TIMEOUT and flushes any not-yet-yielded products.
const SEARCH_TIMEOUT = Symbol('searchTimeout');
const { promise: timeoutPromise, handle: timeoutHandle } =
this.armSearchTimeout(SEARCH_TIMEOUT);
try {
const results = await this.queryProductsWithCache(this.query, fetchLimit);
if (!results || results.length === 0) {
this.logger.log(`No query results found`);
return;
}
// Drop any products the user has ignored, then slice back down to the
// user-visible limit. Uses the same dual-read (identity + legacy URL) as
// getProductData/partitionForBatch so the check is consistent wherever it
// fires first.
const survivors: ProductBuilder<T>[] = [];
for (const builder of results) {
if (survivors.length >= this.limit) break;
if (this.isExcluded(builder)) {
this.logger.debug('Skipping excluded product (pre-detail)', {
url: builder.get('url'),
});
continue;
}
survivors.push(builder);
}
this.products = survivors;
const queue = new Queue(this.maxConcurrentRequests, this.minConcurrentCycle);
// Each task fetches a product's detail data and finishes it, tagged with its index so the
// yield loop can track which products are still outstanding when the budget elapses.
const pending = new Map<number, Promise<{ index: number; finished: Maybe<T> }>>();
this.products.forEach((product, index) => {
pending.set(
index,
queue.run(async () => {
// If the budget already elapsed, skip the (now-aborted) detail fetch — the timeout
// handler below emits this product's basic data directly.
if (this.controller.signal.aborted) {
return { index, finished: undefined };
}
try {
const builder = await this.getProductData(product);
const finished = builder ? await this.finishProduct(builder) : undefined;
return { index, finished };
} catch (e: unknown) {
this.logger.error('Error processing product', { error: e, product });
incrementParseError(this.supplierName);
return { index, finished: undefined };
}
}),
);
});
// As each task resolves, yield the product. Race against the optional search-time budget;
// when it elapses, emit every not-yet-yielded product with its basic (query-phase) data so
// the rows still show — the enrichment for those simply wasn't cached, so a later search
// (served from the query cache) re-fetches their detail data.
const yielded = new Set<number>();
while (pending.size > 0) {
const result = await Promise.race(
timeoutPromise ? [...pending.values(), timeoutPromise] : [...pending.values()],
);
if (result === SEARCH_TIMEOUT) {
for (let index = 0; index < this.products.length; index++) {
if (yielded.has(index)) continue;
// The builder is enriched in place, so this carries whatever detail data was set
// before the abort, falling back to the basic query-phase fields otherwise.
const finished = await this.finishProduct(this.products[index]);
if (finished) {
yield finished;
}
}
break;
}
pending.delete(result.index);
yielded.add(result.index);
if (result.finished) {
yield result.finished;
}
}
} finally {
if (timeoutHandle !== undefined) {
clearTimeout(timeoutHandle);
}
}
}
ProtectedfinishFinalizes a partial product by adding computed properties and validating the result. This method:
The ProductBuilder instance containing the partial product to finalize
Promise resolving to a complete Product object or void if validation fails
// Example with a valid partial product
const builder = new ProductBuilder<Product>(this.baseURL);
builder
.setBasicInfo("Sodium Chloride", "/products/nacl", "ChemSupplier")
.setPricing(29.99, "USD", "$")
.setQuantity(500, "g");
const finishedProduct = await this.finishProduct(builder);
if (finishedProduct) {
console.log("Finalized product:", {
title: finishedProduct.title,
price: finishedProduct.price,
quantity: finishedProduct.quantity,
uom: finishedProduct.uom,
usdPrice: finishedProduct.usdPrice,
baseQuantity: finishedProduct.baseQuantity
});
}
// Example with an invalid partial product
const invalidBuilder = new ProductBuilder<Product>(this.baseURL);
invalidBuilder.setBasicInfo("Sodium Chloride", "/products/nacl", "ChemSupplier");
// Missing required fields
const invalidProduct = await this.finishProduct(invalidBuilder);
if (!invalidProduct) {
console.log("Failed to finalize product - missing required fields");
}
protected async finishProduct(product: ProductBuilder<T>): Promise<Maybe<T>> {
if (!isMinimalProduct(product.dump())) {
this.logger.warn('Unable to finish product - Minimum data not set', { product });
return;
}
// Dev guard: every builder should carry a stamped `cacheKey` (set at parse
// time via `setCacheKey(getUniqueProductKey(item))`). A missing key means a
// supplier implemented `getUniqueProductKey` but forgot to stamp — it would
// silently disable per-product caching and precise exclusion for that
// product. Surface it loudly in dev/tests.
if (IS_DEV_BUILD && product.get('cacheKey') == null) {
this.logger.error('finishProduct| product is missing a stamped cacheKey', {
supplier: this.supplierName,
url: product.get('url'),
});
}
// Set the country and shipping scope of the supplier
// have different restrictions on different products or countries.
product.setSupplierCountry(this.country);
product.setSupplierShipping(this.shipping);
if (this.paymentMethods.length > 0) {
product.setSupplierPaymentMethods(this.paymentMethods);
}
// Marketplace storefronts. The "*only" methods point users away from the supplier's own
// (restricted) site, so a missing store URL is a misconfiguration — surface it loudly in dev.
// The plain "ebay"/"amazon" methods instead drive an informational "more products there"
// notice; the store URL is optional, so stamp it only when present and don't warn.
if (this.paymentMethods.includes('ebayonly')) {
if (IS_DEV_BUILD && !this.ebayStoreURL) {
this.logger.error("finishProduct| supplier declares 'ebayonly' but sets no ebayStoreURL", {
supplier: this.supplierName,
});
}
product.setSupplierEbayStoreURL(this.ebayStoreURL);
} else if (this.paymentMethods.includes('ebay') && this.ebayStoreURL) {
product.setSupplierEbayStoreURL(this.ebayStoreURL);
}
if (this.paymentMethods.includes('amazononly')) {
if (IS_DEV_BUILD && !this.amazonStoreURL) {
this.logger.error(
"finishProduct| supplier declares 'amazononly' but sets no amazonStoreURL",
{ supplier: this.supplierName },
);
}
product.setSupplierAmazonStoreURL(this.amazonStoreURL);
} else if (this.paymentMethods.includes('amazon') && this.amazonStoreURL) {
product.setSupplierAmazonStoreURL(this.amazonStoreURL);
}
const built = await product.build();
return built;
}
ProtectedproductThe stable per-product cache/exclusion key for a builder: the identity
stamped on it at parse time (getUniqueProductKey →
setCacheKey), hashed with the supplier name via
getProductIdentityKey. Returns undefined when no identity was
stamped (so callers can skip the identity cache/exclusion path).
The product builder
The identity cache key, or undefined when unstamped
const key = this.productIdentityKey(builder); // md5({key, supplier}) or undefined
protected productIdentityKey(product: ProductBuilder<T>): string | undefined {
const identity = product.get('cacheKey');
if (typeof identity === 'string' && identity.length > 0) {
return this.cache.getProductIdentityCacheKey(identity);
}
return undefined;
}
ProtectedisWhether a product is on the user's ignore list, matched by its identity key (productIdentityKey) — the same key the "Ignore Product" action writes.
The product builder to check
true if the product matches an ignore-list entry
if (this.isExcluded(builder)) continue; // skip ignored product
protected isExcluded(product: ProductBuilder<T>): boolean {
const identityKey = this.productIdentityKey(product);
return identityKey !== undefined && this.excludedProductKeys.has(identityKey);
}
ProtectedpartitionPartitions query-phase builders for a batch supplier (one that enriches
details up front rather than per-product in getProductData). Runs a
three-way split: ignored products are dropped from both results;
cache hits (found in the product-detail cache by their stamped
identity) are hydrated in place via setData and kept in survivors but
excluded from misses; everything else is a miss, kept in both. The
caller enriches only misses, then caches them, and returns survivors.
Skips the cache lookup entirely when skipProductDetailCache is true (pure-search suppliers), so those still drop ignored products but treat every survivor as needing no enrichment.
The query-phase builders (each already setCacheKey-stamped)
{ survivors, misses } — see above
const { survivors, misses } = await this.partitionForBatch(builders);
await this.enrichVariants(misses);
await this.cacheProductBuilders(misses);
return survivors;
protected async partitionForBatch(
products: ProductBuilder<T>[],
): Promise<{ survivors: ProductBuilder<T>[]; misses: ProductBuilder<T>[] }> {
const survivors: ProductBuilder<T>[] = [];
const misses: ProductBuilder<T>[] = [];
for (const product of products) {
if (this.isExcluded(product)) {
continue;
}
survivors.push(product);
if (this.skipProductDetailCache) {
continue;
}
const key = this.productIdentityKey(product);
const cached = key ? await this.cache.getCachedProductData(key) : undefined;
if (isCachedProductData<T>(cached)) {
product.setData(cached);
} else {
misses.push(product);
}
}
return { survivors, misses };
}
ProtectedcacheWrites each enriched builder to the product-detail cache under its stamped
identity key. No-ops for suppliers with skipProductDetailCache true,
for aborted searches, and for builders shouldCacheProductData
rejects (e.g. a fetch that hit a noCacheStatusCode). Used by batch
suppliers after enriching their misses.
The enriched builders to persist
A promise that resolves once all writes complete
await this.cacheProductBuilders(misses);
protected async cacheProductBuilders(products: ProductBuilder<T>[]): Promise<void> {
if (this.skipProductDetailCache || this.controller.signal.aborted) {
return;
}
await Promise.all(
products.map(async (product) => {
const key = this.productIdentityKey(product);
if (key && this.shouldCacheProductData(product)) {
await this.cache.cacheProductData(key, product.dump());
}
}),
);
}
ProtectedhrefTakes in either a relative or absolute URL and returns an absolute URL. This is useful for when you aren't sure if the link (retrieved from parsed text, a setting, an element, an anchor value, etc) is absolute or not. Using relative links will result in http://chrome-extension://... being added to the link.
URL object or string
Optionalparams: Maybe<RequestParams>The parameters to add to the URL.
Optionalhost: stringThe host to use for overrides (eg: needing to call a different host for an API)
absolute URL
this.href('/some/path')
// https://supplier_base_url.com/some/path
this.href('https://supplier_base_url.com/some/path', null, 'another_host.com')
// https://another_host.com/some/path
this.href('/some/path', { a: 'b', c: 'd' }, 'another_host.com')
// http://another_host.com/some/path?a=b&c=d
this.href('https://supplier_base_url.com/some/path')
// https://supplier_base_url.com/some/path
this.href(new URL('https://supplier_base_url.com/some/path'))
// https://supplier_base_url.com/some/path
this.href('/some/path', { a: 'b', c: 'd' })
// https://supplier_base_url.com/some/path?a=b&c=d
this.href('https://supplier_base_url.com/some/path', new URLSearchParams({ a: 'b', c: 'd' }))
// https://supplier_base_url.com/some/path?a=b&c=d
protected href(path: string | URL, params?: Maybe<RequestParams>, host?: string): string {
const href = new URL(path, this.baseURL);
if (host) {
href.host = host;
}
if (params && Object.keys(params).length > 0) {
href.search = this.buildSearchParams(params).toString();
}
return String(href);
}
ProtectedbuildRecursively serializes an object into URLSearchParams, encoding nested
objects with bracket notation (parent[child]=value). Fixes the case that
new URLSearchParams(obj) mishandles — it stringifies a nested object to
"[object Object]" — so a params object with nested filters serializes
correctly. Primitive values (and arrays, which serialize comma-joined like
URLSearchParams does) are appended directly.
The params object to serialize
The URLSearchParams to append into (defaults to a fresh instance)
The key prefix used while recursing into nested objects
The populated URLSearchParams
this.buildSearchParams({ q: "acid", filter: { size: "500g" } }).toString();
// "q=acid&filter%5Bsize%5D=500g" (i.e. filter[size]=500g)
protected buildSearchParams(
obj: Record<string, unknown>,
params: URLSearchParams = new URLSearchParams(),
prefix: string = '',
): URLSearchParams {
for (const [key, value] of Object.entries(obj)) {
// Bracket-nest the key as depth grows: parent[child][grandchild]...
const formKey = prefix ? `${prefix}[${key}]` : key;
if (isPopulatedObject(value)) {
this.buildSearchParams(value, params, formKey);
} else {
params.append(formKey, String(value));
}
}
return params;
}
ProtectedgetRetrieves product data with caching support. Similar to getProductData but allows for additional parameters to be included in the cache key.
The ProductBuilder instance to get data for
The function to use for fetching product data
Promise resolving to the updated ProductBuilder or void if fetch fails
const builder = new ProductBuilder<Product>(this.baseURL);
builder.setBasicInfo("Acetone", "/products/acetone", "ChemSupplier");
const updatedBuilder = await supplier.getProductDataWithCache(
builder,
async (b) => {
// Custom fetching logic
return b;
},
);
protected async getProductDataWithCache(
product: ProductBuilder<T>,
fetcher: (builder: ProductBuilder<T>) => Promise<ProductBuilder<T> | void>,
): Promise<ProductBuilder<T> | void> {
const url = product.get('url');
if (typeof url !== 'string') {
this.logger.error('[SupplierBase > getProductDataWithCache] Invalid URL in product:', {
url,
});
return undefined;
}
// Skip products the user has ignored (matched by identity key).
if (this.isExcluded(product)) {
this.logger.debug('[SupplierBase > getProductDataWithCache] Skipping excluded product', {
url,
supplierName: this.supplierName,
});
return undefined;
}
// Key by the supplier's stable identity, stamped on the builder at parse
// time. Absent only if a supplier failed to stamp one, in which case this
// product simply isn't cached.
const cacheKey = this.productIdentityKey(product);
this.logger.debug(
'[SupplierBase > getProductDataWithCache] Product detail cache key:',
cacheKey,
{
url,
},
);
try {
if (!this.skipProductDetailCache && cacheKey) {
const cachedData = await this.cache.getCachedProductData(cacheKey);
if (isCachedProductData<T>(cachedData)) {
product.setData(cachedData);
return product;
}
}
// Cache miss (or caching skipped): run setup (memoized) so any state
// subclasses rely on is ready before the fetcher reads it, then fetch.
await this.ensureSetup();
let resultBuilder: ProductBuilder<T> | void = undefined;
try {
resultBuilder = await fetcher(product);
} catch (err: unknown) {
this.logger.error(
'[SupplierBase > getProductDataWithCache] Error in product detail fetcher:',
err,
);
incrementParseError(this.supplierName);
return undefined;
}
if (resultBuilder) {
incrementProductCount(this.supplierName);
// Skip caching when the search was aborted (e.g. maxAllowableSearchTimeSec) — the enrichment
// fetch was cancelled, so the data is incomplete and a later search should retry it.
if (
cacheKey &&
!this.skipProductDetailCache &&
!this.controller.signal.aborted &&
this.shouldCacheProductData(resultBuilder)
) {
await this.cache.cacheProductData(cacheKey, resultBuilder.dump());
}
}
return resultBuilder;
} catch (outerErr: unknown) {
this.logger.error(
'[SupplierBase > getProductDataWithCache] Error in getProductDataWithCache:',
outerErr,
);
incrementParseError(this.supplierName);
return undefined;
}
}
ProtectedproductThe key under which a product's detail-fetch failures are recorded/looked up: its permalink if set, otherwise its processing URL. Subclasses that fetch supplemental data should record failures under this same key so shouldCacheProductData can match them.
The product builder
The fetch key, or undefined when neither permalink nor url is a string
this.productFetchKey(builder); // "https://www.aladdinsci.com/us_en/x.html"
protected productFetchKey(product: ProductBuilder<T>): string | undefined {
const key = product.get('permalink') ?? product.get('url');
return typeof key === 'string' ? key : undefined;
}
ProtectedrecordRecords the HTTP status of a failed product-detail fetch so shouldCacheProductData can skip caching it. Non-HTTP failures (no status) are not recorded.
The product fetch key (see productFetchKey)
OptionalhttpStatus: numberThe HTTP status of the failure, if it was an HttpError
void
this.recordFetchFailure(permalink, 429);
protected recordFetchFailure(key: string, httpStatus?: number): void {
if (typeof httpStatus === 'number') {
this.failedFetchStatuses.set(key, httpStatus);
}
}
ProtectedshouldDecides whether a freshly-fetched product's detail data should be written to the cache. Skips caching when the product's last detail fetch failed with a noCacheStatusCodes status (default 429), so a later search retries it instead of serving the incomplete cached entry. The product is still listed regardless.
The product builder about to be cached
True to cache the product data, false to skip caching it
this.shouldCacheProductData(builder); // false when the detail fetch hit a 429
protected shouldCacheProductData(product: ProductBuilder<T>): boolean {
const key = this.productFetchKey(product);
if (key === undefined) {
return true;
}
const failedStatus = this.failedFetchStatuses.get(key);
return failedStatus === undefined || !this.noCacheStatusCodes.includes(failedStatus);
}
ProtectedgroupGroups variants of a product by their title
Array of product listings from search results
Array of product listings with grouped variants
Create a generic method for this, the same method is used in Synthetika and could be of use with LoudWolf.
const results = await this.queryProducts("sodium chloride");
const grouped = this.groupVariants(results);
// grouped is an array of product listings with grouped variants
protected groupVariants<R>(data: R[]): R[] {
const variants: GroupedItem<R>[] = data
.map((item) => {
const title = this.titleSelector(item);
if (!title) {
this.logger.error('No title found in product:', { item });
return undefined;
}
const groupId = stripQuantityFromString(title.replace(/(?<=\d{1,3})\s(?=\d{3})/g, ''));
const groupIdWithoutSpaces = groupId.replace(/[\s-]/g, '');
return { ...item, groupId: groupIdWithoutSpaces };
})
.filter((item): item is GroupedItem<R> => item !== undefined);
const products = Object.groupBy(variants, (item) => item.groupId);
return Object.values(products)
.filter((product): product is GroupedItem<R>[] => product !== undefined)
.map((product) => {
const main = product.splice(0, 1)[0];
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { groupId, ...newObject } = main;
// R is unconstrained, so TS widens newObject.variants to
// R["variants"] & (R[] | undefined), which it cannot prove product
// (GroupedItem<R>[]) satisfies; assert to the destructured property type.
newObject.variants = product as typeof newObject.variants;
return newObject;
})
.filter((item): item is GroupedItem<R> => item !== undefined);
}
ProtectedbackgroundRuns an HTTP request from the extension's background service worker instead of this
(page) context, sidestepping the CORS restrictions that apply to extension pages.
The target host must be granted in the manifest host_permissions. Returns a real
Response (text/JSON bodies only). Independent of fetch — it does not
share the request counter, hard limit, or WAF retry logic.
The absolute URL to request.
Optionalinit: BackgroundFetchInitOptional serializable request options (method, headers, body, etc.).
A Response reconstructed from the worker's reply.
// Inside a supplier method, e.g. scraping an asset blocked by page CORS:
const homepage = await this.backgroundFetch("https://chemsavers.com/");
const html = await homepage.text();
const search = await this.backgroundFetch("https://api.example.com/search", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ q: "acid" }),
});
const results = search.ok ? await search.json() : undefined;
protected async backgroundFetch(url: string, init?: BackgroundFetchInit): Promise<Response> {
this.logger.debug(`Background fetching: ${url}`);
return backgroundFetch(url, init);
}
ProtectedfetchInternal fetch method with request counting and decorator. Tracks request count and enforces hard limits on HTTP requests.
Arguments to pass to fetchDecorator (usually a Request or URL and options)
The response from the fetchDecorator
Error if request count exceeds hard limit
// Example usage inside a subclass:
const response = await this.fetch(new Request('https://example.com'));
if (response.ok) {
const data = await response.json();
console.log(data);
}
// With custom request options
const response = await this.fetch(
new Request('https://example.com', {
headers: { 'Accept': 'application/json' }
})
);
protected async fetch(
...args: Parameters<typeof fetchDecorator>
): Promise<FetchDecoratorResponse> {
const [input] = args;
this.logger.debug(`Fetching: ${input}`);
// One initial attempt plus up to `challengeRetryLimit` retries. A 403 from
// a WAF cookie handshake plants a cookie on the first hit (stored because
// credentials:"include"); the retry sends it back and usually passes.
const maxAttempts = 1 + Math.max(0, this.challengeRetryLimit);
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
// Each attempt is a real network request, so it counts toward the hard
// limit. For non-retrying suppliers (maxAttempts === 1) this is
// identical to the previous single increment.
this.requestCount++;
if (this.requestCount > this.httpRequestHardLimit) {
this.logger.warn('Request count exceeded hard limit', { requestCount: this.requestCount });
incrementFailure(this.supplierName);
throw new Error('Request count exceeded hard limit');
}
try {
const response = await fetchDecorator(...args);
this.logger.debug(`Response Status: ${response.status}`);
this.logger.debug('response hash:', response.requestHash);
if (typeof response.data === 'string' && response.data?.length === 0) {
throw new EmptyResponseError(`Invalid response: ${response.data}`);
}
incrementSuccess(this.supplierName);
return response;
} catch (error: unknown) {
if (this.shouldRetryChallenge(error) && attempt < maxAttempts) {
this.logger.warn('Retrying after 403 (WAF cookie handshake)', {
attempt,
maxAttempts,
input,
});
await sleep(this.challengeRetryDelayMs);
continue;
}
incrementFailure(this.supplierName);
throw error;
}
}
// Unreachable: the loop always returns on success or throws on the final
// failed attempt. Present only to satisfy the return-type checker.
throw new Error('fetch: exhausted retries without resolving');
}
ProtectedgetDerives the stable unique key for a LeroChem search-result card: the
data-id-product attribute (the PrestaShop product id, which survives the
query→detail transition). When the card carries no id, the same numeric id
is recovered from the product URL's last segment, falling back to the full
href only if that fails — so the key is always a non-empty string.
The raw LeroChem article.product-miniature Element
The card's product id, or its product href when no id is present
this.getUniqueProductKey(element); // "208"
protected getUniqueProductKey(data: Element): string {
const productId = data.getAttribute('data-id-product');
if (productId) {
return productId;
}
const href = data.querySelector('.product-title a')?.getAttribute('href');
return this.productIdFromUrl(href) ?? this.href(String(href ?? ''));
}
PrivateproductExtracts the numeric PrestaShop product id from a LeroChem product URL — the
leading digits of the last path segment (e.g.
".../pagrindinis/126-abs-acid-labsa-96-l.html" -> "126"). Used as a backup
source for the product id when the data-id-product attribute is absent.
The product URL (absolute or relative), or null/undefined
The numeric product id, or undefined when the URL has no id segment
this.productIdFromUrl("https://lerochem.eu/en/pagrindinis/126-abs-acid.html"); // "126"
private productIdFromUrl(url: string | null | undefined): Maybe<string> {
return url?.match(/\/(\d+)-[^/]*\.html/)?.[1] ?? undefined;
}
ProtectedtitleExtracts the product title from a LeroChem search-result card. Used by the base fuzzy filter to score each card against the query.
The article.product-miniature Element
The product title, or undefined when the card has no title link
this.titleSelector(card); // "SULFURIC ACID (tech grade), 98%, L"
protected titleSelector(data: Element): Maybe<string> {
if (!data) {
this.logger.error('No data for product', { data });
return undefined;
}
return data.querySelector('.product-title a')?.textContent?.trim() ?? undefined;
}
ProtectedqueryQueries LeroChem products for a search term. Fetches the first PrestaShop search-results page, reads how many pages the pagination reports, then fetches further pages only while more fuzzy matches are still needed (never beyond the reported total or the httpRequestHardLimit request cap). The collected cards are fuzzy-filtered and turned into product builders for the top matches.
The search term to query products for
The maximum number of results to return
Promise resolving to product builders, or void when the search fails
const results = await supplier.queryProducts("acetone", 5);
console.log(results?.length); // up to 5
protected async queryProducts(
query: string,
limit: number = this.limit,
): Promise<ProductBuilder<Product>[] | void> {
const firstPage = await this.httpGetHtml({
path: '/en/search',
params: { controller: 'search', s: query },
});
if (!firstPage) {
this.logger.error('No search response', { query });
return;
}
const cards: Element[] = this.parseSearchCards(firstPage);
const totalPages = Math.min(this.parseTotalPages(firstPage), this.httpRequestHardLimit);
this.logger.info('Search pagination', { query, totalPages });
for (let page = 2; page <= totalPages && this.fuzzyFilterAst(cards).length < limit; page++) {
const pageResponse = await this.httpGetHtml({
path: '/en/search',
params: { controller: 'search', s: query, page },
});
if (!pageResponse) {
this.logger.warn('No search response for page', { query, page });
break;
}
cards.push(...this.parseSearchCards(pageResponse));
}
if (cards.length === 0) {
this.logger.log('No products found', { query });
return;
}
const fuzzResults = this.fuzzyFilterAst(cards);
this.logger.info('fuzzResults:', { query, count: fuzzResults.length });
return this.initProductBuilders(fuzzResults.slice(0, limit));
}
ProtectedparseDetermines how many result pages a LeroChem search returned, from the
nav.pagination block. Prefers the exact "Showing X-Y of Z item(s)" count
(which is not truncated when the page links collapse behind an ellipsis),
and falls back to the highest numbered page link. Returns 1 when there is no
pagination (a single page of results).
The raw search-results HTML
The total number of result pages (at least 1)
this.parseTotalPages(html); // 9 (for "Showing 1-24 of 201 item(s)")
protected parseTotalPages(html: string): number {
const dom = createDOM(html);
const summary = dom.querySelector('nav.pagination')?.textContent ?? '';
const match = summary.match(/Showing\s+(\d+)\s*[-–]\s*(\d+)\s+of\s+(\d+)/i);
if (match) {
const perPage = Number(match[2]) - Number(match[1]) + 1;
const total = Number(match[3]);
if (perPage > 0 && total > 0) {
return Math.max(1, Math.ceil(total / perPage));
}
}
const pageNumbers = Array.from(
dom.querySelectorAll('nav.pagination a[aria-label^="Page"]'),
).map((anchor) => Number(anchor.getAttribute('aria-label')?.replace(/\D+/g, '')));
const highest = pageNumbers.filter((n) => Number.isFinite(n) && n > 0);
return highest.length > 0 ? Math.max(...highest) : 1;
}
ProtectedparseParses a PrestaShop search-results HTML page into its product cards.
The raw search-results HTML
The article.product-miniature Elements on the page
const cards = this.parseSearchCards(html);
console.log(cards.length); // up to 24
protected parseSearchCards(html: string): Element[] {
const dom = createDOM(html);
if (!dom) {
throw new Error('No data found when loading search HTML');
}
return Array.from(dom.querySelectorAll('article.product-miniature'));
}
ProtectedinitInitializes product builders from LeroChem search-result cards. Each card yields the product id, title, URL, thumbnail, and the "From" price shown in the listing (the authoritative price is refreshed from the product page in getProductData).
The article.product-miniature Elements to convert
Product builders seeded with the listing data
const builders = this.initProductBuilders(cards);
const product = await builders[0].build();
console.log(product.title, product.price);
protected initProductBuilders(elements: Element[]): ProductBuilder<Product>[] {
return mapDefined(elements, (element: Element) => {
const anchor = element.querySelector('.product-title a');
const title = anchor?.textContent?.trim();
const url = anchor?.getAttribute('href');
if (!title || !url) {
this.logger.error('Card missing title or URL', { element });
return;
}
const builder = new ProductBuilder<Product>(this.baseURL);
const image =
element.querySelector('.product-image img')?.getAttribute('data-full-size-image-url') ??
element.querySelector('.product-image img')?.getAttribute('src');
const currency = this.cardCurrency(element);
builder
.setBasicInfo(title, url, this.supplierName)
.setID(element.getAttribute('data-id-product') ?? this.productIdFromUrl(url))
.setCacheKey(this.getUniqueProductKey(element))
.setImage(image)
.setPrice(this.cardPrice(element))
.setCurrencyCode(currency)
.setCurrencySymbol(CURRENCY_SYMBOL_MAP[currency]);
return builder;
});
}
PrivatecardReads the numeric "From" price from a search-result card. PrestaShop renders
the price as <span content="EUR"></span><span content="5">5.00 €</span>;
the numeric content value is used to avoid parsing the currency-suffixed
display text.
The article.product-miniature Element
The numeric price string (e.g. "5"), or undefined when absent
this.cardPrice(card); // "5"
private cardPrice(element: Element): Maybe<string> {
const contents = Array.from(
element.querySelectorAll('.product-price-and-shipping .price span[content]'),
);
return (
contents
.map((span) => span.getAttribute('content'))
.find((content) => content != null && /^\d+(\.\d+)?$/.test(content)) ?? undefined
);
}
PrivatecardReads the currency code from a search-result card, defaulting to EUR (the shop's only currency) when the microdata span is absent.
The article.product-miniature Element
The ISO currency code, e.g. "EUR"
this.cardCurrency(card); // "EUR"
private cardCurrency(element: Element): string {
const span = element.querySelector('.product-price-and-shipping .price span[content="EUR"]');
return span?.getAttribute('content') ?? 'EUR';
}
ProtectedgetFetches and parses a LeroChem product page into a complete product. Pulls
data from the schema.org Product ld+json, the #product-details
data-product dataset, the #description spec table, and the per-size
refresh AJAX endpoint.
The product builder seeded from the search listing
Promise resolving to the enriched builder, or void on failure
const full = await supplier.getProductData(builder);
console.log(full?.dump().cas, full?.dump().quantity);
protected async getProductData(
product: ProductBuilder<Product>,
): Promise<ProductBuilder<Product> | void> {
return this.getProductDataWithCache(product, async (builder) => {
if (typeof builder === 'undefined') {
this.logger.error('No products to get data for', { builder });
return;
}
const productResponse = await this.httpGetHtml({ path: builder.get('url') });
if (!productResponse) {
this.logger.warn('No product response', { builder });
return;
}
const dom = createDOM(productResponse);
// schema.org Product ld+json: sku, image, price, currency, availability.
const schema = SchemaOrgData.fromDocument(dom);
const product = schema.first('Product');
const offer = schema.all('Offer')[0];
if (product) {
builder.setSku(product.sku);
// The ld+json sku/mpn equal the numeric product id — a backup for the id set from the card.
builder.setID(product.sku);
builder.setImage(Array.isArray(product.image) ? product.image[0] : product.image);
}
if (offer) {
if (offer.price != null) {
builder.setPrice(offer.price);
}
const currencyCode = typeof offer.priceCurrency === 'string' ? offer.priceCurrency : 'EUR';
builder.setCurrencyCode(currencyCode);
builder.setCurrencySymbol(CURRENCY_SYMBOL_MAP[currencyCode]);
// SchemaOrgData already strips the schema.org enum prefix (".../PreOrder" -> "PreOrder").
if (typeof offer.availability === 'string') {
builder.setAvailability(offer.availability);
}
}
// #product-details data-product: authoritative price, default size, description.
const dataProduct = this.parseDataProduct(dom);
if (dataProduct) {
// data-product `id`/`id_product` is another backup for the numeric product id.
builder.setID(dataProduct.id_product ?? dataProduct.id);
if (typeof dataProduct.price_amount === 'number') {
builder.setPrice(dataProduct.price_amount);
}
builder.setDescription(
this.descriptionToText(dataProduct.description) ?? dataProduct.meta_description,
);
builder.setShortDescription(
dataProduct.meta_description ??
this.shortDescriptionToText(dataProduct.description_short),
);
const defaultQty = this.defaultSizeQuantity(dataProduct);
if (defaultQty) {
builder.setQuantity(defaultQty);
}
// CAS most reliably lives in the meta description ("… CAS <number> …").
builder.setCAS(
firstMap(
(text) => findCAS(text),
[dataProduct.meta_description, dataProduct.name, dataProduct.description].filter(
(value): value is string => typeof value === 'string',
),
),
);
}
// The product name carries the purity ("… 98%, L" -> "98%"); findPurity
// (via setPurity) ignores the pack unit and pulls the percentage.
builder.setPurity(builder.get('title'));
builder.setGrade(this.gradeFromName(builder.get('title')));
// The #description spec table carries the most reliable CAS, formula, and molar mass.
this.applyDataTable(builder, dom);
// The short-description block links the COA and the SDS (labelled "Declaration").
this.applyDocumentLinks(builder, dom);
const variants = await this.parseVariants(builder, dom);
if (variants.length > 0) {
builder.setVariants(variants);
}
return builder;
});
}
PrivategradeExtracts a chemical grade from a LeroChem product name. Many names tag a
grade in the title (e.g. "(tech grade)", "food grade", "E300 food grade"),
which findPurity's generic grade fallback does not recognise, so a curated
set of grade words is matched here.
The product name/title (or any value)
The normalised grade phrase (e.g. "tech grade"), or undefined
this.gradeFromName("SULFURIC ACID (tech grade), 98%, L"); // "tech grade"
private gradeFromName(name: unknown): Maybe<string> {
if (typeof name !== 'string') {
return undefined;
}
const match = name.match(
/\b(tech(?:nical)?|food|feed|reagent|analytical|industrial|cosmetic|pharmaceutical|laboratory|lab)\s+grade\b/i,
);
return match ? match[0].toLowerCase() : undefined;
}
PrivateparseReads and parses the #product-details data-product dataset from a
product page or a refresh response fragment.
The parsed Document containing a #product-details element
The data-product object, or undefined when absent/invalid
this.parseDataProduct(dom)?.price_amount; // 5.2
private parseDataProduct(dom: Document): Maybe<LeroChemDataProduct> {
const raw = dom.querySelector('#product-details')?.getAttribute('data-product');
const parsed = tryParseJson(raw);
return isLeroChemDataProduct(parsed) ? parsed : undefined;
}
PrivatedefaultParses the default (selected) combination's size into a quantity object,
e.g. the "1 L" attribute becomes { quantity: 1, uom: "l" }.
The parsed data-product dataset
The parsed quantity, or undefined when no size attribute is present
this.defaultSizeQuantity(dataProduct); // { quantity: 1, uom: "l" }
private defaultSizeQuantity(dataProduct: LeroChemDataProduct): Maybe<QuantityObject> {
const label = Object.values(dataProduct.attributes ?? {})[0]?.name;
return label ? (parseQuantity(label) ?? undefined) : undefined;
}
PrivatedescriptionConverts the data-product description HTML into plain text: strips every
<table> (LeroChem embeds a spec table that duplicates the detail fields, and
a hazard table) and runs the remainder through htmlToAscii.
The raw description HTML (or any value)
The plain-text description, or undefined when empty/absent
this.descriptionToText("<table>…</table><p>Some acid.</p>"); // "Some acid."
private descriptionToText(html: Maybe<string>): Maybe<string> {
if (typeof html !== 'string' || html.length === 0) {
return undefined;
}
const dom = createDOM(html);
for (const table of Array.from(dom.querySelectorAll('table'))) {
table.remove();
}
const text = htmlToAscii(dom.body?.innerHTML ?? html);
return text.length > 0 ? text : undefined;
}
PrivateshortConverts the data-product description_short HTML into plain text: removes
the COA/Declaration document links (they are surfaced separately as
coaUrl/sdsUrl) and runs the remainder through htmlToAscii.
The raw short-description HTML (or any value)
The plain-text short description, or undefined when empty/absent
this.shortDescriptionToText(
'<p><a href="/COA.pdf">CERTIFICATE OF ANALYSIS</a></p><p>ABS acid.</p>',
); // "ABS acid."
private shortDescriptionToText(html: Maybe<string>): Maybe<string> {
if (typeof html !== 'string' || html.length === 0) {
return undefined;
}
const dom = createDOM(html);
for (const anchor of Array.from(dom.querySelectorAll('a[href]'))) {
const text = anchor.textContent ?? '';
const href = anchor.getAttribute('href') ?? '';
if (
/certificate of analysis|declaration/i.test(text) ||
/\/COA[\s%/]|declaration/i.test(href)
) {
(anchor.closest('p') ?? anchor).remove();
}
}
const text = htmlToAscii(dom.body?.innerHTML ?? html);
return text.length > 0 ? text : undefined;
}
PrivateapplyExtracts the datasheet links from the #product-description-short block:
the Certificate of Analysis (COA) and the SDS (which LeroChem labels
"Declaration"). Links are matched by anchor text or href, so the id suffix
(product-description-short-<id>) does not need to be known.
The product builder to enrich
The parsed product page Document
Nothing; mutates the builder in place
this.applyDocumentLinks(builder, dom);
// builder.dump().coaUrl -> ".../COA Sulfuric acid.pdf"
// builder.dump().sdsUrl -> ".../DECLARATION Sulfuric acid 98.pdf"
private applyDocumentLinks(builder: ProductBuilder<Product>, dom: Document): void {
const anchors = Array.from(dom.querySelectorAll('[id^="product-description-short"] a[href]'));
for (const anchor of anchors) {
const href = anchor.getAttribute('href');
if (!href) {
continue;
}
const text = anchor.textContent?.trim() ?? '';
if (/certificate of analysis/i.test(text) || /\/COA[\s%/]/i.test(href)) {
builder.setCoaUrl(href);
} else if (/declaration/i.test(text) || /declaration/i.test(href)) {
builder.setSDSUrl(href);
}
}
}
PrivateapplyReads the #description spec table — a two-column label/value grid — and
applies the structured fields it carries: the CAS number (more reliable than
scraping the description), the molecular formula, and the molar mass. Labels
are English: "CAS", "Formula", and "Molar mass" (e.g. "326,49", using a
decimal comma).
The product builder to enrich
The parsed product page Document
Nothing; mutates the builder in place
this.applyDataTable(builder, dom);
// builder.dump().cas -> "27176-87-0", builder.dump().moleweight -> 326.49
private applyDataTable(builder: ProductBuilder<Product>, dom: Document): void {
const rows = Array.from(dom.querySelectorAll('#description table tbody tr'));
for (const row of rows) {
const cells = row.querySelectorAll('td');
const label = cells[0]?.textContent?.replace(/\s+/g, ' ').replace(/:\s*$/, '').trim();
const value = cells[1]?.textContent?.replace(/\s+/g, ' ').trim();
if (!label || !value) {
continue;
}
if (/^CAS$/i.test(label)) {
const cas = findCAS(value);
if (cas) {
builder.setCAS(cas);
}
} else if (/^Formula$/i.test(label)) {
// The cell tags atom counts as <sub>, which render across lines; collapse
// that inter-tag whitespace so findFormulaInHtml matches the whole formula
// and converts the tags to Unicode subscripts. Fall back to subscripting
// the flattened text if no tagged formula is found.
const cellHtml = cells[1]?.innerHTML?.replace(/\s+/g, '');
const formula =
(cellHtml ? findFormulaInHtml(cellHtml) : undefined) ??
formatFormula(value.replace(/\s+/g, ''));
builder.setData({ formula });
} else if (/^IUPAC$/i.test(label)) {
builder.setIupacName(value);
} else if (/^Molar mass$/i.test(label)) {
// e.g. "98,079 g/mol" or "326,49" — strip the unit; parseLocalizedNumber
// resolves the European decimal comma.
builder.setMoleweight(parseLocalizedNumber(value.replace(/[^\d.,]/g, '')));
}
}
}
PrivateparseBuilds the size/price variants for a LeroChem product. The default size is
priced from the product page; every other size is priced with a PrestaShop
refresh AJAX call (controller=product&action=refresh). Currency is left
unset so each variant inherits the parent product's currency at build time.
When the page lacks a session token or product id, only the default size is
returned.
The product builder (source of the default price)
The parsed product page Document
The parsed variants in the page's (ascending-size) order
const variants = await this.parseVariants(builder, dom);
// [{ id: "934", title: "1 L", quantity: 1, uom: "l", price: 5 }, ...]
private async parseVariants(
builder: ProductBuilder<Product>,
dom: Document,
): Promise<Partial<Variant>[]> {
const radios = Array.from(dom.querySelectorAll('.product-variants input.input-radio'));
if (radios.length === 0) {
return [];
}
const token = dom
.querySelector('#add-to-cart-or-refresh input[name="token"]')
?.getAttribute('value');
const idProduct =
dom.querySelector('#product_page_product_id')?.getAttribute('value') ??
dom.querySelector('#add-to-cart-or-refresh input[name="id_product"]')?.getAttribute('value');
const defaultPrice = builder.get('price');
const variants = await Promise.all(
radios.map(async (radio) => {
const groupId = radio.getAttribute('name')?.match(/group\[(\d+)\]/)?.[1];
const valueId = radio.getAttribute('value');
const label = radio.parentElement?.querySelector('.radio-label')?.textContent?.trim();
if (!groupId || !valueId) {
return undefined;
}
const qty = label ? parseQuantity(label) : undefined;
const base: Partial<Variant> = {
id: valueId,
title: label ?? undefined,
quantity: qty?.quantity,
uom: qty?.uom,
};
// The checked radio is the default size, already priced from the page.
if (radio.hasAttribute('checked')) {
return { ...base, price: typeof defaultPrice === 'number' ? defaultPrice : undefined };
}
if (!token || !idProduct) {
return base;
}
const price = await this.fetchVariantPrice(idProduct, token, groupId, valueId);
return { ...base, price };
}),
);
return variants.filter((variant): variant is Partial<Variant> => variant !== undefined);
}
PrivatefetchPrices a single non-default size via the PrestaShop product refresh AJAX
endpoint, reading the combination's price_amount from the returned
product_details fragment.
The PrestaShop product id
The per-session product form token
The attribute group id (e.g. "5" for Size/capacity)
The attribute value id for the size to price
The variant price, or undefined when the refresh fails
await this.fetchVariantPrice("208", "abc123", "5", "27"); // 22
private async fetchVariantPrice(
idProduct: string,
token: string,
groupId: string,
valueId: string,
): Promise<Maybe<number>> {
const response = await this.httpPostJson({
path: '/en/index.php',
params: {
controller: 'product',
token,
id_product: idProduct,
id_customization: '0',
group: { [groupId]: valueId },
qty: '1',
},
body: 'ajax=1&action=refresh&quantity_wanted=1',
headers: {
'Content-Type': 'application/x-www-form-urlencoded; charset=UTF-8',
'X-Requested-With': 'XMLHttpRequest',
},
});
if (!isLeroChemVariantRefresh(response)) {
this.logger.warn('Invalid variant refresh response', { idProduct, groupId, valueId });
return undefined;
}
const dataProduct = this.parseDataProduct(createDOM(response.product_details ?? ''));
return typeof dataProduct?.price_amount === 'number' ? dataProduct.price_amount : undefined;
}
Color used to visually tag this supplier's log output (and available for
charts/UI). Defaults to a stable palette color derived from the class name
via getSupplierColor, so no supplier has to set one. Override by
assigning a hex string in a subclass constructor (also call
this.logger.setColor(this.color) there to recolor the already-built logger).
Protected ReadonlyminThe minimum match percentage for a product to be considered a match.
Protected OptionalfuzzRuntime override resolved from userSettings.fuzzScorerOverride. When
set, fuzzyFilter uses this instead of this.fuzzScorer. Undefined
(the default) means "use whatever the supplier class picked". Mutated
by setFuzzScorerOverride so it can't be readonly.
Protected OptionalparsedParsed advanced-search query, set per-instance by SupplierFactory from the
user's input. When absent (e.g. a directly-constructed test supplier),
getAst lazily parses this.query instead.
Protected OptionalresolvedSMILES/structure query terms resolved to their chemical identifiers, keyed by
the raw search term. Resolved once per search by SupplierFactory (network-
bound, via NCI Cactus/PubChem) and shared with every supplier so none of them
re-resolve. Undefined when the query has no structure terms. Only suppliers
that filter by structure (currently Ambeed) read this; others ignore it.
Protected Static ReadonlysupportsWhether this supplier can search its own site by a CAS number / molecular formula / SMILES directly. Default false: the supplier matches product names, not identifiers, so when the query is one of these identifier types (see detectTermType) the base swaps it for the chemical name the factory resolved (see effectiveQuery). Set the relevant flag true on a supplier whose search natively accepts that identifier (e.g. Ambeed), so it receives the raw identifier unchanged.
Protected Static ReadonlysupportsSee supportsCAS.
Protected Static ReadonlysupportsSee supportsCAS.
ProtectedfuzzyRuntime flag resolved from userSettings.fuzzyFilteringDisabled. When true,
fuzzyFilterAst skips fuzzball scoring: plain queries return raw supplier
results and advanced queries are filtered only by the boolean predicate via
case-insensitive substring matching.
Protected ReadonlyfuzzyFuzzy strategy. When true (the default), fuzzyFilter/fuzzyFilterAst
rank candidates by fuzz score and keep them all (in score order) instead of dropping
anything below minMatchPercentage; the base search pipeline then caps the list
to limit, so the highest-scoring matches survive. This avoids dropping clear
matches whose ratio-style score falls under the cutoff purely because the title dwarfs
the query. Set to false on a supplier to restore the hard-cutoff behavior.
Protected ReadonlymaxMaximum number of backend search requests the keyword-only fallback issues.
Protected ReadonlysupportsWhether queryProducts handles an advanced (boolean) query natively in a
single request — true for suppliers that translate the AST into a server-side
query (Wix, Shopify, Chemsavers/Typesense, LiMac/FreeFind). When false (the
default), queryProductsWithCache drives the keyword-only fallback:
one backend search per positive OR-group, unioned and deduped, with the full
boolean predicate enforced client-side by fuzzyFilterAst.
Protected ReadonlyskipOpt-out flag for the per-product detail cache. Left false (the default),
every supplier caches its per-product detail data — the safe default, since
forgetting to set this just yields harmless redundant caching, never a
silent cache regression.
Set true only on a supplier that resolves every field in the initial
search (a passthrough getProductData with no per-product fetch): for those
the per-product cache saves nothing, so
getProductData/getProductDataWithCache/partitionForBatch
skip the product-detail cache read+write. The query cache still serves
repeat searches, and getUniqueProductKey is still used for
exclusions. Mark the concrete pure-search supplier (not a shared base
class), so a base's fetching subclass keeps caching by default.
Optional ReadonlyebayThe supplier's eBay storefront. Subclasses that list "ebayonly" in
SupplierBase.paymentMethods must override this; see
src/suppliers/__tests__/storeOnlyPaymentMethods.test.ts, which enforces the pairing that
TypeScript can't express.
Optional ReadonlyamazonThe supplier's Amazon storefront. Required alongside "amazononly", exactly as
ebayStoreURL is for "ebayonly".
Protected Static Optional ReadonlyshipsThe countries to which the supplier ships.
public readonly shipsTo: CountryCode[] = ["US", "CN", "NL"];
protected static readonly shipsTo?: CountryCode[];
Protected Static Optional ReadonlyapiOptional external API hostname used by some suppliers (e.g., Typesense,
Searchanise). When set, automatically included in requiredHosts for
permission checks.
ProtectedqueryString to query for (product name, CAS, etc.). The search term that will be used to find products. Set during construction and used throughout the supplier's lifecycle.
ProtectedbaseThe base search parameters that are always included in search requests. These parameters are merged with any additional search parameters when making requests to the supplier's API.
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
this.baseSearchParams = {
format: "json",
version: "2.0"
};
}
}
protected baseSearchParams: Record<string, string | number> = {};
ProtectedcontrollerThe AbortController instance used to manage and cancel ongoing requests. This allows for cancellation of in-flight requests when needed, such as when a new search is started or the supplier is disposed.
const controller = new AbortController();
const supplier = new MySupplier("acetone", 5, controller);
// Later, to cancel all pending requests:
controller.abort();
protected controller: AbortController;
ProtectedlimitThe maximum number of results to return for a search query. This is not a limit on HTTP requests, but rather the number of products that will be returned to the caller.
const supplier = new MySupplier("acetone", 5); // Limit to 5 results
for await (const product of supplier) {
// Will yield at most 5 products
}
protected limit: number;
ProtectedproductsThe products that are currently being built by the supplier. This array holds ProductBuilder instances that are in the process of being transformed into complete Product objects.
await supplier.queryProducts("acetone");
console.log(`Building ${supplier.products.length} products`);
for (const builder of supplier.products) {
const product = await builder.build();
console.log("Built product:", product.title);
}
protected products: ProductBuilder<T>[] = [];
ProtectedrequestCounter for HTTP requests made during the current query execution. This is used to track the number of requests and ensure we don't exceed the httpRequestHardLimit.
0
await supplier.queryProducts("acetone");
console.log(`Made ${supplier.requestCount} requests`);
if (supplier.requestCount >= supplier.httpRequestHardLimit) {
console.log("Reached request limit");
}
protected requestCount: number = 0;
ProtectedminMinimum number of milliseconds between two consecutive tasks
protected minConcurrentCycle: number = 100;
ProtectedmaxMaximum wall-clock time (in seconds) a single supplier's execute() search may run.
Once exceeded, any in-flight and pending product-detail requests are aborted and the search
stops yielding new products — only those already collected are returned. Measured from the
start of execute(), so it also bounds a slow initial query. Set to 0 (the default) to
disable the limit. Override per supplier for sources that are slow or rate-limit-prone.
0 (disabled)
protected maxAllowableSearchTimeSec: number = 0;
ProtectedheadersHTTP headers used as a basis for all requests to the supplier. These headers are merged with any request-specific headers when making HTTP requests.
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
this.headers = {
"Accept": "application/json",
"User-Agent": "ChemPal/1.0"
};
}
}
protected headers: HeadersInit = {};
Protected ReadonlyrequiredCookies that must be written into the browser jar before any request runs
— e.g. a currency or session-preference cookie the backend reads. Seeded
once per instance by ensureSetup (before setup) via chrome.cookies,
since the Cookie request header is on the fetch-forbidden list and can't
be set through this.headers. Each entry's url defaults to baseURL.
Subclasses override this instead of hand-rolling a setup that calls
chrome.cookies.set directly.
[]
class MySupplier extends SupplierBase<Partial<Product>, Product> {
protected readonly requiredCookies: SupplierCookieSeed[] = [
{ name: "currency", value: "2" },
];
}
protected readonly requiredCookies: SupplierCookieSeed[] = [];
Protected ReadonlychallengeNumber of times fetch retries a request that comes back 403. Some
suppliers sit behind a WAF that 403s the first hit while planting a
session cookie (a "cookie handshake"); because every request now sets
credentials: "include", that cookie lands in the jar and the retry
carries it back, usually passing. We can't gate on the Set-Cookie
header (it's fetch-forbidden and invisible to JS), so this per-supplier
flag is the gate — 0 (the default) means never retry. Only enable it
for suppliers known to do this handshake.
0
protected readonly challengeRetryLimit: number = 0;
Protected ReadonlychallengeDelay in milliseconds between 403 challenge retries. Gives the WAF a
brief beat before re-requesting with the freshly-planted cookie.
300
protected readonly challengeRetryDelayMs: number = 300;
ProtectedloggerLogger for the supplier. Initialized in the constructor with the name of the inheriting class.
ProtectedproductDefault values for products. These will get overridden if they're found in the product data.
ProtectedcacheCache instance for this supplier.
Initialized after construction by initCache() (called from
SupplierFactory once supplierName is set). The ! assertion is safe
here because every code path that reads this.cache
(queryProductsWithCache, getProductData, getProductDataWithCache)
runs only after execute() is called on a factory-built instance, and the
factory always calls initCache() before handing the instance out.
ProtectednoHTTP status codes that, when hit while fetching a product's detail data, prevent that
product from being cached (see shouldCacheProductData). Mirrors
userSettings.noCacheStatusCodes; set by initCache. Defaults to [429].
Protected ReadonlyfailedMaps a product's fetch key (permalink, falling back to its processing URL) to the HTTP status of its last failed detail fetch. Populated by subclasses via recordFetchFailure and consulted by shouldCacheProductData. Per-search, since the factory builds a fresh supplier instance for each search.
ProtectedexcludedProduct-data cache keys the user has explicitly excluded via the "Ignore
Product" context menu action. Loaded once per execute() from
storage.local so membership checks are synchronous on the hot path
(see getProductData). Newly-ignored products take effect on the next
search, which matches the stated feature requirement.
Static ReadonlysupplierThe name of the supplier (used for display name, lists, etc).
Static ReadonlybaseThe base URL for the supplier.
Static ReadonlyshippingThe shipping scope of the supplier. Used to determine the shipping scope of the supplier.
Static ReadonlycountryThe country code of the supplier. Used to determine the currency and other country-specific information.
Static ReadonlypaymentThe payment methods accepted by the supplier. Used to determine the payment methods accepted by the supplier.
ProtectedqueryIf the products first require a query of a search page that gets iterated over, those results are stored here. Acts as a cache for the initial search results before they are processed into full product objects.
ProtectedhttpMaximum number of HTTP requests allowed per search query. This is a hard limit to prevent excessive requests to the supplier's API. If this limit is reached, the supplier will stop making new requests.
50
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
this.httpRequestHardLimit = 100; // Allow more requests
}
}
protected httpRequestHardLimit: number = 50;
ProtectedhttpProtectedmaxNumber of requests to process in parallel when fetching product details. This controls the batch size for concurrent requests to avoid overwhelming the supplier's API and the user's bandwidth.
10
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
// Process 5 requests at a time
this.maxConcurrentRequests = 5;
}
}
protected maxConcurrentRequests: number = 5;
Protected ReadonlyfuzzFuzz scorer used by fuzzyFilter to score each candidate's title against
the query. Any function from fuzzball with the
(str1, str2, opts?) => number shape works. Subclasses override this when
a supplier's title format needs a different scorer (e.g. a catalog that
pads titles with boilerplate might prefer partial_ratio). Defaults to
WRatio.
Overridable at runtime from userSettings.fuzzScorerOverride — see
setFuzzScorerOverride and fuzzyFilter below. The user's Advanced
settings selection wins over this subclass default when set.
Supplier implementation for LeroChem, a Lithuania-based chemical supplier running on PrestaShop (lerochem.eu). Like the Warchem supplier it is HTML-based, but the pages embed rich JSON: search results are parsed from the
article.product-miniaturecards, and each product page carries a schema.orgProductld+jsonblock plus a#product-detailsdata-productdataset. Per-size prices are fetched from the PrestaShop productrefreshAJAX endpoint.Type Param: S
The supplier-specific product type (Partial)
Type Param: T
The common Product type that all suppliers map to
Example
Source