Instance view of the static supplierName (keeps this.supplierName working).
Instance view of the static baseURL (keeps this.baseURL working).
PrivatesupplierThis supplier's own class, typed for static metadata access. this.constructor
is otherwise typed Function (no custom statics), so the instance getters below
read the concrete class's static fields through this narrowed view.
ProtectedsupportsInstance view of the static supportsCAS flag (keeps this.supportsCAS working).
ProtectedsupportsInstance view of the static supportsFormula flag.
ProtectedsupportsInstance view of the static supportsSMILES flag.
Instance view of the static shipping scope (keeps this.shipping working).
Instance view of the static country (keeps this.country working).
Instance view of the static paymentMethods (keeps this.paymentMethods working).
ProtectedshipsInstance view of the static shipsTo allowlist (keeps this.shipsTo working).
ProtectedapiInstance view of the static apiURL (keeps this.apiURL working).
StaticrequiredAll host origin patterns required for this supplier to function.
Automatically includes baseURL and, if defined, apiURL. Used by the
factory to check chrome permissions before querying.
Instance view of the static requiredHosts.
ProtectedeffectiveThe single query term this supplier should search by. Normally the raw query, but when the query is a CAS/formula/SMILES identifier the supplier can't search for itself (per supportsCAS/supportsFormula/supportsSMILES) and the factory resolved it, it's the broadest resolved name — the one likely to yield the most store results — so a name-only supplier can answer "Na6O18P6" by searching "Sodium hexametaphosphate". Matching then scores results against every candidate (see effectiveQueryCandidates/fuzzyScore). Falls back to the raw query when the type is supported, the query is a plain name, or nothing was resolved.
The effective search query.
// this.query === "10124-56-8", supportsCAS === false, resolved names available
this.effectiveQuery; // "Sodium hexametaphosphate"
Creates a new instance of the supplier base class. Initializes the supplier with query parameters, request limits, and abort controller. Sets up logging and default product values.
The search term to query products for
The maximum number of results to return (default: 5)
Optionalcontroller: AbortControllerAbortController instance for managing request cancellation
// Create a supplier with default limit
const supplier = new MySupplier("sodium chloride", undefined, new AbortController());
// Create a supplier with custom limit
const supplier = new MySupplier("acetone", 10, new AbortController());
// Create a supplier and handle cancellation
const controller = new AbortController();
const supplier = new MySupplier("ethanol", 5, controller);
// Later, to cancel all pending requests:
controller.abort();
Determines whether this supplier ships to the given country. Prefers the
explicit shipsTo allowlist when the supplier declares one; otherwise falls
back to the coarse shipping scope — "worldwide"/"international" ship
anywhere, while "domestic"/"local" ship only within the supplier's own
country.
The user's location as an ISO 3166-1 alpha-2 country code.
True if the supplier ships to location, false otherwise.
// Supplier with shipsTo = ["US", "CA"]:
supplier.shipsToCountry("US"); // true
supplier.shipsToCountry("DE"); // false
// Domestic US supplier (no shipsTo):
supplier.shipsToCountry("US"); // true
supplier.shipsToCountry("DE"); // false
// Worldwide supplier (no shipsTo):
supplier.shipsToCountry("DE"); // true
public shipsToCountry(location: CountryCode): boolean {
// Pass an explicit meta object rather than `this.constructor` so the
// (protected) `shipsTo` static doesn't clash with SupplierStaticMeta's public
// shape; the getters read the concrete class's static values.
return SupplierBase.shipsToCountryStatic(
{ shipping: this.shipping, country: this.country, shipsTo: this.shipsTo },
location,
);
}
StaticshipsWhether a supplier with the given static shipping metadata ships to
location. Shared by the instance shipsToCountry and SupplierFactory
so the UI can test shipping compatibility from a supplier's static fields
without instantiating it.
The supplier's static shipping/country/shipsTo metadata.
Destination country (ISO 3166-1 alpha-2).
True when the supplier ships to location.
public static shipsToCountryStatic(meta: SupplierStaticMeta, location: CountryCode): boolean {
if (meta.shipsTo) {
return meta.shipsTo.includes(location);
}
switch (meta.shipping) {
case 'worldwide':
case 'international':
return true;
case 'domestic':
case 'local':
return meta.country === location;
default:
return true;
}
}
Initializes the cache for the supplier. This is called after construction to ensure supplierName is set.
The cache is initialized with the supplier's name and is used to store both query results and product data. This method should be called after the supplier's name is set to ensure proper cache key generation.
class MySupplier extends SupplierBase<Product> {
constructor() {
super("acetone", 5);
// supplierName is set here
this.initCache(); // Initialize cache after supplierName is set
}
}
public initCache(
enabled: boolean = true,
doNotCacheEmptyResults: boolean = false,
cacheTtlMinutes: number = 0,
noCacheStatusCodes: number[] = [429],
): void {
this.cache = new SupplierCache(
this.supplierName,
this.constructor.name,
enabled,
doNotCacheEmptyResults,
cacheTtlMinutes,
);
// Stored on the supplier (not the cache): the decision is made at cache-write time in
// getProductData(WithCache), where the per-product fetch status is known.
this.noCacheStatusCodes = noCacheStatusCodes ?? [429];
}
Applies (or clears) a runtime override for the fuzz scorer. Driven by
userSettings.fuzzScorerOverride — when the user picks a scorer in the
Advanced drawer section, SupplierFactory calls this on each instance
so the choice takes effect uniformly across every supplier.
Silently ignores unknown names so an outdated / corrupted setting can't blow up the search flow — callers fall back to the subclass default.
Name of a scorer from FUZZ_SCORERS, or undefined to
clear the override and use the subclass default.
const supplier = new MySupplier("acetone", 5, controller);
supplier.setFuzzScorerOverride("token_set_ratio");
// fuzzyFilter now uses token_set_ratio regardless of MySupplier's default
supplier.setFuzzScorerOverride(undefined);
// back to MySupplier's default
public setFuzzScorerOverride(name: string | undefined): void {
console.debug('setFuzzScorerOverride', { name });
if (isFuzzScorerName(name)) {
this.fuzzScorerOverride = FUZZ_SCORERS[name];
} else {
this.fuzzScorerOverride = undefined;
}
}
Applies a runtime override for supplierSearchTimeBudgetSec, driven by
userSettings.supplierSearchTimeBudgetSec (set in the Advanced settings section). Accepts the raw
setting value (which may arrive as a string from the number input). An absent, empty, or
invalid value is ignored so the supplier keeps its class default; a valid non-negative number
(including 0 to disable the limit) replaces it.
The override in seconds, or any value (invalid/empty input is ignored)
supplier.setSupplierSearchTimeBudgetSec(60); // cap searches at 60s
supplier.setSupplierSearchTimeBudgetSec(""); // no-op, keep the per-supplier default
public setSupplierSearchTimeBudgetSec(value: unknown): void {
if (value === undefined || value === null || value === '') {
return;
}
const seconds = Number(value);
if (!Number.isNaN(seconds) && seconds >= 0) {
this.supplierSearchTimeBudgetSec = seconds;
}
}
Sets the parsed advanced-search query for this instance. Called by
SupplierFactory once per search so every supplier shares the same parse of
the user's input.
The parsed query, or undefined to clear it.
supplier.setParsedQuery(parseSearchQuery("Sodium OR Potassium"));
public setParsedQuery(parsed: ParsedSearchQuery | undefined): void {
this.parsedQuery = parsed;
}
Applies a runtime override for fuzzyFilteringDisabled, driven by
userSettings.fuzzyFilteringDisabled. When true, fuzzball scoring is skipped
and only the boolean predicate (substring matching) is applied.
True to disable fuzzy filtering, false (default) to keep it.
supplier.setFuzzyFilteringDisabled(true); // show raw/boolean-only results
public setFuzzyFilteringDisabled(value: boolean): void {
this.fuzzyFilteringDisabled = value === true;
}
Sets the map of resolved structure terms for this instance. Called by
SupplierFactory once per search so every supplier shares one resolution of
any SMILES/structure terms instead of each hitting the network.
Map of raw search term → resolved structure, or undefined when none.
supplier.setResolvedStructures(new Map([["CCO", { name: "ethanol", cas: ["64-17-5"] }]]));
public setResolvedStructures(resolved: ReadonlyMap<string, ResolvedStructure> | undefined): void {
this.resolvedStructures = resolved;
this.effectiveQueryCandidatesCache = undefined;
}
ProtectedeffectiveThe candidate queries this supplier matches against. Normally just
[this.query], but when the query is a CAS/formula/SMILES identifier the
supplier can't search itself (per supportsCAS/supportsFormula/
supportsSMILES), it's the factory-resolved chemical-name candidates —
the compound Title plus cleaned synonyms — so a product can be matched by
whichever name scores best (see fuzzyScore). Falls back to
[this.query] when the type is supported, the query is a plain name, or
nothing was resolved. Memoized.
The candidate queries, best first (never empty).
// this.query === "10124-56-8", supportsCAS === false, resolution available
this.effectiveQueryCandidates(); // ["Hexasodium hexametaphosphate", "Sodium hexametaphosphate", …]
protected effectiveQueryCandidates(): string[] {
if (this.effectiveQueryCandidatesCache === undefined) {
this.effectiveQueryCandidatesCache = this.resolveEffectiveQueryCandidates();
}
return this.effectiveQueryCandidatesCache;
}
ProtectedgetReturns the parsed query for this instance. Uses the effectiveQuery, so
an identifier query the supplier can't search is parsed as its resolved name.
Otherwise lazily parses this.query when SupplierFactory did not set a
parsed query (e.g. in unit tests that construct a supplier directly).
The parsed search query.
const { isAdvanced, ast } = this.getAst();
protected getAst(): ParsedSearchQuery {
const effective = this.effectiveQuery;
if (effective !== this.query) {
return parseSearchQuery(effective);
}
return this.parsedQuery ?? parseSearchQuery(this.query);
}
ProtectedhttpRetrieves HTTP headers from a URL using a HEAD request. Useful for checking content types, caching headers, and other metadata without downloading the full response.
The URL to fetch headers from
Promise resolving to the response headers or void if request fails
// Basic usage
const headers = await supplier.httpGetHeaders('https://example.com/product/123');
if (headers) {
console.log('Content-Type:', headers['content-type']);
}
// With error handling
try {
const headers = await supplier.httpGetHeaders('https://example.com/product/123');
if (headers) {
console.log('Headers:', headers);
}
} catch (err) {
console.error('Failed to fetch headers:', err);
}
protected async httpGetHeaders(url: string | URL): Promise<Maybe<HeadersInit>> {
const requestObj = new Request(this.href(url), {
signal: this.controller.signal,
headers: new Headers(this.headers),
referrer: this.baseURL,
referrerPolicy: 'strict-origin-when-cross-origin',
body: null,
method: 'HEAD',
mode: 'cors',
credentials: 'include',
});
try {
const httpResponse = await this.fetch(requestObj);
return Object.fromEntries(httpResponse.headers.entries()) satisfies HeadersInit;
} catch (error: unknown) {
if (error instanceof Error && error.name === 'AbortError') {
this.logger.warn('Request was aborted', { error, signal: this.controller.signal });
this.controller.abort('Abort signal detected');
} else {
this.logger.error('Error received during fetch:', {
error,
signal: this.controller.signal,
});
}
return;
}
}
ProtectedhttpSends a POST request to the given URL with the given body and headers. Handles request setup, error handling, and response caching.
The request configuration options
Promise resolving to the Response object or void if request fails
// Basic POST request
const response = await supplier.httpPost({
path: '/api/v1/products',
body: { name: 'Test Chemical' }
});
// POST with custom headers
const response = await supplier.httpPost({
path: '/api/v1/products',
body: { name: 'Test Chemical' },
headers: {
'Authorization': 'Bearer token123',
'Content-Type': 'application/json'
}
});
// POST with custom host and params
const response = await supplier.httpPost({
path: '/api/v1/products',
host: 'api.example.com',
body: { name: 'Test Chemical' },
params: { version: '2' }
});
// Error handling
try {
const response = await supplier.httpPost({ path: '/api/v1/products', body: { name: 'Test' } });
if (response && response.ok) {
const data = await response.json();
console.log('Created:', data);
}
} catch (err) {
console.error('POST failed:', err);
}
protected async httpPost({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<Response>> {
this.logger.log('httpPost| Requesting:', {
path,
host,
body,
params,
headers,
});
const method = 'POST';
const mode = 'cors';
const referrer = this.baseURL;
const referrerPolicy = 'strict-origin-when-cross-origin';
const signal = this.controller.signal;
const headersObj = new Headers({
...this.headers,
...headers,
});
let bodyStr = null;
if (body instanceof FormData) {
headersObj.set('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8');
bodyStr = body;
} else if (typeof body === 'string') {
bodyStr = body;
} else if (typeof body === 'object' && body !== null) {
bodyStr = JSON.stringify(body);
}
const url = this.href(path, params, host);
const requestObj = new Request(url, {
signal,
headers: headersObj,
referrer,
referrerPolicy,
body: bodyStr,
method,
mode,
credentials: 'include',
});
// Fetch the goods
const httpResponse = await this.fetch(requestObj);
if (!isHttpResponse(httpResponse) || !httpResponse.ok) {
const badResponse = await httpResponse.text();
this.logger.error('Invalid POST response: ', badResponse);
throw new TypeError(`Invalid POST response: ${String(httpResponse)}`);
}
return httpResponse;
}
ProtectedhttpSends a POST request with the body encoded as multipart/form-data.
Converts the given object into a FormData instance (one field per
key/value pair) before delegating to httpPost.
The request configuration options. body must be a non-null object.
Promise resolving to the Response object or void if the request fails
TypeError - If body is not an object, or the response is not a valid HTTP response
const response = await this.httpPostFormData({
path: "/api/v1/cart",
body: { productId: "123", quantity: "2" },
});
protected async httpPostFormData({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<Response>> {
if (typeof body !== 'object' || body === null) {
throw new TypeError('httpPostFormData| Body must be an object');
}
headers = {
...headers,
'Content-Type': 'application/x-www-form-urlencoded',
};
const formData = new FormData();
for (const [key, value] of Object.entries(body)) {
formData.append(key, value);
}
const httpResponse = await this.httpPost({ path, host, body: formData, params, headers });
if (!isHttpResponse(httpResponse) || !httpResponse.ok) {
const badResponse = await httpResponse?.text();
this.logger.error('Invalid POST response: ', badResponse);
throw new TypeError(`Invalid POST response: ${String(httpResponse)}`);
}
this.logger.log('httpPostFormData| Successfully sent POST request to:', path);
return httpResponse;
}
ProtectedhttpSends a POST request and returns the response as a JSON object.
The parameters for the POST request.
The response from the POST request as a JSON object.
// Basic usage
const data = await supplier.httpPostJson({
path: '/api/v1/products',
body: { name: 'John' }
});
// With custom headers and error handling
try {
const data = await supplier.httpPostJson({
path: '/api/v1/products',
body: { name: 'John' },
headers: { 'Authorization': 'Bearer token123' }
});
if (data) {
console.log('Created:', data);
}
} catch (err) {
console.error('POST JSON failed:', err);
}
protected async httpPostJson({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<JsonValue>> {
const httpResponse = await this.httpPost({ path, host, body, params, headers });
if (!isJsonResponse(httpResponse) || !httpResponse.ok) {
this.logger.error('httpPostJson| Invalid POST response: ', {
httpResponse,
path,
host,
body,
params,
headers,
});
throw new TypeError(`httpPostJson| Invalid POST response: ${httpResponse}`);
}
return await httpResponse.json();
}
ProtectedhttpSends a POST request and returns the response as a HTML string.
The request configuration options
Promise resolving to the HTML response as a string or void if request fails
TypeError - If the response is not valid HTML content
// Basic usage
const html = await supplier.httpPostHtml({
path: '/api/v1/products',
body: { name: 'John' }
});
protected async httpPostHtml({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<string>> {
const httpResponse = await this.httpPost({ path, host, body, params, headers });
if (!isHtmlResponse(httpResponse)) {
throw new TypeError(`httpPostHtml| Invalid POST response: ${httpResponse}`);
}
return await httpResponse.text();
}
ProtectedhttpSends a GET request to the given URL with the specified options. Handles request setup, error handling, and response caching.
The request configuration options
Promise resolving to the Response object or void if request fails
// Basic GET request
const response = await supplier.httpGet({
path: '/products/search',
params: { query: 'sodium chloride' }
});
// GET with custom headers
const response = await supplier.httpGet({
path: '/products/search',
headers: { 'Accept': 'application/json' }
});
// GET with custom host
const response = await supplier.httpGet({
path: '/products/search',
host: 'api.example.com',
params: { category: 'chemicals' }
});
// Error handling
try {
const response = await supplier.httpGet({ path: '/products/search' });
if (response && response.ok) {
const data = await response.json();
console.log('Products:', data);
}
} catch (err) {
console.error('GET failed:', err);
}
protected async httpGet({
path,
params,
headers,
host,
rethrowErrors,
}: RequestOptions): Promise<Maybe<Response>> {
// Check if the request has been aborted before proceeding
if (this.controller.signal.aborted) {
this.logger.warn('Request was aborted before fetch', {
signal: this.controller.signal,
});
return;
}
const headersRaw = { ...this.headers };
Object.assign(headersRaw, {
accept: [
'text/html',
'application/xhtml+xml',
'application/xml;q=0.9',
'image/avif',
'image/webp',
'image/apng',
'*/*;q=0.8',
].join(','),
...(headers ?? {}),
});
const requestObj = new Request(this.href(path, params, host), {
signal: this.controller.signal,
headers: new Headers(headersRaw),
referrer: this.baseURL,
referrerPolicy: 'no-referrer',
body: null,
method: 'GET',
mode: 'cors',
credentials: 'include',
redirect: 'follow',
});
try {
// Fetch the goods
const httpResponse = await this.fetch(requestObj.url, requestObj);
const responseHeaders = Object.fromEntries(
httpResponse.headers.entries(),
) satisfies HeadersInit;
this.logger.debug('responseHeaders:', responseHeaders);
this.logger.debug('responseHeaders.location:', responseHeaders.location);
return httpResponse;
} catch (error: unknown) {
if (error instanceof Error && error.name === 'AbortError') {
this.logger.warn('Request was aborted', { error, signal: this.controller.signal });
this.controller.abort('Abort signal detected');
return;
}
this.logger.error('Error received during fetch:', {
error,
signal: this.controller.signal,
});
// Opt-in: surface the failure (e.g. an HttpError 429) so the caller can apply
// status-aware retry/backoff. Default behavior remains swallow-and-return-undefined.
if (rethrowErrors) {
throw error;
}
return;
}
}
ProtectedfuzzyScores a single string against the current search query (this.query)
using the active fuzz scorer — the user's fuzzScorerOverride when set,
otherwise the supplier's fuzzScorer. Returns a 0–100 similarity score
(higher is closer). Useful for suppliers that can only fuzz-match after a
secondary request reveals the real product name, e.g. when the search
index only exposes coarse category breadcrumbs.
The text to score against this.query.
A similarity score from 0 (no match) to 100 (identical).
// this.query === "sodium borohydride"
this.fuzzyScore("Sodium borohydride, min 95%"); // ~90
this.fuzzyScore("Acetone"); // ~10
protected fuzzyScore(text: string): number {
const activeScorer = this.fuzzScorerOverride ?? this.fuzzScorer;
// Score against every effective-query candidate and keep the best — for an
// identifier query these are the resolved name + synonyms, so a product
// matches whichever name it names. A plain query has a single candidate.
let best = 0;
for (const candidate of this.effectiveQueryCandidates()) {
const score = activeScorer(candidate, text);
if (score > best) {
best = score;
}
}
return best;
}
ProtectedfuzzyFilters an array of data using fuzzy string matching to find items that closely match a query string. Uses the WRatio algorithm from fuzzball for string similarity comparison.
The search string to match against
Array of data objects to search through
Minimum match percentage (0-100) for a match to be included (default: 55)
Array of matching data objects with added fuzzy match metadata
// Example with simple string array
const products = [
{ title: "Sodium Chloride", price: 29.99 },
{ title: "Sodium Hydroxide", price: 39.99 },
{ title: "Potassium Chloride", price: 19.99 }
];
const matches = this.fuzzyFilter("sodium chloride", products);
// Returns: [
// {
// title: "Sodium Chloride",
// price: 29.99,
// _fuzz: { score: 100, idx: 0 }
// },
// {
// title: "Sodium Hydroxide",
// price: 39.99,
// _fuzz: { score: 85, idx: 1 }
// }
// ]
// Example with custom minMatchPercentage
const strictMatches = this.fuzzyFilter("sodium chloride", products, 90);
// Returns only exact matches with score >= 90
// Example with different data structure
const chemicals = [
{ name: "NaCl", formula: "Sodium Chloride" },
{ name: "NaOH", formula: "Sodium Hydroxide" }
];
// Override titleSelector to use formula field
this.titleSelector = (data) => data.formula;
const formulaMatches = this.fuzzyFilter("sodium chloride", chemicals);
protected fuzzyFilter<X>(
query: string,
data: X[],
minMatchPercentage: number = this.minMatchPercentage,
): X[] {
// User's Advanced-settings override wins over the subclass default.
const activeScorer = this.fuzzScorerOverride ?? this.fuzzScorer;
// console.log(
// `[fuzzyFilter] ${this.supplierName} query="${query}" — scorer comparison (cutoff=${minMatchPercentage})`,
// );
if (IS_DEV_BUILD) {
this.showFuzzScorerComparisonTable(query, data);
}
if (this.fuzzyFilterRankOnly) {
// Rank every candidate by score (no cutoff) and return them in score order; the
// caller slices the top N. Avoids dropping clear matches whose ratio-style score
// falls under minMatchPercentage purely because the title dwarfs the query.
return extract(query, data, {
scorer: activeScorer,
processor: this.titleSelector,
sortBySimilarity: true,
}).map(([obj, score, idx]) => this.attachFuzz(obj, score, idx));
}
const results = extract(query, data, {
scorer: activeScorer,
processor: this.titleSelector,
cutoff: minMatchPercentage,
sortBySimilarity: true,
}).reduce<FuzzyMatchResult<X>[]>((acc, [obj, score, idx]) => {
if (score < minMatchPercentage) {
this.logger.debug('fuzzyFilter: score below minimum match percentage, excluding product', {
product: obj,
score,
idx,
minMatchPercentage: minMatchPercentage,
});
return acc;
}
acc[idx] = Object.assign(obj, { _fuzz: { score, idx }, matchPercentage: score });
return acc;
}, []);
this.logger.debug('[fuzzyFilter]', {
supplierName: this.supplierName,
query,
minMatchPercentage,
activeScorer,
results,
});
// Get rid of any empty items that didn't match closely enough
return results.filter((item) => !!item);
}
ProtectedattachAnnotates an item with its fuzz score, returning it as a FuzzyMatchResult.
Mutates obj in place (rather than spreading) so non-plain inputs — e.g. DOM
Elements fuzzed by HTML-scraping suppliers — keep their prototype and methods. The
as is needed because a direct Object.assign on an unconstrained X widens it away.
The matched item to annotate.
The fuzz score (0–100).
The item's index in the source list.
obj extended with _fuzz and matchPercentage.
this.attachFuzz({ name: "NaCl" }, 92, 0);
// => { name: "NaCl", _fuzz: { score: 92, idx: 0 }, matchPercentage: 92 }
protected attachFuzz<X>(obj: X, score: number, idx: number): FuzzyMatchResult<X> {
const result = obj as FuzzyMatchResult<X>;
result._fuzz = { score, idx };
result.matchPercentage = score;
return result;
}
ProtectedmatchReads the fuzzy/AST match score that fuzzyFilterAst/attachFuzz stamped onto a
scored search item, for handing to builder.setMatchPercentage in a supplier's
initProductBuilders. Returns undefined when the item was never scored (e.g. fuzzy filtering
disabled on a plain query) — setMatchPercentage ignores that, leaving the score unset.
A scored search item (the raw supplier item after fuzzyFilterAst).
The stamped match percentage, or undefined when the item carries no score.
builder.setMatchPercentage(this.matchScoreOf(item));
protected matchScoreOf(item: unknown): number | undefined {
if (isPopulatedObject(item) && typeof item.matchPercentage === 'number') {
return item.matchPercentage;
}
return undefined;
}
ProtectedfuzzyAdvanced-search-aware companion to fuzzyFilter. Filters data using
the parsed query (getAst) so boolean operators (AND/OR/NOT) and
nesting are honored, and respects the fuzzyFilteringDisabled toggle.
Behavior matrix:
data unchanged (raw supplier results).Array of raw search-result objects to filter.
Minimum leaf match score when fuzzing is on.
The filtered (and, when fuzzing, ranked) subset, each item tagged
with _fuzz/matchPercentage like fuzzyFilter.
// this.query === "Sodium OR Potassium"
const matches = this.fuzzyFilterAst(products);
protected fuzzyFilterAst<X>(
data: X[],
minMatchPercentage: number = this.minMatchPercentage,
): X[] {
const parsed = this.getAst();
if (!parsed.isAdvanced) {
// Plain query: no filtering when disabled; the multi-candidate path for a
// resolved identifier query (several candidate names); else the legacy
// single-query fuzzy path.
if (this.fuzzyFilteringDisabled) {
return data;
}
if (this.effectiveQueryCandidates().length > 1) {
return this.fuzzyFilterCandidates(data, minMatchPercentage);
}
return this.fuzzyFilter(parsed.raw.trim(), data, minMatchPercentage);
}
const scorer = this.fuzzScorerOverride ?? this.fuzzScorer;
// Rank-only floors the leaf score at 0 so predicate-matching items are never dropped
// for a low fuzz score; the sort below still ranks them. Otherwise enforce the cutoff.
const threshold = this.fuzzyFilteringDisabled
? 1
: this.fuzzyFilterRankOnly
? 0
: minMatchPercentage;
const fuzzyWords = !this.fuzzyFilteringDisabled;
const matched = data.reduce<FuzzyMatchResult<X>[]>((acc, obj, idx) => {
const title = this.titleSelector(obj) ?? '';
const score = scoreAstMatch(title, parsed.ast, { scorer, threshold, fuzzyWords });
if (score === null) {
return acc;
}
acc.push(this.attachFuzz(obj, score, idx));
return acc;
}, []);
// Rank by relevance when fuzzing; preserve backend order when disabled.
if (!this.fuzzyFilteringDisabled) {
matched.sort((a, b) => (b.matchPercentage ?? 0) - (a.matchPercentage ?? 0));
}
return matched;
}
ProtectedfuzzyAdvanced-search-aware, keep-or-drop companion to fuzzyScore for
suppliers that re-filter a single title after a detail fetch (e.g. LiMac).
Returns the score to keep the item, or null to drop it, honoring both the
parsed query and the fuzzyFilteringDisabled toggle:
minMatchPercentage;
advanced query keeps items satisfying the predicate with fuzzy leaf scores.The text (e.g. a detail-page product name) to score.
A 0–100 score to keep the item, or null to drop it.
// this.query === "acid AND NOT boric"
this.fuzzyScoreAst("Sulfuric acid"); // a number
this.fuzzyScoreAst("Boric acid"); // null
protected fuzzyScoreAst(text: string): number | null {
const parsed = this.getAst();
if (!parsed.isAdvanced) {
if (this.fuzzyFilteringDisabled) {
return 100;
}
const score = this.fuzzyScore(text);
return score >= this.minMatchPercentage ? score : null;
}
const scorer = this.fuzzScorerOverride ?? this.fuzzScorer;
const threshold = this.fuzzyFilteringDisabled ? 1 : this.minMatchPercentage;
return scoreAstMatch(text, parsed.ast, {
scorer,
threshold,
fuzzyWords: !this.fuzzyFilteringDisabled,
});
}
ProtectedderiveDerives the backend search terms for a keyword-only supplier from an advanced query: one representative term per positive OR-group (the longest — most selective — token of each AND-group), de-duplicated and capped at maxFallbackQueries. Returns an empty array when there are no positive terms (e.g. a purely negative query), in which case the caller falls back to a single raw search.
The de-duplicated, capped list of backend search terms.
// this.query === "(Sodium OR Potassium) AND Hydroxide"
this.deriveFallbackTerms(); // ["Hydroxide", "Hydroxide"] -> ["Hydroxide"] (deduped)
protected deriveFallbackTerms(): string[] {
const groups = extractOrGroups(this.getAst().ast);
// How many AND-groups each term appears in — a term shared across groups (a
// common factor, e.g. "Hydroxide" in "(Sodium OR Potassium) AND Hydroxide")
// covers more of the query in fewer requests, so prefer it; break ties by
// length (more selective). Picking one representative term per group keeps
// each request's result set a superset of that group's matches.
const frequency = new Map<string, number>();
for (const group of groups) {
for (const term of new Set(group)) {
frequency.set(term, (frequency.get(term) ?? 0) + 1);
}
}
const terms = groups
.map(
(group) =>
group.slice().sort((a, b) => {
const byFrequency = (frequency.get(b) ?? 0) - (frequency.get(a) ?? 0);
return byFrequency !== 0 ? byFrequency : b.length - a.length;
})[0],
)
.filter((term): term is string => Boolean(term));
return [...new Set(terms)].slice(0, this.maxFallbackQueries);
}
ProtectedhttpMakes an HTTP GET request and returns the response as a string. Handles request configuration, error handling, and HTML parsing.
The request configuration options
Promise resolving to the HTML response as a string or void if request fails
TypeError - If the response is not valid HTML content
// Basic GET request
const html = await this.httpGetHtml({
path: "/api/products",
params: { search: "sodium" }
});
// GET request with custom headers
const html = await this.httpGetHtml({
path: "/api/products",
headers: {
"Authorization": "Bearer token123",
"Accept": "text/html"
}
});
// GET request with custom host
const html = await this.httpGetHtml({
path: "/products",
host: "api.supplier.com",
params: { limit: 10 }
});
protected async httpGetHtml({
path,
params,
headers,
host,
}: RequestOptions): Promise<Maybe<string>> {
const httpResponse = await this.httpGet({ path, params, headers, host });
if (!isHtmlResponse(httpResponse)) {
throw new TypeError(`httpGetHtml| Invalid GET response: ${httpResponse}`);
}
return await httpResponse.text();
}
ProtectedhttpMakes an HTTP GET request and returns the response as parsed JSON. Handles request configuration, error handling, and JSON parsing.
The request configuration options
Promise resolving to the parsed JSON response or void if request fails
TypeError - If the response is not valid JSON content
// Basic GET request
const data = await supplier.httpGetJson({ path: '/api/products', params: { search: 'sodium' } });
// GET request with custom headers
const data = await supplier.httpGetJson({
path: '/api/products',
headers: {
'Authorization': 'Bearer token123',
'Accept': 'application/json'
}
});
// GET request with custom host
const data = await supplier.httpGetJson({
path: '/products',
host: 'api.supplier.com',
params: { limit: 10 }
});
// Error handling
try {
const data = await supplier.httpGetJson({ path: '/api/products' });
if (data) {
console.log('Products:', data);
}
} catch (error) {
console.error('Failed to fetch products:', error);
}
protected async httpGetJson({
path,
params,
headers = {},
host,
}: RequestOptions): Promise<Maybe<JsonValue>> {
if (
typeof headers?.accept === 'undefined' ||
!Array.isArray(headers?.accept) ||
!headers?.accept.includes('application/json')
) {
headers.accept = ['application/json', 'text/plain', '*/*'].join(',');
}
const httpRequest = await this.httpGet({ path, params, headers, host });
if (!isJsonResponse(httpRequest)) {
const badResponse = isHttpResponse(httpRequest) ? await httpRequest.text() : undefined;
this.logger.error('Invalid HTTP GET JSON response:', {
badResponse,
httpRequest,
path,
params,
headers,
host,
});
return;
}
return await httpRequest.json();
}
ProtectedqueryExecutes a product search query with caching support. First checks the cache for existing results, then falls back to the actual query if needed. The limit parameter is only used for the actual query and doesn't affect caching.
The search term to query products for
The maximum number of results to return (defaults to instance limit)
Promise resolving to array of product builders or void if search fails
// Basic usage with default limit
const results = await supplier.queryProductsWithCache("acetone");
if (results) {
console.log(`Found ${results.length} products`);
}
// With custom limit
const results = await supplier.queryProductsWithCache("acetone", 10);
if (results) {
for (const builder of results) {
const product = await builder.build();
console.log(product.title, product.price);
}
}
protected async queryProductsWithCache(
query: string,
limit: number = this.limit,
): Promise<ProductBuilder<T>[] | void> {
// Check cache first (processed product data)
this.logger.debug(
'queryProductsWithCache: called for',
this.supplierName,
'query:',
query,
'limit:',
limit,
);
const key = this.cache.generateCacheKey(query);
const cached = await this.cache.getCachedQueryEntry(key);
this.logger.debug('queryProductsWithCache: cache hit:', !!cached, 'key:', key);
if (cached) {
const cachedLimit = cached.__cacheMetadata.limit;
const insufficientLimit = typeof cachedLimit === 'number' && cachedLimit < limit;
if (!insufficientLimit) {
this.logger.debug('Returning cached query results');
// Re-initialize product builders from cached processed data
return ProductBuilder.createFromCache<T>(this.baseURL, cached.data.slice(0, limit));
}
// Cached entry was built with a smaller limit than requested — drop it and re-query below.
this.logger.debug('Invalidating query cache due to insufficient limit', {
cachedLimit,
requestedLimit: limit,
});
await deleteSupplierQueryCacheEntry(key);
}
// If not in cache, perform the actual query. Run setup first so any
// subclass state it mutates (headers, localStorage, tokens, etc.) is
// in place before `queryProducts` reads it. Memoized, so this is cheap
// on repeat calls within the same supplier instance.
await this.ensureSetup();
const results = await this.queryProductsResolved(query, limit);
if (results) {
// Store processed results in cache (dumped/serialized form) and the limit used
await this.cache.cacheQueryResults(
query,
results.map((b) => b.dump()),
limit,
);
}
return results;
}
ProtectedqueryResolves the supplier's queryProducts for the current search, applying the
keyword-only advanced-search fallback when needed. For a plain query, or for
a supplier that handles boolean queries natively
(supportsNativeAdvancedSearch), this is a single queryProducts
call. For an advanced query on a keyword-only backend, it issues one search
per derived OR-group term (deriveFallbackTerms) and unions the
results, deduping by product URL/ID. Each queryProducts batch already
enforces the full boolean predicate via fuzzyFilterAst, so the union
is the set of products matching the whole query. Honors
httpRequestHardLimit; the fetches run inside execute()'s existing
supplierSearchTimeBudgetSec race so an aborted controller cancels them.
The raw search query.
The per-supplier result limit.
The (possibly unioned) product builders, or void.
protected async queryProductsResolved(
query: string,
limit: number,
): Promise<ProductBuilder<T>[] | void> {
const parsed = this.getAst();
if (!parsed.isAdvanced || this.supportsNativeAdvancedSearch) {
return this.queryProducts(query, limit);
}
const terms = this.deriveFallbackTerms();
if (terms.length <= 1) {
// Nothing to union (single positive term, or a purely negative query):
// run the raw query once and let fuzzyFilterAst enforce the predicate.
return this.queryProducts(terms[0] ?? query, limit);
}
const seen = new Set<string>();
const union: ProductBuilder<T>[] = [];
for (const term of terms) {
if (this.requestCount >= this.httpRequestHardLimit) {
this.logger.warn('queryProductsResolved: httpRequestHardLimit reached, stopping fallback', {
term,
requestCount: this.requestCount,
});
break;
}
const batch = await this.queryProducts(term, limit);
if (!batch) {
continue;
}
for (const builder of batch) {
const key = String(builder.get('url') ?? builder.get('id') ?? builder.get('title') ?? '');
if (key === '' || seen.has(key)) {
continue;
}
seen.add(key);
union.push(builder);
}
}
return union.length > 0 ? union : undefined;
}
Executes the supplier's search query and returns the results. This method will execute all results concurrently (to the limits set in the supplier class), and resolve to an array of product objects.
Promise resolving to an array of products
This method is used to execute the supplier's search query and return the results.
public async *execute(): AsyncGenerator<T, void, undefined> {
// setup() is not called eagerly here — it's run lazily from the
// phase-boundary gates inside `queryProductsWithCache` and
// `getProductData` / `getProductDataWithCache`. A fully cached search
// never reaches those gates, so setup's token/cookie/permission
// requests are skipped entirely.
// Snapshot the user's ignore list once per search. Any product whose
// exclusion key matches an entry here is dropped before the detail phase
// runs (see the filter after queryProductsWithCache below).
this.excludedProductKeys = await loadExcludedProductKeys();
// Over-fetch by the number of previously-ignored products belonging to
// this supplier so that, in the worst case where every ignored product
// appears in the top of the query result set, we still end up with
// `this.limit` survivors after filtering. The queryProductsWithCache
// cache invalidates itself when the requested limit exceeds the cached
// limit, so this is safe.
const excludedForSupplier = await countExcludedProductsForSupplier(this.supplierName);
const fetchLimit = this.limit + excludedForSupplier;
incrementSearchQueryCount(this.supplierName);
// Optional per-supplier search-time budget; see armSearchTimeout. When it elapses the
// race below wins via SEARCH_TIMEOUT and flushes any not-yet-yielded products.
const SEARCH_TIMEOUT = Symbol('searchTimeout');
const { promise: timeoutPromise, handle: timeoutHandle } =
this.armSearchTimeout(SEARCH_TIMEOUT);
try {
const results = await this.queryProductsWithCache(this.query, fetchLimit);
if (!results || results.length === 0) {
this.logger.log(`No query results found`);
return;
}
// Drop any products the user has ignored, then slice back down to the
// user-visible limit. Uses the same dual-read (identity + legacy URL) as
// getProductData/partitionForBatch so the check is consistent wherever it
// fires first.
const survivors: ProductBuilder<T>[] = [];
for (const builder of results) {
if (survivors.length >= this.limit) break;
if (this.isExcluded(builder)) {
this.logger.debug('Skipping excluded product (pre-detail)', {
url: builder.get('url'),
});
continue;
}
survivors.push(builder);
}
this.products = survivors;
const queue = new Queue(this.maxConcurrentRequests, this.minConcurrentCycle);
// Each task fetches a product's detail data and finishes it, tagged with its index so the
// yield loop can track which products are still outstanding when the budget elapses.
const pending = new Map<number, Promise<{ index: number; finished: Maybe<T> }>>();
this.products.forEach((product, index) => {
pending.set(
index,
queue.run(async () => {
// If the budget already elapsed, skip the (now-aborted) detail fetch — the timeout
// handler below emits this product's basic data directly.
if (this.controller.signal.aborted) {
return { index, finished: undefined };
}
try {
const builder = await this.getProductData(product);
const finished = builder ? await this.finishProduct(builder) : undefined;
return { index, finished };
} catch (e: unknown) {
this.logger.error('Error processing product', { error: e, product });
incrementParseError(this.supplierName);
return { index, finished: undefined };
}
}),
);
});
// As each task resolves, yield the product. Race against the optional search-time budget;
// when it elapses, emit every not-yet-yielded product with its basic (query-phase) data so
// the rows still show — the enrichment for those simply wasn't cached, so a later search
// (served from the query cache) re-fetches their detail data.
const yielded = new Set<number>();
while (pending.size > 0) {
const result = await Promise.race(
timeoutPromise ? [...pending.values(), timeoutPromise] : [...pending.values()],
);
if (result === SEARCH_TIMEOUT) {
for (let index = 0; index < this.products.length; index++) {
if (yielded.has(index)) continue;
// The builder is enriched in place, so this carries whatever detail data was set
// before the abort, falling back to the basic query-phase fields otherwise.
const finished = await this.finishProduct(this.products[index]);
if (finished) {
yield finished;
}
}
break;
}
pending.delete(result.index);
yielded.add(result.index);
if (result.finished) {
yield result.finished;
}
}
} finally {
if (timeoutHandle !== undefined) {
clearTimeout(timeoutHandle);
}
}
}
ProtectedfinishFinalizes a partial product by adding computed properties and validating the result. This method:
The ProductBuilder instance containing the partial product to finalize
Promise resolving to a complete Product object or void if validation fails
// Example with a valid partial product
const builder = new ProductBuilder<Product>(this.baseURL);
builder
.setBasicInfo("Sodium Chloride", "/products/nacl", "ChemSupplier")
.setPricing(29.99, "USD", "$")
.setQuantity(500, "g");
const finishedProduct = await this.finishProduct(builder);
if (finishedProduct) {
console.log("Finalized product:", {
title: finishedProduct.title,
price: finishedProduct.price,
quantity: finishedProduct.quantity,
uom: finishedProduct.uom,
usdPrice: finishedProduct.usdPrice,
baseQuantity: finishedProduct.baseQuantity
});
}
// Example with an invalid partial product
const invalidBuilder = new ProductBuilder<Product>(this.baseURL);
invalidBuilder.setBasicInfo("Sodium Chloride", "/products/nacl", "ChemSupplier");
// Missing required fields
const invalidProduct = await this.finishProduct(invalidBuilder);
if (!invalidProduct) {
console.log("Failed to finalize product - missing required fields");
}
protected async finishProduct(product: ProductBuilder<T>): Promise<Maybe<T>> {
if (!isMinimalProduct(product.dump())) {
this.logger.warn('Unable to finish product - Minimum data not set', { product });
return;
}
// Dev guard: every builder should carry a stamped `cacheKey` (set at parse
// time via `setCacheKey(getUniqueProductKey(item))`). A missing key means a
// supplier implemented `getUniqueProductKey` but forgot to stamp — it would
// silently disable per-product caching and precise exclusion for that
// product. Surface it loudly in dev/tests.
if (IS_DEV_BUILD && product.get('cacheKey') == null) {
this.logger.error('finishProduct| product is missing a stamped cacheKey', {
supplier: this.supplierName,
url: product.get('url'),
});
}
// Set the country and shipping scope of the supplier
// have different restrictions on different products or countries.
product.setSupplierCountry(this.country);
product.setSupplierShipping(this.shipping);
if (this.paymentMethods.length > 0) {
product.setSupplierPaymentMethods(this.paymentMethods);
}
// Marketplace storefronts. The "*only" methods point users away from the supplier's own
// (restricted) site, so a missing store URL is a misconfiguration — surface it loudly in dev.
// The plain "ebay"/"amazon" methods instead drive an informational "more products there"
// notice; the store URL is optional, so stamp it only when present and don't warn.
if (this.paymentMethods.includes('ebayonly')) {
if (IS_DEV_BUILD && !this.ebayStoreURL) {
this.logger.error("finishProduct| supplier declares 'ebayonly' but sets no ebayStoreURL", {
supplier: this.supplierName,
});
}
product.setSupplierEbayStoreURL(this.ebayStoreURL);
} else if (this.paymentMethods.includes('ebay') && this.ebayStoreURL) {
product.setSupplierEbayStoreURL(this.ebayStoreURL);
}
if (this.paymentMethods.includes('amazononly')) {
if (IS_DEV_BUILD && !this.amazonStoreURL) {
this.logger.error(
"finishProduct| supplier declares 'amazononly' but sets no amazonStoreURL",
{ supplier: this.supplierName },
);
}
product.setSupplierAmazonStoreURL(this.amazonStoreURL);
} else if (this.paymentMethods.includes('amazon') && this.amazonStoreURL) {
product.setSupplierAmazonStoreURL(this.amazonStoreURL);
}
const built = await product.build();
return built;
}
ProtectedproductThe stable per-product cache/exclusion key for a builder: the identity
stamped on it at parse time (getUniqueProductKey →
setCacheKey), hashed with the supplier name via
getProductIdentityKey. Returns undefined when no identity was
stamped (so callers can skip the identity cache/exclusion path).
The product builder
The identity cache key, or undefined when unstamped
const key = this.productIdentityKey(builder); // md5({key, supplier}) or undefined
protected productIdentityKey(product: ProductBuilder<T>): string | undefined {
const identity = product.get('cacheKey');
if (typeof identity === 'string' && identity.length > 0) {
return this.cache.getProductIdentityCacheKey(identity);
}
return undefined;
}
ProtectedisWhether a product is on the user's ignore list, matched by its identity key (productIdentityKey) — the same key the "Ignore Product" action writes.
The product builder to check
true if the product matches an ignore-list entry
if (this.isExcluded(builder)) continue; // skip ignored product
protected isExcluded(product: ProductBuilder<T>): boolean {
const identityKey = this.productIdentityKey(product);
return identityKey !== undefined && this.excludedProductKeys.has(identityKey);
}
ProtectedpartitionPartitions query-phase builders for a batch supplier (one that enriches
details up front rather than per-product in getProductData). Runs a
three-way split: ignored products are dropped from both results;
cache hits (found in the product-detail cache by their stamped
identity) are hydrated in place via setData and kept in survivors but
excluded from misses; everything else is a miss, kept in both. The
caller enriches only misses, then caches them, and returns survivors.
Skips the cache lookup entirely when skipProductDetailCache is true (pure-search suppliers), so those still drop ignored products but treat every survivor as needing no enrichment.
The query-phase builders (each already setCacheKey-stamped)
{ survivors, misses } — see above
const { survivors, misses } = await this.partitionForBatch(builders);
await this.enrichVariants(misses);
await this.cacheProductBuilders(misses);
return survivors;
protected async partitionForBatch(
products: ProductBuilder<T>[],
): Promise<{ survivors: ProductBuilder<T>[]; misses: ProductBuilder<T>[] }> {
const survivors: ProductBuilder<T>[] = [];
const misses: ProductBuilder<T>[] = [];
for (const product of products) {
if (this.isExcluded(product)) {
continue;
}
survivors.push(product);
if (this.skipProductDetailCache) {
continue;
}
const key = this.productIdentityKey(product);
const cached = key ? await this.cache.getCachedProductData(key) : undefined;
if (isCachedProductData<T>(cached)) {
product.setData(cached);
} else {
misses.push(product);
}
}
return { survivors, misses };
}
ProtectedcacheWrites each enriched builder to the product-detail cache under its stamped
identity key. No-ops for suppliers with skipProductDetailCache true,
for aborted searches, and for builders shouldCacheProductData
rejects (e.g. a fetch that hit a noCacheStatusCode). Used by batch
suppliers after enriching their misses.
The enriched builders to persist
A promise that resolves once all writes complete
await this.cacheProductBuilders(misses);
protected async cacheProductBuilders(products: ProductBuilder<T>[]): Promise<void> {
if (this.skipProductDetailCache || this.controller.signal.aborted) {
return;
}
await Promise.all(
products.map(async (product) => {
const key = this.productIdentityKey(product);
if (key && this.shouldCacheProductData(product)) {
await this.cache.cacheProductData(key, product.dump());
}
}),
);
}
ProtectedhrefTakes in either a relative or absolute URL and returns an absolute URL. This is useful for when you aren't sure if the link (retrieved from parsed text, a setting, an element, an anchor value, etc) is absolute or not. Using relative links will result in http://chrome-extension://... being added to the link.
URL object or string
Optionalparams: Maybe<RequestParams>The parameters to add to the URL.
Optionalhost: stringThe host to use for overrides (eg: needing to call a different host for an API)
absolute URL
this.href('/some/path')
// https://supplier_base_url.com/some/path
this.href('https://supplier_base_url.com/some/path', null, 'another_host.com')
// https://another_host.com/some/path
this.href('/some/path', { a: 'b', c: 'd' }, 'another_host.com')
// http://another_host.com/some/path?a=b&c=d
this.href('https://supplier_base_url.com/some/path')
// https://supplier_base_url.com/some/path
this.href(new URL('https://supplier_base_url.com/some/path'))
// https://supplier_base_url.com/some/path
this.href('/some/path', { a: 'b', c: 'd' })
// https://supplier_base_url.com/some/path?a=b&c=d
this.href('https://supplier_base_url.com/some/path', new URLSearchParams({ a: 'b', c: 'd' }))
// https://supplier_base_url.com/some/path?a=b&c=d
protected href(path: string | URL, params?: Maybe<RequestParams>, host?: string): string {
const href = new URL(path, this.baseURL);
if (host) {
href.host = host;
}
if (params && Object.keys(params).length > 0) {
href.search = this.buildSearchParams(params).toString();
}
return String(href);
}
ProtectedbuildRecursively serializes an object into URLSearchParams, encoding nested
objects with bracket notation (parent[child]=value). Fixes the case that
new URLSearchParams(obj) mishandles — it stringifies a nested object to
"[object Object]" — so a params object with nested filters serializes
correctly. Primitive values (and arrays, which serialize comma-joined like
URLSearchParams does) are appended directly.
The params object to serialize
The URLSearchParams to append into (defaults to a fresh instance)
The key prefix used while recursing into nested objects
The populated URLSearchParams
this.buildSearchParams({ q: "acid", filter: { size: "500g" } }).toString();
// "q=acid&filter%5Bsize%5D=500g" (i.e. filter[size]=500g)
protected buildSearchParams(
obj: Record<string, unknown>,
params: URLSearchParams = new URLSearchParams(),
prefix: string = '',
): URLSearchParams {
for (const [key, value] of Object.entries(obj)) {
// Bracket-nest the key as depth grows: parent[child][grandchild]...
const formKey = prefix ? `${prefix}[${key}]` : key;
if (isPopulatedObject(value)) {
this.buildSearchParams(value, params, formKey);
} else {
params.append(formKey, String(value));
}
}
return params;
}
ProtectedgetRetrieves product data with caching support. Similar to getProductData but allows for additional parameters to be included in the cache key.
The ProductBuilder instance to get data for
The function to use for fetching product data
Promise resolving to the updated ProductBuilder or void if fetch fails
const builder = new ProductBuilder<Product>(this.baseURL);
builder.setBasicInfo("Acetone", "/products/acetone", "ChemSupplier");
const updatedBuilder = await supplier.getProductDataWithCache(
builder,
async (b) => {
// Custom fetching logic
return b;
},
);
protected async getProductDataWithCache(
product: ProductBuilder<T>,
fetcher: (builder: ProductBuilder<T>) => Promise<ProductBuilder<T> | void>,
): Promise<ProductBuilder<T> | void> {
const url = product.get('url');
if (typeof url !== 'string') {
this.logger.error('[SupplierBase > getProductDataWithCache] Invalid URL in product:', {
url,
});
return undefined;
}
// Skip products the user has ignored (matched by identity key).
if (this.isExcluded(product)) {
this.logger.debug('[SupplierBase > getProductDataWithCache] Skipping excluded product', {
url,
supplierName: this.supplierName,
});
return undefined;
}
// Key by the supplier's stable identity, stamped on the builder at parse
// time. Absent only if a supplier failed to stamp one, in which case this
// product simply isn't cached.
const cacheKey = this.productIdentityKey(product);
this.logger.debug(
'[SupplierBase > getProductDataWithCache] Product detail cache key:',
cacheKey,
{
url,
},
);
try {
if (!this.skipProductDetailCache && cacheKey) {
const cachedData = await this.cache.getCachedProductData(cacheKey);
if (isCachedProductData<T>(cachedData)) {
product.setData(cachedData);
return product;
}
}
// Cache miss (or caching skipped): run setup (memoized) so any state
// subclasses rely on is ready before the fetcher reads it, then fetch.
await this.ensureSetup();
let resultBuilder: ProductBuilder<T> | void = undefined;
try {
resultBuilder = await fetcher(product);
} catch (err: unknown) {
this.logger.error(
'[SupplierBase > getProductDataWithCache] Error in product detail fetcher:',
err,
);
incrementParseError(this.supplierName);
return undefined;
}
if (resultBuilder) {
incrementProductCount(this.supplierName);
// Skip caching when the search was aborted (e.g. supplierSearchTimeBudgetSec) — the enrichment
// fetch was cancelled, so the data is incomplete and a later search should retry it.
if (
cacheKey &&
!this.skipProductDetailCache &&
!this.controller.signal.aborted &&
this.shouldCacheProductData(resultBuilder)
) {
await this.cache.cacheProductData(cacheKey, resultBuilder.dump());
}
}
return resultBuilder;
} catch (outerErr: unknown) {
this.logger.error(
'[SupplierBase > getProductDataWithCache] Error in getProductDataWithCache:',
outerErr,
);
incrementParseError(this.supplierName);
return undefined;
}
}
ProtectedproductThe key under which a product's detail-fetch failures are recorded/looked up: its permalink if set, otherwise its processing URL. Subclasses that fetch supplemental data should record failures under this same key so shouldCacheProductData can match them.
The product builder
The fetch key, or undefined when neither permalink nor url is a string
this.productFetchKey(builder); // "https://www.aladdinsci.com/us_en/x.html"
protected productFetchKey(product: ProductBuilder<T>): string | undefined {
const key = product.get('permalink') ?? product.get('url');
return typeof key === 'string' ? key : undefined;
}
ProtectedrecordRecords the HTTP status of a failed product-detail fetch so shouldCacheProductData can skip caching it. Non-HTTP failures (no status) are not recorded.
The product fetch key (see productFetchKey)
OptionalhttpStatus: numberThe HTTP status of the failure, if it was an HttpError
void
this.recordFetchFailure(permalink, 429);
protected recordFetchFailure(key: string, httpStatus?: number): void {
if (typeof httpStatus === 'number') {
this.failedFetchStatuses.set(key, httpStatus);
}
}
ProtectedshouldDecides whether a freshly-fetched product's detail data should be written to the cache. Skips caching when the product's last detail fetch failed with a noCacheStatusCodes status (default 429), so a later search retries it instead of serving the incomplete cached entry. The product is still listed regardless.
The product builder about to be cached
True to cache the product data, false to skip caching it
this.shouldCacheProductData(builder); // false when the detail fetch hit a 429
protected shouldCacheProductData(product: ProductBuilder<T>): boolean {
const key = this.productFetchKey(product);
if (key === undefined) {
return true;
}
const failedStatus = this.failedFetchStatuses.get(key);
return failedStatus === undefined || !this.noCacheStatusCodes.includes(failedStatus);
}
ProtectedgroupGroups variants of a product by their title
Array of product listings from search results
Array of product listings with grouped variants
Create a generic method for this, the same method is used in Synthetika and could be of use with LoudWolf.
const results = await this.queryProducts("sodium chloride");
const grouped = this.groupVariants(results);
// grouped is an array of product listings with grouped variants
protected groupVariants<R>(data: R[]): R[] {
const variants: GroupedItem<R>[] = data
.map((item) => {
const title = this.titleSelector(item);
if (!title) {
this.logger.error('No title found in product:', { item });
return undefined;
}
const groupId = stripQuantityFromString(title.replace(/(?<=\d{1,3})\s(?=\d{3})/g, ''));
const groupIdWithoutSpaces = groupId.replace(/[\s-]/g, '');
return { ...item, groupId: groupIdWithoutSpaces };
})
.filter((item): item is GroupedItem<R> => item !== undefined);
const products = Object.groupBy(variants, (item) => item.groupId);
return Object.values(products)
.filter((product): product is GroupedItem<R>[] => product !== undefined)
.map((product) => {
const main = product.splice(0, 1)[0];
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { groupId, ...newObject } = main;
// R is unconstrained, so TS widens newObject.variants to
// R["variants"] & (R[] | undefined), which it cannot prove product
// (GroupedItem<R>[]) satisfies; assert to the destructured property type.
newObject.variants = product as typeof newObject.variants;
return newObject;
})
.filter((item): item is GroupedItem<R> => item !== undefined);
}
ProtectedbackgroundRuns an HTTP request from the extension's background service worker instead of this
(page) context, sidestepping the CORS restrictions that apply to extension pages.
The target host must be granted in the manifest host_permissions. Returns a real
Response (text/JSON bodies only). Independent of fetch — it does not
share the request counter, hard limit, or WAF retry logic.
The absolute URL to request.
Optionalinit: BackgroundFetchInitOptional serializable request options (method, headers, body, etc.).
A Response reconstructed from the worker's reply.
// Inside a supplier method, e.g. scraping an asset blocked by page CORS:
const homepage = await this.backgroundFetch("https://chemsavers.com/");
const html = await homepage.text();
const search = await this.backgroundFetch("https://api.example.com/search", {
method: "POST",
headers: { "content-type": "application/json" },
body: JSON.stringify({ q: "acid" }),
});
const results = search.ok ? await search.json() : undefined;
protected async backgroundFetch(url: string, init?: BackgroundFetchInit): Promise<Response> {
this.logger.debug(`Background fetching: ${url}`);
return backgroundFetch(url, init);
}
ProtectedfetchInternal fetch method with request counting and decorator. Tracks request count and enforces hard limits on HTTP requests.
Arguments to pass to fetchDecorator (usually a Request or URL and options)
The response from the fetchDecorator
Error if request count exceeds hard limit
// Example usage inside a subclass:
const response = await this.fetch(new Request('https://example.com'));
if (response.ok) {
const data = await response.json();
console.log(data);
}
// With custom request options
const response = await this.fetch(
new Request('https://example.com', {
headers: { 'Accept': 'application/json' }
})
);
protected async fetch(
...args: Parameters<typeof fetchDecorator>
): Promise<FetchDecoratorResponse> {
const [input] = args;
this.logger.debug(`Fetching: ${input}`);
// One initial attempt plus up to `challengeRetryLimit` retries. A 403 from
// a WAF cookie handshake plants a cookie on the first hit (stored because
// credentials:"include"); the retry sends it back and usually passes.
const maxAttempts = 1 + Math.max(0, this.challengeRetryLimit);
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
// Each attempt is a real network request, so it counts toward the hard
// limit. For non-retrying suppliers (maxAttempts === 1) this is
// identical to the previous single increment.
this.requestCount++;
if (this.requestCount > this.httpRequestHardLimit) {
this.logger.warn('Request count exceeded hard limit', { requestCount: this.requestCount });
incrementFailure(this.supplierName);
throw new Error('Request count exceeded hard limit');
}
try {
const response = await fetchDecorator(...args);
this.logger.debug(`Response Status: ${response.status}`);
this.logger.debug('response hash:', response.requestHash);
if (typeof response.data === 'string' && response.data?.length === 0) {
throw new EmptyResponseError(`Invalid response: ${response.data}`);
}
incrementSuccess(this.supplierName);
return response;
} catch (error: unknown) {
if (this.shouldRetryChallenge(error) && attempt < maxAttempts) {
this.logger.warn('Retrying after 403 (WAF cookie handshake)', {
attempt,
maxAttempts,
input,
});
await sleep(this.challengeRetryDelayMs);
continue;
}
incrementFailure(this.supplierName);
throw error;
}
}
// Unreachable: the loop always returns on success or throws on the final
// failed attempt. Present only to satisfy the return-type checker.
throw new Error('fetch: exhausted retries without resolving');
}
ProtectedgetDerives the unique product key from a Macklin product variant: its item_id
(the same value passed to .setID), stable across the query→detail
transition.
The raw Macklin product variant
The product's item_id
this.getUniqueProductKey(product); // "12345"
protected getUniqueProductKey(data: MacklinProductVariant): string {
return String(data.item_id);
}
ProtectedsetupSets up the Macklin API client by:
void
MacklinApiError If the timestamp request fails or response is invalid
protected async setup(): Promise<void> {
await this.validateAndUpdateTimestamp();
if (!this.localStorage.soleId) {
this.localStorage.soleId = this.generateString(16, 16);
}
if (!this.localStorage.MklUserToken) {
this.localStorage.MklUserToken = '';
}
// Update headers to match api-client.js exactly, using defensive string conversion
this.headers = {
'X-Agent': 'web',
'X-User-Token': this.ensureStringHeader(this.localStorage.MklUserToken),
'X-Device-Id': this.ensureStringHeader(this.localStorage.soleId),
'X-Language': 'en',
'X-Timestamp': '',
};
}
ProtectedqueryQueries the Macklin API for products matching the search term. Handles the complex response structure where products are grouped by CAS number and may have multiple variants per CAS number.
The search term to find products
Maximum number of products to return (after fuzzy filtering a search limited to 90 results))
Array of ProductBuilder instances or void if the request fails
MacklinApiError if the API request fails or response is invalid
protected async queryProducts(
query: string,
limit: number = this.limit,
): Promise<ProductBuilder<Product>[] | void> {
this.limit = limit;
// Reset the per-search batches so a new query doesn't reuse the previous
// result set's product/list and product/info lookups.
this.productListBatch = undefined;
this.productInfoBatch = undefined;
const searchRequest: unknown = await this.request<MacklinSearchResultProducts>(
`/api/item/search`,
{
params: { keyword: query, limit: 90, page: 1 },
},
);
if (!isMacklinSearchResult<MacklinSearchResultProducts>(searchRequest)) {
this.logger.warn('Invalid API response format');
return;
}
// Flatten the array of arrays into a single array of products
const products = Object.values(searchRequest.list).map((item) => item[0]);
const fuzzFiltered = this.fuzzyFilterAst<MacklinProductVariant>(products);
this.logger.debug('fuzzFiltered:', { query, searchRequest, products, fuzzFiltered });
const processed = fuzzFiltered.slice(0, limit);
return this.initProductBuilders(processed);
}
ProtectedtitleExtracts the English name from a Macklin product variant. Used by the base class to display product titles.
The product variant to extract the title from
The English name of the product
protected titleSelector(data: MacklinProductVariant): string {
return data.item_en_name;
}
ProtectedgetFetches detailed product information from the Macklin API. This includes pricing, stock levels, and delivery information that isn't available in the search results.
The ProductBuilder instance to enrich with details
The enriched ProductBuilder or void if the request fails
MacklinApiError if the API request fails or response is invalid
protected async getProductData(
product: ProductBuilder<Product>,
): Promise<ProductBuilder<Product> | void> {
const itemCode = product.get('uuid');
return this.getProductDataWithCache(product, async (builder) => {
// Each endpoint is fetched as its own bounded batch shared across all
// products (list -> info -> sds), so the requests run in distinct phases
// rather than interleaving per product. Interleaving let some products'
// list calls get pushed to the very end of the burst, outside the
// server's signing window, where they were rejected.
// `/api/product/list` returns every pack-size variant of the product.
const listVariants = (await this.getProductListBatch()).get(itemCode);
// Keep only purchasable variants (in stock), then map each to a Variant.
// A product with no in-stock variant has nothing to show.
const variants = mapDefined(
(listVariants ?? [])
.filter((detail) => Number(detail.product_stock) > 0)
.sort((a, b) => Number(a.product_price) - Number(b.product_price)),
(detail) => {
const built = this.toVariant(detail);
return built ? { detail, variant: built } : undefined;
},
);
// Bail with `void` (not the half-built builder) when there's nothing
// usable. Returning the builder here would cache a price-less product
// (see getProductDataWithCache), poisoning the cache so the product stays
// broken on every later search. `void` skips the cache write and lets the
// next search retry.
if (variants.length === 0) {
this.logger.warn('No in-stock product/list variants for product:', itemCode);
return undefined;
}
// The cheapest in-stock variant is the product's headline price/size; the
// rest are attached as selectable variants.
const [primary] = variants;
builder.setPricing(primary.detail.product_price, 'CNY', CURRENCY_SYMBOL_MAP.CNY);
if (primary.variant.quantity != null && primary.variant.uom != null) {
builder.setQuantity(primary.variant.quantity, primary.variant.uom);
}
builder.setAvailability(AVAILABILITY.IN_STOCK);
builder.setDescription(primary.detail.item_en_specification);
builder.setVariants(variants.map(({ variant }) => variant));
// Molecular weight comes from `/api/product/info`.
const info = (await this.getProductInfoBatch()).get(itemCode);
builder.setMoleweight(info?.item.chem_mw);
// SDS lookup runs last — one request, and only for products that
// already have the minimum required data (a request spent on a product
// that will be dropped anyway is wasted).
if (isMinimalProduct(builder.dump())) {
builder.setSDSUrl(await this.sdsSearch(itemCode));
}
return builder;
});
}
PrivatetoConverts a single /api/product/list variant into a Partial<Variant>. The
pack size is parsed from product_code (formatted ${item_code}-${quantity},
e.g. I929937-100mg); the product-level product_id/product_code become
the variant's uuid/sku. Returns undefined when the quantity can't be
parsed, so the caller can skip it.
The product-list variant to convert
The variant, or undefined when the pack size can't be parsed
this.toVariant({ product_code: "I929937-100mg", item_code: "I929937", ... });
// -> { uuid: 94613254, sku: "I929937-100mg", title: "... 100mg", quantity: 100, uom: "mg", ... }
private toVariant(detail: MacklinProductDetails): Partial<Variant> | undefined {
const prefix = `${detail.item_code}-`;
const quantityLabel = detail.product_code.startsWith(prefix)
? detail.product_code.slice(prefix.length)
: detail.product_code;
const quantity = parseQuantity(quantityLabel);
if (!quantity) {
return undefined;
}
return {
id: detail.product_id,
uuid: detail.product_id,
sku: detail.product_code,
title: quantityLabel,
price: Number(detail.product_price),
currencyCode: 'CNY',
currencySymbol: CURRENCY_SYMBOL_MAP.CNY,
quantity: quantity.quantity,
uom: quantity.uom,
};
}
PrivategetFetches /api/product/list for every product in the current result set as
one bounded, memoized batch keyed by item code, storing the first (cheapest)
variant. Running the list calls as a single phase — rather than letting each
ride the per-product detail queue — keeps every product's list request in
the same burst window; previously the later products' list calls were pushed
to the end of the run (behind the info/SDS phases) and rejected by the
server. The batch is created once and shared.
A map of item code to its full list of product variants (only codes with a valid, non-empty list response are present)
const variants = (await this.getProductListBatch()).get("T819228");
// variants?.[0].product_price -> "30.00"
private async getProductListBatch(): Promise<Map<string, MacklinProductDetails[]>> {
if (!this.productListBatch) {
this.productListBatch = (async () => {
const codes = this.products
.map((product) => product.get('uuid'))
.filter((uuid): uuid is string => typeof uuid === 'string');
const queue = new Queue(this.maxConcurrentRequests, this.minConcurrentCycle);
const variantsByCode = new Map<string, MacklinProductDetails[]>();
await Promise.all(
codes.map((code) =>
queue.run(async () => {
const response = await this.request<MacklinProductDetails>('/api/product/list', {
params: { code },
});
if (isMacklinProductDetailsResponse(response) && response.list.length > 0) {
variantsByCode.set(code, response.list);
}
}),
),
);
return variantsByCode;
})();
}
return this.productListBatch;
}
PrivategetFetches a product's chemistry data from the product info endpoint
(POST /api/product/info, body { item_code }). Returns undefined rather
than throwing when the request fails or the response isn't the expected
shape, so a missing info payload never drops the product.
The product's item code
The validated product info, or undefined when unavailable
const info = await this.getProductInfo("P866188");
// info?.item.chem_mw -> "252.13"
private async getProductInfo(itemCode: string): Promise<MacklinProductInfo | undefined> {
try {
const data = await this.request<MacklinProductInfo>('/api/product/info', {
method: 'POST',
body: { item_code: itemCode },
});
return isMacklinProductInfo(data) ? data : undefined;
} catch (error: unknown) {
this.logger.debug('product/info fetch failed', { itemCode, error });
return undefined;
}
}
PrivategetFetches /api/product/info for every product in the current result set as
one bounded, memoized batch keyed by item code. Running the info calls as
their own phase (rather than interleaving a POST after each product's list
GET) keeps the number of concurrently in-flight signed requests inside the
browser's per-host connection limit — interleaved list+info bursts were
pushing the trailing requests outside the server's window, which rejected
them as "Signature failed". The batch is created once and shared, so every
per-product detail fetch awaits the same set of results.
A map of item code to its product info (only codes with a valid info payload are present)
const info = (await this.getProductInfoBatch()).get("T819228");
// info?.item.chem_mw -> "252.13"
private async getProductInfoBatch(): Promise<Map<string, MacklinProductInfo>> {
if (!this.productInfoBatch) {
this.productInfoBatch = (async () => {
const codes = this.products
.map((product) => product.get('uuid'))
.filter((uuid): uuid is string => typeof uuid === 'string');
const queue = new Queue(this.maxConcurrentRequests, this.minConcurrentCycle);
const infoByCode = new Map<string, MacklinProductInfo>();
await Promise.all(
codes.map((code) =>
queue.run(async () => {
const info = await this.getProductInfo(code);
if (info) {
infoByCode.set(code, info);
}
}),
),
);
return infoByCode;
})();
}
return this.productInfoBatch;
}
PrivatesdsLooks up a product's SDS (MSDS) document URL by its item code. The endpoint
returns code: 200 with data.url when a sheet exists, or an error code
with data: [] when it doesn't.
Example request URL: https://api.macklin.cn/api/msds/search?lang=en&keyword=P866188×tampe=...
The product's item code (the API's keyword param)
The SDS document URL, or undefined when none is available
const url = await this.sdsSearch("P866188");
// "https://www.macklin.cn/pdf/msds/download?lang=en&id=23884&..."
private async sdsSearch(itemCode: string): Promise<string | undefined> {
const response = await this.request<unknown>('/api/msds/search', {
params: { keyword: itemCode, lang: 'en' },
});
if (isMacklinMsdsSearchResponse(response)) {
return response.url;
}
this.logger.debug('No SDS document for item', { itemCode, response });
return undefined;
}
ProtectedinitCreates ProductBuilder instances from Macklin product variants. This is the final step in the product search process, converting the API response into a format that can be used by the rest of the application. Pricing, molecular weight, and the SDS URL are filled in later, per product, by getProductData.
Array of product variants to convert
Array of ProductBuilder instances
protected initProductBuilders(data: MacklinProductVariant[]): ProductBuilder<Product>[] {
return data.map((product) =>
new ProductBuilder(this.baseURL)
.setBasicInfo(
`${product.item_en_name}, ${product.item_specification}`,
`${this.baseURL}/en/products/${product.item_code}`,
this.supplierName,
)
.setMatchPercentage(this.matchScoreOf(product))
.setID(product.item_id)
.setCacheKey(this.getUniqueProductKey(product))
.setUUID(product.item_code)
.setCAS(product.cas)
.setSmiles(product.smile_code)
.setFormula(product.chem_mf)
.setImage(product.item_upimg)
.setConcentration(product.item_specification),
);
}
PrivategenerateGenerates a random string for use in device IDs and user tokens. Can operate in two modes:
Optionallength: numberOptional length for random string mode
OptionalcharSetSize: numberOptional size of character set to use
A random string
const string = this.generateString(20);
// "5Wf70hQ0y1akc8rTQ8ps"
const uuid = this.generateString();
// "550e8400-e29b-41d4-a716-446655440000"
private generateString(length?: number, charSetSize?: number): string {
const chars = '0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz';
const size = charSetSize || chars.length;
if (length) {
// Random string mode
return Array.from({ length }, () => chars[Math.floor(Math.random() * size)]).join('');
}
// UUID mode
const uuid = new Array(36).fill('');
uuid[8] = uuid[13] = uuid[18] = uuid[23] = '-';
uuid[14] = '4';
for (let i = 0; i < 36; i++) {
if (!uuid[i]) {
const random = Math.floor(16 * Math.random());
uuid[i] = chars[19 === i ? (3 & random) | 8 : random];
}
}
return uuid.join('');
}
PrivatesignStep 1 & 2: Signature Generation Implements the core signature generation process by:
Request headers to sign
Request parameters to sign
The final request signature
private signRequest(headers: MacklinRequestHeaders, params: RequestParams): string {
// Sort and filter headers exactly like api-client.js
const headerString =
Object.entries(headers)
.filter(([key, value]) => {
return (
key !== 'Content-Type' && value !== '' && value != null && typeof value !== 'object'
);
})
.sort(([a], [b]) => a.toLowerCase().localeCompare(b.toLowerCase()))
.map(([key, value]) => `${key.toLowerCase()}=${value}`)
.join('&') + `&salt=${this.SALT}`;
// Sort and filter params exactly like api-client.js
const paramString =
Object.entries(params)
.filter(([, value]) => {
return value !== '' && value != null && typeof value !== 'object';
})
.sort(([a], [b]) => a.toLowerCase().localeCompare(b.toLowerCase()))
.map(([key, value]) => `${key}=${String(value).trim()}`)
.join('&') + `&salt=${this.SALT}`;
// Debug logging to match api-client.js
this.logger.debug('Headers for signing:', headers);
this.logger.debug('Params for signing:', params);
this.logger.debug('Header string:', headerString.toLowerCase());
this.logger.debug('Param string:', paramString.toLowerCase());
const headerHash = md5(headerString.toLowerCase());
const paramHash = md5(paramString.toLowerCase());
const finalSignature = headerHash + paramHash;
this.logger.debug('Header hash:', headerHash);
this.logger.debug('Param hash:', paramHash);
this.logger.debug('Final signature:', finalSignature);
return finalSignature;
}
PrivaterequestStep 3: Request Processing The main request handler that:
The API endpoint to call
Request configuration
The API response
ApiError If the request fails or response is invalid
TimeoutError If the request times out
private async request<T>(path: string, options: MacklinApiRequestOptions = {}): Promise<T> {
try {
if (!isTimestampStorage(this.localStorage.MklTmKey)) {
throw new Error('Missing or invalid timestamp in localStorage');
}
// X-Timestamp must track the *current* server time, not the value synced
// at setup. The server compares it against its own clock with a tight
// tolerance, and the frozen value goes stale across the batched phases
// (list -> info -> sds) — by the SDS phase (the last requests) the gap is
// large enough that the server rejects the call. Derive the current server
// time from the live clock plus the server/client drift measured at sync.
// This is equivalent to what the site does by re-fetching /api/timestamp
// right before each call, but without the extra round-trips.
const tm = this.localStorage.MklTmKey;
const timestamp = Math.round(Date.now() / 1000) + (tm.serverTm - tm.clientTm);
// Create a fresh headers object to avoid any potential array concatenation
const headers: MacklinRequestHeaders = {
'X-Agent': 'web',
'X-User-Token': this.ensureStringHeader(this.localStorage.MklUserToken),
'X-Device-Id': this.ensureStringHeader(this.localStorage.soleId),
'X-Language': 'en',
'X-Timestamp': this.ensureStringHeader(timestamp),
};
// Handle auth headers exactly like api-client.js
if (isAuthRequiredEndpoint(path)) {
headers['X-User-Token'] = this.ensureStringHeader(this.localStorage.MklUserToken);
}
// Handle language parameter exactly like api-client.js
if (options.params?.lang) {
headers['X-Language'] = this.ensureStringHeader(options.params.lang);
}
// Add any additional headers from options, ensuring they're strings
if (options.headers) {
Object.entries(options.headers).forEach(([key, value]) => {
headers[key] = this.ensureStringHeader(value);
});
}
// Add timestamp to request exactly like api-client.js
const requestTimestamp = this.generateRequestTimestamp();
const params: RequestParams = { ...options.params };
let body = options.body;
if (options.method === 'GET' || !options.method) {
params.timestampe = requestTimestamp;
} else if (body) {
body = { ...body, timestampe: requestTimestamp };
}
// Sign the request
const signature = this.signRequest(
headers,
options.method === 'GET' || !options.method ? params : (body ?? {}),
);
headers.sign = signature;
this.lastSignature = signature;
// Debug logging to match api-client.js
this.logger.debug('Full request URL:', this.href(path, params, this.apiURL));
this.logger.debug('Request headers:', headers);
this.logger.debug('Request params:', params);
this.logger.debug('Request body:', body);
// GET endpoints carry everything in the query string; POST/PUT/DELETE
// send a JSON body. Dispatch to the matching HTTP method. Use the
// lower-level helpers (not the *Json variants) so the raw Response — and
// its X-Ratelimit-* headers — is available before the body is parsed.
const httpResponse =
options.method && options.method !== 'GET'
? await this.httpPost({
path,
headers: { ...headers, 'Content-Type': 'application/json' },
params,
body: body ? JSON.stringify(body) : undefined,
host: this.apiURL,
})
: await this.httpGet({
path,
headers,
params,
host: this.apiURL,
});
if (!httpResponse) {
throw new MacklinApiError('No response from Macklin API');
}
this.trackRateLimit(httpResponse.headers);
const response: unknown = await httpResponse.json();
if (!isMacklinApiResponse<T>(response)) {
throw new MacklinApiError('Invalid API response format');
}
// Handle authentication errors exactly like api-client.js
if (isAuthCheckEndpoint(path) && response.code === 1005) {
throw new MacklinApiError('Authentication required');
}
// Anything other than 200 is a failure (e.g. 504 "Signature failed").
// The data payload is empty/invalid in that case; downstream typeguards
// drop it, but surface the failure here so it isn't silent.
if (response.code !== 200) {
this.logger.warn('Macklin API returned a non-success code', {
path,
response,
code: response.code,
message: response.message,
});
}
return response.data;
} catch (error) {
if (error instanceof TimeoutError) {
throw error;
}
if (error instanceof MacklinApiError) {
throw error;
}
throw new MacklinApiError('API request failed', undefined, undefined, error);
}
}
PrivatetrackTracks the Macklin API rate limit from a response's X-Ratelimit-Limit /
X-Ratelimit-Remaining headers. Logs a warning the first time usage crosses
50%, 75%, and 90%, and an error once the limit is exhausted. Each tier is
logged at most once per server window — the tracking resets when the
remaining count rises again (the window rolled over).
The response headers to read the rate-limit values from
this.trackRateLimit(httpResponse.headers);
// console.warn: "Macklin API rate limit: 90/360 remaining (75% used)"
private trackRateLimit(headers: Headers): void {
const limit = Number(headers.get('x-ratelimit-limit'));
const remaining = Number(headers.get('x-ratelimit-remaining'));
if (!Number.isFinite(limit) || !Number.isFinite(remaining) || limit <= 0) {
return;
}
// A rise in remaining means the server's rate-limit window reset; allow the
// tiers to be reported again.
if (remaining > this.rateLimitLastRemaining) {
this.rateLimitTierLogged = 0;
}
this.rateLimitLastRemaining = remaining;
const usedPct = ((limit - remaining) / limit) * 100;
const tier =
remaining <= 0 ? 100 : usedPct >= 90 ? 90 : usedPct >= 75 ? 75 : usedPct >= 50 ? 50 : 0;
if (tier <= this.rateLimitTierLogged) {
return;
}
this.rateLimitTierLogged = tier;
const message = `Macklin API rate limit: ${remaining}/${limit} remaining (${Math.round(usedPct)}% used)`;
if (tier === 100) {
this.logger.error(message);
} else {
this.logger.warn(message);
}
}
PrivatefetchStep 4.1: Server Timestamp Management Fetches and stores the server timestamp to maintain time synchronization. Called when local timestamp is missing or expired (over 800 seconds old, or 13 minutes).
The server timestamp
Add the timestamp to the chrome.storage.local
MacklinApiError If the timestamp request fails or response is invalid
private async fetchServerTimestamp(): Promise<number> {
const response = await this.httpGetJson({
path: `/api/timestamp`,
host: this.apiURL,
});
if (!isMacklinApiResponse<TimestampResponse>(response)) {
throw new MacklinApiError('Invalid API response format');
}
this.logger.debug('serverTimestamp response:', response);
const clientTime = Math.round(Date.now() / 1000);
const timestampData: TimestampStorage = {
serverTm: response.data.timestamp,
clientTm: clientTime,
};
this.localStorage.MklTmKey = timestampData;
return timestampData.serverTm;
}
Gets the spec sheet url. Should return: https://www.macklin.cn/pdf/specification/download?lang=en&item_code=S867696
PrivategenerateStep 4.2: Request Timestamp Generation Generates a unique timestamp for each request using either:
The timestamp string sent verbatim in both GET query strings and
POST bodies. It must stay a string: the site builds it as
Date.now() + signatureDigits (string concatenation), and the value is
40+ digits — doing real addition collapses it into scientific notation
(e.g. 3.9e+43), which the server rejects with "Signature failed".
private generateRequestTimestamp(): string {
if (this.lastSignature) {
// Mirrors the site's `Date.now() + t.join("")`: string concatenation of
// the current time and the previous signature's digits.
const digits = this.lastSignature.match(/\d+/g)?.join('') ?? '';
return `${Date.now()}${digits}`;
}
// First request (no prior signature): plain current time + small offset.
return String(Date.now() + Math.floor(Math.random()) + Math.ceil(Math.random()));
}
PrivatevalidateStep 4.3: Timestamp Validation and Update Manages the timestamp lifecycle:
The current valid timestamp as a string
private async validateAndUpdateTimestamp(): Promise<string> {
const currentTime = Math.round(Date.now() / 1000);
const storedTimestamp = isTimestampStorage(this.localStorage.MklTmKey)
? this.localStorage.MklTmKey
: null;
if (
!storedTimestamp ||
currentTime > storedTimestamp.clientTm + this.TIMESTAMP_REFRESH_THRESHOLD
) {
delete this.localStorage.MklTmKey;
return String(await this.fetchServerTimestamp());
}
return String(storedTimestamp.serverTm);
}
PrivateensureEnsures header values are always strings, handling arrays and null values. This prevents issues with header concatenation and type mismatches.
The header value to convert
A string representation of the value
this.ensureStringHeader(["value"]) // "value"
this.ensureStringHeader(null) // ""
this.ensureStringHeader(123) // "123"
private ensureStringHeader(value: unknown): string {
if (Array.isArray(value)) {
// If it's an array, take the first value
return String(value[0] || '');
}
return String(value || '');
}
Color used to visually tag this supplier's log output (and available for
charts/UI). Defaults to a stable palette color derived from the class name
via getSupplierColor, so no supplier has to set one. Override by
assigning a hex string in a subclass constructor (also call
this.logger.setColor(this.color) there to recolor the already-built logger).
Protected ReadonlyminThe minimum match percentage for a product to be considered a match.
Protected ReadonlyfuzzFuzz scorer used by fuzzyFilter to score each candidate's title against
the query. Any function from fuzzball with the
(str1, str2, opts?) => number shape works. Subclasses override this when
a supplier's title format needs a different scorer (e.g. a catalog that
pads titles with boilerplate might prefer partial_ratio). Defaults to
WRatio.
Overridable at runtime from userSettings.fuzzScorerOverride — see
setFuzzScorerOverride and fuzzyFilter below. The user's Advanced
settings selection wins over this subclass default when set.
Protected OptionalfuzzRuntime override resolved from userSettings.fuzzScorerOverride. When
set, fuzzyFilter uses this instead of this.fuzzScorer. Undefined
(the default) means "use whatever the supplier class picked". Mutated
by setFuzzScorerOverride so it can't be readonly.
Protected OptionalparsedParsed advanced-search query, set per-instance by SupplierFactory from the
user's input. When absent (e.g. a directly-constructed test supplier),
getAst lazily parses this.query instead.
Protected OptionalresolvedSMILES/structure query terms resolved to their chemical identifiers, keyed by
the raw search term. Resolved once per search by SupplierFactory (network-
bound, via NCI Cactus/PubChem) and shared with every supplier so none of them
re-resolve. Undefined when the query has no structure terms. Only suppliers
that filter by structure (currently Ambeed) read this; others ignore it.
Protected Static ReadonlysupportsSee supportsCAS.
ProtectedfuzzyRuntime flag resolved from userSettings.fuzzyFilteringDisabled. When true,
fuzzyFilterAst skips fuzzball scoring: plain queries return raw supplier
results and advanced queries are filtered only by the boolean predicate via
case-insensitive substring matching.
Protected ReadonlyfuzzyFuzzy strategy. When true (the default), fuzzyFilter/fuzzyFilterAst
rank candidates by fuzz score and keep them all (in score order) instead of dropping
anything below minMatchPercentage; the base search pipeline then caps the list
to limit, so the highest-scoring matches survive. This avoids dropping clear
matches whose ratio-style score falls under the cutoff purely because the title dwarfs
the query. Set to false on a supplier to restore the hard-cutoff behavior.
Protected ReadonlymaxMaximum number of backend search requests the keyword-only fallback issues.
Protected ReadonlysupportsWhether queryProducts handles an advanced (boolean) query natively in a
single request — true for suppliers that translate the AST into a server-side
query (Wix, Shopify, Chemsavers/Typesense, LiMac/FreeFind). When false (the
default), queryProductsWithCache drives the keyword-only fallback:
one backend search per positive OR-group, unioned and deduped, with the full
boolean predicate enforced client-side by fuzzyFilterAst.
Protected ReadonlyskipOpt-out flag for the per-product detail cache. Left false (the default),
every supplier caches its per-product detail data — the safe default, since
forgetting to set this just yields harmless redundant caching, never a
silent cache regression.
Set true only on a supplier that resolves every field in the initial
search (a passthrough getProductData with no per-product fetch): for those
the per-product cache saves nothing, so
getProductData/getProductDataWithCache/partitionForBatch
skip the product-detail cache read+write. The query cache still serves
repeat searches, and getUniqueProductKey is still used for
exclusions. Mark the concrete pure-search supplier (not a shared base
class), so a base's fetching subclass keeps caching by default.
Optional ReadonlyebayThe supplier's eBay storefront. Subclasses that list "ebayonly" in
SupplierBase.paymentMethods must override this; see
src/suppliers/__tests__/storeOnlyPaymentMethods.test.ts, which enforces the pairing that
TypeScript can't express.
Optional ReadonlyamazonThe supplier's Amazon storefront. Required alongside "amazononly", exactly as
ebayStoreURL is for "ebayonly".
Protected Static Optional ReadonlyshipsThe countries to which the supplier ships.
public readonly shipsTo: CountryCode[] = ["US", "CN", "NL"];
protected static readonly shipsTo?: CountryCode[];
ProtectedqueryString to query for (product name, CAS, etc.). The search term that will be used to find products. Set during construction and used throughout the supplier's lifecycle.
ProtectedbaseThe base search parameters that are always included in search requests. These parameters are merged with any additional search parameters when making requests to the supplier's API.
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
this.baseSearchParams = {
format: "json",
version: "2.0"
};
}
}
protected baseSearchParams: Record<string, string | number> = {};
ProtectedcontrollerThe AbortController instance used to manage and cancel ongoing requests. This allows for cancellation of in-flight requests when needed, such as when a new search is started or the supplier is disposed.
const controller = new AbortController();
const supplier = new MySupplier("acetone", 5, controller);
// Later, to cancel all pending requests:
controller.abort();
protected controller: AbortController;
ProtectedlimitThe maximum number of results to return for a search query. This is not a limit on HTTP requests, but rather the number of products that will be returned to the caller.
const supplier = new MySupplier("acetone", 5); // Limit to 5 results
for await (const product of supplier) {
// Will yield at most 5 products
}
protected limit: number;
ProtectedproductsThe products that are currently being built by the supplier. This array holds ProductBuilder instances that are in the process of being transformed into complete Product objects.
await supplier.queryProducts("acetone");
console.log(`Building ${supplier.products.length} products`);
for (const builder of supplier.products) {
const product = await builder.build();
console.log("Built product:", product.title);
}
protected products: ProductBuilder<T>[] = [];
ProtectedhttpMaximum number of HTTP requests allowed per search query. This is a hard limit to prevent excessive requests to the supplier's API. If this limit is reached, the supplier will stop making new requests.
50
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
this.httpRequestHardLimit = 100; // Allow more requests
}
}
protected httpRequestHardLimit: number = 50;
ProtectedrequestCounter for HTTP requests made during the current query execution. This is used to track the number of requests and ensure we don't exceed the httpRequestHardLimit.
0
await supplier.queryProducts("acetone");
console.log(`Made ${supplier.requestCount} requests`);
if (supplier.requestCount >= supplier.httpRequestHardLimit) {
console.log("Reached request limit");
}
protected requestCount: number = 0;
ProtectedmaxNumber of requests to process in parallel when fetching product details. This controls the batch size for concurrent requests to avoid overwhelming the supplier's API and the user's bandwidth.
10
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
// Process 5 requests at a time
this.maxConcurrentRequests = 5;
}
}
protected maxConcurrentRequests: number = 3;
ProtectedminMinimum number of milliseconds between two consecutive tasks
protected minConcurrentCycle: number = 100;
ProtectedsupplierMaximum wall-clock time (in seconds) a single supplier's execute() search may run.
Once exceeded, any in-flight and pending product-detail requests are aborted and the search
stops yielding new products — only those already collected are returned. Measured from the
start of execute(), so it also bounds a slow initial query. Defaults to
search.supplierSearchTimeBudgetSec from config.json; set to 0 to disable the limit.
Override per supplier for sources that are slow or rate-limit-prone.
search.supplierSearchTimeBudgetSec (config.json)
protected supplierSearchTimeBudgetSec: number = search.supplierSearchTimeBudgetSec;
Protected ReadonlyrequiredCookies that must be written into the browser jar before any request runs
— e.g. a currency or session-preference cookie the backend reads. Seeded
once per instance by ensureSetup (before setup) via chrome.cookies,
since the Cookie request header is on the fetch-forbidden list and can't
be set through this.headers. Each entry's url defaults to baseURL.
Subclasses override this instead of hand-rolling a setup that calls
chrome.cookies.set directly.
[]
class MySupplier extends SupplierBase<Partial<Product>, Product> {
protected readonly requiredCookies: SupplierCookieSeed[] = [
{ name: "currency", value: "2" },
];
}
protected readonly requiredCookies: SupplierCookieSeed[] = [];
Protected ReadonlychallengeNumber of times fetch retries a request that comes back 403. Some
suppliers sit behind a WAF that 403s the first hit while planting a
session cookie (a "cookie handshake"); because every request now sets
credentials: "include", that cookie lands in the jar and the retry
carries it back, usually passing. We can't gate on the Set-Cookie
header (it's fetch-forbidden and invisible to JS), so this per-supplier
flag is the gate — 0 (the default) means never retry. Only enable it
for suppliers known to do this handshake.
0
protected readonly challengeRetryLimit: number = 0;
Protected ReadonlychallengeDelay in milliseconds between 403 challenge retries. Gives the WAF a
brief beat before re-requesting with the freshly-planted cookie.
300
protected readonly challengeRetryDelayMs: number = 300;
ProtectedloggerLogger for the supplier. Initialized in the constructor with the name of the inheriting class.
ProtectedproductDefault values for products. These will get overridden if they're found in the product data.
ProtectedcacheCache instance for this supplier.
Initialized after construction by initCache() (called from
SupplierFactory once supplierName is set). The ! assertion is safe
here because every code path that reads this.cache
(queryProductsWithCache, getProductData, getProductDataWithCache)
runs only after execute() is called on a factory-built instance, and the
factory always calls initCache() before handing the instance out.
ProtectednoHTTP status codes that, when hit while fetching a product's detail data, prevent that
product from being cached (see shouldCacheProductData). Mirrors
userSettings.noCacheStatusCodes; set by initCache. Defaults to [429].
Protected ReadonlyfailedMaps a product's fetch key (permalink, falling back to its processing URL) to the HTTP status of its last failed detail fetch. Populated by subclasses via recordFetchFailure and consulted by shouldCacheProductData. Per-search, since the factory builds a fresh supplier instance for each search.
ProtectedexcludedProduct-data cache keys the user has explicitly excluded via the "Ignore
Product" context menu action. Loaded once per execute() from
storage.local so membership checks are synchronous on the hot path
(see getProductData). Newly-ignored products take effect on the next
search, which matches the stated feature requirement.
Static ReadonlysupplierName of supplier (for display purposes)
Static ReadonlybaseBase URL for HTTP(s) requests
Static ReadonlyapiThe host of the Macklin API.
Static ReadonlyshippingShipping scope for Macklin
Static ReadonlycountryThe country code of the supplier.
Static ReadonlypaymentThe payment methods accepted by the supplier. Used to determine the payment methods accepted by the supplier.
ProtectedqueryOverride the type of queryResults to use our specific type
ProtectedhttpUsed to keep track of how many requests have been made to the supplier.
Private ReadonlyTIMESTAMP_Private ReadonlySALTThe salt used to sign requests.
Private ReadonlyDEFAULT_PrivatelocalLocal storage object for the Macklin API client.
Add the timestamp to the chrome.storage.local
private localStorage: Record<string, unknown> = {};
PrivatelastThe last signature used for the request
Private OptionalproductMemoized batch of /api/product/list lookups for the current result set,
keyed by item code. Created once on first access so every product's detail
fetch shares the same bounded list phase (see getProductListBatch).
Private OptionalproductMemoized batch of /api/product/info lookups for the current result set,
keyed by item code. Created once on first access so every product's detail
fetch shares the same bounded info phase (see getProductInfoBatch).
PrivaterateHighest rate-limit usage tier (50/75/90/100) already logged this window.
PrivaterateLast X-Ratelimit-Remaining seen; a rise signals the server window reset.
Protected Static ReadonlysupportsMacklin supports CAS numbers searches.
Protected Static ReadonlysupportsMacklin supports formula searches.
ProtectedheadersHTTP headers used as a basis for all queries.
Macklin is a Chineese based chemical supply company. This module handles API requests with custom authentication and request signing for every call.
Remarks
Macklins client side code is very different from the other platforms. Looking at the API structure and authentication pattern, this appears to be a custom implementation rather than a standard ecommerce platform or CMS. The
/api/timestampendpoint with the specific authentication flow (using device IDs, custom signing with salt, and timestamp synchronization) is not a common pattern in major platforms. The specific implementation (withMklTmKeyinlocalStorage, the saltndksyr9834@#$32ndsfu, and the custom MD5-like transformation) suggests this is a custom-built platform, likely Macklin's own ecommerce system, rather than a standard off-the-shelf solution.Request Signature Generation Process
The Macklin API requires a custom signature for each request to ensure authenticity. The signature is generated through the following steps:
Step 1: Header String Generation
Step 2: Parameter String Generation
Step 3: Final Signature
Step 4: Timestamp Handling
Storage units
All persistent state lives in three keys the site keeps in
localStorage(mirrored here by the in-memorylocalStoragefield). Each one feeds the request headers below:MklTmKey— the server/client clock sync, stored as{ serverTm, clientTm }.serverTmis the Unix timestamp returned byGET /api/timestamp;clientTmis the local clock (Date.now() / 1000) captured at the moment of that fetch. It is refetched when missing or older thanTIMESTAMP_REFRESH_THRESHOLD(800 s). TheX-Timestampheader is derived from it asnow + (serverTm - clientTm), so it tracks server time across the batched request phases without a/api/timestampround-trip before every call. Written byfetchServerTimestamp, validated byisTimestampStorage.soleId— a stable per-browser device identifier: a random 16-character string generated once (viagenerateString) and reused thereafter. Sent verbatim as theX-Device-Idheader.MklUserToken— the logged-in user's session token. The site sets it on login and leaves it empty ('') for an anonymous session. Sent as theX-User-Tokenheader. ChemPal only browses anonymously, so this is normally empty; the site itself attaches it only on the authenticated endpoints (seeAuthRequiredEndpoints).Request headers
x-device-id— copied from thesoleIdstorage key above.x-user-token— copied from theMklUserTokenstorage key above.x-timestamp— the current server time derived fromMklTmKeyabove.sign— the per-request signature (headerHash + paramHash) built by the process described above. Recomputed for every call and stashed inlastSignatureto seed the next request'stimestampe.Request parameter
timestampe— a per-request nonce (the API's own spelling — not a typo to "fix"). Carried in the GET query string or the POST body. The first request usesDate.now()plus a small random offset; every later request usesDate.now()string-concatenated with the digits pulled from the previous signature (seegenerateRequestTimestamp). It must stay a string: the value is 40+ digits, so numeric addition would collapse it into scientific notation and the server would reject the signature.Known Macklin API response codes/messages: Successful: 200 "查询成功" ("Query successful") "成功" ("success") "Success" "success" "获取地区列表成功" ("Successfully retrieved list of regions") "200 ok!" Error: 201 "请输入正确的项目号！" ("Please enter the correct project number!") 401 "The field is required."
504 "Signature expired"
"Signature failed"
"Signature invalid"
1005 "Please login and try again"
1114 "The product number does not exist, please enter the full product number!"
1108 "If necessary, please contact customer service: 4006238666!" (failed call to specification endpoint)
1202 ??
3002 ??
Example
Source