All host origin patterns required for this supplier to function.
Automatically includes baseURL and, if defined, apiURL. Used by the
factory to check chrome permissions before querying.
Creates a new instance of the supplier base class. Initializes the supplier with query parameters, request limits, and abort controller. Sets up logging and default product values.
The search term to query products for
The maximum number of results to return (default: 5)
Optionalcontroller: AbortControllerAbortController instance for managing request cancellation
// Create a supplier with default limit
const supplier = new MySupplier("sodium chloride", undefined, new AbortController());
// Create a supplier with custom limit
const supplier = new MySupplier("acetone", 10, new AbortController());
// Create a supplier and handle cancellation
const controller = new AbortController();
const supplier = new MySupplier("ethanol", 5, controller);
// Later, to cancel all pending requests:
controller.abort();
Initializes the cache for the supplier. This is called after construction to ensure supplierName is set.
The cache is initialized with the supplier's name and is used to store both query results and product data. This method should be called after the supplier's name is set to ensure proper cache key generation.
class MySupplier extends SupplierBase<Product> {
constructor() {
super("acetone", 5);
// supplierName is set here
this.initCache(); // Initialize cache after supplierName is set
}
}
public initCache(
enabled: boolean = true,
doNotCacheEmptyResults: boolean = false,
cacheTtlMinutes: number = 0,
): void {
this.cache = new SupplierCache(
this.supplierName,
this.constructor.name,
enabled,
doNotCacheEmptyResults,
cacheTtlMinutes,
);
}
Applies (or clears) a runtime override for the fuzz scorer. Driven by
userSettings.fuzzScorerOverride — when the user picks a scorer in the
Advanced drawer section, SupplierFactory calls this on each instance
so the choice takes effect uniformly across every supplier.
Silently ignores unknown names so an outdated / corrupted setting can't blow up the search flow — callers fall back to the subclass default.
Name of a scorer from FUZZ_SCORERS, or undefined to
clear the override and use the subclass default.
const supplier = new MySupplier("acetone", 5, controller);
supplier.setFuzzScorerOverride("token_set_ratio");
// fuzzyFilter now uses token_set_ratio regardless of MySupplier's default
supplier.setFuzzScorerOverride(undefined);
// back to MySupplier's default
public setFuzzScorerOverride(name: string | undefined): void {
if (isFuzzScorerName(name)) {
this.fuzzScorerOverride = FUZZ_SCORERS[name];
} else {
this.fuzzScorerOverride = undefined;
}
}
ProtectedsetupPlaceholder for any setup that needs to be done before the query is made. Override this in subclasses if you need to perform setup (e.g., authentication, token fetching).
A promise that resolves when the setup is complete.
await supplier.setup();
protected async setup(): Promise<void> {}
ProtectedhttpRetrieves HTTP headers from a URL using a HEAD request. Useful for checking content types, caching headers, and other metadata without downloading the full response.
The URL to fetch headers from
Promise resolving to the response headers or void if request fails
// Basic usage
const headers = await supplier.httpGetHeaders('https://example.com/product/123');
if (headers) {
console.log('Content-Type:', headers['content-type']);
}
// With error handling
try {
const headers = await supplier.httpGetHeaders('https://example.com/product/123');
if (headers) {
console.log('Headers:', headers);
}
} catch (err) {
console.error('Failed to fetch headers:', err);
}
protected async httpGetHeaders(url: string | URL): Promise<Maybe<HeadersInit>> {
const requestObj = new Request(this.href(url), {
signal: this.controller.signal,
headers: new Headers(this.headers),
referrer: this.baseURL,
referrerPolicy: "strict-origin-when-cross-origin",
body: null,
method: "HEAD",
mode: "cors",
credentials: "include",
});
try {
const httpResponse = await this.fetch(requestObj);
return Object.fromEntries(httpResponse.headers.entries()) satisfies HeadersInit;
} catch (error: unknown) {
if (error instanceof Error && error.name === "AbortError") {
this.logger.warn("Request was aborted", { error, signal: this.controller.signal });
this.controller.abort();
} else {
this.logger.error("Error received during fetch:", {
error,
signal: this.controller.signal,
});
}
return;
}
}
ProtectedhttpSends a POST request to the given URL with the given body and headers. Handles request setup, error handling, and response caching.
The request configuration options
Promise resolving to the Response object or void if request fails
// Basic POST request
const response = await supplier.httpPost({
path: '/api/v1/products',
body: { name: 'Test Chemical' }
});
// POST with custom headers
const response = await supplier.httpPost({
path: '/api/v1/products',
body: { name: 'Test Chemical' },
headers: {
'Authorization': 'Bearer token123',
'Content-Type': 'application/json'
}
});
// POST with custom host and params
const response = await supplier.httpPost({
path: '/api/v1/products',
host: 'api.example.com',
body: { name: 'Test Chemical' },
params: { version: '2' }
});
// Error handling
try {
const response = await supplier.httpPost({ path: '/api/v1/products', body: { name: 'Test' } });
if (response && response.ok) {
const data = await response.json();
console.log('Created:', data);
}
} catch (err) {
console.error('POST failed:', err);
}
protected async httpPost({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<Response>> {
const method = "POST";
const mode = "cors";
const referrer = this.baseURL;
const referrerPolicy = "strict-origin-when-cross-origin";
const signal = this.controller.signal;
const bodyStr = typeof body === "string" ? body : (JSON.stringify(body) ?? null);
const headersObj = new Headers({
...this.headers,
...headers,
});
const url = this.href(path, params, host);
const requestObj = new Request(url, {
signal,
headers: headersObj,
referrer,
referrerPolicy,
body: bodyStr,
method,
mode,
credentials: "include",
});
// Fetch the goods
const httpResponse = await this.fetch(requestObj);
if (!isHttpResponse(httpResponse) || !httpResponse.ok) {
const badResponse = await httpResponse.text();
this.logger.error("Invalid POST response: ", badResponse);
throw new TypeError(`Invalid POST response: ${httpResponse?.toString()}`);
}
return httpResponse;
}
ProtectedhttpSends a POST request and returns the response as a JSON object.
The parameters for the POST request.
The response from the POST request as a JSON object.
// Basic usage
const data = await supplier.httpPostJson({
path: '/api/v1/products',
body: { name: 'John' }
});
// With custom headers and error handling
try {
const data = await supplier.httpPostJson({
path: '/api/v1/products',
body: { name: 'John' },
headers: { 'Authorization': 'Bearer token123' }
});
if (data) {
console.log('Created:', data);
}
} catch (err) {
console.error('POST JSON failed:', err);
}
protected async httpPostJson({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<JsonValue>> {
const httpResponse = await this.httpPost({ path, host, body, params, headers });
if (!isJsonResponse(httpResponse) || !httpResponse.ok) {
this.logger.error("httpPostJson| Invalid POST response: ", {
httpResponse,
path,
host,
body,
params,
headers,
});
throw new TypeError(`httpPostJson| Invalid POST response: ${httpResponse}`);
}
return await httpResponse.json();
}
ProtectedhttpSends a POST request and returns the response as a HTML string.
The request configuration options
Promise resolving to the HTML response as a string or void if request fails
TypeError - If the response is not valid HTML content
// Basic usage
const html = await supplier.httpPostHtml({
path: '/api/v1/products',
body: { name: 'John' }
});
protected async httpPostHtml({
path,
host,
body,
params,
headers,
}: RequestOptions): Promise<Maybe<string>> {
const httpResponse = await this.httpPost({ path, host, body, params, headers });
if (!isHtmlResponse(httpResponse)) {
throw new TypeError(`httpPostHtml| Invalid POST response: ${httpResponse}`);
}
return await httpResponse.text();
}
ProtectedhttpSends a GET request to the given URL with the specified options. Handles request setup, error handling, and response caching.
The request configuration options
Promise resolving to the Response object or void if request fails
// Basic GET request
const response = await supplier.httpGet({
path: '/products/search',
params: { query: 'sodium chloride' }
});
// GET with custom headers
const response = await supplier.httpGet({
path: '/products/search',
headers: { 'Accept': 'application/json' }
});
// GET with custom host
const response = await supplier.httpGet({
path: '/products/search',
host: 'api.example.com',
params: { category: 'chemicals' }
});
// Error handling
try {
const response = await supplier.httpGet({ path: '/products/search' });
if (response && response.ok) {
const data = await response.json();
console.log('Products:', data);
}
} catch (err) {
console.error('GET failed:', err);
}
protected async httpGet({
path,
params,
headers,
host,
}: RequestOptions): Promise<Maybe<Response>> {
// Check if the request has been aborted before proceeding
if (this.controller.signal.aborted) {
this.logger.warn("Request was aborted before fetch", {
signal: this.controller.signal,
});
return;
}
const headersRaw = { ...this.headers };
Object.assign(headersRaw, {
accept: [
"text/html",
"application/xhtml+xml",
"application/xml;q=0.9",
"image/avif",
"image/webp",
"image/apng",
"*/*;q=0.8",
].join(","),
...(headers ?? {}),
});
const requestObj = new Request(this.href(path, params, host), {
signal: this.controller.signal,
headers: new Headers(headersRaw),
referrer: this.baseURL,
referrerPolicy: "no-referrer",
body: null,
method: "GET",
mode: "cors",
credentials: "include",
redirect: "follow",
});
try {
// Fetch the goods
const httpResponse = await this.fetch(requestObj.url, requestObj);
const responseHeaders = Object.fromEntries(
httpResponse.headers.entries(),
) satisfies HeadersInit;
this.logger.debug("responseHeaders:", responseHeaders);
this.logger.debug("responseHeaders.location:", responseHeaders.location);
return httpResponse;
} catch (error: unknown) {
if (error instanceof Error && error.name === "AbortError") {
this.logger.warn("Request was aborted", { error, signal: this.controller.signal });
this.controller.abort();
} else {
this.logger.error("Error received during fetch:", {
error,
signal: this.controller.signal,
});
}
return;
}
}
ProtectedfuzzyFilters an array of data using fuzzy string matching to find items that closely match a query string. Uses the WRatio algorithm from fuzzball for string similarity comparison.
The search string to match against
Array of data objects to search through
Minimum match percentage (0-100) for a match to be included (default: 55)
Array of matching data objects with added fuzzy match metadata
// Example with simple string array
const products = [
{ title: "Sodium Chloride", price: 29.99 },
{ title: "Sodium Hydroxide", price: 39.99 },
{ title: "Potassium Chloride", price: 19.99 }
];
const matches = this.fuzzyFilter("sodium chloride", products);
// Returns: [
// {
// title: "Sodium Chloride",
// price: 29.99,
// _fuzz: { score: 100, idx: 0 }
// },
// {
// title: "Sodium Hydroxide",
// price: 39.99,
// _fuzz: { score: 85, idx: 1 }
// }
// ]
// Example with custom minMatchPercentage
const strictMatches = this.fuzzyFilter("sodium chloride", products, 90);
// Returns only exact matches with score >= 90
// Example with different data structure
const chemicals = [
{ name: "NaCl", formula: "Sodium Chloride" },
{ name: "NaOH", formula: "Sodium Hydroxide" }
];
// Override titleSelector to use formula field
this.titleSelector = (data) => data.formula;
const formulaMatches = this.fuzzyFilter("sodium chloride", chemicals);
protected fuzzyFilter<X>(
query: string,
data: X[],
minMatchPercentage: number = this.minMatchPercentage,
): X[] {
// User's Advanced-settings override wins over the subclass default.
const activeScorer = this.fuzzScorerOverride ?? this.fuzzScorer;
// console.log(
// `[fuzzyFilter] ${this.supplierName} query="${query}" — scorer comparison (cutoff=${minMatchPercentage})`,
// );
if (IS_DEV_BUILD) {
this.showFuzzScorerComparisonTable(query, data);
}
const results = extract(query, data, {
scorer: activeScorer,
processor: this.titleSelector,
cutoff: minMatchPercentage,
sortBySimilarity: true,
}).reduce<FuzzyMatchResult<X>[]>((acc, [obj, score, idx]) => {
if (score < minMatchPercentage) {
this.logger.debug("fuzzyFilter: score below minimum match percentage, excluding product", {
product: obj,
score,
idx,
minMatchPercentage: minMatchPercentage,
});
return acc;
}
// eslint-disable-next-line @typescript-eslint/naming-convention
acc[idx] = Object.assign(obj, { _fuzz: { score, idx }, matchPercentage: score });
return acc;
}, []);
this.logger.debug("[fuzzyFilter]", {
supplierName: this.supplierName,
query,
minMatchPercentage,
activeScorer,
results,
});
// Get rid of any empty items that didn't match closely enough
return results.filter((item) => !!item);
}
ProtectedhttpMakes an HTTP GET request and returns the response as a string. Handles request configuration, error handling, and HTML parsing.
The request configuration options
Promise resolving to the HTML response as a string or void if request fails
TypeError - If the response is not valid HTML content
// Basic GET request
const html = await this.httpGetHtml({
path: "/api/products",
params: { search: "sodium" }
});
// GET request with custom headers
const html = await this.httpGetHtml({
path: "/api/products",
headers: {
"Authorization": "Bearer token123",
"Accept": "text/html"
}
});
// GET request with custom host
const html = await this.httpGetHtml({
path: "/products",
host: "api.supplier.com",
params: { limit: 10 }
});
protected async httpGetHtml({
path,
params,
headers,
host,
}: RequestOptions): Promise<Maybe<string>> {
const httpResponse = await this.httpGet({ path, params, headers, host });
if (!isHtmlResponse(httpResponse)) {
throw new TypeError(`httpGetHtml| Invalid GET response: ${httpResponse}`);
}
return await httpResponse.text();
}
ProtectedhttpMakes an HTTP GET request and returns the response as parsed JSON. Handles request configuration, error handling, and JSON parsing.
The request configuration options
Promise resolving to the parsed JSON response or void if request fails
TypeError - If the response is not valid JSON content
// Basic GET request
const data = await supplier.httpGetJson({ path: '/api/products', params: { search: 'sodium' } });
// GET request with custom headers
const data = await supplier.httpGetJson({
path: '/api/products',
headers: {
'Authorization': 'Bearer token123',
'Accept': 'application/json'
}
});
// GET request with custom host
const data = await supplier.httpGetJson({
path: '/products',
host: 'api.supplier.com',
params: { limit: 10 }
});
// Error handling
try {
const data = await supplier.httpGetJson({ path: '/api/products' });
if (data) {
console.log('Products:', data);
}
} catch (error) {
console.error('Failed to fetch products:', error);
}
protected async httpGetJson({
path,
params,
headers,
host,
}: RequestOptions): Promise<Maybe<JsonValue>> {
const httpRequest = await this.httpGet({ path, params, headers, host });
if (!isJsonResponse(httpRequest)) {
const badResponse = isHttpResponse(httpRequest) ? await httpRequest.text() : undefined;
this.logger.error("Invalid HTTP GET JSON response:", {
badResponse,
httpRequest,
path,
params,
headers,
host,
});
return;
}
return await httpRequest.json();
}
ProtectedqueryExecutes a product search query with caching support. First checks the cache for existing results, then falls back to the actual query if needed. The limit parameter is only used for the actual query and doesn't affect caching.
The search term to query products for
The maximum number of results to return (defaults to instance limit)
Promise resolving to array of product builders or void if search fails
// Basic usage with default limit
const results = await supplier.queryProductsWithCache("acetone");
if (results) {
console.log(`Found ${results.length} products`);
}
// With custom limit
const results = await supplier.queryProductsWithCache("acetone", 10);
if (results) {
for (const builder of results) {
const product = await builder.build();
console.log(product.title, product.price);
}
}
protected async queryProductsWithCache(
query: string,
limit: number = this.limit,
): Promise<ProductBuilder<T>[] | void> {
// Check cache first (processed product data)
this.logger.debug(
"queryProductsWithCache: called for",
this.supplierName,
"query:",
query,
"limit:",
limit,
);
const key = this.cache.generateCacheKey(query);
const cached = await this.cache.getCachedQueryEntry(key);
this.logger.debug("queryProductsWithCache: cache hit:", !!cached, "key:", key);
if (cached) {
// If the cached limit is less than the requested limit, invalidate the cache
if (
typeof cached.__cacheMetadata.limit === "number" &&
cached.__cacheMetadata.limit < limit
) {
this.logger.debug("Invalidating query cache due to insufficient limit", {
cachedLimit: cached.__cacheMetadata.limit,
requestedLimit: limit,
});
await deleteSupplierQueryCacheEntry(key);
} else {
this.logger.debug("Returning cached query results");
// Re-initialize product builders from cached processed data
return ProductBuilder.createFromCache<T>(this.baseURL, cached.data.slice(0, limit));
}
}
// If not in cache, perform the actual query. Run setup first so any
// subclass state it mutates (headers, localStorage, tokens, etc.) is
// in place before `queryProducts` reads it. Memoized, so this is cheap
// on repeat calls within the same supplier instance.
await this.ensureSetup();
const results = await this.queryProducts(query, limit);
if (results) {
// Store processed results in cache (dumped/serialized form) and the limit used
await this.cache.cacheQueryResults(
query,
results.map((b) => b.dump()),
limit,
);
}
return results;
}
Executes the supplier's search query and returns the results. This method will execute all results concurrently (to the limits set in the supplier class), and resolve to an array of product objects.
Promise resolving to an array of products
This method is used to execute the supplier's search query and return the results.
public async *execute(): AsyncGenerator<T, void, undefined> {
// setup() is not called eagerly here — it's run lazily from the
// phase-boundary gates inside `queryProductsWithCache` and
// `getProductData` / `getProductDataWithCache`. A fully cached search
// never reaches those gates, so setup's token/cookie/permission
// requests are skipped entirely.
// Snapshot the user's ignore list once per search. Any product whose
// exclusion key matches an entry here is dropped before the detail phase
// runs (see the filter after queryProductsWithCache below).
this.excludedProductKeys = await loadExcludedProductKeys();
// Over-fetch by the number of previously-ignored products belonging to
// this supplier so that, in the worst case where every ignored product
// appears in the top of the query result set, we still end up with
// `this.limit` survivors after filtering. The queryProductsWithCache
// cache invalidates itself when the requested limit exceeds the cached
// limit, so this is safe.
const excludedForSupplier = await countExcludedProductsForSupplier(this.supplierName);
const fetchLimit = this.limit + excludedForSupplier;
incrementSearchQueryCount(this.supplierName);
const results = await this.queryProductsWithCache(this.query, fetchLimit);
if (!results || results.length === 0) {
this.logger.log(`No query results found`);
return;
}
// Drop any products the user has ignored, then slice back down to the
// user-visible limit. Uses the same key shape as getProductData so
// whichever side catches the exclusion first, the check is consistent.
const survivors: ProductBuilder<T>[] = [];
for (const builder of results) {
if (survivors.length >= this.limit) break;
const rawUrl = builder.get("url");
if (typeof rawUrl !== "string") {
survivors.push(builder);
continue;
}
const exclusionUrl = this.href(rawUrl);
const exclusionKey = getProductExclusionKey(exclusionUrl, this.supplierName);
if (this.excludedProductKeys.has(exclusionKey)) {
this.logger.debug("Skipping excluded product (pre-detail)", {
url: rawUrl,
exclusionUrl,
exclusionKey,
});
continue;
}
survivors.push(builder);
}
this.products = survivors;
const queue = new Queue(this.maxConcurrentRequests, this.minConcurrentCycle);
// Create an array of promises, each yielding a product as soon as it's ready
const tasks = this.products.map((product) =>
queue.run(async () => {
try {
this.logger.debug(`Product data for ${this.supplierName}:`, product);
const builder = await this.getProductData(product);
if (!builder) return;
this.logger.debug(`Builder data for ${this.supplierName}:`, builder);
const finished = await this.finishProduct(builder);
this.logger.debug(`Finished product data for ${this.supplierName}:`, finished);
if (finished) {
return finished;
}
} catch (e: unknown) {
this.logger.error("Error processing product", { error: e, product });
incrementParseError(this.supplierName);
}
}),
);
// As each promise resolves, yield the product
const resultsSet = new Set(tasks);
while (resultsSet.size > 0) {
const finished = await Promise.race(resultsSet);
// Remove the finished promise from the set
for (const t of resultsSet) {
if ((await Promise.resolve(t)) === finished) {
resultsSet.delete(t);
break;
}
}
if (finished) {
yield finished;
}
}
}
ProtectedfinishFinalizes a partial product by adding computed properties and validating the result. This method:
The ProductBuilder instance containing the partial product to finalize
Promise resolving to a complete Product object or void if validation fails
// Example with a valid partial product
const builder = new ProductBuilder<Product>(this.baseURL);
builder
.setBasicInfo("Sodium Chloride", "/products/nacl", "ChemSupplier")
.setPricing(29.99, "USD", "$")
.setQuantity(500, "g");
const finishedProduct = await this.finishProduct(builder);
if (finishedProduct) {
console.log("Finalized product:", {
title: finishedProduct.title,
price: finishedProduct.price,
quantity: finishedProduct.quantity,
uom: finishedProduct.uom,
usdPrice: finishedProduct.usdPrice,
baseQuantity: finishedProduct.baseQuantity
});
}
// Example with an invalid partial product
const invalidBuilder = new ProductBuilder<Product>(this.baseURL);
invalidBuilder.setBasicInfo("Sodium Chloride", "/products/nacl", "ChemSupplier");
// Missing required fields
const invalidProduct = await this.finishProduct(invalidBuilder);
if (!invalidProduct) {
console.log("Failed to finalize product - missing required fields");
}
protected async finishProduct(product: ProductBuilder<T>): Promise<Maybe<T>> {
if (!isMinimalProduct(product.dump())) {
this.logger.warn("Unable to finish product - Minimum data not set", { product });
return;
}
// Set the country and shipping scope of the supplier
// have different restrictions on different products or countries.
product.setSupplierCountry(this.country);
product.setSupplierShipping(this.shipping);
if (this.paymentMethods.length > 0) {
product.setSupplierPaymentMethods(this.paymentMethods);
}
const built = await product.build();
return built;
}
ProtectedhrefTakes in either a relative or absolute URL and returns an absolute URL. This is useful for when you aren't sure if the link (retrieved from parsed text, a setting, an element, an anchor value, etc) is absolute or not. Using relative links will result in http://chrome-extension://... being added to the link.
URL object or string
Optionalparams: Maybe<RequestParams>The parameters to add to the URL.
Optionalhost: stringThe host to use for overrides (eg: needing to call a different host for an API)
absolute URL
this.href('/some/path')
// https://supplier_base_url.com/some/path
this.href('https://supplier_base_url.com/some/path', null, 'another_host.com')
// https://another_host.com/some/path
this.href('/some/path', { a: 'b', c: 'd' }, 'another_host.com')
// http://another_host.com/some/path?a=b&c=d
this.href('https://supplier_base_url.com/some/path')
// https://supplier_base_url.com/some/path
this.href(new URL('https://supplier_base_url.com/some/path'))
// https://supplier_base_url.com/some/path
this.href('/some/path', { a: 'b', c: 'd' })
// https://supplier_base_url.com/some/path?a=b&c=d
this.href('https://supplier_base_url.com/some/path', new URLSearchParams({ a: 'b', c: 'd' }))
// https://supplier_base_url.com/some/path?a=b&c=d
protected href(path: string | URL, params?: Maybe<RequestParams>, host?: string): string {
const href = new URL(path, this.baseURL);
if (host) {
href.host = host;
}
if (params && Object.keys(params).length > 0) {
href.search = new URLSearchParams(
Object.entries(params).reduce<QueryParams>((acc, [key, value]) => {
acc[key] = String(value);
return acc;
}, {}),
).toString();
}
return href.toString();
}
ProtectedgetRetrieves product data with caching support. Similar to getProductData but allows for additional parameters to be included in the cache key.
The ProductBuilder instance to get data for
The function to use for fetching product data
Optionalparams: QueryParamsOptional parameters to include in the cache key
Promise resolving to the updated ProductBuilder or void if fetch fails
const builder = new ProductBuilder<Product>(this.baseURL);
builder.setBasicInfo("Acetone", "/products/acetone", "ChemSupplier");
// Use custom fetcher with additional params
const updatedBuilder = await supplier.getProductDataWithCache(
builder,
async (b) => {
// Custom fetching logic
return b;
},
{ version: "2.0" }
);
protected async getProductDataWithCache(
product: ProductBuilder<T>,
fetcher: (builder: ProductBuilder<T>) => Promise<ProductBuilder<T> | void>,
params?: QueryParams,
): Promise<ProductBuilder<T> | void> {
const url = product.get("url");
if (typeof url !== "string") {
this.logger.error("Invalid URL in product:", { url });
return undefined;
}
// See getProductData above: normalize only for the exclusion key so it
// lines up with the absolute URL the UI context menu stores.
const exclusionUrl = this.href(url);
const shouldExclude = await shouldExcludeProduct(exclusionUrl, this.supplierName);
if (shouldExclude) {
this.logger.debug("Skipping excluded product", {
url,
exclusionUrl,
supplierName: this.supplierName,
});
return undefined;
}
const cacheKey = this.cache.getProductDataCacheKey(url, params);
this.logger.debug("[SupplierBase] Product detail cache key:", cacheKey, "for url:", url);
try {
const cachedData = await this.cache.getCachedProductData(cacheKey);
if (cachedData) {
// Safe: cache only stores values previously produced by ProductBuilder<T>.dump(),
// so the round-tripped shape is structurally a Partial<T>.
product.setData(cachedData as Partial<T>);
return product;
}
// Cache miss: run setup (memoized) so any state subclasses rely on is
// ready before the fetcher reads it, then call the fetcher.
await this.ensureSetup();
let resultBuilder: ProductBuilder<T> | void = undefined;
try {
resultBuilder = await fetcher(product);
} catch (err: unknown) {
this.logger.error("Error in product detail fetcher:", err);
incrementParseError(this.supplierName);
return undefined;
}
if (resultBuilder) {
incrementProductCount(this.supplierName);
await this.cache.cacheProductData(cacheKey, resultBuilder.dump());
}
return resultBuilder;
} catch (outerErr: unknown) {
this.logger.error("Error in getProductDataWithCache:", outerErr);
incrementParseError(this.supplierName);
return undefined;
}
}
ProtectedgroupGroups variants of a product by their title
Array of product listings from search results
Array of product listings with grouped variants
Create a generic method for this, the same method is used in Synthetika and could be of use with LoudWolf.
const results = await this.queryProducts("sodium chloride");
const grouped = this.groupVariants(results);
// grouped is an array of product listings with grouped variants
protected groupVariants<R>(data: R[]): R[] {
const variants: GroupedItem<R>[] = data
.map((item) => {
const title = this.titleSelector(item);
if (!title) {
this.logger.error("No title found in product:", { item });
return undefined;
}
const groupId = stripQuantityFromString(title.replace(/(?<=\d{1,3})\s(?=\d{3})/g, ""));
const groupIdWithoutSpaces = groupId.replace(/[\s-]/g, "");
return { ...item, groupId: groupIdWithoutSpaces };
})
.filter((item): item is GroupedItem<R> => item !== undefined);
const products = Object.groupBy(variants, (item) => item.groupId);
return Object.values(products)
.filter((product): product is GroupedItem<R>[] => product !== undefined)
.map((product) => {
const main = product.splice(0, 1)[0];
// eslint-disable-next-line @typescript-eslint/no-unused-vars
const { groupId, ...newObject } = main;
newObject.variants = product as GroupedItem<R>["variants"];
return newObject;
})
.filter((item): item is GroupedItem<R> => item !== undefined);
}
ProtectedfetchInternal fetch method with request counting and decorator. Tracks request count and enforces hard limits on HTTP requests.
Arguments to pass to fetchDecorator (usually a Request or URL and options)
The response from the fetchDecorator
Error if request count exceeds hard limit
// Example usage inside a subclass:
const response = await this.fetch(new Request('https://example.com'));
if (response.ok) {
const data = await response.json();
console.log(data);
}
// With custom request options
const response = await this.fetch(
new Request('https://example.com', {
headers: { 'Accept': 'application/json' }
})
);
protected async fetch(
...args: Parameters<typeof fetchDecorator>
): Promise<FetchDecoratorResponse> {
const [input] = args;
this.logger.debug(`Fetching: ${input}`);
// One initial attempt plus up to `challengeRetryLimit` retries. A 403 from
// a WAF cookie handshake plants a cookie on the first hit (stored because
// credentials:"include"); the retry sends it back and usually passes.
const maxAttempts = 1 + Math.max(0, this.challengeRetryLimit);
for (let attempt = 1; attempt <= maxAttempts; attempt++) {
// Each attempt is a real network request, so it counts toward the hard
// limit. For non-retrying suppliers (maxAttempts === 1) this is
// identical to the previous single increment.
this.requestCount++;
if (this.requestCount > this.httpRequestHardLimit) {
this.logger.warn("Request count exceeded hard limit", { requestCount: this.requestCount });
incrementFailure(this.supplierName);
throw new Error("Request count exceeded hard limit");
}
try {
const response = await fetchDecorator(...args);
this.logger.debug(`Response Status: ${response.status}`);
this.logger.debug("response hash:", response.requestHash);
if (typeof response.data === "string" && response.data?.length === 0) {
throw new EmptyResponseError(`Invalid response: ${response.data}`);
}
incrementSuccess(this.supplierName);
return response;
} catch (error: unknown) {
if (this.shouldRetryChallenge(error) && attempt < maxAttempts) {
this.logger.warn("Retrying after 403 (WAF cookie handshake)", {
attempt,
maxAttempts,
input,
});
await new Promise((resolve) => setTimeout(resolve, this.challengeRetryDelayMs));
continue;
}
incrementFailure(this.supplierName);
throw error;
}
}
// Unreachable: the loop always returns on success or throws on the final
// failed attempt. Present only to satisfy the return-type checker.
throw new Error("fetch: exhausted retries without resolving");
}
ProtectedqueryExecutes a product search query and returns matching products
Search term to look for
The maximum number of results to query for
Promise resolving to array of product objects or void if search fails
// Search for sodium chloride with a limit of 10 results
const products = await this.queryProducts("sodium chloride", 10);
if (products) {
console.log(`Found ${products.length} products`);
for (const product of products) {
const builtProduct = await product.build();
console.log(builtProduct.title, builtProduct.price);
}
} else {
console.log("No products found or search failed");
}
protected async queryProducts(
query: string,
limit: number = this.limit,
): Promise<ProductBuilder<Product>[] | void> {
/* The code snippet is performing the following actions: */
const params = {
limit: 50,
page: 1,
};
const products: SynthetikaProduct[] = [];
// Iterate through the pages to collect all of the products in the search results, but
// limit it to something reasonable (5 pages, for now)
for (let i = 1; i <= 5; i++) {
const pageResponse = await this.httpGetJson({
path: `/webapi/front/en_US/products/usd/search/${urlencode(query)}`,
params: { ...params, page: i },
});
assertIsSynthetikaSearchResponse(pageResponse);
// Add these to the products array, but filter out the non-whitelisted categories first
products.push(
...pageResponse.list.filter((product) =>
this.includeCategories.includes(product.category.id),
),
);
if (pageResponse.pages <= i) break;
}
const fuzzFiltered = this.fuzzyFilter<SynthetikaProduct>(query, products);
const grouped = this.groupVariants<SynthetikaProduct>(fuzzFiltered);
return this.initProductBuilders(grouped);
}
ProtectedtitleSelects the title/name of a product from the search response
Product object from search response
Title or name of the product
protected titleSelector(data: SynthetikaProduct): string {
return data.name;
}
ProtectedavailabilityConverts the availability string from Synthetika to an AVAILABILITY enum value. I wrote a bash script that iterated over their collections, then the products in those collections, getting the availability for each. I only found the below:
Availability string from Synthetika
AVAILABILITY enum value
const availability = "na wyczerpaniu";
const availabilityEnum = availabilityConverter(availability);
console.log(availabilityEnum);
protected availabilityConverter(availability: string): AVAILABILITY {
switch (availability.toLowerCase().trim()) {
case "na wyczerpaniu":
return AVAILABILITY.LIMITED_STOCK;
case "large quantity":
case "średnia ilość":
return AVAILABILITY.IN_STOCK;
case "tymczasowo niedostępny":
return AVAILABILITY.UNAVAILABLE;
case "nie ma w sprzedaży":
return AVAILABILITY.OUT_OF_STOCK;
default:
this.logger.warn("Unknown availability - Defaulting to UNKNOWN", { availability });
return AVAILABILITY.UNKNOWN;
}
}
ProtectedinitInitialize product builders from Synthetika search response data. Transforms product listings into ProductBuilder instances, handling:
Array of product listings from search results
Array of ProductBuilder instances initialized with product data
protected initProductBuilders(
data: SynthetikaProduct[],
): ProductBuilder<Product & { variants?: Variant[] }>[] {
return mapDefined(data, (product) => {
const productBuilder = new ProductBuilder(this.baseURL);
const availability = this.availabilityConverter(product.availability.name);
if (
[AVAILABILITY.LIMITED_STOCK, AVAILABILITY.IN_STOCK].includes(availability) === false ||
product.can_buy === false
) {
this.logger.warn("Product not in stock - Skipping", {
availability,
product,
can_buy: product.can_buy,
});
return;
}
productBuilder
.setBasicInfo(product.name, product.url, this.supplierName)
.setDescription(product.shortDescription)
.setID(product.id)
.setAvailability(this.availabilityConverter(product.availability.name))
.setSku(product.code)
.setUUID(product.code);
const quantity = firstMap(parseQuantity, [
product.name,
product.description,
product.weight.weight,
]);
if (quantity) productBuilder.setQuantity(quantity);
const price = parsePrice(product.price.gross.final);
if (price) productBuilder.setPricing(price.price, price.currencyCode, price.currencySymbol);
if (Array.isArray(product.variants) && product.variants.length > 0) {
productBuilder.setVariants(
product.variants.map((v) => {
const price = parsePrice(v.price?.gross.final ?? "");
return {
title: v.name,
price: price?.price ?? 0,
quantity: parseQuantity(v.name ?? "")?.quantity ?? 0,
uom: parseQuantity(v.name ?? "")?.uom ?? "",
url: v.url,
};
}),
);
}
return productBuilder;
});
}
ProtectedparseParses the description HTML and returns a record of properties
The description HTML to parse
A record of properties
protected parseDescriptionHTML(descriptionHTML: string): Record<string, string> {
const descriptionDOM = createDOM(descriptionHTML);
const properties: Record<string, string> = {};
// Find all <strong> elements that end with ":"
descriptionDOM.querySelectorAll("strong").forEach((strong) => {
const label = strong.textContent?.trim();
if (!label || !label.endsWith(":")) return;
const key = label.slice(0, -1); // strip the colon
// The value is the text that follows the <strong> inside its parent
const parent = strong.parentElement;
let value = "";
// Walk sibling nodes after the <strong>
let node = strong.nextSibling;
while (node) {
if (node.nodeType === Node.TEXT_NODE) {
value += node.textContent;
} else if (node.nodeName === "BR") {
break; // stop at line break — next field starts here
} else {
value += node.textContent;
}
node = node.nextSibling;
}
properties[key] = value.trim();
});
return properties;
}
ProtectedgetProcess the product data and return a ProductBuilder instance
The ProductBuilder instance to process
Promise resolving to a ProductBuilder instance or void
protected async getProductData(
product: ProductBuilder<Product & { variants?: Variant[] }>,
): Promise<ProductBuilder<Product> | void> {
console.log("[synthetika] getProductData init", { product });
return this.getProductDataWithCache(product, async (builder) => {
if (builder instanceof ProductBuilder === false) {
this.logger.warn("Invalid product object - Expected ProductBuilder instance:", {
builder,
product,
});
return;
}
const productURL = this.href(`/webapi/front/en_US/products/usd/${builder.get("id")}`);
const productResponse = await this.httpGetJson({
path: productURL,
});
console.log("[synthetika] productResponse", {
builder,
product,
productURL,
productResponse,
});
// Run the minimal check first, so if that fails we can bail early.
if (!isSynthetikaProduct(productResponse)) {
this.logger.warn("Product Response body did not satisfy product typeguard:", {
productResponse,
builder,
product,
productURL,
});
return;
}
let quantityObject: QuantityObject | undefined;
if ("options_configuration" in productResponse) {
quantityObject = mapDefined(
productResponse.options_configuration[0].values,
(value: SynthetikaConfigurationOptionValueSchema) => {
return parseQuantity(value.name);
},
)
.sort((a, b) => (a?.quantity ?? 0) - (b?.quantity ?? 0))
.at(0);
console.log("[synthetika] quantityObject (from configurationOptions", { quantityObject });
}
if (!quantityObject) {
quantityObject =
firstMap<string, QuantityObject | undefined>(parseQuantity, [
productResponse.name,
productResponse.description,
productResponse.weight.weight,
]) ?? undefined;
if (!quantityObject) {
this.logger.warn("Failed to parse quantity from product response", {
productResponse,
builder,
product,
productURL,
parsedValues: [
productResponse.name,
productResponse.description,
productResponse.weight.weight,
],
});
return;
}
}
builder.setQuantity(quantityObject);
if (builder.get("price") === undefined) {
const price = parsePrice(productResponse.price.gross.final);
if (!price) {
this.logger.warn("Failed to parse price from product response", {
productResponse,
builder,
product,
productURL,
parsedValue: productResponse.price.gross.final,
});
return;
}
builder.setPricing(price.price, price.currencyCode, price.currencySymbol);
}
const descriptionProperties = this.parseDescriptionHTML(productResponse.description);
console.log("[synthetika] descriptionProperties", {
descriptionProperties,
builder,
product,
productURL,
});
if (Object.keys(descriptionProperties).length > 0) {
if ("CAS Number" in descriptionProperties) {
builder.setCAS(descriptionProperties["CAS Number"]);
}
if ("Sum Formula" in descriptionProperties) {
builder.setFormula(descriptionProperties["Sum Formula"]);
}
}
// if (variants.length > 0) {
// builder.setVariants(variants);
// }
console.log("[synthetika] builder", { builder, product });
return builder;
});
}
Protected ReadonlyminThe minimum match percentage for a product to be considered a match.
Protected ReadonlyfuzzFuzz scorer used by fuzzyFilter to score each candidate's title against
the query. Any function from fuzzball with the
(str1, str2, opts?) => number shape works. Subclasses override this when
a supplier's title format needs a different scorer (e.g. a catalog that
pads titles with boilerplate might prefer partial_ratio). Defaults to
ratio.
Overridable at runtime from userSettings.fuzzScorerOverride — see
setFuzzScorerOverride and fuzzyFilter below. The user's Advanced
settings selection wins over this subclass default when set.
Protected OptionalfuzzRuntime override resolved from userSettings.fuzzScorerOverride. When
set, fuzzyFilter uses this instead of this.fuzzScorer. Undefined
(the default) means "use whatever the supplier class picked". Mutated
by setFuzzScorerOverride so it can't be readonly.
Protected Optional ReadonlyapiOptional external API hostname used by some suppliers (e.g., Typesense,
Searchanise). When set, automatically included in requiredHosts for
permission checks.
ProtectedqueryString to query for (product name, CAS, etc.). The search term that will be used to find products. Set during construction and used throughout the supplier's lifecycle.
ProtectedbaseThe base search parameters that are always included in search requests. These parameters are merged with any additional search parameters when making requests to the supplier's API.
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
this.baseSearchParams = {
format: "json",
version: "2.0"
};
}
}
protected baseSearchParams: Record<string, string | number> = {};
ProtectedcontrollerThe AbortController instance used to manage and cancel ongoing requests. This allows for cancellation of in-flight requests when needed, such as when a new search is started or the supplier is disposed.
const controller = new AbortController();
const supplier = new MySupplier("acetone", 5, controller);
// Later, to cancel all pending requests:
controller.abort();
protected controller: AbortController;
ProtectedlimitThe maximum number of results to return for a search query. This is not a limit on HTTP requests, but rather the number of products that will be returned to the caller.
const supplier = new MySupplier("acetone", 5); // Limit to 5 results
for await (const product of supplier) {
// Will yield at most 5 products
}
protected limit: number;
ProtectedproductsThe products that are currently being built by the supplier. This array holds ProductBuilder instances that are in the process of being transformed into complete Product objects.
await supplier.queryProducts("acetone");
console.log(`Building ${supplier.products.length} products`);
for (const builder of supplier.products) {
const product = await builder.build();
console.log("Built product:", product.title);
}
protected products: ProductBuilder<T>[] = [];
ProtectedhttpMaximum number of HTTP requests allowed per search query. This is a hard limit to prevent excessive requests to the supplier's API. If this limit is reached, the supplier will stop making new requests.
50
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
this.httpRequestHardLimit = 100; // Allow more requests
}
}
protected httpRequestHardLimit: number = 50;
ProtectedrequestCounter for HTTP requests made during the current query execution. This is used to track the number of requests and ensure we don't exceed the httpRequestHardLimit.
0
await supplier.queryProducts("acetone");
console.log(`Made ${supplier.requestCount} requests`);
if (supplier.requestCount >= supplier.httpRequestHardLimit) {
console.log("Reached request limit");
}
protected requestCount: number = 0;
ProtectedmaxNumber of requests to process in parallel when fetching product details. This controls the batch size for concurrent requests to avoid overwhelming the supplier's API and the user's bandwidth.
10
class MySupplier extends SupplierBase<Product> {
constructor() {
super();
// Process 5 requests at a time
this.maxConcurrentRequests = 5;
}
}
protected maxConcurrentRequests: number = 3;
ProtectedminMinimum number of milliseconds between two consecutive tasks
protected minConcurrentCycle: number = 100;
Protected ReadonlyrequiredCookies that must be written into the browser jar before any request runs
— e.g. a currency or session-preference cookie the backend reads. Seeded
once per instance by ensureSetup (before setup) via chrome.cookies,
since the Cookie request header is on the fetch-forbidden list and can't
be set through this.headers. Each entry's url defaults to baseURL.
Subclasses override this instead of hand-rolling a setup that calls
chrome.cookies.set directly.
[]
class MySupplier extends SupplierBase<Partial<Product>, Product> {
protected readonly requiredCookies: SupplierCookieSeed[] = [
{ name: "currency", value: "2" },
];
}
protected readonly requiredCookies: SupplierCookieSeed[] = [];
Protected ReadonlychallengeNumber of times fetch retries a request that comes back 403. Some
suppliers sit behind a WAF that 403s the first hit while planting a
session cookie (a "cookie handshake"); because every request now sets
credentials: "include", that cookie lands in the jar and the retry
carries it back, usually passing. We can't gate on the Set-Cookie
header (it's fetch-forbidden and invisible to JS), so this per-supplier
flag is the gate — 0 (the default) means never retry. Only enable it
for suppliers known to do this handshake.
0
protected readonly challengeRetryLimit: number = 0;
Protected ReadonlychallengeDelay in milliseconds between 403 challenge retries. Gives the WAF a
brief beat before re-requesting with the freshly-planted cookie.
300
protected readonly challengeRetryDelayMs: number = 300;
ProtectedloggerLogger for the supplier. Initialized in the constructor with the name of the inheriting class.
ProtectedproductDefault values for products. These will get overridden if they're found in the product data.
ProtectedcacheCache instance for this supplier.
Initialized after construction by initCache() (called from
SupplierFactory once supplierName is set). The ! assertion is safe
here because every code path that reads this.cache
(queryProductsWithCache, getProductData, getProductDataWithCache)
runs only after execute() is called on a factory-built instance, and the
factory always calls initCache() before handing the instance out.
ProtectedexcludedProduct-data cache keys the user has explicitly excluded via the "Ignore
Product" context menu action. Loaded once per execute() from
storage.local so membership checks are synchronous on the hot path
(see getProductData). Newly-ignored products take effect on the next
search, which matches the stated feature requirement.
ReadonlysupplierName of supplier (for display purposes)
ReadonlybaseBase URL for HTTP(s) requests
ReadonlyshippingShipping scope for Synthetika
ReadonlycountryThe country code of the supplier
ReadonlypaymentThe payment methods accepted by the supplier. Used to determine the payment methods accepted by the supplier.
ProtectedqueryOverride the type of queryResults to use our specific type
ProtectedhttpUsed to keep track of how many requests have been made to the supplier
ProtectedheadersHTTP headers used as a basis for all queries
Private ReadonlyincludeList of categories to include when filtering through the search results. This list made by sorting through https://synthetikaeu.com/webapi/front/en_US/categories/list/?limit=50
private readonly includeCategories: number[] = [
16, // Hydrides
17, // Antimony
18, // Barium
21, // Inorganic compounds
25, // Salts
30, // Ammonium
39, // Organic compounds
40, // Aldehydes
42, // Alcohols
43, // Aminoacids
44, // Amines
45, // Acyl Halides
46, // Esters
47, // Ethers
48, // Oxides
50, // Phenols
51, // Halogens
52, // Ketones
53, // Carboxylic acids
55, // Nitroles
57, // Polymers
58, // Terlenes
59, // Hydrocarbons
60, // Saturated
62, // Silicoorgano and Phosphoorganocompounds
64, // Nitro Compounds
67, // Others
68, // Bismuth,
70, // Cesium
71, // Chromium
72, // Tin
74, // Aluminium
75, // Zarconium
76, // Cadmium
77, // Lanthanum
78, // Lithium
79, // Magnesium
80, // Manganese
81, // Copper
83, // Nickel
84, // Lead
87, // Mercury
89, // Silver
90, // Strontium
94, // Calcium
96, // Iron
97, // Mixed salts
98, // Sodium
99, // Hydroxides
100, // Potassium
102, // Elements
103, // Metals
104, // Nonmetals
105, // Heterocycles
106, // Cobalt
108, // Amides
109, // Zinc
110, // Aromatic
111, // Anhydrides
112, // Organic salts
113, // Lactams
114, // Lactones
115, // Sugars
116, // Glycols
117, // Others
118, // Mixed solvents
119, // Chelates,
120, // Colorimetric reagents
273, // Alkyl nitriles
278, // "*"
279, // Chemicals sorted by use
281, // Solvents
290, // Industrial solvents
292, // Pyrotechnics
293, // Chemicals sorted by industry
293, // Oxidizers
294, // Reducing agents
295, // Metal powders
297, // Reducing agents for synthesis
298, // Oxidizing agents for synthesis
300, // Bases
301, // Classic acids
302, // Alkylating agents
303, // Halogenating reagents
304, // Mineral acids
305, // Cesium
306, // Nitrostyrenes
307, // Nitroalkanes
308, // Nitrobenzenes
309, // Catalysts
];
Supplier implementation for Synthetika
Remarks
Synthetika is a chemical supplier that sells a wide range of chemicals out of Poland. Their website seems to be using Shopper, which is a CMS popular with Polish ecommerce stores.
Their Shopper instance does have a public facing API (at /webapo/front). Most of the pages are limited to 50 results, which may require pagination.
Additionally, they don't seem to use "variants" very much. The same product with just slightly different quantities is just a different product, which fills up the results pretty quickly. I wrote the groupVariants function to group the products that are very similar all into a single product with an array of variants. This is done by removing the quantity from the title/name, then removing all spaces and dashes to give us a temporary unique identifier for the group. Then grouping that by that identifier.
Links:
Example API Endpoints:
Example
Source