Class SupplierCache

Remarks

Benefits of this Caching Approach:

1. Granular Data Reuse:

   • Product detail data is cached independently of search results
   • Same product details can be reused across different search queries
   • Example: If "sodium chloride" and "NaCl" searches return the same product, the product details are only fetched and cached once

2. Flexible Result Limits:

   • Search results are cached without limit constraints
   • Same cached results can be used for different limit requests
   • Example: A search for "acetone" can return 5, 10, or 20 results using the same cached data, just sliced differently

3. Storage Efficiency:

   • Avoids duplicate storage of product details
   • Reduces storage requirements by ~50-80% compared to caching complete results for each query
   • Example: 100 searches returning the same 10 products would store product details 100 times in a naive cache, but only 10 times in this system

4. Improved Performance:

   • Faster subsequent searches due to partial cache hits
   • Reduced API calls by reusing cached product details
   • Example: A new search that includes previously cached products can skip their detail fetches

5. Better Cache Invalidation:

   • Product details can be invalidated independently
   • Search results remain valid even if some product details are updated
   • Example: Price updates only require re-fetching affected product details, not all search results

6. Optimized Network Usage:

   • Minimizes redundant API calls
   • Reduces bandwidth usage by avoiding re-fetching unchanged data
   • Example: Product descriptions rarely change, so they can be cached longer than prices

The cache system uses two separate storage keys:

1. supplier_query_cache - Stores search query results
2. supplier_product_data_cache - Stores detailed product data

Cache Storage Format:

• Query Cache:

  {
    [cacheKey: string]: {
      data: T[],                    // The actual cached results
      __cacheMetadata: {            // Metadata about the cache entry
        cachedAt: number,           // Timestamp when cached
        version: number,            // Cache format version
        query: string,              // Original search query
        supplier: string,           // Supplier name
        resultCount: number,        // Number of results
        limit: number              // Limit used for query
      }
    }
  }
  Copy

• Product Data Cache:

  {
    [cacheKey: string]: {
      data: Record<string, unknown>, // The cached product data
      timestamp: number             // When the data was cached
    }
  }
  Copy

Cache Key Generation:

• Query Cache Keys: Generated using base64 encoding of ${query}:${supplierName} Falls back to a simple hash if base64 is unavailable
• Product Data Cache Keys: Generated using MD5 hash of JSON stringified object containing:

  {
    url: string,                    // Product URL
    params?: Record<string, string>, // Optional request parameters
    supplier: string                // Supplier name
  }
  Copy

Cache Expiration & Management:

• Size Limits:
  • Query Cache: Maximum 100 entries
  • Product Data Cache: Maximum 100 entries
• Eviction Policy: Least Recently Used (LRU)
  • When cache is full, oldest entries are removed based on timestamp
• Cache Invalidation:
  • Query cache entries are invalidated if:
    1. Cache version changes (CACHE_VERSION constant)
    2. Requested limit is greater than cached limit
  • Product data cache entries are refreshed on access (timestamp updated to prevent premature eviction)

Cache Triggering: The cache is automatically triggered in two main scenarios:

1. Query Results Caching (via queryProductsWithCache):

   // Inside SupplierBase class
   protected async queryProductsWithCache(query: string, limit: number) {
     // 1. Check cache first
     const key = this.cache.generateCacheKey(query, this.supplierName);
     const cached = await this.getCachedQueryResults(key);
   
     if (cached) {
       // 2a. Return cached results if valid
       return ProductBuilder.createFromCache(cached.data);
     }
   
     // 2b. If not in cache, perform actual query
     const results = await this.queryProducts(query, limit);
   
     // 3. Cache the new results
     if (results) {
       await this.cache.cacheQueryResults(
         query,
         this.supplierName,
         results.map(b => b.dump()),
         limit
       );
     }
   
     return results;
   }
   Copy

2. Product Data Caching (via getProductData):

   // Inside SupplierBase class
   protected async getProductData(product: ProductBuilder<T>) {
     const url = product.get("url");
     const cacheKey = this.cache.getProductDataCacheKey(url, this.supplierName);
   
     // 1. Check cache first
     const cachedData = await this.cache.getCachedProductData(cacheKey);
     if (cachedData) {
       // 2a. Return cached data if available
       product.setData(cachedData);
       return product;
     }
   
     // 2b. If not in cache, fetch fresh data
     const resultBuilder = await this.getProductDataWithCache(product);
   
     // 3. Cache the new data
     if (resultBuilder) {
       await this.cache.cacheProductData(
         cacheKey,
         resultBuilder.dump()
       );
     }
   
     return resultBuilder;
   }
   Copy

Cache Flow:

1. Query Results:

   • Triggered by supplier's search operations
   • Cached after initial search results are processed (smaller cache than if the raw result was cached or the fully processed data)
   • Retrieved before making new search requests
   • Invalidated when search parameters change (not including timestamps in the URL)

2. Product Data:

   • Triggered when fetching detailed product information
   • Cached after successful product data retrieval (output of ProductBuilder.dump())
   • Retrieved before making new product detail requests
   • Automatically refreshed on access to prevent eviction

Usage Example:

// Initialize cache for a supplier
const cache = new SupplierCache("ChemSupplier");

// Cache query results
await cache.cacheQueryResults(
  "sodium chloride",
  "ChemSupplier",
  results,
  10
);

// Cache product data
await cache.cacheProductData(
  cacheKey,
  productData
);

// Retrieve cached data
const cachedData = await cache.getCachedProductData(cacheKey);
Copy