product-import-ai/agent.md

Catalog Maker Project Notes

Purpose

This project replaces the Excel/VBA workflow named Catalog maker - 20 with a Node.js local web application. The UI is a compact product catalog editor for loading manufacturer products, combinations, source mappings and parsed supplier data.

Non-negotiable rules

• Live DB is never written to. Live DB access is read-only through the existing endpoint.
• Local DB may later allow controlled writes, but writes must remain explicit, logged and protected by transaction/dry-run settings.
• Never print, commit or share endpoint tokens, usernames or passwords.
• Never use Live DB as a fallback when Local DB is selected. If Local DB is unavailable, show an error and keep the UI running.
• Keep the development server on port 3404 unless the user explicitly requests another port.
• Do not delete or revert user files or unrelated changes.

Git workflow

• After every completed implementation step, run the relevant checks, create a focused commit and push it to the configured remote repository.
• Use short, descriptive commit messages that state the completed change.
• Never commit .env, config/local.json, passwords, tokens, the Excel workbook or database dumps.
• Before the first push, initialize Git if needed, configure the project remote and create an initial baseline commit from safe files only.
• If a push fails, do not expose credentials in the terminal output or in chat; report the failure without printing secrets.

UI inspection workflow

• Use Playwright for browser inspections and visual verification of the local application.
• Run npm run test:ui after frontend or layout changes.
• Check for page errors, visible controls, modal behavior and important responsive states.
• Keep screenshots and traces in Playwright's ignored test-results/ and playwright-report/ directories; do not commit them.
• A UI change is not complete until the relevant Playwright checks pass.

Styling workflow

• Use Tailwind CSS for new styling and UI changes.
• Keep shared tokens in src/tailwind.css using @theme and base/component layers.
• Run npm run css:build after styling changes; public/styles.css is generated output.
• Prefer Tailwind utility classes and Tailwind-style component patterns for buttons, panels, dialogs and responsive layouts.
• Whenever a reusable pattern appears, prefer Tailwind component patterns (@layer components with @apply) over handwritten legacy CSS; keep one-off layout details as utility classes where practical.
• Whenever possible, use Tailwind UI / Tailwind Plus application components and patterns for shells, multi-column layouts, sidebars, drawers, dialogs, navigation and forms; adapt them to this project's dense catalog workflow instead of inventing a separate visual pattern.
• Multi-column panels must use a shared collapse pattern: collapsing a panel removes its grid track, lets the adjacent workspace expand, preserves an accessible expand/collapse control, and stacks predictably on mobile.
• When the left sidebar is closed on desktop, its open control keeps a dedicated narrow rail; it must not be positioned inside the secondary product-details panel. On mobile, the rail collapses and the control may overlay the stacked layout.
• Top headers for the sidebar, secondary panel and main workspace must use one shared fixed height so the columns align visually across the application.
• All top panel headers must use the shared panel-header component class, including identical background, border, typography, spacing and dark/light theme behavior. Do not style Catalog controls, Product details or Main workspace headers independently.
• Panel collapse/open controls must use the shared panel-control-button style; do not create separate visual variants for X, chevrons or future panel toggles.
• Verify shared panel-control styling through computed browser styles, not only by checking class names.
• Use Playwright to inspect the rendered result after every visual change at desktop and narrow viewport sizes.

TypeScript workflow

• All application, service, SQL, workflow, script and test source files must use TypeScript (.ts).
• Run the project through tsx during development and compile the browser entrypoint from public/app.ts to public/app.js before serving it.
• Do not add new JavaScript source files. Generated browser JavaScript is build output only.
• Keep database safety rules and existing runtime behavior unchanged during the migration; add types incrementally to migrated modules.

Browser scraping workflow

• Use Playwright for all supplier/manufacturer website scraping and browser page inspection.
• Do not implement new scraping with raw fetch, regex-only HTML parsing, Selenium or direct HTTP shortcuts.
• Scrapers must use controlled browser navigation, selectors, explicit waits and bounded request counts.
• Keep scraping slow and observable, preserve source URL/status information, and handle blocked pages or missing selectors as explicit errors.
• Supplier data remains a local draft until the user explicitly confirms an Apply / Save action.

Runtime

• Start the web app with npm run dev.
• Open http://127.0.0.1:3404/.
• The current server is src/server.ts, run through tsx.
• Configuration is loaded from config/local.json, merged over config/local.example.json.
• .env exists for local MariaDB notes, but the application does not yet load .env automatically. Do not assume .env is active until configuration loading is explicitly wired to it.

Database architecture

There are three selectable data sources:

Local 9bplus DB

• Uses the imported local MariaDB database named 9bplus.
• This is the only database allowed to receive explicit catalog writes.
• Writes remain disabled unless the user enables the local permission switches.

Live 9bplus DB

• Uses the existing 9b-plus API endpoint configured in config/local.json.
• Credentials are secret and must stay server-side.
• Must remain read-only.

Local scrape DB

• Uses the separate local MariaDB database named catalog_scrape.
• Stores scraper source configuration, scrape runs, discovered products and assets.
• It must never be used as a fallback for catalog reads. When selected in the UI, existing catalog reads continue using Local 9bplus DB until dedicated scrape-storage endpoints are added.
• The schema is versioned in sql/local-scrape-db.sql and can be created with npm run db:create-scrape.

Local MariaDB connection

• MariaDB is installed locally as service MariaDB.
• Host: localhost / 127.0.0.1.
• Port: 3306.
• Database: 9bplus.
• A full dump from 9bplus (1).sql.gz was imported locally.
• Verified local contents: about 266 tables, 74 manufacturers, 8,888 products and 48,206 product combinations.
• The import was local only and did not contact Live DB.
• The project currently needs to be switched to database.driver = "mariadb" and supplied with local credentials before the UI can read this database.

Database safety behavior

• The UI has Local 9bplus DB, Live 9bplus DB and Local scrape DB settings.
• The UI sends the selected mode using the X-Database-Mode request header.
• Backend data API routes reject a Local DB request when MariaDB is not configured instead of silently using the endpoint.
• Backend data API routes reject a Live DB request when the project is configured only for MariaDB.
• Static HTML/CSS/JS files must remain available even when a database is not configured.
• The top status shows the selected source, connection state, permissions and execution mode.
• Selecting Live 9bplus DB or Local scrape DB disables catalog write permissions in the UI.

Important current files

• src/server.ts: HTTP server, API routes, mode guard and status endpoint.
• src/config.ts: configuration loading and database settings.
• src/endpoint/endpoint-client.ts: endpoint queries and MariaDB routing.
• src/db/mariadb-client.ts: MariaDB pool, read queries, local writes and connection test.
• src/services/catalog-products.ts: product, combinations, mapping, source and picture workflow.
• src/services/browser-adapter.ts: Playwright navigation, scraping and visible browser inspection.
• public/index.html: compact catalog UI and settings modal.
• public/app.ts: typed browser source compiled to public/app.js.
• public/styles.css: compact modern layout.
• config/local.example.json: safe configuration template.
• config/local.json: local secrets and machine-specific configuration; never expose it.

Current UI behavior

• Manufacturer and language selects load from the selected database.
• Product data loads when manufacturer changes or navigation arrows are used.
• Get product info remains a separate action.
• EAN scan input searches on Enter.
• Find sources searches once and caches candidates for the current product; a second click reopens cached results.
• Source candidates show how they were found, including EAN/reference/name and color context.
• Each source row has its own Get pictures action.
• Product source cache is cleared when a new product loads.
• Hudy picture parsing uses the correct product color variant, checks EANs and extracts large gallery images.
• Idealo uses the browser adapter and is intended mainly as a picture source.
• Image preview is square.
• Settings is at the bottom of the left panel; database status is directly above it.

Settings

The settings modal contains:

• Local DB / Live DB selection.
• Local DB connection fields: host, port, database, user and password.
• Test connection, which performs a read-only SELECT VERSION() against local MariaDB only.
• Local insert/update/delete permission switches.
• Transaction confirmation or dry-run preview mode.
• Live DB permissions are always read-only.

Local UI settings are stored in browser localStorage. They are not yet a replacement for server-side config/local.json credentials.

Next safe step

Wire the local connection settings into server configuration without exposing Live credentials, then set the active local database driver to MariaDB and test manufacturer loading from the imported 9bplus database. After that, verify that switching to Live uses only read-only endpoint queries.

Verification commands

npm run build:types
npm run build:browser
npm run css:build
npm test

For a local MariaDB check, use the installed client against 127.0.0.1:3306 and never the Live endpoint.