product-import-ai/agent.md

Catalog Maker Project Notes

Purpose

This project replaces the Excel/VBA workflow named Catalog maker - 20 with a Node.js local web application. The UI is a compact product catalog editor for loading manufacturer products, combinations, source mappings and parsed supplier data.

Non-negotiable rules

• Live DB is never written to. Live DB access is read-only through the existing endpoint.
• Local DB may later allow controlled writes, but writes must remain explicit, logged and protected by transaction/dry-run settings.
• Never print, commit or share endpoint tokens, usernames or passwords.
• Never use Live DB as a fallback when Local DB is selected. If Local DB is unavailable, show an error and keep the UI running.
• The selected catalog database mode is authoritative for catalog data calls. When Local 9bplus DB is selected, manufacturers, languages, products, combinations and all other catalog reads must use the local MariaDB connection; no endpoint or Live DB call is allowed. When Live 9bplus DB is selected, catalog reads use the existing endpoint and remain read-only.
• Mapping, source configuration, scraper settings, browser-engine choices and scraped draft data belong to Local scrape DB (catalog_scrape) and must be read from there, not from Live DB or the product catalog database.
• Keep the development server on port 3404 unless the user explicitly requests another port.
• Do not delete or revert user files or unrelated changes.

Git workflow

• After every completed implementation step, run the relevant checks, create a focused commit and push it to the configured remote repository.
• Use short, descriptive commit messages that state the completed change.
• Never commit .env, config/local.json, passwords, tokens, the Excel workbook or database dumps.
• Before the first push, initialize Git if needed, configure the project remote and create an initial baseline commit from safe files only.
• If a push fails, do not expose credentials in the terminal output or in chat; report the failure without printing secrets.

UI inspection workflow

• Use Playwright for browser inspections and visual verification of the local application.
• Run npm run test:ui after frontend or layout changes.
• Check for page errors, visible controls, modal behavior and important responsive states.
• Keep screenshots and traces in Playwright's ignored test-results/ and playwright-report/ directories; do not commit them.
• A UI change is not complete until the relevant Playwright checks pass.

Styling workflow

• Use Tailwind CSS for new styling and UI changes.
• Keep shared tokens in src/tailwind.css using @theme and base/component layers.
• Run npm run css:build after styling changes; public/styles.css is generated output.
• Prefer Tailwind utility classes and Tailwind-style component patterns for buttons, panels, dialogs and responsive layouts.
• Whenever a reusable pattern appears, prefer Tailwind component patterns (@layer components with @apply) over handwritten legacy CSS; keep one-off layout details as utility classes where practical.
• Whenever possible, use Tailwind UI / Tailwind Plus application components and patterns for shells, multi-column layouts, sidebars, drawers, dialogs, navigation and forms; adapt them to this project's dense catalog workflow instead of inventing a separate visual pattern.
• Multi-column panels must use a shared collapse pattern: collapsing a panel removes its grid track, lets the adjacent workspace expand, preserves an accessible expand/collapse control, and stacks predictably on mobile.
• When the left sidebar is closed on desktop, its open control keeps a dedicated narrow rail; it must not be positioned inside the secondary product-details panel. On mobile, the rail collapses and the control may overlay the stacked layout.
• Top headers for the sidebar, secondary panel and main workspace must use one shared fixed height so the columns align visually across the application.
• All top panel headers must use the shared panel-header component class, including identical background, border, typography, spacing and dark/light theme behavior. Do not style Catalog controls, Product details or Main workspace headers independently.
• Catalog controls is the visual master for the top header bar. Product details and Main workspace must match it exactly in title color, font, height, padding, border, background and control-button treatment in both themes.
• The three top panel headers must share the same top baseline and visual bounds. The sidebar header must compensate for the sidebar's inner padding so it is not offset from the product-details and main-workspace headers.
• Collapse/expand arrow buttons must keep writing-mode: horizontal-tb when their parent panel uses vertical writing mode, so the arrow glyph remains visually directional.
• Panel collapse/open controls must use the shared panel-control-button style; do not create separate visual variants for X, chevrons or future panel toggles.
• Verify shared panel-control styling through computed browser styles, not only by checking class names.
• Use Playwright to inspect the rendered result after every visual change at desktop and narrow viewport sizes.
• Every actionable button and selector must use an icon before its text whenever a suitable symbol exists. Reuse the existing symbols from public/icons.svg; do not create text-only controls when an appropriate icon exists.
• Button icons must be placed before the label in DOM order, with the icon and label centered together as one group. Keep the control's chevron or disclosure arrow in its own fixed right-side slot so it does not shift the centered label.
• Use one shared icon size, gap, alignment, hover state and disabled state for equivalent controls. Do not mix icon-only, text-only and icon-after-text variants for the same control family.
• Before finishing any UI change, inspect the rendered page with Playwright and verify that icons are visible, precede their labels, remain centered, and do not overlap or drift between controls. Update this file when a new reusable visual rule is discovered.
• Custom select controls may use a native select as an interaction layer, but the native label must be visually transparent when a separate display label is rendered; never show the same control text twice.
• Runtime button labels must update the existing direct child label span instead of replacing button.textContent; state changes must never remove the button icon.
• Parser tools are always visible in the left panel. Do not hide, collapse or replace the parser action buttons with a toggle; the user must see the complete parser action set at all times.
• Mapping status and mapping source details belong beside the database connection status at the bottom of the sidebar. Style it as a shared sidebar section header with icon, centered status and a fixed right chevron; keep the existing mapping behavior.

TypeScript workflow

• All application, service, SQL, workflow, script and test source files must use TypeScript (.ts).
• Run the project through tsx during development and compile the browser entrypoint from public/app.ts to public/app.js before serving it.
• Do not add new JavaScript source files. Generated browser JavaScript is build output only.
• Keep database safety rules and existing runtime behavior unchanged during the migration; add types incrementally to migrated modules.

Browser scraping workflow

• Use Playwright for all supplier/manufacturer website scraping and browser page inspection.
• Do not implement new scraping with raw fetch, regex-only HTML parsing, Selenium or direct HTTP shortcuts.
• Scrapers must use controlled browser navigation, selectors, explicit waits and bounded request counts.
• The configured browser engine for a source is authoritative. If a source says firefox, launch Firefox or fail loudly; do not silently fall back to Chromium, Edge or another browser.
• Visible source windows, URL health checks, source lookup and picture parsing must all use the same configured browser engine from that source's database setting.
• Source candidate Open actions in the UI must call the backend source opener with manufacturerId, sourceKey and URL. Do not use plain target="_blank" source links for mapped supplier/manufacturer sources because that ignores the configured browser engine.
• Keep scraping slow and observable, preserve source URL/status information, and handle blocked pages or missing selectors as explicit errors.
• Supplier data remains a local draft until the user explicitly confirms an Apply / Save action.

Runtime

• Start the web app with npm run dev.
• Open http://127.0.0.1:3404/.
• The current server is src/server.ts, run through tsx.
• Configuration is loaded from config/local.json, merged over config/local.example.json.
• .env is loaded automatically by src/config.ts for local MariaDB settings. Never expose its values. The explicit local write capability flag is DB_LOCAL_ALLOW_WRITES; it never enables Live DB writes.

Database architecture

There are three selectable data sources:

Local 9bplus DB

• Uses the imported local MariaDB database named 9bplus.
• This is the only database allowed to receive explicit catalog writes.
• Writes remain disabled unless the user enables the local permission switches.

Live 9bplus DB

• Uses the existing 9b-plus API endpoint configured in config/local.json.
• Credentials are secret and must stay server-side.
• Must remain read-only.

Local scrape DB

• Uses the separate local MariaDB database named catalog_scrape.
• Stores scraper source configuration, manufacturer-to-source mapping, scrape runs, discovered products and assets.
• Supplier/manufacturer website settings such as source URL, search templates, fallback mode, browser engine, headless mode, wait time, notes and enabled/priority flags belong in Local scrape DB, not in the product catalog database.
• On first manufacturer selection, mapping status may be loaded quietly from Local scrape DB in the background without opening browsers or running Playwright URL checks. Page reloads should reuse the cached mapping status instead of doing the mapping check again. Manual manufacturer changes, including switching away and back to the same manufacturer, must load mapping from DB again. Browser/search work starts only after explicit user actions such as Find sources, Get pictures, or opening a source.
• The Settings modal should expose an Open Adminer action beside Local scrape DB; it opens the local Adminer URL from ADMINER_URL / databaseTool.url for inspecting local MariaDB databases.
• The Settings modal may also expose Open DBGate beside Adminer for a richer local database browser. DBGate should stay local, start with npm run dbgate, and use DBGATE_URL / dbGate.url.
• npm run dbgate must start local-only DBGate without auth prompts and with two predefined local MariaDB connections from .env: Local 9bplus DB (DB_NAME) and Local scrape DB (SCRAPE_DB_NAME). Do not hardcode or commit passwords; pass them through environment variables at process start.
• Start Adminer only through npm run adminer; the script downloads the ignored local tools/adminer/adminer.php file when missing and runs it on the configured local Adminer URL. Adminer requires local PHP CLI.
• Adminer login help may show non-secret local connection fields such as system, host, port, database and user, but must never render or print the actual password. Show password source as .env DB_PASSWORD instead.
• It must never be used as a fallback for catalog reads. When selected in the UI, existing catalog reads continue using Local 9bplus DB until dedicated scrape-storage endpoints are added.
• The schema is versioned in sql/local-scrape-db.sql and can be created with npm run db:create-scrape.
• Existing legacy mapping rows from Local 9bplus DB can be copied into Local scrape DB with npm run db:migrate-mapping-to-scrape.

Local MariaDB connection

• MariaDB is installed locally as service MariaDB.
• Host: localhost / 127.0.0.1.
• Port: 3306.
• Database: 9bplus.
• A full dump from 9bplus (1).sql.gz was imported locally.
• Verified local contents: about 266 tables, 74 manufacturers, 8,888 products and 48,206 product combinations.
• The import was local only and did not contact Live DB.
• The project currently needs to be switched to database.driver = "mariadb" and supplied with local credentials before the UI can read this database.

Database safety behavior

• The UI has Local 9bplus DB, Live 9bplus DB and Local scrape DB settings.
• The UI sends the selected mode using the X-Database-Mode request header.
• The backend must resolve that header before every API database operation and route the complete request through the selected source. A Local DB request must use queryMariaDb; a Live DB request must use queryEndpoint; there is no silent fallback or mixed-source response.
• Backend data API routes reject a Local DB request when MariaDB is not configured instead of silently using the endpoint.
• Backend data API routes reject a Live DB request when the project is configured only for MariaDB.
• Product endpoints read catalog data from the selected catalog database. Mapping endpoints read scraper source mapping from Local scrape DB while still using the selected catalog database only for catalog/product data.
• Static HTML/CSS/JS files must remain available even when a database is not configured.
• The top status shows the selected source, connection state, permissions and execution mode.
• Selecting Live 9bplus DB or Local scrape DB disables catalog write permissions in the UI.

Important current files

• src/server.ts: HTTP server, API routes, mode guard and status endpoint.
• src/config.ts: configuration loading and database settings.
• src/endpoint/endpoint-client.ts: endpoint queries and MariaDB routing.
• src/db/mariadb-client.ts: MariaDB pool, read queries, local writes and connection test.
• src/services/catalog-products.ts: product, combinations, mapping, source and picture workflow.
• src/services/browser-adapter.ts: Playwright navigation, scraping and visible browser inspection.
• public/index.html: compact catalog UI and settings modal.
• public/app.ts: typed browser source compiled to public/app.js.
• public/styles.css: compact modern layout.
• config/local.example.json: safe configuration template.
• config/local.json: local secrets and machine-specific configuration; never expose it.

Current UI behavior

• Manufacturer and language selects load from the selected database.
• Product data loads when manufacturer changes or navigation arrows are used.
• Get product info remains a separate action.
• EAN scan input searches on Enter.
• Find sources searches once and caches candidates for the current product; a second click reopens cached results.
• The source-candidates popup must provide a Reload action beside Close; Reload bypasses only the current product's cached source result and re-runs the source lookup using the current local mapping configuration. It must not switch database modes or write to Live DB.
• Source candidates show how they were found, including EAN/reference/name and color context.
• Each source row has its own Get pictures action.
• Product source cache is cleared when a new product loads.
• Hudy picture parsing uses the correct product color variant, checks EANs and extracts large gallery images.
• Idealo uses the browser adapter and is intended mainly as a picture source.
• Image preview is square.
• Settings is at the bottom of the left panel; database status is directly above it.

Settings

The settings modal contains:

• Local DB / Live DB selection.
• Local DB connection fields: host, port, database, user and password.
• Test connection, which performs a read-only SELECT VERSION() against local MariaDB only.
• Local insert/update/delete permission switches.
• Transaction confirmation or dry-run preview mode.
• Live DB permissions are always read-only.

Local UI settings are stored in browser localStorage. They are not yet a replacement for server-side config/local.json credentials.

Next safe step

Wire the local connection settings into server configuration without exposing Live credentials, then set the active local database driver to MariaDB and test manufacturer loading from the imported 9bplus database. After that, verify that switching to Live uses only read-only endpoint queries.

Verification commands

npm run build:types
npm run build:browser
npm run css:build
npm test

For a local MariaDB check, use the installed client against 127.0.0.1:3306 and never the Live endpoint.