179 lines
12 KiB
Markdown
179 lines
12 KiB
Markdown
# Catalog Maker Project Notes
|
|
|
|
## Purpose
|
|
|
|
This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a Node.js local web application. The UI is a compact product catalog editor for loading manufacturer products, combinations, source mappings and parsed supplier data.
|
|
|
|
## Non-negotiable rules
|
|
|
|
- Live DB is never written to. Live DB access is read-only through the existing endpoint.
|
|
- Local DB may later allow controlled writes, but writes must remain explicit, logged and protected by transaction/dry-run settings.
|
|
- Never print, commit or share endpoint tokens, usernames or passwords.
|
|
- Never use Live DB as a fallback when Local DB is selected. If Local DB is unavailable, show an error and keep the UI running.
|
|
- Keep the development server on port `3404` unless the user explicitly requests another port.
|
|
- Do not delete or revert user files or unrelated changes.
|
|
|
|
## Git workflow
|
|
|
|
- After every completed implementation step, run the relevant checks, create a focused commit and push it to the configured remote repository.
|
|
- Use short, descriptive commit messages that state the completed change.
|
|
- Never commit `.env`, `config/local.json`, passwords, tokens, the Excel workbook or database dumps.
|
|
- Before the first push, initialize Git if needed, configure the project remote and create an initial baseline commit from safe files only.
|
|
- If a push fails, do not expose credentials in the terminal output or in chat; report the failure without printing secrets.
|
|
|
|
## UI inspection workflow
|
|
|
|
- Use Playwright for browser inspections and visual verification of the local application.
|
|
- Run `npm run test:ui` after frontend or layout changes.
|
|
- Check for page errors, visible controls, modal behavior and important responsive states.
|
|
- Keep screenshots and traces in Playwright's ignored `test-results/` and `playwright-report/` directories; do not commit them.
|
|
- A UI change is not complete until the relevant Playwright checks pass.
|
|
|
|
## Styling workflow
|
|
|
|
- Use Tailwind CSS for new styling and UI changes.
|
|
- Keep shared tokens in `src/tailwind.css` using `@theme` and base/component layers.
|
|
- Run `npm run css:build` after styling changes; `public/styles.css` is generated output.
|
|
- Prefer Tailwind utility classes and Tailwind-style component patterns for buttons, panels, dialogs and responsive layouts.
|
|
- Whenever a reusable pattern appears, prefer Tailwind component patterns (`@layer components` with `@apply`) over handwritten legacy CSS; keep one-off layout details as utility classes where practical.
|
|
- Whenever possible, use Tailwind UI / Tailwind Plus application components and patterns for shells, multi-column layouts, sidebars, drawers, dialogs, navigation and forms; adapt them to this project's dense catalog workflow instead of inventing a separate visual pattern.
|
|
- Multi-column panels must use a shared collapse pattern: collapsing a panel removes its grid track, lets the adjacent workspace expand, preserves an accessible expand/collapse control, and stacks predictably on mobile.
|
|
- When the left sidebar is closed on desktop, its open control keeps a dedicated narrow rail; it must not be positioned inside the secondary product-details panel. On mobile, the rail collapses and the control may overlay the stacked layout.
|
|
- Top headers for the sidebar, secondary panel and main workspace must use one shared fixed height so the columns align visually across the application.
|
|
- All top panel headers must use the shared `panel-header` component class, including identical background, border, typography, spacing and dark/light theme behavior. Do not style `Catalog controls`, `Product details` or `Main workspace` headers independently.
|
|
- `Catalog controls` is the visual master for the top header bar. Product details and Main workspace must match it exactly in title color, font, height, padding, border, background and control-button treatment in both themes.
|
|
- The three top panel headers must share the same top baseline and visual bounds. The sidebar header must compensate for the sidebar's inner padding so it is not offset from the product-details and main-workspace headers.
|
|
- Collapse/expand arrow buttons must keep `writing-mode: horizontal-tb` when their parent panel uses vertical writing mode, so the arrow glyph remains visually directional.
|
|
- Panel collapse/open controls must use the shared `panel-control-button` style; do not create separate visual variants for `X`, chevrons or future panel toggles.
|
|
- Verify shared panel-control styling through computed browser styles, not only by checking class names.
|
|
- Use Playwright to inspect the rendered result after every visual change at desktop and narrow viewport sizes.
|
|
- Every actionable button and selector must use an icon before its text whenever a suitable symbol exists. Reuse the existing symbols from `public/icons.svg`; do not create text-only controls when an appropriate icon exists.
|
|
- Button icons must be placed before the label in DOM order, with the icon and label centered together as one group. Keep the control's chevron or disclosure arrow in its own fixed right-side slot so it does not shift the centered label.
|
|
- Use one shared icon size, gap, alignment, hover state and disabled state for equivalent controls. Do not mix icon-only, text-only and icon-after-text variants for the same control family.
|
|
- Before finishing any UI change, inspect the rendered page with Playwright and verify that icons are visible, precede their labels, remain centered, and do not overlap or drift between controls. Update this file when a new reusable visual rule is discovered.
|
|
- Custom select controls may use a native select as an interaction layer, but the native label must be visually transparent when a separate display label is rendered; never show the same control text twice.
|
|
|
|
## TypeScript workflow
|
|
|
|
- All application, service, SQL, workflow, script and test source files must use TypeScript (`.ts`).
|
|
- Run the project through `tsx` during development and compile the browser entrypoint from `public/app.ts` to `public/app.js` before serving it.
|
|
- Do not add new JavaScript source files. Generated browser JavaScript is build output only.
|
|
- Keep database safety rules and existing runtime behavior unchanged during the migration; add types incrementally to migrated modules.
|
|
|
|
## Browser scraping workflow
|
|
|
|
- Use Playwright for all supplier/manufacturer website scraping and browser page inspection.
|
|
- Do not implement new scraping with raw `fetch`, regex-only HTML parsing, Selenium or direct HTTP shortcuts.
|
|
- Scrapers must use controlled browser navigation, selectors, explicit waits and bounded request counts.
|
|
- Keep scraping slow and observable, preserve source URL/status information, and handle blocked pages or missing selectors as explicit errors.
|
|
- Supplier data remains a local draft until the user explicitly confirms an Apply / Save action.
|
|
|
|
## Runtime
|
|
|
|
- Start the web app with `npm run dev`.
|
|
- Open `http://127.0.0.1:3404/`.
|
|
- The current server is `src/server.ts`, run through `tsx`.
|
|
- Configuration is loaded from `config/local.json`, merged over `config/local.example.json`.
|
|
- `.env` exists for local MariaDB notes, but the application does not yet load `.env` automatically. Do not assume `.env` is active until configuration loading is explicitly wired to it.
|
|
|
|
## Database architecture
|
|
|
|
There are three selectable data sources:
|
|
|
|
### Local 9bplus DB
|
|
|
|
- Uses the imported local MariaDB database named `9bplus`.
|
|
- This is the only database allowed to receive explicit catalog writes.
|
|
- Writes remain disabled unless the user enables the local permission switches.
|
|
|
|
### Live 9bplus DB
|
|
|
|
- Uses the existing 9b-plus API endpoint configured in `config/local.json`.
|
|
- Credentials are secret and must stay server-side.
|
|
- Must remain read-only.
|
|
|
|
### Local scrape DB
|
|
|
|
- Uses the separate local MariaDB database named `catalog_scrape`.
|
|
- Stores scraper source configuration, scrape runs, discovered products and assets.
|
|
- It must never be used as a fallback for catalog reads. When selected in the UI, existing catalog reads continue using Local 9bplus DB until dedicated scrape-storage endpoints are added.
|
|
- The schema is versioned in `sql/local-scrape-db.sql` and can be created with `npm run db:create-scrape`.
|
|
|
|
### Local MariaDB connection
|
|
|
|
- MariaDB is installed locally as service `MariaDB`.
|
|
- Host: `localhost` / `127.0.0.1`.
|
|
- Port: `3306`.
|
|
- Database: `9bplus`.
|
|
- A full dump from `9bplus (1).sql.gz` was imported locally.
|
|
- Verified local contents: about 266 tables, 74 manufacturers, 8,888 products and 48,206 product combinations.
|
|
- The import was local only and did not contact Live DB.
|
|
- The project currently needs to be switched to `database.driver = "mariadb"` and supplied with local credentials before the UI can read this database.
|
|
|
|
## Database safety behavior
|
|
|
|
- The UI has `Local 9bplus DB`, `Live 9bplus DB` and `Local scrape DB` settings.
|
|
- The UI sends the selected mode using the `X-Database-Mode` request header.
|
|
- Backend data API routes reject a Local DB request when MariaDB is not configured instead of silently using the endpoint.
|
|
- Backend data API routes reject a Live DB request when the project is configured only for MariaDB.
|
|
- Static HTML/CSS/JS files must remain available even when a database is not configured.
|
|
- The top status shows the selected source, connection state, permissions and execution mode.
|
|
- Selecting Live 9bplus DB or Local scrape DB disables catalog write permissions in the UI.
|
|
|
|
## Important current files
|
|
|
|
- `src/server.ts`: HTTP server, API routes, mode guard and status endpoint.
|
|
- `src/config.ts`: configuration loading and database settings.
|
|
- `src/endpoint/endpoint-client.ts`: endpoint queries and MariaDB routing.
|
|
- `src/db/mariadb-client.ts`: MariaDB pool, read queries, local writes and connection test.
|
|
- `src/services/catalog-products.ts`: product, combinations, mapping, source and picture workflow.
|
|
- `src/services/browser-adapter.ts`: Playwright navigation, scraping and visible browser inspection.
|
|
- `public/index.html`: compact catalog UI and settings modal.
|
|
- `public/app.ts`: typed browser source compiled to `public/app.js`.
|
|
- `public/styles.css`: compact modern layout.
|
|
- `config/local.example.json`: safe configuration template.
|
|
- `config/local.json`: local secrets and machine-specific configuration; never expose it.
|
|
|
|
## Current UI behavior
|
|
|
|
- Manufacturer and language selects load from the selected database.
|
|
- Product data loads when manufacturer changes or navigation arrows are used.
|
|
- `Get product info` remains a separate action.
|
|
- EAN scan input searches on Enter.
|
|
- `Find sources` searches once and caches candidates for the current product; a second click reopens cached results.
|
|
- Source candidates show how they were found, including EAN/reference/name and color context.
|
|
- Each source row has its own `Get pictures` action.
|
|
- Product source cache is cleared when a new product loads.
|
|
- Hudy picture parsing uses the correct product color variant, checks EANs and extracts large gallery images.
|
|
- Idealo uses the browser adapter and is intended mainly as a picture source.
|
|
- Image preview is square.
|
|
- Settings is at the bottom of the left panel; database status is directly above it.
|
|
|
|
## Settings
|
|
|
|
The settings modal contains:
|
|
|
|
- Local DB / Live DB selection.
|
|
- Local DB connection fields: host, port, database, user and password.
|
|
- `Test connection`, which performs a read-only `SELECT VERSION()` against local MariaDB only.
|
|
- Local insert/update/delete permission switches.
|
|
- Transaction confirmation or dry-run preview mode.
|
|
- Live DB permissions are always read-only.
|
|
|
|
Local UI settings are stored in browser localStorage. They are not yet a replacement for server-side `config/local.json` credentials.
|
|
|
|
## Next safe step
|
|
|
|
Wire the local connection settings into server configuration without exposing Live credentials, then set the active local database driver to MariaDB and test manufacturer loading from the imported `9bplus` database. After that, verify that switching to Live uses only read-only endpoint queries.
|
|
|
|
## Verification commands
|
|
|
|
```powershell
|
|
npm run build:types
|
|
npm run build:browser
|
|
npm run css:build
|
|
npm test
|
|
```
|
|
|
|
For a local MariaDB check, use the installed client against `127.0.0.1:3306` and never the Live endpoint.
|