168 lines
9.8 KiB
Markdown
168 lines
9.8 KiB
Markdown
# Catalog Maker Project Notes
|
|
|
|
## Purpose
|
|
|
|
This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a Node.js local web application. The UI is a compact product catalog editor for loading manufacturer products, combinations, source mappings and parsed supplier data.
|
|
|
|
## Non-negotiable rules
|
|
|
|
- Live DB is never written to. Live DB access is read-only through the existing endpoint.
|
|
- Local DB may later allow controlled writes, but writes must remain explicit, logged and protected by transaction/dry-run settings.
|
|
- Never print, commit or share endpoint tokens, usernames or passwords.
|
|
- Never use Live DB as a fallback when Local DB is selected. If Local DB is unavailable, show an error and keep the UI running.
|
|
- Keep the development server on port `3404` unless the user explicitly requests another port.
|
|
- Do not delete or revert user files or unrelated changes.
|
|
|
|
## Git workflow
|
|
|
|
- After every completed implementation step, run the relevant checks, create a focused commit and push it to the configured remote repository.
|
|
- Use short, descriptive commit messages that state the completed change.
|
|
- Never commit `.env`, `config/local.json`, passwords, tokens, the Excel workbook or database dumps.
|
|
- Before the first push, initialize Git if needed, configure the project remote and create an initial baseline commit from safe files only.
|
|
- If a push fails, do not expose credentials in the terminal output or in chat; report the failure without printing secrets.
|
|
|
|
## UI inspection workflow
|
|
|
|
- Use Playwright for browser inspections and visual verification of the local application.
|
|
- Run `npm run test:ui` after frontend or layout changes.
|
|
- Check for page errors, visible controls, modal behavior and important responsive states.
|
|
- Keep screenshots and traces in Playwright's ignored `test-results/` and `playwright-report/` directories; do not commit them.
|
|
- A UI change is not complete until the relevant Playwright checks pass.
|
|
|
|
## Styling workflow
|
|
|
|
- Use Tailwind CSS for new styling and UI changes.
|
|
- Keep shared tokens in `src/tailwind.css` using `@theme` and base/component layers.
|
|
- Run `npm run css:build` after styling changes; `public/styles.css` is generated output.
|
|
- Prefer Tailwind utility classes and Tailwind-style component patterns for buttons, panels, dialogs and responsive layouts.
|
|
- Whenever a reusable pattern appears, prefer Tailwind component patterns (`@layer components` with `@apply`) over handwritten legacy CSS; keep one-off layout details as utility classes where practical.
|
|
- Whenever possible, use Tailwind UI / Tailwind Plus application components and patterns for shells, multi-column layouts, sidebars, drawers, dialogs, navigation and forms; adapt them to this project's dense catalog workflow instead of inventing a separate visual pattern.
|
|
- Multi-column panels must use a shared collapse pattern: collapsing a panel removes its grid track, lets the adjacent workspace expand, preserves an accessible expand/collapse control, and stacks predictably on mobile.
|
|
- When the left sidebar is closed on desktop, its open control keeps a dedicated narrow rail; it must not be positioned inside the secondary product-details panel. On mobile, the rail collapses and the control may overlay the stacked layout.
|
|
- Top headers for the sidebar, secondary panel and main workspace must use one shared fixed height so the columns align visually across the application.
|
|
- Use Playwright to inspect the rendered result after every visual change at desktop and narrow viewport sizes.
|
|
|
|
## TypeScript workflow
|
|
|
|
- All application, service, SQL, workflow, script and test source files must use TypeScript (`.ts`).
|
|
- Run the project through `tsx` during development and compile the browser entrypoint from `public/app.ts` to `public/app.js` before serving it.
|
|
- Do not add new JavaScript source files. Generated browser JavaScript is build output only.
|
|
- Keep database safety rules and existing runtime behavior unchanged during the migration; add types incrementally to migrated modules.
|
|
|
|
## Browser scraping workflow
|
|
|
|
- Use Playwright for all supplier/manufacturer website scraping and browser page inspection.
|
|
- Do not implement new scraping with raw `fetch`, regex-only HTML parsing, Selenium or direct HTTP shortcuts.
|
|
- Scrapers must use controlled browser navigation, selectors, explicit waits and bounded request counts.
|
|
- Keep scraping slow and observable, preserve source URL/status information, and handle blocked pages or missing selectors as explicit errors.
|
|
- Supplier data remains a local draft until the user explicitly confirms an Apply / Save action.
|
|
|
|
## Runtime
|
|
|
|
- Start the web app with `npm run dev`.
|
|
- Open `http://127.0.0.1:3404/`.
|
|
- The current server is `src/server.ts`, run through `tsx`.
|
|
- Configuration is loaded from `config/local.json`, merged over `config/local.example.json`.
|
|
- `.env` exists for local MariaDB notes, but the application does not yet load `.env` automatically. Do not assume `.env` is active until configuration loading is explicitly wired to it.
|
|
|
|
## Database architecture
|
|
|
|
There are three selectable data sources:
|
|
|
|
### Local 9bplus DB
|
|
|
|
- Uses the imported local MariaDB database named `9bplus`.
|
|
- This is the only database allowed to receive explicit catalog writes.
|
|
- Writes remain disabled unless the user enables the local permission switches.
|
|
|
|
### Live 9bplus DB
|
|
|
|
- Uses the existing 9b-plus API endpoint configured in `config/local.json`.
|
|
- Credentials are secret and must stay server-side.
|
|
- Must remain read-only.
|
|
|
|
### Local scrape DB
|
|
|
|
- Uses the separate local MariaDB database named `catalog_scrape`.
|
|
- Stores scraper source configuration, scrape runs, discovered products and assets.
|
|
- It must never be used as a fallback for catalog reads. When selected in the UI, existing catalog reads continue using Local 9bplus DB until dedicated scrape-storage endpoints are added.
|
|
- The schema is versioned in `sql/local-scrape-db.sql` and can be created with `npm run db:create-scrape`.
|
|
|
|
### Local MariaDB connection
|
|
|
|
- MariaDB is installed locally as service `MariaDB`.
|
|
- Host: `localhost` / `127.0.0.1`.
|
|
- Port: `3306`.
|
|
- Database: `9bplus`.
|
|
- A full dump from `9bplus (1).sql.gz` was imported locally.
|
|
- Verified local contents: about 266 tables, 74 manufacturers, 8,888 products and 48,206 product combinations.
|
|
- The import was local only and did not contact Live DB.
|
|
- The project currently needs to be switched to `database.driver = "mariadb"` and supplied with local credentials before the UI can read this database.
|
|
|
|
## Database safety behavior
|
|
|
|
- The UI has `Local 9bplus DB`, `Live 9bplus DB` and `Local scrape DB` settings.
|
|
- The UI sends the selected mode using the `X-Database-Mode` request header.
|
|
- Backend data API routes reject a Local DB request when MariaDB is not configured instead of silently using the endpoint.
|
|
- Backend data API routes reject a Live DB request when the project is configured only for MariaDB.
|
|
- Static HTML/CSS/JS files must remain available even when a database is not configured.
|
|
- The top status shows the selected source, connection state, permissions and execution mode.
|
|
- Selecting Live 9bplus DB or Local scrape DB disables catalog write permissions in the UI.
|
|
|
|
## Important current files
|
|
|
|
- `src/server.ts`: HTTP server, API routes, mode guard and status endpoint.
|
|
- `src/config.ts`: configuration loading and database settings.
|
|
- `src/endpoint/endpoint-client.ts`: endpoint queries and MariaDB routing.
|
|
- `src/db/mariadb-client.ts`: MariaDB pool, read queries, local writes and connection test.
|
|
- `src/services/catalog-products.ts`: product, combinations, mapping, source and picture workflow.
|
|
- `src/services/browser-adapter.ts`: Playwright navigation, scraping and visible browser inspection.
|
|
- `public/index.html`: compact catalog UI and settings modal.
|
|
- `public/app.ts`: typed browser source compiled to `public/app.js`.
|
|
- `public/styles.css`: compact modern layout.
|
|
- `config/local.example.json`: safe configuration template.
|
|
- `config/local.json`: local secrets and machine-specific configuration; never expose it.
|
|
|
|
## Current UI behavior
|
|
|
|
- Manufacturer and language selects load from the selected database.
|
|
- Product data loads when manufacturer changes or navigation arrows are used.
|
|
- `Get product info` remains a separate action.
|
|
- EAN scan input searches on Enter.
|
|
- `Find sources` searches once and caches candidates for the current product; a second click reopens cached results.
|
|
- Source candidates show how they were found, including EAN/reference/name and color context.
|
|
- Each source row has its own `Get pictures` action.
|
|
- Product source cache is cleared when a new product loads.
|
|
- Hudy picture parsing uses the correct product color variant, checks EANs and extracts large gallery images.
|
|
- Idealo uses the browser adapter and is intended mainly as a picture source.
|
|
- Image preview is square.
|
|
- Settings is at the bottom of the left panel; database status is directly above it.
|
|
|
|
## Settings
|
|
|
|
The settings modal contains:
|
|
|
|
- Local DB / Live DB selection.
|
|
- Local DB connection fields: host, port, database, user and password.
|
|
- `Test connection`, which performs a read-only `SELECT VERSION()` against local MariaDB only.
|
|
- Local insert/update/delete permission switches.
|
|
- Transaction confirmation or dry-run preview mode.
|
|
- Live DB permissions are always read-only.
|
|
|
|
Local UI settings are stored in browser localStorage. They are not yet a replacement for server-side `config/local.json` credentials.
|
|
|
|
## Next safe step
|
|
|
|
Wire the local connection settings into server configuration without exposing Live credentials, then set the active local database driver to MariaDB and test manufacturer loading from the imported `9bplus` database. After that, verify that switching to Live uses only read-only endpoint queries.
|
|
|
|
## Verification commands
|
|
|
|
```powershell
|
|
npm run build:types
|
|
npm run build:browser
|
|
npm run css:build
|
|
npm test
|
|
```
|
|
|
|
For a local MariaDB check, use the installed client against `127.0.0.1:3306` and never the Live endpoint.
|