Migrate project to TypeScript and Playwright scraping
This commit is contained in:
@@ -38,11 +38,26 @@ This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a N
|
||||
- Whenever a reusable pattern appears, prefer Tailwind component patterns (`@layer components` with `@apply`) over handwritten legacy CSS; keep one-off layout details as utility classes where practical.
|
||||
- Use Playwright to inspect the rendered result after every visual change at desktop and narrow viewport sizes.
|
||||
|
||||
## TypeScript workflow
|
||||
|
||||
- All application, service, SQL, workflow, script and test source files must use TypeScript (`.ts`).
|
||||
- Run the project through `tsx` during development and compile the browser entrypoint from `public/app.ts` to `public/app.js` before serving it.
|
||||
- Do not add new JavaScript source files. Generated browser JavaScript is build output only.
|
||||
- Keep database safety rules and existing runtime behavior unchanged during the migration; add types incrementally to migrated modules.
|
||||
|
||||
## Browser scraping workflow
|
||||
|
||||
- Use Playwright for all supplier/manufacturer website scraping and browser page inspection.
|
||||
- Do not implement new scraping with raw `fetch`, regex-only HTML parsing, Selenium or direct HTTP shortcuts.
|
||||
- Scrapers must use controlled browser navigation, selectors, explicit waits and bounded request counts.
|
||||
- Keep scraping slow and observable, preserve source URL/status information, and handle blocked pages or missing selectors as explicit errors.
|
||||
- Supplier data remains a local draft until the user explicitly confirms an Apply / Save action.
|
||||
|
||||
## Runtime
|
||||
|
||||
- Start the web app with `npm run dev`.
|
||||
- Open `http://127.0.0.1:3404/`.
|
||||
- The current server is `src/server.js`.
|
||||
- The current server is `src/server.ts`, run through `tsx`.
|
||||
- Configuration is loaded from `config/local.json`, merged over `config/local.example.json`.
|
||||
- `.env` exists for local MariaDB notes, but the application does not yet load `.env` automatically. Do not assume `.env` is active until configuration loading is explicitly wired to it.
|
||||
|
||||
@@ -78,13 +93,14 @@ There are two selectable data sources:
|
||||
|
||||
## Important current files
|
||||
|
||||
- `src/server.js`: HTTP server, API routes, mode guard and status endpoint.
|
||||
- `src/config.js`: configuration loading and database settings.
|
||||
- `src/endpoint/endpoint-client.js`: endpoint queries and MariaDB routing.
|
||||
- `src/db/mariadb-client.js`: MariaDB pool, read queries, local writes and connection test.
|
||||
- `src/services/catalog-products.js`: product, combinations, mapping, source and picture workflow.
|
||||
- `src/server.ts`: HTTP server, API routes, mode guard and status endpoint.
|
||||
- `src/config.ts`: configuration loading and database settings.
|
||||
- `src/endpoint/endpoint-client.ts`: endpoint queries and MariaDB routing.
|
||||
- `src/db/mariadb-client.ts`: MariaDB pool, read queries, local writes and connection test.
|
||||
- `src/services/catalog-products.ts`: product, combinations, mapping, source and picture workflow.
|
||||
- `src/services/browser-adapter.ts`: Playwright navigation, scraping and visible browser inspection.
|
||||
- `public/index.html`: compact catalog UI and settings modal.
|
||||
- `public/app.js`: UI state, API calls, Local/Live mode header, local settings and source workflow.
|
||||
- `public/app.ts`: typed browser source compiled to `public/app.js`.
|
||||
- `public/styles.css`: compact modern layout.
|
||||
- `config/local.example.json`: safe configuration template.
|
||||
- `config/local.json`: local secrets and machine-specific configuration; never expose it.
|
||||
@@ -124,9 +140,9 @@ Wire the local connection settings into server configuration without exposing Li
|
||||
## Verification commands
|
||||
|
||||
```powershell
|
||||
node --check src/server.js
|
||||
node --check src/db/mariadb-client.js
|
||||
node --check public/app.js
|
||||
npm run build:types
|
||||
npm run build:browser
|
||||
npm run css:build
|
||||
npm test
|
||||
```
|
||||
|
||||
|
||||
Reference in New Issue
Block a user