Move source mapping to scrape database
This commit is contained in:
@@ -10,7 +10,8 @@ This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a N
|
||||
- Local DB may later allow controlled writes, but writes must remain explicit, logged and protected by transaction/dry-run settings.
|
||||
- Never print, commit or share endpoint tokens, usernames or passwords.
|
||||
- Never use Live DB as a fallback when Local DB is selected. If Local DB is unavailable, show an error and keep the UI running.
|
||||
- The selected database mode is authoritative for every database call. When `Local 9bplus DB` is selected, manufacturers, languages, products, combinations, mapping, source configuration and all other catalog reads must use the local MariaDB connection; no endpoint or Live DB call is allowed. When `Live 9bplus DB` is selected, every catalog call must use the existing endpoint and remain read-only. Do not mix sources within one workflow.
|
||||
- The selected catalog database mode is authoritative for catalog data calls. When `Local 9bplus DB` is selected, manufacturers, languages, products, combinations and all other catalog reads must use the local MariaDB connection; no endpoint or Live DB call is allowed. When `Live 9bplus DB` is selected, catalog reads use the existing endpoint and remain read-only.
|
||||
- Mapping, source configuration, scraper settings, browser-engine choices and scraped draft data belong to `Local scrape DB` (`catalog_scrape`) and must be read from there, not from Live DB or the product catalog database.
|
||||
- Keep the development server on port `3404` unless the user explicitly requests another port.
|
||||
- Do not delete or revert user files or unrelated changes.
|
||||
|
||||
@@ -102,9 +103,11 @@ There are three selectable data sources:
|
||||
### Local scrape DB
|
||||
|
||||
- Uses the separate local MariaDB database named `catalog_scrape`.
|
||||
- Stores scraper source configuration, scrape runs, discovered products and assets.
|
||||
- Stores scraper source configuration, manufacturer-to-source mapping, scrape runs, discovered products and assets.
|
||||
- Supplier/manufacturer website settings such as source URL, search templates, fallback mode, browser engine, headless mode, wait time, notes and enabled/priority flags belong in Local scrape DB, not in the product catalog database.
|
||||
- It must never be used as a fallback for catalog reads. When selected in the UI, existing catalog reads continue using Local 9bplus DB until dedicated scrape-storage endpoints are added.
|
||||
- The schema is versioned in `sql/local-scrape-db.sql` and can be created with `npm run db:create-scrape`.
|
||||
- Existing legacy mapping rows from `Local 9bplus DB` can be copied into Local scrape DB with `npm run db:migrate-mapping-to-scrape`.
|
||||
|
||||
### Local MariaDB connection
|
||||
|
||||
@@ -124,7 +127,7 @@ There are three selectable data sources:
|
||||
- The backend must resolve that header before every API database operation and route the complete request through the selected source. A Local DB request must use `queryMariaDb`; a Live DB request must use `queryEndpoint`; there is no silent fallback or mixed-source response.
|
||||
- Backend data API routes reject a Local DB request when MariaDB is not configured instead of silently using the endpoint.
|
||||
- Backend data API routes reject a Live DB request when the project is configured only for MariaDB.
|
||||
- Mapping endpoints follow the same rule as product endpoints and must read mapping tables from the selected database.
|
||||
- Product endpoints read catalog data from the selected catalog database. Mapping endpoints read scraper source mapping from Local scrape DB while still using the selected catalog database only for catalog/product data.
|
||||
- Static HTML/CSS/JS files must remain available even when a database is not configured.
|
||||
- The top status shows the selected source, connection state, permissions and execution mode.
|
||||
- Selecting Live 9bplus DB or Local scrape DB disables catalog write permissions in the UI.
|
||||
|
||||
Reference in New Issue
Block a user