rajch_ales/product-import-ai
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Store reusable scraper capabilities in scrape DB

Browse Source
This commit is contained in:
rajch_ales
2026-08-06 01:01:55 +02:00
parent b11ada9c91
commit 9e3b4de4e2
7 changed files with 411 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
agent.md
+2
View File
@@ -120,6 +120,8 @@ There are three selectable data sources:
- Uses the separate local MariaDB database named `catalog_scrape`.
- Stores scraper source configuration, manufacturer-to-source mapping, scrape runs, discovered products and assets.
- Supplier/manufacturer website settings such as source URL, search templates, fallback mode, browser engine, headless mode, wait time, notes and enabled/priority flags belong in Local scrape DB, not in the product catalog database.
- Source extraction rules are stored as reusable capabilities in Local scrape DB: `search`, `pictures`, `texts` and `features`. The code is only the generic engine; source-specific selectors, parser keys, minimum picture quality, browser engine choices and target strategies must live in DB so the same Hudy/Idealo rules can be reused for other manufacturers.
- Manufacturer-to-source mapping may enable or override individual capabilities, for example Hudy `pictures/texts/features` and Idealo `pictures`. Do not duplicate the same source configuration per manufacturer when a reusable source capability already exists.
- On first manufacturer selection, mapping status may be loaded quietly from Local scrape DB in the background without opening browsers or running Playwright URL checks. Page reloads should reuse the cached mapping status instead of doing the mapping check again. Manual manufacturer changes, including switching away and back to the same manufacturer, must load mapping from DB again. Browser/search work starts only after explicit user actions such as `Find sources`, `Get pictures`, or opening a source.
- The Settings modal should expose an `Open Adminer` action beside `Local scrape DB`; it opens the local Adminer URL from `ADMINER_URL` / `databaseTool.url` for inspecting local MariaDB databases.
- The Settings modal may also expose `Open DBGate` beside Adminer for a richer local database browser. DBGate should stay local, start with `npm run dbgate`, and use `DBGATE_URL` / `dbGate.url`.