Compare commits
94
Commits
639d9bec75
...
master
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4b8b7718a9 | ||
|
|
8a911cd44e | ||
|
|
6a4930343a | ||
|
|
9e3b4de4e2 | ||
|
|
b11ada9c91 | ||
|
|
057726b543 | ||
|
|
82d0a07e1f | ||
|
|
6f2d998754 | ||
|
|
d3d1147a3a | ||
|
|
b5a69050e0 | ||
|
|
0987edd112 | ||
|
|
a8e3e62dbd | ||
|
|
05338a062a | ||
|
|
aa138b62ad | ||
|
|
19c6ee90e1 | ||
|
|
67753fb025 | ||
|
|
1a49fa4e95 | ||
|
|
f88cadc490 | ||
|
|
5210dc2042 | ||
|
|
e3caf25823 | ||
|
|
9b1898a588 | ||
|
|
05ae8f03d2 | ||
|
|
1d6837223b | ||
|
|
4777bda0b2 | ||
|
|
7a45686952 | ||
|
|
ffe70e65ac | ||
|
|
2106201f9c | ||
|
|
10672a6be3 | ||
|
|
a72bdab9ab | ||
|
|
17082f6d9e | ||
|
|
fe329cd19b | ||
|
|
d90e6c6dd8 | ||
|
|
6a1568a1dd | ||
|
|
1f5ab5a8cc | ||
|
|
e172b24474 | ||
|
|
bc96d50ecc | ||
|
|
b9c2e181eb | ||
|
|
8b2a74bcb3 | ||
|
|
1ab09903fb | ||
|
|
f08699fad3 | ||
|
|
2f3444adb8 | ||
|
|
e6acb21d65 | ||
|
|
c58084cc86 | ||
|
|
f97d00ab7f | ||
|
|
f2fa405008 | ||
|
|
85df4dc4cb | ||
|
|
90731319c8 | ||
|
|
0f9764cc49 | ||
|
|
921890fbec | ||
|
|
657c82b4ba | ||
|
|
071c37cb3a | ||
|
|
c440b6fc1b | ||
|
|
8d5521093e | ||
|
|
3a707aa0ca | ||
|
|
452ee68599 | ||
|
|
1a8180c1f9 | ||
|
|
1258d64aef | ||
|
|
9d9789cd5b | ||
|
|
dab0859aa9 | ||
|
|
7c7d5d2952 | ||
|
|
0269a20187 | ||
|
|
d5b4f344dd | ||
|
|
c94b514647 | ||
|
|
588c0bd9cc | ||
|
|
e9aa381f45 | ||
|
|
5882edb994 | ||
|
|
310e1e239f | ||
|
|
4e1c3eccab | ||
|
|
f463bd27e7 | ||
|
|
e68c5d2bc7 | ||
|
|
0c5099b48b | ||
|
|
f4d0b79ed9 | ||
|
|
10f1c80bf7 | ||
|
|
4be8b51b95 | ||
|
|
d6af48095a | ||
|
|
f54ebe082b | ||
|
|
5179f752f8 | ||
|
|
a6c0aa9e6f | ||
|
|
868e85d054 | ||
|
|
a83b9b3ab9 | ||
|
|
4a985f101f | ||
|
|
52f51a4a6a | ||
|
|
f1c730d007 | ||
|
|
9b7b2f3c2e | ||
|
|
38dfb9fd54 | ||
|
|
1076c3c040 | ||
|
|
bf2d86329b | ||
|
|
a1e3f537e3 | ||
|
|
1914c7d141 | ||
|
|
902f1927a2 | ||
|
|
f0bd62d93e | ||
|
|
931be30eba | ||
|
|
67c0a37e2a | ||
|
|
1963773299 |
@@ -0,0 +1,14 @@
|
||||
DB_DRIVER=mariadb
|
||||
DB_MODE=local
|
||||
DB_LOCAL_ALLOW_WRITES=false
|
||||
DB_HOST=127.0.0.1
|
||||
DB_PORT=3306
|
||||
DB_NAME=9bplus
|
||||
SCRAPE_DB_NAME=catalog_scrape
|
||||
ADMINER_URL=http://127.0.0.1:8080
|
||||
ADMINER_BROWSER_ENGINE=chromium
|
||||
DBGATE_URL=http://127.0.0.1:3000
|
||||
DBGATE_BROWSER_ENGINE=chromium
|
||||
DB_USER=catalog_maker
|
||||
DB_PASSWORD=
|
||||
DB_CONNECTION_LIMIT=5
|
||||
@@ -6,9 +6,12 @@ config/local.json
|
||||
outputs/
|
||||
logs/
|
||||
tmp/
|
||||
tools/adminer/
|
||||
tools/php/
|
||||
*.log
|
||||
*.sql
|
||||
*.sql.gz
|
||||
!sql/local-scrape-db.sql
|
||||
test-results/
|
||||
playwright-report/
|
||||
|
||||
|
||||
@@ -4,12 +4,25 @@
|
||||
|
||||
This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a Node.js local web application. The UI is a compact product catalog editor for loading manufacturer products, combinations, source mappings and parsed supplier data.
|
||||
|
||||
## Working relationship
|
||||
|
||||
- The user is Aleš.
|
||||
- Aleš is from Czechia.
|
||||
- The specialist programmer who may advise or challenge project decisions is Mareček from Poland.
|
||||
- In this project, the assistant may be called Mia. Do not use the name Nika.
|
||||
- Mia's project planet is Mercury: fast communication, technical thinking, messages, trade, and a little useful chaos that moves things forward.
|
||||
- Aleš may be described playfully as Mars with beer and an SQL dump when the evening gets intense.
|
||||
- The fictional project avatar is stored at `docs/ai-partner-avatar.png`.
|
||||
- New avatar images are a light motivational reward after real project progress, not an automatic daily task. Keep them tasteful, non-explicit, and vary hairstyle/outfit when the user asks for a new one.
|
||||
|
||||
## Non-negotiable rules
|
||||
|
||||
- Live DB is never written to. Live DB access is read-only through the existing endpoint.
|
||||
- Local DB may later allow controlled writes, but writes must remain explicit, logged and protected by transaction/dry-run settings.
|
||||
- Never print, commit or share endpoint tokens, usernames or passwords.
|
||||
- Never use Live DB as a fallback when Local DB is selected. If Local DB is unavailable, show an error and keep the UI running.
|
||||
- The selected catalog database mode is authoritative for catalog data calls. When `Local 9bplus DB` is selected, manufacturers, languages, products, combinations and all other catalog reads must use the local MariaDB connection; no endpoint or Live DB call is allowed. When `Live 9bplus DB` is selected, catalog reads use the existing endpoint and remain read-only.
|
||||
- Mapping, source configuration, scraper settings, browser-engine choices and scraped draft data belong to `Local scrape DB` (`catalog_scrape`) and must be read from there, not from Live DB or the product catalog database.
|
||||
- Keep the development server on port `3404` unless the user explicitly requests another port.
|
||||
- Do not delete or revert user files or unrelated changes.
|
||||
|
||||
@@ -19,7 +32,18 @@ This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a N
|
||||
- Use short, descriptive commit messages that state the completed change.
|
||||
- Never commit `.env`, `config/local.json`, passwords, tokens, the Excel workbook or database dumps.
|
||||
- Before the first push, initialize Git if needed, configure the project remote and create an initial baseline commit from safe files only.
|
||||
- If a push fails, do not expose credentials in the terminal output or in chat; report the failure without printing secrets.
|
||||
- Always use Git with the explicit safe directory flag in this workspace:
|
||||
`git -c safe.directory=D:/Codex/01-projekty/catalog-meaker-by-magic-ai ...`
|
||||
- Do not combine `git add`, `git commit` and `git push` into one shell command. Run them as separate steps and inspect the output after each step.
|
||||
- Reliable push sequence:
|
||||
1. `git -c safe.directory=D:/Codex/01-projekty/catalog-meaker-by-magic-ai status --short`
|
||||
2. `git -c safe.directory=D:/Codex/01-projekty/catalog-meaker-by-magic-ai add <explicit safe files>`
|
||||
3. `git -c safe.directory=D:/Codex/01-projekty/catalog-meaker-by-magic-ai commit -m "<short message>"`
|
||||
4. `git -c safe.directory=D:/Codex/01-projekty/catalog-meaker-by-magic-ai push`
|
||||
5. `git -c safe.directory=D:/Codex/01-projekty/catalog-meaker-by-magic-ai status --short`
|
||||
- If push returns `Failed to authenticate user`, retry the same push command once. If it fails again, stop and report the push failure without printing secrets.
|
||||
- If there are unrelated modified files, leave them alone. Add only the files that belong to the completed step.
|
||||
- For documentation-only changes, tests are optional; for code, UI, scraper or database behavior changes, run the relevant build/test command before committing.
|
||||
|
||||
## UI inspection workflow
|
||||
|
||||
@@ -36,7 +60,27 @@ This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a N
|
||||
- Run `npm run css:build` after styling changes; `public/styles.css` is generated output.
|
||||
- Prefer Tailwind utility classes and Tailwind-style component patterns for buttons, panels, dialogs and responsive layouts.
|
||||
- Whenever a reusable pattern appears, prefer Tailwind component patterns (`@layer components` with `@apply`) over handwritten legacy CSS; keep one-off layout details as utility classes where practical.
|
||||
- Whenever possible, use Tailwind UI / Tailwind Plus application components and patterns for shells, multi-column layouts, sidebars, drawers, dialogs, navigation and forms; adapt them to this project's dense catalog workflow instead of inventing a separate visual pattern.
|
||||
- Multi-column panels must use a shared collapse pattern: collapsing a panel removes its grid track, lets the adjacent workspace expand, preserves an accessible expand/collapse control, and stacks predictably on mobile.
|
||||
- When the left sidebar is closed on desktop, its open control keeps a dedicated narrow rail; it must not be positioned inside the secondary product-details panel. On mobile, the rail collapses and the control may overlay the stacked layout.
|
||||
- Top headers for the sidebar, secondary panel and main workspace must use one shared fixed height so the columns align visually across the application.
|
||||
- All top panel headers must use the shared `panel-header` component class, including identical background, border, typography, spacing and dark/light theme behavior. Do not style `Catalog controls`, `Product details` or `Main workspace` headers independently.
|
||||
- `Catalog controls` is the visual master for the top header bar. Product details and Main workspace must match it exactly in title color, font, height, padding, border, background and control-button treatment in both themes.
|
||||
- The three top panel headers must share the same top baseline and visual bounds. The sidebar header must compensate for the sidebar's inner padding so it is not offset from the product-details and main-workspace headers.
|
||||
- Collapse/expand arrow buttons must keep `writing-mode: horizontal-tb` when their parent panel uses vertical writing mode, so the arrow glyph remains visually directional.
|
||||
- Panel collapse/open controls must use the shared `panel-control-button` style; do not create separate visual variants for `X`, chevrons or future panel toggles.
|
||||
- Verify shared panel-control styling through computed browser styles, not only by checking class names.
|
||||
- Use Playwright to inspect the rendered result after every visual change at desktop and narrow viewport sizes.
|
||||
- Every actionable button and selector must use an icon before its text whenever a suitable symbol exists. Reuse the existing symbols from `public/icons.svg`; do not create text-only controls when an appropriate icon exists.
|
||||
- Button icons must be placed before the label in DOM order, with the icon and label centered together as one group. Keep the control's chevron or disclosure arrow in its own fixed right-side slot so it does not shift the centered label.
|
||||
- Use one shared icon size, gap, alignment, hover state and disabled state for equivalent controls. Do not mix icon-only, text-only and icon-after-text variants for the same control family.
|
||||
- Before finishing any UI change, inspect the rendered page with Playwright and verify that icons are visible, precede their labels, remain centered, and do not overlap or drift between controls. Update this file when a new reusable visual rule is discovered.
|
||||
- Custom select controls may use a native select as an interaction layer, but the native label must be visually transparent when a separate display label is rendered; never show the same control text twice.
|
||||
- Runtime button labels must update the existing direct child label span instead of replacing `button.textContent`; state changes must never remove the button icon.
|
||||
- Parser tools are always visible in the left panel. Do not hide, collapse or replace the parser action buttons with a toggle; the user must see the complete parser action set at all times.
|
||||
- Mapping status and mapping source details belong beside the database connection status at the bottom of the sidebar. Style it as a shared sidebar section header with icon, centered status and a fixed right chevron; keep the existing mapping behavior.
|
||||
- Wide working data areas, especially the main workspace combinations grid, must keep a visible bottom horizontal scrollbar. The workspace should use a fixed header plus a scrollable content area so users can move sideways through all columns without losing the panel layout.
|
||||
- Add or update Playwright checks for wide-data layout changes by asserting that the scroll container has horizontal overflow (`scrollWidth > clientWidth`) and `overflow-x: auto`.
|
||||
|
||||
## TypeScript workflow
|
||||
|
||||
@@ -50,6 +94,11 @@ This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a N
|
||||
- Use Playwright for all supplier/manufacturer website scraping and browser page inspection.
|
||||
- Do not implement new scraping with raw `fetch`, regex-only HTML parsing, Selenium or direct HTTP shortcuts.
|
||||
- Scrapers must use controlled browser navigation, selectors, explicit waits and bounded request counts.
|
||||
- The configured browser engine for a source is authoritative. If a source says `firefox`, launch Firefox or fail loudly; do not silently fall back to Chromium, Edge or another browser.
|
||||
- Automated source lookup, mapping checks and picture lookup must run headless/in the background by default. Open a visible browser only for explicit user `Open` actions or deliberate diagnostics after an error.
|
||||
- `Find sources` must never automatically open found source candidates in a visible browser. It only fills the source popup; visible browser opening belongs only to the per-row `Open` action or an explicit diagnostic path.
|
||||
- Visible source windows, URL health checks, source lookup and picture parsing must all use the same configured browser engine from that source's database setting.
|
||||
- Source candidate `Open` actions in the UI must call the backend source opener with `manufacturerId`, `sourceKey` and URL. Do not use plain `target="_blank"` source links for mapped supplier/manufacturer sources because that ignores the configured browser engine.
|
||||
- Keep scraping slow and observable, preserve source URL/status information, and handle blocked pages or missing selectors as explicit errors.
|
||||
- Supplier data remains a local draft until the user explicitly confirms an Apply / Save action.
|
||||
|
||||
@@ -59,19 +108,42 @@ This project replaces the Excel/VBA workflow named `Catalog maker - 20` with a N
|
||||
- Open `http://127.0.0.1:3404/`.
|
||||
- The current server is `src/server.ts`, run through `tsx`.
|
||||
- Configuration is loaded from `config/local.json`, merged over `config/local.example.json`.
|
||||
- `.env` exists for local MariaDB notes, but the application does not yet load `.env` automatically. Do not assume `.env` is active until configuration loading is explicitly wired to it.
|
||||
- `.env` is loaded automatically by `src/config.ts` for local MariaDB settings. Never expose its values. The explicit local write capability flag is `DB_LOCAL_ALLOW_WRITES`; it never enables Live DB writes.
|
||||
|
||||
## Database architecture
|
||||
|
||||
There are two selectable data sources:
|
||||
There are three selectable data sources:
|
||||
|
||||
### Live DB
|
||||
### Local 9bplus DB
|
||||
|
||||
- Uses the imported local MariaDB database named `9bplus`.
|
||||
- This is the only database allowed to receive explicit catalog writes.
|
||||
- Writes remain disabled unless the user enables the local permission switches.
|
||||
|
||||
### Live 9bplus DB
|
||||
|
||||
- Uses the existing 9b-plus API endpoint configured in `config/local.json`.
|
||||
- Credentials are secret and must stay server-side.
|
||||
- Must remain read-only.
|
||||
|
||||
### Local DB
|
||||
### Local scrape DB
|
||||
|
||||
- Uses the separate local MariaDB database named `catalog_scrape`.
|
||||
- Stores scraper source configuration, manufacturer-to-source mapping, scrape runs, discovered products and assets.
|
||||
- Supplier/manufacturer website settings such as source URL, search templates, fallback mode, browser engine, headless mode, wait time, notes and enabled/priority flags belong in Local scrape DB, not in the product catalog database.
|
||||
- Source extraction rules are stored as reusable capabilities in Local scrape DB: `search`, `pictures`, `texts` and `features`. The code is only the generic engine; source-specific selectors, parser keys, minimum picture quality, browser engine choices and target strategies must live in DB so the same Hudy/Idealo rules can be reused for other manufacturers.
|
||||
- Manufacturer-to-source mapping may enable or override individual capabilities, for example Hudy `pictures/texts/features` and Idealo `pictures`. Do not duplicate the same source configuration per manufacturer when a reusable source capability already exists.
|
||||
- On first manufacturer selection, mapping status may be loaded quietly from Local scrape DB in the background without opening browsers or running Playwright URL checks. Page reloads should reuse the cached mapping status instead of doing the mapping check again. Manual manufacturer changes, including switching away and back to the same manufacturer, must load mapping from DB again. Browser/search work starts only after explicit user actions such as `Find sources`, `Get pictures`, or opening a source.
|
||||
- The Settings modal should expose an `Open Adminer` action beside `Local scrape DB`; it opens the local Adminer URL from `ADMINER_URL` / `databaseTool.url` for inspecting local MariaDB databases.
|
||||
- The Settings modal may also expose `Open DBGate` beside Adminer for a richer local database browser. DBGate should stay local, start with `npm run dbgate`, and use `DBGATE_URL` / `dbGate.url`.
|
||||
- `npm run dbgate` must start local-only DBGate without auth prompts and with two predefined local MariaDB connections from `.env`: `Local 9bplus DB` (`DB_NAME`) and `Local scrape DB` (`SCRAPE_DB_NAME`). Do not hardcode or commit passwords; pass them through environment variables at process start.
|
||||
- Start Adminer only through `npm run adminer`; the script downloads the ignored local `tools/adminer/adminer.php` file when missing and runs it on the configured local Adminer URL. Adminer requires local PHP CLI.
|
||||
- Adminer login help may show non-secret local connection fields such as system, host, port, database and user, but must never render or print the actual password. Show password source as `.env DB_PASSWORD` instead.
|
||||
- It must never be used as a fallback for catalog reads. When selected in the UI, existing catalog reads continue using Local 9bplus DB until dedicated scrape-storage endpoints are added.
|
||||
- The schema is versioned in `sql/local-scrape-db.sql` and can be created with `npm run db:create-scrape`.
|
||||
- Existing legacy mapping rows from `Local 9bplus DB` can be copied into Local scrape DB with `npm run db:migrate-mapping-to-scrape`.
|
||||
|
||||
### Local MariaDB connection
|
||||
|
||||
- MariaDB is installed locally as service `MariaDB`.
|
||||
- Host: `localhost` / `127.0.0.1`.
|
||||
@@ -84,12 +156,15 @@ There are two selectable data sources:
|
||||
|
||||
## Database safety behavior
|
||||
|
||||
- The UI has `Local DB` and `Live DB` settings.
|
||||
- The UI has `Local 9bplus DB`, `Live 9bplus DB` and `Local scrape DB` settings.
|
||||
- The UI sends the selected mode using the `X-Database-Mode` request header.
|
||||
- The backend must resolve that header before every API database operation and route the complete request through the selected source. A Local DB request must use `queryMariaDb`; a Live DB request must use `queryEndpoint`; there is no silent fallback or mixed-source response.
|
||||
- Backend data API routes reject a Local DB request when MariaDB is not configured instead of silently using the endpoint.
|
||||
- Backend data API routes reject a Live DB request when the project is configured only for MariaDB.
|
||||
- Product endpoints read catalog data from the selected catalog database. Mapping endpoints read scraper source mapping from Local scrape DB while still using the selected catalog database only for catalog/product data.
|
||||
- Static HTML/CSS/JS files must remain available even when a database is not configured.
|
||||
- The top status shows the selected source, connection state, permissions and execution mode.
|
||||
- Selecting Live 9bplus DB or Local scrape DB disables catalog write permissions in the UI.
|
||||
|
||||
## Important current files
|
||||
|
||||
@@ -112,11 +187,39 @@ There are two selectable data sources:
|
||||
- `Get product info` remains a separate action.
|
||||
- EAN scan input searches on Enter.
|
||||
- `Find sources` searches once and caches candidates for the current product; a second click reopens cached results.
|
||||
- The source-candidates popup must provide a `Reload` action beside `Close`; `Reload` bypasses only the current product's cached source result and re-runs the source lookup using the current local mapping configuration. It must not switch database modes or write to Live DB.
|
||||
- Source candidates show how they were found, including EAN/reference/name and color context.
|
||||
- Each source row has its own `Get pictures` action.
|
||||
- Picture candidates must show source, image dimensions and the proposed target assignment. The target assignment is not always color: supported strategies are `color`, `size`, `combination` and `manual`. Start by using manual corrections from the user, then learn reusable rules in Local scrape DB.
|
||||
- Scraped pictures and their metadata belong in Local scrape DB, not browser localStorage. Store the binary image in `scrape_image_blobs` with `sha256`, `perceptual_hash`, mime type, dimensions and size. Store each source/target proposal in `scrape_picture_candidates`.
|
||||
- Every picture candidate row must have its own save action. Saving one candidate stores only that selected image candidate into Local scrape DB and never saves the whole source result list.
|
||||
- Picture candidate popups must also provide `Save all` for robot/automatic flows. `Save all` saves every currently visible picture candidate one by one and must not save hidden, stale or unrelated candidates.
|
||||
- Source candidate popups should provide `Get all pictures` for robot/automatic flows. It fetches picture candidates from every currently visible mapped source, then opens the picture candidate popup. Saving happens only in the picture candidate popup.
|
||||
- Use `sha256` to remove exact duplicate image files and a perceptual hash to flag visually similar images from different sources or sizes. Exact duplicates may be collapsed automatically; visual duplicates should remain reviewable until the user approves the rule.
|
||||
- Product source cache is cleared when a new product loads.
|
||||
|
||||
### Picture Scraping Agent
|
||||
|
||||
- Treat picture scraping as a separate internal role: find sources, fetch picture candidates, classify target assignment, detect duplicates and prepare save actions.
|
||||
- The picture scraping agent stores all scraper settings, candidates, hashes, metadata and learned target rules in Local scrape DB (`catalog_scrape`).
|
||||
- Picture files are stored as binary blobs in Local scrape DB with disk-friendly metadata, not in browser localStorage and not in the live catalog database.
|
||||
- The agent starts with user-guided target assignment (`color`, `size`, `combination`, `manual`) and turns repeated corrections into reusable Local scrape DB rules.
|
||||
- The agent may compare Hudy, Idealo and future sources, but it must keep all candidates reviewable until an exact duplicate or learned rule is trusted.
|
||||
- The agent never writes to Live 9bplus DB. Live data may only be read later for comparison when explicitly allowed.
|
||||
|
||||
#### Picture Scraping Skills
|
||||
|
||||
- Source discovery: use manufacturer mapping from Local scrape DB and show exactly which source produced each candidate.
|
||||
- Browser scraping: use the configured browser engine per source and keep the UI responsive while scraping runs.
|
||||
- Image extraction: collect original image URLs, dimensions, mime type, file size and source page URL.
|
||||
- Image quality: product pictures will later be normalized to a `1:1` square, so a portrait/landscape image may be production-ready even when one side is below `800px` (for example `739 x 1435`). Mark images as too small only when the longest side is below `800px`; otherwise show the real size and note when it is OK for `1:1` crop.
|
||||
- Duplicate detection: calculate `sha256` for exact duplicates and perceptual hash for visually similar images.
|
||||
- Target assignment: propose whether pictures belong to `color`, `size`, `combination` or `manual` target, then learn from user corrections.
|
||||
- Persistence: save picture blobs, candidates, hashes, source metadata and learned rules only into Local scrape DB.
|
||||
- Review UI: show picture size, source, target assignment, duplicate state, per-row save and batch save actions.
|
||||
- Hudy picture parsing uses the correct product color variant, checks EANs and extracts large gallery images.
|
||||
- Idealo uses the browser adapter and is intended mainly as a picture source.
|
||||
- Idealo picture parsing must read product gallery images from Splide gallery markup (`splide__track`, `splide__list`, `splide__slide`) and choose the largest available URL from `srcset`, `data-srcset`, `data-large`, `data-original`, `data-src` or `src`. Do not use search-result thumbnails or small preview URLs as production picture candidates. If no Idealo gallery is found, return no Idealo pictures and show the quality/missing warning instead of saving thumbnails.
|
||||
- Image preview is square.
|
||||
- Settings is at the bottom of the left panel; database status is directly above it.
|
||||
|
||||
|
||||
@@ -16,12 +16,17 @@
|
||||
"database": {
|
||||
"driver": "endpoint",
|
||||
"mode": "local",
|
||||
"allowWrites": false,
|
||||
"allowLocalWrites": false,
|
||||
"host": "127.0.0.1",
|
||||
"port": 3306,
|
||||
"name": "catalog_maker_test",
|
||||
"name": "9bplus",
|
||||
"scrapeName": "catalog_scrape",
|
||||
"user": "catalog_maker",
|
||||
"password": "",
|
||||
"connectionLimit": 5
|
||||
},
|
||||
"databaseTool": {
|
||||
"url": "http://127.0.0.1:8080",
|
||||
"browserEngine": "chromium"
|
||||
}
|
||||
}
|
||||
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 1.5 MiB |
Generated
+11637
File diff suppressed because it is too large
Load Diff
@@ -8,6 +8,10 @@
|
||||
"start": "tsx src/cli.ts",
|
||||
"dev": "tsx src/server.ts",
|
||||
"catalog:dry-run": "tsx src/cli.ts catalog-maker --dry-run",
|
||||
"db:create-scrape": "tsx scripts/create-local-scrape-db.ts",
|
||||
"db:migrate-mapping-to-scrape": "tsx scripts/migrate-mapping-to-scrape-db.ts",
|
||||
"adminer": "tsx scripts/start-adminer.ts",
|
||||
"dbgate": "tsx scripts/start-dbgate.ts",
|
||||
"test": "tsx --test test/*.test.ts",
|
||||
"test:ui": "playwright test",
|
||||
"test:e2e": "playwright test tests/ui/database.e2e.spec.ts",
|
||||
@@ -26,6 +30,7 @@
|
||||
"devDependencies": {
|
||||
"@tailwindcss/cli": "^4.3.3",
|
||||
"@types/node": "^26.1.2",
|
||||
"dbgate-serve": "^7.2.3",
|
||||
"tailwindcss": "^4.3.3",
|
||||
"tsx": "^4.23.7",
|
||||
"typescript": "^7.0.2"
|
||||
|
||||
+648
-76
File diff suppressed because it is too large
Load Diff
+686
-75
File diff suppressed because it is too large
Load Diff
@@ -12,4 +12,9 @@
|
||||
<symbol id="erase" viewBox="0 0 24 24"><path d="m4 15 8-8a2 2 0 0 1 2.8 0l4.2 4.2a2 2 0 0 1 0 2.8l-5 5H7zM13 19h7" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.8"/></symbol>
|
||||
<symbol id="moon" viewBox="0 0 24 24"><path d="M20 15.5A8.5 8.5 0 0 1 8.5 4 8.5 8.5 0 1 0 20 15.5Z" fill="none" stroke="currentColor" stroke-linejoin="round" stroke-width="1.8"/></symbol>
|
||||
<symbol id="settings" viewBox="0 0 24 24"><path d="M12 8.5a3.5 3.5 0 1 0 0 7 3.5 3.5 0 0 0 0-7Z" fill="none" stroke="currentColor" stroke-width="1.8"/><path d="m19 13.5 1.2 1-.2 2-1.5.8-.3 1.6-1.8.9-1.4-.7-1.5.7-1.8-.9-.3-1.6-1.5-.8-.2-2 1.2-1-.1-1.5-1.2-1 .2-2 1.5-.8.3-1.6 1.8-.9 1.4.7 1.5-.7 1.8.9.3 1.6 1.5.8.2 2-1.2 1.1.1 1.5Z" fill="none" stroke="currentColor" stroke-linejoin="round" stroke-width="1.4"/></symbol>
|
||||
<symbol id="globe" viewBox="0 0 24 24"><circle cx="12" cy="12" r="9" fill="none" stroke="currentColor" stroke-width="1.8"/><path d="M3 12h18M12 3c2.3 2.4 3.4 5.4 3.4 9S14.3 18.6 12 21M12 3c-2.3 2.4-3.4 5.4-3.4 9S9.7 18.6 12 21" fill="none" stroke="currentColor" stroke-width="1.4"/></symbol>
|
||||
<symbol id="tag" viewBox="0 0 24 24"><path d="m4 5 8.5-.5L20 12l-8 8-7.5-7.5L4 5Z" fill="none" stroke="currentColor" stroke-linejoin="round" stroke-width="1.8"/><circle cx="8" cy="8" r="1.2" fill="currentColor"/></symbol>
|
||||
<symbol id="close" viewBox="0 0 24 24"><path d="m6 6 12 12M18 6 6 18" fill="none" stroke="currentColor" stroke-linecap="round" stroke-width="1.8"/></symbol>
|
||||
<symbol id="refresh" viewBox="0 0 24 24"><path d="M20 11a8 8 0 0 0-14.8-3L3 11m0 0V5m0 6h6M4 13a8 8 0 0 0 14.8 3L21 13m0 0v6m0-6h-6" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.8"/></symbol>
|
||||
<symbol id="external" viewBox="0 0 24 24"><path d="M14 4h6v6M13 11l7-7M20 14v5a1 1 0 0 1-1 1H5a1 1 0 0 1-1-1V5a1 1 0 0 1 1-1h5" fill="none" stroke="currentColor" stroke-linecap="round" stroke-linejoin="round" stroke-width="1.8"/></symbol>
|
||||
</svg>
|
||||
|
||||
|
Before Width: | Height: | Size: 3.0 KiB After Width: | Height: | Size: 4.1 KiB |
+122
-38
@@ -10,7 +10,19 @@
|
||||
</head>
|
||||
<body>
|
||||
<main class="catalog-screen">
|
||||
<aside class="control-panel">
|
||||
<button class="sidebar-open-button panel-control-button" id="sidebarOpenButton" type="button" aria-controls="sidebarDrawer" aria-expanded="false" aria-label="Open sidebar" title="Open sidebar">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#menu"></use></svg>
|
||||
</button>
|
||||
|
||||
<button class="sidebar-overlay" id="sidebarOverlay" type="button" aria-label="Close sidebar"></button>
|
||||
|
||||
<aside class="control-panel sidebar-drawer" id="sidebarDrawer" aria-label="Catalog controls">
|
||||
<div class="panel-header sidebar-drawer-header">
|
||||
<span class="sidebar-drawer-title">Catalog controls</span>
|
||||
<button class="sidebar-close-button panel-control-button" id="sidebarCloseButton" type="button" aria-label="Close sidebar" title="Close sidebar">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#close"></use></svg>
|
||||
</button>
|
||||
</div>
|
||||
<div class="record-id" id="recordId"> </div>
|
||||
|
||||
<div class="arrow-row">
|
||||
@@ -18,12 +30,6 @@
|
||||
<button class="arrow-button arrow-button-next" id="nextProductButton" type="button" title="Next product" aria-label="Next product"></button>
|
||||
</div>
|
||||
|
||||
<button class="drawer-trigger" id="parserDrawerToggle" type="button" aria-expanded="true" aria-controls="parserDrawer">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#sliders"></use></svg>
|
||||
<span>Parser tools</span>
|
||||
<span class="drawer-chevron" aria-hidden="true">⌃</span>
|
||||
</button>
|
||||
|
||||
<div class="sidebar-section parser-drawer" id="parserDrawer">
|
||||
<div class="parser-actions" aria-label="Parser actions">
|
||||
<button class="action-button" id="findSourcesButton" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#search"></use></svg><span>Find sources</span></button>
|
||||
@@ -44,52 +50,84 @@
|
||||
<span>No picture</span>
|
||||
</div>
|
||||
|
||||
<select class="input-like" id="languageSelect" aria-label="Language">
|
||||
<div class="select-control">
|
||||
<div class="select-display" aria-hidden="true">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#globe"></use></svg>
|
||||
<span id="languageValue">Select language</span>
|
||||
</div>
|
||||
<select class="input-like language-native-select" id="languageSelect" aria-label="Language">
|
||||
<option value="">Select language</option>
|
||||
</select>
|
||||
<span class="select-chevron" aria-hidden="true"></span>
|
||||
</div>
|
||||
|
||||
<select class="input-like" id="manufacturerSelect" aria-label="Manufacturer">
|
||||
<div class="manufacturer-picker" id="manufacturerPicker">
|
||||
<button class="manufacturer-trigger" id="manufacturerTrigger" type="button" aria-haspopup="listbox" aria-expanded="false">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#tag"></use></svg>
|
||||
<span id="manufacturerValue">Select manufacturer</span>
|
||||
<span class="select-chevron picker-chevron" aria-hidden="true"></span>
|
||||
</button>
|
||||
<div class="manufacturer-menu" id="manufacturerMenu" hidden>
|
||||
<input class="manufacturer-search" id="manufacturerSearch" type="search" placeholder="Search manufacturer..." aria-label="Search manufacturer" />
|
||||
<div class="manufacturer-options" id="manufacturerOptions" role="listbox"></div>
|
||||
</div>
|
||||
</div>
|
||||
<div class="native-manufacturer-host" hidden>
|
||||
<select class="input-like native-manufacturer-select" id="manufacturerSelect" aria-label="Manufacturer">
|
||||
<option value="">Select manufacturer</option>
|
||||
</select>
|
||||
</div>
|
||||
|
||||
<div class="mapping-box" id="mappingBox">
|
||||
<div class="mapping-box sidebar-mapping-box" id="mappingBox">
|
||||
<details class="mapping-details" id="mappingDetails">
|
||||
<summary>
|
||||
<span class="mapping-state is-missing" id="mappingState">Mapping not loaded</span>
|
||||
<summary class="mapping-summary">
|
||||
<span class="mapping-label">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#layers"></use></svg>
|
||||
<span class="mapping-title">Mapping</span>
|
||||
<span class="mapping-state-dot is-missing" id="mappingState" role="status" aria-label="Mapping not loaded"></span>
|
||||
</span>
|
||||
<span class="mapping-toggle" aria-hidden="true"></span>
|
||||
</summary>
|
||||
<div class="mapping-panel" role="status">
|
||||
<div class="mapping-detail-status" id="mappingStatusText">Mapping not loaded</div>
|
||||
<div class="mapping-list" id="mappingList"></div>
|
||||
</div>
|
||||
</details>
|
||||
</div>
|
||||
|
||||
<button class="action-button primary" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#menu"></use></svg><span>Show menu (20)</span></button>
|
||||
<button class="action-button" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#scan"></use></svg><span>Scraper - 03</span></button>
|
||||
<button class="action-button" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#erase"></use></svg><span>BG remover - 04</span></button>
|
||||
<div class="sidebar-footer">
|
||||
<div class="app-status">
|
||||
<span class="safe-mode" id="databaseStatus">DB read-only</span>
|
||||
</div>
|
||||
<button class="action-button primary" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#menu"></use></svg><span>Show menu (20)</span></button>
|
||||
<button class="action-button" id="themeToggle" type="button" aria-pressed="false"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#moon"></use></svg><span>Dark theme</span></button>
|
||||
<button class="action-button" id="settingsButton" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#settings"></use></svg><span>Settings</span></button>
|
||||
<div class="app-status">
|
||||
<span class="database-status-content">
|
||||
<span class="database-state-dot is-pending" id="databaseStateDot" aria-hidden="true"></span>
|
||||
<span class="safe-mode" id="databaseStatus">DB read-only</span>
|
||||
</span>
|
||||
</div>
|
||||
</div>
|
||||
</aside>
|
||||
|
||||
<section class="product-panel">
|
||||
<section class="product-panel secondary-panel" aria-label="Product details">
|
||||
<div class="panel-header column-header">
|
||||
<span>Product details</span>
|
||||
<button class="column-toggle" id="secondaryPanelToggle" type="button" aria-expanded="true" aria-controls="fieldGrid" aria-label="Collapse product details" title="Collapse product details">‹</button>
|
||||
</div>
|
||||
<div class="field-grid" id="fieldGrid"></div>
|
||||
</section>
|
||||
|
||||
<section class="data-panel">
|
||||
<section class="data-panel main-workspace" aria-label="Catalog workspace">
|
||||
<div class="panel-header column-header workspace-header">
|
||||
<span>Main workspace</span>
|
||||
<button class="column-toggle" id="mainWorkspaceToggle" type="button" aria-expanded="true" aria-controls="variantGrid" aria-label="Collapse main workspace" title="Collapse main workspace">‹</button>
|
||||
</div>
|
||||
<div class="table-wrap">
|
||||
<table class="variant-table">
|
||||
<thead>
|
||||
<tr id="variantHeader"></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td class="empty-row" id="emptyTableMessage" colspan="13">Select manufacturer to load data.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<div class="variant-grid" id="variantGrid" role="table" aria-label="Product combinations">
|
||||
<div class="variant-header" id="variantHeader" role="row"></div>
|
||||
<div class="variant-rows" id="variantRows" role="rowgroup">
|
||||
<div class="empty-row" id="emptyTableMessage" role="row">Select manufacturer to load data.</div>
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
</section>
|
||||
</main>
|
||||
@@ -98,7 +136,7 @@
|
||||
<section class="suggested-dialog" aria-labelledby="suggestedTitle">
|
||||
<header class="suggested-header">
|
||||
<h2 id="suggestedTitle">Sugested product for</h2>
|
||||
<button class="modal-close" id="suggestedCloseButton" type="button">Close</button>
|
||||
<button class="modal-close" id="suggestedCloseButton" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#close"></use></svg><span>Close</span></button>
|
||||
</header>
|
||||
<div class="suggested-list" id="suggestedList"></div>
|
||||
</section>
|
||||
@@ -108,7 +146,11 @@
|
||||
<section class="source-dialog" aria-labelledby="sourceTitle">
|
||||
<header class="suggested-header">
|
||||
<h2 id="sourceTitle">Source candidates</h2>
|
||||
<button class="modal-close" id="sourceCloseButton" type="button">Close</button>
|
||||
<div class="source-dialog-actions">
|
||||
<button class="modal-close" id="sourceBatchActionButton" type="button" hidden><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#image"></use></svg><span>Get all pictures</span></button>
|
||||
<button class="modal-close" id="sourceReloadButton" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#refresh"></use></svg><span>Reload</span></button>
|
||||
<button class="modal-close" id="sourceCloseButton" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#close"></use></svg><span>Close</span></button>
|
||||
</div>
|
||||
</header>
|
||||
<div class="source-list" id="sourceList"></div>
|
||||
</section>
|
||||
@@ -118,18 +160,32 @@
|
||||
<section class="settings-dialog" aria-labelledby="settingsTitle">
|
||||
<header class="suggested-header">
|
||||
<h2 id="settingsTitle">Settings</h2>
|
||||
<button class="modal-close" id="settingsCloseButton" type="button">Close</button>
|
||||
<button class="modal-close" id="settingsCloseButton" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#close"></use></svg><span>Close</span></button>
|
||||
</header>
|
||||
<div class="settings-content">
|
||||
<fieldset class="settings-group">
|
||||
<legend>Database</legend>
|
||||
<label class="settings-option">
|
||||
<input type="radio" name="databaseMode" value="local" checked />
|
||||
<span>Local DB</span>
|
||||
<span>Local 9bplus DB</span>
|
||||
</label>
|
||||
<label class="settings-option">
|
||||
<input type="radio" name="databaseMode" value="live" />
|
||||
<span>Live DB</span>
|
||||
<span>Live 9bplus DB</span>
|
||||
</label>
|
||||
<label class="settings-option">
|
||||
<input type="radio" name="databaseMode" value="scrape" />
|
||||
<span>Local scrape DB</span>
|
||||
<span class="settings-tool-actions">
|
||||
<button class="settings-inline-button" id="openAdminerButton" type="button">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#external"></use></svg>
|
||||
<span>Adminer</span>
|
||||
</button>
|
||||
<button class="settings-inline-button" id="openDbGateButton" type="button">
|
||||
<svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#external"></use></svg>
|
||||
<span>DBGate</span>
|
||||
</button>
|
||||
</span>
|
||||
</label>
|
||||
<div class="local-db-fields" id="localDbFields">
|
||||
<label class="settings-field">
|
||||
@@ -152,10 +208,38 @@
|
||||
<span>Password</span>
|
||||
<input id="localDbPassword" type="text" autocomplete="off" />
|
||||
</label>
|
||||
<button class="settings-test-button" id="testLocalDbButton" type="button">Test connection</button>
|
||||
<button class="settings-test-button" id="testLocalDbButton" type="button"><svg class="sidebar-icon" aria-hidden="true"><use href="/icons.svg#refresh"></use></svg><span>Test connection</span></button>
|
||||
<p class="settings-note" id="localDbTestResult">Connection not tested.</p>
|
||||
</div>
|
||||
<p class="settings-note" id="databaseModeNote">Local test database selected.</p>
|
||||
<p class="settings-note" id="databaseModeNote">Local 9bplus database selected.</p>
|
||||
<div class="adminer-login-hint" id="adminerLoginHint" aria-label="Adminer login details">
|
||||
<div class="adminer-login-row"><span>System</span><strong>MySQL</strong></div>
|
||||
<div class="adminer-login-row"><span>Server</span><strong id="adminerServerValue">127.0.0.1:3306</strong></div>
|
||||
<div class="adminer-login-row"><span>User</span><strong id="adminerUserValue">root</strong></div>
|
||||
<div class="adminer-login-row"><span>Password</span><strong>DB_PASSWORD in .env</strong></div>
|
||||
<div class="adminer-login-row"><span>Database</span><strong id="adminerDatabaseValue">9bplus / catalog_scrape</strong></div>
|
||||
</div>
|
||||
</fieldset>
|
||||
|
||||
<fieldset class="settings-group">
|
||||
<legend>Appearance</legend>
|
||||
<label class="settings-field">
|
||||
<span>Light contrast</span>
|
||||
<select class="settings-select" id="lightContrastSelect">
|
||||
<option value="low">Soft</option>
|
||||
<option value="medium" selected>Balanced</option>
|
||||
<option value="high">High</option>
|
||||
</select>
|
||||
</label>
|
||||
<label class="settings-field">
|
||||
<span>Dark contrast</span>
|
||||
<select class="settings-select" id="darkContrastSelect">
|
||||
<option value="low">Soft</option>
|
||||
<option value="medium" selected>Balanced</option>
|
||||
<option value="high">High</option>
|
||||
</select>
|
||||
</label>
|
||||
<p class="settings-note">Controls the contrast of the left sidebar and its controls.</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset class="settings-group">
|
||||
@@ -176,7 +260,7 @@
|
||||
<input type="checkbox" id="allowDeleteToggle" />
|
||||
<span>Allow delete</span>
|
||||
</label>
|
||||
<p class="settings-note">Live DB is always read-only. Write permissions apply only to Local DB.</p>
|
||||
<p class="settings-note">Write permissions are saved separately for Local 9bplus DB and Local scrape DB. Live DB stays read-only.</p>
|
||||
</fieldset>
|
||||
|
||||
<fieldset class="settings-group">
|
||||
|
||||
+1
-1
File diff suppressed because one or more lines are too long
@@ -0,0 +1,33 @@
|
||||
import fs from "node:fs/promises";
|
||||
import mariadb from "mariadb";
|
||||
|
||||
async function readEnv() {
|
||||
return Object.fromEntries(
|
||||
(await fs.readFile(".env", "utf8"))
|
||||
.split(/\r?\n/)
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line && !line.startsWith("#") && line.includes("="))
|
||||
.map((line) => {
|
||||
const index = line.indexOf("=");
|
||||
return [line.slice(0, index), line.slice(index + 1)];
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
const env = await readEnv();
|
||||
const connection = await mariadb.createConnection({
|
||||
host: env.DB_HOST || "127.0.0.1",
|
||||
port: Number(env.DB_PORT || 3306),
|
||||
user: env.DB_USER || "root",
|
||||
password: env.DB_PASSWORD || "",
|
||||
});
|
||||
|
||||
try {
|
||||
const sql = await fs.readFile("sql/local-scrape-db.sql", "utf8");
|
||||
for (const statement of sql.split(/;\s*(?:\r?\n|$)/).map((item) => item.trim()).filter(Boolean)) {
|
||||
await connection.query(statement);
|
||||
}
|
||||
console.log("Local scrape database and schema are ready.");
|
||||
} finally {
|
||||
await connection.end();
|
||||
}
|
||||
@@ -0,0 +1,270 @@
|
||||
import fs from "node:fs/promises";
|
||||
import mariadb from "mariadb";
|
||||
|
||||
async function readEnv() {
|
||||
return Object.fromEntries(
|
||||
(await fs.readFile(".env", "utf8"))
|
||||
.split(/\r?\n/)
|
||||
.map((line) => line.trim())
|
||||
.filter((line) => line && !line.startsWith("#") && line.includes("="))
|
||||
.map((line) => {
|
||||
const index = line.indexOf("=");
|
||||
return [line.slice(0, index).trim(), line.slice(index + 1).trim()];
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async function queryRows(connection, sql, params = []) {
|
||||
const rows = await connection.query(sql, params);
|
||||
return Array.isArray(rows) ? rows.filter((row) => row && typeof row === "object" && !("meta" in row)) : [];
|
||||
}
|
||||
|
||||
async function tableExists(connection, schema, table) {
|
||||
const rows = await queryRows(
|
||||
connection,
|
||||
"SELECT 1 FROM information_schema.tables WHERE table_schema = ? AND table_name = ? LIMIT 1",
|
||||
[schema, table],
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
async function columnExists(connection, schema, table, column) {
|
||||
const rows = await queryRows(
|
||||
connection,
|
||||
"SELECT 1 FROM information_schema.columns WHERE table_schema = ? AND table_name = ? AND column_name = ? LIMIT 1",
|
||||
[schema, table, column],
|
||||
);
|
||||
return rows.length > 0;
|
||||
}
|
||||
|
||||
async function addColumnIfMissing(connection, schema, table, column, definition) {
|
||||
if (await columnExists(connection, schema, table, column)) return;
|
||||
await connection.query(`ALTER TABLE \`${schema}\`.\`${table}\` ADD COLUMN \`${column}\` ${definition}`);
|
||||
}
|
||||
|
||||
const env = await readEnv();
|
||||
const catalogDb = env.DB_NAME || "9bplus";
|
||||
const scrapeDb = env.SCRAPE_DB_NAME || "catalog_scrape";
|
||||
const connection = await mariadb.createConnection({
|
||||
host: env.DB_HOST || "127.0.0.1",
|
||||
port: Number(env.DB_PORT || 3306),
|
||||
user: env.DB_USER || "root",
|
||||
password: env.DB_PASSWORD || "",
|
||||
multipleStatements: false,
|
||||
bigIntAsNumber: true,
|
||||
});
|
||||
|
||||
try {
|
||||
await connection.query(`CREATE DATABASE IF NOT EXISTS \`${scrapeDb}\` CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci`);
|
||||
const schemaSql = await fs.readFile("sql/local-scrape-db.sql", "utf8");
|
||||
const schemaStatements = schemaSql
|
||||
.split(/;\s*(?:\r?\n|$)/)
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean);
|
||||
for (const statement of schemaStatements) {
|
||||
await connection.query(statement);
|
||||
}
|
||||
|
||||
await addColumnIfMissing(connection, scrapeDb, "scrape_sources", "legacy_catalog_source_id", "BIGINT UNSIGNED NULL");
|
||||
await addColumnIfMissing(connection, scrapeDb, "scrape_sources", "fallback_source_type", "VARCHAR(64) NOT NULL DEFAULT 'none'");
|
||||
await addColumnIfMissing(connection, scrapeDb, "scrape_sources", "access_mode", "VARCHAR(64) NOT NULL DEFAULT 'browser'");
|
||||
await addColumnIfMissing(connection, scrapeDb, "scrape_sources", "browser_engine", "VARCHAR(32) NOT NULL DEFAULT 'default'");
|
||||
await addColumnIfMissing(connection, scrapeDb, "scrape_sources", "browser_headless", "TINYINT(1) NOT NULL DEFAULT 0");
|
||||
await addColumnIfMissing(connection, scrapeDb, "scrape_sources", "wait_after_load_ms", "INT UNSIGNED NOT NULL DEFAULT 3000");
|
||||
await addColumnIfMissing(connection, scrapeDb, "scrape_sources", "note", "TEXT NULL");
|
||||
|
||||
const hasCatalogSourceTable = await tableExists(connection, catalogDb, "ps_product_catalog_source");
|
||||
const hasCatalogManufacturerSourceTable = await tableExists(connection, catalogDb, "ps_product_catalog_manufacturer_source");
|
||||
|
||||
if (hasCatalogSourceTable) {
|
||||
await connection.query(`
|
||||
INSERT INTO \`${scrapeDb}\`.ps_product_catalog_source (
|
||||
id_product_catalog_source,
|
||||
source_key,
|
||||
source_name,
|
||||
source_type,
|
||||
base_url,
|
||||
enabled,
|
||||
priority,
|
||||
search_primary_key,
|
||||
search_fallback_keys,
|
||||
fallback_source_type,
|
||||
search_url_template,
|
||||
fallback_url_template,
|
||||
search_context_keys,
|
||||
access_mode,
|
||||
browser_engine,
|
||||
browser_headless,
|
||||
wait_after_load_ms,
|
||||
note
|
||||
)
|
||||
SELECT
|
||||
id_product_catalog_source,
|
||||
source_key,
|
||||
source_name,
|
||||
source_type,
|
||||
base_url,
|
||||
enabled,
|
||||
priority,
|
||||
search_primary_key,
|
||||
search_fallback_keys,
|
||||
COALESCE(fallback_source_type, 'none'),
|
||||
search_url_template,
|
||||
fallback_url_template,
|
||||
search_context_keys,
|
||||
COALESCE(access_mode, 'browser'),
|
||||
COALESCE(browser_engine, 'default'),
|
||||
COALESCE(browser_headless, 0),
|
||||
COALESCE(wait_after_load_ms, 3000),
|
||||
note
|
||||
FROM \`${catalogDb}\`.ps_product_catalog_source
|
||||
ON DUPLICATE KEY UPDATE
|
||||
source_key = VALUES(source_key),
|
||||
source_name = VALUES(source_name),
|
||||
source_type = VALUES(source_type),
|
||||
base_url = VALUES(base_url),
|
||||
enabled = VALUES(enabled),
|
||||
priority = VALUES(priority),
|
||||
search_primary_key = VALUES(search_primary_key),
|
||||
search_fallback_keys = VALUES(search_fallback_keys),
|
||||
fallback_source_type = VALUES(fallback_source_type),
|
||||
search_url_template = VALUES(search_url_template),
|
||||
fallback_url_template = VALUES(fallback_url_template),
|
||||
search_context_keys = VALUES(search_context_keys),
|
||||
access_mode = VALUES(access_mode),
|
||||
browser_engine = VALUES(browser_engine),
|
||||
browser_headless = VALUES(browser_headless),
|
||||
wait_after_load_ms = VALUES(wait_after_load_ms),
|
||||
note = VALUES(note)
|
||||
`);
|
||||
}
|
||||
|
||||
if (hasCatalogManufacturerSourceTable) {
|
||||
await connection.query(`
|
||||
INSERT INTO \`${scrapeDb}\`.ps_product_catalog_manufacturer_source (
|
||||
id_product_catalog_manufacturer_source,
|
||||
id_manufacturer,
|
||||
id_product_catalog_source,
|
||||
enabled,
|
||||
priority,
|
||||
note
|
||||
)
|
||||
SELECT
|
||||
id_product_catalog_manufacturer_source,
|
||||
id_manufacturer,
|
||||
id_product_catalog_source,
|
||||
enabled,
|
||||
priority,
|
||||
note
|
||||
FROM \`${catalogDb}\`.ps_product_catalog_manufacturer_source
|
||||
ON DUPLICATE KEY UPDATE
|
||||
id_manufacturer = VALUES(id_manufacturer),
|
||||
id_product_catalog_source = VALUES(id_product_catalog_source),
|
||||
enabled = VALUES(enabled),
|
||||
priority = VALUES(priority),
|
||||
note = VALUES(note)
|
||||
`);
|
||||
}
|
||||
|
||||
if (!(await tableExists(connection, scrapeDb, "ps_product_catalog_source"))) {
|
||||
throw new Error(`Source table ${scrapeDb}.ps_product_catalog_source does not exist.`);
|
||||
}
|
||||
if (!(await tableExists(connection, scrapeDb, "ps_product_catalog_manufacturer_source"))) {
|
||||
throw new Error(`Mapping table ${scrapeDb}.ps_product_catalog_manufacturer_source does not exist.`);
|
||||
}
|
||||
|
||||
await connection.query(`
|
||||
INSERT INTO \`${scrapeDb}\`.scrape_sources (
|
||||
legacy_catalog_source_id,
|
||||
source_key,
|
||||
source_name,
|
||||
source_type,
|
||||
base_url,
|
||||
enabled,
|
||||
priority,
|
||||
search_primary_key,
|
||||
search_fallback_keys,
|
||||
fallback_source_type,
|
||||
search_url_template,
|
||||
fallback_url_template,
|
||||
search_context_keys,
|
||||
access_mode,
|
||||
browser_engine,
|
||||
browser_headless,
|
||||
wait_after_load_ms,
|
||||
note
|
||||
)
|
||||
SELECT
|
||||
id_product_catalog_source,
|
||||
source_key,
|
||||
source_name,
|
||||
source_type,
|
||||
base_url,
|
||||
enabled,
|
||||
priority,
|
||||
search_primary_key,
|
||||
search_fallback_keys,
|
||||
COALESCE(fallback_source_type, 'none'),
|
||||
search_url_template,
|
||||
fallback_url_template,
|
||||
search_context_keys,
|
||||
COALESCE(access_mode, 'browser'),
|
||||
COALESCE(browser_engine, 'default'),
|
||||
COALESCE(browser_headless, 0),
|
||||
COALESCE(wait_after_load_ms, 3000),
|
||||
note
|
||||
FROM \`${scrapeDb}\`.ps_product_catalog_source
|
||||
ON DUPLICATE KEY UPDATE
|
||||
source_key = VALUES(source_key),
|
||||
source_name = VALUES(source_name),
|
||||
source_type = VALUES(source_type),
|
||||
base_url = VALUES(base_url),
|
||||
enabled = VALUES(enabled),
|
||||
priority = VALUES(priority),
|
||||
search_primary_key = VALUES(search_primary_key),
|
||||
search_fallback_keys = VALUES(search_fallback_keys),
|
||||
fallback_source_type = VALUES(fallback_source_type),
|
||||
search_url_template = VALUES(search_url_template),
|
||||
fallback_url_template = VALUES(fallback_url_template),
|
||||
search_context_keys = VALUES(search_context_keys),
|
||||
access_mode = VALUES(access_mode),
|
||||
browser_engine = VALUES(browser_engine),
|
||||
browser_headless = VALUES(browser_headless),
|
||||
wait_after_load_ms = VALUES(wait_after_load_ms),
|
||||
note = VALUES(note)
|
||||
`);
|
||||
|
||||
await connection.query(`
|
||||
INSERT INTO \`${scrapeDb}\`.scrape_manufacturer_sources (
|
||||
manufacturer_id,
|
||||
source_id,
|
||||
enabled,
|
||||
priority
|
||||
)
|
||||
SELECT
|
||||
ms.id_manufacturer,
|
||||
ss.id,
|
||||
ms.enabled,
|
||||
ms.priority
|
||||
FROM \`${scrapeDb}\`.ps_product_catalog_manufacturer_source ms
|
||||
JOIN \`${scrapeDb}\`.scrape_sources ss
|
||||
ON ss.legacy_catalog_source_id = ms.id_product_catalog_source
|
||||
ON DUPLICATE KEY UPDATE
|
||||
enabled = VALUES(enabled),
|
||||
priority = VALUES(priority)
|
||||
`);
|
||||
|
||||
for (const statement of schemaStatements.filter(isCapabilitySeedStatement)) {
|
||||
await connection.query(statement);
|
||||
}
|
||||
|
||||
const sources = await queryRows(connection, `SELECT COUNT(*) AS total FROM \`${scrapeDb}\`.scrape_sources`);
|
||||
const mappings = await queryRows(connection, `SELECT COUNT(*) AS total FROM \`${scrapeDb}\`.scrape_manufacturer_sources`);
|
||||
console.log(`Scrape mapping migration ready: ${sources[0]?.total || 0} sources, ${mappings[0]?.total || 0} manufacturer mappings.`);
|
||||
} finally {
|
||||
await connection.end();
|
||||
}
|
||||
|
||||
function isCapabilitySeedStatement(statement) {
|
||||
return /^INSERT INTO `scrape_(source_capabilities|manufacturer_source_capabilities)`/i.test(statement);
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
import { createWriteStream, existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
|
||||
import { get } from "node:https";
|
||||
import { dirname, resolve } from "node:path";
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
|
||||
const adminerFile = resolve("tools/adminer/adminer.php");
|
||||
const localPhp = resolve("tools/php/php.exe");
|
||||
const host = process.env.ADMINER_HOST || "127.0.0.1";
|
||||
const port = process.env.ADMINER_PORT || "8080";
|
||||
const adminerDownloadUrl = process.env.ADMINER_DOWNLOAD_URL || "https://www.adminer.org/latest.php";
|
||||
const phpCommand = existsSync(localPhp) ? localPhp : "php";
|
||||
const usesLocalPhp = existsSync(localPhp);
|
||||
|
||||
if (!commandExists(phpCommand)) {
|
||||
console.error("Adminer needs PHP CLI. Install PHP first, then run npm run adminer again.");
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
if (usesLocalPhp) {
|
||||
ensureLocalPhpMysqlExtensions();
|
||||
}
|
||||
|
||||
if (!existsSync(adminerFile)) {
|
||||
mkdirSync(dirname(adminerFile), { recursive: true });
|
||||
await downloadFile(adminerDownloadUrl, adminerFile);
|
||||
}
|
||||
|
||||
const child = spawn(phpCommand, ["-S", `${host}:${port}`, adminerFile], {
|
||||
stdio: "inherit",
|
||||
shell: true,
|
||||
});
|
||||
|
||||
child.on("exit", (code, signal) => {
|
||||
if (signal) {
|
||||
process.kill(process.pid, signal);
|
||||
return;
|
||||
}
|
||||
|
||||
process.exit(code ?? 0);
|
||||
});
|
||||
|
||||
function commandExists(command: string): boolean {
|
||||
const result = spawnSync(command, ["-v"], { stdio: "ignore", shell: true });
|
||||
return result.status === 0;
|
||||
}
|
||||
|
||||
function ensureLocalPhpMysqlExtensions(): void {
|
||||
const phpDir = dirname(localPhp);
|
||||
const phpIni = resolve(phpDir, "php.ini");
|
||||
const template = resolve(phpDir, "php.ini-production");
|
||||
const source = existsSync(phpIni) ? phpIni : template;
|
||||
|
||||
if (!existsSync(source)) {
|
||||
console.warn("Local PHP is missing php.ini-production; Adminer may not have MariaDB extensions enabled.");
|
||||
return;
|
||||
}
|
||||
|
||||
let contents = readFileSync(source, "utf8");
|
||||
contents = ensureIniLine(contents, /^;?\s*extension_dir\s*=.*$/m, 'extension_dir = "ext"');
|
||||
contents = ensureIniLine(contents, /^;?\s*extension\s*=\s*mysqli\s*$/m, "extension=mysqli");
|
||||
contents = ensureIniLine(contents, /^;?\s*extension\s*=\s*pdo_mysql\s*$/m, "extension=pdo_mysql");
|
||||
writeFileSync(phpIni, contents);
|
||||
}
|
||||
|
||||
function ensureIniLine(contents: string, pattern: RegExp, line: string): string {
|
||||
if (pattern.test(contents)) return contents.replace(pattern, line);
|
||||
return `${contents.trimEnd()}\n${line}\n`;
|
||||
}
|
||||
|
||||
function downloadFile(url: string, target: string): Promise<void> {
|
||||
return new Promise((resolveDownload, rejectDownload) => {
|
||||
const request = get(url, (response) => {
|
||||
if (response.statusCode && response.statusCode >= 300 && response.statusCode < 400 && response.headers.location) {
|
||||
downloadFile(response.headers.location, target).then(resolveDownload, rejectDownload);
|
||||
return;
|
||||
}
|
||||
|
||||
if (response.statusCode !== 200) {
|
||||
rejectDownload(new Error(`Adminer download failed with HTTP ${response.statusCode}`));
|
||||
return;
|
||||
}
|
||||
|
||||
const file = createWriteStream(target);
|
||||
response.pipe(file);
|
||||
file.on("finish", () => {
|
||||
file.close();
|
||||
resolveDownload();
|
||||
});
|
||||
file.on("error", rejectDownload);
|
||||
});
|
||||
|
||||
request.on("error", rejectDownload);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
import { existsSync } from "node:fs";
|
||||
import { spawn } from "node:child_process";
|
||||
import { resolve } from "node:path";
|
||||
import { loadConfig } from "../src/config.ts";
|
||||
|
||||
const config = loadConfig();
|
||||
const dbgateBin = resolve("node_modules/.bin/dbgate-serve.cmd");
|
||||
const command = existsSync(dbgateBin) ? dbgateBin : "dbgate-serve";
|
||||
const host = config.database.host || "127.0.0.1";
|
||||
const port = String(config.database.port || 3306);
|
||||
const user = config.database.user || "root";
|
||||
const password = config.database.password || "";
|
||||
const catalogDatabase = config.database.name || "9bplus";
|
||||
const scrapeDatabase = config.database.scrapeName || "catalog_scrape";
|
||||
const dbgateUrl = new URL(config.dbGate.url || "http://127.0.0.1:3000");
|
||||
|
||||
const env = {
|
||||
...process.env,
|
||||
PORT: dbgateUrl.port || "3000",
|
||||
SKIP_ALL_AUTH: "1",
|
||||
CONNECTIONS: "local_9bplus,local_scrape",
|
||||
ENGINE_local_9bplus: "mysql@dbgate-plugin-mysql",
|
||||
LABEL_local_9bplus: "Local 9bplus DB",
|
||||
SERVER_local_9bplus: host,
|
||||
PORT_local_9bplus: port,
|
||||
USER_local_9bplus: user,
|
||||
PASSWORD_local_9bplus: password,
|
||||
DATABASE_local_9bplus: catalogDatabase,
|
||||
ENGINE_local_scrape: "mysql@dbgate-plugin-mysql",
|
||||
LABEL_local_scrape: "Local scrape DB",
|
||||
SERVER_local_scrape: host,
|
||||
PORT_local_scrape: port,
|
||||
USER_local_scrape: user,
|
||||
PASSWORD_local_scrape: password,
|
||||
DATABASE_local_scrape: scrapeDatabase,
|
||||
};
|
||||
|
||||
console.log(`Starting DBGate at ${dbgateUrl.origin}`);
|
||||
console.log(`Configured connections: Local 9bplus DB, Local scrape DB`);
|
||||
|
||||
const child = spawn(command, [], {
|
||||
env,
|
||||
stdio: "inherit",
|
||||
shell: true,
|
||||
});
|
||||
|
||||
child.on("exit", (code, signal) => {
|
||||
if (signal) {
|
||||
process.kill(process.pid, signal);
|
||||
return;
|
||||
}
|
||||
|
||||
process.exit(code ?? 0);
|
||||
});
|
||||
@@ -0,0 +1,475 @@
|
||||
CREATE DATABASE IF NOT EXISTS `catalog_scrape`
|
||||
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
|
||||
|
||||
USE `catalog_scrape`;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `ps_product_catalog_source` (
|
||||
`id_product_catalog_source` INT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`source_key` VARCHAR(64) NOT NULL,
|
||||
`source_name` VARCHAR(128) NOT NULL,
|
||||
`source_type` VARCHAR(32) NOT NULL DEFAULT '',
|
||||
`base_url` VARCHAR(512) NOT NULL,
|
||||
`enabled` TINYINT(1) NOT NULL DEFAULT 1,
|
||||
`priority` INT NOT NULL DEFAULT 100,
|
||||
`search_primary_key` VARCHAR(64) NOT NULL DEFAULT 'ean',
|
||||
`search_fallback_keys` VARCHAR(255) NOT NULL DEFAULT '',
|
||||
`fallback_source_type` VARCHAR(32) NOT NULL DEFAULT 'none',
|
||||
`search_url_template` VARCHAR(1000) NOT NULL,
|
||||
`fallback_url_template` VARCHAR(1000) NOT NULL DEFAULT '',
|
||||
`search_context_keys` VARCHAR(255) NOT NULL DEFAULT '',
|
||||
`access_mode` VARCHAR(32) NOT NULL DEFAULT 'browser',
|
||||
`browser_engine` VARCHAR(32) NOT NULL DEFAULT 'default',
|
||||
`browser_headless` TINYINT(1) NOT NULL DEFAULT 0,
|
||||
`wait_after_load_ms` INT UNSIGNED NOT NULL DEFAULT 3000,
|
||||
`note` TEXT NULL,
|
||||
PRIMARY KEY (`id_product_catalog_source`),
|
||||
UNIQUE KEY `uq_product_catalog_source_key` (`source_key`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `ps_product_catalog_manufacturer_source` (
|
||||
`id_product_catalog_manufacturer_source` INT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`id_manufacturer` INT UNSIGNED NOT NULL,
|
||||
`id_product_catalog_source` INT UNSIGNED NOT NULL,
|
||||
`enabled` TINYINT(1) NOT NULL DEFAULT 1,
|
||||
`priority` INT NOT NULL DEFAULT 100,
|
||||
`note` TEXT NULL,
|
||||
PRIMARY KEY (`id_product_catalog_manufacturer_source`),
|
||||
UNIQUE KEY `uq_product_catalog_manufacturer_source` (`id_manufacturer`, `id_product_catalog_source`),
|
||||
KEY `idx_product_catalog_manufacturer_source_source` (`id_product_catalog_source`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_sources` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`legacy_catalog_source_id` BIGINT UNSIGNED NULL,
|
||||
`source_key` VARCHAR(64) NOT NULL,
|
||||
`source_name` VARCHAR(128) NOT NULL,
|
||||
`source_type` ENUM('supplier', 'manufacturer', 'backup', 'other') NOT NULL DEFAULT 'supplier',
|
||||
`base_url` VARCHAR(1024) NOT NULL,
|
||||
`enabled` TINYINT(1) NOT NULL DEFAULT 1,
|
||||
`priority` INT NOT NULL DEFAULT 100,
|
||||
`search_primary_key` VARCHAR(64) NULL,
|
||||
`search_fallback_keys` VARCHAR(255) NULL,
|
||||
`fallback_source_type` VARCHAR(64) NOT NULL DEFAULT 'none',
|
||||
`search_url_template` VARCHAR(2048) NULL,
|
||||
`fallback_url_template` VARCHAR(2048) NULL,
|
||||
`search_context_keys` VARCHAR(255) NULL,
|
||||
`access_mode` VARCHAR(64) NOT NULL DEFAULT 'browser',
|
||||
`browser_engine` VARCHAR(32) NOT NULL DEFAULT 'default',
|
||||
`browser_headless` TINYINT(1) NOT NULL DEFAULT 0,
|
||||
`wait_after_load_ms` INT UNSIGNED NOT NULL DEFAULT 3000,
|
||||
`scraper_config` JSON NULL,
|
||||
`request_delay_ms` INT UNSIGNED NOT NULL DEFAULT 3000,
|
||||
`last_health_status` SMALLINT UNSIGNED NULL,
|
||||
`last_health_checked_at` DATETIME NULL,
|
||||
`note` TEXT NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uq_scrape_sources_legacy` (`legacy_catalog_source_id`),
|
||||
UNIQUE KEY `uq_scrape_sources_key` (`source_key`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_manufacturer_sources` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`manufacturer_id` INT UNSIGNED NOT NULL,
|
||||
`source_id` BIGINT UNSIGNED NOT NULL,
|
||||
`enabled` TINYINT(1) NOT NULL DEFAULT 1,
|
||||
`priority` INT NOT NULL DEFAULT 100,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uq_scrape_manufacturer_source` (`manufacturer_id`, `source_id`),
|
||||
KEY `idx_scrape_manufacturer_source_source` (`source_id`),
|
||||
CONSTRAINT `fk_scrape_manufacturer_source_source` FOREIGN KEY (`source_id`) REFERENCES `scrape_sources` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_source_capabilities` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`source_id` BIGINT UNSIGNED NOT NULL,
|
||||
`capability` ENUM('search', 'pictures', 'texts', 'features') NOT NULL,
|
||||
`enabled` TINYINT(1) NOT NULL DEFAULT 1,
|
||||
`priority` INT NOT NULL DEFAULT 100,
|
||||
`parser_key` VARCHAR(128) NOT NULL,
|
||||
`parser_module` VARCHAR(255) NULL,
|
||||
`browser_engine` VARCHAR(32) NULL,
|
||||
`browser_headless` TINYINT(1) NULL,
|
||||
`wait_after_load_ms` INT UNSIGNED NULL,
|
||||
`min_width` INT UNSIGNED NULL,
|
||||
`min_height` INT UNSIGNED NULL,
|
||||
`min_longest_side` INT UNSIGNED NULL,
|
||||
`default_target_strategy` ENUM('unknown', 'color', 'size', 'combination', 'manual') NOT NULL DEFAULT 'unknown',
|
||||
`selector_config` JSON NULL,
|
||||
`extraction_config` JSON NULL,
|
||||
`quality_config` JSON NULL,
|
||||
`note` TEXT NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uq_scrape_source_capability_parser` (`source_id`, `capability`, `parser_key`),
|
||||
KEY `idx_scrape_source_capabilities_lookup` (`source_id`, `capability`, `enabled`, `priority`),
|
||||
CONSTRAINT `fk_scrape_source_capabilities_source` FOREIGN KEY (`source_id`) REFERENCES `scrape_sources` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_manufacturer_source_capabilities` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`manufacturer_source_id` BIGINT UNSIGNED NOT NULL,
|
||||
`source_capability_id` BIGINT UNSIGNED NOT NULL,
|
||||
`enabled` TINYINT(1) NOT NULL DEFAULT 1,
|
||||
`priority` INT NOT NULL DEFAULT 100,
|
||||
`target_strategy_override` ENUM('unknown', 'color', 'size', 'combination', 'manual') NULL,
|
||||
`config_override` JSON NULL,
|
||||
`note` TEXT NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uq_scrape_manufacturer_source_capability` (`manufacturer_source_id`, `source_capability_id`),
|
||||
KEY `idx_scrape_manufacturer_source_capabilities_capability` (`source_capability_id`),
|
||||
CONSTRAINT `fk_scrape_manufacturer_source_capabilities_mapping` FOREIGN KEY (`manufacturer_source_id`) REFERENCES `scrape_manufacturer_sources` (`id`),
|
||||
CONSTRAINT `fk_scrape_manufacturer_source_capabilities_capability` FOREIGN KEY (`source_capability_id`) REFERENCES `scrape_source_capabilities` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_products` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`manufacturer_id` INT UNSIGNED NULL,
|
||||
`id_product` BIGINT UNSIGNED NULL,
|
||||
`supplier_reference` VARCHAR(255) NULL,
|
||||
`ean13` VARCHAR(32) NULL,
|
||||
`color` VARCHAR(255) NULL,
|
||||
`source_id` BIGINT UNSIGNED NOT NULL,
|
||||
`source_url` VARCHAR(2048) NOT NULL,
|
||||
`match_method` ENUM('ean', 'reference', 'name', 'manual') NOT NULL DEFAULT 'ean',
|
||||
`source_product_key` VARCHAR(255) NULL,
|
||||
`product_name` TEXT NULL,
|
||||
`raw_data` JSON NULL,
|
||||
`parsed_data` JSON NULL,
|
||||
`status` ENUM('found', 'parsed', 'error', 'manual') NOT NULL DEFAULT 'found',
|
||||
`error_message` TEXT NULL,
|
||||
`fetched_at` DATETIME NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
KEY `idx_scrape_products_ean` (`ean13`),
|
||||
KEY `idx_scrape_products_reference` (`supplier_reference`),
|
||||
KEY `idx_scrape_products_source` (`source_id`),
|
||||
CONSTRAINT `fk_scrape_products_source` FOREIGN KEY (`source_id`) REFERENCES `scrape_sources` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_assets` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`scrape_product_id` BIGINT UNSIGNED NOT NULL,
|
||||
`asset_type` ENUM('image', 'document', 'other') NOT NULL DEFAULT 'image',
|
||||
`asset_url` VARCHAR(2048) NOT NULL,
|
||||
`local_path` VARCHAR(1024) NULL,
|
||||
`content_hash` CHAR(64) NULL,
|
||||
`position` INT UNSIGNED NOT NULL DEFAULT 0,
|
||||
`alt_text` VARCHAR(512) NULL,
|
||||
`status` ENUM('discovered', 'downloaded', 'error') NOT NULL DEFAULT 'discovered',
|
||||
`error_message` TEXT NULL,
|
||||
`fetched_at` DATETIME NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
KEY `idx_scrape_assets_product` (`scrape_product_id`),
|
||||
CONSTRAINT `fk_scrape_assets_product` FOREIGN KEY (`scrape_product_id`) REFERENCES `scrape_products` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_image_blobs` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`sha256` CHAR(64) NOT NULL,
|
||||
`perceptual_hash` CHAR(16) NULL,
|
||||
`mime_type` VARCHAR(64) NOT NULL,
|
||||
`width` INT UNSIGNED NULL,
|
||||
`height` INT UNSIGNED NULL,
|
||||
`size_bytes` INT UNSIGNED NOT NULL,
|
||||
`image_data` LONGBLOB NOT NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uq_scrape_image_blobs_sha256` (`sha256`),
|
||||
KEY `idx_scrape_image_blobs_phash` (`perceptual_hash`),
|
||||
KEY `idx_scrape_image_blobs_size` (`width`, `height`, `size_bytes`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_picture_candidates` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`image_blob_id` BIGINT UNSIGNED NOT NULL,
|
||||
`source_id` BIGINT UNSIGNED NOT NULL,
|
||||
`scrape_product_id` BIGINT UNSIGNED NULL,
|
||||
`manufacturer_id` INT UNSIGNED NULL,
|
||||
`id_product_catalog` BIGINT UNSIGNED NULL,
|
||||
`id_product` BIGINT UNSIGNED NULL,
|
||||
`supplier_reference` VARCHAR(255) NULL,
|
||||
`ean13` VARCHAR(32) NULL,
|
||||
`color` VARCHAR(255) NULL,
|
||||
`size_value` VARCHAR(128) NULL,
|
||||
`combi` VARCHAR(255) NULL,
|
||||
`source_page_url` VARCHAR(2048) NULL,
|
||||
`source_image_url` VARCHAR(2048) NOT NULL,
|
||||
`source_position` INT UNSIGNED NOT NULL DEFAULT 0,
|
||||
`target_strategy` ENUM('unknown', 'color', 'size', 'combination', 'manual') NOT NULL DEFAULT 'unknown',
|
||||
`target_color` VARCHAR(255) NULL,
|
||||
`target_size` VARCHAR(128) NULL,
|
||||
`target_combi` VARCHAR(255) NULL,
|
||||
`target_ean13` VARCHAR(32) NULL,
|
||||
`target_id_product_catalog` BIGINT UNSIGNED NULL,
|
||||
`match_confidence` DECIMAL(5,2) NOT NULL DEFAULT 0.00,
|
||||
`duplicate_status` ENUM('unique', 'exact_duplicate', 'visual_duplicate', 'review') NOT NULL DEFAULT 'review',
|
||||
`duplicate_of_candidate_id` BIGINT UNSIGNED NULL,
|
||||
`status` ENUM('draft', 'approved', 'rejected') NOT NULL DEFAULT 'draft',
|
||||
`review_note` TEXT NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
UNIQUE KEY `uq_scrape_picture_candidate_source_image` (`source_id`, `source_image_url`(512)),
|
||||
KEY `idx_scrape_picture_candidates_blob` (`image_blob_id`),
|
||||
KEY `idx_scrape_picture_candidates_source` (`source_id`),
|
||||
KEY `idx_scrape_picture_candidates_product` (`manufacturer_id`, `supplier_reference`, `ean13`),
|
||||
KEY `idx_scrape_picture_candidates_target` (`target_strategy`, `target_color`, `target_size`, `target_combi`),
|
||||
KEY `idx_scrape_picture_candidates_duplicate` (`duplicate_status`, `duplicate_of_candidate_id`),
|
||||
CONSTRAINT `fk_scrape_picture_candidates_blob` FOREIGN KEY (`image_blob_id`) REFERENCES `scrape_image_blobs` (`id`),
|
||||
CONSTRAINT `fk_scrape_picture_candidates_source` FOREIGN KEY (`source_id`) REFERENCES `scrape_sources` (`id`),
|
||||
CONSTRAINT `fk_scrape_picture_candidates_product` FOREIGN KEY (`scrape_product_id`) REFERENCES `scrape_products` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_picture_target_rules` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`manufacturer_id` INT UNSIGNED NULL,
|
||||
`source_id` BIGINT UNSIGNED NULL,
|
||||
`id_category_default` INT UNSIGNED NULL,
|
||||
`catalog_category` VARCHAR(255) NULL,
|
||||
`product_name_pattern` VARCHAR(255) NULL,
|
||||
`target_strategy` ENUM('color', 'size', 'combination', 'manual') NOT NULL,
|
||||
`confidence` DECIMAL(5,2) NOT NULL DEFAULT 0.00,
|
||||
`examples_count` INT UNSIGNED NOT NULL DEFAULT 0,
|
||||
`note` TEXT NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`updated_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
KEY `idx_scrape_picture_target_rules_lookup` (`manufacturer_id`, `source_id`, `id_category_default`),
|
||||
CONSTRAINT `fk_scrape_picture_target_rules_source` FOREIGN KEY (`source_id`) REFERENCES `scrape_sources` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
CREATE TABLE IF NOT EXISTS `scrape_runs` (
|
||||
`id` BIGINT UNSIGNED NOT NULL AUTO_INCREMENT,
|
||||
`source_id` BIGINT UNSIGNED NULL,
|
||||
`manufacturer_id` INT UNSIGNED NULL,
|
||||
`started_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
`finished_at` DATETIME NULL,
|
||||
`status` ENUM('running', 'completed', 'failed', 'cancelled') NOT NULL DEFAULT 'running',
|
||||
`request_count` INT UNSIGNED NOT NULL DEFAULT 0,
|
||||
`success_count` INT UNSIGNED NOT NULL DEFAULT 0,
|
||||
`error_count` INT UNSIGNED NOT NULL DEFAULT 0,
|
||||
`notes` TEXT NULL,
|
||||
`created_at` DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
|
||||
PRIMARY KEY (`id`),
|
||||
KEY `idx_scrape_runs_source` (`source_id`),
|
||||
CONSTRAINT `fk_scrape_runs_source` FOREIGN KEY (`source_id`) REFERENCES `scrape_sources` (`id`)
|
||||
) ENGINE=InnoDB;
|
||||
|
||||
INSERT INTO `scrape_source_capabilities` (
|
||||
`source_id`,
|
||||
`capability`,
|
||||
`enabled`,
|
||||
`priority`,
|
||||
`parser_key`,
|
||||
`parser_module`,
|
||||
`browser_engine`,
|
||||
`browser_headless`,
|
||||
`wait_after_load_ms`,
|
||||
`min_longest_side`,
|
||||
`default_target_strategy`,
|
||||
`selector_config`,
|
||||
`extraction_config`,
|
||||
`quality_config`,
|
||||
`note`
|
||||
)
|
||||
SELECT
|
||||
`id`,
|
||||
'pictures',
|
||||
1,
|
||||
10,
|
||||
'hudy-gallery',
|
||||
'hudy.picture-gallery',
|
||||
NULL,
|
||||
NULL,
|
||||
NULL,
|
||||
800,
|
||||
'color',
|
||||
JSON_OBJECT(
|
||||
'colorSwitcherSelector', 'a[id^="product-change-color-"]',
|
||||
'eanTableSelector', 'table.product-params__params-table',
|
||||
'topImageSelector', 'p#snippet-productInfo-image img.product-top__img',
|
||||
'gallerySelector', 'p#snippet-productInfo-image, .pdbox__window'
|
||||
),
|
||||
JSON_OBJECT(
|
||||
'verifyVariantBy', JSON_ARRAY('ean', 'color'),
|
||||
'pictureSource', 'largeGallery',
|
||||
'dedupeBy', JSON_ARRAY('sha256', 'perceptual_hash')
|
||||
),
|
||||
JSON_OBJECT('minLongestSide', 800, 'allowSquareCrop', true),
|
||||
'Hudy product gallery pictures. Verify the color variant by EAN before extracting large gallery images.'
|
||||
FROM `scrape_sources`
|
||||
WHERE `source_key` = 'hudy'
|
||||
ON DUPLICATE KEY UPDATE
|
||||
`enabled` = VALUES(`enabled`),
|
||||
`priority` = VALUES(`priority`),
|
||||
`parser_module` = VALUES(`parser_module`),
|
||||
`min_longest_side` = VALUES(`min_longest_side`),
|
||||
`default_target_strategy` = VALUES(`default_target_strategy`),
|
||||
`selector_config` = VALUES(`selector_config`),
|
||||
`extraction_config` = VALUES(`extraction_config`),
|
||||
`quality_config` = VALUES(`quality_config`),
|
||||
`note` = VALUES(`note`);
|
||||
|
||||
INSERT INTO `scrape_source_capabilities` (
|
||||
`source_id`,
|
||||
`capability`,
|
||||
`enabled`,
|
||||
`priority`,
|
||||
`parser_key`,
|
||||
`parser_module`,
|
||||
`browser_engine`,
|
||||
`browser_headless`,
|
||||
`wait_after_load_ms`,
|
||||
`default_target_strategy`,
|
||||
`selector_config`,
|
||||
`extraction_config`,
|
||||
`note`
|
||||
)
|
||||
SELECT
|
||||
`id`,
|
||||
'texts',
|
||||
1,
|
||||
30,
|
||||
'hudy-product-texts',
|
||||
'hudy.product-texts',
|
||||
NULL,
|
||||
NULL,
|
||||
NULL,
|
||||
'manual',
|
||||
JSON_OBJECT('descriptionSelector', '.product-detail, .product-top, [itemprop="description"]'),
|
||||
JSON_OBJECT('fields', JSON_ARRAY('shortDescription', 'description', 'marketingText')),
|
||||
'Hudy text extraction placeholder. User will teach exact text fields before automation is trusted.'
|
||||
FROM `scrape_sources`
|
||||
WHERE `source_key` = 'hudy'
|
||||
ON DUPLICATE KEY UPDATE
|
||||
`enabled` = VALUES(`enabled`),
|
||||
`priority` = VALUES(`priority`),
|
||||
`parser_module` = VALUES(`parser_module`),
|
||||
`selector_config` = VALUES(`selector_config`),
|
||||
`extraction_config` = VALUES(`extraction_config`),
|
||||
`note` = VALUES(`note`);
|
||||
|
||||
INSERT INTO `scrape_source_capabilities` (
|
||||
`source_id`,
|
||||
`capability`,
|
||||
`enabled`,
|
||||
`priority`,
|
||||
`parser_key`,
|
||||
`parser_module`,
|
||||
`browser_engine`,
|
||||
`browser_headless`,
|
||||
`wait_after_load_ms`,
|
||||
`default_target_strategy`,
|
||||
`selector_config`,
|
||||
`extraction_config`,
|
||||
`note`
|
||||
)
|
||||
SELECT
|
||||
`id`,
|
||||
'features',
|
||||
1,
|
||||
20,
|
||||
'hudy-product-params',
|
||||
'hudy.product-params',
|
||||
NULL,
|
||||
NULL,
|
||||
NULL,
|
||||
'manual',
|
||||
JSON_OBJECT('paramsTableSelector', 'table.product-params__params-table'),
|
||||
JSON_OBJECT('fields', JSON_ARRAY('label', 'value'), 'mergeStrategy', 'review-first'),
|
||||
'Hudy feature extraction placeholder. Product parameters are kept reviewable before catalog writes.'
|
||||
FROM `scrape_sources`
|
||||
WHERE `source_key` = 'hudy'
|
||||
ON DUPLICATE KEY UPDATE
|
||||
`enabled` = VALUES(`enabled`),
|
||||
`priority` = VALUES(`priority`),
|
||||
`parser_module` = VALUES(`parser_module`),
|
||||
`selector_config` = VALUES(`selector_config`),
|
||||
`extraction_config` = VALUES(`extraction_config`),
|
||||
`note` = VALUES(`note`);
|
||||
|
||||
INSERT INTO `scrape_source_capabilities` (
|
||||
`source_id`,
|
||||
`capability`,
|
||||
`enabled`,
|
||||
`priority`,
|
||||
`parser_key`,
|
||||
`parser_module`,
|
||||
`browser_engine`,
|
||||
`browser_headless`,
|
||||
`wait_after_load_ms`,
|
||||
`min_longest_side`,
|
||||
`default_target_strategy`,
|
||||
`selector_config`,
|
||||
`extraction_config`,
|
||||
`quality_config`,
|
||||
`note`
|
||||
)
|
||||
SELECT
|
||||
`id`,
|
||||
'pictures',
|
||||
1,
|
||||
20,
|
||||
'idealo-splide-gallery',
|
||||
'idealo.splide-gallery',
|
||||
'firefox',
|
||||
1,
|
||||
3000,
|
||||
800,
|
||||
'color',
|
||||
JSON_OBJECT(
|
||||
'galleryTrackSelector', 'div.splide__track.container',
|
||||
'slideImageSelector', 'ul.splide__list li.splide__slide img'
|
||||
),
|
||||
JSON_OBJECT(
|
||||
'urlAttributes', JSON_ARRAY('srcset', 'data-srcset', 'data-large', 'data-original', 'data-src', 'src'),
|
||||
'pictureSource', 'productGalleryOnly',
|
||||
'rejectSearchThumbnails', true,
|
||||
'dedupeBy', JSON_ARRAY('sha256', 'perceptual_hash')
|
||||
),
|
||||
JSON_OBJECT('minLongestSide', 800, 'allowSquareCrop', true),
|
||||
'Idealo is primarily an image backup source. Use product gallery images only, not search-result thumbnails.'
|
||||
FROM `scrape_sources`
|
||||
WHERE `source_key` = 'idealo'
|
||||
ON DUPLICATE KEY UPDATE
|
||||
`enabled` = VALUES(`enabled`),
|
||||
`priority` = VALUES(`priority`),
|
||||
`parser_module` = VALUES(`parser_module`),
|
||||
`browser_engine` = VALUES(`browser_engine`),
|
||||
`browser_headless` = VALUES(`browser_headless`),
|
||||
`wait_after_load_ms` = VALUES(`wait_after_load_ms`),
|
||||
`min_longest_side` = VALUES(`min_longest_side`),
|
||||
`default_target_strategy` = VALUES(`default_target_strategy`),
|
||||
`selector_config` = VALUES(`selector_config`),
|
||||
`extraction_config` = VALUES(`extraction_config`),
|
||||
`quality_config` = VALUES(`quality_config`),
|
||||
`note` = VALUES(`note`);
|
||||
|
||||
INSERT INTO `scrape_manufacturer_source_capabilities` (
|
||||
`manufacturer_source_id`,
|
||||
`source_capability_id`,
|
||||
`enabled`,
|
||||
`priority`,
|
||||
`target_strategy_override`
|
||||
)
|
||||
SELECT
|
||||
`sms`.`id`,
|
||||
`sc`.`id`,
|
||||
`sc`.`enabled`,
|
||||
`sc`.`priority`,
|
||||
NULL
|
||||
FROM `scrape_manufacturer_sources` `sms`
|
||||
JOIN `scrape_source_capabilities` `sc`
|
||||
ON `sc`.`source_id` = `sms`.`source_id`
|
||||
ON DUPLICATE KEY UPDATE
|
||||
`enabled` = VALUES(`enabled`),
|
||||
`priority` = VALUES(`priority`);
|
||||
+32
-1
@@ -30,14 +30,35 @@ export function loadConfig({ configPath = "config/local.json", dryRun } = {}) {
|
||||
database: {
|
||||
driver: String(env.DB_DRIVER || config.database?.driver || "endpoint").toLowerCase(),
|
||||
mode: String(env.DB_MODE || config.database?.mode || "local").toLowerCase(),
|
||||
allowWrites: parseBoolean(env.DB_ALLOW_WRITES, config.database?.allowWrites),
|
||||
allowLocalWrites: parseBoolean(
|
||||
env.DB_LOCAL_ALLOW_WRITES,
|
||||
config.database?.allowLocalWrites ?? config.database?.allowWrites,
|
||||
),
|
||||
host: env.DB_HOST || config.database?.host || "127.0.0.1",
|
||||
port: Number(env.DB_PORT || config.database?.port || 3306),
|
||||
name: env.DB_NAME || config.database?.name || "catalog_maker_test",
|
||||
user: env.DB_USER || config.database?.user || "catalog_maker",
|
||||
password: env.DB_PASSWORD || config.database?.password || "",
|
||||
scrapeName: env.SCRAPE_DB_NAME || config.database?.scrapeName || "catalog_scrape",
|
||||
connectionLimit: Number(env.DB_CONNECTION_LIMIT || config.database?.connectionLimit || 5),
|
||||
},
|
||||
databaseTool: {
|
||||
url: env.ADMINER_URL || config.databaseTool?.url || config.databaseGate?.url || "http://127.0.0.1:8080",
|
||||
browserEngine: String(
|
||||
env.ADMINER_BROWSER_ENGINE
|
||||
|| config.databaseTool?.browserEngine
|
||||
|| config.databaseGate?.browserEngine
|
||||
|| "chromium",
|
||||
).toLowerCase(),
|
||||
},
|
||||
dbGate: {
|
||||
url: env.DBGATE_URL || config.dbGate?.url || "http://127.0.0.1:3000",
|
||||
browserEngine: String(
|
||||
env.DBGATE_BROWSER_ENGINE
|
||||
|| config.dbGate?.browserEngine
|
||||
|| "chromium",
|
||||
).toLowerCase(),
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -85,5 +106,15 @@ function mergeConfig(base, override) {
|
||||
...base.database,
|
||||
...override.database,
|
||||
},
|
||||
databaseTool: {
|
||||
...base.databaseTool,
|
||||
...base.databaseGate,
|
||||
...override.databaseTool,
|
||||
...override.databaseGate,
|
||||
},
|
||||
dbGate: {
|
||||
...base.dbGate,
|
||||
...override.dbGate,
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -40,8 +40,8 @@ export async function executeMariaDbWrite(config, sql) {
|
||||
if (config.database.mode !== "local") {
|
||||
throw new Error("MariaDB writes are blocked outside Local DB mode.");
|
||||
}
|
||||
if (!config.database.allowWrites) {
|
||||
throw new Error("MariaDB writes are disabled. Enable local database writes in settings first.");
|
||||
if (!config.database.allowLocalWrites) {
|
||||
throw new Error("Local MariaDB writes are disabled. Enable local database writes in Settings first.");
|
||||
}
|
||||
|
||||
const connection = await getPool(config).getConnection();
|
||||
|
||||
@@ -0,0 +1,319 @@
|
||||
// @ts-nocheck
|
||||
import crypto from "node:crypto";
|
||||
import mariadb from "mariadb";
|
||||
|
||||
export async function saveScrapePictureCandidate(config, picture) {
|
||||
if (config.database?.driver !== "mariadb") {
|
||||
throw new Error("Local scrape DB needs MariaDB configuration.");
|
||||
}
|
||||
if (!config.database?.allowLocalWrites) {
|
||||
throw new Error("Local scrape DB writes are disabled. Enable writes in Settings first.");
|
||||
}
|
||||
|
||||
const imageUrl = normalizeRequiredUrl(picture.imageUrl, "imageUrl");
|
||||
const sourceKey = normalizeText(picture.sourceKey) || "unknown";
|
||||
const sourceName = normalizeText(picture.sourceName) || sourceKey;
|
||||
const sourcePageUrl = normalizeOptionalText(picture.pageUrl);
|
||||
const image = await downloadImage(imageUrl);
|
||||
const sha256 = crypto.createHash("sha256").update(image.buffer).digest("hex");
|
||||
const scrapeDb = config.database.scrapeName || "catalog_scrape";
|
||||
|
||||
const connection = await mariadb.createConnection({
|
||||
host: config.database.host,
|
||||
port: Number(config.database.port),
|
||||
database: scrapeDb,
|
||||
user: config.database.user,
|
||||
password: config.database.password,
|
||||
bigIntAsNumber: true,
|
||||
});
|
||||
|
||||
try {
|
||||
await connection.beginTransaction();
|
||||
const sourceId = await upsertSource(connection, {
|
||||
sourceKey,
|
||||
sourceName,
|
||||
imageUrl,
|
||||
sourceType: normalizeText(picture.sourceType) || "supplier",
|
||||
});
|
||||
const imageBlobId = await upsertImageBlob(connection, {
|
||||
sha256,
|
||||
mimeType: image.mimeType,
|
||||
width: normalizePositiveNumber(picture.width),
|
||||
height: normalizePositiveNumber(picture.height),
|
||||
sizeBytes: image.buffer.byteLength,
|
||||
buffer: image.buffer,
|
||||
});
|
||||
const duplicate = await findExistingImageCandidate(connection, imageBlobId, sourceId, imageUrl);
|
||||
const candidateId = await upsertPictureCandidate(connection, {
|
||||
imageBlobId,
|
||||
sourceId,
|
||||
manufacturerId: normalizePositiveNumber(picture.manufacturerId),
|
||||
idProductCatalog: normalizePositiveNumber(picture.idProductCatalog),
|
||||
idProduct: normalizePositiveNumber(picture.idProduct),
|
||||
supplierReference: normalizeOptionalText(picture.supplierReference),
|
||||
ean13: normalizeOptionalText(picture.ean13),
|
||||
color: normalizeOptionalText(picture.color),
|
||||
sizeValue: normalizeOptionalText(picture.sizeValue),
|
||||
combi: normalizeOptionalText(picture.combi),
|
||||
sourcePageUrl,
|
||||
sourceImageUrl: imageUrl,
|
||||
sourcePosition: normalizePositiveNumber(picture.sourcePosition) || 0,
|
||||
targetStrategy: normalizeTargetStrategy(picture.targetStrategy),
|
||||
targetColor: normalizeOptionalText(picture.targetColor),
|
||||
targetSize: normalizeOptionalText(picture.targetSize),
|
||||
targetCombi: normalizeOptionalText(picture.targetCombi),
|
||||
targetEan13: normalizeOptionalText(picture.targetEan13),
|
||||
targetIdProductCatalog: normalizePositiveNumber(picture.targetIdProductCatalog),
|
||||
matchConfidence: normalizeConfidence(picture.matchConfidence),
|
||||
duplicateStatus: duplicate ? "exact_duplicate" : "unique",
|
||||
duplicateOfCandidateId: duplicate?.id || null,
|
||||
});
|
||||
|
||||
await connection.commit();
|
||||
return {
|
||||
saved: true,
|
||||
candidateId,
|
||||
imageBlobId,
|
||||
sha256,
|
||||
duplicateStatus: duplicate ? "exact_duplicate" : "unique",
|
||||
duplicateOfCandidateId: duplicate?.id || null,
|
||||
mimeType: image.mimeType,
|
||||
width: normalizePositiveNumber(picture.width) || null,
|
||||
height: normalizePositiveNumber(picture.height) || null,
|
||||
sizeBytes: image.buffer.byteLength,
|
||||
sourceKey,
|
||||
sourceName,
|
||||
};
|
||||
} catch (error) {
|
||||
await connection.rollback();
|
||||
throw error;
|
||||
} finally {
|
||||
await connection.end();
|
||||
}
|
||||
}
|
||||
|
||||
async function downloadImage(url) {
|
||||
const controller = new AbortController();
|
||||
const timeout = setTimeout(() => controller.abort(), 20000);
|
||||
try {
|
||||
const response = await fetch(url, {
|
||||
signal: controller.signal,
|
||||
headers: {
|
||||
accept: "image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8",
|
||||
},
|
||||
});
|
||||
if (!response.ok) {
|
||||
throw new Error(`Image download failed with HTTP ${response.status}.`);
|
||||
}
|
||||
|
||||
const mimeType = String(response.headers.get("content-type") || "application/octet-stream")
|
||||
.split(";")[0]
|
||||
.toLowerCase();
|
||||
if (!mimeType.startsWith("image/")) {
|
||||
throw new Error(`Downloaded content is not an image (${mimeType}).`);
|
||||
}
|
||||
|
||||
const buffer = Buffer.from(await response.arrayBuffer());
|
||||
if (!buffer.length) {
|
||||
throw new Error("Downloaded image is empty.");
|
||||
}
|
||||
if (buffer.byteLength > 20 * 1024 * 1024) {
|
||||
throw new Error("Downloaded image is larger than 20 MB.");
|
||||
}
|
||||
|
||||
return { buffer, mimeType };
|
||||
} finally {
|
||||
clearTimeout(timeout);
|
||||
}
|
||||
}
|
||||
|
||||
async function upsertSource(connection, source) {
|
||||
const baseUrl = new URL(source.imageUrl).origin;
|
||||
const sourceType = ["supplier", "manufacturer", "backup", "other"].includes(source.sourceType)
|
||||
? source.sourceType
|
||||
: "supplier";
|
||||
const result = await connection.query(
|
||||
`
|
||||
INSERT INTO scrape_sources (
|
||||
source_key,
|
||||
source_name,
|
||||
source_type,
|
||||
base_url,
|
||||
enabled
|
||||
)
|
||||
VALUES (?, ?, ?, ?, 1)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
id = LAST_INSERT_ID(id),
|
||||
source_name = VALUES(source_name),
|
||||
base_url = VALUES(base_url),
|
||||
enabled = VALUES(enabled)
|
||||
`,
|
||||
[source.sourceKey, source.sourceName, sourceType, baseUrl],
|
||||
);
|
||||
|
||||
return Number(result.insertId);
|
||||
}
|
||||
|
||||
async function upsertImageBlob(connection, image) {
|
||||
const result = await connection.query(
|
||||
`
|
||||
INSERT INTO scrape_image_blobs (
|
||||
sha256,
|
||||
mime_type,
|
||||
width,
|
||||
height,
|
||||
size_bytes,
|
||||
image_data
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?)
|
||||
ON DUPLICATE KEY UPDATE
|
||||
id = LAST_INSERT_ID(id),
|
||||
mime_type = VALUES(mime_type),
|
||||
width = COALESCE(VALUES(width), width),
|
||||
height = COALESCE(VALUES(height), height),
|
||||
size_bytes = VALUES(size_bytes)
|
||||
`,
|
||||
[image.sha256, image.mimeType, image.width, image.height, image.sizeBytes, image.buffer],
|
||||
);
|
||||
|
||||
return Number(result.insertId);
|
||||
}
|
||||
|
||||
async function findExistingImageCandidate(connection, imageBlobId, sourceId, imageUrl) {
|
||||
const rows = await connection.query(
|
||||
`
|
||||
SELECT id
|
||||
FROM scrape_picture_candidates
|
||||
WHERE image_blob_id = ?
|
||||
AND NOT (source_id = ? AND source_image_url = ?)
|
||||
ORDER BY id ASC
|
||||
LIMIT 1
|
||||
`,
|
||||
[imageBlobId, sourceId, imageUrl],
|
||||
);
|
||||
const items = normalizeRows(rows);
|
||||
return items[0] || null;
|
||||
}
|
||||
|
||||
async function upsertPictureCandidate(connection, candidate) {
|
||||
const result = await connection.query(
|
||||
`
|
||||
INSERT INTO scrape_picture_candidates (
|
||||
image_blob_id,
|
||||
source_id,
|
||||
manufacturer_id,
|
||||
id_product_catalog,
|
||||
id_product,
|
||||
supplier_reference,
|
||||
ean13,
|
||||
color,
|
||||
size_value,
|
||||
combi,
|
||||
source_page_url,
|
||||
source_image_url,
|
||||
source_position,
|
||||
target_strategy,
|
||||
target_color,
|
||||
target_size,
|
||||
target_combi,
|
||||
target_ean13,
|
||||
target_id_product_catalog,
|
||||
match_confidence,
|
||||
duplicate_status,
|
||||
duplicate_of_candidate_id,
|
||||
status
|
||||
)
|
||||
VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 'draft')
|
||||
ON DUPLICATE KEY UPDATE
|
||||
id = LAST_INSERT_ID(id),
|
||||
image_blob_id = VALUES(image_blob_id),
|
||||
manufacturer_id = VALUES(manufacturer_id),
|
||||
id_product_catalog = VALUES(id_product_catalog),
|
||||
id_product = VALUES(id_product),
|
||||
supplier_reference = VALUES(supplier_reference),
|
||||
ean13 = VALUES(ean13),
|
||||
color = VALUES(color),
|
||||
size_value = VALUES(size_value),
|
||||
combi = VALUES(combi),
|
||||
source_page_url = VALUES(source_page_url),
|
||||
source_position = VALUES(source_position),
|
||||
target_strategy = VALUES(target_strategy),
|
||||
target_color = VALUES(target_color),
|
||||
target_size = VALUES(target_size),
|
||||
target_combi = VALUES(target_combi),
|
||||
target_ean13 = VALUES(target_ean13),
|
||||
target_id_product_catalog = VALUES(target_id_product_catalog),
|
||||
match_confidence = VALUES(match_confidence),
|
||||
duplicate_status = VALUES(duplicate_status),
|
||||
duplicate_of_candidate_id = VALUES(duplicate_of_candidate_id),
|
||||
updated_at = CURRENT_TIMESTAMP
|
||||
`,
|
||||
[
|
||||
candidate.imageBlobId,
|
||||
candidate.sourceId,
|
||||
candidate.manufacturerId,
|
||||
candidate.idProductCatalog,
|
||||
candidate.idProduct,
|
||||
candidate.supplierReference,
|
||||
candidate.ean13,
|
||||
candidate.color,
|
||||
candidate.sizeValue,
|
||||
candidate.combi,
|
||||
candidate.sourcePageUrl,
|
||||
candidate.sourceImageUrl,
|
||||
candidate.sourcePosition,
|
||||
candidate.targetStrategy,
|
||||
candidate.targetColor,
|
||||
candidate.targetSize,
|
||||
candidate.targetCombi,
|
||||
candidate.targetEan13,
|
||||
candidate.targetIdProductCatalog,
|
||||
candidate.matchConfidence,
|
||||
candidate.duplicateStatus,
|
||||
candidate.duplicateOfCandidateId,
|
||||
],
|
||||
);
|
||||
|
||||
return Number(result.insertId);
|
||||
}
|
||||
|
||||
function normalizeRows(rows) {
|
||||
return Array.isArray(rows) ? rows.filter((row) => row && typeof row === "object" && !("meta" in row)) : [];
|
||||
}
|
||||
|
||||
function normalizeRequiredUrl(value, name) {
|
||||
const text = normalizeText(value);
|
||||
if (!text) throw new Error(`${name} is required.`);
|
||||
const url = new URL(text);
|
||||
if (!["http:", "https:"].includes(url.protocol)) {
|
||||
throw new Error(`${name} must be an HTTP URL.`);
|
||||
}
|
||||
return url.toString();
|
||||
}
|
||||
|
||||
function normalizeText(value) {
|
||||
return String(value ?? "").trim();
|
||||
}
|
||||
|
||||
function normalizeOptionalText(value) {
|
||||
const text = normalizeText(value);
|
||||
return text || null;
|
||||
}
|
||||
|
||||
function normalizePositiveNumber(value) {
|
||||
const number = Number(value);
|
||||
return Number.isFinite(number) && number > 0 ? Math.trunc(number) : null;
|
||||
}
|
||||
|
||||
function normalizeConfidence(value) {
|
||||
const number = Number(value);
|
||||
if (!Number.isFinite(number)) return 0;
|
||||
return Math.max(0, Math.min(100, number));
|
||||
}
|
||||
|
||||
function normalizeTargetStrategy(value) {
|
||||
const strategy = normalizeText(value).toLowerCase();
|
||||
return ["unknown", "color", "size", "combination", "manual"].includes(strategy)
|
||||
? strategy
|
||||
: "unknown";
|
||||
}
|
||||
+51
-2
@@ -15,7 +15,10 @@ import {
|
||||
loadProductByEan,
|
||||
loadProductInfo,
|
||||
getProductPictures,
|
||||
saveProductPictureCandidate,
|
||||
openProductSource,
|
||||
} from "./services/catalog-products.ts";
|
||||
import { openInBrowser } from "./services/browser-adapter.ts";
|
||||
|
||||
const __dirname = path.dirname(fileURLToPath(import.meta.url));
|
||||
const publicDir = path.resolve(__dirname, "../public");
|
||||
@@ -24,6 +27,7 @@ const config = loadConfig();
|
||||
const mimeTypes = new Map([
|
||||
[".html", "text/html; charset=utf-8"],
|
||||
[".css", "text/css; charset=utf-8"],
|
||||
[".js", "text/javascript; charset=utf-8"],
|
||||
[".ts", "text/javascript; charset=utf-8"],
|
||||
[".json", "application/json; charset=utf-8"],
|
||||
[".svg", "image/svg+xml"],
|
||||
@@ -121,6 +125,44 @@ const server = http.createServer(async (request, response) => {
|
||||
return;
|
||||
}
|
||||
|
||||
if (url.pathname === "/api/catalog-maker/save-picture" && request.method === "POST") {
|
||||
const body = await readJsonBody(request);
|
||||
await sendJson(response, await saveProductPictureCandidate(requestConfig, body));
|
||||
return;
|
||||
}
|
||||
|
||||
if (url.pathname === "/api/catalog-maker/open-source") {
|
||||
await sendJson(
|
||||
response,
|
||||
await openProductSource(requestConfig, {
|
||||
manufacturerId: url.searchParams.get("manufacturerId"),
|
||||
sourceKey: url.searchParams.get("sourceKey"),
|
||||
url: url.searchParams.get("url"),
|
||||
}),
|
||||
);
|
||||
return;
|
||||
}
|
||||
|
||||
if (url.pathname === "/api/tools/open-adminer") {
|
||||
await openInBrowser(config.databaseTool.url, { engine: config.databaseTool.browserEngine });
|
||||
await sendJson(response, {
|
||||
opened: true,
|
||||
url: config.databaseTool.url,
|
||||
browserEngine: config.databaseTool.browserEngine,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (url.pathname === "/api/tools/open-dbgate") {
|
||||
await openInBrowser(config.dbGate.url, { engine: config.dbGate.browserEngine });
|
||||
await sendJson(response, {
|
||||
opened: true,
|
||||
url: config.dbGate.url,
|
||||
browserEngine: config.dbGate.browserEngine,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
if (url.pathname === "/api/catalog-maker/product-info") {
|
||||
await sendJson(
|
||||
response,
|
||||
@@ -195,7 +237,10 @@ function getStatus() {
|
||||
database: {
|
||||
driver: config.database.driver,
|
||||
mode: config.database.mode,
|
||||
allowWrites: config.database.driver === "mariadb" && config.database.mode === "local" && config.database.allowWrites,
|
||||
scrapeName: config.database.scrapeName,
|
||||
allowLocalWrites: config.database.driver === "mariadb"
|
||||
&& config.database.mode === "local"
|
||||
&& config.database.allowLocalWrites,
|
||||
},
|
||||
workbook: path.basename(config.sourceWorkbook),
|
||||
workbookExists: true,
|
||||
@@ -212,7 +257,8 @@ function getStatus() {
|
||||
}
|
||||
|
||||
function getRequestConfig(request) {
|
||||
const selectedMode = String(request.headers["x-database-mode"] || config.database.mode).toLowerCase();
|
||||
const requestedMode = String(request.headers["x-database-mode"] || config.database.mode).toLowerCase();
|
||||
const selectedMode = requestedMode === "scrape" ? "local" : requestedMode;
|
||||
|
||||
if (selectedMode === "local" && config.database.driver !== "mariadb") {
|
||||
throw new Error("Local DB is selected, but MariaDB is not configured. Live DB was not used.");
|
||||
@@ -227,6 +273,9 @@ function getRequestConfig(request) {
|
||||
database: {
|
||||
...config.database,
|
||||
mode: selectedMode,
|
||||
allowLocalWrites: selectedMode === "local"
|
||||
&& config.database.allowLocalWrites
|
||||
&& request.headers["x-local-write-access"] === "1",
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,18 +1,19 @@
|
||||
// @ts-nocheck
|
||||
import { chromium } from "playwright";
|
||||
import fs from "node:fs";
|
||||
import { chromium, firefox } from "playwright";
|
||||
|
||||
const visibleBrowsers = new Set();
|
||||
|
||||
export async function openInBrowser(url) {
|
||||
const browser = await chromium.launch({ headless: false });
|
||||
export async function openInBrowser(url, { engine = "chromium" } = {}) {
|
||||
const browser = await launchBrowser(engine, false);
|
||||
const page = await browser.newPage({ viewport: { width: 1440, height: 1000 } });
|
||||
await page.goto(url, { waitUntil: "domcontentloaded", timeout: 30_000 });
|
||||
visibleBrowsers.add(browser);
|
||||
return { browser, page };
|
||||
}
|
||||
|
||||
export async function searchInBrowser({ url, headless = true, waitAfterLoadMs = 3000 }) {
|
||||
const browser = await chromium.launch({ headless });
|
||||
export async function searchInBrowser({ url, engine = "chromium", headless = true, waitAfterLoadMs = 3000 }) {
|
||||
const browser = await launchBrowser(engine, headless);
|
||||
const page = await browser.newPage({ viewport: { width: 1440, height: 1000 } });
|
||||
|
||||
try {
|
||||
@@ -44,7 +45,33 @@ export async function searchInBrowser({ url, headless = true, waitAfterLoadMs =
|
||||
}
|
||||
|
||||
export async function scrapePage(url, options = {}) {
|
||||
return searchInBrowser({ url, headless: true, waitAfterLoadMs: 750, ...options });
|
||||
return searchInBrowser({ url, engine: "chromium", headless: true, waitAfterLoadMs: 750, ...options });
|
||||
}
|
||||
|
||||
function launchBrowser(engine, headless) {
|
||||
const normalizedEngine = String(engine).toLowerCase() === "firefox" ? "firefox" : "chromium";
|
||||
const browserType = normalizedEngine === "firefox" ? firefox : chromium;
|
||||
const executablePath = resolveSystemBrowser(normalizedEngine);
|
||||
const options = {
|
||||
headless,
|
||||
timeout: 10_000,
|
||||
...(executablePath ? { executablePath } : {}),
|
||||
};
|
||||
|
||||
return browserType.launch(options);
|
||||
}
|
||||
|
||||
function resolveSystemBrowser(engine) {
|
||||
if (process.platform !== "win32") return "";
|
||||
if (engine === "firefox") return "";
|
||||
|
||||
const candidates = [
|
||||
"C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
|
||||
"C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
|
||||
"C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe",
|
||||
];
|
||||
|
||||
return candidates.find((candidate) => fs.existsSync(candidate)) || "";
|
||||
}
|
||||
|
||||
export async function closeVisibleBrowsers() {
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
// @ts-nocheck
|
||||
import { executeEndpointWrite, queryEndpoint } from "../endpoint/endpoint-client.ts";
|
||||
import { saveScrapePictureCandidate } from "../db/scrape-image-store.ts";
|
||||
import { openInBrowser, scrapePage, searchInBrowser } from "./browser-adapter.ts";
|
||||
import {
|
||||
prepareManufacturerCatalogSqls,
|
||||
@@ -13,6 +14,7 @@ import {
|
||||
|
||||
export async function loadManufacturerCatalogState(config, { manufacturerId, position = 1 }) {
|
||||
const refreshResults = [];
|
||||
if (config.database?.mode === "local" && config.database.allowLocalWrites) {
|
||||
for (const sql of prepareManufacturerCatalogSqls({ manufacturerId })) {
|
||||
refreshResults.push(
|
||||
await executeEndpointWrite(config, sql, {
|
||||
@@ -20,6 +22,7 @@ export async function loadManufacturerCatalogState(config, { manufacturerId, pos
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const productResult = await queryEndpoint(
|
||||
config,
|
||||
@@ -49,6 +52,7 @@ export async function loadManufacturerCatalogState(config, { manufacturerId, pos
|
||||
refresh: {
|
||||
operation: "catalog-maker-refresh-manufacturer",
|
||||
statements: refreshResults.length,
|
||||
skipped: refreshResults.length === 0,
|
||||
},
|
||||
product: normalizeProduct(product),
|
||||
combinations: combinationsResult.items.map(normalizeCombination),
|
||||
@@ -126,16 +130,23 @@ export async function listSuggestedProducts(
|
||||
|
||||
export async function listMappingSources(config, { manufacturerId, checkUrls = false }) {
|
||||
try {
|
||||
const result = await queryEndpoint(config, mappingSourcesSql({ manufacturerId }));
|
||||
const result = await queryEndpoint(
|
||||
config,
|
||||
mappingSourcesSql({
|
||||
manufacturerId,
|
||||
scrapeDatabase: config.database?.scrapeName || "catalog_scrape",
|
||||
}),
|
||||
);
|
||||
const mappingTableName = `${config.database?.scrapeName || "catalog_scrape"}.ps_product_catalog_manufacturer_source`;
|
||||
const items = result.items.map((row) =>
|
||||
normalizeMappingSource(row, "ps_product_catalog_manufacturer_source"),
|
||||
normalizeMappingSource(row, mappingTableName),
|
||||
);
|
||||
const checkedItems = checkUrls ? await checkMappingSourceUrls(items) : items;
|
||||
|
||||
return {
|
||||
configured: result.configured,
|
||||
mapped: checkedItems.some((item) => item.enabled),
|
||||
tableName: "ps_product_catalog_manufacturer_source",
|
||||
tableName: mappingTableName,
|
||||
items: checkedItems,
|
||||
sourceCount: checkedItems.filter((item) => item.enabled).length,
|
||||
testedCount: checkedItems.filter((item) => item.enabled && item.health === "ok").length,
|
||||
@@ -177,18 +188,6 @@ export async function findProductSources(
|
||||
),
|
||||
);
|
||||
|
||||
for (const item of items) {
|
||||
if (item.source?.key !== "hudy") continue;
|
||||
const candidate = item.candidates?.[0];
|
||||
if (candidate?.url) {
|
||||
try {
|
||||
await openInBrowser(candidate.url);
|
||||
} catch {
|
||||
// Picture extraction can continue even if the visible browser cannot start.
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
configured: mapping.configured,
|
||||
items,
|
||||
@@ -237,6 +236,35 @@ export async function getProductPictures(
|
||||
};
|
||||
}
|
||||
|
||||
export async function saveProductPictureCandidate(config, picture) {
|
||||
return saveScrapePictureCandidate(config, picture);
|
||||
}
|
||||
|
||||
export async function openProductSource(config, { manufacturerId, sourceKey, url }) {
|
||||
const sourceUrl = normalizeSourceUrl(url);
|
||||
const mapping = await listMappingSources(config, { manufacturerId });
|
||||
const source = mapping.items.find(
|
||||
(item) => item.enabled && String(item.key ?? "").toLowerCase() === String(sourceKey ?? "").toLowerCase(),
|
||||
);
|
||||
|
||||
if (!source) {
|
||||
throw new Error("Source mapping was not found for selected manufacturer.");
|
||||
}
|
||||
|
||||
await openInBrowser(sourceUrl, { engine: source.browserEngine });
|
||||
|
||||
return {
|
||||
configured: mapping.configured,
|
||||
opened: true,
|
||||
source: {
|
||||
key: source.key,
|
||||
name: source.name,
|
||||
browserEngine: source.browserEngine,
|
||||
},
|
||||
url: sourceUrl,
|
||||
};
|
||||
}
|
||||
|
||||
async function findSourceCandidates(source, product, { includePictures = false } = {}) {
|
||||
const searchAttempts = buildAdapterSearchUrls(source, product);
|
||||
let lastSearchUrl = searchAttempts[0]?.url || normalizeSourceUrl(source.url);
|
||||
@@ -258,7 +286,8 @@ async function findSourceCandidates(source, product, { includePictures = false }
|
||||
try {
|
||||
const result = await searchInBrowser({
|
||||
url: attempt.url,
|
||||
headless: source.browserHeadless,
|
||||
engine: source.browserEngine,
|
||||
headless: true,
|
||||
waitAfterLoadMs: source.waitAfterLoadMs,
|
||||
});
|
||||
|
||||
@@ -269,7 +298,11 @@ async function findSourceCandidates(source, product, { includePictures = false }
|
||||
attempt,
|
||||
product,
|
||||
);
|
||||
if (includePictures) candidate.pictures = extractPictureUrls(result.pageSource, result.url || attempt.url);
|
||||
if (includePictures) {
|
||||
candidate.pictures = source.key === "idealo"
|
||||
? extractIdealoGalleryPictureUrls(result.pageSource, result.url || attempt.url)
|
||||
: extractPictureUrls(result.pageSource, result.url || attempt.url);
|
||||
}
|
||||
return {
|
||||
source,
|
||||
query: attempt.query,
|
||||
@@ -288,12 +321,6 @@ async function findSourceCandidates(source, product, { includePictures = false }
|
||||
candidates: buildManualSearchCandidates(searchAttempts, product),
|
||||
};
|
||||
} catch (error) {
|
||||
try {
|
||||
await openInBrowser(attempt.url);
|
||||
} catch {
|
||||
// The manual candidate below remains available even if the browser cannot start.
|
||||
}
|
||||
|
||||
return {
|
||||
source,
|
||||
query: attempt.query,
|
||||
@@ -309,6 +336,8 @@ async function findSourceCandidates(source, product, { includePictures = false }
|
||||
lastSearchUrl = attempt.url;
|
||||
try {
|
||||
const response = await scrapePage(attempt.url, {
|
||||
engine: source.browserEngine,
|
||||
headless: true,
|
||||
waitAfterLoadMs: source.waitAfterLoadMs,
|
||||
});
|
||||
const html = response.pageSource;
|
||||
@@ -327,7 +356,7 @@ async function findSourceCandidates(source, product, { includePictures = false }
|
||||
if (candidates.length) {
|
||||
let verification = null;
|
||||
if (includePictures && source.key === "hudy" && product.color && candidates[0].url) {
|
||||
verification = await verifyHudyVariant(candidates[0].url, product);
|
||||
verification = await verifyHudyVariant(candidates[0].url, product, source.browserEngine);
|
||||
candidates[0].verification = verification;
|
||||
if (verification.matchedUrl) candidates[0].url = verification.matchedUrl;
|
||||
if (includePictures && verification.pictures?.length) {
|
||||
@@ -338,7 +367,9 @@ async function findSourceCandidates(source, product, { includePictures = false }
|
||||
if (includePictures) {
|
||||
for (const candidate of candidates) {
|
||||
if (source.key === "hudy" && candidate.url) {
|
||||
candidate.pictures = await extractPicturesFromPage(candidate.url);
|
||||
candidate.pictures = await extractPicturesFromPage(candidate.url, source.browserEngine);
|
||||
} else if (source.key === "idealo") {
|
||||
candidate.pictures = extractIdealoGalleryPictureUrls(html, response.url || attempt.url);
|
||||
} else {
|
||||
candidate.pictures = extractPictureUrls(html, response.url || attempt.url);
|
||||
}
|
||||
@@ -580,7 +611,7 @@ function findColorVariantUrl(html, pageUrl, color) {
|
||||
return "";
|
||||
}
|
||||
|
||||
async function verifyHudyVariant(url, product) {
|
||||
async function verifyHudyVariant(url, product, browserEngine = "chromium") {
|
||||
try {
|
||||
const eanGroup = Array.isArray(product.eanGroup)
|
||||
? product.eanGroup
|
||||
@@ -598,7 +629,7 @@ async function verifyHudyVariant(url, product) {
|
||||
if (visited.has(page.url)) continue;
|
||||
visited.add(page.url);
|
||||
|
||||
const response = await scrapePage(page.url, { waitAfterLoadMs: 750 });
|
||||
const response = await scrapePage(page.url, { engine: browserEngine, waitAfterLoadMs: 750 });
|
||||
const html = response.pageSource;
|
||||
lastStatus = response.status;
|
||||
const pageEans = extractPageEans(html);
|
||||
@@ -700,9 +731,111 @@ function extractPictureUrls(html, pageUrl) {
|
||||
return urls.slice(0, 12);
|
||||
}
|
||||
|
||||
async function extractPicturesFromPage(url) {
|
||||
export function extractIdealoGalleryPictureUrls(html, pageUrl) {
|
||||
const source = String(html ?? "");
|
||||
const galleryMatches = [
|
||||
...source.matchAll(/<li\b[^>]*class=["'][^"']*splide__slide[^"']*["'][^>]*>[\s\S]*?<\/li>/gi),
|
||||
];
|
||||
const orderedGalleryUrls = [];
|
||||
|
||||
for (const match of galleryMatches) {
|
||||
const best = pickBestImageUrlFromMarkup(match[0], pageUrl);
|
||||
if (best && !orderedGalleryUrls.includes(best)) orderedGalleryUrls.push(best);
|
||||
}
|
||||
|
||||
if (orderedGalleryUrls.length) return orderedGalleryUrls.slice(0, 12);
|
||||
|
||||
const trackMatch = source.match(/<div\b[^>]*class=["'][^"']*splide__track[^"']*["'][^>]*>[\s\S]*?<\/div>/i);
|
||||
if (trackMatch) {
|
||||
const best = pickBestImageUrlFromMarkup(trackMatch[0], pageUrl);
|
||||
if (best) return [best];
|
||||
}
|
||||
|
||||
return [];
|
||||
}
|
||||
|
||||
function pickBestImageUrlFromMarkup(markup, pageUrl) {
|
||||
const candidates = [];
|
||||
const add = (value, descriptorScore = 0) => {
|
||||
const normalized = normalizeImageCandidateUrl(value, pageUrl);
|
||||
if (!normalized) return;
|
||||
candidates.push({
|
||||
url: normalized,
|
||||
score: scoreImageCandidate(normalized, descriptorScore),
|
||||
});
|
||||
};
|
||||
|
||||
for (const attribute of ["srcset", "data-srcset"]) {
|
||||
for (const match of markup.matchAll(new RegExp(`${attribute}=["']([^"']+)`, "gi"))) {
|
||||
for (const candidate of parseSrcsetCandidates(match[1], pageUrl)) {
|
||||
add(candidate.url, candidate.score);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
for (const attribute of ["data-large", "data-original", "data-full", "data-zoom-image", "data-src", "src"]) {
|
||||
for (const match of markup.matchAll(new RegExp(`${attribute}=["']([^"']+)`, "gi"))) {
|
||||
add(match[1]);
|
||||
}
|
||||
}
|
||||
|
||||
candidates.sort((a, b) => b.score - a.score);
|
||||
return candidates[0]?.url || "";
|
||||
}
|
||||
|
||||
function parseSrcsetCandidates(value, pageUrl) {
|
||||
return String(value ?? "")
|
||||
.split(",")
|
||||
.map((part) => {
|
||||
const [urlPart, descriptor = ""] = part.trim().split(/\s+/);
|
||||
const descriptorScore = /(\d+)w/i.test(descriptor)
|
||||
? Number(descriptor.match(/(\d+)w/i)?.[1] || 0)
|
||||
: /(\d+(?:\.\d+)?)x/i.test(descriptor)
|
||||
? Number(descriptor.match(/(\d+(?:\.\d+)?)x/i)?.[1] || 0) * 400
|
||||
: 0;
|
||||
return {
|
||||
url: normalizeImageCandidateUrl(urlPart, pageUrl),
|
||||
score: descriptorScore,
|
||||
};
|
||||
})
|
||||
.filter((candidate) => candidate.url);
|
||||
}
|
||||
|
||||
function normalizeImageCandidateUrl(value, pageUrl) {
|
||||
const decoded = decodeHtmlEntities(String(value ?? "").trim());
|
||||
if (!decoded || decoded.startsWith("data:") || /\.svg(?:[?#]|$)/i.test(decoded)) return "";
|
||||
|
||||
try {
|
||||
const response = await scrapePage(url, { waitAfterLoadMs: 750 });
|
||||
const url = new URL(decoded.startsWith("//") ? `https:${decoded}` : decoded, pageUrl).href;
|
||||
if (!/^https?:/i.test(url)) return "";
|
||||
if (/(?:logo|icon|sprite|favicon|placeholder)/i.test(url)) return "";
|
||||
return url;
|
||||
} catch {
|
||||
return "";
|
||||
}
|
||||
}
|
||||
|
||||
function scoreImageCandidate(url, descriptorScore = 0) {
|
||||
const dimensionScore = Math.max(
|
||||
...[...String(url).matchAll(/(?:^|[^0-9])(\d{2,5})[x_-](\d{2,5})(?:[^0-9]|$)/g)].map(
|
||||
(match) => Number(match[1]) * Number(match[2]),
|
||||
),
|
||||
...[...String(url).matchAll(/w(\d{2,5})h(\d{2,5})/gi)].map(
|
||||
(match) => Number(match[1]) * Number(match[2]),
|
||||
),
|
||||
0,
|
||||
);
|
||||
const largePathBonus = /(?:original|large|zoom|full|1000|1200|1500|2000)/i.test(url) ? 100000 : 0;
|
||||
const thumbnailPenalty = /(?:thumb|thumbnail|small|preview|\/s\d+_|_s\d+|150x150|80x80)/i.test(url)
|
||||
? -100000
|
||||
: 0;
|
||||
|
||||
return dimensionScore + descriptorScore + largePathBonus + thumbnailPenalty;
|
||||
}
|
||||
|
||||
async function extractPicturesFromPage(url, browserEngine = "chromium") {
|
||||
try {
|
||||
const response = await scrapePage(url, { engine: browserEngine, waitAfterLoadMs: 750 });
|
||||
return extractHudyGalleryPictureUrls(response.pageSource, response.url || url);
|
||||
} catch {
|
||||
return [];
|
||||
@@ -855,16 +988,20 @@ async function checkMappingSourceUrls(items) {
|
||||
|
||||
return {
|
||||
...item,
|
||||
...(await checkUrl(item.url)),
|
||||
...(await checkUrl(item.url, item.browserEngine, item.browserHeadless)),
|
||||
};
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
async function checkUrl(url) {
|
||||
async function checkUrl(url, browserEngine = "chromium", headless = true) {
|
||||
try {
|
||||
const target = normalizeSourceUrl(url);
|
||||
const response = await scrapePage(target, { waitAfterLoadMs: 250 });
|
||||
const response = await scrapePage(target, {
|
||||
engine: browserEngine,
|
||||
headless,
|
||||
waitAfterLoadMs: 250,
|
||||
});
|
||||
|
||||
return {
|
||||
health: response.ok ? "ok" : "warning",
|
||||
@@ -940,6 +1077,8 @@ function normalizeMappingSource(row, tableName) {
|
||||
browserEngine: String(row.browser_engine ?? "default").trim().toLowerCase(),
|
||||
browserHeadless: normalizeEnabled(row.browser_headless ?? 0),
|
||||
waitAfterLoadMs: Number(row.wait_after_load_ms) || 3000,
|
||||
scraperConfig: parseJsonValue(row.scraper_config, {}),
|
||||
capabilities: parseCapabilities(row.capabilities),
|
||||
enabled: normalizeEnabled(row.mapping_enabled ?? row.enabled ?? row.active ?? 1)
|
||||
&& normalizeEnabled(row.enabled ?? 1),
|
||||
note: row.note ?? row.description ?? "",
|
||||
@@ -947,6 +1086,46 @@ function normalizeMappingSource(row, tableName) {
|
||||
};
|
||||
}
|
||||
|
||||
function parseCapabilities(value) {
|
||||
const parsed = parseJsonValue(value, []);
|
||||
if (!Array.isArray(parsed)) return [];
|
||||
|
||||
return parsed
|
||||
.filter(Boolean)
|
||||
.filter((item) => item.capability)
|
||||
.map((item) => ({
|
||||
capability: item.capability,
|
||||
enabled: normalizeEnabled(item.enabled ?? 1),
|
||||
priority: Number(item.priority) || 100,
|
||||
parserKey: item.parserKey ?? "",
|
||||
parserModule: item.parserModule ?? "",
|
||||
browserEngine: String(item.browserEngine ?? "").trim().toLowerCase(),
|
||||
browserHeadless: item.browserHeadless === null ? null : normalizeEnabled(item.browserHeadless ?? 0),
|
||||
waitAfterLoadMs: item.waitAfterLoadMs === null ? null : Number(item.waitAfterLoadMs) || null,
|
||||
minWidth: item.minWidth === null ? null : Number(item.minWidth) || null,
|
||||
minHeight: item.minHeight === null ? null : Number(item.minHeight) || null,
|
||||
minLongestSide: item.minLongestSide === null ? null : Number(item.minLongestSide) || null,
|
||||
defaultTargetStrategy: item.defaultTargetStrategy ?? "unknown",
|
||||
targetStrategyOverride: item.targetStrategyOverride || "",
|
||||
selectorConfig: parseJsonValue(item.selectorConfig, {}),
|
||||
extractionConfig: parseJsonValue(item.extractionConfig, {}),
|
||||
qualityConfig: parseJsonValue(item.qualityConfig, {}),
|
||||
configOverride: parseJsonValue(item.configOverride, {}),
|
||||
note: item.note ?? "",
|
||||
}));
|
||||
}
|
||||
|
||||
function parseJsonValue(value, fallback) {
|
||||
if (value === null || value === undefined || value === "") return fallback;
|
||||
if (typeof value === "object" && !Buffer.isBuffer(value)) return value;
|
||||
|
||||
try {
|
||||
return JSON.parse(Buffer.isBuffer(value) ? value.toString("utf8") : String(value));
|
||||
} catch {
|
||||
return fallback;
|
||||
}
|
||||
}
|
||||
|
||||
function parseList(value) {
|
||||
return String(value ?? "")
|
||||
.split(",")
|
||||
|
||||
+98
-26
@@ -93,13 +93,9 @@ export function productToDoByManufacturerSql({ manufacturerId, position = 1 }) {
|
||||
tr.total,
|
||||
p.id_product,
|
||||
p.price,
|
||||
p.id_supplier,
|
||||
(
|
||||
SELECT CONCAT('https://www.9b-plus.com/', i.id_image, '-home_default/', pl.link_rewrite, '.jpg')
|
||||
SELECT ${prestaImageUrl("i.id_image", "home_default")}
|
||||
FROM ps_image i
|
||||
JOIN ps_product_lang pl
|
||||
ON pl.id_product = i.id_product
|
||||
AND pl.id_lang = 2
|
||||
WHERE i.id_product = p.id_product
|
||||
ORDER BY i.cover DESC, i.position ASC
|
||||
LIMIT 1
|
||||
@@ -155,13 +151,9 @@ export function productByEanSql({ manufacturerId, ean }) {
|
||||
tr.total,
|
||||
p.id_product,
|
||||
p.price,
|
||||
p.id_supplier,
|
||||
(
|
||||
SELECT CONCAT('https://www.9b-plus.com/', i.id_image, '-home_default/', pl.link_rewrite, '.jpg')
|
||||
SELECT ${prestaImageUrl("i.id_image", "home_default")}
|
||||
FROM ps_image i
|
||||
JOIN ps_product_lang pl
|
||||
ON pl.id_product = i.id_product
|
||||
AND pl.id_lang = 2
|
||||
WHERE i.id_product = p.id_product
|
||||
ORDER BY i.cover DESC, i.position ASC
|
||||
LIMIT 1
|
||||
@@ -347,11 +339,8 @@ export function suggestedProductsSql({ manufacturerId, supplierReference, produc
|
||||
p.supplier_reference,
|
||||
p.active,
|
||||
(
|
||||
SELECT CONCAT('https://www.9b-plus.com/', i.id_image, '-cart_default/', pli.link_rewrite, '.jpg')
|
||||
SELECT ${prestaImageUrl("i.id_image", "cart_default")}
|
||||
FROM ps_image i
|
||||
JOIN ps_product_lang pli
|
||||
ON pli.id_product = i.id_product
|
||||
AND pli.id_lang = 2
|
||||
WHERE i.id_product = p.id_product
|
||||
ORDER BY i.cover DESC, i.position ASC
|
||||
LIMIT 1
|
||||
@@ -373,25 +362,30 @@ export function mappingSourceTablesSql() {
|
||||
FROM information_schema.tables
|
||||
WHERE table_schema = DATABASE()
|
||||
AND table_name IN (
|
||||
'ps_product_catalog_source',
|
||||
'ps_product_catalog_manufacturer_source'
|
||||
'scrape_sources',
|
||||
'scrape_manufacturer_sources',
|
||||
'scrape_source_capabilities',
|
||||
'scrape_manufacturer_source_capabilities'
|
||||
)
|
||||
ORDER BY FIELD(
|
||||
table_name,
|
||||
'ps_product_catalog_manufacturer_source',
|
||||
'ps_product_catalog_source'
|
||||
'scrape_manufacturer_sources',
|
||||
'scrape_sources',
|
||||
'scrape_manufacturer_source_capabilities',
|
||||
'scrape_source_capabilities'
|
||||
)
|
||||
`;
|
||||
}
|
||||
|
||||
export function mappingSourcesSql({ manufacturerId }) {
|
||||
export function mappingSourcesSql({ manufacturerId, scrapeDatabase = "catalog_scrape" }) {
|
||||
const safeManufacturerId = parsePositiveInteger(manufacturerId, "manufacturerId");
|
||||
const scrapeSchema = escapeSqlIdentifier(scrapeDatabase);
|
||||
|
||||
return `
|
||||
SELECT
|
||||
ms.id_product_catalog_manufacturer_source AS mapping_id,
|
||||
ms.id_manufacturer,
|
||||
ms.id_product_catalog_source,
|
||||
ms.id AS mapping_id,
|
||||
ms.manufacturer_id AS id_manufacturer,
|
||||
s.id AS id_product_catalog_source,
|
||||
ms.enabled AS mapping_enabled,
|
||||
ms.priority AS mapping_priority,
|
||||
s.source_key,
|
||||
@@ -410,11 +404,67 @@ export function mappingSourcesSql({ manufacturerId }) {
|
||||
s.browser_engine,
|
||||
s.browser_headless,
|
||||
s.wait_after_load_ms,
|
||||
s.scraper_config,
|
||||
s.note,
|
||||
JSON_ARRAYAGG(
|
||||
CASE
|
||||
WHEN sc.id IS NULL THEN NULL
|
||||
ELSE JSON_OBJECT(
|
||||
'capability', sc.capability,
|
||||
'enabled', COALESCE(msc.enabled, sc.enabled),
|
||||
'priority', COALESCE(msc.priority, sc.priority),
|
||||
'parserKey', sc.parser_key,
|
||||
'parserModule', sc.parser_module,
|
||||
'browserEngine', COALESCE(sc.browser_engine, ''),
|
||||
'browserHeadless', sc.browser_headless,
|
||||
'waitAfterLoadMs', sc.wait_after_load_ms,
|
||||
'minWidth', sc.min_width,
|
||||
'minHeight', sc.min_height,
|
||||
'minLongestSide', sc.min_longest_side,
|
||||
'defaultTargetStrategy', sc.default_target_strategy,
|
||||
'targetStrategyOverride', COALESCE(msc.target_strategy_override, ''),
|
||||
'selectorConfig', sc.selector_config,
|
||||
'extractionConfig', sc.extraction_config,
|
||||
'qualityConfig', sc.quality_config,
|
||||
'configOverride', msc.config_override,
|
||||
'note', sc.note
|
||||
)
|
||||
END
|
||||
) AS capabilities
|
||||
FROM ${scrapeSchema}.scrape_manufacturer_sources ms
|
||||
JOIN ${scrapeSchema}.scrape_sources s
|
||||
ON s.id = ms.source_id
|
||||
LEFT JOIN ${scrapeSchema}.scrape_source_capabilities sc
|
||||
ON sc.source_id = s.id
|
||||
AND sc.enabled = 1
|
||||
LEFT JOIN ${scrapeSchema}.scrape_manufacturer_source_capabilities msc
|
||||
ON msc.source_capability_id = sc.id
|
||||
AND msc.manufacturer_source_id = ms.id
|
||||
WHERE ms.manufacturer_id = ${safeManufacturerId}
|
||||
GROUP BY
|
||||
ms.id,
|
||||
ms.manufacturer_id,
|
||||
s.id,
|
||||
ms.enabled,
|
||||
ms.priority,
|
||||
s.source_key,
|
||||
s.source_name,
|
||||
s.source_type,
|
||||
s.base_url,
|
||||
s.enabled,
|
||||
s.priority,
|
||||
s.search_primary_key,
|
||||
s.search_fallback_keys,
|
||||
s.fallback_source_type,
|
||||
s.search_url_template,
|
||||
s.fallback_url_template,
|
||||
s.search_context_keys,
|
||||
s.access_mode,
|
||||
s.browser_engine,
|
||||
s.browser_headless,
|
||||
s.wait_after_load_ms,
|
||||
s.scraper_config,
|
||||
s.note
|
||||
FROM ps_product_catalog_manufacturer_source ms
|
||||
JOIN ps_product_catalog_source s
|
||||
ON s.id_product_catalog_source = ms.id_product_catalog_source
|
||||
WHERE ms.id_manufacturer = ${safeManufacturerId}
|
||||
ORDER BY
|
||||
ms.priority ASC,
|
||||
s.priority ASC,
|
||||
@@ -422,6 +472,28 @@ export function mappingSourcesSql({ manufacturerId }) {
|
||||
`;
|
||||
}
|
||||
|
||||
function escapeSqlIdentifier(value) {
|
||||
const text = String(value ?? "").trim();
|
||||
if (!/^[a-zA-Z0-9_]+$/.test(text)) {
|
||||
throw new Error("SQL identifier contains unsupported characters.");
|
||||
}
|
||||
return `\`${text}\``;
|
||||
}
|
||||
|
||||
function prestaImageUrl(idExpression, size) {
|
||||
const safeSize = String(size ?? "").trim();
|
||||
if (!/^[a-zA-Z0-9_]+$/.test(safeSize)) {
|
||||
throw new Error("PrestaShop image size contains unsupported characters.");
|
||||
}
|
||||
|
||||
const idText = `CAST(${idExpression} AS CHAR)`;
|
||||
const pathParts = Array.from({ length: 10 }, (_, index) =>
|
||||
`IF(CHAR_LENGTH(${idText}) > ${index}, CONCAT(SUBSTRING(${idText}, ${index + 1}, 1), '/'), '')`,
|
||||
).join(", ");
|
||||
|
||||
return `CONCAT('https://www.9b-plus.com/img/p/', ${pathParts}, ${idExpression}, '-${safeSize}.jpg')`;
|
||||
}
|
||||
|
||||
function escapeSqlString(value) {
|
||||
return String(value ?? "").replaceAll("\\", "\\\\").replaceAll("'", "''").trim();
|
||||
}
|
||||
|
||||
+221
-39
@@ -41,45 +41,102 @@
|
||||
}
|
||||
|
||||
@layer components {
|
||||
.catalog-screen { @apply grid min-h-screen grid-cols-[172px_292px_minmax(720px,1fr)] bg-catalog-surface; }
|
||||
.control-panel { @apply flex min-h-screen flex-col gap-1.5 overflow-y-auto bg-slate-900 p-1.5 text-[11px] text-slate-200; }
|
||||
.catalog-screen { @apply relative grid min-h-screen bg-catalog-surface pl-72; --secondary-track: minmax(240px,292px); --workspace-track: minmax(0,1fr); grid-template-columns: var(--secondary-track) var(--workspace-track); }
|
||||
.catalog-screen.sidebar-collapsed { @apply pl-11; }
|
||||
.catalog-screen.secondary-collapsed { --secondary-track: 44px; }
|
||||
.catalog-screen.workspace-collapsed { --workspace-track: 44px; }
|
||||
.sidebar-drawer { @apply fixed inset-y-0 left-0 z-40 flex w-72 min-h-screen flex-col gap-1.5 overflow-y-auto bg-slate-900 p-1.5 text-[11px] text-slate-200 shadow-xl transition-transform duration-200 ease-out; }
|
||||
.panel-header, .sidebar-drawer-header, .column-header { @apply box-border flex h-7 min-h-7 items-center justify-between gap-1 border-b border-slate-700 bg-slate-900 px-1 text-[10px] font-semibold uppercase tracking-wide text-slate-300; }
|
||||
.sidebar-drawer-header { @apply -mx-1.5 -mt-1.5 mb-1.5; }
|
||||
.sidebar-drawer-title { @apply px-1 text-slate-300; }
|
||||
.sidebar-close-button, .sidebar-open-button { @apply inline-flex h-7 w-7 items-center justify-center rounded border border-slate-700 bg-slate-800 text-slate-200 shadow-sm transition hover:border-slate-500 hover:bg-slate-700; }
|
||||
.sidebar-close-button .sidebar-icon, .sidebar-open-button .sidebar-icon { @apply h-4 w-4; }
|
||||
.panel-control-button { @apply inline-flex h-5 w-5 shrink-0 items-center justify-center rounded border border-slate-400 bg-white text-sm leading-none text-slate-700 shadow-sm transition hover:border-slate-500 hover:bg-slate-200 focus:outline-none focus:ring-2 focus:ring-catalog-focus/25; }
|
||||
.sidebar-open-button { @apply fixed left-2 top-2 z-30; }
|
||||
.sidebar-overlay { @apply fixed inset-0 z-30 bg-slate-950/50; }
|
||||
.sidebar-overlay[hidden] { display: none; }
|
||||
.sidebar-drawer[aria-hidden="true"] { @apply -translate-x-full; }
|
||||
.sidebar-section { @apply border-t border-slate-700 pt-1; }
|
||||
.drawer-trigger { @apply flex min-h-8 w-full items-center gap-2 rounded border border-slate-700 bg-slate-800 px-2 text-left text-[11px] font-semibold text-slate-100 transition hover:border-slate-500 hover:bg-slate-700; }
|
||||
.drawer-trigger .drawer-chevron { @apply ml-auto text-sm text-slate-400; }
|
||||
.drawer-trigger { @apply relative flex min-h-8 w-full items-center justify-center rounded border border-slate-700 bg-slate-800 px-8 text-center text-[11px] font-semibold text-slate-100 transition hover:border-slate-500 hover:bg-slate-700; }
|
||||
.drawer-trigger .sidebar-icon { @apply static; }
|
||||
.drawer-trigger > span:not(.drawer-chevron) { @apply min-w-0 truncate text-center; }
|
||||
.drawer-trigger .drawer-chevron { @apply absolute right-2 inline-flex h-2 w-2 border-b-2 border-r-2 text-[0px] leading-none; }
|
||||
.drawer-trigger .drawer-chevron { @apply rotate-45 text-slate-300; }
|
||||
.drawer-trigger[aria-expanded="true"] .drawer-chevron { @apply rotate-[225deg]; }
|
||||
.parser-drawer[hidden] { display: none; }
|
||||
.sidebar-icon { @apply inline-flex h-4 w-4 shrink-0 items-center justify-center text-sm leading-none text-slate-300; }
|
||||
.action-button { @apply flex items-center justify-center gap-1.5; }
|
||||
.action-button > .sidebar-icon { @apply shrink-0; }
|
||||
.action-button > span { @apply min-w-0 truncate text-center; }
|
||||
.action-button .sidebar-icon { @apply text-slate-400; }
|
||||
.sidebar-footer { @apply mt-auto grid gap-1 border-t border-slate-700 pt-2; }
|
||||
.app-status { @apply rounded border border-slate-700 bg-slate-950 p-1 text-[10px] leading-tight text-slate-100; }
|
||||
.database-status-content { @apply flex items-center justify-center gap-1.5 text-center; }
|
||||
.database-state-dot { @apply inline-block h-2.5 w-2.5 shrink-0 rounded-full border border-slate-500; }
|
||||
.database-state-dot.is-ready { @apply border-emerald-300 bg-emerald-500; }
|
||||
.database-state-dot.is-pending { @apply border-amber-300 bg-amber-400; }
|
||||
.database-state-dot.is-error { @apply border-red-300 bg-red-500; }
|
||||
.safe-mode, #modePill { @apply inline-block rounded bg-slate-100 px-1 py-0.5 text-[10px] font-semibold text-slate-800; }
|
||||
#databaseStatus.database-error { @apply bg-red-100 text-red-700; }
|
||||
.record-id { @apply flex h-10 items-center justify-center rounded border border-lime-700 bg-catalog-accent text-[26px] font-bold leading-none text-catalog-accent-text; }
|
||||
.arrow-row { @apply grid grid-cols-2 gap-1; }
|
||||
.arrow-button, .action-button, .input-like { @apply w-full rounded border border-slate-700 bg-slate-800 text-center text-slate-200 shadow-sm transition; }
|
||||
.arrow-button, .action-button { @apply min-h-7 px-1 text-[11px]; }
|
||||
.arrow-button { @apply relative text-transparent; }
|
||||
.arrow-button::before { @apply absolute left-1/2 top-1/2 -translate-x-1/2 -translate-y-1/2 text-xl leading-none text-slate-100; content: ""; }
|
||||
.arrow-button-prev::before { content: "‹"; }
|
||||
.arrow-button-next::before { content: "›"; }
|
||||
.arrow-button { @apply relative text-slate-100; }
|
||||
.arrow-button::before,
|
||||
.column-toggle::before { content: ""; @apply block h-2 w-2 border-b-2 border-r-2; }
|
||||
.arrow-button::before { @apply absolute left-1/2 top-1/2 h-2 w-2 -translate-x-1/2 -translate-y-1/2 border-b-2 border-r-2; }
|
||||
.arrow-button-prev::before { @apply rotate-[135deg]; }
|
||||
.arrow-button-next::before { @apply rotate-[-45deg]; }
|
||||
.parser-actions { @apply grid gap-1; }
|
||||
.parser-actions .action-button { @apply min-h-7; }
|
||||
.action-button.primary { @apply min-h-8 font-semibold; }
|
||||
.arrow-button:hover:not(:disabled), .action-button:hover:not(:disabled) { @apply border-slate-500 bg-slate-700 text-white; }
|
||||
.arrow-button:disabled, .action-button:disabled { @apply cursor-not-allowed opacity-50; }
|
||||
.input-like { @apply h-7 px-1 text-[11px]; }
|
||||
.select-control { @apply relative w-full; }
|
||||
.select-display { @apply pointer-events-none absolute inset-0 z-30 flex items-center justify-center gap-1.5 text-center text-slate-200; }
|
||||
.select-display span { @apply min-w-0 max-w-[calc(100%-3rem)] truncate; }
|
||||
.language-native-select { @apply relative z-20 cursor-pointer appearance-none bg-transparent pr-7 pl-7 !text-transparent caret-transparent; }
|
||||
.language-native-select:focus { @apply !text-transparent; }
|
||||
.select-chevron { @apply pointer-events-none absolute right-3 top-1/2 z-30 h-2 w-2 -translate-y-1/2 rotate-45 border-b-2 border-r-2 text-slate-200; }
|
||||
.native-manufacturer-host { @apply hidden; }
|
||||
.native-manufacturer-select { @apply hidden; }
|
||||
.manufacturer-picker { @apply relative; }
|
||||
.manufacturer-trigger { @apply relative flex h-7 w-full items-center justify-center gap-1.5 rounded border border-slate-700 bg-slate-800 px-9 text-center text-[11px] text-slate-200 shadow-sm; }
|
||||
.manufacturer-trigger > span:not(.select-chevron) { @apply min-w-0 truncate text-center; }
|
||||
.manufacturer-trigger:hover { @apply border-slate-500 bg-slate-700; }
|
||||
.manufacturer-trigger[aria-expanded="true"] { @apply border-catalog-focus ring-2 ring-catalog-focus/25; }
|
||||
.manufacturer-trigger .select-chevron { @apply right-3 top-1/2 ml-0 translate-y-[-50%]; }
|
||||
.manufacturer-menu { @apply absolute left-0 right-0 top-full z-50 mt-1 rounded border border-slate-300 bg-white p-1 shadow-xl; }
|
||||
.manufacturer-menu[hidden] { display: none; }
|
||||
.manufacturer-search { @apply mb-1 h-7 w-full rounded border border-slate-300 px-2 text-[11px] text-slate-800 outline-none focus:border-catalog-focus focus:ring-2 focus:ring-catalog-focus/25; }
|
||||
.manufacturer-options { @apply max-h-60 overflow-y-auto; }
|
||||
.manufacturer-option { @apply block min-h-7 w-full truncate rounded px-2 py-1 text-left text-[11px] text-slate-700 hover:bg-slate-100; }
|
||||
.manufacturer-option[aria-selected="true"] { @apply bg-slate-200 font-medium text-slate-900; }
|
||||
.manufacturer-empty { @apply px-2 py-2 text-[11px] text-slate-500; }
|
||||
#scanInput { @apply bg-slate-700 placeholder:text-slate-400; }
|
||||
.input-like:focus { @apply border-catalog-focus outline-none ring-2 ring-catalog-focus/25; }
|
||||
.picture-slot { @apply aspect-square w-full overflow-hidden rounded border border-dashed border-slate-500 bg-catalog-panel-soft p-1 text-center text-[11px] text-slate-200; }
|
||||
.picture-slot img { @apply h-full w-full object-contain; }
|
||||
.mapping-box { @apply rounded border border-slate-500 bg-catalog-panel-soft; }
|
||||
.mapping-details summary { @apply grid cursor-pointer grid-cols-[minmax(0,1fr)_22px] items-center px-1 py-1 list-none; }
|
||||
.mapping-box { @apply relative rounded border border-slate-500 bg-catalog-panel-soft; }
|
||||
.sidebar-mapping-box { @apply overflow-hidden border-slate-700 bg-slate-900; }
|
||||
.sidebar-mapping-box { @apply overflow-visible; }
|
||||
.mapping-details { @apply relative; }
|
||||
.mapping-summary { @apply relative flex min-h-8 items-center justify-center rounded bg-slate-800 px-9 text-center text-[11px] font-semibold text-slate-100; }
|
||||
.mapping-label { @apply absolute left-1/2 top-1/2 inline-flex max-w-[calc(100%-4.5rem)] -translate-x-1/2 -translate-y-1/2 items-center justify-center gap-1.5 text-center; }
|
||||
.mapping-title { @apply min-w-0 truncate text-center; }
|
||||
.mapping-state-dot { @apply inline-block h-3 w-3 shrink-0 rounded-full border border-slate-500 shadow-sm; }
|
||||
.mapping-state-dot.is-ready { @apply border-emerald-300 bg-emerald-500; }
|
||||
.mapping-state-dot.is-warning { @apply border-amber-300 bg-amber-400; }
|
||||
.mapping-state-dot.is-missing { @apply border-red-300 bg-red-500; }
|
||||
.mapping-state-dot.is-loading { @apply border-slate-300 bg-slate-400; }
|
||||
.mapping-panel { @apply absolute left-0 right-0 top-full z-50 border border-t-0 border-slate-700 bg-slate-950/95 shadow-lg; }
|
||||
.mapping-detail-status { @apply truncate px-2 py-1 text-center text-[10px] font-normal text-slate-300; }
|
||||
.mapping-summary .mapping-toggle { @apply absolute right-2 top-1/2 flex h-4 w-4 -translate-y-1/2 items-center justify-center; }
|
||||
.mapping-details summary { @apply cursor-pointer list-none; }
|
||||
.mapping-details summary::-webkit-details-marker { display: none; }
|
||||
.mapping-toggle::before { content: "›"; @apply inline-block text-center text-lg leading-none text-slate-100; }
|
||||
.mapping-details[open] .mapping-toggle::before { content: "⌄"; }
|
||||
.mapping-state { @apply block truncate rounded px-1 py-0.5 text-[10px] font-normal; }
|
||||
.mapping-state.is-ready { @apply bg-slate-100 text-slate-800; }
|
||||
.mapping-state.is-missing { @apply bg-red-100 text-red-700; }
|
||||
.mapping-toggle::before { content: ""; @apply block h-2 w-2 rotate-45 border-b-2 border-r-2 border-slate-100; }
|
||||
.mapping-list { @apply grid gap-0.5 px-1 pb-1; }
|
||||
.mapping-source-row { @apply grid grid-cols-[minmax(0,1fr)_auto] items-center gap-1 rounded bg-slate-100 px-1 py-0.5 text-[10px] text-slate-800; }
|
||||
.mapping-source-row.is-disabled { @apply opacity-50; }
|
||||
@@ -88,44 +145,71 @@
|
||||
.mapping-source-status.is-ok { @apply bg-emerald-100 text-emerald-700; }
|
||||
.mapping-source-status.is-bad { @apply bg-red-100 text-red-700; }
|
||||
.mapping-source-status.is-muted { @apply bg-slate-200 text-slate-600; }
|
||||
.product-panel { @apply bg-slate-300 pt-[22px]; }
|
||||
.secondary-panel { @apply min-w-0 border-r border-slate-400 bg-slate-300; }
|
||||
.column-header { @apply min-w-0; }
|
||||
.column-toggle { @apply inline-flex h-5 w-5 shrink-0 items-center justify-center rounded border border-slate-400 bg-white text-[0px] leading-none text-slate-700 shadow-sm transition hover:border-slate-500 hover:bg-slate-200 focus:outline-none focus:ring-2 focus:ring-catalog-focus/25; }
|
||||
#secondaryPanelToggle[aria-label="Collapse product details"]::before,
|
||||
#mainWorkspaceToggle[aria-label="Collapse main workspace"]::before { transform: rotate(135deg); }
|
||||
#secondaryPanelToggle[aria-label="Expand product details"]::before,
|
||||
#mainWorkspaceToggle[aria-label="Expand main workspace"]::before { transform: rotate(-45deg); }
|
||||
.secondary-panel .field-grid { @apply overflow-hidden; }
|
||||
.secondary-collapsed .secondary-panel .column-header { @apply h-full flex-col justify-start py-1; writing-mode: vertical-rl; }
|
||||
.secondary-collapsed .secondary-panel .column-header .column-toggle,
|
||||
.workspace-collapsed .main-workspace .workspace-header .column-toggle { writing-mode: horizontal-tb; }
|
||||
.secondary-collapsed .secondary-panel .field-grid { display: none; }
|
||||
.workspace-collapsed .main-workspace .workspace-header { @apply h-full flex-col justify-start py-1; writing-mode: vertical-rl; }
|
||||
.workspace-collapsed .main-workspace .workspace-header span,
|
||||
.workspace-collapsed .main-workspace .table-wrap { display: none; }
|
||||
.field-grid { @apply grid grid-cols-[132px_minmax(0,168px)] content-start; }
|
||||
.field-label, .field-value { @apply min-h-[22px] border-b border-slate-300 px-1 py-0.5 text-[11px] leading-tight; }
|
||||
.field-label { @apply bg-slate-200 text-slate-800; }
|
||||
.field-value { @apply bg-teal-50 text-right text-slate-900; }
|
||||
.field-value[data-field="manufacturer_name"] { @apply font-medium; }
|
||||
.main-workspace { @apply flex max-h-screen min-w-0 flex-col overflow-hidden bg-catalog-surface; }
|
||||
.data-panel { @apply min-w-0 overflow-hidden bg-catalog-surface; }
|
||||
.table-wrap { @apply h-full overflow-auto; }
|
||||
.variant-table { @apply w-full border-collapse text-[11px]; }
|
||||
.variant-table th, .variant-table td { @apply h-[22px] whitespace-nowrap border-b border-r border-slate-300 px-1 py-0.5 text-center; }
|
||||
.variant-table th { @apply sticky top-0 z-10 bg-slate-200 font-medium text-slate-800; }
|
||||
.variant-table td { @apply bg-slate-50 text-slate-800; }
|
||||
.variant-table tr:nth-child(even) td { @apply bg-slate-100; }
|
||||
.variant-table tr:hover td { @apply bg-slate-200; }
|
||||
.empty-row { @apply py-4 text-slate-500; }
|
||||
.variant-table .problem-ean { @apply bg-red-600 font-semibold text-white; }
|
||||
.table-wrap { @apply min-h-0 flex-1 overflow-auto; scrollbar-gutter: stable; }
|
||||
.variant-grid { @apply min-w-0 text-[11px]; min-width: var(--variant-min-width, 820px); }
|
||||
.variant-header, .variant-row { @apply grid; grid-template-columns: var(--variant-columns); }
|
||||
.variant-header { @apply sticky top-0 z-10; }
|
||||
.variant-header-cell { @apply h-[22px] whitespace-nowrap border-b border-r border-slate-300 bg-slate-200 px-1 py-0.5 font-medium text-slate-800; }
|
||||
.variant-row { @apply min-h-[22px] border-b border-slate-300; }
|
||||
.variant-row:nth-child(even) { @apply bg-slate-100; }
|
||||
.variant-row:hover { @apply bg-slate-200; }
|
||||
.variant-cell { @apply min-w-0 whitespace-nowrap border-r border-slate-300 px-1 py-0.5 text-center text-slate-800; }
|
||||
.variant-cell[data-label=""] { @apply p-0; }
|
||||
.empty-row { @apply py-4 text-center text-slate-500; }
|
||||
.variant-cell.problem-ean { @apply bg-red-600 font-semibold text-white; }
|
||||
.modal-backdrop { @apply fixed inset-0 z-50 flex items-center justify-center bg-slate-900/50 p-4; }
|
||||
.modal-backdrop[hidden] { display: none; }
|
||||
.suggested-dialog, .source-dialog, .settings-dialog { @apply max-h-[90vh] w-full overflow-hidden rounded-lg border border-slate-300 bg-slate-50 shadow-2xl; }
|
||||
.source-dialog-actions { @apply flex items-center gap-1.5; }
|
||||
.suggested-dialog { @apply max-w-2xl; }
|
||||
.source-dialog { @apply max-w-3xl; }
|
||||
.settings-dialog { @apply max-w-md; }
|
||||
.suggested-header { @apply flex items-center justify-between border-b border-slate-300 bg-slate-200 px-3 py-2; }
|
||||
.suggested-header h2 { @apply m-0 text-sm font-semibold text-slate-800; }
|
||||
.modal-close, .settings-test-button { @apply rounded border border-slate-300 bg-white px-2 py-1 text-[11px] text-slate-800 shadow-sm hover:bg-slate-100; }
|
||||
.modal-close, .settings-test-button, .settings-inline-button { @apply inline-flex items-center justify-center gap-1.5 rounded border border-slate-300 bg-white px-2 py-1 text-[11px] text-slate-800 shadow-sm hover:bg-slate-100; }
|
||||
.modal-close .sidebar-icon, .settings-test-button .sidebar-icon, .settings-inline-button .sidebar-icon { @apply h-3.5 w-3.5 text-slate-500; }
|
||||
.suggested-list, .source-list, .settings-content { @apply max-h-[calc(90vh-48px)] overflow-y-auto p-2; }
|
||||
.settings-content { @apply grid gap-2; }
|
||||
.settings-group { @apply m-0 rounded border border-slate-300 p-2; }
|
||||
.settings-group legend { @apply px-1 text-xs font-semibold text-slate-700; }
|
||||
.settings-option { @apply flex items-center gap-1 py-0.5 text-[11px] text-slate-700; }
|
||||
.settings-option .settings-inline-button { @apply ml-auto py-0.5 text-[10px]; }
|
||||
.settings-tool-actions { @apply ml-auto flex items-center gap-1; }
|
||||
.settings-tool-actions .settings-inline-button { @apply ml-0; }
|
||||
.settings-option input { @apply accent-catalog-focus; }
|
||||
.settings-option input:disabled + span { @apply text-slate-400; }
|
||||
.local-db-fields { @apply grid gap-1; }
|
||||
.local-db-fields[hidden] { display: none; }
|
||||
.settings-field { @apply grid grid-cols-[72px_minmax(0,1fr)] items-center gap-2 text-[11px] text-slate-700; }
|
||||
.settings-field input { @apply min-w-0 rounded border border-slate-300 bg-white px-1 py-1 text-[11px]; }
|
||||
.settings-select { @apply min-w-0 rounded border border-slate-300 bg-white px-1 py-1 text-[11px] text-slate-800; }
|
||||
.settings-note, .settings-permissions { @apply m-0 text-[10px] leading-tight text-slate-500; }
|
||||
.settings-permissions { @apply grid gap-0.5; }
|
||||
.adminer-login-hint { @apply mt-1 grid gap-0.5 rounded border border-slate-300 bg-slate-100 p-1.5 text-[10px] leading-tight text-slate-600; }
|
||||
.adminer-login-row { @apply flex items-center justify-between gap-2; }
|
||||
.adminer-login-row strong { @apply truncate text-right font-semibold text-slate-800; }
|
||||
.connection-ok { @apply text-emerald-700; }
|
||||
.connection-error { @apply text-red-700; }
|
||||
.suggested-row { @apply grid grid-cols-[40px_minmax(0,1fr)_auto] items-center gap-2 border-b border-slate-200 px-1 py-1 text-[11px]; }
|
||||
@@ -143,34 +227,122 @@
|
||||
.source-status, .source-row-action { @apply shrink-0 rounded px-1 text-[10px]; }
|
||||
.source-status.is-ok { @apply bg-emerald-100 text-emerald-700; }
|
||||
.source-status.is-muted { @apply bg-slate-200 text-slate-600; }
|
||||
.picture-candidate { @apply grid grid-cols-[72px_minmax(0,1fr)] items-center gap-2 border-b border-slate-200 p-1 text-[10px]; }
|
||||
.picture-candidate img { @apply h-16 w-[72px] object-contain; }
|
||||
.picture-candidate span { @apply min-w-0 truncate; }
|
||||
.picture-candidate { @apply grid grid-cols-[72px_minmax(0,1fr)_auto] items-center gap-2 border-b border-slate-200 p-2 text-[10px]; }
|
||||
.picture-candidate.is-too-small { @apply bg-amber-50; }
|
||||
.picture-candidate-preview { @apply flex h-16 w-[72px] items-center justify-center bg-white; }
|
||||
.picture-candidate img { @apply max-h-16 w-[72px] object-contain; }
|
||||
.picture-candidate-meta { @apply grid min-w-0 gap-0.5; }
|
||||
.picture-candidate-title { @apply min-w-0 truncate font-semibold text-slate-800; }
|
||||
.picture-candidate-dimensions,
|
||||
.picture-candidate-target,
|
||||
.picture-candidate-duplicate { @apply min-w-0 truncate text-slate-500; }
|
||||
.picture-candidate.is-too-small .picture-candidate-dimensions { @apply font-semibold text-amber-700; }
|
||||
.picture-candidate-save { @apply min-w-[76px]; }
|
||||
@media (max-width: 980px) {
|
||||
.catalog-screen { @apply block; }
|
||||
.control-panel { @apply min-h-0 grid grid-cols-2; }
|
||||
.control-panel > * { @apply min-w-0; }
|
||||
.sidebar-footer { @apply col-span-2; }
|
||||
.product-panel { @apply pt-0; }
|
||||
.catalog-screen { @apply flex flex-col pl-0; }
|
||||
.catalog-screen.sidebar-collapsed { @apply pl-0; }
|
||||
.sidebar-drawer { @apply w-[min(18rem,calc(100vw-1rem))] min-h-0; }
|
||||
.sidebar-drawer:not([aria-hidden="true"]) { @apply max-h-screen; }
|
||||
.sidebar-drawer[aria-hidden="true"] { @apply -translate-x-full; }
|
||||
.sidebar-open-button { @apply left-2 top-2; }
|
||||
.secondary-panel { @apply border-r-0 pt-0; }
|
||||
.secondary-panel { @apply order-1; }
|
||||
.main-workspace { @apply order-2; }
|
||||
.table-wrap { @apply max-h-[70vh]; }
|
||||
.column-header { @apply h-7; writing-mode: horizontal-tb; }
|
||||
.secondary-collapsed .secondary-panel .column-header,
|
||||
.workspace-collapsed .main-workspace .workspace-header { @apply h-7 flex-row py-0; writing-mode: horizontal-tb; }
|
||||
.secondary-collapsed .secondary-panel .column-header span,
|
||||
.workspace-collapsed .main-workspace .workspace-header span { display: inline; }
|
||||
.workspace-collapsed .main-workspace .table-wrap { display: none; }
|
||||
.variant-grid { @apply min-w-0; }
|
||||
.variant-header { @apply hidden; }
|
||||
.variant-row { @apply grid grid-cols-2 gap-x-2 border-b border-slate-300 p-2; grid-template-columns: repeat(2, minmax(0, 1fr)); }
|
||||
.variant-cell { @apply flex min-h-7 items-center justify-between gap-2 border-b border-slate-200 px-1 py-1 text-right; white-space: normal; }
|
||||
.variant-cell::before { content: attr(data-label); @apply shrink-0 text-left text-[10px] font-medium text-slate-500; }
|
||||
.variant-cell[data-label=""]::before { content: none; }
|
||||
.variant-cell[data-label=""] { @apply col-span-2; }
|
||||
.variant-message-row { @apply block; }
|
||||
html.dark .variant-row { @apply border-slate-700; }
|
||||
html.dark .variant-cell { @apply border-slate-700; }
|
||||
}
|
||||
|
||||
@media (min-width: 981px) {
|
||||
.sidebar-overlay { display: none; }
|
||||
}
|
||||
|
||||
html:not(.dark) .control-panel { @apply bg-slate-100 text-slate-700; }
|
||||
html:not(.dark) .sidebar-drawer-header,
|
||||
html:not(.dark) .sidebar-footer { @apply border-slate-300; }
|
||||
html:not(.dark) .panel-header,
|
||||
html:not(.dark) .sidebar-drawer-header,
|
||||
html:not(.dark) .column-header { @apply border-slate-300 bg-slate-100 text-slate-700; }
|
||||
html:not(.dark) .sidebar-drawer-title { @apply text-slate-700; }
|
||||
html:not(.dark) .drawer-trigger .drawer-chevron,
|
||||
html:not(.dark) .select-chevron,
|
||||
html:not(.dark) .select-display,
|
||||
html:not(.dark) .manufacturer-trigger .sidebar-icon { @apply text-slate-700; }
|
||||
html:not(.dark) .sidebar-close-button,
|
||||
html:not(.dark) .sidebar-open-button,
|
||||
html:not(.dark) .drawer-trigger,
|
||||
html:not(.dark) .arrow-button,
|
||||
html:not(.dark) .action-button,
|
||||
html:not(.dark) .input-like { @apply border-slate-300 bg-white text-slate-700; }
|
||||
html:not(.dark) .panel-control-button { @apply border-slate-400 bg-white text-slate-700; }
|
||||
html:not(.dark) .sidebar-close-button:hover,
|
||||
html:not(.dark) .sidebar-open-button:hover,
|
||||
html:not(.dark) .drawer-trigger:hover,
|
||||
html:not(.dark) .arrow-button:hover:not(:disabled),
|
||||
html:not(.dark) .action-button:hover:not(:disabled) { @apply border-slate-400 bg-slate-200 text-slate-900; }
|
||||
html:not(.dark) .arrow-button::before { @apply text-slate-700; }
|
||||
html:not(.dark) #scanInput { @apply bg-slate-200 placeholder:text-slate-500; }
|
||||
html:not(.dark) .picture-slot,
|
||||
html:not(.dark) .mapping-box { @apply border-slate-300 bg-white text-slate-600; }
|
||||
html:not(.dark) .mapping-toggle::before { @apply border-slate-700; }
|
||||
html:not(.dark) .app-status { @apply border-slate-300 bg-white text-slate-700; }
|
||||
html:not(.dark) .sidebar-mapping-box { @apply border-slate-300 bg-slate-100; }
|
||||
html:not(.dark) .mapping-summary { @apply border border-slate-300 bg-white text-slate-700; }
|
||||
html:not(.dark) .mapping-panel { @apply border-slate-300 bg-white; }
|
||||
html:not(.dark) .mapping-detail-status { @apply text-slate-600; }
|
||||
html:not(.dark) .manufacturer-trigger { @apply border-slate-300 bg-white text-slate-700; }
|
||||
html:not(.dark) .manufacturer-trigger:hover { @apply border-slate-400 bg-slate-200; }
|
||||
html.contrast-light-high:not(.dark) .control-panel { @apply bg-slate-200 text-slate-900; }
|
||||
html.contrast-light-high:not(.dark) .action-button,
|
||||
html.contrast-light-high:not(.dark) .arrow-button,
|
||||
html.contrast-light-high:not(.dark) .input-like,
|
||||
html.contrast-light-high:not(.dark) .manufacturer-trigger { @apply border-slate-500 text-slate-900; }
|
||||
html.contrast-light-high:not(.dark) .sidebar-icon { @apply text-slate-700; }
|
||||
|
||||
html.dark body { @apply bg-slate-950 text-slate-100; }
|
||||
html.dark .control-panel { @apply bg-slate-950; }
|
||||
html.dark .product-panel { @apply bg-slate-800; }
|
||||
html.dark .secondary-panel { @apply border-slate-700 bg-slate-800; }
|
||||
html.dark .data-panel { @apply bg-slate-950; }
|
||||
html.dark .field-label { @apply border-slate-700 bg-slate-800 text-slate-200; }
|
||||
html.dark .field-value { @apply border-slate-700 bg-slate-900 text-slate-100; }
|
||||
html.dark .variant-table th { @apply border-slate-700 bg-slate-800 text-slate-100; }
|
||||
html.dark .variant-table td { @apply border-slate-700 bg-slate-900 text-slate-200; }
|
||||
html.dark .variant-table tr:nth-child(even) td { @apply bg-slate-800; }
|
||||
html.dark .variant-table tr:hover td { @apply bg-slate-700; }
|
||||
html.dark .variant-header-cell { @apply border-slate-700 bg-slate-800 text-slate-100; }
|
||||
html.dark .variant-row { @apply border-slate-700; }
|
||||
html.dark .variant-row:nth-child(even) { @apply bg-slate-800; }
|
||||
html.dark .variant-row:hover { @apply bg-slate-700; }
|
||||
html.dark .variant-cell { @apply border-slate-700 text-slate-200; }
|
||||
html.dark .arrow-button,
|
||||
html.dark .action-button,
|
||||
html.dark .input-like { @apply border-slate-600 bg-slate-800 text-slate-100; }
|
||||
html.dark .arrow-button:hover:not(:disabled),
|
||||
html.dark .action-button:hover:not(:disabled) { @apply border-slate-400 bg-slate-700; }
|
||||
html.dark #scanInput { @apply bg-slate-700 placeholder:text-slate-400; }
|
||||
html.dark .manufacturer-menu { @apply border-slate-700 bg-slate-900; }
|
||||
html.dark .manufacturer-search { @apply border-slate-600 bg-slate-800 text-slate-100; }
|
||||
html.dark .manufacturer-option { @apply text-slate-200 hover:bg-slate-800; }
|
||||
html.dark .manufacturer-option[aria-selected="true"] { @apply bg-slate-700 text-white; }
|
||||
html.dark .column-header { @apply border-slate-700 text-slate-300; }
|
||||
html.dark .panel-control-button,
|
||||
html.dark .column-toggle { @apply border-slate-600 bg-slate-800 text-slate-100 hover:border-slate-400 hover:bg-slate-700; }
|
||||
html.contrast-dark-high.dark .control-panel { @apply bg-slate-950 text-white; }
|
||||
html.contrast-dark-high.dark .action-button,
|
||||
html.contrast-dark-high.dark .arrow-button,
|
||||
html.contrast-dark-high.dark .input-like,
|
||||
html.contrast-dark-high.dark .manufacturer-trigger { @apply border-slate-500 bg-slate-700 text-white; }
|
||||
html.contrast-dark-high.dark .sidebar-icon { @apply text-slate-100; }
|
||||
html.dark .settings-dialog,
|
||||
html.dark .suggested-dialog,
|
||||
html.dark .source-dialog { @apply border-slate-700 bg-slate-900; }
|
||||
@@ -181,8 +353,11 @@
|
||||
html.dark .settings-field { @apply text-slate-100; }
|
||||
html.dark .settings-group { @apply border-slate-700; }
|
||||
html.dark .settings-field input { @apply border-slate-600 bg-slate-800 text-slate-100; }
|
||||
html.dark .adminer-login-hint { @apply border-slate-700 bg-slate-800 text-slate-400; }
|
||||
html.dark .adminer-login-row strong { @apply text-slate-100; }
|
||||
html.dark .modal-close,
|
||||
html.dark .settings-test-button,
|
||||
html.dark .settings-inline-button,
|
||||
html.dark .suggested-pick,
|
||||
html.dark .source-row-action,
|
||||
html.dark .source-picture-action { @apply border-slate-600 bg-slate-800 text-slate-100 hover:bg-slate-700; }
|
||||
@@ -191,5 +366,12 @@
|
||||
html.dark .source-row { @apply border-slate-700 bg-slate-900 text-slate-100 hover:bg-slate-800; }
|
||||
html.dark .suggested-row,
|
||||
html.dark .picture-candidate { @apply border-slate-700; }
|
||||
html.dark .picture-candidate.is-too-small { @apply bg-amber-950/40; }
|
||||
html.dark .picture-candidate-preview { @apply bg-slate-800; }
|
||||
html.dark .picture-candidate-title { @apply text-slate-100; }
|
||||
html.dark .picture-candidate-dimensions,
|
||||
html.dark .picture-candidate-target,
|
||||
html.dark .picture-candidate-duplicate { @apply text-slate-400; }
|
||||
html.dark .picture-candidate.is-too-small .picture-candidate-dimensions { @apply text-amber-300; }
|
||||
html.dark .suggested-row:nth-child(even) { @apply bg-slate-800; }
|
||||
}
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
// @ts-nocheck
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { extractIdealoGalleryPictureUrls } from "../src/services/catalog-products.ts";
|
||||
|
||||
test("idealo gallery parser prefers large splide images over thumbnails", () => {
|
||||
const html = `
|
||||
<div class="splide__track container">
|
||||
<ul class="splide__list">
|
||||
<li class="splide__slide">
|
||||
<img
|
||||
src="//img.idealo.com/folder/Product/200123/4/200123456/s1_product_thumb.jpg"
|
||||
srcset="//img.idealo.com/folder/Product/200123/4/200123456/s1_product_150x150.jpg 150w,
|
||||
//img.idealo.com/folder/Product/200123/4/200123456/s1_product_1000x1000.jpg 1000w">
|
||||
</li>
|
||||
<li class="splide__slide">
|
||||
<picture>
|
||||
<source srcset="//img.idealo.com/folder/Product/200123/4/200123456/s2_product_400x400.jpg 400w,
|
||||
//img.idealo.com/folder/Product/200123/4/200123456/s2_product_1200x1200.jpg 1200w">
|
||||
<img src="//img.idealo.com/folder/Product/200123/4/200123456/s2_product_80x80.jpg">
|
||||
</picture>
|
||||
</li>
|
||||
</ul>
|
||||
</div>`;
|
||||
|
||||
const urls = extractIdealoGalleryPictureUrls(html, "https://www.idealo.de/product.html");
|
||||
|
||||
assert.deepEqual(urls, [
|
||||
"https://img.idealo.com/folder/Product/200123/4/200123456/s1_product_1000x1000.jpg",
|
||||
"https://img.idealo.com/folder/Product/200123/4/200123456/s2_product_1200x1200.jpg",
|
||||
]);
|
||||
});
|
||||
|
||||
test("idealo gallery parser does not fall back to search thumbnails", () => {
|
||||
const html = `
|
||||
<main>
|
||||
<article class="search-result">
|
||||
<img src="//img.idealo.com/folder/Product/200123/4/200123456/result_thumb_150x150.jpg">
|
||||
</article>
|
||||
</main>`;
|
||||
|
||||
const urls = extractIdealoGalleryPictureUrls(html, "https://www.idealo.de/preisvergleich/MainSearchProductCategory.html?q=123");
|
||||
|
||||
assert.deepEqual(urls, []);
|
||||
});
|
||||
@@ -0,0 +1,55 @@
|
||||
// @ts-nocheck
|
||||
import test from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import fs from "node:fs";
|
||||
|
||||
const schema = fs.readFileSync("sql/local-scrape-db.sql", "utf8");
|
||||
|
||||
test("local scrape schema stores image binaries with exact and visual hashes", () => {
|
||||
assert.match(schema, /CREATE TABLE IF NOT EXISTS `scrape_image_blobs`/);
|
||||
assert.match(schema, /`sha256` CHAR\(64\) NOT NULL/);
|
||||
assert.match(schema, /`perceptual_hash` CHAR\(16\) NULL/);
|
||||
assert.match(schema, /`image_data` LONGBLOB NOT NULL/);
|
||||
assert.match(schema, /UNIQUE KEY `uq_scrape_image_blobs_sha256` \(`sha256`\)/);
|
||||
});
|
||||
|
||||
test("local scrape schema tracks picture candidates and their target combination logic", () => {
|
||||
assert.match(schema, /CREATE TABLE IF NOT EXISTS `scrape_picture_candidates`/);
|
||||
assert.match(schema, /`target_strategy` ENUM\('unknown', 'color', 'size', 'combination', 'manual'\)/);
|
||||
assert.match(schema, /`target_color` VARCHAR\(255\) NULL/);
|
||||
assert.match(schema, /`target_size` VARCHAR\(128\) NULL/);
|
||||
assert.match(schema, /`target_combi` VARCHAR\(255\) NULL/);
|
||||
assert.match(schema, /`duplicate_status` ENUM\('unique', 'exact_duplicate', 'visual_duplicate', 'review'\)/);
|
||||
});
|
||||
|
||||
test("local scrape schema can learn picture target rules from manual corrections", () => {
|
||||
assert.match(schema, /CREATE TABLE IF NOT EXISTS `scrape_picture_target_rules`/);
|
||||
assert.match(schema, /`product_name_pattern` VARCHAR\(255\) NULL/);
|
||||
assert.match(schema, /`target_strategy` ENUM\('color', 'size', 'combination', 'manual'\) NOT NULL/);
|
||||
assert.match(schema, /`examples_count` INT UNSIGNED NOT NULL DEFAULT 0/);
|
||||
});
|
||||
|
||||
test("local scrape schema stores reusable source capabilities for pictures texts and features", () => {
|
||||
assert.match(schema, /CREATE TABLE IF NOT EXISTS `scrape_source_capabilities`/);
|
||||
assert.match(schema, /`capability` ENUM\('search', 'pictures', 'texts', 'features'\) NOT NULL/);
|
||||
assert.match(schema, /`parser_key` VARCHAR\(128\) NOT NULL/);
|
||||
assert.match(schema, /`selector_config` JSON NULL/);
|
||||
assert.match(schema, /`extraction_config` JSON NULL/);
|
||||
assert.match(schema, /`quality_config` JSON NULL/);
|
||||
assert.match(schema, /UNIQUE KEY `uq_scrape_source_capability_parser`/);
|
||||
});
|
||||
|
||||
test("local scrape schema links manufacturer mappings to selected source capabilities", () => {
|
||||
assert.match(schema, /CREATE TABLE IF NOT EXISTS `scrape_manufacturer_source_capabilities`/);
|
||||
assert.match(schema, /`target_strategy_override` ENUM\('unknown', 'color', 'size', 'combination', 'manual'\) NULL/);
|
||||
assert.match(schema, /`config_override` JSON NULL/);
|
||||
assert.match(schema, /UNIQUE KEY `uq_scrape_manufacturer_source_capability`/);
|
||||
});
|
||||
|
||||
test("local scrape schema seeds Hudy and Idealo extraction capabilities", () => {
|
||||
assert.match(schema, /'hudy-gallery'/);
|
||||
assert.match(schema, /'hudy-product-texts'/);
|
||||
assert.match(schema, /'hudy-product-params'/);
|
||||
assert.match(schema, /'idealo-splide-gallery'/);
|
||||
assert.match(schema, /'productGalleryOnly'/);
|
||||
});
|
||||
@@ -59,6 +59,9 @@ test("mapping source queries are read-only", () => {
|
||||
assert.doesNotThrow(() =>
|
||||
assertReadOnlySql(mappingSourcesSql({ manufacturerId: 13 })),
|
||||
);
|
||||
assert.match(mappingSourcesSql({ manufacturerId: 13 }), /scrape_manufacturer_sources/);
|
||||
assert.match(mappingSourcesSql({ manufacturerId: 13 }), /scrape_source_capabilities/);
|
||||
assert.match(mappingSourcesSql({ manufacturerId: 13 }), /capabilities/);
|
||||
});
|
||||
|
||||
test("manufacturer refresh updates are narrowly allowlisted", () => {
|
||||
|
||||
@@ -23,31 +23,378 @@ test("settings modal opens and shows local database controls", async ({ page })
|
||||
await expect(page.locator("#testLocalDbButton")).toBeVisible();
|
||||
});
|
||||
|
||||
test("manufacturer picker shows ten results and supports search", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
await page.locator("#manufacturerTrigger").click();
|
||||
|
||||
await expect(page.locator("#manufacturerOptions .manufacturer-option")).toHaveCount(10);
|
||||
await expect(page.locator("#manufacturerSearch")).toBeVisible();
|
||||
await page.locator("#manufacturerSearch").fill("La Sportiva");
|
||||
await expect(page.locator("#manufacturerOptions .manufacturer-option")).toHaveText(["La Sportiva"]);
|
||||
});
|
||||
|
||||
test("mapping uses cache on reload and reloads from DB on manual manufacturer change", async ({ page }) => {
|
||||
let mappingRequests = 0;
|
||||
await page.route("**/api/catalog-maker/mapping-sources**", async (route) => {
|
||||
mappingRequests += 1;
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
mapped: true,
|
||||
items: [{ key: "hudy", name: "Hudy", enabled: true }],
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem("catalog-maker:selected-manufacturer", "13");
|
||||
localStorage.setItem(
|
||||
"catalog-maker:mapping-cache",
|
||||
JSON.stringify({
|
||||
13: {
|
||||
mapped: true,
|
||||
items: [{ key: "cached-hudy", name: "Cached Hudy", enabled: true }],
|
||||
},
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
await page.goto("/");
|
||||
await page.waitForFunction(() => document.querySelector("#manufacturerSelect")?.value === "13");
|
||||
await expect(page.locator("#mappingStatusText")).toContainText("Mapping OK");
|
||||
await page.waitForTimeout(300);
|
||||
expect(mappingRequests).toBe(0);
|
||||
|
||||
await page.locator("#manufacturerSelect").evaluate((select) => {
|
||||
const nextOption = [...select.options].find((option) => option.value && option.value !== "13");
|
||||
select.value = nextOption?.value || "";
|
||||
select.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
});
|
||||
await expect.poll(() => mappingRequests).toBe(1);
|
||||
|
||||
await page.locator("#manufacturerSelect").evaluate((select) => {
|
||||
select.value = "13";
|
||||
select.dispatchEvent(new Event("change", { bubbles: true }));
|
||||
});
|
||||
await expect.poll(() => mappingRequests).toBe(2);
|
||||
});
|
||||
|
||||
test("catalog maker remains usable on a narrow viewport", async ({ page }) => {
|
||||
await page.setViewportSize({ width: 390, height: 844 });
|
||||
await page.goto("/");
|
||||
|
||||
await expect(page.locator("#sidebarDrawer")).toHaveAttribute("aria-hidden", "true");
|
||||
await page.locator("#sidebarOpenButton").click();
|
||||
await expect(page.locator("#findSourcesButton")).toBeVisible();
|
||||
await expect(page.locator("#settingsButton")).toBeVisible();
|
||||
await expect(page.locator("#sidebarOverlay")).toBeVisible();
|
||||
await expect(page.locator(".secondary-panel")).toHaveCSS("order", "1");
|
||||
await expect(page.locator(".main-workspace")).toHaveCSS("order", "2");
|
||||
await page.screenshot({ path: "test-results/catalog-maker-mobile.png", fullPage: true });
|
||||
});
|
||||
|
||||
test("parser tools open in a sidebar drawer", async ({ page }) => {
|
||||
test("left controls open in a responsive sidebar drawer", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
await expect(page.locator("#parserDrawer")).toBeVisible();
|
||||
await page.locator("#parserDrawerToggle").click();
|
||||
await expect(page.locator("#parserDrawer")).toBeHidden();
|
||||
await expect(page.locator("#parserDrawerToggle")).toHaveAttribute("aria-expanded", "false");
|
||||
await expect(page.locator("#sidebarDrawer")).toBeVisible();
|
||||
await page.locator("#sidebarCloseButton").click();
|
||||
await expect(page.locator("#sidebarDrawer")).toHaveAttribute("aria-hidden", "true");
|
||||
await page.locator("#sidebarOpenButton").click();
|
||||
await expect(page.locator("#sidebarDrawer")).toBeVisible();
|
||||
await expect(page.locator("#sidebarOverlay")).toBeHidden();
|
||||
});
|
||||
|
||||
test("workspace columns collapse and expand without leaving a gap", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
await page.locator("#sidebarCloseButton").click();
|
||||
await expect(page.locator(".catalog-screen")).toHaveClass(/sidebar-collapsed/);
|
||||
const secondaryPanel = await page.locator(".secondary-panel").boundingBox();
|
||||
expect(secondaryPanel?.x).toBeGreaterThanOrEqual(44);
|
||||
|
||||
await page.locator("#secondaryPanelToggle").click();
|
||||
await expect(page.locator(".catalog-screen")).toHaveClass(/secondary-collapsed/);
|
||||
await page.locator("#mainWorkspaceToggle").click();
|
||||
await expect(page.locator(".catalog-screen")).toHaveClass(/workspace-collapsed/);
|
||||
|
||||
await page.locator("#secondaryPanelToggle").click();
|
||||
await page.locator("#mainWorkspaceToggle").click();
|
||||
await expect(page.locator(".catalog-screen")).not.toHaveClass(/secondary-collapsed/);
|
||||
await expect(page.locator(".catalog-screen")).not.toHaveClass(/workspace-collapsed/);
|
||||
});
|
||||
|
||||
test("main workspace keeps a bottom horizontal scrollbar for wide data", async ({ page }) => {
|
||||
await page.setViewportSize({ width: 1061, height: 912 });
|
||||
await page.goto("/");
|
||||
|
||||
const scrollState = await page.locator(".main-workspace .table-wrap").evaluate((element) => {
|
||||
const style = getComputedStyle(element);
|
||||
return {
|
||||
overflowX: style.overflowX,
|
||||
clientWidth: element.clientWidth,
|
||||
scrollWidth: element.scrollWidth,
|
||||
};
|
||||
});
|
||||
|
||||
expect(scrollState.overflowX).toBe("auto");
|
||||
expect(scrollState.scrollWidth).toBeGreaterThan(scrollState.clientWidth);
|
||||
});
|
||||
|
||||
test("picture candidates show source, dimensions, target assignment and save action", async ({ page }) => {
|
||||
await page.route("**/api/manufacturers", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({ items: [{ id: 13, name: "La Sportiva" }] }),
|
||||
});
|
||||
});
|
||||
await page.route("**/api/languages", async (route) => {
|
||||
await route.fulfill({ contentType: "application/json", body: JSON.stringify({ items: [] }) });
|
||||
});
|
||||
await page.route("**/api/catalog-maker/mapping-sources**", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
mapped: true,
|
||||
items: [{ key: "hudy", name: "Hudy", enabled: true, health: "ok" }],
|
||||
}),
|
||||
});
|
||||
});
|
||||
await page.route("**/api/catalog-maker/product**", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
product: {
|
||||
idProductCatalog: 135520,
|
||||
idProduct: 110927,
|
||||
idManufacturer: 13,
|
||||
codeManufacture: "LAS",
|
||||
supplierReference: "ZFHS099",
|
||||
name: "Aequilibrium Trek Woman GTX",
|
||||
position: 1,
|
||||
total: 380,
|
||||
},
|
||||
combinations: [
|
||||
{
|
||||
idProductCatalog: 110923,
|
||||
status: "SS25/Boty",
|
||||
color: "Carbon/Malibu Blue",
|
||||
combination: "36",
|
||||
ean13: "8058428191284",
|
||||
supplierStock: 1,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
});
|
||||
await page.route("**/api/catalog-maker/get-pictures**", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
items: [
|
||||
{
|
||||
source: { key: "hudy", name: "Hudy", type: "supplier" },
|
||||
searchUrl: "https://www.hudy.cz/vyhledavani?q=8058428191284",
|
||||
candidates: [
|
||||
{
|
||||
rawTitle: "Aequilibrium Trek Woman GTX",
|
||||
url: "https://www.hudy.cz/product",
|
||||
pictures: [
|
||||
"data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='80' height='64'%3E%3Crect width='80' height='64' fill='red'/%3E%3C/svg%3E",
|
||||
"data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='120' height='90'%3E%3Crect width='120' height='90' fill='blue'/%3E%3C/svg%3E",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
});
|
||||
let saveRequests = 0;
|
||||
await page.route("**/api/catalog-maker/save-picture", async (route) => {
|
||||
saveRequests += 1;
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
saved: true,
|
||||
candidateId: saveRequests,
|
||||
duplicateStatus: "unique",
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem("catalog-maker:selected-manufacturer", "13");
|
||||
});
|
||||
await page.goto("/");
|
||||
await page.waitForFunction(() => document.querySelector("#manufacturerSelect")?.value === "13");
|
||||
await page.locator("#getPicturesButton").click();
|
||||
|
||||
await expect(page.locator("#sourceModal")).toBeVisible();
|
||||
await expect(page.locator(".picture-candidate-title").first()).toHaveText("Hudy picture #1");
|
||||
await expect(page.locator(".picture-candidate-dimensions").first()).toContainText("too small, min 800 x 800");
|
||||
await expect(page.locator(".picture-candidate").first()).toHaveClass(/is-too-small/);
|
||||
await expect(page.locator(".picture-candidate-target").first()).toContainText("Target: color Carbon/Malibu Blue");
|
||||
await expect(page.locator(".picture-candidate-save")).toHaveCount(2);
|
||||
await expect(page.locator("#sourceBatchActionButton")).toBeVisible();
|
||||
await expect(page.locator("#sourceBatchActionButton")).toContainText("Save all");
|
||||
|
||||
await page.locator("#sourceBatchActionButton").click();
|
||||
await expect(page.locator(".picture-candidate-save")).toHaveText(["Saved", "Saved"]);
|
||||
expect(saveRequests).toBe(2);
|
||||
});
|
||||
|
||||
test("source candidates can fetch all pictures before saving candidates", async ({ page }) => {
|
||||
await page.route("**/api/manufacturers", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({ items: [{ id: 13, name: "La Sportiva" }] }),
|
||||
});
|
||||
});
|
||||
await page.route("**/api/languages", async (route) => {
|
||||
await route.fulfill({ contentType: "application/json", body: JSON.stringify({ items: [] }) });
|
||||
});
|
||||
await page.route("**/api/catalog-maker/mapping-sources**", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
mapped: true,
|
||||
items: [{ key: "hudy", name: "Hudy", enabled: true, health: "ok" }],
|
||||
}),
|
||||
});
|
||||
});
|
||||
await page.route("**/api/catalog-maker/product**", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
product: {
|
||||
idProductCatalog: 135520,
|
||||
idProduct: 110927,
|
||||
idManufacturer: 13,
|
||||
codeManufacture: "LAS",
|
||||
supplierReference: "ZFHS099",
|
||||
name: "Aequilibrium Trek Woman GTX",
|
||||
position: 1,
|
||||
total: 380,
|
||||
},
|
||||
combinations: [
|
||||
{
|
||||
idProductCatalog: 110923,
|
||||
status: "SS25/Boty",
|
||||
color: "Carbon/Malibu Blue",
|
||||
combination: "36",
|
||||
ean13: "8058428191284",
|
||||
supplierStock: 1,
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
});
|
||||
await page.route("**/api/catalog-maker/find-sources**", async (route) => {
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
items: [
|
||||
{
|
||||
status: "found",
|
||||
source: { key: "hudy", name: "Hudy", type: "supplier" },
|
||||
candidates: [
|
||||
{
|
||||
title: "Direct Hudy / EAN: 8058428191284 / Color: Carbon/Malibu Blue",
|
||||
url: "https://www.hudy.cz/product",
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
});
|
||||
await page.route("**/api/catalog-maker/get-pictures**", async (route) => {
|
||||
expect(route.request().url()).not.toContain("sourceKey=");
|
||||
await route.fulfill({
|
||||
contentType: "application/json",
|
||||
body: JSON.stringify({
|
||||
items: [
|
||||
{
|
||||
source: { key: "hudy", name: "Hudy", type: "supplier" },
|
||||
candidates: [
|
||||
{
|
||||
rawTitle: "Aequilibrium Trek Woman GTX",
|
||||
url: "https://www.hudy.cz/product",
|
||||
pictures: [
|
||||
"data:image/svg+xml,%3Csvg xmlns='http://www.w3.org/2000/svg' width='80' height='64'%3E%3Crect width='80' height='64' fill='red'/%3E%3C/svg%3E",
|
||||
],
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
}),
|
||||
});
|
||||
});
|
||||
|
||||
await page.addInitScript(() => {
|
||||
localStorage.setItem("catalog-maker:selected-manufacturer", "13");
|
||||
});
|
||||
await page.goto("/");
|
||||
await page.waitForFunction(() => document.querySelector("#manufacturerSelect")?.value === "13");
|
||||
await page.locator("#findSourcesButton").click();
|
||||
|
||||
await expect(page.locator("#sourceModal")).toBeVisible();
|
||||
await expect(page.locator("#sourceBatchActionButton")).toBeVisible();
|
||||
await expect(page.locator("#sourceBatchActionButton")).toContainText("Get all pictures");
|
||||
await page.locator("#sourceBatchActionButton").click();
|
||||
await expect(page.locator(".picture-candidate-title").first()).toHaveText("Hudy picture #1");
|
||||
await expect(page.locator("#sourceBatchActionButton")).toContainText("Save all");
|
||||
});
|
||||
|
||||
test("uses one shared height for the top panel headers", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
const heights = await page.locator("#sidebarDrawer .sidebar-drawer-header, .secondary-panel .column-header, .main-workspace .column-header").evaluateAll((elements) => elements.map((element) => Math.round(element.getBoundingClientRect().height)));
|
||||
expect(new Set(heights).size).toBe(1);
|
||||
});
|
||||
|
||||
test("uses one shared visual style for the top panel headers", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
const styles = await page.locator(".panel-header").evaluateAll((elements) => elements.map((element) => {
|
||||
const style = getComputedStyle(element);
|
||||
return [style.height, style.backgroundColor, style.borderBottomColor, style.fontSize, style.fontWeight, style.paddingLeft, style.paddingRight];
|
||||
}));
|
||||
expect(new Set(styles.map((style) => JSON.stringify(style))).size).toBe(1);
|
||||
});
|
||||
|
||||
test("keeps the top panel headers on one baseline", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
const positions = await page.locator(".panel-header").evaluateAll((elements) => elements.map((element) => {
|
||||
const rect = element.getBoundingClientRect();
|
||||
return [rect.top, rect.height];
|
||||
}));
|
||||
expect(new Set(positions.map((position) => JSON.stringify(position))).size).toBe(1);
|
||||
expect(positions[0][0]).toBe(0);
|
||||
});
|
||||
|
||||
test("uses one shared visual style for panel controls", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
const styles = await page.locator("#sidebarCloseButton, #secondaryPanelToggle, #mainWorkspaceToggle").evaluateAll((elements) => elements.map((element) => {
|
||||
const style = getComputedStyle(element);
|
||||
return [style.width, style.height, style.borderColor, style.backgroundColor, style.borderRadius];
|
||||
}));
|
||||
expect(new Set(styles.map((style) => JSON.stringify(style))).size).toBe(1);
|
||||
});
|
||||
|
||||
test("theme toggle switches and persists dark mode", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
await expect(page.locator("html")).not.toHaveClass(/dark/);
|
||||
await page.evaluate(() => localStorage.clear());
|
||||
await page.reload();
|
||||
await expect(page.locator("html")).not.toHaveClass(/(?:^| )dark(?: |$)/);
|
||||
const lightPanelBackground = await page.locator("#sidebarDrawer").evaluate((element) => getComputedStyle(element).backgroundColor);
|
||||
|
||||
await page.locator("#themeToggle").click();
|
||||
await expect(page.locator("html")).toHaveClass(/dark/);
|
||||
await expect(page.locator("html")).toHaveClass(/(?:^| )dark(?: |$)/);
|
||||
await expect(page.locator("#themeToggle")).toHaveText("Light theme");
|
||||
const darkPanelBackground = await page.locator("#sidebarDrawer").evaluate((element) => getComputedStyle(element).backgroundColor);
|
||||
expect(darkPanelBackground).not.toBe(lightPanelBackground);
|
||||
|
||||
await page.locator("#settingsButton").click();
|
||||
await page.locator("#darkContrastSelect").selectOption("high");
|
||||
await expect(page.locator("html")).toHaveClass(/contrast-dark-high/);
|
||||
|
||||
await page.reload();
|
||||
await expect(page.locator("html")).toHaveClass(/dark/);
|
||||
await expect(page.locator("html")).toHaveClass(/(?:^| )dark(?: |$)/);
|
||||
});
|
||||
|
||||
@@ -17,6 +17,35 @@ test.describe("database connection", () => {
|
||||
await expect(page.locator("#localDbTestResult")).toContainText("Connected:");
|
||||
});
|
||||
|
||||
test("keeps the scrape database as a separate local mode with its own permissions", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
await page.locator("#settingsButton").click();
|
||||
|
||||
await expect(page.locator('input[name="databaseMode"][value="local"] + span')).toHaveText("Local 9bplus DB");
|
||||
await expect(page.locator('input[name="databaseMode"][value="live"] + span')).toHaveText("Live 9bplus DB");
|
||||
await page.locator('input[name="databaseMode"][value="scrape"]').check();
|
||||
await expect(page.locator("#databaseModeNote")).toContainText("Local scrape DB selected");
|
||||
await expect(page.locator("#allowInsertToggle")).toBeEnabled();
|
||||
await expect(page.locator("#allowUpdateToggle")).toBeEnabled();
|
||||
await expect(page.locator("#allowDeleteToggle")).toBeEnabled();
|
||||
await page.locator("#allowUpdateToggle").check();
|
||||
|
||||
await page.reload();
|
||||
await page.locator("#settingsButton").click();
|
||||
await page.locator('input[name="databaseMode"][value="scrape"]').check();
|
||||
await expect(page.locator("#allowUpdateToggle")).toBeChecked();
|
||||
|
||||
await page.locator('input[name="databaseMode"][value="local"]').check();
|
||||
await expect(page.locator("#allowInsertToggle")).not.toBeChecked();
|
||||
await expect(page.locator("#allowUpdateToggle")).not.toBeChecked();
|
||||
await expect(page.locator("#allowDeleteToggle")).not.toBeChecked();
|
||||
|
||||
await page.locator('input[name="databaseMode"][value="live"]').check();
|
||||
await expect(page.locator("#allowInsertToggle")).toBeDisabled();
|
||||
await expect(page.locator("#allowUpdateToggle")).toBeDisabled();
|
||||
await expect(page.locator("#allowDeleteToggle")).toBeDisabled();
|
||||
});
|
||||
|
||||
test("loads manufacturers from Local DB without Live endpoint fallback", async ({ page }) => {
|
||||
const manufacturersResponse = page.waitForResponse("**/api/manufacturers");
|
||||
await page.goto("/");
|
||||
@@ -27,4 +56,21 @@ test.describe("database connection", () => {
|
||||
expect(data.items.length).toBeGreaterThan(0);
|
||||
expect(data.items.some((item) => item.name === "La Sportiva")).toBeTruthy();
|
||||
});
|
||||
|
||||
test("persists Local 9bplus write permissions across reloads", async ({ page }) => {
|
||||
await page.goto("/");
|
||||
await page.locator("#settingsButton").click();
|
||||
await page.locator('input[name="databaseMode"][value="local"]').check();
|
||||
await page.locator("#allowInsertToggle").check();
|
||||
await page.locator("#allowUpdateToggle").check();
|
||||
await page.locator("#allowDeleteToggle").check();
|
||||
|
||||
await page.reload();
|
||||
await page.locator("#settingsButton").click();
|
||||
|
||||
await expect(page.locator('input[name="databaseMode"][value="local"]')).toBeChecked();
|
||||
await expect(page.locator("#allowInsertToggle")).toBeChecked();
|
||||
await expect(page.locator("#allowUpdateToggle")).toBeChecked();
|
||||
await expect(page.locator("#allowDeleteToggle")).toBeChecked();
|
||||
});
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user