How to Import Products from Any Online Store? – Product Import Crawler for OpenCart

💡 Quick note: At the end of this article, I’ve included a ready-to-install module package. If you’re just looking for a plug-and-play solution (“install & import”), feel free to scroll straight to the bottom 😉

 

1. Introduction

I built this module because I kept running into the same wall: suppliers with no API, no XML feeds, no database access—nothing. Just HTML. And when you’re asked to import thousands of products under those conditions, manual work is simply not an option if you care about time, budget, or sanity 😅

In real projects, this approach has been a game changer. I’ve used it to import entire catalogs from wholesale stores where the only available source was the frontend. In one case, I had to handle over 7,000 products with zero structured data available, and this exact solution allowed me to deliver without increasing project costs. I described that process in detail here 👉 https://www.designcart.pl/case-study/301-7210-produktow-brak-api-i-72-mln-znakow-do-przetlumaczenia-jak-to-zrobilem-nie-podnoszac-kosztow-projektu.html

Let’s put this into perspective:
if adding a single product in OpenCart takes ~1 minute, importing 6000 products manually means 100+ hours of work. With crawling, we reduce that to a fraction of the time. That’s not optimization—that’s a completely different scale of operation 🚀

 

What this module actually does

At its core, this is a product import crawler for OpenCart that allows me to:

• import products directly from HTML-based supplier stores
• automate catalog creation without structured data sources
• reuse the same logic across multiple projects and suppliers

 

How it works (high-level)

The architecture is intentionally simple but powerful:

• OCMOD – injects the module into OpenCart without touching core files
• Crawler – scans listing pages and product pages using CSS selectors
• Sandbox – stores raw scraped data before import
• Importer – creates actual OpenCart products from selected records

 

Key concept: 2-step process

I never import data directly from the crawler into the catalog. That’s risky.

Instead, I use a controlled 2-step pipeline:

1. Scan → collect and normalize data into a sandbox
2. Import → manually select and push products into the catalog

This gives me:

• full control over data quality
• ability to debug selectors
• protection against bad imports

If you’ve ever been in a situation where:

• you have multiple supplier stores
• each has thousands of products
• and zero integration options

then crawling isn’t a hack — it’s the only sensible solution.

 

2. Architecture of the Solution

Before jumping into code, I always step back and design the big picture. This module isn’t just “some scraper glued into OpenCart”—it’s a structured pipeline that separates concerns and keeps things maintainable over time.

I’m building this on OpenCart, because that’s the platform I use in client projects. But let’s be clear:
👉 the architecture itself is platform-agnostic.
You can port this approach to any open-source e-commerce (PrestaShop, WooCommerce, Magento) with minimal conceptual changes.

 

2.1 System Components

Here’s how I break the system down in practice:

🧩 OCMOD (install.xml)

This is the entry point into OpenCart.

• injects UI into category form and list
• hooks into controller lifecycle
• avoids modifying core files

👉 This is critical. I never touch core—everything goes through OCMOD.

⚙️ Backend (Controller + Model)

This is where the orchestration happens.

• controllers handle requests (scan, import, settings)
• models encapsulate logic (crawler, importer, DB ops)
• everything is split so I can extend it later without chaos

🕷️ Crawler (HTML Parsing)

This is the engine.

• fetches listing pages
• extracts product URLs
• parses product pages using CSS selectors
• builds structured payload (name, price, image, etc.)

👉 No API? No problem. HTML is the API 😎

🧪 Sandbox (Database Layer)

I never import directly into the catalog.

Instead:

• every scanned product goes into a sandbox table
• data is normalized and stored as JSON
• duplicates are prevented via URL uniqueness

This gives me a safe buffer between scraping and real data.

📦 Importer

This is the second stage.

• reads selected sandbox records
• calculates price (including margins)
• creates OpenCart products
• updates metadata (original price, source URL, code)

👉 Important: importer is deterministic and controlled—not automatic.

🖥️ Admin UI

Everything is exposed in the OpenCart admin:

• category-level configuration (selectors, URLs, margins)
• scan screen (preview + diagnostics)
• sandbox table with selectable records
• import action

This is what makes the tool usable in real projects—not just a dev script.

 

🧠 Why this architecture works

Because it separates:

• data collection (crawler)
• data validation (sandbox)
• data execution (importer)

That separation is what keeps large imports safe and predictable.

2.2 Module Structure

Here’s how the module is organized on disk:

dc_crawling/
├── install.xml          # OCMOD — wstrzyknięcia w core
├── install.sql          # ALTER + CREATE (ręcznie w MySQL)
└── upload/              # Nakładka na katalog główny sklepu
    ├── admin/
    │   ├── controller/catalog/dc_crawling_form.php   # dołączany przed output formularza kategorii
    │   ├── controller/catalog/dc_crawling_list.php   # dołączany przed output listy kategorii
    │   ├── controller/dc_crawling/import.php
    │   ├── controller/dc_crawling/settings.php
    │   ├── model/catalog/dc_crawling.php
    │   ├── model/dc_crawling/crawler.php
    │   ├── model/dc_crawling/importer.php
    │   ├── view/template/dc_crawling/...
    │   └── language/...
    └── system/library/simple_html_dom.php

This structure follows a simple rule I always stick to:

• everything that extends OpenCart goes into upload/
• everything that modifies OpenCart goes into install.xml
• everything that touches the database goes into install.sql

👉 Once you understand this layout, navigating and extending the module becomes trivial.

 

3. Installation

This is the part where things can either go smoothly… or burn 30 minutes of your life for no good reason 😅
I’ll walk you through exactly how I install this module in real projects—and where things usually break.

 

3.1 OCMOD Installation

First, the core of everything:

<?xml version="1.0" encoding="utf-8"?>
<modification>
	<name>Design Cart Crawling</name>
	<code>dc_crawling</code>
	<version>1.0.0</version>
	<author>Design Cart</author>
	<link>https://designcart.pl</link>

	<file path="admin/controller/catalog/category.php">
		<operation>
			<search><![CDATA[$this->model_catalog_category->addCategory($this->request->post);]]></search>
			<add position="replace"><![CDATA[$category_id = $this->model_catalog_category->addCategory($this->request->post);

			$this->load->model('catalog/dc_crawling');
			$this->model_catalog_dc_crawling->saveCategoryImportFromPost($category_id, $this->request->post);]]></add>
		</operation>
		<operation>
			<search><![CDATA[$this->model_catalog_category->editCategory($this->request->get['category_id'], $this->request->post);]]></search>
			<add position="after"><![CDATA[
			$this->load->model('catalog/dc_crawling');
			$this->model_catalog_dc_crawling->saveCategoryImportFromPost($this->request->get['category_id'], $this->request->post);]]></add>
		</operation>
		<operation>
			<search><![CDATA[$this->response->setOutput($this->load->view('catalog/category_list', $data));]]></search>
			<add position="before"><![CDATA[		if (is_file(DIR_APPLICATION . 'controller/catalog/dc_crawling_list.php')) {
			require_once(DIR_APPLICATION . 'controller/catalog/dc_crawling_list.php');
		}
]]></add>
		</operation>
		<operation>
			<search><![CDATA[$this->response->setOutput($this->load->view('catalog/category_form', $data));]]></search>
			<add position="before"><![CDATA[		if (is_file(DIR_APPLICATION . 'controller/catalog/dc_crawling_form.php')) {
			require_once(DIR_APPLICATION . 'controller/catalog/dc_crawling_form.php');
		}
]]></add>
		</operation>
	</file>

	<file path="admin/view/template/catalog/category_form.twig">
		<operation>
			<search><![CDATA[<li><a href="#tab-seo" data-toggle="tab">{{ tab_seo }}</a></li>]]></search>
			<add position="before"><![CDATA[          <li><a href="#tab-dc-crawling" data-toggle="tab">{{ tab_dc_crawling }}</a></li>
]]></add>
		</operation>
		<operation>
			<search><![CDATA[<div class="tab-pane" id="tab-seo">]]></search>
			<add position="before"><![CDATA[          <div class="tab-pane" id="tab-dc-crawling">{{ crawling_settings_form }}</div>

]]></add>
		</operation>
	</file>

	<file path="admin/view/template/catalog/category_list.twig">
		<operation>
			<search><![CDATA[<td class="text-right"><a href="/{{ category.edit }}" data-toggle="tooltip" title="{{ button_edit }}" class="btn btn-primary"><i class="fa fa-pencil"></i></a></td>]]></search>
			<add position="replace"><![CDATA[<td class="text-right"><a href="/{{ category.edit }}" data-toggle="tooltip" title="{{ button_edit }}" class="btn btn-primary"><i class="fa fa-pencil"></i></a> <a href="/{{ category.dc_import }}" data-toggle="tooltip" title="{{ button_dc_import }}" class="btn btn-info"><i class="fa fa-download"></i></a></td>]]></add>
		</operation>
		<operation>
			<search><![CDATA[<td class="text-right">{{ column_action }}</td>]]></search>
			<add position="before"><![CDATA[            <td class="text-left">{{ column_last_import }}</td>
]]></add>
		</operation>
		<operation>
			<search><![CDATA[<td class="text-right">{{ category.sort_order }}</td>]]></search>
			<add position="after"><![CDATA[            <td class="text-left">{% if category.date_import %}{{ category.date_import }}{% else %}—{% endif %}</td>
]]></add>
		</operation>
		<operation>
			<search><![CDATA[<td class="text-center" colspan="4">{{ text_no_results }}</td>]]></search>
			<add position="replace"><![CDATA[<td class="text-center" colspan="5">{{ text_no_results }}</td>]]></add>
		</operation>
	</file>
</modification>

 

How OCMOD actually works

If you’ve worked with OpenCart before, you know this—but if not, here’s the practical view:

• OCMOD is a runtime patch system
• it modifies core files without touching them physically
• it works by:
  • searching for specific code fragments
  • injecting / replacing / appending logic

👉 Think of it like a controlled “monkey patching” layer.

 

What this module modifies

In this project, I’m using OCMOD to hook directly into the category workflow, because that’s where the crawler config lives.

🔧 Controller injection (category.php)

I inject logic into:

• addCategory → to save crawler config on create
• editCategory → to update config
• before setOutput → to attach additional UI data

👉 This is where I plug in:

• saving JSON config
• loading crawler settings into $data

🎨 Twig template injection

I also extend the admin UI:

• category_form.twig → adds a new “Import crawling” tab
• category_list.twig → adds:
  • import button
  • “last import date” column

👉 This is key: I don’t build a separate module UI—I extend existing workflows.

 

⚠️ Common installation problem (very important)

Sometimes OpenCart will block file modifications during install.

You’ll see errors like:

• permission denied
• file path not allowed
• modification skipped

This happens because of restricted installer paths.

 

🔧 Fix (temporary workaround)

Edit this file:

admin/controller/marketplace/install.php

Find the array:

$allowed = array(

Then add these paths:

'admin/controller/catalog/',
'admin/controller/dc_crawling/',
'admin/model/catalog/',
'admin/model/dc_crawling/',
'admin/view/template/dc_crawling/',

 

🧠 Why this is needed

OpenCart installer blocks writing to certain directories for security.

Since this module:

• injects into catalog/category
• adds custom controllers/models/views

👉 we need to temporarily whitelist those paths.

 

⚠️ Important

This change is ONLY needed during installation.

After installing the module:

• you can safely revert the changes
• OCMOD modifications will still work

 

✅ After installation

Don’t forget:

• go to Extensions → Modifications
• click Refresh

Without this, OCMOD changes won’t apply.

If something doesn’t show up in admin after install,
👉 90% of the time it’s either:

• missing permissions
• or you forgot to refresh modifications

Been there too many times 😉

 

3.2 Database Installation

The OCMOD part only connects the module to OpenCart.
Now I need to prepare the database layer — this is where crawler configuration, sandbox records, and import metadata will live.

-- Design Cart Crawling — uruchom w phpMyAdmin lub konsoli MySQL (dopasuj prefix jeśli inny niż oc_)

ALTER TABLE `oc_category`
  ADD COLUMN `import` MEDIUMTEXT NULL AFTER `image`,
  ADD COLUMN `date_import` DATETIME NULL AFTER `import`;

CREATE TABLE IF NOT EXISTS `oc_dc_crawling_sandbox` (
  `sandbox_id` int(11) NOT NULL AUTO_INCREMENT,
  `category_id` int(11) NOT NULL,
  `source_url` varchar(1024) NOT NULL,
  `payload` MEDIUMTEXT NOT NULL,
  `date_scanned` datetime NOT NULL,
  `imported` tinyint(1) NOT NULL DEFAULT 0,
  PRIMARY KEY (`sandbox_id`),
  KEY `category_id` (`category_id`),
  UNIQUE KEY `uniq_category_url` (`category_id`, `source_url`(333))
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

CREATE TABLE IF NOT EXISTS `oc_dc_crawling_settings` (
  `settings_id` int(11) NOT NULL AUTO_INCREMENT,
  `margin_percent` decimal(15,4) NOT NULL DEFAULT 0.0000,
  `defaults_json` MEDIUMTEXT NOT NULL,
  `date_modified` datetime NOT NULL,
  PRIMARY KEY (`settings_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_general_ci;

INSERT INTO `oc_dc_crawling_settings` (`margin_percent`, `defaults_json`, `date_modified`)
SELECT 0, '{}', NOW()
WHERE NOT EXISTS (SELECT 1 FROM `oc_dc_crawling_settings` LIMIT 1);

-- Dodatkowe pola produktu (hurtownia ↔ sklep), jak w crawlerze yakamoz:
-- `price` w oc_product = cena sprzedaży (po marży); `original_price` = cena z źródła.
ALTER TABLE `oc_product`
  ADD COLUMN `original_price` decimal(15,4) NOT NULL DEFAULT 0.0000 AFTER `price`,
  ADD COLUMN `url` varchar(1024) NOT NULL DEFAULT '' AFTER `image`,
  ADD COLUMN `code` varchar(128) NOT NULL DEFAULT '' AFTER `mpn`;

-- Opcjonalnie (częściowa baza): jeśli kolumny w `oc_category` i `oc_product` są już dodane,
-- pomiń oba ALTER i uruchom tylko bloki CREATE TABLE / INSERT powyżej — są idempotentne.

The most important thing here is the table prefix.

OpenCart installations usually use oc_, but many production shops use a custom prefix. So before running the SQL, I always check the real database prefix and replace oc_ if needed.

For example:

oc_category

may need to become:

ocstore_category

or whatever prefix the project uses.

ALTER vs CREATE

This SQL does two different things:

• ALTER TABLE extends existing OpenCart tables
• CREATE TABLE adds new module-specific tables

The ALTER TABLE statements add extra fields to:

• category — for crawler configuration and last import date
• product — for original price, source URL, and supplier code

The CREATE TABLE statements create the module’s own storage, especially the sandbox table.

That sandbox is important because I don’t want scanned products to go directly into the catalog. First I collect them, inspect them, and only then import selected records.

Idempotency

Where possible, the SQL is written to be safe to run more than once.

In practice, that means:

• tables should use IF NOT EXISTS
• default settings should avoid duplicate inserts
• existing columns may need to be skipped manually if they already exist

So if OpenCart says that a column already exists, I don’t panic — I just remove or skip that specific ALTER TABLE line and continue with the rest.

Source: module documentation describes the required SQL setup, prefix replacement, and skipping existing columns during installation.

 

3.3 Admin Permissions

After installing files and database tables, I still need to give OpenCart admin users access to the new routes.

In System → Users → User Groups, I edit the admin group and enable both Access Permission and Modify Permission for:

dc_crawling/import
dc_crawling/setting

Without this step, the module may be installed correctly, but clicking the import or settings screen can still end with a permission error.

The crawler also checks modify permission for catalog/category, because the import workflow is connected directly with category configuration.

 

4. Integration with OpenCart (OCMOD Deep Dive)

This is the part I care about the most when building OpenCart extensions.
If the integration is messy, the whole module becomes fragile after the first update.

My rule is simple:
👉 inject as little as possible, and do the rest in my own files.

 

4.1 How install.xml Works

Here are the key fragments responsible for integrating with category.php:

My approach to OCMOD (practical, not theoretical)

When I write OCMODs, I avoid:

• large code injections
• complex inline logic
• modifying multiple unrelated places

Instead, I usually:

• inject one require/include
• and move all logic into my own file

👉 This keeps things:

• readable
• debuggable
• safe during updates

 

Why category.php?

Because I’m attaching crawler configuration directly to categories.

So I hook into:

• category form (UI)
• category save (data persistence)
• category list (import actions)

 

🔑 Stable anchor point (very important)

When choosing where to inject, I always look for guaranteed lines—things that won’t change across OpenCart versions.

This is one of them:

$this->response->setOutput($this->load->view('catalog/category_form', $data));

👉 This line exists in basically every OpenCart 3.x installation.

So what I do is:

• search for this line
• inject before it

 

Why “before output” works so well

Because at that moment:

• $data is already fully prepared
• all core logic is done
• I can safely modify or extend $data

👉 My injected file runs as the last step before rendering.

That means I can:

• append my own data ($data['dc_import'])
• modify existing values
• inject UI blocks

Without interfering with core logic.

 

4.2 Hooks Explained

Here are the three key hook types I use:

🔹 before output

<operation>
			<search><![CDATA[$this->response->setOutput($this->load->view('catalog/category_list', $data));]]></search>
			<add position="before"><![CDATA[		if (is_file(DIR_APPLICATION . 'controller/catalog/dc_crawling_list.php')) {
			require_once(DIR_APPLICATION . 'controller/catalog/dc_crawling_list.php');
		}
]]></add>
		</operation>

<operation>
			<search><![CDATA[$this->response->setOutput($this->load->view('catalog/category_form', $data));]]></search>
			<add position="before"><![CDATA[		if (is_file(DIR_APPLICATION . 'controller/catalog/dc_crawling_form.php')) {
			require_once(DIR_APPLICATION . 'controller/catalog/dc_crawling_form.php');
		}
]]></add>
		</operation>

Used for:

• injecting UI data
• attaching templates
• modifying $data before rendering

👉 This is where I plug in:

• crawler config into category form
• additional columns in category list

🔹 after editCategory

<operation>
			<search><![CDATA[$this->model_catalog_category->editCategory($this->request->get['category_id'], $this->request->post);]]></search>
			<add position="after"><![CDATA[
			$this->load->model('catalog/dc_crawling');
			$this->model_catalog_dc_crawling->saveCategoryImportFromPost($this->request->get['category_id'], $this->request->post);]]></add>
		</operation>

Used for:

• saving crawler configuration after category edit

Flow:

1. OpenCart saves category
2. my hook runs
3. I store crawler JSON into DB

👉 This keeps my logic separate from core save process.

🔹 replace addCategory

<operation>
			<search><![CDATA[$this->model_catalog_category->addCategory($this->request->post);]]></search>
			<add position="replace"><![CDATA[$category_id = $this->model_catalog_category->addCategory($this->request->post);

			$this->load->model('catalog/dc_crawling');
			$this->model_catalog_dc_crawling->saveCategoryImportFromPost($category_id, $this->request->post);]]></add>
		</operation>

Used when I need to extend behavior during category creation.

Instead of rewriting everything, I:

• replace just enough to capture category_id
• immediately call my own method

👉 Minimal intrusion, maximum control.

 

🧠 Why this strategy works

Because:

• I rely on stable anchors (like setOutput)
• I inject only entry points, not full logic
• I keep all complexity in my own files

This gives me:

• compatibility across OpenCart 3.x versions
• easy debugging
• safe upgrades

If you’ve ever had an OCMOD break after a minor OpenCart update,
you already know why this approach matters 😉

 

5. Database Design

I treat the database layer as the backbone of the whole crawler.
The crawler can deal with messy HTML — the database cannot. It has to stay predictable and clean 🧱

 

5.1 Extending the category Table

ALTER TABLE `oc_category`
ADD COLUMN `import` MEDIUMTEXT NULL,
ADD COLUMN `date_import` DATETIME NULL;

Here I store crawler configuration directly inside the category.

• import → JSON config for crawling (URLs, selectors, margin, etc.)
• date_import → timestamp of the last successful import

👉 I use JSON on purpose — it gives me flexibility without constantly changing the schema.

 

5.2 Sandbox Table

CREATE TABLE IF NOT EXISTS `oc_dc_crawling_sandbox` (
 `sandbox_id` INT(11) NOT NULL AUTO_INCREMENT,
 `category_id` INT(11) NOT NULL,
 `source_url` VARCHAR(1024) NOT NULL,
 `payload` MEDIUMTEXT NOT NULL,
 `date_scanned` DATETIME NOT NULL,
 `imported` TINYINT(1) NOT NULL DEFAULT 0,
 PRIMARY KEY (`sandbox_id`),
 UNIQUE KEY `uniq_category_url` (`category_id`, `source_url`(333))
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

This is where all scanned products go first.

Why sandbox?

Because importing directly from crawler output is a bad idea.

Instead:

1. scan data
2. store in sandbox
3. review/select
4. import

Upsert logic

The key part is:

UNIQUE KEY `uniq_category_url` (`category_id`, `source_url`(333))

This allows me to safely use:

ON DUPLICATE KEY UPDATE

So:

• existing product → update
• new product → insert

👉 No duplicates, clean dataset.

 

5.3 Settings Table

CREATE TABLE IF NOT EXISTS `oc_dc_crawling_settings` (
 `settings_id` INT(11) NOT NULL AUTO_INCREMENT,
 `margin_percent` DECIMAL(10,2) NOT NULL DEFAULT 0,
 `defaults_json` MEDIUMTEXT NULL,
 PRIMARY KEY (`settings_id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;

This is where I store global configuration.

Typical use:

• global margin
• default product values (JSON)

Example logic:

final price = base price + (global margin + category margin)

👉 Clean separation: global vs per-category config.

 

5.4 Extending the product Table

ALTER TABLE `oc_product`
ADD COLUMN `original_price` DECIMAL(15,4) NULL,
ADD COLUMN `url` VARCHAR(1024) NULL,
ADD COLUMN `code` VARCHAR(255) NULL;

These fields are extremely useful in real-world projects.

• original_price → raw supplier price
• url → source product URL
• code → supplier SKU / identifier

👉 This gives me traceability.

When something breaks or needs updating, I can always go back to the source without guessing 🔎

 

🧠 Why this structure works

Because everything is separated properly:

• config → category.import
• raw data → sandbox
• global settings → settings table
• final products → product

That separation is what makes large-scale imports safe.

👉 First collect
👉 then verify
👉 then import

Skip that flow, and sooner or later you’ll corrupt your catalog 😅

 

6. Category-Based Crawl Configuration

The crawler is configured per category because every supplier behaves differently.
One category may come from one wholesale store, another category from a completely different HTML structure — different URLs, different selectors, different pricing logic 🕷️

That’s why I don’t hardcode selectors globally. I attach them directly to the OpenCart category.

6.1 JSON Schema

This is the structure I store in the category.import field:

{
 "urls": [
 "https://supplier-store.com/category/page"
 ],
 "url": "https://supplier-store.com/category/page",
 "pagination": ".pagination",
 "product_url": ".product-grid .product-thumb a",
 "product_name": "h1",
 "product_image": ".product-image img",
 "product_code": ".sku",
 "product_price": ".price",
 "product_description": ".description",
 "product_producer": ".manufacturer",
 "margin_percent": "20"
}

urls

This is the list of listing pages I want to scan.

I use an array because one OpenCart category can be built from multiple supplier listing URLs.

Example:

{
 "urls": [
 "https://supplier.com/tools",
 "https://supplier.com/power-tools",
 "https://supplier.com/accessories"
 ]
}

The legacy (my old solution) url field is still supported, but internally I normalize everything into urls.

Selectors

Selectors tell the crawler where to find data in supplier HTML.

The most important one is:

"product_url": ".product-grid .product-thumb a"

Without it, the crawler has no product pages to visit.

Then I map product fields:

"product_name": "h1",
"product_image": ".product-image img",
"product_code": ".sku",
"product_price": ".price",
"product_description": ".description",
"product_producer": ".manufacturer"

This is the part that makes the module flexible.
If the supplier changes HTML, I don’t rewrite PHP — I update selectors in admin.

Margin

"margin_percent": "20"

This is the category-level margin.

Later, during import, I combine it with the global margin:

final margin = global margin + category margin

That gives me simple but powerful pricing control per category 💰

6.2 Admin UI

This file prepares crawler data for the OpenCart category form:

<?php
$this->load->language('catalog/dc_crawling');

$data['tab_dc_crawling'] = $this->language->get('tab_dc_crawling');

foreach (array(
	'tx_head_category',
	'tx_head_product',
	'entry_import_url',
	'entry_import_pagination',
	'entry_import_product_url',
	'entry_import_product_name',
	'entry_import_product_image',
	'entry_import_product_code',
	'entry_import_product_price',
	'entry_import_product_description',
	'entry_import_product_producer',
	'entry_margin_percent',
	'tx_pagination',
	'tx_product_url',
	'tx_product_name',
	'tx_product_image',
	'tx_product_code',
	'tx_product_price',
	'tx_product_description',
	'tx_product_producer',
	'help_urls',
	'button_add_url',
) as $_dc_k) {
	$data[$_dc_k] = $this->language->get($_dc_k);
}

$this->load->model('catalog/dc_crawling');

if (isset($this->request->post['import']) && is_array($this->request->post['import'])) {
	$data['import'] = $this->request->post['import'];
} elseif (isset($category_info) && !empty($category_info)) {
	$_dc_raw = isset($category_info['import']) ? $category_info['import'] : '';
	$data['import'] = json_decode($_dc_raw, true);

	if (!is_array($data['import'])) {
		$data['import'] = array();
	}
} else {
	$data['import'] = array();
}

$data['import'] = array_merge($this->model_catalog_dc_crawling->getImportDefaults(), $data['import']);

if (!isset($data['import']['urls']) || !is_array($data['import']['urls'])) {
	$data['import']['urls'] = array();
}

$_dc_legacy = '';

if (isset($data['import']['url'])) {
	$_dc_legacy = trim((string)$data['import']['url']);
}

if ($_dc_legacy === '' && isset($data['import']['URL'])) {
	$_dc_legacy = trim((string)$data['import']['URL']);
}

if ($_dc_legacy !== '') {
	$_dc_seen = false;

	foreach ($data['import']['urls'] as $_dc_existing) {
		if (trim((string)$_dc_existing) === $_dc_legacy) {
			$_dc_seen = true;
			break;
		}
	}

	if (!$_dc_seen) {
		$data['import']['urls'][] = $_dc_legacy;
	}

	$data['import']['url'] = $_dc_legacy;
}

$_dc_filtered = array();

foreach ($data['import']['urls'] as $_dc_u) {
	$_dc_u = trim((string)$_dc_u);

	if ($_dc_u !== '') {
		$_dc_filtered[] = $_dc_u;
	}
}

if (!$_dc_filtered && $_dc_legacy !== '') {
	$_dc_filtered = array($_dc_legacy);
}

$data['import']['urls'] = $_dc_filtered ? array_values(array_unique($_dc_filtered)) : array('');

$data['dc_import'] = $this->model_catalog_dc_crawling->prepareImportForForm($data['import']);

$data['crawling_settings_form'] = $this->load->view('dc_crawling/category/settings_form', $data);

How data gets into $data

This file runs inside the category controller context, right before rendering the category form.

That means I already have access to:

• $data
• $category_info
• $this->request->post
• OpenCart language/model loaders

The logic is simple:

1. load language labels
2. load crawler model
3. check POST data first
4. fallback to saved category.import JSON
5. merge with defaults
6. normalize legacy url / URL into urls
7. prepare final data for Twig

The important line is:

$data['dc_import'] = $this->model_catalog_dc_crawling->prepareImportForForm($data['import']);

I intentionally pass the prepared config as dc_import.

How Twig receives dc_import

Twig receives the prepared config through:

$data['crawling_settings_form'] = $this->load->view('dc_crawling/category/settings_form', $data);

So inside settings_form.twig, I can use:

dc_import.urls
dc_import.product_url
dc_import.product_name
dc_import.margin_percent

I avoid using import directly in Twig because import can be awkward as a template keyword.

👉 Small detail, but these small details save debugging time later.

 

7. Crawler Engine

The crawler is the part that does the dirty work: it visits listing pages, finds product URLs, opens product pages, extracts data, and saves everything into the sandbox.

I don’t treat crawling as a “one-shot import”. That would be too risky.
For me, crawling is only the first stage of the pipeline 🕷️

 

7.1 Fetching Listing Pages

The scan starts from category configuration.

Each category can have multiple source URLs, so I iterate over urls:

foreach ($import['urls'] as $seed) {
$seed = trim($seed);

if ($seed === '') {
continue;
}

$this->collectProductsForSeed($seed, $import, $urls, $errors, $hints);
}

Each seed is one listing page from the supplier store.

Inside collectProductsForSeed(), I:

1. fetch listing HTML
2. parse it with simple_html_dom
3. detect pagination
4. collect product links from every listing page

This is the moment where the crawler turns supplier HTML into a list of product URLs.

 

7.2 Pagination

private function buildPaginationListingUrls($category_url, $domain, $root, $pagination_selector) {
$out = array();
$els = $root->find($pagination_selector);

if (!is_array($els)) {
$els = $els ? array($els) : array();
}

$last_page = 1;
$found_numeric = false;

foreach ($els as $e) {
$n = (int)trim($e->plaintext);

if ($n > 0) {
$found_numeric = true;

if ($n > $last_page) {
$last_page = $n;
}
}
}

if ($found_numeric) {
for ($i = 1; $i <= $last_page; $i++) {
$out[] = rtrim($category_url, '/') . '/' . $i;
}

return array_values(array_unique($out));
}

foreach ($els as $e) {
$anchors = $e->find('a');

if (!is_array($anchors)) {
$anchors = $anchors ? array($anchors) : array();
}

if (!$anchors && $e->node instanceof DOMElement && strtolower($e->node->nodeName) === 'a') {
$anchors = array($e);
}

foreach ($anchors as $a) {
$href = isset($a->href) ? $a->href : '';

if ($href === '' || $href === '#') {
continue;
}

$abs = $this->mergeHrefWithDomain($domain, $category_url, $href);

if ($abs !== '') {
$out[] = $abs;
}
}
}

return array_values(array_unique($out));
}

I support two pagination modes.

Numeric mode

If the pagination selector returns plain numbers like:

1 2 3 4 5

I treat the highest number as the last page and generate URLs like:

/category/1
/category/2
/category/3

This is useful for simple supplier stores where pagination follows a predictable URL pattern.

Link mode

If pagination contains normal <a href="/"> links, I extract those links instead.

This is more flexible because many stores use URLs like:

?page=2
/page/2
?start=40

The crawler then normalizes relative URLs into absolute URLs using:

$abs = $this->mergeHrefWithDomain($domain, $category_url, $href);

👉 This way I don’t care whether the supplier gives me absolute or relative links.

 

7.3 HTML Parsing

private function extractProductPayload($doc, array $import, $product_page_url) {
$name = '';

if (trim($import['product_name']) !== '') {
$h = $doc->find(trim($import['product_name']), 0);

if ($h) {
$name = $this->plaintextAfterStrippingInnerDivs($h);
$name = html_entity_decode(trim($name), ENT_QUOTES, 'UTF-8');
}
}

if (stripos($name, 'bezpłatna dostawa') !== false) {
$name = trim(substr($name, 0, stripos($name, 'bezpłatna dostawa')));
}

$price_raw = '';

if (trim($import['product_price']) !== '') {
$pe = $doc->find(trim($import['product_price']), 0);
$price_raw = $pe ? $pe->plaintext : '';
$price_raw = str_replace(array(',', ' ', "\xc2\xa0"), array('.', '', ''), $price_raw);
$price_raw = trim($price_raw);
}

$image = '';

if (trim($import['product_image']) !== '') {
$image = $this->extractImage($doc, trim($import['product_image']), $product_page_url);
}

$description = '';

if (trim($import['product_description']) !== '') {
$de = $doc->find(trim($import['product_description']), 0);
$description = $de ? trim($de->innertext) : '';
}

$producer = '';

if (trim($import['product_producer']) !== '') {
$pr = $doc->find(trim($import['product_producer']), 0);
$producer = $pr ? trim($pr->plaintext) : '';
}

$code = '';

if (trim($import['product_code']) !== '') {
$ce = $doc->find(trim($import['product_code']), 0);
$code = $ce ? trim($ce->plaintext) : '';
}

return array(
'name' => $name,
'image' => $image,
'code' => $code,
'price_raw' => $price_raw,
'description' => $description,
'manufacturer' => $producer,
);
}

For parsing, I use simple_html_dom.

The idea is straightforward:

• admin provides CSS-like selectors
• crawler opens product HTML
• crawler extracts only fields that have selectors configured

So if I configure:

{
 "product_name": "h1",
 "product_price": ".price",
 "product_description": ".description"
}

then the crawler knows exactly where to look.

The payload is intentionally simple:

return array(
'name' => $name,
'image' => $image,
'code' => $code,
'price_raw' => $price_raw,
'description' => $description,
'manufacturer' => $producer,
);

👉 At this point I don’t create products yet. I only collect clean, structured data.

 

7.4 Saving to Sandbox

public function upsertSandboxProduct($category_id, $source_url, array $payload) {
$this->db->query("INSERT INTO " . DB_PREFIX . "dc_crawling_sandbox SET category_id = '" . (int)$category_id . "', source_url = '" . $this->db->escape(utf8_substr($source_url, 0, 1024)) . "', payload = '" . $this->db->escape(json_encode($payload, JSON_UNESCAPED_UNICODE | JSON_UNESCAPED_SLASHES)) . "', date_scanned = NOW(), imported = 0 ON DUPLICATE KEY UPDATE payload = VALUES(payload), date_scanned = NOW()");
}

At the beginning, I considered saving scan results into flat files — for example inside /tmp.

It could work. It might even be fast.

But then I asked myself:

What if I need pagination, filtering, sorting, preview, import status, or reusing scan results later?

With flat files, I’d probably end up fighting filenames and directory structures.
Not worth it 😅

So I moved the sandbox into the database.

 

Why database sandbox makes more sense

My project assumes up to around 4000 products in sandbox at once.
That is not a scary number for MySQL.

And the database gives me:

• easy sorting
• easy pagination
• import flags
• duplicate prevention
• predictable querying
• better future extensibility

For me, sandbox is basically a working cache.

I scan the supplier store once, save results, and then work with local data.

That matters because crawling is expensive:

• it loads the supplier server
• it loads my server
• it takes time
• it can hit timeouts or rate limits

So instead of scanning the same store again and again, I scan once and reuse sandbox data.

👉 This also gives me manual control: I can select only the products I actually want to import.

That’s the whole point of this architecture:

scan once → store locally → review → import selected products

 

8. Product Importer

The importer is the second stage of the pipeline.
At this point I already have scanned products in the sandbox, so now I can safely decide what should become a real OpenCart product 📦

 

8.1 Import Logic

public function importSandboxProducts(array $sandbox_ids, $category_id) {
$this->load->model('catalog/dc_crawling');
$this->load->model('catalog/product');
$this->load->model('localisation/language');

$settings_row = $this->model_catalog_dc_crawling->getSettingsRow();
$defaults = $settings_row['defaults'];

if (!is_array($defaults)) {
$defaults = array();
}

$pack = $this->model_catalog_dc_crawling->getCategoryImportDecoded((int)$category_id);
$import_margin = isset($pack['import']['margin_percent']) ? (float)$pack['import']['margin_percent'] : 0;
$margin_total = (float)$settings_row['margin_percent'] + $import_margin;

$languages = $this->model_localisation_language->getLanguages();

$imported = 0;
$marked = array();

foreach ($sandbox_ids as $sandbox_id) {
$row = $this->model_catalog_dc_crawling->getSandbox((int)$sandbox_id);

if (!$row || (int)$row['category_id'] !== (int)$category_id || !empty($row['imported'])) {
continue;
}

$p = isset($row['payload_decoded']) && is_array($row['payload_decoded']) ? $row['payload_decoded'] : array();

$name = isset($p['name']) ? trim($p['name']) : '';

if ($name === '') {
continue;
}

$original_price = $this->parsePrice(isset($p['price_raw']) ? $p['price_raw'] : '0');
$price = $original_price * (1 + $margin_total / 100);

$source_url = isset($row['source_url']) ? trim($row['source_url']) : '';

$scraped_code = isset($p['code']) ? trim((string)$p['code']) : '';
$warehouse_code = $scraped_code !== '' ? $scraped_code : substr(md5($source_url), 0, 100);

$model = $scraped_code !== '' ? $scraped_code : ('dc-' . (int)$sandbox_id);

$manufacturer_id = $this->resolveManufacturerId(isset($p['manufacturer']) ? $p['manufacturer'] : '', $defaults);

$product_description = array();

foreach ($languages as $language) {
$lang_id = (int)$language['language_id'];

$product_description[$lang_id] = array(
'name' => $name,
'description' => isset($p['description']) ? $p['description'] : '',
'meta_title' => $name,
'meta_description' => '',
'meta_keyword' => '',
'tag' => '',
);
}

$image_path = '';

if (!empty($p['image'])) {
$image_path = $this->downloadImage($p['image']);
}

$data = array_merge($this->baseProductDefaults($defaults), array(
'model' => $model,
'sku' => $model,
'price' => round($price, 4),
'manufacturer_id' => (int)$manufacturer_id,
'product_description' => $product_description,
'product_store' => isset($defaults['product_store']) && is_array($defaults['product_store']) ? $defaults['product_store'] : array(0),
'product_category' => array((int)$category_id),
'image' => $image_path,
));

$product_id = (int)$this->model_catalog_product->addProduct($data);

if ($product_id) {
$this->db->query("UPDATE " . DB_PREFIX . "product SET original_price = '" . (float)$original_price . "', `url` = '" . $this->db->escape(utf8_substr($source_url, 0, 1020)) . "', `code` = '" . $this->db->escape(utf8_substr($warehouse_code, 0, 128)) . "' WHERE product_id = '" . $product_id . "'");
$marked[] = (int)$sandbox_id;
$imported++;
}
}

if ($marked) {
$this->model_catalog_dc_crawling->markSandboxImported($marked);
}

if ($imported > 0) {
$this->model_catalog_dc_crawling->touchCategoryDateImport((int)$category_id);
}

return $imported;
}

The validation is intentionally defensive.

Before I create a product, I check:

• sandbox record exists
• record belongs to the current category
• record was not imported before
• product name is not empty

if (!$row || (int)$row['category_id'] !== (int)$category_id || !empty($row['imported'])) {
continue;
}

This protects me from wrong IDs, repeated imports, and broken crawler output.

The actual OpenCart product is created here:

$product_id = (int)$this->model_catalog_product->addProduct($data);

After that, I update my custom tracking fields:

$this->db->query("UPDATE " . DB_PREFIX . "product SET original_price = '" . (float)$original_price . "', `url` = '" . $this->db->escape(utf8_substr($source_url, 0, 1020)) . "', `code` = '" . $this->db->escape(utf8_substr($warehouse_code, 0, 128)) . "' WHERE product_id = '" . $product_id . "'");

👉 This gives me traceability: I always know where the imported product came from.

 

8.2 Price Calculation

The pricing logic is simple on purpose:

price = original_price * (1 + margin / 100)

In code:

$original_price = $this->parsePrice(isset($p['price_raw']) ? $p['price_raw'] : '0');
$price = $original_price * (1 + $margin_total / 100);

margin_total is built from:

$margin_total = (float)$settings_row['margin_percent'] + $import_margin;

So I can combine:

• global margin
• category-level margin

This is practical because I often need different pricing logic per product group 💰

 

8.3 Data Mapping

Manufacturer

private function resolveManufacturerId($name, array $defaults) {
$name = trim($name);

if ($name !== '') {
$q = $this->db->query("SELECT manufacturer_id FROM " . DB_PREFIX . "manufacturer WHERE name = '" . $this->db->escape(utf8_substr($name, 0, 64)) . "' LIMIT 1");

if ($q->num_rows) {
return (int)$q->row['manufacturer_id'];
}
}

if (isset($defaults['manufacturer_id'])) {
return (int)$defaults['manufacturer_id'];
}

return 0;
}

I first try to match the scraped manufacturer name with an existing OpenCart manufacturer.
If there is no match, I fall back to the default manufacturer from settings.

SKU / Model

$scraped_code = isset($p['code']) ? trim((string)$p['code']) : '';
$warehouse_code = $scraped_code !== '' ? $scraped_code : substr(md5($source_url), 0, 100);

$model = $scraped_code !== '' ? $scraped_code : ('dc-' . (int)$sandbox_id);

If the supplier gives me a product code, I use it.
If not, I generate a fallback from the source URL or sandbox ID.

👉 That way every product still gets a usable model / sku.

Image

private function downloadImage($url) {
$this->load->model('tool/image');

$bin = @file_get_contents($url, false, stream_context_create(array(
'http' => array('timeout' => 20, 'user_agent' => 'Mozilla/5.0 (compatible; DesignCartCrawling/1.0)'),
)));

if ($bin === false || $bin === '') {
return '';
}

$path = parse_url($url, PHP_URL_PATH);
$ext = strtolower(pathinfo($path, PATHINFO_EXTENSION));

if (!in_array($ext, array('jpg', 'jpeg', 'png', 'gif', 'webp'), true)) {
$ext = 'jpg';
}

$dir = 'catalog/dc_crawling/';

if (!is_dir(DIR_IMAGE . $dir)) {
@mkdir(DIR_IMAGE . $dir, 0755, true);
}

$file = $dir . 'i_' . bin2hex(random_bytes(8)) . '.' . $ext;
$full = DIR_IMAGE . $file;

if (@file_put_contents($full, $bin) === false) {
return '';
}

return $file;
}

Images are downloaded into:

image/catalog/dc_crawling/

I don’t reuse the supplier URL directly as the product image, because that would make my client’s shop dependent on the supplier’s server.

Local image copy is safer, faster, and easier to maintain ⚡

 

9. UI and Workflow

The UI is intentionally simple.
I don’t want the client—or myself—to jump between five different places just to run an import. Everything starts from the OpenCart category list 🖥️

 

9.1 Category List

This file extends category rows with:

• last import date
• direct import URL
• import button support

<?php
$this->load->language('catalog/dc_crawling');

$data['column_last_import'] = $this->language->get('column_last_import');
$data['button_dc_import'] = $this->language->get('button_dc_import');

$_dc_url = '';

if (isset($this->request->get['sort'])) {
$_dc_url .= '&sort=' . $this->request->get['sort'];
}

if (isset($this->request->get['order'])) {
$_dc_url .= '&order=' . $this->request->get['order'];
}

if (isset($this->request->get['page'])) {
$_dc_url .= '&page=' . $this->request->get['page'];
}

$this->load->model('catalog/dc_crawling');

foreach ($data['categories'] as &$_dc_cat) {
$_dc_pack = $this->model_catalog_dc_crawling->getCategoryImportDecoded((int)$_dc_cat['category_id']);

$_dc_cat['date_import'] = $_dc_pack['date_import'] ? date('Y-m-d H:i', strtotime($_dc_pack['date_import'])) : '';
$_dc_cat['dc_import'] = $this->url->link('dc_crawling/import', 'user_token=' . $this->session->data['user_token'] . '&category_id=' . (int)$_dc_cat['category_id'] . $_dc_url, true);
}

unset($_dc_cat);

What I like about this approach is that the category list becomes the operational dashboard.

From there, I can immediately see:

• which category was imported recently
• which category needs an update
• where to start the scan/import flow

👉 No separate menu needed. The workflow stays where the category work already happens.

 

9.2 Import Screen

This controller handles both actions:

• scan
• import

<?php
/**
 * Import with sandbox scan — UI + POST.
 * Crawling logic is handled by the dc_crawling/crawler model.
 */
class ControllerDcCrawlingImport extends Controller {
public function index() {
$this->load->language('dc_crawling/import');

if (!$this->user->hasPermission('modify', 'catalog/category')) {
$this->response->redirect($this->url->link('error/permission', 'user_token=' . $this->session->data['user_token'], true));
}

$this->document->setTitle($this->language->get('heading_title'));

if (($this->request->server['REQUEST_METHOD'] == 'POST') && isset($this->request->post['action'])) {
if ($this->request->post['action'] === 'scan') {
$this->runScan();
} elseif ($this->request->post['action'] === 'import') {
$this->runImport();
}
}

$this->getPage();
}

private function runScan() {
$category_id = isset($this->request->get['category_id']) ? (int)$this->request->get['category_id'] : 0;

if (!$category_id) {
return;
}

$this->load->model('catalog/dc_crawling');
$this->load->model('dc_crawling/crawler');

$pack = $this->model_catalog_dc_crawling->getCategoryImportDecoded($category_id);

try {
$result = $this->model_dc_crawling_crawler->scanCategory($category_id, $pack['import']);

$saved = isset($result['saved']) ? (int)$result['saved'] : 0;
$links = isset($result['product_urls']) ? (int)$result['product_urls'] : 0;
$fetches = isset($result['listing_fetches']) ? (int)$result['listing_fetches'] : 0;

$this->session->data['success'] = sprintf($this->language->get('text_scan_done_detail'), $saved, $links, $fetches);

$this->session->data['dc_crawl_scan_report'] = array(
'saved' => $saved,
'product_urls' => $links,
'listing_fetches' => $fetches,
'errors' => isset($result['errors']) && is_array($result['errors']) ? $result['errors'] : array(),
'hints' => isset($result['hints']) && is_array($result['hints']) ? $result['hints'] : array(),
);
} catch (Exception $e) {
$this->session->data['error'] = $this->language->get('error_scan') . ' ' . $e->getMessage();
}

$this->response->redirect($this->url->link('dc_crawling/import', 'user_token=' . $this->session->data['user_token'] . '&category_id=' . $category_id, true));
}

private function runImport() {
$category_id = isset($this->request->get['category_id']) ? (int)$this->request->get['category_id'] : 0;

if (!$category_id || empty($this->request->post['selected']) || !is_array($this->request->post['selected'])) {
$this->response->redirect($this->url->link('dc_crawling/import', 'user_token=' . $this->session->data['user_token'] . '&category_id=' . $category_id, true));

return;
}

$this->load->model('dc_crawling/importer');

$ids = array_map('intval', $this->request->post['selected']);

$n = $this->model_dc_crawling_importer->importSandboxProducts($ids, $category_id);

$this->session->data['success'] = sprintf($this->language->get('text_import_done'), (int)$n);

$this->response->redirect($this->url->link('dc_crawling/import', 'user_token=' . $this->session->data['user_token'] . '&category_id=' . $category_id, true));
}

private function getPage() {
$category_id = isset($this->request->get['category_id']) ? (int)$this->request->get['category_id'] : 0;

if (!$category_id) {
$this->response->redirect($this->url->link('catalog/category', 'user_token=' . $this->session->data['user_token'], true));

return;
}

$data['user_token'] = $this->session->data['user_token'];
$data['category_id'] = $category_id;

$data['action_scan'] = $this->url->link('dc_crawling/import', 'user_token=' . $this->session->data['user_token'] . '&category_id=' . $category_id, true);
$data['action_import'] = $data['action_scan'];

$this->load->model('catalog/dc_crawling');

$cfg = $this->model_catalog_dc_crawling->getCategoryImportDecoded($category_id);

$preview_urls = isset($cfg['import']['urls']) && is_array($cfg['import']['urls']) ? $cfg['import']['urls'] : array();

$data['import_preview_urls'] = array_values(array_filter($preview_urls, function ($u) {
return is_string($u) && trim($u) !== '';
}));

$data['import_preview_selectors'] = array(
'pagination' => isset($cfg['import']['pagination']) ? $cfg['import']['pagination'] : '',
'product_url' => isset($cfg['import']['product_url']) ? $cfg['import']['product_url'] : '',
'product_name' => isset($cfg['import']['product_name']) ? $cfg['import']['product_name'] : '',
);

$rows = $this->model_catalog_dc_crawling->getSandboxByCategory($category_id);

$data['rows'] = array();

foreach ($rows as $row) {
$p = isset($row['payload_decoded']) && is_array($row['payload_decoded']) ? $row['payload_decoded'] : array();

$data['rows'][] = array(
'sandbox_id' => (int)$row['sandbox_id'],
'source_url' => $row['source_url'],
'name' => isset($p['name']) ? $p['name'] : '',
'price_raw' => isset($p['price_raw']) ? $p['price_raw'] : '',
'date' => $row['date_scanned'],
'imported' => !empty($row['imported']),
);
}

$data['header'] = $this->load->controller('common/header');
$data['column_left'] = $this->load->controller('common/column_left');
$data['footer'] = $this->load->controller('common/footer');

$this->response->setOutput($this->load->view('dc_crawling/import', $data));
}
}

I usually keep this controller thin.

It doesn’t parse HTML.
It doesn’t create products directly.
It only routes actions to the right model.

That makes debugging much easier:

• crawler bugs → crawler.php
• product creation bugs → importer.php
• UI / action bugs → import.php

 

9.3 Workflow

The whole workflow is simple and safe:

1. Configuration

I configure the category:

• source URLs
• pagination selector
• product URL selector
• product data selectors
• category margin

This is done once per category.

2. Scan

Then I click scan.

The system:

• fetches listing pages
• detects pagination
• collects product URLs
• opens product pages
• extracts product payload

Nothing is imported yet.

3. Sandbox

All scanned products land in the sandbox table.

This gives me a working preview:

• name
• price
• source URL
• scan date
• imported status

👉 This is where I can catch selector mistakes before touching the real catalog.

4. Import

Finally, I select products and import only what I need.

The importer:

• validates selected sandbox rows
• calculates prices
• downloads images
• creates OpenCart products
• marks sandbox rows as imported
• updates last import date

The final flow looks like this:

configure category → scan supplier store → review sandbox → import selected products

And that’s the main reason this module works well in real projects:
I keep the speed of automation, but I don’t give up control ✅

 

10. Limits and Safety

Crawling without limits is dangerous.
One bad selector or unexpected pagination can easily trigger thousands of requests.

That’s why I enforce hard limits directly in the crawler logic 🚧

$max_listing_pages = 250;
$max_product_pages = 800;

$listing_pages_count = 0;
$product_pages_count = 0;

// iterate listing pages
foreach ($listing_urls as $listing_url) {
if ($listing_pages_count >= $max_listing_pages) {
$hints[] = 'Listing page limit reached (' . $max_listing_pages . ')';
break;
}

$listing_pages_count++;

// ... fetch listing HTML ...

foreach ($product_urls as $product_url) {
if ($product_pages_count >= $max_product_pages) {
$hints[] = 'Product page limit reached (' . $max_product_pages . ')';
break 2;
}

$product_pages_count++;

// ... fetch product HTML ...
}
}

Why this matters

• break → stops scanning more listing pages
• break 2 → stops the entire crawling process (listing + products)

👉 This protects against:

• infinite pagination loops
• broken selectors returning garbage
• accidental full-site crawling

And most importantly:

• protects your server
• protects supplier server
• protects your time 😅

 

 

11. System Limitations

This crawler solves a very specific problem.
And like every real-world tool, it has trade-offs.

 

❌ No SPA support

This works only with server-rendered HTML.

It will NOT work with:

• React / Vue apps
• JS-rendered product data
• APIs hidden behind frontend

👉 If the data isn’t in raw HTML → crawler won’t see it.

 

❌ No automatic deduplication

Each sandbox record creates a new product.

There is no automatic merge based on:

• SKU
• EAN
• product name

👉 This is intentional.

Auto-deduplication in real projects usually creates more problems than it solves.

 

❌ Dependency on HTML structure

This is the trade-off for flexibility.

The crawler depends on:

• CSS selectors
• DOM structure

If supplier changes layout:

• selectors break
• scan returns empty data

👉 Fix = update selectors, not PHP.

 

🧠 Why I accept these limitations

Because this tool exists for one reason:

importing products when there is no API, no XML, no structured data

And in that scenario, this approach is still the most efficient one 🚀

 

12. Extensions and Future Development

This module is not “finished” — it’s a foundation I keep extending in real projects.

The next step I’m actively working on is price synchronization.

Right now, the crawler imports products once.
But in real-world stores, prices change constantly. So the natural evolution is:

• re-scan existing products
• match them by source URL or code
• update price without creating duplicates

👉 This turns the crawler into a lightweight integration layer, not just a one-time importer.

 

Real-time product translation 🌍

The second direction is something I’ve already tested in production:

• automatic translation of product data
• support for multiple languages during import
• generating descriptions per language on the fly

Instead of:

• importing one language
• then manually translating

I can:

• import once
• generate multi-language content immediately

👉 This is especially powerful for stores targeting multiple markets.

 

What’s next

I’ll cover both topics in detail in a separate tutorial on my blog:

• price updates (re-import logic)
• multilingual product generation

Because those two features completely change how scalable this system becomes.

 

Download & Github

Download | Github

 

Summary

What I built

I built a practical product import crawler for OpenCart that:

• works without API, XML, or database access
• uses HTML as a data source
• separates scan → sandbox → import
• gives full control over what enters the catalog

This is not just a script.
It’s a structured system designed for real client projects.

 

Where to use it

This solution works best when:

• suppliers provide only HTML
• there is no API or export feed
• you need to import hundreds or thousands of products
• you want control over data before publishing

👉 In other words: messy real-world integrations.

 

When NOT to use it

I would NOT use this approach if:

• the supplier provides a stable API
• XML/CSV feeds are available
• the data structure is already clean and standardized

In those cases:

👉 use native integrations — they’re faster and more reliable.

 

Final thought

This approach exists for one reason:

when there is no clean way to integrate — I create one.

And in that space, crawling is not a workaround.
It’s often the most effective solution 🚀