=== DataStream API Sync ===
Contributors: utahhans
Tags: api, json, sync, import, rest api
Requires at least: 6.0
Tested up to: 7.0
Requires PHP: 7.4
Stable tag: 2.1.2
License: GPLv3
License URI: https://www.gnu.org/licenses/gpl-3.0.html

Pull any API or feed — JSON, XML, RSS, CSV — into WordPress on a schedule. Show it anywhere, or turn it into posts. No code required.

== Description ==

DataStream mirrors external JSON APIs into your WordPress database and keeps them fresh on a schedule — then lets you put that data on any page, or turn it into real posts.

**Setup is one URL.** Paste your API's address into the Connect Wizard and DataStream inspects the real response: it finds the records (wherever they hide), picks the ID field by testing which one is actually unique, detects each field's type (dates, images, prices, links), spots pagination, and — if the API wants a key — figures out where the key goes by trying the standard places. You see the evidence for every decision, preview your real data, and finish with a draft page already showing it.

Unlike page-load fetchers, DataStream syncs to your own database in the background: pages render instantly from local data, your site keeps working when the API is down, and the API sees one polite scheduled request instead of one per visitor.

**Connect (per stream)**

* Any JSON API endpoint — GET or POST.
* Auth: bearer token, HTTP Basic, API key (header or query parameter), or any custom headers.
* Keep credentials out of the database with wp-config constants (`DATASTREAM_TOKEN_{STREAM}`).
* Pagination: page-number or offset, with automatic stop on empty or repeating pages.
* Built for fickle APIs: automatic retry on transient failures, HTTP 429 rate limits respected via Retry-After, UTF-8 BOM tolerated, oversized responses capped, exponential backoff per stream after repeated failures — one dead API never blocks the others.
* A "Preview & Detect Fields" button shows exactly what the API returns and which fields you can display, before anything syncs.

**Map**

* Auto-detects common response shapes (`data`, `items`, `results`, `records`, `entries`) or takes an explicit dot path like `response.dataItems` — case-sensitive, as your API intended.
* Configurable ID field per stream; payloads stored faithfully (keys keep their exact case).
* Works with single-object APIs (weather-style endpoints) too.

**Display**

* `[datastream_list stream="products" template="table" fields="name,price"]` — four bundled templates: list, table, cards, raw JSON. Dot paths reach nested values (`company.name`).
* `[datastream_field id="123" field="title"]` for single values.
* A server-rendered **DataStream List** block with live preview.
* Template tags: `datastream_get_records()`, `datastream_get_record()`, `datastream_get_field()`.
* Every value escaped at output.

**Free vs Pro**

Free includes everything above for **one stream**, syncing as often as **every 15 minutes** — a complete product, not a demo.

Pro adds:

* **Unlimited streams** — sync many APIs side by side, each with its own auth, mapping, and backoff.
* **API to Posts** — records become real posts of any post type (created once, updated on every sync, never duplicated), with every top-level field stored as `datastream_*` post meta — ready for WP_Query, ACF, and page builders.
* **WP-CLI** — `wp datastream sync`, `status`, `streams`, `purge`.
* **5-minute sync interval** and deeper pagination (50 pages vs 5).

**Safe by default**

* Requests to private/local network addresses blocked unless you opt in (`DATASTREAM_ALLOW_PRIVATE_URLS`).
* Responses size-capped (5 MB default). Credentials never logged.
* Uninstalling keeps your data unless you explicitly opt in to deletion.
* All admin and REST surfaces require `manage_options`.

== Frequently Asked Questions ==

= Which APIs work? =

Any endpoint returning JSON over HTTP(S): wrapped or bare arrays, or a single object. Verified against the WordPress REST API (syncing another site's posts, paginated), GitHub, Open Library, Open-Meteo, and store-style product APIs.

= How is this different from plugins that fetch the API when the page loads? =

DataStream syncs on a schedule into your own database. Pages render from local data (fast, no API latency), your site keeps working during API outages, and rate limits stop being a page-load problem.

= How do I show the data on a page? =

Add the DataStream List block, or:

`[datastream_list stream="products" limit="10" template="table" fields="title,price,rating.rate"]`

Use Preview & Detect Fields on the stream's settings to see which fields exist — click a field to copy it.

= Can records become real posts? =

Yes (Pro): pick a post type per stream, choose which fields become the title and content, and every sync creates or updates the matching posts — never duplicates. Top-level fields land in `datastream_*` post meta, so `WP_Query`, ACF, and page builders can use them directly.

= Does it work with SwiftQueue? =

Yes — they're built to compose. DataStream schedules everything through standard WP-Cron, which is exactly the layer SwiftQueue manages. When SwiftQueue is active, DataStream registers its sync as low-priority background work so API fetches yield to checkout and other urgent tasks. Neither plugin requires the other.

= The API broke / rate limited me. What happens? =

One transient failure is retried once. An HTTP 429 pauses that stream for exactly as long as the API's Retry-After asks. Repeated failures back the stream off exponentially (15 minutes doubling to a 6-hour cap) — other streams keep syncing, and a manual Sync Now always works and resets the backoff. The health panel explains what's paused and why.

= Does it work when WP-Cron is disabled? =

Yes — if `DISABLE_WP_CRON` is set, make sure a system cron calls `wp-cron.php`; the health panel warns about this.

== External services ==

DataStream makes outbound requests in two distinct situations. One you
configure yourself; the other is the licensing service.

= The APIs and feeds you point it at =

This is the plugin's whole purpose: it fetches the URLs *you* enter, on the
schedule you choose, and stores the results in your database. It never
contacts an endpoint you did not configure, and it never contacts one at all
until you save a stream.

What is sent to those URLs is what you set up: the request itself, plus any
authentication you entered for that stream (bearer token, API key header, or
an OAuth 2.0 client credential). Nothing about your site, your visitors or
your posts is added to those requests. Requests to private and local network
addresses are blocked by default.

Because you choose those endpoints, their terms and privacy policies are
whatever the provider publishes — please check them for any API you connect.

= Freemius (licensing, payments and updates) =

This plugin bundles the Freemius SDK, the service that handles licensing,
payments and plugin updates for DataStream.

What is sent, and when:

* **On activation, only if you agree.** The first screen asks whether to share
  your site URL, your admin email address, and basic environment details (PHP
  and WordPress versions, active theme and plugins). Choosing "Skip" sends none
  of it, and the plugin works exactly the same either way.
* **When you activate a paid licence.** Your licence key and site URL are sent
  so the licence can be validated and counted against your allowance.
* **When checking for updates.** The site URL and the installed version are
  sent so the correct update can be offered.

Nothing is sent for visitors to your site, and no page content, post content,
customer data, or data synced from your APIs is ever transmitted to Freemius.

Service: Freemius, Inc. — https://freemius.com
Terms: https://freemius.com/terms/
Privacy policy: https://freemius.com/privacy/

== Screenshots ==

1. The Connect Wizard starts with one field — paste a URL, or pick something from the sample gallery to try it in seconds.
2. The "found it" moment: DataStream inspects the real response and shows its evidence — which array holds the records, which field is genuinely unique, where the API key goes.
3. Detected fields become clickable chips, with a live preview of your actual data before anything syncs.
4. The finished front-end: a grid of synced records with images, rendered from your own database.
5. The dashboard's per-stream health cards — last sync, records stored, and why a stream is paused if it is.
6. Real WooCommerce products created from a product API, with prices, stock and categories mapped.
7. The DataStream List block in the editor, rendering live in the preview.
8. The Advanced form for a stream: auth, mapping, pagination and destinations, grouped into sections.

== Changelog ==

= 2.1.2 =

* Plugin Check (all five categories, including plugin_repo) is now clean against the staged free copy: current phpcs sniff names, unslash/sanitize on admin error notices, and justified custom-table DB ignores.
* No functional changes to syncing or display.

= 2.1.1 =

* Compliance pass before submission: WordPress.org derives a plugin's expected slug from its display name, so the plugin name and readme title are now "DataStream API Sync", matching the text domain and folder exactly.
* Uninstall cleanup moved out of uninstall.php into a class so it runs whichever way the plugin is removed.
* readme: added the External services disclosure, screenshots and upgrade notice.
* No functional changes to syncing or display.

= 2.1.0 =

* New: display pagination — page_size="10" renders numbered page links (server-rendered, cache-friendly).
* New: sorting — orderby="price:number" order="asc" sorts before limiting, so "top 3 by price" means exactly that.
* New: searchable="1" adds a type-to-filter box above any list, table, or grid.
* New: style presets (clean, bordered, striped, compact) and a grid image-aspect option.
* New: dashboard health cards — one per stream with freshness, last-10 sync dots, plain-language failure notes, and one-click sync.
* New: first run opens the Connect Wizard automatically; the wizard gained a step indicator, animated detection evidence, keyboard/screen-reader flow, and error messages that explain what to do.
* Improved: the Advanced stream form is now sectioned and conditional — WooCommerce fields only appear when WooCommerce does, and category/custom-field mappings are simple dropdown rows (no JSON anywhere).
* Improved: secrets now have explicit Clear buttons.
* Compatibility: the full test suite passes on PHP 7.4 and the plugin is live-verified on WordPress 6.0 — the minimums we claim are minimums we test.
* Developer: translation template (POT) now ships in /languages.

= 2.0.0 =

* New: formats — XML, RSS/Atom, and CSV/TSV flow through the same pipeline as JSON. Paste any of them into the wizard; the format is sniffed from the response (XXE-safe XML with attributes preserved as @keys, curated feed mapping, delimiter-sniffing CSV with quoted fields).
* New (Pro): destination mapping — any field to any taxonomy (terms created as needed, "Clothing > Shirts" builds hierarchies) and any field to any meta/ACF key.
* New (Pro): WooCommerce product preset — price, SKU, and stock (with stock status) written the way WooCommerce expects; simple product type set automatically.
* New (Pro): image sideloading — record images download into the Media Library once (deduped by source URL), attach as featured images, and skip unchanged images on re-sync.
* New (Pro): stale-record actions — when records disappear from the API, optionally draft or trash their mapped posts, guarded by the auto-purge age so partial syncs never mass-remove content.
* New (Pro): OAuth 2.0 — client-credentials and refresh-token grants with automatic token caching and renewal.
* Fixed: a payload field literally named "content" no longer replaces the whole record (a 1.2.0 heuristic that silently discarded every other field — RSS feeds always have one).

= 1.5.0 =

* New: Connect Wizard — paste a URL, DataStream detects the records, ID field, field types, and pagination from the real response, shows its evidence, previews your data, and can create a ready-made draft page. A "Try a fun one" gallery and popular-setup recipes (another WordPress site, GitHub releases) get first-timers going instantly.
* New: auth probing — paste an API key and DataStream finds where it belongs (Bearer, X-Api-Key, or query parameter) by trying them.
* New: typed display formatting — fields="photo:image,price:number,published:date,site:url" renders real images, formatted numbers and dates, and safe links; new image-led "grid" template.
* New: fingerprint IDs — APIs whose records have no unique field now sync via content fingerprint instead of failing.
* New: block editor shows clickable detected-field chips (with types) for the chosen stream.
* New: activity log in Diagnostics & Tools; a single dismissible notice when a stream keeps failing; Site Health test for stream status.
* Improved: pagination detection reads metadata objects (info/meta) and Link headers.

= 1.4.0 =

* New: streams — each API is its own stream with independent connection, auth, mapping, pagination, and failure backoff. Free syncs 1 stream; Pro unlimited.
* New: API to Posts (Pro) — records become posts of any post type with queryable datastream_* meta, updated in place on every sync.
* New: WP-CLI (Pro) — wp datastream sync / status / streams / purge.
* New: pagination (page-number and offset) with automatic stop on empty or repeating pages.
* New: auth options — HTTP Basic, API key in header or query parameter; per-stream wp-config token constants.
* New: Preview & Detect Fields tool on every stream.
* New: fickle-API handling — one automatic retry on transient failures, Retry-After-aware 429 backoff, UTF-8 BOM stripping.
* New: best-effort batches — one bad record no longer discards the rest (all-or-nothing available via filter).
* New: 5-minute sync interval (Pro).
* Changed: Connection/Mapping settings moved into the Streams tab; existing single-API configurations migrate automatically.

= 1.3.0 =

* New: display layer — shortcodes, block, template tags with four templates.
* New: SSRF guard, response size cap, failure backoff, bearer token constant, opt-in uninstall.
* Fixed: payload keys stored with exact case; camelCase JSON paths resolve; large numeric IDs preserved.
* Fixed: batch transactions actually commit on InnoDB.

= 1.2.0 =

* Ingestion engine, admin dashboard, health diagnostics, REST API.

== Upgrade Notice ==

= 2.1.2 =
Plugin Check cleanup so the WordPress.org zip matches the review. No settings change, no re-sync, no action needed.

= 2.1.1 =
Housekeeping for the WordPress.org listing. No settings change, no re-sync, no action needed.

= 2.1.0 =
Adds display pagination, sorting and search, a rebuilt Connect Wizard, and per-stream health cards. Safe upgrade: no settings change and no re-sync required.
