This article brings all three together in one place. We walk through what each workflow does, why it matters, and what it produces. The Jupyter notebooks are provided alongside this article so you can download them, configure the parameters, and run them directly.

Here is what we built:

1. Survivorship-Bias-Free U.S. Equity Database. The entire U.S. stock market, including delisted companies, enriched with sector classifications, company names, and index membership flags.

2. Point-in-Time Index Membership. Reconstructing which stocks actually belonged to an index (S&P 500, Nasdaq 100, Russell 3000) on any given date in history, with entry/exit dates and re-entry tracking.

3. Complete Futures Database Pipeline. Contract specifications, individual contract histories, continuous adjusted and unadjusted series, and volume decomposition across all futures markets.

Each section below explains the problem, describes the approach, and shows what you get at the end.

Part 1: Building a Survivorship-Bias-Free U.S. Equity Database

Why This Matters

If you backtest a stock strategy using only companies that trade today, you are testing on the survivors. Companies that went bankrupt, got acquired, or were delisted for any reason are missing from your dataset. This is survivorship bias, and it consistently makes strategies look better than they actually were.

The fix is straightforward: include every company that ever traded, whether it is still active or not. Norgate Data maintains a complete archive of delisted U.S. equities alongside the active ones, and their Python API lets you pull both.

What the Notebook Does

The workflow is a 12-step Jupyter notebook that starts small and scales up.

Starting small. The notebook first downloads price data for a few well-known tickers (AAPL, TSLA, GOOG) so you can see how the Norgate API works. It uses total-return-adjusted prices, which account for dividends and stock splits, and returns a clean pandas DataFrame with OHLCV columns.

Enriching with metadata. Once you have the raw prices, the notebook adds three layers of information to each row:

• Company name using Norgate’s security name lookup. Adds a Security_Name column (e.g., “Apple Inc”).

• GICS sector using the Global Industry Classification Standard. Adds a Sector column (e.g., “Information Technology”).

• Index membership using Norgate’s index constituent time series. Adds In_S&P_500 and In_Russell_3000 columns as binary flags (1 = member on that date, 0 = not).

Scaling to the full market. The notebook then retrieves every symbol from both the US Equities and US Equities Delisted databases, filters to equities only (excluding ETFs, REITs, etc.), and runs the same enrichment pipeline across the entire universe.

Merge and export. Finally, active and delisted data are combined into a single DataFrame, sorted by date, and saved as a .parquet file.

What You Get

A single .parquet file containing every U.S. equity, active and delisted, with daily OHLCV prices, company names, GICS sectors, and point-in-time S&P 500 and Russell 3000 membership flags. Ready for backtesting.

Part 2: Point-in-Time Index Membership Tables

Why This Matters

Even with a survivorship-bias-free price database, there is another subtle bias that can distort your backtests: index membership bias.

If you want to backtest a strategy on “S&P 500 stocks,” which S&P 500 do you use? Today’s constituents? That is the wrong answer. The S&P 500 composition changes constantly. Companies are added when they grow large enough and removed when they shrink, get acquired, or go bankrupt. Using today’s list to represent the past means you are implicitly selecting stocks that performed well enough to remain in the index.

What you actually need is a point-in-time record: on January 15, 2018, which exact stocks were in the S&P 500? On March 3, 2020? The answer is different for every date.

What the Notebook Does

The core concept is membership spells: continuous periods during which a stock belongs to an index. A stock can have multiple spells if it was removed and later re-added.

Configuration. You define which indices to analyze (e.g., $NDX for Nasdaq 100, $SPX for S&P 500) and the date range. The notebook explains how to look up the correct Norgate symbol codes using their Data Viewer tool.

Symbol universe. The notebook pulls all active and delisted U.S. equity symbols, because restricting to currently active stocks would reintroduce survivorship bias.

Spell detection. For each symbol and index pair, the notebook retrieves the daily index constituent time series from Norgate (a binary 1/0 signal indicating membership on each date) and scans it sequentially. When a stock enters the index, a new spell begins. When it leaves, the spell ends on the last day the stock was still a member.

A few design decisions worth noting:

• Pre-existing members (stocks already in the index on the start date) get a blank entry date, since the algorithm cannot determine the original entry from within the window.

• Exit date is the last day the stock was still in the index, not the first day it dropped out.

• A membership counter tracks re-entries: a stock with membership number 2 entered the index, was removed, and entered again.

Full scan. The notebook runs this function across every symbol in the entire U.S. equity universe for each configured index. Scanning approximately 40,000 symbols per index takes about 4 to 5 minutes each. The results are combined into a single DataFrame and exported as CSV.

What You Get

A CSV file (e.g., ndx_spx_constituents_2015_2025.csv) containing one row per membership spell. Each row tells you exactly when a stock entered and exited a specific index. This file can be directly joined with your price data to filter your backtest to only the stocks that were actually in the index on each date.

Part 3: Complete Futures Database Pipeline

Why This Matters

Futures data is fundamentally different from equity data. You are not dealing with one continuous time series per instrument. You have dozens or hundreds of individual contracts per market, each with its own expiry date, and you need to stitch them together into continuous series while keeping track of which specific contract you are in at any given time.

Getting this wrong leads to phantom P&L from roll gaps, incorrect position sizing from using adjusted prices for dollar calculations, and missed roll signals from ignoring volume transitions between contracts.

This workflow builds four Parquet files that cover the full spectrum: from raw contract metadata to a fully enriched continuous series with volume decomposition.

What the Notebook Does

The notebook produces four Parquet files, each building on the previous one.

Layer 1: Contract Specifications. The pipeline starts by pulling metadata for every futures market: tick size, point value, margin requirements, currency, exchange, and classification group. This is the foundation for position sizing and risk calculations.

Layer 2: Individual Contracts. All individual futures contracts are then downloaded, including expired ones. Each contract is queried for its full OHLCV series with open interest. The download runs in parallel with 10 workers since there are roughly 27,000 individual contract symbols producing around 11 million rows.

Layer 3: Continuous Adjusted and Unadjusted Series. This is where it gets subtle. Norgate provides two types of continuous symbols for each market:

You need both. Adjusted prices for signal generation (no artificial jumps at roll dates) and unadjusted prices for position sizing (real dollar values). The notebook merges them into a single row so each date has both representations available.

Layer 4: Volume Decomposition. The final layer enriches the continuous series with volume from the first-to-expire and second-to-expire contracts, which (as volume shifts from the front-month to the next contract) is critical for understanding proper roll dynamics.

The pipeline parses individual contract symbols (e.g., ES-2025M) to determine expiry dates, builds a sorted map of contracts per commodity, and then for each date finds the two nearest active contracts and records their volumes.

What You Get

Four Parquet files in the norgate_output/ directory:

Download and Run

All three workflows are provided as Jupyter notebooks alongside this article. Download them, adjust the configuration in the first cell (date range, indices, output paths), and run.

Conclusion

Data quality is not glamorous, but it determines whether your research conclusions are real or artifacts. Survivorship bias, index membership bias, and incorrect futures roll handling are three of the most common ways backtests can lie to you.

These three workflows address each of those problems directly. They produce clean, structured, Parquet-based datasets that you can plug into any backtesting framework or research pipeline.

Download the notebooks, run them, adapt them, and build on them.

Subscribe now

Disclaimer
This publication is provided by Concretum Group for informational, educational, and research purposes only. It does not constitute investment, financial, legal, or tax advice, nor a recommendation to buy or sell any security, instrument, strategy, or investment product. All investments involve risk, including possible loss of principal. Past performance, backtested performance, and historical analysis are not reliable indicators of future results. Readers should conduct their own research and consult qualified professionals before making investment decisions.

Full disclaimer: https://concretumgroup.com/disclaimer/

The source of the divergence was not the strategy itself.

It was the data.

Across providers, we repeatedly observed:

• phantom highs and lows

• post-close bars on early-close sessions

• stale bars

• missing intraday intervals

• differing timestamp conventions

• inconsistent venue aggregation

This follow-up focuses on the practical side of the problem:

Can these anomalies be detected systematically, and can any of them be corrected safely?

In many cases, yes.

This article introduces concretum_tests, a lightweight Python CLI used internally at Concretum Group to audit and validate 1-minute OHLCV datasets.

Download concretum_tests

Subscribe now

The tool performs four core checks:

• Early-close leakage

• Phantom highs and lows

• Strong stale bars

• Missing intraday bars

Where corrections are defensible, the tool can also generate reconstructed datasets.

The Tool

concretum_tests is intentionally lightweight.

It scans intraday OHLCV CSVs, reports structural anomalies, and optionally applies corrections where those corrections are defensible.

No APIs.
No databases.
No external infrastructure.

Just CSVs and a terminal.

Installation

pip install polars rich exchange-calendars

Usage

python concretum_tests.py
python concretum_tests.py Data/Raw/TQQQ_Polygon_1min.csv
python concretum_tests.py --zb 3 --zw 3 --zv 4
python concretum_tests.py --no-fix
python concretum_tests.py --provider MyFeed --bar-label end

The tool expects standard 1-minute OHLCV CSVs:

datetime,open,high,low,close,volume
2016-04-04 09:30:00,105.65,105.70,105.32,105.48,27391
2016-04-04 09:31:00,105.44,105.46,105.33,105.44,5066

Different providers use different timestamp conventions. Most label bars at the start of the minute, while others such as IQFeed label bars at the end. The CLI automatically detects major providers from the filename and applies the appropriate session logic.

Early-Close Leakage

A small number of NYSE trading sessions close early each year, typically at 13:00 ET.

Black Friday, Christmas Eve, and the session before Independence Day are the most common examples.

Some providers truncate these sessions correctly. Others continue reporting bars until the normal 16:00 ET close.

Those additional bars are structurally invalid for strategies assuming standard session boundaries.

The example above shows the same half-day session across two providers. IBKR correctly terminates the session at the official close, while Alpaca continues emitting bars afterward.

Unlike many intraday anomalies, early-close leakage is fully deterministic.

The NYSE calendar is authoritative. A bar either occurs before the official close or it does not.

Because of that, the correction is clean:

• load the official exchange calendar

• identify early-close sessions

• remove all bars after the cutoff

No interpolation or reconstruction is required.

The tool uses the exchange_calendars package to retrieve the official NYSE schedule, including both recurring and one-off early closes.

Phantom Highs and Lows

Phantom spikes are isolated intraday extremes that appear in one provider but not others.

They are typically single-bar events with unusually large wick extensions and weak accompanying volume.

In the example above, the detected phantom lows occur during otherwise stable intraday price action and leave almost no footprint in volume.

These anomalies matter because they can directly affect:

• breakout systems

• stop-loss execution

• ATR calculations

• volatility estimates

• intraday signal generation

Even a single phantom wick can materially alter a backtest.

Why They Exist

U.S. equities trade across fragmented venues:

• exchanges

• ECNs

• dark pools

• off-exchange internalizers

Trades may also arrive late to the consolidated tape.

Different providers apply different filtering and aggregation rules to those prints. Depending on the feed and consolidation logic, isolated highs or lows may appear in one dataset while remaining absent in another.

This is why phantom spikes are not always simple “bad ticks.” In some cases they are legitimate prints that only certain providers preserve.

Detection Methodology

The detector uses three simultaneous conditions.

For phantom highs:

Condition 1: Body Baseline

The upper wick must exceed z_b times the largest candle body observed that day.

This identifies spikes that are geometrically inconsistent with normal intraday price movement.

Condition 2: Wick Baseline

The wick must also exceed z_w times the session’s median upper wick.

This measures anomaly size relative to the day’s normal wick behavior.

Condition 3: Volume Coherence

The associated volume must remain below z_v times the session’s median volume.

This is the key coherence filter.

Large price dislocations typically attract participation. If a bar produces an extreme wick while volume remains unusually weak, the move becomes statistically suspicious.

All three conditions must trigger simultaneously.

The first two conditions identify geometric outliers. The third filters out legitimate high-volatility bars that occur on elevated participation.

Default thresholds are:

z_b = 2
z_w = 2
z_v = 5

Threshold Trade-Offs

There is no universally correct threshold configuration.

Tighter thresholds reduce false positives but miss more anomalies. Looser thresholds capture more suspicious bars at the cost of flagging legitimate volatility.

The figure above illustrates the trade-off directly using the same trading session under two parameter configurations.

The CLI exposes all thresholds as configurable parameters:

--zb
--zw
--zv

In practice, the recommended workflow is:

1. start with defaults

2. inspect flagged bars manually

3. adjust thresholds based on the acceptable false-positive / false-negative balance

Phantom Correction Methodology

For strategies sensitive to intraday extremes, detected phantom spikes can optionally be reconstructed.

The correction intentionally avoids predictive modeling.

Instead, the anomalous wick is replaced with a statistically typical wick for that session.

The reconstruction rules are:

Phantom high:
new_high = max(open, close) + median_upper_wick

Phantom low:
new_low = min(open, close) - median_lower_wick

The reconstruction preserves:

• candle direction

• candle body

• session structure

while removing the isolated outlier.

Importantly, the corrected value is not exchange-reported market data.

It is a reconstructed estimate and should always remain identifiable within any research workflow.

Stale Bars and Missing Data

Stale bars and missing bars are two representations of the same underlying problem: insufficient trading information during a given interval.

A strong stale bar occurs when:

open = high = low = close

Some providers emit synthetic stale bars when no trade occurs. Others omit the interval entirely, creating gaps in the minute grid.

The examples above show identical market halts represented differently across providers:

• one emits flat zero-volume stale bars

• the other emits missing intervals

The underlying market state is identical.

Only the representation changes.

This distinction becomes especially important during volatility events and LULD halts.

Detecting Structural Gaps

The stale-bar check is straightforward:

• identify bars where all OHLC values are identical

Missing bars require interval analysis:

• compute elapsed time between consecutive rows

• flag any gap larger than one minute

The tool uses each session’s observed first and last timestamp as bounds, allowing early-close sessions and overnight gaps to be handled naturally.

Why Missing Data Is Never Auto-Filled

Unlike phantom spikes, where there is clearly too much anomalous information, stale and missing regions contain too little information.

Any fill becomes synthetic.

That introduces a serious asymmetry.

A fabricated fill can create:

• impossible executions

• synthetic volatility

• false stop-outs

• trades that never existed

A backtest that appears tradable but is structurally impossible is significantly more dangerous than a backtest that simply flags unreliable sessions.

For that reason, the CLI never fills missing bars automatically.

It reports them.

It does not invent them.

Example Output

The CLI produces a structured terminal report summarizing all integrity checks and anomaly detections.

Download concretum_tests

A typical run includes:

• schema validation

• OHLC consistency checks

• early-close leakage detection

• phantom event counts

• stale-bar statistics

• missing-gap statistics

Corrected datasets can also be generated for:

• early-close cleanup

• phantom reconstruction

Stale and missing intervals are never modified.

Final Thoughts

Intraday market data is not a commodity.

Different providers make different decisions about:

• venue inclusion

• trade consolidation

• timestamp labeling

• inactive intervals

• delayed prints

• session boundaries

Those decisions propagate directly into backtests.

As shown throughout this research series, two datasets can both appear valid while still producing materially different strategy outcomes.

Once these structural issues are identified and controlled, provider divergence narrows significantly.

But the broader implication remains unchanged:

Data is not a neutral input.

If you found this article useful, share it with a friend. Feel free to leave a comment and reach out via direct message or email at info@concretumgroup.com for any questions.

Share

Disclaimer
This publication is provided by Concretum Group for informational, educational, and research purposes only. It does not constitute investment, financial, legal, or tax advice, nor a recommendation to buy or sell any security, instrument, strategy, or investment product. All investments involve risk, including possible loss of principal. Past performance, backtested performance, and historical analysis are not reliable indicators of future results. Readers should conduct their own research and consult qualified professionals before making investment decisions.

Full disclaimer: https://concretumgroup.com/disclaimer/

Quantitative research depends heavily on historical data, and the choice of data source, together with a robust procedure for retrieving and organizing it, is rarely a detail. It is one of the foundations on which every backtest stands or falls.

In the past, we have written extensively about how to source data for stocks and ETFs, from daily to intraday strategies (see here). In this article we turn to a different and considerably trickier asset class: futures contracts.

Futures have, for a long time, been the domain of institutional investors.

In recent years, however, the industry’s push toward smaller, retail-friendly products has made them increasingly accessible to well-capitalized individual traders.

The catch, from a data perspective, is that futures behave very differently from cash equities.

Stocks and ETFs do not expire; futures contracts do, and they do so on a regular cycle. As each contract approaches expiration, positions are rolled into the next active one, and the resulting price discontinuities, analogous to dividend gaps on equities, must be handled carefully when stitching together a continuous historical series.

Most professional data providers already adjust for these gaps and supply, alongside individual contract histories, a continuous time series in which the first-to-expire contract is systematically replaced by the next active one.

A usable futures database in Python typically relies on three core datasets: contract specifications, individual contract histories, and a continuous series.

The challenge is not access. It is structure.

These datasets follow different conventions and are not aligned by default, which makes it surprisingly easy to combine them incorrectly or to use the wrong prices when running a backtest.

The data we use here comes from Norgate Data, which offers long historical coverage across more than 100 futures markets, with some contracts going back to the 1980s.

This article shows how to retrieve those three datasets and organize them into a single, consistent pipeline.

What You Will Learn

By the end of this article, you will be able to retrieve contract specifications, individual contract data, and continuous futures series from Norgate, and to combine them into a set of Parquet files ready for research and backtesting.

A practical note before starting: the article walks through the pipeline step by step, and the companion notebook contains the full code. We recommend reading both from a desktop or laptop rather than a mobile device, since some of the code blocks and column-by-column descriptions are easier to follow on a wider screen, and trying the procedure end-to-end naturally requires a Python environment.

Code: Norgate_Futures_Data_Download

Subscribe now

Futures Database Output Example

Output Columns

01_contract_specs.parquet

• Symbol
  Norgate continuous futures symbol (e.g. &ES_CCB for E-mini S&P 500 back-adjusted).

• Name
  Full security name as provided by Norgate Data.

• Exchange
  Exchange where the contract trades (e.g. CME, NYMEX, ICE).

• Group
  Broadest Norgate classification (e.g. Currency, Energy, Grains).

• Contract Size
  Value of a one-point move in the contract’s currency.

• Tick Size
  Minimum price increment for the contract.

• Tick Value
  Value of one tick in the contract’s currency. Calculated as:
  Tick Size × Contract Size.

• Point Value
  Same as Contract Size.

• Currency
  Contract denomination (e.g. USD, GBP, EUR).

• Margin
  Exchange margin requirement in the contract’s currency.

02_individual_contracts_ohlcv.parquet

• Date
  Trading date.

• Open, High, Low, Close
  Price data for the individual contract.

• Volume
  Traded volume for this individual contract.

• Open Interest
  Open interest for this individual contract.

• Symbol
  Individual contract symbol (e.g. ES-2024Z for Dec 2024 E-mini S&P).

03_continuous_adjusted_and_unadjusted.parquet

• Date
  Trading date.

• Open, High, Low, Close
  Back-adjusted prices, shifted to eliminate gaps at contract rolls.

• Volume
  Total volume across all active contracts.

• Delivery Month
  Delivery month of the current front-month contract, encoded as YYYYMM.

• Open Interest
  Total open interest across all active contracts for this market.

• Symbol
  Continuous symbol (e.g. &ES_CCB). The _CCB suffix denotes a back-adjusted series.

• Close Unadjusted
  Closing price of the front-month contract (unadjusted).

04_continuous_with_volume_decomposition.parquet

• (All columns from 03)
  Same structure as the back-adjusted series

• FirstVolume
  Volume of the first-to-expire (front-month) contract.

• FirstContract
  Symbol of the first-to-expire contract (e.g. ES-2025M).

• SecondVolume
  Volume of the second-to-expire contract.

• SecondContract
  Symbol of the second-to-expire contract (e.g. ES-2025U).

Download & Run the Code

The full pipeline is provided as a single Jupyter notebook.

Quick Setup

1. Start Norgate Data

Open the Norgate Data Updater and keep it running in the background. This application provides the connection required for the Python API, so the code will not run unless it is active.

2. Enable Required Databases

Inside the Norgate Data Updater, navigate to the Database section and ensure that both the Futures and Continuous Futures databases are fully active.

The Futures database contains all individual contracts, including expired ones, while Continuous Futures provides the stitched front-month and back-adjusted series. If either is missing, select it, download it, and wait until it is fully available before proceeding.

3. Run the Notebook

After adjusting the configuration in the first cell (such as date range or output settings), run the entire notebook. In Jupyter, this can be done using the “Run All” option.

The notebook will download the data, process it, and generate the four Parquet files automatically in the output folder.

How the Workflow Operates

1. Contract Specifications

The workflow begins by retrieving contract metadata such as tick size, point value, margin, and currency for each market. This provides the foundation needed for sizing and risk calculations.

2. Individual Contracts

All individual futures contracts are then downloaded. Each contract is queried once to obtain the full OHLCV series with open interest. These are merged into a single dataset.

3. Continuous Futures

Continuous futures series are constructed by combining back-adjusted prices with the corresponding unadjusted front-month data. This produces a dataset where adjusted prices can be used for signal generation, while real prices are preserved for accurate sizing calculations.

The merge strips the _CCB suffix to create a BaseSymbol, then joins on Date + BaseSymbol so each adjusted row picks up the unadjusted Close and Volume from its non-_CCB counterpart.

4. Volume Decomposition

The final step enriches the continuous series with volume information drawn from the two contracts closest to expiry.

We flag this as a separate stage because of a subtlety in how Norgate exposes volume on its continuous contracts. In our experience, the volume column attached to a Norgate continuous series does not correspond to the front-month contract whose price is currently driving the series. It reports, instead, the sum of volume traded across the entire term structure for that market on that date, that is, the aggregate of all active contracts rather than the one effectively quoted by the continuous price.

The rationale behind this design choice is not clear but it is worth knowing about: a user pulling volume off a continuous series and assuming it reflects the front-month is, in fact, looking at curve-wide activity. For most analytical purposes (open interest concentration, roll dynamics, liquidity at the front of the curve) this is not what one wants.

To recover the more granular picture, our pipeline reconstructs front-month and second-month volumes separately for each date. It parses individual contract symbols into base code and expiry, builds a per-commodity map of contracts sorted by expiration, and then matches each date in the continuous series to the two nearest active contracts, recording their volumes alongside the continuous price.

Conclusion

A futures database is, in many ways, plumbing. It does not generate alpha on its own, and yet every backtest, every signal, every sizing decision sits on top of it. In our experience, the most stubborn bugs in systematic research rarely come from the strategy logic. They come from quiet inconsistencies in the data layer: a misaligned roll date, an unadjusted price used where an adjusted one was expected, a volume column that means something different from what one assumed.

The four-file structure presented here is the layout we have converged on after using Norgate across several research projects. Contract specifications anchor sizing and margin calculations. Individual contract histories preserve the full term structure whenever an analysis requires it. The continuous series, by carrying adjusted and unadjusted prices on the same row, lets signal generation and execution coexist without ambiguity. The volume decomposition layer, finally, bridges the continuous series back to its underlying contracts, which matters for any work touching liquidity, rolls, or carry.

The pipeline is intentionally simple, and that simplicity is the point. Once the data layer is solid and reproducible, the interesting part, the research itself, can finally begin.

We are aware that there is no single correct way to organize a futures database, and the structure that works best depends on the kind of research one runs on top of it. If your setup looks different, or if you have run into edge cases we have not, we would love to hear about it.

If you are looking for similar pipelines on other asset classes, our website has a dedicated coding section where we share the procedures we use to download stock data from various providers, both at daily and intraday frequency. Futures are the latest addition to that toolkit, and more will follow.

Concretum Coding

If you found this article useful, feel free to share it with a colleague or a friend who works with futures data, or reach out at info@concretumgroup.com for questions.

Disclaimer
This publication is provided by Concretum Group for informational, educational, and research purposes only. It does not constitute investment, financial, legal, or tax advice, nor a recommendation to buy or sell any security, instrument, strategy, or investment product. All investments involve risk, including possible loss of principal. Past performance, backtested performance, and historical analysis are not reliable indicators of future results. Readers should conduct their own research and consult qualified professionals before making investment decisions.

Full disclaimer: https://concretumgroup.com/disclaimer/

Introduction

This article shows how to run an Opening Range Breakout (ORB) strategy backtest in Python using Alpaca, with more than 10 years of intraday data.

In a previous article, we implemented the ORB strategy from Can Day Trading Really Be Profitable? (Zarattini & Aziz, 2023) using Massive (previously Polygon.io). The main limitation was data coverage. The free tier only provides two years of history, which restricts meaningful testing.

While extending this, we found something unexpected. The same strategy, with identical code and parameters, produced materially different results across data providers.

We investigated this separately:

→ Can Your Data Provider Be Trusted?

This article delivers on the original goal: a reproducible backtest using long-horizon intraday data, with access to SIP-quality data where required.

How to Run It Yourself

Running the backtest requires only a free API key and a browser.

These are the only three steps required:

1. Create a free Alpaca account and generate your API keys (paper trading keys work)

2. Open the notebook in Google Colab and paste your keys into the setup section

3. Click Run All

Open Notebook (10Y Free Data)

Data downloads in a few minutes and is cached. The full backtest runs in under a minute.

Subscribe now

Why Alpaca?

Alpaca provides long-horizon intraday data with a simple and developer-friendly infrastructure.

The free plan offers up to ~10 years of historical intraday data at 1-minute frequency, making it particularly attractive for rapid research and prototyping. However, this data is based on the IEX feed (a single-venue source) and therefore does not represent the full consolidated market.

For full market coverage, Alpaca also provides access to consolidated SIP data (CTA and UTP) through its paid plans, enabling NBBO-based aggregation across venues when higher data fidelity is required.

The data pipeline is straightforward. Daily bars can be requested fully adjusted for splits and dividends, while intraday bars remain unadjusted. This avoids the need for a separate adjustment process and keeps the implementation consistent with how trades would actually occur.

Access is available through a standard REST API, and the rate limits are sufficient to download large historical datasets in a relatively short time.

What Changed from the Previous Version

Relative to the Massive-based implementation:

• Historical coverage extends to 10+ years (since 2016)

• Dividend adjustments are handled natively

• The backtest spans a full market cycle instead of a short window

• Stop logic includes both H/L and ATR-based methods

The strategy itself is unchanged. Only the data handling and implementation details differ.

ORB Strategy Logic

The Opening Range Breakout strategy follows a simple structure:

• Use the first 5 minutes to determine direction

• Enter at the open of the 6th bar

• Apply either an H/L-based or ATR-based stop

• Hold until stop or market close

• Size positions to risk 1% of equity, capped at 4x leverage

Strategy Parameters

All parameters are fully configurable in the notebook and can be adjusted to test different setups:

• Opening range: 5 minutes

• H/L stop: range high/low

• ATR stop: 5% of 14-day ATR

• Profit target (H/L mode): 10R

• Profit target (ATR mode): none (hold to close)

• Risk per trade: 1% of equity

• Max leverage: 4x

• Starting capital: $25,000

• Commission/Slippage: $0.0035 per share per side

Walkthrough of the Notebook

The notebook runs the full pipeline from data download to performance analysis. You only need to configure it once.

Section 1: Setup & Configuration

This is the only section you edit:

API_KEY_ID     = os.environ.get("ALPACA_API_KEY_ID", "")
API_SECRET_KEY = os.environ.get("ALPACA_API_SECRET_KEY", "")
TICKER         = "TQQQ"
START_DATE     = "2016-01-01"
END_DATE       = "2026-04-20"

Everything else runs automatically.

Section 2: Data Download

The notebook pulls two datasets from Alpaca:

• Intraday data
  1-minute OHLCV bars, unadjusted, filtered to regular trading hours

• Daily data
  Fully adjusted OHLC data, used for ATR calculations

On the first run, both datasets are saved locally as CSV files.

Subsequent runs load from cache, so you do not re-download data.

Section 3: Load Data and Compute ATR

Daily data is used to compute ATR% (14-day ATR ÷ daily open, lagged by one day).

This gives a normalized volatility measure that is not affected by splits or dividends.

At trade time, the ratio is converted back into dollar terms using the current intraday open.

The stop distance is then computed as:

stop width = stop_atr × ATR% × (intraday open / entry)

This keeps the stop consistent with current price levels while preserving the stability of adjusted data.

Compared to the previous implementation, this removes the need for manual dividend adjustments and keeps the pipeline cleaner.

Section 4: Filter Early-Close Days

Half-days (e.g. Black Friday, Christmas Eve) are filtered using exchange_calendars.

Some datasets include post-close bars. If not removed, these can introduce invalid trades.

Data Quality Note
Early-close leakage can distort results and is a common source of inconsistency across providers.

Section 5 and 6. Backtest Engine

The strategy runs day by day:

• Determine direction from the opening range

• Enter at the open of the 6th bar

• Compute stop (H/L or ATR) and position size

• Walk forward through intraday data to detect exits

• Close at stop, gap, or end of day

• Update equity

Both stop modes are run separately for comparison.

Section 7: Performance Analysis

The notebook outputs:

• Summary metrics (CAGR, volatility, Sharpe, drawdown, hit rate)

• Monthly return heatmap

• Log-scale equity curve with out-of-sample period highlighted

Section 8 and 9: Run and Results

Results are displayed for both H/L and ATR modes under identical conditions.

The out-of-sample period begins after February 17, 2023, allowing direct evaluation beyond the original study window.

Conclusion

This implementation extends the original Massive-based backtest with a longer data history, a cleaner data pipeline, and a complete ATR-based stop.

The code is fully reproducible. Modify the parameters, change the ticker, and test it yourself.

→ Full Code: Google Colab Notebook

Subscribe now

If you found this article useful, feel free to leave a comment or reach out via direct message or email at info@concretumgroup.com for any questions.

Scope and limitations

This study evaluates the ORB strategy under a specific set of assumptions, instruments, and market conditions. Results are not universal and depend on data quality, execution assumptions, and implementation details. Small changes in these inputs can lead to materially different outcomes. Independent validation is recommended.

Note: intraday results may be sensitive to the underlying data source. The free dataset used here is based on IEX and may differ from consolidated SIP data, particularly for strategies relying on intraday highs, lows, or volume.

Disclosure

This analysis is conducted independently for research and educational purposes. We are not affiliated with, sponsored by, or compensated by any data provider referenced in this study.

Disclaimer
This publication is provided by Concretum Group for informational, educational, and research purposes only. It does not constitute investment, financial, legal, or tax advice, nor a recommendation to buy or sell any security, instrument, strategy, or investment product. All investments involve risk, including possible loss of principal. Past performance, backtested performance, and historical analysis are not reliable indicators of future results. Readers should conduct their own research and consult qualified professionals before making investment decisions.

Full disclaimer: https://concretumgroup.com/disclaimer/

Kraken’s public archive is one of the few free sources that gets surprisingly close to that standard, with intraday data stretching back to 2013 across a wide range of trading pairs.

What you’ll learn

At Concretum, we have made a significant effort to make this entire process transparent and reproducible by sharing it publicly.

In this piece, we walk through how to transform Kraken’s raw archive files into a single research-ready dataset, how to validate the output against an independent exchange, and where the resulting database proves genuinely useful in practice.

What you’ll need

From a practical standpoint, you will need access to Kraken’s public archive files, Python to run the consolidation and validation scripts, and sufficient local storage to handle several large compressed downloads.

Because some of the source files are multiple gigabytes in size, the initial download step is handled manually rather than through a fully automated pipeline.

The full technical guide, including the scripts and implementation notes, is available on our website

Access Full Code

This makes it possible to evaluate how much alpha survives when signals are formed shortly before the close and executed via Market on Close orders, versus the more common workflow of computing signals at the close and executing at the next morning’s open.

What you’ll learn

The focus here is building a database that combines both standard daily market data with a near-close intraday snapshot at 15:45 Eastern Time. The objective is to have data to test strategies in a way that better reflects how realistic execution-flows work in practice, while also keeping return calculations internally consistent.

What you’ll need

To build it properly, you need a Python environment and a data source that can supply unadjusted daily prices, split-adjusted closes, dividend records, and 15-minute intraday bars. For this implementation, we use Massive (formerly Polygon Data).

The full technical guide and accompanying Google Colab notebook are available on our website.

Access Full Code

The names that appear in a major index today are, almost by definition, the ones that survived. Firms that failed, shrank, merged, or were delisted along the way disappear from view entirely, and their absence makes historical tests look better than the real opportunity set ever was.

What you’ll learn

Our coding guide walks through how to reconstruct historical index membership in Python using Norgate Data, so that on any given date we can know with precision which stocks were actually part of an index and which were not. Our procedure also accounts for stocks that leave and later re-enter an index, an important detail for anyone planning to use the output for long-term backtests.

What you’ll need

To reproduce the process, you need Norgate Data set up with access to both active and delisted equities, together with Python and enough patience to process a full symbol universe across one or more indices. The whole workflow relies on dynamic point-in-time constituency data.

The full technical guide and accompanying Google Colab notebook are available on our website.

Access Full Code

The reason is straightforward: many failed projects disappear from standard data feeds, while the surviving assets remain easy to query: the result is a historical sample that quietly excludes many of the crypto market’s most unsuccessful stories…

What you’ll learn

Our article explains how to build a cryptocurrency database (using data from CoinMarketCap) that includes both successful projects and the large universe of tokens that have become mostly inactive, failed, or were delisted.

On top of that, it also covers how to enrich OHLCV with metadata so that the final output can also be analyzed by category if needed.

What you’ll need

To do this at full scale, you need access to the CoinMarketCap API, Python for data collection and batching, and enough runtime to process a very large universe of identifiers. The full build is not a quick scrape: it is a long-running collection job that has to respect API limits and save progress as it goes.

While the process is undeniably time-consuming, it is an indispensable step for anyone aiming to conduct proper, unbiased backtesting in the crypto space.

The full methodology, sample notebooks, and Google Colab resources are available on our website.

Access Full Code

Opening Range Breakout (ORB) strategies are built around the idea that when price leaves that opening range decisively, the move tends to carry further through the session.

What you’ll learn

Today we show readers how to build a configurable ORB backtesting engine in Python, using Polygon.io data. Our emphasis is not on hard-coding one fixed parameter set, but on creating a framework that lets the user vary assumptions in a controlled way and observe how the results change.

What you’ll need

In practice, that means you’ll need a Python environment, access to Polygon minute and daily data, and a clear idea regarding the strategy inputs to be tested.

Reviewing some of our earlier work on opening-range breakout strategies can be extremely beneficial, such as Can Day Trading Really be Profitable and A Profitable Day Trading Strategy For The U.S. Equity Market.

The complete technical guide and Google Colab notebook are available on our website

Access Full Code

Companies that failed are gone, the survivors remain, and the resulting sample looks healthier than the real market ever did: this is probably one of the oldest and most common forms of bias in quantitative research data.

What you’ll learn

The objective of this guide is to build a broad stock dataset in Python (leveraging Norgate Data) that includes both active and delisted names, and later enriching that price history with company names, sector classifications, and point-in-time index membership.

What you’ll need

To follow the process, you need Python, a Norgate Data installation with the relevant historical databases enabled, and access to both active and delisted equity symbols.

The workflow also assumes that you want more than raw prices, since part of the value comes from attaching sector and index context to each security.

The complete guide and Jupyter notebook are available on our website for free.

Access Full Code

For this backtest, we use Alpaca as our data provider. Unlike Polygon, which limits free access to two years of intraday data, Alpaca offers up to eight years of free historical data for a wide range of US stocks and ETFs.

What you’ll learn

The article walks through a Python backtest for a trading strategy on SPY that combines intraday momentum with mean reversion in overnight gaps. Along the way, it covers how the signals are defined, how volatility is used to scale exposure, and how the strategy holds up against a passive S&P 500 investment.

What you’ll need

To follow along, you only need Python and a free Alpaca API key, which provides access to historical intraday and daily SPY data (dividend records are also required to keep the total-return calculation accurate).

The implementation draws on a handful of indicators such as VWAP (the volume-weighted average price) and rolling volatility measures, so some familiarity with these concepts will be helpful.

The complete technical guide and Google Colab notebook are available on our website.

Access Full Code

What you’ll learn

This article pair covers the same industry-based trend-following framework implemented in both Python and MATLAB, using the 48 industry portfolios from Kenneth French’s data library. Across both implementations, readers are walked through the data preparation, signal construction, volatility-scaled position sizing, and performance evaluation over a long historical sample.

What you’ll need

To follow along, you only need Python installed on your machine or a MATLAB license, no proprietary data is required. All return series are drawn from Kenneth French’s publicly available data library, including the daily industry returns and the corresponding market and risk-free rate series.

The strategy draws on a set of standard technical indicators (i.e. Donchian channels and Keltner bands), therefore we think some familiarity with them will serve you well.

Both implementations are available for free on our website.

Full Python Implementation

Full Matlab Implementation

By making the backtesting procedure for our paper Beat the Market publicly available in both Python and MATLAB, we hope that readers can engage with the strategy not just conceptually, but hands-on.

In our experience, becoming familiar with the implementation is often where the deepest understanding comes from.

What you’ll learn

By working through this article pair, readers will learn how to translate our SPY intraday momentum strategy from Beat the Market into a fully reproducible backtest (in both Python and MATLAB), using data from Polygon.io.

That means building the indicators from scratch, applying the trading rules in code, and evaluating performance in a way that can be verified and cross-checked across both implementations.

What you’ll need

To follow along, you only need Python (or a MATLAB license) and a free Polygon.io API key, which provides access to minute-level and daily SPY data along with dividend records.

The implementation works entirely within the free tier, though it does require some patience: API rate limits need to be properly respected during the data collection step.

Both implementations are available for free on our website.

Full Python Implementation

Full Matlab Implementation