Loading ...

Building an AI Part 1: The Core Framework

03 September 2026

Leveraging Artificial Intelligence is no longer optional for businesses looking to maintain a competitive edge. The ability to process and analyze time-series data—from sales figures to user engagement metrics—is critical for accurate forecasting and strategic planning. This guide gets straight to the point, breaking down the essential technical framework required to build a reliable AI foundation. We'll use a sophisticated Python script as a real-world example to demonstrate the core principles of data preparation and modeling in both TensorFlow and PyTorch, providing your team with the foundational knowledge to drive data-centric business decisions.

Prerequisites 🛠️

Before you begin, ensure your development environment is set up correctly. This guide assumes you have the following:

  • Python: Version 3.12 or newer.

  • pip: The latest version to ensure compatibility with modern packages. You can upgrade it by running: python -m pip install --upgrade pip.

You will need to install several key libraries. You can install them all with the following commands:

pip install pandas scikit-learn numpy

pip install tensorflow #or below

pip install torch

 

The Role of Pandas: The Data Scientist's Toolkit 🐼

Before data ever reaches a machine learning model, it needs to be loaded, cleaned, and manipulated. For this, pandas is the undisputed champion in the Python ecosystem. It provides the DataFrame, a powerful and intuitive two-dimensional table structure that is perfectly suited for the kind of tabular and time-series data used in the script.

Why is pandas essential?

  • Effortless Data Handling: It simplifies loading data from various sources (like CSVs or APIs) into a clean, human-readable format.

  • Time-Series Powerhouse: Pandas has incredibly powerful, built-in tools for working with time-series data. The script uses it to reindex data to different timeframes (e.g., 1-minute, 5-minute, 15-minute), which is crucial for multi-timeframe analysis.

  • Easy Data Cleaning: Real-world data is messy. Functions like .fillna(0) provide a simple way to handle missing values, preventing errors during model training.

  • Seamless Integration: Pandas DataFrames work hand-in-hand with virtually every other scientific library. The script seamlessly passes data from pandas into scikit-learn for scaling, and then into NumPy arrays to be converted into TensorFlow or PyTorch tensors.

  • Vectorized Operations: Operations in pandas are highly optimized. Calculating a percentage change across thousands of rows is a simple, one-line command that executes with remarkable speed.

In short, pandas acts as the universal backbone for data preparation. It provides the tools to wrangle raw data into the pristine, structured format that machine learning models require.

Feature Scaling and Normalization ⚖️

Machine learning models perform best when their input features share a similar scale. This prevents features with larger ranges from unfairly dominating the learning process. Normalization is the process of rescaling data to fit a specific range.

Standard Scaling

This method rescales data to have a mean of 0 and a standard deviation of 1. It's highly effective when your data is roughly normally distributed. The provided script uses StandardScaler from scikit-learn, a common, framework-agnostic approach.

# The scikit-learn approach works for both frameworks

from sklearn.preprocessing import StandardScaler

 

# scaler is fit on training data

scaler = StandardScaler().fit(training_data)

 

# The scaler is then used to transform data before feeding it into the model

scaled_data = scaler.transform(input_data)

Percentage Change Normalization

Sometimes, the relative change between data points is more important than the absolute values. This is where pandas' efficiency shines before the data is even converted to a tensor.

TensorFlow/Keras Example:

import pandas as pd

import tensorflow as tf

 

# Assuming 'data' is a pandas DataFrame with a 'Close' column

# Pandas makes this calculation trivial

data['pct_change'] = data['Close'].pct_change().fillna(0)

 

# The normalized column is then converted to a TensorFlow tensor

input_tensor = tf.convert_to_tensor(data['pct_change'].values, dtype=tf.float32)

PyTorch Example: The logic is identical, with the final array being converted to a PyTorch tensor.

import pandas as pd

import torch

 

# The pandas calculation remains the same

data['pct_change'] = data['Close'].pct_change().fillna(0)

 

# Convert the numpy array to a PyTorch tensor

input_tensor = torch.tensor(data['pct_change'].values, dtype=torch.float32)

 

From Timestamps to Sequences: Preparing Data for Time-Series Models 🧠

To predict future values, a model needs to see the recent past. This requires converting a flat list of data into a dataset of overlapping sequences. The script uses a straightforward Python function for this, which works seamlessly with both frameworks.

The create_sequence_dataset function slides a window across the data, creating numerous samples. Each sample contains a sequence of historical data (e.g., 60 events) and the corresponding target outcome.

def create_sequence_dataset(X_data, y_data, time_steps):

    Xs, ys = [], []

    for i in range(len(X_data) - time_steps):

        Xs.append(X_data.iloc[i:(i + time_steps)].values)

        ys.append(y_data.iloc[i + time_steps - 1])

    return np.array(Xs), np.array(ys)

Once the numpy arrays are created, they are converted into the native tensor format for each framework before being fed into the model.

TensorFlow/Keras: X_train_seq_tf = tf.convert_to_tensor(X_train_seq, dtype=tf.float32)

PyTorch: X_train_seq_torch = torch.tensor(X_train_seq, dtype=torch.float32)

Performance Simulation with a Dose of Reality 📉

A historical simulation tests your AI's strategy on past data to see how it would have performed. The run_trial_backtest function in the script provides a high-level, framework-agnostic logic for this. This simulation sits above the core model and can be used to evaluate any model, regardless of whether it was built in TensorFlow or PyTorch.

Key realistic details from the script include:

  • Operational Costs: The simulation accounts for potential execution variance and transaction costs.

  • Dynamic Resource Management: The magnitude of an action is based on a small, fixed percentage of total available resources.

  • Post-Failure Cooldown: After a negative outcome, the system pauses before taking new action, preventing compounded errors.

Building Beyond the Basics with Custom Layers 🏗️

Both TensorFlow and PyTorch offer extensive flexibility for creating custom layers, allowing you to design novel architectures tailored to your specific problem.

Custom Layer: Transformer Block

The TransformerBlock is a complex layer perfect for capturing relationships in sequential data.

TensorFlow/Keras Example: The script defines a custom Keras Layer, subclassing tf.keras.layers.Layer and implementing the call method for the forward pass logic.

import tensorflow as tf

from tensorflow.keras.layers import Layer, MultiHeadAttention, Dense, LayerNormalization, Dropout

 

class TransformerBlock(Layer):

PyTorch Example: In PyTorch, you subclass torch.nn.Module and define the forward pass in the forward method. The architecture is identical.

Custom Layer: Percentage Change

For a simpler example, a layer that calculates the percentage change can be useful for on-the-fly feature engineering directly within the model.

TensorFlow/Keras Example:

import tensorflow as tf

from tensorflow.keras.layers import Layer

 

PyTorch Example:

import torch

import torch.nn as nn

import torch.nn.functional as F

 

class PercentageChange(nn.Module):