In machine learning, the train-test split is a critical technique used to evaluate the performance of a model. It involves dividing your dataset into two subsets:

• Training Set: This subset is used to train the model, allowing it to learn patterns from the data.
• Testing Set: This subset is used to evaluate the model’s performance after training, helping us understand how well the model generalizes to new, unseen data.

By splitting the data, we reduce the risk of overfitting, where a model performs well on the training data but poorly on new data. A proper split ensures that the model is tested on data it has not seen during training, providing a more realistic measure of its effectiveness.

Here are the main reasons why splitting the data is necessary:

• Avoid Overfitting: Training and testing on the same data can cause the model to memorize patterns (overfit), and it will not perform well on unseen data.
• Validate the Model: By testing on a separate set, you can evaluate how well your model generalizes to new data, which is essential for real-world applications.

In Python, we use the train_test_split function from the scikit-learn library to split the data. Here’s how it works:

# Import necessary library
from sklearn.model_selection import train_test_split

# Split the dataset
X = data.drop('medv', axis=1)  # Features (without target variable 'medv')
y = data['medv']  # Target variable ('medv')

# Train-test split (80% train, 20% test)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Check the shapes of the resulting datasets
X_train.shape, X_test.shape, y_train.shape, y_test.shape
  

1. Data Preparation: We define X as the features (everything except the target variable) and y as the target variable (the house prices).
2. train_test_split: The function splits the data into 80% training and 20% testing using the test_size=0.2 argument. The random_state=42 ensures reproducibility of the split.
3. Shape Check: The shapes of the training and testing sets are displayed to confirm the split (e.g., 80% training, 20% testing).

After splitting the data, you can now proceed to build your model. For example, if you’re using Linear Regression, you can:

# Import the Linear Regression model
from sklearn.linear_model import LinearRegression

# Create and train the model
model = LinearRegression()
model.fit(X_train, y_train)

# Predict using the test data
y_pred = model.predict(X_test)

# Evaluate the model's performance
from sklearn.metrics import mean_squared_error, r2_score

mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

print(f'Mean Squared Error: {mse}')
print(f'R-squared: {r2}')