Project: Newspaper Classifier on Bias

Today I managed to train my first BERT model using Huggingface Transformers and datasets in just 2 hours. Here's a quick rundown of how I did it:

1. Data Collection
2. Data Preparation
3. Model Training

It was surprisingly simpler and easier than expected. It's fascinating how much we can procrastinate even the simplest tasks. Here's what I learned from each stage:

Data Collection

To collect data, I focused on news agencies that have a clear bias towards one side. This helped me quickly label them as.

I initially tried using scrapy, a great web scraping library. Although it seems like a powerful tool, it took me too long to learn and I couldn't get anything useful in time for this project.

Next, I experimented with playwright, but I quickly encountered issues with reCAPTCHA, which blocked my progress.

Then, I attempted to use Google News combined with newspaper3k to extract articles, but once again, I was blocked.

In the end, I resorted to using the reader mode on the Brave browser and manually collecting all the data. While this worked, it left me with a relatively small dataset.

Data Preparation

After labeling the data, I created a Huggingface dataset and tokenized the text:

from transformers import AutoTokenizer
import datasets

tokenizer = AutoTokenizer.from_pretrained("distilbert/distilbert-base-uncased")

def tokenize_function(examples):
    return tokenizer(examples["text"], padding=True, truncation=True)

dataset = datasets.Dataset.from_pandas(articles_df)
dataset = dataset.train_test_split(test_size=0.2)
tokenized_dataset = dataset.map(tokenize_function, batched=True)
print(tokenized_dataset)

Finally, I trained the model, which turned out to be quite straightforward using Huggingface.

First, I defined the model:

from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert/distilbert-base-uncased",
    num_labels=2,
    id2label=id2label,
    label2id=label2id,
)

Next, I defined the output path:

from pathlib import Path

models_dir = Path("models")
models_dir.mkdir(parents=True, exist_ok=True)

model_save_name = "articles-bias-detection"
model_save_dir = Path(models_dir, model_save_name)

model_save_dir

Then I set up the training arguments:

from transformers import TrainingArguments

training_args = TrainingArguments(
    output_dir=model_save_dir,
    evaluation_strategy="epoch",
    learning_rate=2e-5,
    per_device_train_batch_size=8,
    per_device_eval_batch_size=8,
    num_train_epochs=3,
    weight_decay=0.01,
    eval_strategy="epoch",
    save_strategy="epoch",
    save_total_limit=3,
    seed=42,
    load_best_model_at_end=True,
    logging_strategy="epoch",
    report_to="none",
)

Afterward, I defined the trainer:

from transformers import Trainer

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"],
    eval_dataset=tokenized_dataset["test"],
    tokenizer=tokenizer,
)

Finally, I trained the model:

trainer.train()

Once the training was complete, I tested the model:

# Trying out the model 

from transformers import pipeline

classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)

As you can see however, the accuracy is very low, meaning the model is not efficient. At 50% accuracy, thats literally a coin toss. Meaning my model is just barely better than a coin toss accuracy...

This project was a great learning experience, and I look forward to refining and improving it further.