Any engineer involved in machine learning projects has likely hit the wall of “not having enough data” at some point. Even with cutting-edge algorithms and sufficient computing resources, if you lack data to train on, the model becomes nothing more than an unused treasure. This is similar to having premium ingredients and cooking utensils but being unable to cook because you’re missing the key ingredients.
In the past, I too faced the bitter decision in developing an anomaly detection system where I had plenty of “normal data” but almost no “anomalous data.” In such cases, traditional data collection methods alone have limits in terms of both time and cost. This is where “synthetic data” generation using LLMs (Large Language Models) has been gaining attention.
This time, I’ll delve deeply into why this technology is necessary, how it works, and how to actually implement automatic high-quality data generation using Python, drawing on my own experience.
Why synthetic data is needed in the first place
Traditionally, “data augmentation” was common for addressing data shortages. For image recognition, this means techniques like rotating images or adding noise to increase data volume. However, simple replacement or rule-based augmentation has limitations for text data and structured data. Word replacement without considering context can actually become noise that hinders model learning.
This is where the approach using LLM capabilities becomes a major turning point. Since LLMs learn language rules and context from vast text corpora, they can naturally generate “plausible” data based on specific domains or patterns. The biggest difference from existing methods is that it doesn’t just “pad” data but “creates” new variations.
It becomes a powerful weapon especially in the following scenarios:
- Resolving class imbalance: When there’s bias in the frequency of events, such as in anomaly detection or specific sentiment analysis.
- Privacy protection: In fields like healthcare and finance where real data usage is strictly regulated.
- Rare case simulation: Generating error patterns that rarely occur in reality but are critical for systems.
Synthetic data generation with LLMs is positioned as a strategic approach to enhance model robustness, not just a cost-cutting measure.
Technical explanation: How LLM-based data generation works
From a technical perspective, LLM-based data generation consists of a combination of “prompt engineering” and “validation loops.” Simply asking an LLM to “create data” carries the risk of generating biased or logically inconsistent data.
To obtain high-quality synthetic data, you typically build a pipeline like the following:
The key to this process is the “validation layer.” Instead of blindly accepting data generated by the LLM, you ensure reliability by programmatically checking the format or scoring quality with another lightweight model.
Additionally, the “Few-Shot Prompting” technique is used during generation. By including several examples of the data you want to generate in the prompt, the LLM can more accurately mimic data distribution and style. Furthermore, research has shown that for complex data generation tasks, accuracy improves by encouraging Chain-of-Thought and having the LLM output while thinking about the reasons for data generation.
Implementation example: Automatic generation pipeline with Python
Let’s look at specific code. Here, we’ll assume a scenario where we generate customer support inquiry data and use it for classification tasks.
We’ll use Python and the OpenAI API, but with an implementation mindful of production environments, including error handling, logging, and output validation. The key is not just outputting text but generating structured data in JSON format and validating it with Pydantic.
First, install the necessary libraries and configure settings.
import asyncio
import logging
import json
from typing import List, Dict, Any
from pydantic import BaseModel, ValidationError
from openai import AsyncOpenAI
import os
# Logging configuration
logging.basicConfig(
level=logging.INFO,
format='%(asctime)s - %(levelname)s - %(message)s'
)
logger = logging.getLogger(__name__)
# Get API key from environment variables (in practice, use .env file, etc.)
client = AsyncOpenAI(api_key=os.getenv("OPENAI_API_KEY"))
# Data model definition (structure of the data we want to generate)
class TicketData(BaseModel):
id: int
category: str # Example: billing, technical issue, account
sentiment: str # Example: positive, neutral, negative
text: str
class SyntheticDataGenerator:
def __init__(self, model_name: str = "gpt-4o"):
self.model_name = model_name
self.generated_count = 0
self.error_count = 0
def _create_prompt(self, num_examples: int = 5) -> str:
"""Create prompt. Include Few-Shot examples"""
examples = """
Example 1:
Category: 請求, Sentiment: ネガティブ
Text: 請求書がまだ届いていません。いつになりますか？
Example 2:
Category: 技術的問題, Sentiment: ポジティブ
Text: 新機能の使い方がとても分かりやすくて助かりました。
Example 3:
Category: アカウント, Sentiment: ニュートラル
Text: パスワードを変更したいのですが、手順を教えてください。
"""
return f"""
You are a customer support data generation assistant.
Based on the examples below, generate customer support ticket data.
Return the data as a valid JSON list.
Each object must have "id", "category", "sentiment", and "text" keys.
Assign consecutive numbers to "id".
{examples}
Generate {num_examples} new unique examples in JSON list format.
"""
async def _generate_with_retry(self, prompt: str, max_retries: int = 3) -> str:
"""Generation method including retry logic"""
for attempt in range(max_retries):
try:
response = await client.chat.completions.create(
model=self.model_name,
messages=[
{"role": "system", "content": "You are a helpful data generator."},
{"role": "user", "content": prompt}
],
temperature=0.7, # Balance between creativity and consistency
response_format={"type": "json_object"} # Force JSON mode
)
return response.choices[0].message.content
except Exception as e:
logger.warning(f"Attempt {attempt + 1} failed: {e}")
if attempt == max_retries - 1:
raise
await asyncio.sleep(2 ** attempt) # Exponential Backoff
return ""
def _validate_data(self, raw_data: str) -> List[TicketData]:
"""Validate generated JSON data"""
try:
data_json = json.loads(raw_data)
if "items" in data_json:
data_list = data_json["items"]
elif isinstance(data_json, list):
data_list = data_json
else:
# Handle single objects or unexpected structures
logger.warning("Unexpected JSON structure, trying to adapt...")
return []
validated_tickets = []
for item in data_list:
try:
ticket = TicketData(**item)
validated_tickets.append(ticket)
except ValidationError as ve:
logger.error(f"Validation error for item {item}: {ve}")
self.error_count += 1
return validated_tickets
except json.JSONDecodeError as e:
logger.error(f"JSON decode error: {e}")
self.error_count += 1
return []
async def generate_batch(self, batch_size: int = 5) -> List[TicketData]:
"""Generate and validate one batch of data"""
logger.info(f"Generating batch of {batch_size} items...")
prompt = self._create_prompt(batch_size)
try:
raw_content = await self._generate_with_retry(prompt)
validated_data = self._validate_data(raw_content)
self.generated_count += len(validated_data)
logger.info(f"Successfully generated and validated {len(validated_data)} items.")
return validated_data
except Exception as e:
logger.error(f"Failed to generate batch: {e}")
return []
async def main():
generator = SyntheticDataGenerator()
all_tickets = []
# Generate a total of 20 data items (4 batches)
for _ in range(4):
batch = await generator.generate_batch(batch_size=5)
all_tickets.extend(batch)
# Short wait to avoid API rate limits
await asyncio.sleep(1)
logger.info(f"Generation complete. Total valid: {generator.generated_count}, Errors: {generator.error_count}")
# Display sample results
for ticket in all_tickets[:3]:
print(f"ID: {ticket.id} | Cat: {ticket.category} | Sent: {ticket.sentiment}")
print(f"Text: {ticket.text}\n")
if __name__ == "__main__":
asyncio.run(main())An important point in this code is specifying response_format={"type": "json_object"}. This forces the LLM to always output in JSON format instead of text, making subsequent program processing much easier. Additionally, using a Pydantic model automatically detects missing fields or type mismatches, maintaining dataset quality.
Business use case: Fraud detection in financial institutions
Now that we’ve covered the technical details, how does this work in actual business? Let’s take the improvement of “fraud detection systems” in financial institutions as a specific example.
A major challenge facing many financial institutions is “overwhelming shortage of fraud transaction data.” Over 99.9% of credit card usage data consists of normal transactions, with fraudulent usage being extremely rare. When training a machine learning model on this imbalanced data, the model learns that “predicting all transactions as normal gives a 99.9% accuracy rate,” causing it to miss fraud.
This is where synthetic data using LLMs helps. By having LLMs learn past fraud patterns (sudden IP address changes, high-value payments in unusual time zones, consecutive usage in specific commercial categories, etc.), we generate thousands of “plausible” fraud transaction patterns that don’t exist in reality. Mixing these into the training data helps the model capture the subtle characteristics of fraud.
At a fintech company I know, introducing this method improved fraud detection recall by about 15%, successfully preventing hundreds of millions of yen in annual damages. One factor behind this effectiveness is that LLMs can create patterns close to “new methods” not included in the real data.
Frequently Asked Questions
Q: Is synthetic data inferior in quality compared to real data?
A: Properly generated synthetic data maintains the statistical properties of real data while having less noise and accurate labeling, so it can sometimes improve model performance over real data for specific tasks. A major advantage of synthetic data is its ability to remove human errors and biases contained in real data.
Q: How much does LLM-based data generation cost?
A: Cost depends on the model used and the amount generated, but using small-scale or open-source models locally can significantly reduce API usage fees. It’s common to perform fine-tuning after initial dataset construction, and the total cost performance is very high.
Q: Is synthetic data generation safe for data containing personal information?
A: Yes, a major advantage of synthetic data generation is privacy protection. Even if patterns from original data are learned, the generated data is new data that cannot identify real individuals, making it effective for compliance with regulations like GDPR. However, appropriate consideration of differential privacy may be necessary to guarantee complete anonymization.
Summary
- Synthetic data generation with LLMs is a powerful means to solve the longstanding machine learning challenges of data shortages and class imbalance.
- What sets it apart from conventional augmentation techniques is its ability to understand context and patterns to “create” new data, not just pad existing data.
- In implementation, engineering considerations like using JSON mode and Pydantic validation determine data quality.
- Even in highly regulated industries like finance, there are track records of improving model accuracy while protecting privacy, resulting in very high business impact.
Recommended Resources
- Book: “Synthetic Data for Deep Learning” (Springer)
- A specialized book covering everything from theoretical background to practical applications of synthetic data generation.
- Tool: Gretel.ai
- A synthetic data generation platform specializing in text and structured data. It has a comprehensive SDK, making it easy for engineers to implement.
- Tool: Mostly AI
- A synthetic data generation tool with particular strengths in privacy protection, with extensive implementation experience in the financial and healthcare industries.
AI Implementation Support & Development Consultation
Are you struggling with LLM-based data generation or building AI models using your company’s data? We provide practical AI implementation support from requirements definition to implementation and maintenance, from an engineer’s perspective. Please feel free to contact us through the inquiry form first.
References
[1]OpenAI API Documentation [2]Self-Instruct: Aligning Language Models with Self-Generated Instructions [3]Synthetic Data: A Primer
