README.md

# Language Translation Dataset

<!--
Language Translation Dataset - README
Author: RSK World
Website: https://rskworld.in
Email: help@rskworld.in
Phone: +91 93305 39277
Copyright © 2016 RSK World. All rights reserved.
-->

## Overview

This dataset contains parallel sentence pairs in multiple languages with aligned translations. Perfect for machine translation, multilingual NLP, and cross-lingual model training.

## Features

- ✅ Parallel sentences
- ✅ Multiple language pairs
- ✅ Aligned translations
- ✅ Training and validation sets
- ✅ Ready for translation models

## Dataset Information

- **Category**: Text Data
- **Difficulty**: Advanced
- **Technologies**: TSV, JSON, Transformers, mBERT, mT5
- **Format**: TSV (Tab-Separated Values) and JSON

## Dataset Structure

The dataset includes parallel sentence pairs in multiple languages:

- English
- Spanish
- French
- German
- And more...

## Files Included

- `data/train.tsv` - Training dataset in TSV format
- `data/validation.tsv` - Validation dataset in TSV format
- `data/train.json` - Training dataset in JSON format
- `data/validation.json` - Validation dataset in JSON format
- `data/sample_data.json` - Sample data for preview
- `scripts/process_data.py` - Python script for data processing
- `scripts/convert_format.py` - Script to convert between TSV and JSON formats

## Usage

### Loading TSV Data

```python
import pandas as pd

# Load training data
train_df = pd.read_csv('data/train.tsv', sep='\t')
print(train_df.head())
```

### Loading JSON Data

```python
import json

# Load training data
with open('data/train.json', 'r', encoding='utf-8') as f:
train_data = json.load(f)
```

### Using with Transformers

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("google/mt5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-small")
```

## Installation

1. Download the dataset from the provided link
2. Extract the files
3. Install required dependencies:

```bash
pip install pandas transformers torch
```

## Processing Scripts

Run the data processing script:

```bash
python scripts/process_data.py
```

Convert between formats:

```bash
python scripts/convert_format.py --input data/train.tsv --output data/train.json
```

## Citation

If you use this dataset in your research, please cite:

```
Language Translation Dataset
RSK World (https://rskworld.in)
2016
```

## Contact

- **Website**: https://rskworld.in
- **Email**: help@rskworld.in
- **Phone**: +91 93305 39277

## License

Copyright © 2016 RSK World. All rights reserved.

---

**Created by RSK World** - Free Programming Resources & Source Code