SETUP.md

# Setup Guide - Language Translation Dataset

<!--
Language Translation Dataset - Setup Guide
Author: RSK World
Website: https://rskworld.in
Email: help@rskworld.in
Phone: +91 93305 39277
Copyright © 2016 RSK World. All rights reserved.
-->

## Quick Start

### 1. Install Dependencies

```bash
pip install -r requirements.txt
```

### 2. Verify Dataset Files

The dataset files should be located in the `data/` directory:

- `data/train.tsv` - Training dataset (TSV format)
- `data/train.json` - Training dataset (JSON format)
- `data/validation.tsv` - Validation dataset (TSV format)
- `data/validation.json` - Validation dataset (JSON format)
- `data/sample_data.json` - Sample data for preview

### 3. Process the Dataset

Run the data processing script:

```bash
python scripts/process_data.py
```

### 4. Analyze the Dataset

Get detailed statistics:

```bash
python scripts/analyze_dataset.py
```

### 5. Convert Formats

Convert between TSV and JSON:

```bash
# TSV to JSON
python scripts/convert_format.py --input data/train.tsv --output data/train.json

# JSON to TSV
python scripts/convert_format.py --input data/train.json --output data/train.tsv
```

### 6. Run Examples

See example usage:

```bash
python examples/example_usage.py
```

## Using with Transformers

### Load with mBERT

```python
from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bert-base-multilingual-cased")
model = AutoModel.from_pretrained("bert-base-multilingual-cased")
```

### Load with mT5

```python
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("google/mt5-small")
model = AutoModelForSeq2SeqLM.from_pretrained("google/mt5-small")
```

## Project Structure

```
language-translation/
├── data/
│ ├── train.tsv
│ ├── train.json
│ ├── validation.tsv
│ ├── validation.json
│ ├── sample_data.json
│ └── METADATA.txt
├── scripts/
│ ├── process_data.py
│ ├── convert_format.py
│ └── analyze_dataset.py
├── examples/
│ └── example_usage.py
├── index.html
├── README.md
├── SETUP.md
├── requirements.txt
├── config.py
└── LICENSE
```

## Support

For questions or support, please contact:

- **Website**: https://rskworld.in
- **Email**: help@rskworld.in
- **Phone**: +91 93305 39277

---

**Created by RSK World** - Free Programming Resources & Source Code