classification_X.npy CLEANUP_GUIDE.md autoencoders.py

classification_X.npy

This file cannot be displayed in the browser.

CLEANUP_GUIDE.md

# Cleanup Guide for __pycache__ and Data Folders

**Author**: RSK World
**Website**: https://rskworld.in
**Email**: help@rskworld.in
**Phone**: +91 93305 39277

## Overview

This guide explains how to manage `__pycache__` folders and the `data` directory in the TensorFlow Deep Learning project.

## __pycache__ Folders

### What are __pycache__ folders?

Python automatically creates `__pycache__` directories to store compiled bytecode (`.pyc` files) for faster module loading. These are generated automatically and don't need to be committed to version control.

### Current Status

- ✅ `__pycache__/` is already in `.gitignore`
- ✅ All `.pyc`, `.pyo`, `.pyd` files are ignored
- ✅ No `__pycache__` folders currently exist in the project

### Cleaning __pycache__ Folders

#### Method 1: Using Python Script (Cross-platform)
```bash
python scripts/clean_cache.py
```

#### Method 2: Using Shell Script (Linux/Mac)
```bash
bash scripts/cleanup.sh
```

#### Method 3: Using Batch Script (Windows)
```cmd
scripts\cleanup.bat
```

#### Method 4: Manual Cleanup

**Windows (PowerShell):**
```powershell
Get-ChildItem -Path . -Filter "__pycache__" -Recurse -Directory | Remove-Item -Recurse -Force
Get-ChildItem -Path . -Filter "*.pyc" -Recurse -File | Remove-Item -Force
```

**Linux/Mac:**
```bash
find . -type d -name "__pycache__" -exec rm -r {} +
find . -type f -name "*.pyc" -delete
```

**Python:**
```python
import os
import shutil

for root, dirs, files in os.walk('.'):
if '__pycache__' in dirs:
shutil.rmtree(os.path.join(root, '__pycache__'))
for file in files:
if file.endswith('.pyc'):
os.remove(os.path.join(root, file))
```

## Data Folder

### Current Structure

The `data/` folder contains:
- Generated datasets (`.npy`, `.csv` files)
- Metadata files (`.json`)
- Visualizations (if generated)
- README.md documentation

### Git Configuration

The `.gitignore` is configured to:
- ✅ Ignore generated data files (`*.npy`, `*.csv`, `*.json`)
- ✅ Keep directory structure (`.gitkeep` file)
- ✅ Keep documentation (`README.md`)

### Data Folder Contents

```
data/
├── .gitkeep # Keeps folder in git
├── README.md # Documentation (tracked)
├── classification_X.npy # Generated (ignored)
├── classification_y.npy # Generated (ignored)
├── regression_X.npy # Generated (ignored)
├── images_X.npy # Generated (ignored)
├── sequences_X.npy # Generated (ignored)
├── tabular.csv # Generated (ignored)
└── *_metadata.json # Generated (ignored)
```

### Regenerating Data

If you need to regenerate all data:
```bash
python scripts/generate_data_standalone.py
```

### Cleaning Data Folder

To remove all generated data (but keep structure):
```bash
# Remove all .npy files
find data -name "*.npy" -delete

# Remove all .csv files
find data -name "*.csv" -delete

# Remove all .json metadata files
find data -name "*_metadata.json" -delete
```

Or use Python:
```python
import os
import glob

data_dir = './data'
for pattern in ['*.npy', '*.csv', '*_metadata.json']:
for file in glob.glob(os.path.join(data_dir, pattern)):
os.remove(file)
print(f"Removed: {file}")
```

## Best Practices

### 1. Before Committing
- Run cleanup script to remove `__pycache__` folders
- Verify `.gitignore` is working correctly
- Don't commit generated data files

### 2. Regular Maintenance
- Clean `__pycache__` folders periodically
- Regenerate data if needed for testing
- Keep data directory structure intact

### 3. CI/CD Integration
Add cleanup to your CI/CD pipeline:
```yaml
# Example GitHub Actions
- name: Clean __pycache__
run: python scripts/clean_cache.py
```

## Troubleshooting

### Issue: __pycache__ folders keep appearing
**Solution**: This is normal! Python creates them automatically. Just run the cleanup script before committing.

### Issue: Data files are being tracked by git
**Solution**: Check `.gitignore` and ensure patterns are correct. Remove tracked files:
```bash
git rm --cached data/*.npy
git rm --cached data/*.csv
```

### Issue: Can't delete __pycache__ folders
**Solution**: Make sure no Python processes are using them. Close IDEs and Python interpreters, then try again.

## Scripts Available

1. **scripts/clean_cache.py** - Python cleanup script (recommended)
2. **scripts/cleanup.sh** - Shell script for Linux/Mac
3. **scripts/cleanup.bat** - Batch script for Windows

## Summary

- ✅ `__pycache__` folders are ignored by git
- ✅ Generated data files are ignored by git
- ✅ Directory structure is preserved
- ✅ Cleanup scripts are available
- ✅ Documentation is tracked

All cleanup operations are safe and won't affect your source code or project structure.

src/autoencoders.py

Raw Download

"""
Autoencoders with TensorFlow
Author: RSK World
Website: https://rskworld.in
Email: help@rskworld.in
Phone: +91 93305 39277

This module demonstrates various autoencoder architectures.
"""

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, Model
import numpy as np
import matplotlib.pyplot as plt

def build_simple_autoencoder(input_shape=(784,), encoding_dim=32):
    """
    Build a simple autoencoder.
    
    Args:
        input_shape: Shape of input data
        encoding_dim: Dimension of encoding layer
    
    Returns:
        Autoencoder model, encoder model, decoder model
    """
    # Encoder
    encoder_input = keras.Input(shape=input_shape)
    encoded = layers.Dense(128, activation='relu')(encoder_input)
    encoded = layers.Dense(64, activation='relu')(encoded)
    encoded = layers.Dense(encoding_dim, activation='relu')(encoded)
    
    encoder = Model(encoder_input, encoded, name='encoder')
    
    # Decoder
    decoder_input = keras.Input(shape=(encoding_dim,))
    decoded = layers.Dense(64, activation='relu')(decoder_input)
    decoded = layers.Dense(128, activation='relu')(decoded)
    decoded = layers.Dense(input_shape[0], activation='sigmoid')(decoded)
    
    decoder = Model(decoder_input, decoded, name='decoder')
    
    # Autoencoder
    autoencoder_input = keras.Input(shape=input_shape)
    encoded_output = encoder(autoencoder_input)
    decoded_output = decoder(encoded_output)
    
    autoencoder = Model(autoencoder_input, decoded_output, name='autoencoder')
    
    autoencoder.compile(
        optimizer='adam',
        loss='binary_crossentropy',
        metrics=['accuracy']
    )
    
    return autoencoder, encoder, decoder

def build_convolutional_autoencoder(input_shape=(28, 28, 1), encoding_dim=32):
    """
    Build a convolutional autoencoder.
    
    Args:
        input_shape: Shape of input images
        encoding_dim: Dimension of encoding layer
    
    Returns:
        Autoencoder model, encoder model, decoder model
    """
    # Encoder
    encoder_input = keras.Input(shape=input_shape)
    x = layers.Conv2D(32, 3, activation='relu', padding='same')(encoder_input)
    x = layers.MaxPooling2D(2, padding='same')(x)
    x = layers.Conv2D(64, 3, activation='relu', padding='same')(x)
    x = layers.MaxPooling2D(2, padding='same')(x)
    x = layers.Conv2D(64, 3, activation='relu', padding='same')(x)
    x = layers.Flatten()(x)
    encoded = layers.Dense(encoding_dim, activation='relu')(x)
    
    encoder = Model(encoder_input, encoded, name='encoder')
    
    # Decoder
    decoder_input = keras.Input(shape=(encoding_dim,))
    x = layers.Dense(7 * 7 * 64, activation='relu')(decoder_input)
    x = layers.Reshape((7, 7, 64))(x)
    x = layers.Conv2DTranspose(64, 3, activation='relu', padding='same')(x)
    x = layers.UpSampling2D(2)(x)
    x = layers.Conv2DTranspose(32, 3, activation='relu', padding='same')(x)
    x = layers.UpSampling2D(2)(x)
    decoded = layers.Conv2DTranspose(1, 3, activation='sigmoid', padding='same')(x)
    
    decoder = Model(decoder_input, decoded, name='decoder')
    
    # Autoencoder
    autoencoder_input = keras.Input(shape=input_shape)
    encoded_output = encoder(autoencoder_input)
    decoded_output = decoder(encoded_output)
    
    autoencoder = Model(autoencoder_input, decoded_output, name='autoencoder')
    
    autoencoder.compile(
        optimizer='adam',
        loss='binary_crossentropy',
        metrics=['accuracy']
    )
    
    return autoencoder, encoder, decoder

def build_variational_autoencoder(input_shape=(784,), latent_dim=2):
    """
    Build a Variational Autoencoder (VAE).
    
    Args:
        input_shape: Shape of input data
        latent_dim: Dimension of latent space
    
    Returns:
        VAE model, encoder model, decoder model
    """
    # Encoder
    encoder_input = keras.Input(shape=input_shape)
    x = layers.Dense(512, activation='relu')(encoder_input)
    x = layers.Dense(256, activation='relu')(x)
    
    z_mean = layers.Dense(latent_dim, name='z_mean')(x)
    z_log_var = layers.Dense(latent_dim, name='z_log_var')(x)
    
    def sampling(args):
        z_mean, z_log_var = args
        batch = tf.shape(z_mean)[0]
        dim = tf.shape(z_mean)[1]
        epsilon = tf.random.normal(shape=(batch, dim))
        return z_mean + tf.exp(0.5 * z_log_var) * epsilon
    
    z = layers.Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])
    
    encoder = Model(encoder_input, [z_mean, z_log_var, z], name='encoder')
    
    # Decoder
    decoder_input = keras.Input(shape=(latent_dim,))
    x = layers.Dense(256, activation='relu')(decoder_input)
    x = layers.Dense(512, activation='relu')(x)
    decoded = layers.Dense(input_shape[0], activation='sigmoid')(x)
    
    decoder = Model(decoder_input, decoded, name='decoder')
    
    # VAE
    vae_input = keras.Input(shape=input_shape)
    z_mean, z_log_var, z = encoder(vae_input)
    vae_output = decoder(z)
    
    # VAE loss
    reconstruction_loss = keras.losses.binary_crossentropy(vae_input, vae_output)
    reconstruction_loss *= input_shape[0]
    kl_loss = 1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var)
    kl_loss = tf.reduce_mean(kl_loss)
    kl_loss *= -0.5
    vae_loss = tf.reduce_mean(reconstruction_loss + kl_loss)
    
    vae = Model(vae_input, vae_output, name='vae')
    vae.add_loss(vae_loss)
    vae.compile(optimizer='adam')
    
    return vae, encoder, decoder

def visualize_reconstructions(autoencoder, test_data, num_samples=10):
    """
    Visualize original and reconstructed images.
    
    Args:
        autoencoder: Trained autoencoder model
        test_data: Test data
        num_samples: Number of samples to visualize
    """
    decoded_imgs = autoencoder.predict(test_data[:num_samples], verbose=0)
    
    n = num_samples
    plt.figure(figsize=(20, 4))
    for i in range(n):
        # Display original
        ax = plt.subplot(2, n, i + 1)
        if len(test_data[i].shape) == 1:
            img_size = int(np.sqrt(test_data[i].shape[0]))
            plt.imshow(test_data[i].reshape(img_size, img_size), cmap='gray')
        else:
            plt.imshow(test_data[i], cmap='gray')
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)
        
        # Display reconstruction
        ax = plt.subplot(2, n, i + 1 + n)
        if len(decoded_imgs[i].shape) == 1:
            img_size = int(np.sqrt(decoded_imgs[i].shape[0]))
            plt.imshow(decoded_imgs[i].reshape(img_size, img_size), cmap='gray')
        else:
            plt.imshow(decoded_imgs[i], cmap='gray')
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)
    
    plt.tight_layout()
    plt.show()

def example_usage():
    """
    Example usage of autoencoder functions.
    """
    # Load sample data
    (X_train, _), (X_test, _) = keras.datasets.mnist.load_data()
    
    # Preprocess data
    X_train = X_train.astype('float32') / 255.0
    X_test = X_test.astype('float32') / 255.0
    X_train = X_train.reshape((len(X_train), np.prod(X_train.shape[1:])))
    X_test = X_test.reshape((len(X_test), np.prod(X_test.shape[1:])))
    
    # Build simple autoencoder
    autoencoder, encoder, decoder = build_simple_autoencoder(
        input_shape=(784,), encoding_dim=32
    )
    
    print("Autoencoder Model:")
    autoencoder.summary()
    
    # Train autoencoder
    history = autoencoder.fit(
        X_train, X_train,
        epochs=10,
        batch_size=256,
        shuffle=True,
        validation_data=(X_test, X_test),
        verbose=1
    )
    
    # Visualize reconstructions
    visualize_reconstructions(autoencoder, X_test, num_samples=10)
    
    return autoencoder, encoder, decoder, history

if __name__ == '__main__':
    print("Autoencoders with TensorFlow")
    print("Author: RSK World - https://rskworld.in")
    autoencoder, encoder, decoder, history = example_usage()

243 lines•8 KB

python

Theme Settings

Color Scheme

Display Options

Font Size

CLEANUP_GUIDE.md

Theme Settings

Color Scheme

Display Options

Font Size

CLEANUP_GUIDE.md

🚀 Support RSK World