classification_X.npy CLEANUP_GUIDE.md autoencoders.py cnns.py

classification_X.npy

This file cannot be displayed in the browser.

CLEANUP_GUIDE.md

# Cleanup Guide for __pycache__ and Data Folders

**Author**: RSK World
**Website**: https://rskworld.in
**Email**: help@rskworld.in
**Phone**: +91 93305 39277

## Overview

This guide explains how to manage `__pycache__` folders and the `data` directory in the TensorFlow Deep Learning project.

## __pycache__ Folders

### What are __pycache__ folders?

Python automatically creates `__pycache__` directories to store compiled bytecode (`.pyc` files) for faster module loading. These are generated automatically and don't need to be committed to version control.

### Current Status

- ✅ `__pycache__/` is already in `.gitignore`
- ✅ All `.pyc`, `.pyo`, `.pyd` files are ignored
- ✅ No `__pycache__` folders currently exist in the project

### Cleaning __pycache__ Folders

#### Method 1: Using Python Script (Cross-platform)
```bash
python scripts/clean_cache.py
```

#### Method 2: Using Shell Script (Linux/Mac)
```bash
bash scripts/cleanup.sh
```

#### Method 3: Using Batch Script (Windows)
```cmd
scripts\cleanup.bat
```

#### Method 4: Manual Cleanup

**Windows (PowerShell):**
```powershell
Get-ChildItem -Path . -Filter "__pycache__" -Recurse -Directory | Remove-Item -Recurse -Force
Get-ChildItem -Path . -Filter "*.pyc" -Recurse -File | Remove-Item -Force
```

**Linux/Mac:**
```bash
find . -type d -name "__pycache__" -exec rm -r {} +
find . -type f -name "*.pyc" -delete
```

**Python:**
```python
import os
import shutil

for root, dirs, files in os.walk('.'):
if '__pycache__' in dirs:
shutil.rmtree(os.path.join(root, '__pycache__'))
for file in files:
if file.endswith('.pyc'):
os.remove(os.path.join(root, file))
```

## Data Folder

### Current Structure

The `data/` folder contains:
- Generated datasets (`.npy`, `.csv` files)
- Metadata files (`.json`)
- Visualizations (if generated)
- README.md documentation

### Git Configuration

The `.gitignore` is configured to:
- ✅ Ignore generated data files (`*.npy`, `*.csv`, `*.json`)
- ✅ Keep directory structure (`.gitkeep` file)
- ✅ Keep documentation (`README.md`)

### Data Folder Contents

```
data/
├── .gitkeep # Keeps folder in git
├── README.md # Documentation (tracked)
├── classification_X.npy # Generated (ignored)
├── classification_y.npy # Generated (ignored)
├── regression_X.npy # Generated (ignored)
├── images_X.npy # Generated (ignored)
├── sequences_X.npy # Generated (ignored)
├── tabular.csv # Generated (ignored)
└── *_metadata.json # Generated (ignored)
```

### Regenerating Data

If you need to regenerate all data:
```bash
python scripts/generate_data_standalone.py
```

### Cleaning Data Folder

To remove all generated data (but keep structure):
```bash
# Remove all .npy files
find data -name "*.npy" -delete

# Remove all .csv files
find data -name "*.csv" -delete

# Remove all .json metadata files
find data -name "*_metadata.json" -delete
```

Or use Python:
```python
import os
import glob

data_dir = './data'
for pattern in ['*.npy', '*.csv', '*_metadata.json']:
for file in glob.glob(os.path.join(data_dir, pattern)):
os.remove(file)
print(f"Removed: {file}")
```

## Best Practices

### 1. Before Committing
- Run cleanup script to remove `__pycache__` folders
- Verify `.gitignore` is working correctly
- Don't commit generated data files

### 2. Regular Maintenance
- Clean `__pycache__` folders periodically
- Regenerate data if needed for testing
- Keep data directory structure intact

### 3. CI/CD Integration
Add cleanup to your CI/CD pipeline:
```yaml
# Example GitHub Actions
- name: Clean __pycache__
run: python scripts/clean_cache.py
```

## Troubleshooting

### Issue: __pycache__ folders keep appearing
**Solution**: This is normal! Python creates them automatically. Just run the cleanup script before committing.

### Issue: Data files are being tracked by git
**Solution**: Check `.gitignore` and ensure patterns are correct. Remove tracked files:
```bash
git rm --cached data/*.npy
git rm --cached data/*.csv
```

### Issue: Can't delete __pycache__ folders
**Solution**: Make sure no Python processes are using them. Close IDEs and Python interpreters, then try again.

## Scripts Available

1. **scripts/clean_cache.py** - Python cleanup script (recommended)
2. **scripts/cleanup.sh** - Shell script for Linux/Mac
3. **scripts/cleanup.bat** - Batch script for Windows

## Summary

- ✅ `__pycache__` folders are ignored by git
- ✅ Generated data files are ignored by git
- ✅ Directory structure is preserved
- ✅ Cleanup scripts are available
- ✅ Documentation is tracked

All cleanup operations are safe and won't affect your source code or project structure.

src/autoencoders.py

Raw Download

"""
Autoencoders with TensorFlow
Author: RSK World
Website: https://rskworld.in
Email: help@rskworld.in
Phone: +91 93305 39277

This module demonstrates various autoencoder architectures.
"""

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers, Model
import numpy as np
import matplotlib.pyplot as plt

def build_simple_autoencoder(input_shape=(784,), encoding_dim=32):
    """
    Build a simple autoencoder.
    
    Args:
        input_shape: Shape of input data
        encoding_dim: Dimension of encoding layer
    
    Returns:
        Autoencoder model, encoder model, decoder model
    """
    # Encoder
    encoder_input = keras.Input(shape=input_shape)
    encoded = layers.Dense(128, activation='relu')(encoder_input)
    encoded = layers.Dense(64, activation='relu')(encoded)
    encoded = layers.Dense(encoding_dim, activation='relu')(encoded)
    
    encoder = Model(encoder_input, encoded, name='encoder')
    
    # Decoder
    decoder_input = keras.Input(shape=(encoding_dim,))
    decoded = layers.Dense(64, activation='relu')(decoder_input)
    decoded = layers.Dense(128, activation='relu')(decoded)
    decoded = layers.Dense(input_shape[0], activation='sigmoid')(decoded)
    
    decoder = Model(decoder_input, decoded, name='decoder')
    
    # Autoencoder
    autoencoder_input = keras.Input(shape=input_shape)
    encoded_output = encoder(autoencoder_input)
    decoded_output = decoder(encoded_output)
    
    autoencoder = Model(autoencoder_input, decoded_output, name='autoencoder')
    
    autoencoder.compile(
        optimizer='adam',
        loss='binary_crossentropy',
        metrics=['accuracy']
    )
    
    return autoencoder, encoder, decoder

def build_convolutional_autoencoder(input_shape=(28, 28, 1), encoding_dim=32):
    """
    Build a convolutional autoencoder.
    
    Args:
        input_shape: Shape of input images
        encoding_dim: Dimension of encoding layer
    
    Returns:
        Autoencoder model, encoder model, decoder model
    """
    # Encoder
    encoder_input = keras.Input(shape=input_shape)
    x = layers.Conv2D(32, 3, activation='relu', padding='same')(encoder_input)
    x = layers.MaxPooling2D(2, padding='same')(x)
    x = layers.Conv2D(64, 3, activation='relu', padding='same')(x)
    x = layers.MaxPooling2D(2, padding='same')(x)
    x = layers.Conv2D(64, 3, activation='relu', padding='same')(x)
    x = layers.Flatten()(x)
    encoded = layers.Dense(encoding_dim, activation='relu')(x)
    
    encoder = Model(encoder_input, encoded, name='encoder')
    
    # Decoder
    decoder_input = keras.Input(shape=(encoding_dim,))
    x = layers.Dense(7 * 7 * 64, activation='relu')(decoder_input)
    x = layers.Reshape((7, 7, 64))(x)
    x = layers.Conv2DTranspose(64, 3, activation='relu', padding='same')(x)
    x = layers.UpSampling2D(2)(x)
    x = layers.Conv2DTranspose(32, 3, activation='relu', padding='same')(x)
    x = layers.UpSampling2D(2)(x)
    decoded = layers.Conv2DTranspose(1, 3, activation='sigmoid', padding='same')(x)
    
    decoder = Model(decoder_input, decoded, name='decoder')
    
    # Autoencoder
    autoencoder_input = keras.Input(shape=input_shape)
    encoded_output = encoder(autoencoder_input)
    decoded_output = decoder(encoded_output)
    
    autoencoder = Model(autoencoder_input, decoded_output, name='autoencoder')
    
    autoencoder.compile(
        optimizer='adam',
        loss='binary_crossentropy',
        metrics=['accuracy']
    )
    
    return autoencoder, encoder, decoder

def build_variational_autoencoder(input_shape=(784,), latent_dim=2):
    """
    Build a Variational Autoencoder (VAE).
    
    Args:
        input_shape: Shape of input data
        latent_dim: Dimension of latent space
    
    Returns:
        VAE model, encoder model, decoder model
    """
    # Encoder
    encoder_input = keras.Input(shape=input_shape)
    x = layers.Dense(512, activation='relu')(encoder_input)
    x = layers.Dense(256, activation='relu')(x)
    
    z_mean = layers.Dense(latent_dim, name='z_mean')(x)
    z_log_var = layers.Dense(latent_dim, name='z_log_var')(x)
    
    def sampling(args):
        z_mean, z_log_var = args
        batch = tf.shape(z_mean)[0]
        dim = tf.shape(z_mean)[1]
        epsilon = tf.random.normal(shape=(batch, dim))
        return z_mean + tf.exp(0.5 * z_log_var) * epsilon
    
    z = layers.Lambda(sampling, output_shape=(latent_dim,))([z_mean, z_log_var])
    
    encoder = Model(encoder_input, [z_mean, z_log_var, z], name='encoder')
    
    # Decoder
    decoder_input = keras.Input(shape=(latent_dim,))
    x = layers.Dense(256, activation='relu')(decoder_input)
    x = layers.Dense(512, activation='relu')(x)
    decoded = layers.Dense(input_shape[0], activation='sigmoid')(x)
    
    decoder = Model(decoder_input, decoded, name='decoder')
    
    # VAE
    vae_input = keras.Input(shape=input_shape)
    z_mean, z_log_var, z = encoder(vae_input)
    vae_output = decoder(z)
    
    # VAE loss
    reconstruction_loss = keras.losses.binary_crossentropy(vae_input, vae_output)
    reconstruction_loss *= input_shape[0]
    kl_loss = 1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var)
    kl_loss = tf.reduce_mean(kl_loss)
    kl_loss *= -0.5
    vae_loss = tf.reduce_mean(reconstruction_loss + kl_loss)
    
    vae = Model(vae_input, vae_output, name='vae')
    vae.add_loss(vae_loss)
    vae.compile(optimizer='adam')
    
    return vae, encoder, decoder

def visualize_reconstructions(autoencoder, test_data, num_samples=10):
    """
    Visualize original and reconstructed images.
    
    Args:
        autoencoder: Trained autoencoder model
        test_data: Test data
        num_samples: Number of samples to visualize
    """
    decoded_imgs = autoencoder.predict(test_data[:num_samples], verbose=0)
    
    n = num_samples
    plt.figure(figsize=(20, 4))
    for i in range(n):
        # Display original
        ax = plt.subplot(2, n, i + 1)
        if len(test_data[i].shape) == 1:
            img_size = int(np.sqrt(test_data[i].shape[0]))
            plt.imshow(test_data[i].reshape(img_size, img_size), cmap='gray')
        else:
            plt.imshow(test_data[i], cmap='gray')
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)
        
        # Display reconstruction
        ax = plt.subplot(2, n, i + 1 + n)
        if len(decoded_imgs[i].shape) == 1:
            img_size = int(np.sqrt(decoded_imgs[i].shape[0]))
            plt.imshow(decoded_imgs[i].reshape(img_size, img_size), cmap='gray')
        else:
            plt.imshow(decoded_imgs[i], cmap='gray')
        ax.get_xaxis().set_visible(False)
        ax.get_yaxis().set_visible(False)
    
    plt.tight_layout()
    plt.show()

def example_usage():
    """
    Example usage of autoencoder functions.
    """
    # Load sample data
    (X_train, _), (X_test, _) = keras.datasets.mnist.load_data()
    
    # Preprocess data
    X_train = X_train.astype('float32') / 255.0
    X_test = X_test.astype('float32') / 255.0
    X_train = X_train.reshape((len(X_train), np.prod(X_train.shape[1:])))
    X_test = X_test.reshape((len(X_test), np.prod(X_test.shape[1:])))
    
    # Build simple autoencoder
    autoencoder, encoder, decoder = build_simple_autoencoder(
        input_shape=(784,), encoding_dim=32
    )
    
    print("Autoencoder Model:")
    autoencoder.summary()
    
    # Train autoencoder
    history = autoencoder.fit(
        X_train, X_train,
        epochs=10,
        batch_size=256,
        shuffle=True,
        validation_data=(X_test, X_test),
        verbose=1
    )
    
    # Visualize reconstructions
    visualize_reconstructions(autoencoder, X_test, num_samples=10)
    
    return autoencoder, encoder, decoder, history

if __name__ == '__main__':
    print("Autoencoders with TensorFlow")
    print("Author: RSK World - https://rskworld.in")
    autoencoder, encoder, decoder, history = example_usage()

243 lines•8 KB

python

src/cnns.py

Raw Download

"""
Convolutional Neural Networks (CNNs) with TensorFlow
Author: RSK World
Website: https://rskworld.in
Email: help@rskworld.in
Phone: +91 93305 39277

This module demonstrates CNN construction for image classification.
"""

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import numpy as np
import matplotlib.pyplot as plt

def create_simple_cnn(input_shape, num_classes):
    """
    Create a simple CNN for image classification.
    
    Args:
        input_shape: Shape of input images (height, width, channels)
        num_classes: Number of output classes
    
    Returns:
        Compiled Keras model
    """
    model = keras.Sequential([
        layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape),
        layers.MaxPooling2D((2, 2)),
        layers.Conv2D(64, (3, 3), activation='relu'),
        layers.MaxPooling2D((2, 2)),
        layers.Conv2D(64, (3, 3), activation='relu'),
        layers.Flatten(),
        layers.Dense(64, activation='relu'),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation='softmax')
    ])
    
    model.compile(
        optimizer='adam',
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )
    
    return model

def create_deep_cnn(input_shape, num_classes):
    """
    Create a deeper CNN with batch normalization and dropout.
    
    Args:
        input_shape: Shape of input images
        num_classes: Number of output classes
    
    Returns:
        Compiled Keras model
    """
    model = keras.Sequential([
        # First block
        layers.Conv2D(32, (3, 3), activation='relu', input_shape=input_shape),
        layers.BatchNormalization(),
        layers.Conv2D(32, (3, 3), activation='relu'),
        layers.MaxPooling2D((2, 2)),
        layers.Dropout(0.25),
        
        # Second block
        layers.Conv2D(64, (3, 3), activation='relu'),
        layers.BatchNormalization(),
        layers.Conv2D(64, (3, 3), activation='relu'),
        layers.MaxPooling2D((2, 2)),
        layers.Dropout(0.25),
        
        # Third block
        layers.Conv2D(128, (3, 3), activation='relu'),
        layers.BatchNormalization(),
        layers.MaxPooling2D((2, 2)),
        layers.Dropout(0.25),
        
        # Dense layers
        layers.Flatten(),
        layers.Dense(512, activation='relu'),
        layers.BatchNormalization(),
        layers.Dropout(0.5),
        layers.Dense(num_classes, activation='softmax')
    ])
    
    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=0.001),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )
    
    return model

def create_resnet_block(x, filters, kernel_size=3, stride=1):
    """
    Create a ResNet-style residual block.
    
    Args:
        x: Input tensor
        filters: Number of filters
        kernel_size: Size of convolution kernel
        stride: Stride of convolution
    
    Returns:
        Output tensor
    """
    shortcut = x
    
    # Main path
    x = layers.Conv2D(filters, kernel_size, strides=stride, padding='same')(x)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    
    x = layers.Conv2D(filters, kernel_size, padding='same')(x)
    x = layers.BatchNormalization()(x)
    
    # Shortcut connection
    if stride != 1 or shortcut.shape[-1] != filters:
        shortcut = layers.Conv2D(filters, 1, strides=stride, padding='same')(shortcut)
        shortcut = layers.BatchNormalization()(shortcut)
    
    x = layers.Add()([x, shortcut])
    x = layers.Activation('relu')(x)
    
    return x

def create_resnet_cnn(input_shape, num_classes):
    """
    Create a ResNet-inspired CNN using functional API.
    
    Args:
        input_shape: Shape of input images
        num_classes: Number of output classes
    
    Returns:
        Compiled Keras model
    """
    inputs = keras.Input(shape=input_shape)
    
    # Initial convolution
    x = layers.Conv2D(64, 7, strides=2, padding='same')(inputs)
    x = layers.BatchNormalization()(x)
    x = layers.Activation('relu')(x)
    x = layers.MaxPooling2D(3, strides=2, padding='same')(x)
    
    # Residual blocks
    x = create_resnet_block(x, 64)
    x = create_resnet_block(x, 64)
    x = create_resnet_block(x, 128, stride=2)
    x = create_resnet_block(x, 128)
    
    # Global average pooling and output
    x = layers.GlobalAveragePooling2D()(x)
    x = layers.Dense(256, activation='relu')(x)
    x = layers.Dropout(0.5)(x)
    outputs = layers.Dense(num_classes, activation='softmax')(x)
    
    model = keras.Model(inputs, outputs)
    
    model.compile(
        optimizer=keras.optimizers.Adam(learning_rate=0.001),
        loss='sparse_categorical_crossentropy',
        metrics=['accuracy']
    )
    
    return model

def visualize_cnn_layers(model, sample_image):
    """
    Visualize CNN layer activations.
    
    Args:
        model: Trained CNN model
        sample_image: Sample image to visualize
    """
    layer_outputs = [layer.output for layer in model.layers[:8]]
    activation_model = keras.Model(inputs=model.input, outputs=layer_outputs)
    
    activations = activation_model.predict(sample_image.reshape(1, *sample_image.shape))
    
    fig, axes = plt.subplots(2, 4, figsize=(15, 7))
    axes = axes.ravel()
    
    for i, activation in enumerate(activations):
        if len(activation.shape) == 4:  # Convolutional layer
            # Take first filter
            axes[i].imshow(activation[0, :, :, 0], cmap='viridis')
            axes[i].set_title(f'Layer {i+1}')
            axes[i].axis('off')
    
    plt.tight_layout()
    plt.show()

def example_usage():
    """
    Example usage of CNN functions.
    """
    # Load CIFAR-10 dataset
    (X_train, y_train), (X_test, y_test) = keras.datasets.cifar10.load_data()
    
    # Preprocess data
    X_train = X_train.astype('float32') / 255.0
    X_test = X_test.astype('float32') / 255.0
    
    # Create model
    model = create_simple_cnn(input_shape=(32, 32, 3), num_classes=10)
    
    # Display model architecture
    model.summary()
    
    # Train model
    history = model.fit(
        X_train, y_train,
        batch_size=128,
        epochs=10,
        validation_data=(X_test, y_test),
        verbose=1
    )
    
    # Evaluate model
    test_loss, test_accuracy = model.evaluate(X_test, y_test, verbose=0)
    print(f'\nTest Accuracy: {test_accuracy:.4f}')
    
    return model, history

if __name__ == '__main__':
    print("Convolutional Neural Networks with TensorFlow")
    print("Author: RSK World - https://rskworld.in")
    model, history = example_usage()

232 lines•6.7 KB

python

Theme Settings

Color Scheme

Display Options

Font Size

CLEANUP_GUIDE.md

Theme Settings

Color Scheme

Display Options

Font Size

CLEANUP_GUIDE.md

🚀 Support RSK World