econometric_data.csv README.md lazy_evaluation.py advanced_queries.py

README.md

# Polars Fast DataFrames



High-performance DataFrame library with Polars for fast data processing, querying, and analysis on large datasets.

## Description

This project demonstrates Polars, a blazingly fast DataFrame library written in Rust. It covers DataFrame operations, lazy evaluation, query optimization, and performance comparisons. Perfect for high-performance data processing and analysis.

## Features

- ⚡ Fast DataFrame operations
- 🔄 Lazy evaluation and optimization
- 💾 Memory-efficient processing
- 🎯 Query optimization
- 🔗 Pandas compatibility
- 📊 Time series operations
- 🔍 Advanced string operations
- 🔗 Multiple join types (Inner, Left, Right, Outer, Anti, Semi)
- 📈 Advanced aggregations
- 🧹 Missing data handling
- 🔄 Data reshaping (Melt, Pivot)
- ✅ Data validation and quality checks
- 📦 Nested data structures (Struct, List)
- 🚀 Performance optimization techniques
- 💼 Real-world analytics examples

## Technologies

- Python
- Polars
- Pandas
- Jupyter Notebook

## Installation

```bash
pip install -r requirements.txt
```

## Project Structure

```
polars-fastdataframes/
├── README.md
├── requirements.txt
├── notebooks/
│ ├── 01_basic_operations.ipynb
│ ├── 02_lazy_evaluation.ipynb
│ ├── 03_performance_comparison.ipynb
│ └── 04_advanced_queries.ipynb
├── scripts/
│ ├── basic_operations.py
│ ├── lazy_evaluation.py
│ ├── performance_comparison.py
│ ├── advanced_queries.py
│ └── data_generator.py
└── data/
└── sample_data.csv
```

## Usage

### Jupyter Notebooks

Launch Jupyter Notebook and open any notebook from the `notebooks/` directory:

```bash
jupyter notebook
```

**Notebooks included:**
- `01_basic_operations.ipynb` - Basic DataFrame operations
- `02_lazy_evaluation.ipynb` - Lazy evaluation and query optimization
- `03_performance_comparison.ipynb` - Polars vs Pandas benchmarks
- `04_advanced_queries.ipynb` - **20+ advanced features** including:
- Time series operations
- Missing data handling
- Advanced string operations
- All join types
- Data validation
- Nested data structures
- Real-world analytics
- Performance optimization

### Python Scripts

Run any script from the `scripts/` directory:

```bash
# Basic operations
python scripts/basic_operations.py

# Lazy evaluation
python scripts/lazy_evaluation.py

# Performance comparison
python scripts/performance_comparison.py

# Advanced queries (NEW!)
python scripts/advanced_queries.py

# Generate sample data
python scripts/data_generator.py
```

## Performance

Polars is designed for speed and efficiency. It uses:
- Apache Arrow columnar memory format
- Query optimization through lazy evaluation
- Parallel processing capabilities
- Zero-copy reads

## License

This project is provided as educational material by RSK World.

## Contact

- Website: https://rskworld.in
- Email: help@rskworld.in
- Phone: +91 93305 39277

scripts/lazy_evaluation.py

Raw Download

"""
Lazy Evaluation and Query Optimization in Polars
Demonstrates lazy evaluation, query planning, and optimization techniques

Author: RSK World
Website: https://rskworld.in
Email: help@rskworld.in
Phone: +91 93305 39277
"""

import polars as pl
import numpy as np
import os
from datetime import datetime, timedelta

def demonstrate_lazy_evaluation():
    """Demonstrate lazy evaluation and query optimization"""
    
    print("=" * 60)
    print("LAZY EVALUATION AND QUERY OPTIMIZATION")
    print("=" * 60)
    
    # Create a sample DataFrame
    print("\n1. Creating a sample DataFrame:")
    df = pl.DataFrame({
        'id': range(1, 10001),
        'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], 10000),
        'value1': np.random.randn(10000) * 100,
        'value2': np.random.randn(10000) * 50,
        'value3': np.random.randint(1, 1000, 10000)
    })
    print(f"DataFrame shape: {df.shape}")
    
    # Convert to LazyFrame
    print("\n2. Converting to LazyFrame:")
    lazy_df = df.lazy()
    print(f"Type: {type(lazy_df)}")
    print("Operations are not executed until .collect() is called")
    
    # Build a complex query
    print("\n3. Building a complex query (not executed yet):")
    query = (lazy_df
        .filter(pl.col('value1') > 50)
        .filter(pl.col('value2') < 20)
        .select(['id', 'category', 'value1', 'value2'])
        .group_by('category')
        .agg([
            pl.col('value1').mean().alias('avg_value1'),
            pl.col('value2').mean().alias('avg_value2'),
            pl.count().alias('count')
        ])
        .sort('avg_value1', descending=True)
    )
    print("Query built successfully!")
    
    # Show query plan
    print("\n4. Query Plan (optimized):")
    print(query.explain())
    
    # Execute the query
    print("\n5. Executing the query:")
    result = query.collect()
    print("Query executed!")
    print(result)
    
    # Demonstrate query optimization benefits
    print("\n6. Query Optimization Benefits:")
    print("- Predicate pushdown: Filters are applied early")
    print("- Projection pushdown: Only needed columns are selected")
    print("- Predicate combination: Multiple filters are combined")
    print("- Join reordering: Joins are optimized for performance")
    
    # Lazy CSV reading
    print("\n7. Lazy CSV Reading (more efficient for large files):")
    try:
        script_dir = os.path.dirname(os.path.abspath(__file__))
        project_root = os.path.dirname(script_dir)
        csv_path = os.path.join(project_root, 'data', 'sample_data.csv')
        lazy_from_csv = pl.scan_csv(csv_path)
        print("LazyFrame from CSV created")
        print("\nQuery plan for CSV operation:")
        csv_query = lazy_from_csv.filter(pl.col('price') > 100).select(['name', 'price'])
        print(csv_query.explain())
        print("\nExecuting CSV query:")
        csv_result = csv_query.collect()
        print(csv_result.head())
    except FileNotFoundError:
        print("Sample data file not found. Run data_generator.py first.")
    
    print("\n" + "=" * 60)
    print("Lazy evaluation demonstration complete!")
    print("=" * 60)

if __name__ == "__main__":
    demonstrate_lazy_evaluation()

97 lines•3.2 KB

python

scripts/advanced_queries.py

Raw Download

"""
Advanced Queries with Polars
Comprehensive examples of advanced Polars operations and patterns

Author: RSK World
Website: https://rskworld.in
Email: help@rskworld.in
Phone: +91 93305 39277
"""

import polars as pl
import numpy as np
from datetime import datetime, timedelta
import time

def demonstrate_time_series():
    """Demonstrate time series operations"""
    print("=" * 60)
    print("TIME SERIES OPERATIONS")
    print("=" * 60)
    
    dates = pl.date_range(datetime(2023, 1, 1), datetime(2023, 12, 31), '1d', eager=True)
    ts_data = pl.DataFrame({
        'date': dates,
        'value': np.random.randn(len(dates)).cumsum() * 100 + 1000,
        'category': np.random.choice(['A', 'B', 'C'], len(dates))
    })
    
    ts_ops = ts_data.with_columns([
        pl.col('value').shift(1).over('category').alias('lag_1'),
        pl.col('value').diff().over('category').alias('diff'),
        pl.col('value').pct_change().over('category').alias('pct_change'),
        pl.col('value').rolling_mean(window_size=7).over('category').alias('rolling_mean_7d')
    ])
    
    print("\nTime series operations sample:")
    print(ts_ops.head(10))

def demonstrate_missing_data():
    """Demonstrate missing data handling"""
    print("\n" + "=" * 60)
    print("MISSING DATA HANDLING")
    print("=" * 60)
    
    data_with_nulls = pl.DataFrame({
        'id': range(1, 11),
        'name': ['Alice', None, 'Charlie', 'David', None, 'Frank', 'Grace', None, 'Ivy', 'Jack'],
        'age': [25, 30, None, 28, 32, None, 29, 31, None, 33],
        'salary': [50000, None, 70000, 55000, 65000, None, 58000, 62000, 51000, None]
    })
    
    print("\nOriginal data:")
    print(data_with_nulls)
    print(f"\nNull counts: {data_with_nulls.null_count()}")
    
    filled = data_with_nulls.with_columns([
        pl.col('name').fill_null('Unknown'),
        pl.col('age').fill_null(pl.col('age').mean()),
        pl.col('salary').fill_null(strategy='forward')
    ])
    
    print("\nAfter filling nulls:")
    print(filled)

def demonstrate_advanced_strings():
    """Demonstrate advanced string operations"""
    print("\n" + "=" * 60)
    print("ADVANCED STRING OPERATIONS")
    print("=" * 60)
    
    text_data = pl.DataFrame({
        'id': range(1, 6),
        'text': ['Hello World', 'Python Programming', 'Data Science', 'Machine Learning', 'Deep Learning'],
        'email': ['user1@example.com', 'user2@test.org', 'admin@company.com', 'info@website.net', 'contact@business.io']
    })
    
    string_ops = text_data.with_columns([
        pl.col('text').str.to_uppercase().alias('upper'),
        pl.col('text').str.len_chars().alias('char_count'),
        pl.col('email').str.extract(r'@(\w+)', 1).alias('domain'),
        pl.col('text').str.contains('Learning').alias('has_learning')
    ])
    
    print("\nString operations:")
    print(string_ops)

def demonstrate_joins():
    """Demonstrate all join types"""
    print("\n" + "=" * 60)
    print("ADVANCED JOIN TYPES")
    print("=" * 60)
    
    customers = pl.DataFrame({
        'customer_id': [1, 2, 3, 4, 5],
        'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve']
    })
    
    orders = pl.DataFrame({
        'order_id': [101, 102, 103, 104, 105, 106],
        'customer_id': [1, 2, 1, 4, 3, 99],
        'amount': [100, 200, 150, 300, 250, 400]
    })
    
    print("\nInner Join:")
    print(customers.join(orders, on='customer_id', how='inner'))
    
    print("\nLeft Join:")
    print(customers.join(orders, on='customer_id', how='left').head())
    
    print("\nAnti Join (customers with no orders):")
    print(customers.join(orders, on='customer_id', how='anti'))

def demonstrate_performance():
    """Demonstrate performance optimization"""
    print("\n" + "=" * 60)
    print("PERFORMANCE OPTIMIZATION")
    print("=" * 60)
    
    large_data = pl.DataFrame({
        'id': range(1, 100001),
        'category': np.random.choice(['A', 'B', 'C', 'D', 'E'], 100000),
        'value1': np.random.randn(100000) * 100,
        'value2': np.random.randn(100000) * 50
    })
    
    print(f"\nDataset shape: {large_data.shape}")
    
    # Eager vs Lazy
    start = time.time()
    eager_result = large_data.filter(pl.col('value1') > 50).group_by('category').agg([pl.col('value1').mean()])
    eager_time = time.time() - start
    
    start = time.time()
    lazy_result = (large_data.lazy()
        .filter(pl.col('value1') > 50)
        .group_by('category')
        .agg([pl.col('value1').mean()])
        .collect()
    )
    lazy_time = time.time() - start
    
    print(f"\nEager execution: {eager_time:.4f} seconds")
    print(f"Lazy execution: {lazy_time:.4f} seconds")
    print(f"Speedup: {eager_time / lazy_time:.2f}x")

def demonstrate_ecommerce_analytics():
    """Real-world e-commerce analytics example"""
    print("\n" + "=" * 60)
    print("E-COMMERCE ANALYTICS EXAMPLE")
    print("=" * 60)
    
    np.random.seed(42)
    dates = pl.date_range(datetime(2023, 1, 1), datetime(2023, 3, 31), '1d', eager=True)
    
    ecommerce = pl.DataFrame({
        'order_id': range(1, 501),
        'date': np.random.choice(dates, 500),
        'product': np.random.choice(['Laptop', 'Phone', 'Tablet'], 500),
        'region': np.random.choice(['North', 'South', 'East', 'West'], 500),
        'quantity': np.random.randint(1, 10, 500),
        'unit_price': np.random.uniform(100, 1000, 500),
        'discount': np.random.uniform(0, 0.3, 500)
    })
    
    ecommerce = ecommerce.with_columns([
        (pl.col('unit_price') * pl.col('quantity') * (1 - pl.col('discount'))).alias('total')
    ])
    
    # Analytics
    analytics = (ecommerce
        .group_by(['product', 'region'])
        .agg([
            pl.col('total').sum().alias('revenue'),
            pl.col('order_id').n_unique().alias('num_orders'),
            pl.col('quantity').sum().alias('total_sold')
        ])
        .sort('revenue', descending=True)
    )
    
    print("\nProduct and Region Analytics:")
    print(analytics)

def run_all_demonstrations():
    """Run all advanced query demonstrations"""
    print("\n" + "=" * 60)
    print("ADVANCED POLARS QUERIES - COMPREHENSIVE DEMONSTRATION")
    print("=" * 60)
    
    demonstrate_time_series()
    demonstrate_missing_data()
    demonstrate_advanced_strings()
    demonstrate_joins()
    demonstrate_performance()
    demonstrate_ecommerce_analytics()
    
    print("\n" + "=" * 60)
    print("All demonstrations complete!")
    print("=" * 60)
    print("\nKey Takeaways:")
    print("- Polars provides powerful operations for time series data")
    print("- Missing data handling is straightforward and flexible")
    print("- String operations are comprehensive and efficient")
    print("- Multiple join types available for different use cases")
    print("- Lazy evaluation provides significant performance benefits")
    print("- Real-world analytics can be done efficiently with Polars")

if __name__ == "__main__":
    run_all_demonstrations()

210 lines•7 KB

python

Theme Settings

Color Scheme

Display Options

Font Size

README.md

Theme Settings

Color Scheme

Display Options

Font Size

README.md

🚀 Support RSK World