Ethiraj Srinivasan

• Co-founded Griffin AI Tech; lead product engineering for InfiniTraq, an Edge AI platform for senior-care monitoring.

• Consulting engagement leading Data Engineering — mentoring a high-performing team and implementing governance, best practices, and process optimizations to deliver scalable, reliable, and high-quality data platforms.
• Architected an end-to-end Spark-based ML pipeline on AWS EMR for logistics anomaly and weight prediction, integrating a distributed ML model and automated feature engineering, achieving 99.9% uptime.

Batch Ingestion Team Leadership

• Led the batch ingestion team managing 23k MySQL-to-Hive pipelines, 6k Hive-to-Hive pipelines, and 2k Hive-to-HBase and HBase-to-Hive pipelines with daily and hourly frequencies.
• Ensured seamless operations on a 14 PB data lake, achieving an outstanding 99.95% job success rate for 31k ingestion pipelines.

Performance Optimization & Resource Utilization

• Maintained 80% memory utilization, processing 1,059,221 GB/hour and 245,414 vCore/hour, with 97% of jobs completing within 30 minutes.
• Optimized cross-IDC Hive-to-Hive pipelines, saving approximately 94,325 GB/hour, 13,487 vCore/hour, and 500 TB of storage.

Platform Upgrade & Cost Optimization

• Upgraded the ingestion platform to Spark 3, resulting in a 9% reduction in processing time and a 14% reduction in resource usage.

Pipeline Development & Monitoring

• Designed and monitored batch, real-time, and cross-country ingestion pipelines, including hourly Hive-to-Hive jobs with filters, incremental dumps, and Data Quality Checks (DQC).

Data Pipeline & Metrics Management

• Designed metrics for job performance and resource utilization, and managed ingestion pipelines handling diverse sources (MySQL, CSV, Kafka) and sinks (HDFS, Hive, Hudi, Kafka) using Spark, DeltaStreamer, Spring Boot, and Maxwell.

Ingestion Applications Development

• Built and maintained key applications: Core Engine (job automation), AutoAdaptation Framework (dynamic optimization), Operations Platform, Data Quality Module, Configuration Service, and Notification Service.

• Data Infrastructure Development — Led the design of a scalable data infrastructure platform, including a data lake using S3, Kinesis, Lambda, and EMR (Spark), and implemented an automated recommendation engine to enhance feeds and follower engagement.
• Big Data Processing & Analytics — Built and monitored PySpark jobs on EMR for diverse business use cases, developed advanced metrics for ranking videos, and conducted graph analysis to identify communities and influencers.
• Real-Time Visualization — Created interactive QuickSight dashboards with Athena as the query engine, providing actionable insights for stakeholders and enabling data-driven decisions.

• Migrated an ad events tracker to Vert.x 3, achieving 50K QPS with optimized throughput using Kinesis and Lambda architecture; built a data lake on S3 integrated with real-time sources, developed PySpark jobs on EMR for analytics.
• Created a reporting tool with Spark (Scala) for attributions, and designed QuickSight dashboards using Athena to deliver actionable insights through visualization and advanced metrics.

• Developed a web application for managing patents with a robust role-based content management system for content manipulation, authorization, and access; additionally, organized and facilitated Open Source meetups to foster community engagement and collaboration.

• Headed a team of two members to develop a collaborative platform for sharing content, ideas, and processes across the Tata group of companies.