As a data engineer, I specialize in designing, developing, and maintaining the infrastructure and systems required to collect, store, process, and analyze large volumes of data. I work closely with data scientists, analysts, and other stakeholders to ensure that data is readily available, accurate, and easily accessible for insights and decision-making.

My expertise lies in various aspects of the data lifecycle, including data ingestion, data transformation, data modeling, and data integration. I am skilled in programming languages such as Python, SQL, and Java, as well as technologies like Apache Hadoop, Spark, and relational and NoSQL databases.

I am experienced in building scalable and robust data pipelines to extract data from various sources, transform it into a suitable format, and load it into data warehouses or data lakes. I also implement data governance practices, ensuring data quality, data security, and compliance with regulatory standards.

In addition, I have a solid understanding of data modeling techniques, both relational and dimensional, to design efficient and optimized databases for analytics and reporting purposes. I work with tools such as ETL (Extract, Transform, Load) frameworks, data integration platforms, and workflow orchestration tools to automate and streamline data processes.

Overall, I am passionate about leveraging data engineering principles and technologies to empower organizations with reliable, accessible, and actionable data for making informed decisions and driving business growth.

I have done my masters in computer science with Data Science specialization

Missouri State University

Work with Data science team to process ETL and provide required data which can be consumed by forecast models to generate future forecast of the inventory required for individual stores.
• Create ETL pipeline to consume forecast output tables and data from different other sources like TeraData, RDBMS, Hive etc to transform and generate analytical output tables which can be leveraged by API to show reports and graphs to UI. Also process adjustment done at UI (override forecast values) in near real time and override the same in output tables (Hive & GCS).
• Build Data Lake by creating various ETL pipeline to extract data from various source, standardize, normalize and make the data production ready.
• Used Jenkins and pipelines which helped us drive all Microservices builds out to the Docker registry and then deployed to Kubernetes.
• Designed infrastructure for AWS application and workflow using Terraform and had done implementation and continuous delivery of AWS infrastructure using Terraform.
• Managed Docker orchestration and Docker containerization using Kubernetes
• Automated builds using Maven and scheduled automated nightly builds using Jenkins. Built Jenkins pipeline to drive all microservices builds out to the Docker registry and then deployed to Kubernetes
• Migrate the entire Data Lake and ETL setup from on-prem cluster to GCP.
• Worked on GCP Data pipeline to configure data loads from SAP HANA to into SQL server.
• Responsible for building scalable distributed data solutions using data proc environment with GCP.
• Scheduling the jobs using Automic workflow.
• Developed and deployed stacks using AWS Cloud Formation Templates (CT) and AWS Terraform.
• Worked on Nifi data pipeline to process large set of data and configured lookups for data validation and integrity
• Performed Complete Migration of Workflows in DPAAS 2.0 to DPAAS 2.1 in Automic

Data Engineer

• Implement AWS Lambdas to drive real-time monitoring dashboard of Kinesis streams.
• Involved in Data Ware house design, data integration and data transformation using Apache Spark and Python.\
• Created/Setup EMR clusters for running data engineering work loads and data scientists.
• Have experience in apache airflow.
• Design and Develop ETL Processes in AWS glue to migrate the data from external sources mostly S3, ORC/parquet /text files into AWS redshift.
• Worked on cloud deployments using maven, docker and Jenkins.
• Created Splunk dashboards for CloudWatch logs and monitored the whole environment using Glass tables and worked on regular alerts.
• Maintained development and production environment through Docker-compose stack deployed on Google Cloud.  
• Developed ETL pipelines in and out of data warehouse using combination of Python and Snowflakes SnowSQL Writing SQL queries against Snowflake.
• Utilized Spark SQL API in PySpark to extract and load data and perform SQL queries.
• Involved in setting up automated jobs and deploying machine learning model using Azure DevOps pipelines.
• Worked on developing, building, and maintaining a highly available, secure, and multi-zone AWS cloud
• Involved in design and deployment of a multitude of Cloud services on AWS stack such as Route53, S3, RDS, Dynamo DB, SNS, SQS, IAM, while focusing on high-availability, fault tolerance, and auto-scaling in AWS CloudFormation.
• Worked on setting-up lambda function along various triggers
• I have used lambda function for infrastructure automation like EC2 automated snapshot creation and daily automated reporting of list of EC2 instance.
• Retrieved data from DBFS into Spark Data Frames, for running predictive analytics on data.
• Used Hive Context which provides a superset of the functionality provided by SQLContext and preferred to write queries using the HiveQL parser to read data from Hive tables.
 
• Modelled Hive partitions extensively for data separation and faster data p

Defined data contracts, and specifications including REST APIs.
• Worked on relational database modelling concepts in SQL, performed query performance tuning.
• Worked on Hive Meta store backup, Partitioning and bucketing techniques in hive to improve the performance and tuning Spark Jobs
• Responsible to build and run resilient data pipelines in production and implemented ETL/ELT to load a multi-terabyte enterprise data warehouse.
• Worked closely with Data science team and understand the requirement clearly and create hive table on HDFS.
• Developed Spark scripts by using python commands as per the requirement.
• Solved performance issues in Spark with understanding of groups, joins and aggregation.
• Scheduling Spark jobs in Hadoop Cluster and Generated detailed design documentation for the source-to-target transformations.
• Experience in using the EMR cluster and various EC2 instance types based on requirements.
• Responsible for loading data from UNIX file systems to HDFS. Installed and configured Hive and written Hive UDFs.
• Worked on integration of lambda with ticketing tools to take automated actions.
• Designed and developed Map Reduce program to analyse & evaluate multiple solutions, considering multiple cost factors across the business as well operational impact on flight historical data.
• Created End-to-end ETL pipeline for data processing for created dashboards to business using PySpark.
• Developing spark programs using python APIs to compare the performance of spark with HIVE and SQL and generated reports monthly and daily basis.
• Developed dataflows and processes for the Data processing using SQL (SparkSQL & Data frames).
• Understood business requirements and prepared design documents, coding, testing and go on live production environment.
• Implemented analytics applications using multiple database technologies, such as relational, multidimensional (OLAP), key-value, document, or graph.
• Built cloud-native applications and supporting technolog

Manoj Gudivaka

#1 Video-Talent Marketplace

Ready to discover your perfect career?