Job Duties: Optimize data pipelines for performance and cost for large scale data lakes. Use Google Cloud Platform native tools including Bigquery, cloud functions, Pubsub, CI/CD, automation, GitHub, Terraform for infrastructure development and deployment. Code in python and call GCP’s rest API’s for integrating data. Build data pipelines in airflow as a service (composer) using various operators. Develop data ingestion/ETL pipeline to load data from data sources identified on daily basis using Data Flow. Design Terraforms and deploy in cloud deployment manager to spin up resources including cloud virtual networks, compute engines in public and private subnets along with Auto Scaler in Google Cloud Platform. Responsible for automating end-to-end data pipeline with metadata, data quality checks and audit to follow the standard and send the alerts to Airflow when any data quality check fails. Develop data orchestrate pipelines using Apache Airflow to ingest incremental data into the data lake. Automate and orchestrate the data pipeline using Airflow; create airflow DAG for data pipeline using various airflow operators and hooks. Create dataproc clusters for spark jobs and continuously optimize data pipelines and BigQuery queries for cost-effectiveness and performance. Define fine-grained access control for GCP resources including Compute Engine instances, Cloud Storage buckets and Bigquery datasets. Monitor and review user access and permissions through audit logs and IAM activity logs. Handle memory issues in dealing with data extracts using Python. Create python report for daily opened ticket details for different teams across portfolio. Build programs using Python and Apache Airflow to execute it in cloud dataflow and to run data validation jobs between raw source file and BIG Query tables. Provide analysis and resolution on the Incidents and users request raised in ServiceNow.  

Work Location: Various unanticipated work locations throughout the United States; relocation may be required. Must be willing to relocate.

Minimum Requirements:

Education: Bachelor – Computer Science or Electronic Engineering (will accept foreign education equivalent)

Experience: Five (5) years