International Business Machines Corporation
Senior Data Engineer - Google Platforms
Hybrid · San Francisco, New York, New York, California
About the role
Employer: (0147) International Business Machines Corporation
As a Data Engineer specializing in Google's data platforms, you will design, build, and maintain data engineering solutions on Google's Cloud ecosystem. You will utilize various Google services to develop batch and real-time data pipelines, manage data migration, and design data layers.
Your primary responsibilities will include:
• Design Data Pipelines: Design and build data pipelines using Google DataProc, DataFlow, PubSub, BigQuery, and other Google services to meet business requirements.
• Develop Data Engineering Solutions: Utilize Google Cloud Storage, BigTable, BigQuery DataProc with Spark and Hadoop, and Google DataFlow with Apache Beam or Python to develop and manage data engineering solutions.
• Manage Data Platforms: Schedule and manage the data platform using Google Cloud Scheduler and Cloud Composer (Airflow) to ensure seamless data pipeline operations.
• Ensure Data Quality: Develop and manage batch and real-time data pipelines for Data Warehouse and Datalake to ensure high-quality data delivery.
• Optimize Data Solutions: Collaborate with stakeholders to optimize data solutions and ensure alignment with business objectives.
This position can be performed from anywhere in the United States of America.