Data Engineer
Description
You will have an important role and be an integral part of a small team that designs, builds, and maintains our Evaluation Cloud. You’ll focus on mission-critical applications, data and data products, and integrations that are the engine of our business. Our systems deal with large amounts of data, data science, analytics, machine learning, and highly interactive and engaging features that ensure our customers can successfully evaluate and execute innovation.
Responsibilities
- Work as part of an implementation team from concept to operations, providing deep technical subject matter expertise for successful deployment.
- Ingest and integrate massive datasets from multiple data sources, while designing and developing solutions for data integration, data modelling, and data inference and insights.
- Design, monitor, and improve development, test, and production infrastructure and automation for data pipelines and data stores.
- Troubleshoot and performance-tuned data pipelines and processes for data ingestion, merging, and integration across multiple technologies and architectures including ETL, ELT, API, and SQL.
- Test and compare competing solutions and provide an informed POV on the best solutions for data ingestion, transformation, storage, retrieval, and insights.
- Work well within the quality and code standards, and engineering practices, in place, and maintained by the team
Qualifications
- 3+ years of data engineering experience required
- 3+ years of Google Cloud Platform (GCP) experience desired. Equivalent cloud platform experience considered.
- 2 + years of coding in Python
- 2+ years of experience working with NoSQL and SQL databases
- Solid understanding of ETL/ELT concepts, data architectures, data modelling, data manipulation languages and techniques, and data query languages and techniques.
- Experience working in GCP-based Big Data deployments (Batch/Real-Time) leveraging Big Query, Big Table, Google Cloud Storage, PubSub, Data Fusion, Dataflow, Dataproc, Airflow, etc.
- In-depth understanding of Google's data product technology and underlying data architectures. Equivalent cloud platform experience considered.
- Google Cloud Platform (Google Cloud Platform) services: BigQuery, DataFlow, Pub/Sub, BigTable, Data Fusion, DataProc, Cloud Compose, Cloud SQL, Compute Engine, Cloud Functions, and App Engine.
related Jobs
Technologies that suit both small and large scale plantations
Machine Learning Engineer
We are looking for a Machine Learning (ML) Engineer to help us create artificial intelligence products. Machine Learning Engineer responsibilities...
Senior Software Engineer
Develop systems by studying operations; designing, developing and implementing software solutions. Lead a team of developers responsible for...
Senior Software Engineer C++
We're seeking a Senior C++ Developer to build backend solutions for our AI and IoT platforms. You should have experience in creating...