Big Data Engineer

Derisk360


Date: 2 weeks ago
City: London
Contract type: Full time
Job Title Big Data Engineer

Location - Bangalore , Chennai , Gurgaon

Job Summary

Experience required - 5 - 8yrs

We are seeking Big Data Engineers to join our offshore team. The ideal candidates will have extensive experience in PySpark, real-time data processing, and Kafka. They should be well-versed in designing and implementing ETL pipelines and working with streaming data frameworks in cloud-based environments.

Key Responsibilities

  • Develop, optimize, and maintain streaming data pipelines using PySpark and Apache Kafka.
  • Design and implement scalable ETL processes for real-time and batch data processing.
  • Work with Amazon EMR, Apache Spark, Apache NIFI, or similar frameworks to build near real-time data pipelines.
  • Develop data solutions and frameworks to handle high-volume, high-velocity data streams.
  • Implement data storage solutions for structured and unstructured data, ensuring efficiency and reliability.
  • Write clean, maintainable, and well-documented code using Python, Groovy, or Java.
  • Optimize data structures and schemas for data processing and retrieval efficiency.
  • Collaborate with cross-functional teams to define, design, and implement data-driven solutions.
  • Troubleshoot and resolve performance bottlenecks and data quality issues.
  • Stay updated with the latest technologies and best practices in big data and streaming architectures.

Required Qualifications & Skills

  • Proficiency in PySpark and strong understanding of distributed computing principles.
  • Hands-on experience with Apache Kafka (including kSQL, Mirror Maker, or similar tools) for real-time data streaming.
  • Strong programming skills in at least one of the following: Python, Groovy, or Java.
  • Good understanding of data structures, ETL design, and data storage solutions.
  • Experience in working with Amazon EMR, Apache Spark, Apache NIFI, or similar streaming data frameworks.
  • Ability to design and implement scalable and high-performance data pipelines.
  • Experience with cloud-based big data solutions (AWS, GCP, or Azure) is a plus.
  • Strong problem-solving and analytical skills.
  • Excellent communication and collaboration abilities.

Preferred Qualifications

  • Experience with SQL and NoSQL databases.
  • Exposure to containerization technologies such as Docker and Kubernetes.
  • Familiarity with CI/CD pipelines for data engineering workflows.
  • Understanding of data governance, security, and compliance best practices.

Why Join Us?

  • Work on cutting-edge real-time streaming and big data technologies.
  • Collaborate with an expert team in a fast-paced, dynamic environment.
  • Competitive compensation and opportunities for professional growth.
  • Gain experience in modern cloud-based data architectures.

If you are passionate about data engineering, big data, and real-time analytics, we invite you to apply and be a part of our innovative team!

Please apply here or share your CV to [email protected]

How to apply

To apply for this job you need to authorize on our website. If you don't have an account yet, please register.

Post a resume

Similar jobs

Graduate Teaching Assistant

TipTopJob, London
5 days ago
Graduate Teaching Assistant Full:Time ASAP Start GBP82.14:GBP100 Per Day Enfield, North LondonReeson Education is looking for an enthusiastic Graduate Teaching Assistant to work in a welcoming primary school in Enfield, starting immediately. If you are a recent graduate eager to step into the world of education, this Graduate Teaching Assistant role will provide invaluable classroom experience.As a Graduate Teaching Assistant,...

Workday Procure to Pay and Strategic Sourcing Consultant (Workday)

Accenture UK & Ireland, London
5 days ago
Accenture is a leading global professional services company, providing a broad range of services in strategy and consulting, interactive, technology and operations, with digital capabilities across all of these services. With our thought leadership and culture of innovation, we apply industry expertise, diverse skills and next-generation technology to each business challenge. We serve clients in more than 120 countries with...

Principal Business Consultant - R&D - Clinical (Remote)

Veeva Systems, London
5 days ago
Veeva Systems is a mission-driven organization and pioneer in industry cloud, helping life sciences companies bring therapies to patients faster. As one of the fastest-growing SaaS companies in history, we surpassed $2B in revenue in our last fiscal year with extensive growth potential ahead.At the heart of Veeva are our values: Do the Right Thing, Customer Success, Employee Success, and...