Big Data Engineer (Cloudera, AWS)

Job Title:

Big Data Engineer (Cloudera, AWS)

Location:

Primarily Remote/ HYB (Monthly Onsite Meetings in Reston, VA and/or Washington, DC)
Employment Type:

Contract

GC/USC

8+ years of experience in Cloudera and AWS Cloud environments with strong expertise in Kafka, NiFi, HBase, Solr, MongoDB, automation scripting, and performance optimization for large-scale enterprise data systems.

Onsite interview needed
Overview:

Big Data Engineer

with strong

Cloudera and AWS

expertise to support a high-availability

Production Support Environment . This role involves managing big data infrastructure, optimizing Cloudera components, and ensuring seamless data migrations and automation. Occasional weekend releases and late-night troubleshooting may be required.
Key Responsibilities:
Maintain and optimize Cloudera environments (CDP, CDSW, Hive, Spark, NiFi).
Perform critical data migrations from

CDH to CDP .
Design and implement

Big Data pipelines

and automated data flows using

Python/R

and

NiFi .
Automate project lifecycle processes and perform incremental upgrades.
Support analytics, ML, and data science use cases across infrastructure and cluster deployments.
Manage

data governance, quality, testing, and documentation .
Use

YARN

for resource allocation and job scheduling.
Manage workflows via

Oozie

and

Hue .
Enforce

security policies

across Hadoop clusters using

Apache Ranger .
Configure and manage

Cloudera Data Science Workbench

via Cloudera Manager.
Troubleshoot

Kerberos ,

TLS/SSL , and data science workload issues.
Provide end-to-end

Big Data / Hadoop support

throughout development and production.
Optimize performance, replication, and connectivity; implement monitoring and automation scripts.
Support and manage

Kafka ,

NiFi ,

HBase ,

Solr , and

Flume

integrations.
Required Skills & Experience:
8+ years

of experience supporting

Cloudera applications

in

AWS Cloud

environments.
Expertise with

MongoDB ,

Kafka ,

NiFi ,

Flume ,

HBase , and

Solr .
Strong knowledge of

Cloud Disaster Recovery

strategies.
Kafka Administration & Troubleshooting

(brokers, Connect, Zookeeper, MQ integration).
Experience with

Kafka Streams API ,

KStreams , and

KTables .
NiFi Administration

(flow management, registry server, controllers, service management).
Integration experience:

NiFi to Kafka/HBase/Solr ,

Flume to Kafka/NiFi/MQ .
Strong skills in

database management ,

query optimization , and

automation scripting .
Hands-on experience supporting

large-scale enterprise data environments .
Preferred Skills:
Experience with

DB2 UDB ,

DB2 mainframe , or

MongoDB .
Exposure to

streaming technologies

such as

Kafka ,

Spark , and

Kudu .
Familiarity with

AI/ML workloads

and

GenAI testing

environments.