Senior Splunk Engineer - Infrastructure Operations
GovCIO · United States
- Hybrid
- Full-time
- $145,000 / year
- United States
Job highlights
- Engineer and operate Splunk environments.
- Monitor and optimize system health and performance.
- Troubleshoot data ingestion and parsing issues.
- Manage user access and provide end-user support.
- Collaborate with engineering on complex issues.
About the role
Senior Splunk Engineer - Infrastructure Operations
GovCIO is currently hiring for a Senior Splunk Engineer - Infrastructure Operations to support the Administrative Office of the US Courts NLS project. The NLS ingests an average of 18-20TB of logging data daily across 60 indexers distributed in 2 data centers. This position is fully remote and located within the United States.
Responsibilities
- Design, implement, and operate Splunk Core, Enterprise Security, IT Service Intelligence (ITSI), Phantom (SOAR), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environments.
- Monitor Splunk health using the Monitoring Console (DMC), including indexer, search head, and cluster master status.
- Track indexing rates, license usage, queue health, and search concurrency to identify performance or ingestion issues.
- Monitor CPU, memory, and disk utilization across all Splunk components for optimal resource usage.
- Respond promptly to health alerts, DMC warnings, or observed anomalies.
- Investigate and resolve common user-reported issues such as access problems, failed searches, or non-triggering alerts.
- Troubleshoot data ingestion, parsing, and indexing issues across Universal Forwarders, Heavy Forwarders, and HEC endpoints.
- Investigate missing/duplicate logs, timestamp errors, or sourcetype misassignments, escalating complex issues.
- Validate new data source onboardings, confirming sourcetype assignment, timestamp accuracy, and field extraction integrity.
- Support data source owners with forwarder deployment, syslog setup, and connectivity troubleshooting.
- Maintain data flow visibility from source to indexer to confirm data completeness and performance.
- Rotate and update credentials, API keys, or tokens used in data inputs, integrations, alerts, and scheduled searches.
- Manage RBAC user and role mappings, handling access requests, entitlement reviews, and permission troubleshooting.
- Provide end-user assistance with SPL searches, reports, alerts, and dashboards, including query optimization.
- Maintain and update knowledge base articles, SOPs, and FAQs for repeatable issues.
- Log and escalate platform or parsing issues to the Engineering team with evidence.
- Open and manage Splunk Support cases for platform bugs, license problems, or critical system faults.
- Monitor and manage ITSI service health, including KPIs, correlation searches, NEAP policies, and summary index latency.
- Troubleshoot ITSI-related issues such as broken KPIs, delayed episodes, or missing notable events.
- Perform capacity management by monitoring index growth, bucket rotation, and frozen data retention.
- Conduct periodic system maintenance, including orphaned object cleanup and knowledge object review.
- Verify and maintain compliance with data governance and retention policies.
- Participate in DR testing and validation to ensure data recovery and HA configurations.
- Document incidents, RCA findings, and preventive actions.
- Collaborate closely with the Engineering team for escalations, investigations, and deployment verifications.
Qualifications
- Bachelor's with 10 years experience (or commensurate experience) OR Master's Degree or higher (in a related discipline) with 7 years experience.
Required Skills And Experience
- Expert skills in Enterprise Security, ITSI, SOAR, and the Splunk product line.
- Able to design, implement, and operate the Splunk Core, Enterprise Security, IT Service Intelligence (i.e., ITSI), Phantom (Security Orchestration, Automation, and Response (SOAR)), Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering environment.
Clearance Required
- Must be able to obtain and maintain AOUSC Public Trust.
Key skills/competency
- Splunk Engineering
- Infrastructure Operations
- Enterprise Security (ES)
- IT Service Intelligence (ITSI)
- Security Orchestration Automation and Response (SOAR)
- Splunk Cloud
- Splunk On-Call
- Multi-Site Index Clustering
- Data Ingestion
- Log Management
Skills & topics
- Splunk Engineer
- Splunk
- Splunk Core
- Enterprise Security
- ITSI
- SOAR
- Splunk Cloud
- Splunk On-Call
- Infrastructure Operations
- Data Ingestion
- Log Management
- Security
- Remote
How to get hired
- Customize your resume: Highlight your expertise in Splunk Core, Enterprise Security, ITSI, and SOAR, tailoring it to the job description's responsibilities.
- Showcase relevant experience: Detail your experience in designing, implementing, and operating complex Splunk environments, including multi-site index clustering and data ingestion.
- Prepare for technical questions: Be ready to discuss troubleshooting data ingestion, parsing issues, and Splunk health monitoring using the Monitoring Console.
- Understand the company culture: Research GovCIO's mission and values to align your responses with their goals, especially regarding government projects and security clearances.
- Highlight clearance readiness: Emphasize your ability to obtain and maintain an AOUSC Public Trust clearance, a critical requirement for this role.
Technical preparation
Master Splunk Core administration and configuration.,Practice designing Splunk Enterprise Security deployments.,Familiarize with ITSI service intelligence capabilities.,Prepare for SOAR and Splunk Cloud scenarios.
Behavioral questions
Describe a complex Splunk issue you resolved.,How do you prioritize multiple urgent requests?,How do you collaborate with engineering teams?,How do you stay updated on Splunk best practices?
Frequently asked questions
- What is the primary focus of the Senior Splunk Engineer role at GovCIO?
- The Senior Splunk Engineer at GovCIO will focus on designing, implementing, and operating various Splunk products, including Splunk Core, Enterprise Security, ITSI, and Phantom, to support the Administrative Office of the US Courts NLS project. This involves ensuring optimal performance, managing data ingestion, and troubleshooting issues within a large-scale logging environment.
- What Splunk modules require expert-level skills for this position?
- Expert-level skills are required for Splunk Core, Enterprise Security (ES), IT Service Intelligence (ITSI), and Phantom (SOAR). Proficiency in managing Splunk Cloud, Splunk On-Call, and Multi-Site Index Clustering is also essential for this Senior Splunk Engineer role.
- Is this a remote position, and what are the location requirements?
- Yes, this is a fully remote position located within the United States. Candidates must be able to obtain and maintain an AOUSC Public Trust clearance.
- What kind of data volume is the Senior Splunk Engineer expected to manage?
- The role involves managing a significant data volume, as the Administrative Office of the US Courts NLS project ingests an average of 18-20TB of logging data daily across 60 indexers.
- What are the typical responsibilities for troubleshooting data ingestion with this Senior Splunk Engineer role?
- Troubleshooting responsibilities include identifying and resolving issues with data ingestion, parsing, and indexing across Universal Forwarders, Heavy Forwarders, and HEC endpoints. This also involves investigating missing or duplicate logs and timestamp errors.
- How does this Senior Splunk Engineer role contribute to system health and performance?
- The engineer actively monitors Splunk health through the Monitoring Console, tracks indexing rates, license usage, and resource utilization (CPU, memory, disk) to identify and address performance bottlenecks or ingestion issues proactively.
- What is the required educational background and experience for the Senior Splunk Engineer position?
- A Bachelor's degree with 10 years of experience, or a Master's degree or higher with 7 years of experience in a related discipline, is required for this Senior Splunk Engineer role.
- What kind of user support can be expected from the Senior Splunk Engineer?