About the job
Job summary
The Office for National Statistics (ONS) is the UK’s largest producer of official statistics, covering a range of key economic, social and demographic topics. These include measuring changes in the value of the UK economy, estimating the size, geographic distribution, and characteristics of the population, and providing indicators of price inflation, employment, earnings, crime, and migration.
Social Surveys Transformation (SST) sits within Surveys Directorate (SD) and is responsible for the transformation of the portfolio of current ONS Social Surveys. Initially focusing on the Transformed Labour Market Survey (TLFS) but will eventually include other surveys within the ONS portfolio of social surveys.
Within SST the processing branch focuses on design, construction and use of processing systems to enable the creation of Reproducible Analytical Pipelines (RAPs) that enable the fast and effective provision of processed survey data to end users. This is all the way from raw survey returns to quality assured micro datasets for analytical use.
Within the branch the technical team is responsible for the continued development of the Configurable Integrated Processing & Assurance System (CIPAS) and associated Management Application for transformed social surveys. This will involve the use of modern open-source tools, such as Python, Pyspark and potentially R, Flask, CSS, Javascript using ONS strategic systems.
Job description
You will be responsible, alongside the Data Processing Integration Lead, for the continued development of CIPAS and associated Management application for transformed social surveys. This will involve the use of modern open-source tools, such as Python, Pyspark, and potentially R, Flask, CSS, Javascript using ONS strategic systems.
The strategic systems currently include the Data Access Platform (DAP) but will over time expand to other cloud based environments such as IDS, GCP and AWS.
The system is being developed to transform raw social survey data into clean, standardised, accurate datasets ready for analytical use. This is to be achieved via the creation of one modularised system, containing standardised methods, that apply rule based, statistical and data science processes to the data.
The system will be developed to be customisable by end users who have little to no coding knowledge to suit each individual pipeline and input dataset. This will be done using a dedicated management application. The result will be one easy to use system with the ability to create multiple RAP, whilst reducing the skills barrier to do so, thereby reducing risk, improving auditability and making processing reproducible.
You will work on the ongoing technical development of CIPAS and associated management application. This will include the transfer of under development statistical and data science-based processes initially focussing on TLFS data but may expand to meet requirements for other surveys in due course.
You will work within a small coding team along with collaborators from other teams, including teams working on shorter term tactical solutions, within the division and across ONS to meet development and operational needs.
You will be required to work in an Agile manner aided via the use of JIRA and Confluence.
Responsibilities
- A senior data engineer designs and leads the implementation of data flows to connect operational systems, data for analytics and business intelligence (BI) systems.
- Recognise opportunities to re-use existing data flows.
- Lead the build of data streaming systems.
- Optimise the code to ensure processes perform optimally.
- Lead work on database management.
- Write PySpark, Python or R code that will systemize the processing of data, based on initial design documents. This includes writing unit / integration tests and managing versions of code with Git.
- Maintain and review documentation for statistical and data processing functions, as well as data processing pipelines.
Person specification
Essential Skills Criteria:
- Programming and build (data engineering): You know how to use agreed standards and tools to design, code, test, correct and document moderate-to-complex programs and scripts from agreed specifications and subsequent iterations. You can collaborate with others to review specifications where appropriate. (Lead Essential Skills Criteria)
- Data Development Process: You can design, build and test data products that are complex or large-scale. You know how to build teams to complete data integration services.
- Data integration design : You know how to select and implement the appropriate technologies to deliver resilient, scalable and future-proofed data solutions.
- Problem Resolution: You know how to respond to problems in databases, data processes, data products and services as they occur. You can initiate actions, monitor services and identify trends to resolve problems. You can determine the appropriate remedy and assist with implementation of it as well as preventative measures.
- Technical understanding (data engineering) : You understand core technical concepts related to the role and can apply them with guidance.
- Testing: You can review requirements, specifications and define test conditions. You can identify issues and risks associated with work while being able to analyse and report test activities and results.
Behaviours
We'll assess you against these behaviours during the selection process:
- Making Effective Decisions
- Working Together
- Communicating and Influencing
Technical skills
We'll assess you against these technical skills during the selection process:
- Programming and Build
- Data Integration Design
Benefits
Alongside your salary of £41,364, Office for National Statistics contributes £11,983 towards you being a member of the Civil Service Defined Benefit Pension scheme. Find out what benefits a Civil Service Pension provides.
The Office for National Statistics is part of the Civil Service, and as such we share a number of key benefits with other departments, whilst also having our own unique offerings to support our 5400 valued colleagues across the business.
This role is part of the cross-government Government Digital and Data (formerly DDaT) profession framework. As a role within Government Digital and Data (formerly DDaT) at the ONS, we also offer benefits such as:
• Protected Learning Time to spend on your personal development and side-projects.
• A supportive and active Community of Practice which you will be expected to contribute to, helping ensure you and your colleagues get the training, development and opportunities you need to progress your careers.
ONS are committed to flexible ways of working that support a healthy work-life balance. ONS has already considered how this job could be right sized for applicants working flexibly and we are happy to explore options with you about working part time, in a job share or flexibly, in line with our hybrid working policies.
Whether you are hearing about us for the first time or already know a bit about our organisation, we hope that the benefits pack attached (bottom of page) will give you a great insight into the benefits and facilities available to our colleagues and our fantastic working culture.
