Data Analysis
We are awash in a sea of distributed data - collected by sensors, from web-based data streams, or derived/inferred from other sources. Making effective use of these data is not always straight forward. The objective is to turn data... into information. Actionable information.
Joint measurements, simultaneous readings, context-aware sensing. Often the volume of data to be analysed can be massive. Sometimes the rate of data is large and the data stream needs to be processed in real time and without lag.
Data Wrangling
Incoming data can be long or wide, unstructured and unformatted. Applying analytical methods to such data will require a homogenisation and massage of data known as wrangling. This can be simple, like making timestamps a consistent format. Or it can be quite involved, involving filtering data, calculating or inferring additional information from partial or combined data sources.
The goal in all cases is to turn the data into a format amenable for further analysis with the tools being used. This could be internal vectors or matrices, structured and validated JSON data, so on. This always depends on the specifics of the project, so a flexible yet rigorous approach is required.
Data Visualisation
Data visualization is the representation of data through use of common graphics. The intention is to convey information in a form that is easily interpreted and digestible. Complex data presented in a clear and understandable way is a key requirement for data-driven insights.
- Visual displays of complex information are useful to effectively convey conclusions to non-technical stakeholders.
- Data-scientists can use visualisations to discover and explain patterns and trends and extract insight.
Effective visualisation can illustrate hidden relationships in broad or unstructured datasets that would otherwise be lost in noise.
Images generated by D3.js | AI/Dalle "Essence of data analysis"
Data Science Services
PIATRA can help turn data into actionable information. We have the tools and experience to digest and process massive datasets - and to present the information in accessible forms - be it for humans to understand or for other processes to use in automation.
Our senior consultants have a strong and demonstrable track-record in the fields of laser-imaging, image processing, diagnostics, measurement technology, statistical analysis, peer-review and publication.
We can pre-process or filter data streams and complex multi-scalar datasets. We can write script tools to streamline often time-consuming processes.
We can implement (and automate) quantitative or qualitative data analysis. Finally, we can produce meaningful visualisations that allow distilled insights and evidence supported conclusions for action.
- Automate the data handling pipeline
- Scripts for pre- and post-processing
- Handle exceptions and filter erroneous data
- Derive context aware information
- Present outcomes in human-readable form
- Generate reports and graphs
- Visualise results
Important Information
Of course. Most things can be automated. The question is whether they should be.
For data analysis, automation is extremely helpful to extract known derived information. That is, when it is known exactly what insights are being sought from the data set. Especially useful when you are looking to generate up-to-date reports based on rapidly changing information, for example.
On the other hand, if you are looking to explore a new data set to extract insights and drive discovery then it may be better to, at most, automate the collection and preprocessing of the data but to have an expert manually explore the data looking for insights not currently known.
Python has some fantastic numerical analysis libraries - Numpy, Scipy. This is in addition to the great array of tools that facilitate the serialisation/deserialisation of raw information.
R is a very popular language finding a good foothold in data-science in recent years. It is primarily related to big data, data mining, and deep learning. R handles both structured and unstructured data elegantly.
MATLAB is also a popular choice for analysis of information, with many statistical tools available for advanced use cases.
The D3 javascript library is very helpful visualisation tool, particularly for structured JSON data.
For sure.
We can write utilities to automatically extract data from API's, web-sockets and even Excel spreadsheets. The data can be refreshed in real-time or by request to an external source.
Basically, if we can access it, derive it or measure it ourselves, it is available for analysis and ready to build insight into your complex problem.